Web Infrastructure.pptVIP

Web Infrastructure /~course/cs501/2008 Hongfei Yan School of EECS, Peking University 3/3/2008 Outline Introduction Google Architecture MapReduce Hadoop References Infrastructure Definition Infrastructure is generally structural elements that provide the framework supporting an entire structure. Allow for services without themselves being part of the production process. e.g. roads allows the transport of raw materials and finished products. Web Infrastructure Definition Cloud computing is a popular phrase that is shorthand for applications that were developed to be rich Internet applications that run on the Internet (or cloud). Google and IBM join CMU, Stanford, Berkeley, California, Maryland, Washingtong will be involved Data Intensive SuperComputing Knowledge Machine Like? operations on the database that can be simple queries or can require complex computations accessing large amounts of data. Internetics Bridging computer science and all application areas Cloud Computing 利用互联网连接的数据中心和服务器进行高效计算、信息存取以及编写程序的系统。 提供存储以及方便操作数据的编程模式， 如MapReduce[Dean and Ghemawat,2004]， Bigtable: A Distributed Storage System for Structured Data 不同于以往的超级计算机， 它除了提供大规模分布式计算外， 还以组织和管理数据为核心工作之一，它获取并且维护持续变化的数据集。 最近Google和Microsft的研究论文[Dean and Ghemawat,2004], [Isard, et al.,2007]给出了利用1800个处理器存取10TB数据的云计算模式。 Motivations Web Search Tianwang Search Engine Web Archive Web Infomall Chinese Digital Assets Library Web Evaluation Chinese Web Information Retrieval Course Laboratory For students or researchers experiment,testing,research For teachers Incorporate ideas behind Web Infrastructure into the course Motivation 200+ processors 200+ terabyte database 1010 total clock cycles 0.1 second response time 5￠ average advertising revenue Google’s Computing Infrastructure System ~ 3 million processors in clusters of ~2000 processors each Commodity parts x86 processors, IDE disks, Ethernet communications Gain reliability through redundancy software management Partitioned workload Data: Web pages, in