A New Data Storage and Service Model of China Web.pdfVIP

A New Data Storage and Service Model of China Web InfoMall1 Hongfei Yan, Lianen Huang, Chong Chen and Zhengmao Xie Computer Networks and Distributed Systems Laboratory, Department of Computer Science and Technology, Peking University, P.R. China {yhf, hle,cc,xzm}@ Abstract The Web consists of enormous pages which is easier vanishing than traditional media such as newspaper, journals. To preserve the web resources, we began the China Web archiving project, named Web InfoMall, from 2001. The paper describes the data storage and service model of Web InfoMall 2.0 to meet the goals of collecting the stuff broadly, storing them perennially, and locating requests efficiently. Currently the Web InfoMall holds 0.7 billion pages (10.6 terabyte) together with 5 terabyte of digital web resources other than web pages, having the ability of collecting more than 1 million pages per day, a storage capacity to hold more than 10 billion pages (about 150 terabyte), and a scheme to manage large numbers of pages. Keywords: web resources, Web InfoMall, distributed crawling, Tianwang storage format, service model Introduction The Internet was introduced to China in 1994, and the first web sites emerged from 1995. From then on, the China Web got an unprecedented development. At the end of 2003, the registered sites amounted to 595,550 with .cn domain suffix [CNNIC,2004]. During the nine years, the China Web enjoys exponential growth, with the same rate to the whole Web. Tianwang[Tianwang,2004] crawled more than 60 million web pages up to the July of 2002, and 105 million pages in January of 2003. According to a quantitative ana