Web数据模型以及获取、存储方法研究.doc

摘　要 Web上信息就像一条河流，从我们身边不停流过。已经有很多人认识到这些信息的价值，从而展开了对Web信息多方面的研究。本文阐述的内容就是基于这些研究，并希望能够对他人的相关工作带来方便。文章围绕的中心是Web数据的高效获取与存储，为此也专门研究了信息在Web上大量重复出现的现象和原因。 本文的主要内容包括： 有关Web信息状况的一些统计数据，结合一些Web的基本概念，这些数据让读者对Web宏观上能有一些具体的认识。这是理解本文其他部分的基础， 提出了同义主机的概念。在Web上有很多不同的域名代表着相同的主机，负责获取网页的系统如果不知道这种信息，就会重复的获取这台主机上的网页。这样导致网络资源和机器资源的浪费，并且对相应的Web服务器也造成额外负担，本文通过分析IP地址与域名的关系，总结出哪些主机名有同义关系，从而避免了网页的重复获取；另外，对于消除Web上存在的重复或相似的网页，本文提出了多种算法，一一进行评测，并选择最好的用于实际的网页消重中。 本文还具体的讨论了Web数据获取系统的设计目标，并给出了计算系统效率的方法。然后参照这些设计目标，比较了两种具体的收集系统结构。 同时，论文还基于Web InfoMall的目标，给出了一种海量网页存储系统的设计方案以及实现的种种考虑。 关键词：Web，网页搜集系统，同义主机，重复网页，网上信息博物馆 Abstract Web is like a river of information, flowing by us endlessly. Now people began to appreciate its value, and research it. This thesis is based on such researches. It focuses how to collect mass web pages and how to store them, as well as some characteristics of Web involving getting and storing data. The following issues are addressed: Some statistic information and basic concepts about Web. They will be useful for readers in understanding following sections. It is a phenomena that many physical hosts (servers) have more than one names representing them on the web. If a Web crawler is not aware of such information, the pages on the same host will be retrieved multiple times. As a result, crawlers’ working is wasted, more bandwidth is consumed, and we can even overload the web servers. This thesis analyzes the relation between IP addresses and domain names and the result gives a crawler indication which host names actually represent the same host, so crawling the same page multiple times can be avoided. To get rid of replica or similar pages obtained from the Web, this thesis proposes a few methods, and they are evaluated carefully. Web data retrieving system, so called crawler is discussed in more detail. Some basic design guidelines and performance metrics is presented. We compare two structures of crawler according these guidelines. At the end of this thesis, Web Infomall, a re