rss、信息聚合、搜索引擎、数据采集.docVIP

RSS、信息聚合、搜索引擎、数据采集 ???? 摘　　要随着网络技术的巨大发展以及互联网使用者的不断增加，互联网己经成为主流的信息发布媒介之一。web2.0的到来，给互联网带来了新的机遇和挑战。如何快速、准确地在网络中找到所需信息，并且以更加人性化的方式为用户提供服务成为网络技术研究的热点。信息聚合和信息挖掘为解决互联网信息过载问题提供了技术支持。RSS(Really Simple Syndication)技术是近几年发展起来的在互联网上被广泛采用的内容包装和投递协议，是在线内容共享的简易方式，各种类别的信息通过RSS发送，被广泛应用于博客、新闻信息、个性化搜索、电子商务等各个领域。本文以一个实际的RSS信息聚合服务项目为背景，通过对RSS协议和Web信息采集系统的研究，阐述了一种基于RSS的并行的信息采集系统，并着重探讨该系统与搜索引擎中Web信息采集系统的不同之处，及其基于自身特点的设计和实现策略。在两年的使用过程中，系统能够正确处理RSS的信息，在并行的情况下，可以对数百万级的RSS Feeds进行及时和有效的采集。具体而言，主要的研究工作如下：首先，研究了RSS协议规范和报文结构，以及搜索引擎中Web信息采集系统的架构，特点和难点；其次，研究了基于RSS的信息采集系统的系统特点和设计目标，并进行了系统总体设计；再次，研究了RSS Feeds的地址识别、编码识别和内容抽取，并着重研究了RSS的文章更新算法和文章去重算法等；最后，对系统进行测试和评估，分析系统的性能和瓶颈。 关键词：RSS、信息聚合、搜索引擎、数据采集?Research and Application of Web Crawler System Based on RSSAbstractWith the tremendous development of network technology and the ever-increasing Internet users, the Internet has become one of the mainstream information by the media release. The arrival of web2.0 has brought new opportunities and challenges to the Internet. How to find the information required quickly and accurately in the network, and provide services for users in a more humane manner becomes the focus of research of Internet.Information polymerization and Information Mining provides a way to solve the problem of information overload. RSS technology is developed in recent years on the Internet, which is widely used in packaging and content delivery. It is a simple way to realize online content sharing. Various types of information are transmitted under RSS. It is widely used in weblogs, news information, personalized search, e-commerce and other fields.The present text, on the basis of RSS information aggregation service projects, discloses a parallel web crawler system based on RSS and mainly focused on the difference between this system and the crawler system of Search Engine as well as the design and implementation strategy based on its own characteristics. All of those are based on the research of RSS and web crawler system.