分布式聚焦网络爬虫系统的设计与实现-计算机应用技术专业论文.docx

浙江大学硕士学位论文 Abstract Abstract More and more people need personalized search service .The appearance of vertical search engine satisfied this kind of requirement.Vertical search engine usual1y need focused crawler，which can crawl the web page by some subjects and rules. Some pωple even want to collect and format special infOIτnation from intemet by focused crawler. This paper introduces a new distributed focused crawler，which has high performance and scalable framework and useful management interface. It also can extract 也e specifically text field from web page into formatted data. Chapter one introduces the research background of focused crawler，include difficulties and the solving strategy. Chapter two introduces the overall design of system framework， which consists of one management server and several crawler servers. Chapter three and Chapter four introduce the implementation of management S町ver and several crawler servers. Chapter five make a system test to 由is sys阳:m. Chapter six summarizes 也is paper and put forward some feasible improvement of the system. Keywords: Distributed System，Focused Crawler，Extract Web Field 11 浙江大学硕士学位论文 目录 目录 摘要... ..? .? ? ... ? ? ? ? ? ? .. .. .. .. 1 Abstract. . .. ... . ... ii 第 1 章绪论...? .. ? ? .. ? .. ? .. ? .. .. 1 1.1 课题背景……… 1 1.2 研究现状………………………………………………...…川 2 1.2.1 聚焦网络爬虫………………… 2 1工2 聚焦网络爬虫的难点………………………………………… ..4 1.2.3聚焦网络爬虫的关键策略… 5 1.3 新的趋势…………………………………… 11 1.4 本文的主要工作及组织结构 11 第 2j袁系统框架设计. ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 13 2.1 需求分析...……………………………… 13 2.2 名词解释………………... 13 2.3系统框架设计……………………………………………… 14 2.3.1 传统的系统框架..……… 14 2.3.2 改进的系统框架………………………………………………..… 15 2.3.3系统软硬件配置川… 17 理服务器的设计……………………………………………川 18 取服务器的设计… 19 2.6 本章小结……………………………………… 21 第 3 章管理服务器的实现.. ? ? ? ? .. .. .. ? .. ? .. .. .. .. .. ? ? .