就业垂直搜索引擎中网络机器人的设计与实现-软件工程专业论文.docxVIP

摘要 随着 Web 技术的飞速发展，网络信息已经呈现出了大爆炸的趋势，通用搜索 引擎在网页捕获，内容索引等方面面临着越来越大的挑战。通用搜索引擎检索结 果包罗万象，与特定领域用户的精确检索需求的矛盾越来越突出。垂直搜索引擎 是专门面向特定领域的搜索引擎，它能够在一定程度上解决这种矛盾。 网络机器人是垂直搜索引擎重要组成部分，是垂直搜索引擎的数据来源，它 捕获网页的数量和质量直接决定了搜索引擎检索结果的召回率和准确率。 文中说明了网络机器人的基本概念，在搜素引擎中所处的位置，发展历史和 当前现状。具体分析了通用搜索引擎网络机器人和垂直搜索引擎网络机器人之间 的区别。垂直搜索引擎所使用的网络机器人一般被称为聚焦网络机器人或者聚焦 爬虫。为处理“主题孤岛”问题，在本文中，结合就业搜索引擎开发的具体情况， 提出了基于 URL 规则，实时监控站点 HUB 页面 URL 列表的可扩展自定制网络机器 人。 文中就网络机器人设计中的一些技术细节进行了讨论，提出了一些提高网络 机器人效率，节省计算机和网络资源的实现方法。 本文创新点包括，设计并且实现了聚焦网络机器人的三层结构；设计并且实 现了聚焦网络机器人的资源优化策略；提出并且实现了列表监控解决主题孤岛的 方法。 关键词： 垂直搜索引擎；网络机器人；主题漫游；并行可扩展 Abstract The rapid development of Web technology, network information has shown a trend of the big bang. For general search engine, many fields such as web page capture, content indexing and so on are facing with increasing challenges. The contradiction of the all-inclusion of the general search engine results and the precise needs of the specific areas of the users is more and more prominent. Vertical search engine which is a professional search engine can solve the conflict in some way. Network Robot is an important part of vertical search engines. It is also the data sources of the search engine. The quantity and quality of the page captured determines the recall and accuracy of the results of the search engine directly. Many as the concept, the status in the whole search engine, the history and the current status are introduced in the article. Differences of network robots in the general search engine and in the vertical search engine are analyzed specifically. The network robot used by a vertical search engine is generally called as a Focused Network Robot. To solve the problem known as ―theme island‖ , contacting the actual situation of our Job Search Engine, we proposed a customizable Network Robot, which bases on the URL-rules and can monitor the URL list of the Hub page real-time. Some technical details in the design of the network robots are discussed and many methods which can improve the efficiency or s