主题搜索引擎设计与实现.doc

摘要 随着网络信息资源数量的急速增长，通用搜索引擎已经很难满足人们对专业化 信息检索服务的要求。主题搜索引擎只搜索与特定主题相关的 Web 资源，能更好地 满足用户对获取信息快速、准确的性能要求。实现主题搜索引擎，核心是设计一个 面向主题的网络爬虫，改进网络爬虫中对页面的过滤方法和网络爬行的策略，将有 助于提高主题搜索引擎的性能。 经过对现有的各种主题相关度判断方法的分析，以基于语义判断的方法为基础， 加入链接标签和元数据等信息作为判断因素，提出了综合使用向量空间模型和链接 标签分析及元数据分析的页面相关度判断的方法。实验表明，该方法可以提高主题 相关度判定的效率和准确率。 在对目前各种搜索策略的研究的基础上，决定采用效率较高的基于内容评定策 略进行网络爬行。针对基于内容评定策略中存在的“近视”问题，设计了一种基于 当前页面 URL 的简单链接结构分析方法，综合使用内容评定和链接结构分析方法作 为网络爬行策略。该方法能在一定程度上缓解 “近视”问题，提高爬虫的效率。 在通用搜索引擎的基础上，通过对其网络爬虫部分进行改进，设计和实现了一 个完整的主题搜索引擎。实验结果表明，上述主题相关度判断方法和爬行策略可以 有效地控制爬虫对主题相关的信息进行爬行，达到了设计要求。 关键词：主题搜索引擎，网络爬虫，爬行策略，页面过滤 I Abstract With the development of the network technology and the rapid growth of Internet information resource, traditional Search Engines can’t satisfy the requirement of personalized information retrieval service. Topical Search Engine only searches the Web resources related with specific theme, so it will do better in special information search. To design a Topical Search Engine, the core task is to design a topical crawler. And improve the method of page filtration and crawling strategy are two most important tasks of topical crawler. After analyzing the relevant judge methods, this paper presents a new method in relativity judgment by integrated use of vector space model, link tags analysis and metadata analysis. Result shows that it may improve the speed of crawler. Based on the research of current crawling strategy, think about the problem of the performance of crawler, the content-based strategy is decided to use in this paper. But there are some shortages such as “short sight” in the crawling strategy. In order to improve the crawling strategy, an easy structure analysis method base on URL content is added to control the crawler. We combine content-based determinant with the simple link analysis and URL tag information as the crawling strategy. This strategy can resolve the “short sight” shortage and improve the performance of crawler. Based on the survey of topical Search Engine and traditional Se