【论文】主题搜索引擎中爬虫搜索策略的研究.pdfVIP

116 2014 ，50（2） Computer Engineering and Applications 计算机工程与应用 主题搜索引擎中爬虫搜索策略的研究 1 1 2 史宝明 ，贺元香 ，吴崇正 1 1 2 SHI Baoming , HE Yuanxiang , WU Chongzheng 1.兰州文理学院 电子信息工程学院，兰州 730000 2.兰州理工大学 计算机与通信学院，兰州 730050 1.School of Electronics and Information Engineering, Lanzhou University of Arts and Science, Lanzhou 730000, China 2.School of Computer and Communication, Lanzhou University of Technology, Lanzhou 730050, China SHI Baoming, HE Yuanxiang, WU Chongzheng. Research on search strategy of web spider in topic-oriented search engines. Computer Engineering and Applications, 2014, 50（2）：116-119. Abstract ：In order to solve the low efficiency problem of traditional focused crawler, web spider always selects the most valuable links to visit, so how to focus the search around a given topic is a key problem. The traditional method always only computes the relevance of the links, but ignores the relevance among the unlabeled URL, now it proposes the algorithm based on link model which combines the seed URL with unlabeled URL to compute the relevance of the other URL, and it deduces the point that initial iterative is insensitivity of the results. Compared with the methods based on traditional algorithm, experimental result proves the performance of the new algorithm is more efficient than the traditional ones. Key words ：web spider; topic-oriented search engine; search strategy; Vector Space Model （VSM ） 摘 要：为了解决传统主题爬虫效率偏低的问题，传统主题爬虫会选择最有价值的链接进行访问，仅简单地计算链 接的相关性，却忽视待分析URL 之间的相关性关系，致使主题爬虫爬取效率较低。提出一种基于链接模型的相关性 判别算法，综合利用有标种子URL 和无标的待判别URL 实现对无标URL 的相关性判别，并推导出迭代初值选取对 结果的不敏感性。实验结果表明，与传统的网络爬虫算法相关性判别方法相比，提出的方法效率更高。 关键词：网络爬虫；主题搜索引擎；搜索策略；向量空间模型 文献标志码：A 中图分类号：TP391 doi ：10.3778/j.issn.1002-8331.1303-0512 1 引言 域、学科信息门户、公司信息中心、行业专家等在内的用 随着互联网的快速发展，信息量爆炸式增长，传统 户，提供了整套的网络信息资源服务解决方案[3] 。其特 的搜索引擎在信息的覆盖率和搜索结