广域网分布式Web爬虫-软件学报.PDFVIP

ISSN 1000-9825, CODEN RUXUEW E-mail: jos@ Journal of Software , Vol.21, No.5, May 2010, pp.1067− 1082 doi: 10.3724/SP.J.1001.2010.03725 Tel/Fax: +86-10 © by Institute of Software , the Chinese Academy of Sciences . All rights reserved. ∗ 广域网分布式 Web 爬虫 + 许 笑 , 张伟哲, 张宏莉, 方滨兴 (哈尔滨工业大学 计算机科学与技术学院,黑龙江 哈尔滨 150001) WAN-Based Distributed Web Crawling + XU Xiao , ZHANG Wei-Zhe, ZHANG Hong-Li, FANG Bin-Xing (School of Computer Science and Technology, Harbin Institue of Technology, Harbin 150001, China) + Corresponding author: E-mail: smilestor@, Xu X, Zhang WZ, Zhang HL, Fang BX. WAN-Based distributed Web crawling. Journal of Software, 2010, 21(5):1067−1082. /1000-9825/3725.htm Abstract : There are three core issues recognized for WAN-based distributed Web crawling systems: Web Partition, Agent collaboration and Agent deployment. Centering around these issues, this paper presents a comprehensive overview of the current strategies adopted by academic and business communities. The experiences, problems and challenges encountered by the WAN-based distributed Web crawlers are classified and discussed in depth. A summary of the current evaluation indicators is also given. Finally, conclusion and some suggestions for future research are put forward. Key words: search engine; WAN-based distributed crawling; Web partition; agent collaboration; agent deployment 摘 要: 分析了广域网分布式Web 爬虫相对于局域网爬虫的诸多优势,提出了广域网分布式Web 爬虫的3 个核心 问题:Web 划分、Agent 协同和Agent 部署.围绕这3 个问题,对目前学术界和商业界出现的多种实现方案和策略进 行了全面的综述,深入讨论了研究中遇到的问题与挑战,并论述了广域网分布式 We