nutch分布式网络爬虫研究与优化 research and optimization of nutch distributed crawler.pdfVIP

ISSN 1673-9418 CODEN JKYTA8 E-mail: fcst@ Journal of Frontiers of Computer Science and Technology 1673-9418/2011/05(01)-0068-07 Tel: +86-10 DOI: 10.3778/j.issn.1673-9418.2011.01.007 Nutch * 1+ 2 2 詹恒飞 , 杨岳湘 , 方 宏 1. 国防科学技术大学 计算机学院, 长沙 410073 2. 国防科学技术大学 信息中心, 长沙 410073 * Research and Optimization of Nutch Distributed Crawler 1+ 2 2 ZHAN Hengfei , YANG Yuexiang , FANG Hong 1. School of Computer Science, National University of Defense Technology, Changsha 410073, China 2. Information Center, National University of Defense Technology, Changsha 410073, China + Corresponding author: E-mail: zhf_a_b@163.com ZHAN Hengfei, YANG Yuexiang, FANG Hong. Research and optimization of Nutch distributed crawler. Journal of Frontiers of Computer Science and Technology, 2011, 5(1): 68-74. Abstract: As a good open-source search engine, Nutch kernel code uses a lot of MapReduce programming models, being used by more and more businesses and organizations to customize their needs in line with the distributed search engine product. As a good search engine, one of the important prerequisites is how to grab network data as much as possible to build indexes. This paper introduces Nutch’s working mechanism based on Hadoop distributed Web crawler, points out its shortcomings and proposes an improved program, which can make Web crawler using network resources more efficiently to capture network data. Experimental results show that it is indeed more effi- cient than the original programs. Key word