基于可扩展哈希算法的并行爬虫动态负载均衡实现计算机科学与技术专业论文.docxVIP

Abs Abstract 哈尔滨工业 哈尔滨工业大学工学硕士学位论文 - - PAGE III - - - PAGE VI - Abstract In recent years, along with the development of Internet, information published through Internet has been enriched greatly. Search Engine has become a necessary tool for daily life. And web crawler, which is an automate program for gathering web pages from Internet, is a critical part for Search Engines. As the s cale of Internet grows, there’re more challenges with the quantity and freshness of the pages that Search Engines gathered. As a result, the requirements of crawlers are higher than ever before. Crawlers not only need to fetch more pages with less resource consumption, but also be able to crawling under constrains. This paper mainly focuses on how to improve crawler performances, and studies load balancing problem of crawler from two aspects: the static work load distribution and the dynamic work load repartition between web crawlers. Firstly, after studying relative technologies of parallel crawler, this paper presents static work load distribution strategy based on extensible hashing and logic two-level node mapping. Extensible hashing system is discussed and an improved lazy algorithm is stated, which avoid splitting buckets and updating hash table continually, caused by unusual distribution of pseudo key values. Secondly, the critical problem of multi-task parallel crawler, dynamic work-load balancing between crawlers, is studied. This paper introduces LW(load weigh) to describe the load state of cluster node more accurately, and then a hyper graph-based model is constructed to accurately model both communication cost in the application and migration cost to move data. Furthermore, it presents an effective multi-level approaches to improve the hyper graph repartitioning qualit y significantly. At last this paper introduces the design issues of a distributed crawler and some key modules’ implementation. Keywords: parallel wed crawling; dynamic load balance; extensible hash; hype