垂直搜索引擎中主题网络爬虫算法的设计与研究计算机技术专业论文.docxVIP

ABSTRACTABSTRACT ABSTRACT ABSTRACT In the era of information globalization，in recent years，traditional general search engines can not meet the needs of people for professional and personalized information． So it is urgent to set up vertical search engine for specific domain．As topic web crawler plays an important role in topic search engine，the design of it has direct impact on the service quality of the search engine．Traditional topic web crawler mainly analyzes the correlation between the candidate URL and the topic through the analysis of all the content of web pages．But web pages contain more and more topics nowadays，this kind of analysis may affect the correlation between the candidate URL and the topic．In recent years，the research about topic crawler has been focused on two aspects：topic relevance algorithm and topic crawler search strategy．In this paper，a hybrid crawler search strategy is proposed，which is superior to the traditional topic web crawler．The main research work includes： (1)On the basis of the research about the related technologies of the topic crawler， this paper states the existing research results，which lays theoretical foundation for the new crawler strategy proposed in the paper． (2)The hierarchial structure of the tree is applied to the process of Filter Bloom duplicate removal，and a multilayer Filter Bloom(MLBF)based the traditional Filter Bloom is proposed to remove the reduplication of ULR．Each layer of Filter Bloom consists of k independent hash function and m bit array，and in this way URL is considered as set divided by“／,transforming the problem of URL duplicate removal into the path problem of tree．The experimental results show that the improved multilayer Filter Bloom has a smaller false positive rate and crawling efficiency is improved． (3)Learning from existing achievements of the studies，this paper puts forward hybrid crawling strategy based on contents and link evaluation，which combines web content evaluation and web link evalu