基于半监督聚类的爬虫在线样本生成算法-计算机科学与技术专业论文.docxVIP

哈尔滨工业大学工学硕士学位论文 哈尔滨工业大学工学硕士学位论文 -II- -II- Abstract In recent years, with the popularity of Internet and its rapid growth, Internet is full of information resources. Although with the help of search engine, you can simplify the process of obtaining information, but in many cases, in order to find specific fields of information, users still have to spend a lot of time and energy, facing even hundreds of thousands of search results a needle in haystack kind of feeling. In order to solve this problem, the focused crawler emerged that it can be user-defined themes, retrieve related pages from the Web pages in an intelligent way, use of machine learning or information retrieval methods for the results of treatment, and extract the most valuable answer. It can be said that, regardless of accessing to a specific area on the theme of information, or building personalized search engine, the focused crawler has become a preferred method. There are many component of focused crawler, and the sample generation system is an important one. This paper firstly introduced some basic concepts and main contents of a generation system in focused crawler. As the core issue of focused crawling is how to determine the weight of candidate url, samples are extremely important to the focused crawler. We analyzed of the existing system the subject of focused crawler and found that their sample generation system have certain problems. After in-depth study of sample pages, we found that the anchor text contains a lot of characteristics of information. In order to extract sample pages in a large number of pages and design a model based on the anchor text, the model uses theme of anchor text as the foundation. We adopt a semi- supervised clustering algorithm to calculate similarity between pages and seed collections to estimate possibility of the former into sample pages. After filtering by the threshold, we get new samples and merge it with the original sample collection, then complete the proce