垂直搜索引擎主要技术研究-计算机系统结构专业论文.docxVIP

Abstract With the explosive growth of information on the Internet, general search engine fail to satisfy special query needs of special user, in this case vertical search engine came into being. In this thesis, several key technologies of vertical search engine are investigated, including focused crawler, web information extraction and purification technology. First of all, a focused crawler based on one-class document classification is designed; and one of the one-class document classification algorithms, the prototype algorithm, is improved by using the least square method to solve the optimal solution of the prototype vector in order to improve the classification accuracy; the experiments show that the improved prototype algorithm significantly improve the classification accuracy. Secondly, two page purification algorithms are proposed based on DIV_DOM model, namely, the heuristic page purification algorithm based on DIV_DOM model and the page purification algorithm base on DIV template tree; and they are applied to the focused crawler in order to evaluate their effects on the performance of the focused crawler; the experiments show that the latter performs better on web purification than the former. Finally, based on the proposed web purification algorithms, another DOM-based web pages structured information extraction technology is proposed in this thesis, and an inductive learning algorithm is used to produce extraction rules automatically; the experiments show that the proposed information extraction algorithms is effective. Keywords: Vertical Search Engine; Focused Crawler; One-class Document Classification; Web Page Purification; Web Information Extraction 独 创 性 声 明 本人声明所呈交的学位论文是本人在导师指导下进行的研究工作及取得的 研究成果。除了文中特别加以标注和致谢的地方外，论文中不包含其他人已经发 表或撰写过的研究成果，也不包含为获得 暨南大学 或其他教育机构的学 位或证书而使用过的材料。与我一同工作的同志对本研究所做的任何贡献均已在 论文中作了明确的说明并表示谢意。 学位论文作者签名： 签字日期： 年 月 日 学位论文版权使用授权书 本学位论文作者完全了解 暨南大学 有关保留、使用学位论文的规 定，有权保留并向国家有关部门或机构送交论文的复印件和磁盘，允许论文被查 阅和借阅。本人授权 暨南大学 可以将学位论文的全部或部分内容编入有 关数据库