一种可扩展高效链接提取模型实现与验证.doc

摘要 随着WWW(World Wide Web)越来越广泛的发展与应用，搜索引擎已经成为人们从中查找信息的重要工具；在搜索引擎的系统实现中，如何通过链接提取发现更多更广的Web资源又是影响搜索引擎性能的重要因素之一。 本文总结了设计链接提取模块所要求的“容错性”、“正确性”、“全面性”、“高效性”和“可扩展性”等五个目标，并从这些角度去分析传统的链接提取方法的不足，并作为改进，提出了一种新的设计思路。 本文将链接提取的过程划分为信息提取，信息加工，信息分析以及信息储存四个过程来进行研究。信息的获取通过HTML文法分析方法从文档中得到初始URI(Uniform Resource Indetifier)数据；信息加工阶段通过运用URI解析算法对初始数据进行精练；然后在信息分析过程中进一步地筛选与过滤；最后将结果存储在一个双链表结构中。 基于上述方法，本文实现了一个新的链接提取模型，并将该模型运用于北京大学天网WWW搜索引擎；在获得足够的实验数据之后，全面的比较了这种新的链接提取模式与传统方法在各项指标上的优劣。结果表明该模型有明显的优势。 关键字：搜索引擎，链接提取，统一资源地址(URI) Abstract As the World Wide Web is becoming more and more popular, search engine has become an essencial tool for people to look for certain information on the web. Among many factors to take into consideration in implementing a search engine, how to find more web resources using URL extractor is a very important one. This essay conludes the basic objectives in the design of URL extractor module, which are robustness, correctness, completeness, effectiveness and expansibility. With these objectives, the essay analyzes the weakness of original design and furthermore, generalizes a new and more powerful design method. The new method is devided into four procedures, that are Information Extraction, Information Refinement, Information Analysis and Information Storage. During the Information Extraction process, the initial URI (Uniform Resource Indentifier) information is extracted from HTML source using HTML BNF analysis method. The initial raw data is refined and normalized in the Information Refinement process and filtered in the Information Analysis phase. Eventually, the remaining correct and useful data is stored in a double-link structure. On implementing the above method, the essay has produced a new URL extractor module, and applies the module on PKU Tianwang WWW search engine,then after acquiring sufficient testing data, completely compares the new module with the original one. According to the comparison result, the new module is much better in many aspects. Keyword: