基于Lucene的中文分词技术分析-软件工程专业论文.docxVIP

摘要 摘要 摘要 随着互联网信息的急剧膨胀，搜索引擎的索引数据库变得越来越大，其开发 与维护成本也越来越高。开源工具包 Lucene 被广泛地应用于构建全文索引与检索 的项目中，但其对中文信息处理的支持不足。本文研究基于 Lucene 的中文分词技 术。首先，介绍了 Lucene 及中文分词模块的核心内容，包括词典机制、分词算法 和歧义处理；然后，设计了根据词语首字建立哈希表，次字及剩余部分建立有序 表的词典结构，其中首字相同的词语采用树的形式来存放；其次，通过使用正、 逆向最长逐字匹配算法对文本进行分词，采集歧义字段，并运用歧义切分规则及 互信息原理进行歧义处理；最后，根据词典机制及分词算法改进 Lucene 中文分词 模块，实现中英文分析器 MyCEAnalyzer。从词典性能和分词性能两方面的实验 数据来看，MyCEAnalyzer 在中文信息处理方面取得了良好效果。 关键词：中文分词 Lucene 逐字匹配算法 歧义处理 Ab Abstract Abstract With the rapid development of Internet, databases of search e ngines become larger and larger, and the cost for implementation and maintenance of the databases are higher and higher. Lucene, an open source toolkit, has been widely used in full- text indexing and retrieving. However, information in Chinese cannot be well processed in Lucene. With this motivation, segmentation of Chinese words based on Lucene is studied in this thesis. First, Lucene and the current Chinese words segmentation method, including dictionary mechanism, segmentation algorithm, and ambiguity process, are briefly introduced. Secondly, a dictionary structure is designed which contains, a hash table established by the first characters, and ordered hierarchy tables constructed by the second and the rest characters, where the words shared by the same first character are stored in the memory as a tree structure. Thirdly, the texts are segmented by the forward and reverse maximum literals matching algorithm, and the ambiguity segments are collected and processed by some ambiguity separating rules and mutual information principles. Finally, according to the dictionary mechanism and the proposed algorithm, Chinese word segmentation module is improved, and an analyzer named MyCEAnalyzer that can process both Chinese and English words is developed. Experimental results which is tested from both the dictionary performance and word segmentation performance aspects show that the tool works well in practice. Keywords: Chinese word segmentation Lucene literal matching algor