基于避免交集型歧义的最大匹配算法改进的研究与实现-计算机软件与理论专业论文.docxVIP

摘 要 在当今信息化世界中，中文信息处理技术已经在各个领域中得到了广泛的应 用。本文主要对中文分词算法中的基于分词词典机制的最大匹配算法进行研究讨 论。由于中文语法语义复杂导致大量歧义字段的出现，文章中提出的最大匹配改 进算法是针对分析原有的最大匹配算法的弊端以及避免交集型歧义字段在切分中 出现错误而改进的，该算法在保证一定切分速率的基础上提高了中文切分的准确 率。 本文描述的中文分词算法是基于避免交集型歧义字段的最大匹配算法的改进 算法。文中首先介绍了目前的中文分词算法，其中就常用的分词算法、分词词典 机制以及歧义字段相关理论三部分来详细讲解目前中文分词算法的基本理论；其 次根据已有分词词典机制描述了基于 Hash 表的反向词典机制；在提出的反向分词 词典机制的基础上改进了最大匹配算法，提出了基于避免交集型歧义的最大匹配 算法的改进算法。 最后论文中实现了简单的中文分词系统，此系统用来进行基于避免交集型歧 义字段的最大匹配改进算法实验，并且用此系统把四种算法进行了实验比较。实 验结果证明基于避免交集型歧义字段的最大匹配改进算法具有较好的性能和实用 性。但是由于本实验没有涉及对于未识别词的识别问题，所以实验结果并没有达 到目前标准的准确率。 关键词：中文信息处理 中文分词算法 最大匹配算法 分词词典机制 交集型歧义 字段 Abstract In todays information world, the Chinese information processing in various large fields has been widely used. This paper focuses on the maximum matching algorithm research and discussion based on the dictionary mechanism on the Chinese word segmentation algorithm. Because of the Chinese semantic complexity, a large number of ambiguity will appear after Chinese segmentation. In this paper the maximum matching improved algorithm is based on the analysis of the maximum matching algorithm so as to avoid ambiguous phrases of overlap type in the segmentation error and improved, in the guarantee rate based on improved Chinesse segmentation accuracy. The Chinese word segmentation algorithm in this paper is based on avoiding overlapping ambiguity string of the maximum matching algorithm. Firstly, this paper introduces the current Chinese word segmentation algorithm, it include commonly used in the word segmentation algorithm, commonly used the dictionary mechanism as well as the theory about ambiguity to explain basic theories of the Chinese word segmentation algorithm; Secondly this paper describe the reverse dictionary mechanism based on the Hash table according to the existing dictionary mechanism, the maximum matching improved algorithm,and the maximum matching improved algorithm for avoid ambiguous phrases of overlap type .This algorithm’s accuracy is improved based on a range of segmentation rate. Fi