面向文本知识理的自适应中文分词算法.pdfVIP

第33 卷第10 期 重 庆 大学 学 报 Vol.33 No. 10 2010 年10 月 Journal of Chongqing University Oct. 2010 : 1000582 (2010) 1011008 冯 永, 贺 迅, 唐 黎, 陈显勇, 陈 贞 (重庆大学 计算机学院,重庆 400044) :针对传统字典匹 分词法在识别新词和特殊词处理方面的不足, 结合2 元统计模型提 出了面向文本知识管理的自适应中文分词算法!!! SACWSASACWSA 在预处理阶段结合应用 有限状态机理论基于连词的分隔方法和分治策略对输入文本进行子句划分, 从而有效降低了分词 算法的复杂度; 在分词阶段应用 2元统计模型, 结合局部概率和全局概率, 完成子句的切分, 从而有 效地提升了新词的识别率并消除了歧义;在后处理阶段,通过建立词性搭 规则来进一步消除2 元 分词结果的歧义SACWSA 主要的特色在于利用 分而治之 的思想来处理长句和长词, 用局部概 率与全局概率相结合来识别生词和消歧通过在不同领域语料库的实验表明,SACWSA 能准确 高效地自动适应不同行业领域的文本知识管理要求 : 知识管理; 文本处理; 统计方法; 自适应算法 : TP182 :A Text knowledge management oriented adaptive Chinese word segmentation algorithms FENG Yong , HE un, TANG Li, CHEN ianyong , CHEN Zhen (College of Computer Science, Chongqing University, Chongqing 400044, P.R. China) Abstract: To overcome the shortcomings of new word recognition and special word processing for the traditional dictionarybased matching algorithm in, text knowledge management oriented adaptive Chinese word segmentation algorithm ( SACWSA) based on 2gram statistical model is presented. . At the preprocessing stage, SACWSA applies finite state machine theory, conjunctionbased partition method and divide conquer strategy to partition long sentences in input text into subsentences, which reduces the algorithm complexity effectively. At theword segmentation stage, 2gram statisticalmodel is employed and combined with partial probability and overall probability to partition the subsentences into words, which improved the recognition rate of new words and eliminated ambiguity. At the postprocessing stage, the matching rules of partofspeech are established to eliminate ambiguity of 2gram word segmentation results furt