基于无指导学习策略的无词表条件下的汉语自动分词.pdfVIP

第27卷 第6期 计 算 机 学 报 Vo1．27 NO．6 2004年 6月 CHINESE JoURNAL oF COMPUTERS June 2004 基于无指导学习策略的无词表条件下的汉语自动分词 孙茂松” 肖 明” 邹嘉彦” (清华大学智能技术与系统国家重点实验室 北京 100084) (香港城市大学语言资讯科学研究中心 香港) 摘 要 探讨了基于无指导学习策略和无词表条件下的汉语自动分词方法，以期对研制开放环境下健壮的分词系统 有所裨益．全部分词知识源自从生语料库中自动获得的汉字 Bigram．在字间互信息和￡一测试差的基础上，提出了一种 将两者线性叠加的新的统计量md，并引人了峰和谷的概念，进而设计了相应的分词算法．大规模开放测试结果显 示，该算法关于字间位置的分词正确率为85．88 ，较单独使用互信息或￡一测试差分别提高了2．47 和5．66 ． 关键词 汉语自动分词；无指导学习；汉字Bigram；互信息与￡一测试的组合 中图法分类号 TP391 Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy SUN Mao—Song XIAO Ming TSOU B K (National Key Laboratory of Intelligent Technology and Systems，Tsinghua University，Beijing 100084) (Language Information Sciences Research Centre，City University of Hong Kong，Hong Kong) Abstract The study of word segmentation using neither a dictionary nor an annotated corpus will help to improve the robustness of Chinese word segmentation systems in practical applica— tions．This paper presents a new statistical measure，md，for estimating the associative strength between two adjacent Chinese characters in terms of a linear combination of two common statisti— cal measures，namely，mutual information and t-test，puts forward the concepts of peak and val— ley of md，and gives an algorithm accordingly．The experimental result on large—scale open tests shows that the proposed algorithm achieves 85．88 segmentation accuracy(defined as the ratio of the number of correctly segmented character 1ocations to the tota1 number of character 1ocations in the testing texts)．The most advantage of the algorithm is that it can easily adapt to new appl