中文分词十年回顾-第四纪研究.PDFVIP

21 　3 Vol.2 1, No.3 2007 5 JO URNA L OF CHINESE INFO RM A TION PROCESSING May, 2007 :1003-0077 (2007)03-0008-012 1 2 , (1., 100080;2., ) 　:过去的十年间, 尤其是2003 年国际中文分词评测活动Bak off 开展以来, 中文自动分词技术有了可喜的进 。其主要表现为:(1)通过“分词规范+词表+分词语料库”的方法,使中文词语在真实文本中得到了可计算的定 义,这是实现计算机自动分词和可比评测的基础;(2)实践证明, 基于手工规则的分词系统在评测中不敌基于统计 学习的分词系统;(3)在Bak off 数据上的评估结果表明, 未登录词造成的分词精度失落至少比分词歧义大5 倍以 上;(4)实验证明,能够大幅度提高未登录词识别性能的字标注统计学习方法优于以往的基于词(或词典)的方法, 并使自动分词系统的精度达到了新高。 :计算机应用;中文信息处理;中文分词;词语定义;未登录词识别;字标注分词方法 :TP39 1　　　　:A Chinese Word Segmentation:A Decade Review 1 2 HUANG Chang-ning , ZH AO Hai (1.Microsof t R s arch A sia, B ijing 100080, China;2.City Univ rsity of Hong Kong, Hong Kong, China) Abstract:During th last d cad , sp cially sinc th First Int rnational Chin s Word S gm ntation Bak off w as h ld in July 2003, th study in automatic Chin s word s gm ntation has b n gr atly improv d.Thos improv - m nts could b summariz d as follow ing:(1)on th computation s ns Chin s words in r al t xt hav b n w ll-d - fin d by “s gm ntation guid lin s + l x icon + s gm nt d corpus”;(2)practical r sults show that p rformanc of statistic s gm ntation sy st ms outp rforms that of handcraft d rul -bas d syst ms;(3)th valuation in t rms of Bak off data shows that th accuracy drop caus d by out-of-vocabulary (OOV )w ords is at l ast fiv tim s gr at r than that of s gm ntation ambiguiti s;(4)th b tt r p rformanc of OOV r cognition th high r accuracy of th s gm ntation syst m in whol , and th accuracy of statistic s gm ntation syst ms with charact r-bas d tagging ap- proach outp rform