结合局部歧义词网格和条件随机场的双层分词法.pdfVIP

Transactions on Computer Science and Technology March 2013, Volume 2, Issue 1, PP.1‐8 A Double-layer Word Segmentation Combined with Local Ambiguity Word Grid and CRF # Lei Chen , Miao Li, Jian Zhang , Weihui Zeng Institute of Intelligent Machines, Chinese Academy of Sciences, Hefei 230031, China #Email: alan.cl@163.com Abstract This paper presents a double-layer model of Chinese word segmentation based on the combination of Local Ambiguity Word Grid and Conditional Random Fields. Firstly, the Local Ambiguity Word Grid algorithm is used to generate rough segmentation results in the lower level. Then, the text is segmented again based on CRF, where the rough results are set as one feature. The Local Ambiguity Word Grid algorithm has the advantage of detecting ambiguity from the process of Chinese word segmentation, while CRF can cope with vocabulary and out-of-vocabulary word equally. Therefore, the hybrid Local Ambiguity Word Grid and CRF approach is the effective resolution for the ambiguity and out-of-vocabulary word. The system is closed tested in the MSRA and PKU testing sets that are provided by the SIGHAN2005 Chinese Language Processing Bakeoff, along with the comparison between four characters and six characters in a set of label. The experiments show that F-measures of the MSRA and PKU testing sets in the closed test reach 97.1% and 95.1% respectively. Additional, the experimental results of open test reveal the practical application of the model. Keywords: Local ambiguity word grid; CRF; Word segmentation 结合局部歧义词网格与条件随机场的双层分词法* # 陈雷 ，李淼，张健，曾伟辉 中国科学院合肥物质科学研究院，智能机械研究所，安徽合肥 230031 摘 要：本文提出了一种结合局部歧义词网格与条件随机场的双层中文