基于有效子串标注的中文分词.PDFVIP

文章编号：1003-0077（2007）05-0000-00 基于有效子串标注的中文分词∗ 1 1 赵海 ，揭春雨 （1.香港城市大学中文翻译及语言学系，香港九龙达之路 83 号） 摘要：由于基于已切分语料的学习方法和体系的兴起，中文分词在本世纪的头几年取得了显著的突破。尤 其是 2003 年国际中文分词评测活动 Bakeoff 开展以来，基于字标注的统计学习方法引起了广泛关注。本文 探讨这一学习框架的推广问题，以一种更为可靠的算法寻找更长的标注单元来实现中文分词的大规模语料 学习，同时改进已有工作的不足。我们提出子串标注的一般化框架，包括两个步骤，一是确定有效子串词 典的迭代最大匹配过滤算法，二是在给定文本上实现子串单元识别的双词典最大匹配算法。该方法的有效 性在 Bakeoff-2005 评测语料上获得了验证。 关键词：中文分词；基于子串标注的分词 中图分类号：TP391 文献标识码：A Effective Subsequence-based Tagging for Chinese Word Segmentation 1 1 Hai Zhao , Chunyu Kit (1. Department of Chinese, Translation and Linguistics, City University of Hong Kong, 83 Tat Avenue, Kowloon, Hong Kong SAR, China) Abstract: The research of automatic Chinese word segmentation has been advancing rapidly in recent years, espe- cially since the First International Chinese Word Segmentation Bakeoff held in 2003. In particular, character-based tagging has claimed a great success in this field. In this paper, we attempt to generalize this method to subse- quence-based tagging. Our goal is to find longer tagging units through a reliable algorithm. We propose a two-step framework to serve this purpose. In the first step, an iterative maximum matching filtering algorithm is applied to obtain an effective subsequence lexicon, while in the second step, a bi-lexicon based maximum matching algo- rithm is used for identifying subsequence units. The effectiveness of this approach is verified by our experiments using two closed test data sets from Bakeoff-2005. Key words: Chinese word segmentation (CWS), subsequence-based tagging approach of CWS 1引言 中文分词技术[13][14][21]随着基于切分语料的机器学习方法的兴起而在最近几年获 得了显著突破。特别是 SIGHAN1举办的国际中文分词评测(International Chinese Word Segmentation Bake