蒙古语有向图形态分析器判别式词干词缀切分.pdfVIP

蒙古语有向图形态分析器的判别式词干词缀切分* 姜文斌 1 吴金星 2,1 乌日力嘎 2,1 那顺乌日图 2 刘群 1 1 （中国科学院计算技术研究所 北京，100190） 2 （内蒙古大学蒙古学学院 呼和浩特，010021 ） (jiangwenbin@ ) 摘 要：蒙古语形态分析中，我们之前的有向图模型[16]取得了较高的性能。这种建模方式以图状结构刻画 句中词干和词缀之间的概率关系，从而借助上下文信息为每个词确定最佳的切分标注候选。为每个词尽可 能地枚举出所有合法的切分标注候选，是有向图模型有效工作的前提。本文提出了一种基于判别式分类的 词干词缀切分策略，与之前基于词干表和词缀表的枚举方案相比，该方法对于词中含有未登录词干的情形 具有更好的泛化能力。以 20 万词规模的三级标注人工语料库为训练数据，采用判别式词干词缀切分的有向 图形态分析器，对于含有未登录词干的情形，词级切分标注正确率提高了7 个百分点。 关键词：蒙古语 词法分析 词性标注 词干提取 有向图 判别式 Discriminative Stem-Affix Segmentation for Directed-Graph-Based Mongolian Lexical Analyzer JIANG Wen-Bin1 WU Jin-Xing2,1 Wuriliga2,1 Nasanurtu2 LIU Qun1 1 （Institute of Computing Technology, Chinese Academy of Sciences, Beijing, P.R.China, 100190 ） 2 （Inner Mongolian University, Huhhot, P.R.China, 010021 ） (jiangwenbin@) Abstract: In Mongolian lexical analysis, directed-graph-based model achieves high performance. This model uses a directed-graph architecture to describe the probabilistic relationship of stems and affixes, thus to determine the best segmented and tagged candidate for each word according to the context. Therefore, it is the basis for a directed-graph-based analyzer to enumerate all legal segmented and tagged candidates for each word. This paper proposes for Mongolian lexical analysis a novel stem-affix segmentation model based on discriminative classification method. Compared with the enumeration strategy based on the stem- and affix sets, this method shows better generalization ability for the words with unknown stems. Using the 3rd-level annotated corpus with about 200,000 words as the training data, the directed-graph-based lexical analyzer with discriminative stem-affix segmentation