256_基于Active Learning的中文分词领域自适应.pdfVIP

基于Active Learning 的中文分词领域自适应 许华婷，张玉洁，杨晓晖，单华，徐金安，陈钰枫 （北京交通大学，北京，100044） 摘要：在新闻领域标注语料上训练的中文分词系统在跨领域时性能会有明显下降。针对目标领域的大规模 标注语料难以获取的问题，本文提出Active learning 算法与n-gram 统计特征相结合的领域自适应方法。该 方法通过对目标领域文本与已有标注语料的差异进行统计分析，选择含有最多未标记过的语言现象的小规 模语料优先进行人工标注，然后再结合大规模文本中的n-gram 统计特征训练目标领域的分词系统。本文采 用了CRF 训练模型，并在100 万句的科技文献领域上，验证了所提方法的有效性，评测数据为人工标注的 300 句科技文献语料。实验结果显示，在科技文献测试语料上，基于Active Learning 训练的分词系统在各 项评测指标上均有提高。 关键词：中文分词；领域自适应；主动学习 中图分类号：TP391 文献标识码：A Domain adaptation of Chinese word segmentation based on active learning XU Huating, ZHANG Yujie, YANG Xiaohui, SHAN Hua, XU Jin’an, CHEN Yufeng (Beijing Jiaotong University, Beijing, 100044, China) Abstract: Chinese word segmentation systems, trained on annotated corpus of newspaper, often obviously decrease in performance when faced with a new domain. Since there is no large scale annotated corpus on the target domain, statistics based methods could not work well. In our approach of this paper, we attack domain adaptation of Chinese word segmentation by combining active learning with the statistical features of n-gram. Our idea is to select such a small amount of data for annotation that the gap from the target domain to the News will be overcome. The word segmentation model is trained again by the corpus added with newly annotated data and the statistical features of n-gram from the raw corpus. We use the CRF model for training and a raw corpus of one million sentences on patent description to verify the proposed approach. For test data, 300 sentences are randomly selected and manually annotated. The experimental results show that the performances of the Chinese word segmentation system based on our approach are improved on each evaluation metrics. Key words: Chinese word se