78_基于word2vec的大中华区词对齐库的构建.pdfVIP

基于word2vec 的大中华区词对齐库的构建 * 王明文，徐雄飞，徐凡，李茂西 （江西师范大学 计算机信息工程学院，南昌 330022） 摘要：本文针对大陆、香港和台湾地区（简称大中华区）存在同一种语义但采用不同词语进行表达的语言 现象进行分析。首先，我们抓取了维基百科以及简繁体新闻网站上的3,200,000 万组大中华区平行句对，手 工标注了一致性程度达到 95% 以上的 10,000 组大中华区平行词对齐语料库。同时，我们提出了一个基于 word2vec 的两阶段的大中华区词对齐模型，该模型采用word2vec 获取大中华区词语的向量表示形式，并融 合了有效的余弦相似度计算方法以及后处理技术。实验结果表明我们提出的大中华区词对齐模型在以上两 种不同文体的词对齐语料库上的F1 值显著优于现有的GIZA++和基于HMM 的基准模型。此外，我们在维 基百科上利用该词对齐模型进一步生成了90,029 组准确率达82.66%的大中华区词语三元组。 关键词：大中华区；词对齐；最长公共子序列；word2vec 中图分类号：TP391 文献标识码：A Building Word Alignment Corpus for the Greater China Region * Wang Mingwen ，Xu Xiongfei ，Xu Fan ，Li Maoxi (School of Computer Information Engineering, Jiangxi Normal University, Nanchang 330022, China) Abstract: We deal with the linguistic phenomenon that different expressions refer to the same semantic meaning among the Mainland China, Hong Kong and Taiwan, a.k.a., the greater China region(GRC). Firstly, we automatically crawl 3.2 million GCR parallel sentences from the wikipedia and the news website with simplified and traditional encoding, and then manually annotate 10,000 GCR parallel word alignment corpora with the annotation agreement above 95%. Meanwhile, we present a 2-phase GCR word alignment model based on word2vec to represent the GCR words’ vector, which incorporate the effective word similarity calculation method along with post-processing technology. Expreiment results on the proposed 2 different word alignment corpus demenstrate the effectiveness of our GCR model which significantly outperforms the current GIZA++ and HMM-based models. Furthermore, we generate 90,029 triples from wikipedia with accuracy over 82.66%. Keywords: the greater China region