LORANGE：一种机器翻译评测自动元评价方法.docVIP

LORANGE：一种机器翻译评测自动元评价方法 　　摘要：本文提出一种LORANGE （Limited ORANGE）自动元评价方法，在语料资源受限的情况下，对目前主流的基于n元语法的BLEUS、ROUGE-L和ROUGE-S的性能进行评价。这种自动元评价方法弥补了传统人工元评价方法的主观性强、代价高昂、不稳定、评测周期长等缺点，有利于机器翻译系统的参数优化调整和缩短系统研发周期。在俄汉双语句对齐的数据集上，采用这种方法对BLEUS、ROUGE-L、ROUGE-S进行性能评估，实验结果显示，ROUGE-L性能最优，ROUGE-S次之，BLEUS性能最差；并且通过与人工元评价结果进行一致性检验，LORANGE自动元评价方法的有效性与可行性得以验证。 　　关键词：LORANGE；BLEUS；ROUGE-L；ROUGE-S；自动元评价 　　中图分类号：TP18 文献标识码：A 文章编号：1009-3044（2017）05-0138-05 　　Abstract： This paper addressed an automatic meta-evaluation method of machine translation （MT） evaluation based on ORANGE-Limited ORANGE. It was adopted as the resources were limited， to evaluate the performances of the three metrics- BLEUS， ROUGE-L and ROUGE-S， which were based on n-gram and widely used in the domain of MT evaluation. Compared with the traditional method operated by human， this method can evaluate metrics automatically without extra human involvement other than using a set of reference translations. It only needed the average rank of the reference， and it was not influenced by the subjective factors. This method costed less and expended less time than the traditional one， which would have advantages to the parameter tuning and shortening system period. In this paper， we use this automatic meta-evaluation method to evaluate BLEUS， ROUGE-L and ROUGE-S on the Russian-Chinese dataset. The result shows that the ROUGE-Ls performance is the best and that of BLEUS is the worst， the ROUGE-S is worse than ROUGE-L but better than BLEUS. Also， the consistance and effectiveness of Limited ORANGE are verified through consistency test with human meta-evaluation results. 　　Key words： LORANGE； BLEUS； ROUGE-L； ROUGE-S； Automatic Meta-evaluation 　　1 概述 　　?C器翻译系统评测在机器翻译领域起着重要的作用。它能够对系统性能进行评估，使系统开发者及时得知系统存在的问题并加以改进；同时可以为用户提供系统译文质量评估的依据。国家语言文字工作委员会发布的机器翻译系统评测规范[1]指出，机器翻译评测一般分为人工评测和自动评测两类。其中，人工评测主要由同时精通源语言和目标语言的专家主观地对系统输出译文的忠实度和流利度进行打分；主观性强，受外界因素影响比较大，代价高昂，评判周期长，无法适应机器翻译系统修改及参数调整的进度，无形中延长了系统研发周期，难以向研发者和用户提供高效的评测结果。于是，研究者更倾向于使用自动评测方法对系统译文进行量化评估。因此自动