Evaluation of Machine Translation Output for an Unknown Source Language Report of an ISLEB.pdfVIP

Evaluation of Machine Translation Output for an Unknown Source Language: Report of an ISLE-Based Investigation Keith J. Miller, The MITRE Corporation, Washington D.C., USA keith@ Donna M. Gates, Carnegie Mellon University, Pittsburgh, USA dmg@ Nancy Underwood, CST, Denmark nancy@cst.ku.dk Josemina Magdalen, The Hebrew University of Jerusalem, Josemina@ Abstract It is often assumed that knowledge of both the source and target languages is necessary in order to evaluate the output of a machine translation (MT) system. This paper reports on an experimental evaluation of Chinese-English MT and Spanish-English MT from output specifically designed for evaluators who do not read or speak Chinese or Spanish. An outline of the characteristics measured and evaluation follows. Keywords Machine Translation, Gisting, Comprehensibility, Fidelity, Evaluation task is the end user trying to accomplish? Following this Introduction suggestion, we spent not an inconsiderable amount of time defining our hypothetical user and the task for which that user had chosen to consider the use of machine The evaluation of the performance of machine translation systems is a topic that has instigated much discussion. translation. This time was not wasted – it helped us to This is partly because in addition to all of the usual concretize many of