Automated Summarization Evaluation with Basic Elements.pdfVIP

Automated Summarization Evaluation with Basic Elements Eduard Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto1 Information Sciences Institute University of Southern California Marina del Rey, CA 90292 {hovy, cyl, liangz}@ Abstract As part of evaluating a summary automatically, it is usual to determine how much of the contents of one or more human- produced ‘ideal’ summaries it contains. Previous automated methods such as ROUGE compare using fixed word ngrams, which are not ideal for a variety of reasons. In this paper we describe a framework in which summary evaluation measures can be instantiated and compared, and we implement a specific evaluation method using very small units of content, called Basic Elements, that address some of the shortcomings of ngrams. This method is tested on DUC 2003, 2004, and 2005 systems and produces very good correlations with human judgments. units), and BE scorers (that assign a score to each BE unit individually). 1. Introduction Experience in Machine Translation and automated 2.1. The BE Procedure speech recognition has shown the great value of an The problem of evaluating the content of a given automated evaluation measure for rapid system growth summary breaks into three distinct sub-problems, and improvement (Papineni et al., 2001