Generic Sentence Fusion is an Ill-Defined Summarization Task.pdf

Generic Sentence Fusion is an Ill-Defined Summarization Task Hal Daume? III and Daniel Marcu Information Sciences Institute University of Southern California 4676 Admiralty Way, Suite 1001 Marina del Rey, CA 90292 {hdaume,marcu}@ Abstract We report on a series of human evaluations of the task of sentence fusion. In this task, a hu- man is given two sentences and asked to produce a single coherent sentence that contains only the important information from the original two. Thus, this is a highly constrained summariza- tion task. Our investigations show that even at this restricted level, there is no measurable agreement between humans regarding what in- formation should be considered important. We further investigate the ability of separate eval- uators to assess summaries, and find similarly disturbing lack of agreement. 1 Introduction and Motivation The practices of automatic summarization vary widely across many dimensions, including source length, summary length, style, source, topic, language, and structure. Most typical are summaries of a single news document down to a headline or short summary, or of a collection of news documents down to a headline or short summary (Hahn and Harman, 2002). A few re- searchers have focused on other aspects of sum- marization, including single sentence (Knight and Marcu, 2002), paragraph or short document (Daume? III and Marcu, 2002), query-focused (Berger and Mittal, 2000), or speech (Hori et al., 2003). The techniques relevant to, and the challenges faced in each of these tasks can be quite dif- ferent. Nevertheless, they all rely on one crit- ical assumption: there exists a notion of (rel- ative) importance between pieces of informa- tion in a document (or utterance), regardless of whether we can detect this or not. Indeed, recent research has looked at this question in de- tail, and can be rather cleanly divided into two partitions. The first partition aims to develop manual evaluation criteria for determining the quality o