Columbia University in the Novelty Track at TREC.pdf

Columbia University in the Novelty Track at TREC 2004 Barry Schiffman Columbia University New York, N.Y. 10027 bschiff@ Kathleen R. McKeown Columbia University New York, N.Y. 10027 kathy@ ABSTRACT Our system for the Novelty Track at TREC 2004 looks be- yond sentence boundaries as well as within sentences to identify novel, nonduplicative passages. It tries to iden- tify text spans of two or more sentences that encompass mini-segments of new information. At the same time, we avoid any pairwise comparison of sentences, but rely on the presence of previously unseen terms to provide evidence of novelty. The system is guided by a number of parameters, both weights and thresholds, that are learned automatically with a randomized hill-climbing algorithm. During learning, we varied the target function to produce configurations that emphasize either precision or recall. We also implemented a straightforward vector-space model as a comparison and to test a combined approach. 1. INTRODUCTION The novelty detection problem seeks an automatic way of identifying any new information in a document, or docu- ments, on a given topic. It is a recent area of inquiry in the Natural Language Processing and Information Retrieval communities and has been explored at the last three meet- ings of the Text Retrieval Conference (TREC), in the Nov- elty Track. After the first Novelty Track in 2002, the National Institute of Standards and Technology (NIST) separated the track into four tasks, two of which combined passage retrieval and novelty filtering and two of which concentrated on novelty filtering, giving participants the choice of whether to do the combined tasks (Tasks 1 and/or 3) or whether to focus on the novelty detection alone (Tasks 2 and/or 4). In the com- bined tasks, participants have to first choose the sentences that are “relevant” to a given topic from a set of documents, and then make a second pass to remove duplicates. In the pure novelty task, participants are given an order