cross-evaluation a new strongmodelstrong for information strongsystemstrong.pdfVIP

Cross-Evaluation: A New Model for Information System Evaluation Ying Sun and Paul B. Kantor 4 Huntington Street, Rutgers University, New Brunswick, NJ 08903. E-mail: {ysun, kantor}@ In this article, we introduce a new information system objective notion of being “related to the stated topic.” This evaluation method and report on its application to a col- approach was pioneered by Cyril Cleverdon in the Cranfield laborative information seeking system, AntWorld. The Experiments (Cleverdon, 1960). It has evolved into the suc- key innovation of the new method is to use precisely the same group of users who work with the system as cessful and influential Text REtrieval Conference (TREC) judges, a system we call Cross-Evaluation. In the new evaluation framework. There is no question that the existence method, we also propose to assess the system at the of TREC, and the applicability of this method, has con- level of task completion. The obvious potential limitation tributed substantially to the growth of information systems. of this method is that individuals may be inclined to At the same time, there have been efforts to extend this think more highly of the materials that they themselves have found and are almost certain to think more highly fundamental paradigm. However, in most cases (an anony- of their own work product than they do of the products mous reviewer has reminded us that this was not a constraint built by others. The keys to neutralizing this problem are of the early Cranfield work) a document is considered to be careful design and a corresponding analytical model objectively either relevant or not relevant for a task. This based on analysis of variance. We model the several