Towards Information Retrieval Measures for Evaluation of Web Search Engines. Unpublished ma.pdfVIP

Towards Information Retrieval Measures for Evaluation of Web Search Engines Jacek Gwizdka and Mark Chignell Department of Mechanical and Industrial Engineering University of Toronto 5 Kings College Rd, Toronto, ON M5S 3G8, CANADA jacek@ie.utoronto.ca, chignel@mie.utoronto.ca Abstract Information retrieval on the Web is very different from retrieval in traditional indexed databases. This difference arises from: the high degree of dynamism of the Web; its hyper-linked character; the absence of a controlled indexing vocabulary; the heterogeneity of document types and authoring styles; the easy access that different types of users may have to it. Thus, since Web retrieval is substantially different from information retrieval, new or revised evaluative measures are required to assess retrieval performance using Web search engines. This paper suggests a number of different measures to evaluate information retrieval from the Web. The motivation behind each of these measures is presented, along with their descriptions and definitions. In the second part of the paper, application of these measures is illustrated in the evaluation of three search engines. The purpose of this paper is not to give the definite prescription for evaluating information retrieval from the Web, but rather to present some examples and to initiate a wider discussion of how to enhance measures of Web search performance. Keywords Web information retrieval, information retrieval measures, evaluation, search engines, precision, user interaction, search length. Introduction Information retrieval systems have been evaluated and compared for many years. Cleverdon (1966) listed six criteria that could be used to evaluate an information retrieval system: (1) coverage, (2) time lag, (3) recall, (4) precision , (5) presentation and (6) user