Evaluation in information retrieval.ppt

12 September 2000 ESSIR 2000 Evaluation in information retrieval Stephen Robertson Microsoft Research Ltd., Cambridge, U.K. Summary The traditional IR evaluation experiment up to and including TREC and a range of problems and issues arising Interactive retrieval Routing and filtering The traditional IR experiment To start with you need: An IR system (or two) A collection of documents A collection of requests Then you run your experiment: Input the documents Put each request to the system Collect the output The traditional IR experiment Then you need to: Evaluate the output, document by document Discover (??) the good documents your system has missed Analyse the results What is a document? Traditionally: a package of information structured by an author The traditional IR experiment What is a request? Traditionally, a description of a topic of interest More properly, a partial representation of an underlying information need or problem (ASK) What is a system? Traditionally, a device which accepts a request and delivers or identifies documents (Note: “device” may be an organisation, may involve people) The traditional IR experiment Possibly bad assumptions about systems: System is pure input-output device (put in the request, get out the answer set) most real searches involve interaction System is program this implies that the user is outside the system – more on this later there are certainly other humans involved (e.g. authors, indexers) The traditional IR experiment Why do we need a complete system? Many tests are really about components But we do not in general know how to evaluate components What is a good (relevant) document? Traditionally, one judged (by an expert) to be on the topic More properly, one judged by the user to be helpful in resolving her/his problem The traditional IR experiment Possibly bad assumptions about relevance: Relevance is binary users are often uncomfortable with yes/no relevance Relevance of a single document can be judged independently