Chapter 3 Retrieval Evaluation.ppt

Hsin-Hsi Chen Chapter 3 Retrieval Evaluation Hsin-Hsi Chen Department of Computer Science and Information Engineering National Taiwan University Evaluation Function analysis Time and space The shorter the response time, the smaller the space used, the better the system is Performance evaluation (for data retrieval) Performance of the indexing structure The interaction with the operating systems The delays in communication channels The overheads introduced by software layers Performance evaluation (for information retrieval) Besides time and space, retrieval performance is an issue Retrieval Performance Evaluation Retrieval task Batch mode The user submits a query and receives an answer back How the answer set is generated Interactive mode The user specifies his information need through a series of interactive steps with the system Aspects User effort characteristics of interface design guidance provided by the system duration of the session Recall and Precision Recall the fraction of the relevant documents which has been retrieved Precision the fraction of the retrieved documents which is relevant precision versus recall curve The user is not usually presented with all the documents in the answer set A at once ExampleRq={d3,d5,d9,d25,d39,d44,d56,d71,d89,d123}Ranking for query q by a retrieval algorithm1. d123 ? 6. d9 ? 11. d382. d84 7. d511 12. d483. d56 ? 8. d129 13. d2504. d6 9. d187 14. d1135. d8 10. d25 ? 15. d3 ? 11 standard recall levels for a query precision versus recall based on 11 standard recall levels: 0%, 10%, 20%, …, 100% 11 standard recall levels for several queries average the precision figures at each recall level P(r): the average precision at the recall level r Nq: the number of queries used Pi(r): the precision at recall level r for the i-th query necessity ofinterpolation procedure Rq={d3,d56,d129}1. d123 6. d9 11. d382. d84 7. d511 12. d483. d56 ? 8. d129 ? 13. d2504. d6 9. d187 14. d1135. d8 10. d25 15.