RetrievalEvaluation.pptVIP

Retrieval Evaluation Modern Information Retrieval, Chapter 3 Ricardo Baeza-Yates, Berthier Ribeiro-Neto 圖書與資訊學刊第29期(1999年5月), 台大圖資所碩士論文, 江玉婷，陳光華 Outline Introduction Retrieval Performance Evaluation Recall and precision Alternative measures Reference Collections TREC Collection CACMISI Collection CF Collection Trends and Research Issues Introduction Type of evaluation Functional analysis phase, and Error analysis phase Performance evaluation Performance evaluation Response time/space required Retrieval performance evaluation The evaluation of how precise is the answer set Retrieval Performance Evaluation 評估以batch query 為主的IR 系統 Precision versus recall curve Rq={d3,d5,d9,d25,d39,d44,d56, d71,d89,d123} P=100% at R=10% P= 66% at R=20% P= 50% at R=30% Precision versus recall curve For a single query Average Over Multiple Queries P(r)=average precision at the recall level r Nq= Number of queries used Pi(r)=The precision at recall level r for the i-th query Interpolated precision Rq={d3,d56,d129} P=33% at R=33% P= 25% at R=66% P= 20% at R=100% P(rj)=max ri≦ r≦ rj+1P(r) Interpolated precision Let rj, j?{0, 1, 2, …, 10}, be a reference to the j-th standard recall level P(rj)=max ri≦ r≦ rj+1P(r) Average recall vs. precision figure Single Value Summaries Average precision versus recall: Compare retrieval algorithms over a set of example queries Sometimes we need to compare individual query’s performance Average precision可能會隱藏演算法中不正常的部分 可能需要知道, 兩個演算法中，對某特定query的performance為何 Need a single value summary The single value should be interpreted as a summary of the corresponding precision versus recall curve Single Value Summaries Average Precision at Seen Relevant Documents Averaging the precision figures obtained after each new relevant document is observed. Example: Figure 3.2, (1+0.66+0.5+0.4+0.3)/5=0.57 此方法對於很快找到相關文件的系統是相當有利的 (相關文件被排在越前面, precision值越高) R-Precision The precision at the R-th position in the ranking R: the total number of re