IR5_so的ngruihua.pptxVIP

Evaluation in Information Retrieval Speaker: Ruihua Song Web Data Management Group, MSR Asia Call For Papers of EVIA 2010 Test collection formation, evaluation metrics, and evaluation environments Statistical issues in retrieval evaluation - User studies and the evaluation of human-computer interaction in information retrieval (HCIR) Evaluation methods for multilingual, multimedia, or mobile information access Novel information access tasks and their evaluation - Evaluation and assessment using implicit user feedback, crowdsourcing, living labs, or inferential methods Evaluation issues in industrial and enterprise retrieval systems Outlines Basics on IR evaluation Introduction of TREC (Text Retrieval Conference) One selected paper Select-the-Best-Ones: A new way to judge relative relevance Motivated Examples Which set is better? S1={r, r, r, n, n} vs. S2={r, r, n, n, n} S3={r} vs. S4={r, r, n} Which ranking list is better? L1=r, r, r, n, n vs. L2=n, n, r, r, r L3=r, n, r, n, h vs. L4=h,n, n, r, r r: relevant n: non-relevant h: highly relevant Precision Recall Precision is fraction of the retrieved document which is relevant Recall is fraction of the relevant document which has been retrieved R (Relevant Set) A (Answer Set) Ra Precision Recall (cont.) Assume there are 10 relevant documents in judgments Example 1: S1={r, r, r, n, n} vs. S2={r, r, n, n, n} P1= 3/5 = 0.6; R1= 3/10 = 0.3 P2= 2/5 = 0.4; R2= 2/10 = 0.2 S1 S2 Example 2: S3={r} vs. S4={r, r, n} P3= 1/1 = 1; R3= 1/10 = 0.1 P4= 2/3 = 0.667; R4= 2/10 = 0.2 ? (F1-Measure) Example 3: L1=r, r, r, n, n vs. L2=n, n, r, r, r ? r: relevant n: non-relevant h: highly relevant Mean Average Precision Defined as the mean of Average Precision for a set of queries Example 3: L1=r, r, r, n, n vs. L2=n, n, r, r, r AP1=(1/1+2/2+3/3)/10=0.3 AP2=(1/3+2/4+3/5)/10=0.143 L1 L2 Other Metrics based on Binary Judgments P@10 (Precision at 10) is the number of relevant documents in the top 10 documents in the ranked list re