Search Engines Information Retrieval in Practice PPT chap8.ppt

Discounted Cumulative Gain DCG is the total gain accumulated at a particular rank p: Alternative formulation: used by some web search companies emphasis on retrieving highly relevant documents DCG Example 10 ranked documents judged on 0-3 relevance scale: 3, 2, 3, 0, 0, 1, 2, 2, 3, 0 discounted gain: 3, 2/1, 3/1.59, 0, 0, 1/2.59, 2/2.81, 2/3, 3/3.17, 0 = 3, 2, 1.89, 0, 0, 0.39, 0.71, 0.67, 0.95, 0 DCG: 3, 5, 6.89, 6.89, 6.89, 7.28, 7.99, 8.66, 9.61, 9.61 Normalized DCG DCG numbers are averaged across a set of queries at specific rank values e.g., DCG at rank 5 is 6.89 and at rank 10 is 9.61 DCG values are often normalized by comparing the DCG at each rank with the DCG value for the perfect ranking makes averaging easier for queries with different numbers of relevant documents NDCG Example Perfect ranking: 3, 3, 3, 2, 2, 2, 1, 0, 0, 0 ideal DCG values: 3, 6, 7.89, 8.89, 9.75, 10.52, 10.88, 10.88, 10.88, 10 NDCG values (divide actual by ideal): 1, 0.83, 0.87, 0.76, 0.71, 0.69, 0.73, 0.8, 0.88, 0.88 NDCG ￡ 1 at any rank position Using Preferences Two rankings described using preferences can be compared using the Kendall tau coefficient (τ ): P is the number of preferences that agree and Q is the number that disagree For preferences derived from binary relevance judgments, can use BPREF BPREF For a query with R relevant documents, only the first R non-relevant documents are considered dr is a relevant document, and Ndr gives the number of non-relevant documents Alternative definition Efficiency Metrics Significance Tests Given the results from a number of queries, how can we conclude that ranking algorithm A is better than algorithm B? A significance test enables us to reject the null hypothesis (no difference) in favor of the alternative hypothesis (B is better than A) the power of a test is the probability that the test will reject the null hypothesis correctly increasing the number of queries in the experiment also increases power of test Significance T