龙星计划课程信息检索OverviewofTextRetrievalPart.pptVIP

龙星计划课程:信息检索?? Overview of Text Retrieval: Part 2 Outline Other retrieval models Implementation of a TR System Applications of TR techniques P-norm (Extended Boolean)(Salton et al. 83) Motivation: how to rank documents with a Boolean query? Intuitions Docs satisfying the query constraint should get the highest ranks Partial satisfaction of query constraint can be used to rank other docs Question: How to capture “partial satisfaction”? P-norm: Basic Ideas Normalized term weights for doc rep ([0,1]) Define similarity between a Boolean query and a doc vector P-norm: Formulas P-norm: Summary A general (and elegant) similarity function for Boolean query and a regular document vector Connecting Boolean model and vector space model with models in between Allowing different “confidence” on Boolean operators (different p for different operators) A model worth more exploration (how to learn optimal p values from feedback?) Probabilistic Retrieval Models Overview of Retrieval Models The Basic Question What is the probability that THIS document is relevant to THIS query? Probability of Relevance Three random variables Query Q Document D Relevance R ? {0,1} Goal: rank D based on P(R=1|Q,D) Evaluate P(R=1|Q,D) Actually, only need to compare P(R=1|Q,D1) with P(R=1|Q,D2), I.e., rank documents Several different ways to refine P(R=1|Q,D) Refining P(R=1|Q,D) Method 1: conditional models Basic idea: relevance depends on how well a query matches a document Define features on Q x D, e.g., #matched terms, # the highest IDF of a matched term, #doclen,.. P(R=1|Q,D)=g(f1(Q,D), f2(Q,D),…,fn(Q,D), ?) Using training data (known relevance judgments) to estimate parameter ? Apply the model to rank new documents Special case: logistic regression Logistic Regression (Cooper 92, Gey 94) Features/Attributes Logistic Regression: Pros Cons Advantages Absolute probability of relevance available May re-use all the past relevance judgments Problems Performance is very sensitive to the selection