龙星计划课程信检索 Overview of Text Retrieval.pptVIP

龙星计划课程:信息检索?? Overview of Text Retrieval Outline Basic Concepts in TR Evaluation of TR Common Components of a TR system Vector Space Retrieval Model Implementation of a TR System Applications of TR techniques Basic Concepts in TR What is Text Retrieval (TR)? There exists a collection of text documents User gives a query to express the information need A retrieval system returns relevant documents to users Known as “search technology” in industry TR vs. Database Retrieval Information Unstructured/free text vs. structured data Ambiguous vs. well-defined semantics Query Ambiguous vs. well-defined semantics Incomplete vs. complete specification Answers Relevant documents vs. matched records TR is an empirically defined problem! TR is Hard! Under/over-specified query Ambiguous: “buying CDs” (money or music?) Incomplete: what kind of CDs? What if “CD” is never mentioned in document? Vague semantics of documents Ambiguity: e.g., word-sense, structural Incomplete: Inferences required Even hard for people! 80% agreement in human judgments TR is “Easy”! TR CAN be easy in a particular case Ambiguity in query/document is RELATIVE to the database So, if the query is SPECIFIC enough, just one keyword may get all the relevant documents PERCEIVED TR performance is usually better than the actual performance Users can NOT judge the completeness of an answer History of TR on One Slide Birth of TR 1945: V. Bush’s article “As we may think” 1957: H. P. Luhn’s idea of word counting and matching Indexing Evaluation Methodology (1960’s) Smart system (G. Salton’s group) Cranfield test collection (C. Cleverdon’s group) Indexing: automatic can be as good as manual (controlled vocabulary) TR Models (1970’s 1980’s) … Large-scale Evaluation Applications (1990’s-Present) TREC (D. Harman E. Voorhees, NIST) Web search, PubMed, … Boundary with related areas are disappearing Short vs. Long Term Info Need Short-term information need (Ad hoc retrieval) “Temporary need”, e.g., info about