信息检索语言模型中的平滑方法(Smoothing_Methods_for_LM_in_IR).ppt

Smoothing Methods for LM in IR Alejandro Figueroa Outline The linguistic phenomena behind the retrieval of documents. Language Modeling Approach. Smoothing methods. Overview. Methods. Parameters setting. Interpolation vs. Back-off. Comparison of methods. Combination of methods. Personal outlook and conclusions. The Linguistic Phenomena behind IR. Information Variation The problem: simply keyword matching is not enough to retrieve the best documents for a query. For example: ?When was Albert Einstein born?? The nobel prize of physics Albert Einstein was born in 1879 in Ulm, Germany. Born: 14 March 1879 in Ulm, Württemberg, Germany. Physics nobel prize Albert Einstein was born at Ulm, in Württemberg, Germany, on March 14, 1879. Died 18 Apr 1955 (born 14 Mar 1879) German-American physicist. The same information can be found in several ways: Information Variation Kinds of variation: Graphic: 14 March 1879“ and 14 Mar 1879“. Morphological:” Physics nobel prize“ Syntactical: “German-American physicist“ Semantic:Albert Einstein was born at Ulm“ and German-American physicist“. Appropriateness: Precision. Economy. Language Modeling Approach Language Modeling The probability that a query Q was generated by a probabilistic model based on a document. Language Modeling Smoothing methods makes use of two probabilites for the model Pu(w|d) and Ps(w|d). Language Modeling Smoothing Methods Overview The problem: Adjust the MLE to compensate data sparseness. The role of smoothing is: LM more accurate. Explain the non-informative words in the query. Goal of the work: How sensitive is retrieval performance to the smoothing of a document LM? How should be the model and the parameters chosen? Overview The unsmoothed model is the MLE: Overview Smoothing: tackles the effect of statistical variability in small training sets. Discounting: the relative frequencies of seen events are discounted; the gained probability mass is then distributed over the unseen words. Smoothing Methods Base