中国计算机学会科前沿讲习班信息检索 Retrieval.pptVIP

* * * * * FSNLP: foundation of statistic Natural language processing * * The user has a document in mind, and generates the query from this document. The equation represents the probability that the document that the user had in mind was in fact this one. * * * * * Relevance model other than query model, Query是它产生的small sample, relevant documents是它产生的大sample * equivalent to query-likelihood approach if simple empirical distribution used for query model More general risk minimization framework has been proposed Zhai and Lafferty 2001 * * * * * Still Bag of words model Here log-scaled tf.idf This values are not calculated on true data Df： 中国 24,100 日本 1860 文化 19600 留学生 1270 * The vector space model * A vector can be normalized (given a length of 1) by dividing each of its components by its length – here we use the L2 norm This maps vectors onto the unit sphere: * Some contents from /~nicholas/676/12LSI.ppt IIR Ch18 Latent : to lie hidden Indexing by Latent Semantic Analysis (1990) by Scott Deerwester ,? Susan T. Dumais ,? George W. Furnas ,? Thomas K. Landauer ,? Richard Harshman Journal of the American Society for Information Science * * /seoptimiser-link-building.php f * Ambiguity and association in natural language * Goals of LSI Similar terms map to similar location in low dimensional space Noise reduction by dimension reduction * T ? maps terms into LSI space ?DT maps documents into LSI space * * minimize the Frobenius norm of the matrix difference X = W ? Wk IIR 18.15 * Similar terms map to similar location in low dimensional space Noise reduction by dimension reduction * Latent Semantic Analysis via SVD * * * * stop words are used uniformly throughout collection, so they tend to appear in the first dimension QUESTION: 我觉得是少了个括号（T?）-1 因为原矩阵S=T*Sigma*D?,现在有S，要求D，就必须左乘矩阵（T?）-1， 这样等式右侧就为D了。 ANSWER: you are right, 新文档空间的向量qn有： T*sigma*qn = q 所以 qn = inv(T*sigma)*q ? inv(T*sigma) = inv(sigma)*inv(T) = inv(sigma)* T 实际上sigma对对角阵，有sigma= sigma;