latent semantic indexing a probabilistic analysis.pptVIP

Latent Semantic Indexing: A probabilistic Analysis Christos Papadimitriou Prabhakar Raghavan, Hisao Tamaki, Santosh Vempala Motivation Application in several areas: querying clustering, identifying topics Other: synonym recognition (TOEFL..) Psychology test essay scoring Motivation Latent Semantic Indexing is Latent: Captures associations which are not explicit Semantic: Represents meaning as a function of similarity to other entities Cool: Lots of spiffy applications, and the potential for some good theory too Overview IR and two classical problems How LSI works Why LSI is effective: A probabilistic analysis Information Retrieval Text corpus with many documents (docs) Given a query, find relevant docs Classical problems: synonymy: missing docs with reference to “automobile” when querying on “car” polysemy: retrieving docs on internet when querying on “surfing” Solution: Represent docs (and queries) by their underlying latent concepts Information Retrieval Represent each document as a word vector Represent corpus as term-document matrix (T-D matrix) A classical method: Create new vector from query terms Find documents with highest dot-product Document vector space Latent Semantic Indexing(LSI) Process term-document (T-D) matrix to expose statistical structure Convert high-dimensional space to lower-dimensional space, throw out noise, keep the good stuff Related to principal component analysis (PCA), multiple dimensional scaling (MDS) Parameters U = universe of terms n = number of terms m = number of docs A = n x m matrix with rank r columns represent docs rows represent terms Singular Value Decomposition(SVD) SVD r is the rank of A D diagonal matrix of the r singular values U and V matrices composed of orthonormal columns SVD is always possible numerical methods for SVD exist run time: O(m n c), where c denotes the average number of words per document T-D Matrix Approximation Synonymy LSI used in several ways: e.g. detecting synonymy A measure of simil