IndexingandRepresentationTheVectorSpaceModel.pptVIP

Slide courtesy Ray Larson Indexing and Representation:The Vector Space Model Document represented by a vector of terms Words (or word stems) Phrases (e.g. computer science) Removes words on “stop list” Documents aren’t about “the” Often assumed that terms are uncorrelated. Correlations between term vectors implies a similarity between documents. For efficiency, an inverted index of terms is often stored. Document RepresentationWhat values to use for terms Boolean (term present /absent) tf (term frequency) - Count of times term occurs in document. The more times a term t occurs in document d the more likely it is that t is relevant to the document. Used alone, favors common words, long documents. df document frequency The more a term t occurs throughout all documents, the more poorly t discriminates between documents tf-idf term frequency * inverse document frequency - High value indicates that the word occurs more often in this document than average. Document Vectors Documents are represented as “bags of words” Represented as vectors when used computationally A vector is like an array of floating point Has direction and magnitude Each vector holds a place for every term in the collection Therefore, most vectors are sparse Vector Representation Documents and Queries are represented as vectors. Position 1 corresponds to term 1, position 2 to term 2, position t to term t Document Vectors nova galaxy heat h’wood film role diet fur 1.0 0.5 0.3 0.5 1.0 1.0 0.8 0.7 0.9 1.0 0.5 1.0 1.0 0.9 1.0 0.5 0.7 0.9 0.6 1.0 0.3 0.2 0.8 0.7 0.5 0.1 0.3 Assigning Weights Want to weight terms highly if they are frequent in relevant documents … BUT infrequent in the collection as a whole Assigning Weights tf x idf measure: term frequency (tf) inverse document frequency (idf) tf x idf Normalize the term weights (so longer do