信息检索六tfidf汇总.pptVIP

Interaction: vectors and phrases Scoring phrases doesn’t fit naturally into the vector space world: “tangerine trees” “marmalade skies” Positional indexes don’t calculate or store tf.idf information for “tangerine trees” Biword indexes treat certain phrases as terms For these, we can pre-compute tf.idf. Theoretical problem of correlated dimensions Problem: we cannot expect end-user formulating queries to know what phrases are indexed We can use a positional index to boost or ensure phrase occurrence Vectors and Boolean queries Vectors and Boolean queries really don’t work together very well In the space of terms, vector proximity selects by spheres: e.g., all docs having cosine similarity ?0.5 to the query Boolean queries on the other hand, select by (hyper-)rectangles and their unions/intersections Round peg - square hole Vector spaces and other operators Vector space queries are apt for no-syntax, bag-of-words queries Clean metaphor for similar-document queries Not a good combination with Boolean, wild-card, positional query operators But … Query language vs. scoring May allow user a certain query language, say Free text basic queries Phrase, wildcard etc. in Advanced Queries. For scoring (oblivious to user) may use all of the above, e.g. for a free text query Highest-ranked hits have query as a phrase Next, docs that have all query terms near each other Then, docs that have some query terms, or all of them spread out, with tf x idf weights for scoring Efficient cosine ranking Find the k docs in the corpus “nearest” to the query ? k largest query-doc cosines. Efficient ranking: Computing a single cosine efficiently. Choosing the k largest cosine values efficiently. Can we do this without computing all n cosines? n = number of documents in collection Efficient cosine ranking What we’re doing in effect: solving the k-nearest neighbor problem for a query vector In general, we do not know how to do this efficiently for high-dimensional spaces But it is solvable for