Document ingestion.ppt

Typical rules in Porter sses ? ss ies ? i ational ? ate tional ? tion Weight of word sensitive rules (m1) EMENT → replacement → replac cement → cement Sec. 2.2.4 Other stemmers Other stemmers exist: Lovins stemmer p.lancs.ac.uk/computing/research/stemming/general/lovins.htm Single-pass, longest suffix removal (about 250 rules) Paice/Husk stemmer Snowball Full morphological analysis (lemmatization) At most modest benefits for retrieval Sec. 2.2.4 Language-specificity The above methods embody transformations that are Language-specific, and often Application-specific These are “plug-in” addenda to the indexing process Both open source and commercial plug-ins are available for handling these Sec. 2.2.4 Does stemming help? English: very mixed results. Helps recall for some queries but harms precision on others E.g., operative (dentistry) ? oper Definitely useful for Spanish, German, Finnish, … 30% performance gains for Finnish! Sec. 2.2.4 Faster postings merges:Skip pointers/Skip lists Recall basic merge Walk through the two postings simultaneously, in time linear in the total number of postings entries 128 31 2 4 8 41 48 64 1 2 3 8 11 17 21 Brutus Caesar 2 8 If the list lengths are m and n, the merge takes O(m+n) operations. Can we do better? Yes (if the index isn’t changing too fast). Sec. 2.3 Augment postings with skip pointers (at indexing time) Why? To skip postings that will not figure in the search results. How? Where do we place skip pointers? 128 2 4 8 41 48 64 31 1 2 3 8 11 17 21 31 11 41 128 Sec. 2.3 Query processing with skip pointers 128 2 4 8 41 48 64 31 1 2 3 8 11 17 21 31 11 41 128 Suppose we’ve stepped through the lists until we process 8 on each list. We match it and advance. We then have 41 and 11 on the lower. 11 is smaller. But the skip successor of 11 on the lower list is 31, so we can skip ahead past the intervening postings. Sec. 2.3 Where do we place skips? Tradeoff: More skips ? shorter skip spans ? more likely to skip. But lots of c