lecture2 词项词典课件.ppt

Document ingestion;Recall the basic indexing pipeline;Parsing a document;Complications: Format/language;Complications: What is a document?;Tokens;Tokenization;Tokenization;Numbers;Tokenization: language issues;Tokenization: language issues;Tokenization: language issues;Terms The things indexed in an IR system;Stop words;Normalization to terms;Normalization: other languages;Normalization: other languages;Case folding;Normalization to terms;Thesauri and soundex;Stemming and Lemmatization;Lemmatization;Stemming;Porter’s algorithm;Typical rules in Porter;Other stemmers;Language-specificity;Does stemming help?;Faster postings merges:Skip pointers/Skip lists;Recall basic merge;Augment postings with skip pointers (at indexing time);Query processing with skip pointers;Where do we place skips?;Placing skips