现代信息检索——倒排文件(2009).pptVIP

IR Components: Indexing and Searching Contents Indexing techniques: Inverted files Suffix arrays Signature files Techniques used to search each type of index Motivation Just like in traditional RDBMS searching for data may be costly In a RDBMS one can take (a lot of) advantage from the well defined structure of (and constraints on) the data Linear scan of the data is not feasible for non-trivial datasets (real life) Motivation Traditional indices, e.g., B-trees, are not well suited for IR Main approaches: Inverted files (or lists) Suffix arrays Signature files Inverted Files There are two main elements: vocabulary – set of unique terms Occurrences – where those terms appear Inverted Files The number of indexed terms is often several orders of magnitude smaller when compared to the documents size (Mbs vs Gbs) The space consumed by the occurrence list is not trivial. Each time the term appears it must be added to a list in the inverted file That may lead to a quite considerable index overhead Inverted Files - layout How Are Inverted Files Created Documents are parsed to extract words (or stems) and these are saved with the Document ID. How Inverted Files are Created After all document have been parsed the inverted file is sorted How Inverted Files are Created Multiple term entries for a single document are merged and frequency information added Inverted Files The file is commonly split into a Dictionary and a Postings file How to Index Word Positions Text: Inverted file Inverted Files - construction Building the index in main memory is not feasible (wouldn’t fit, and swapping would be unbearable) Building it entirely in disk is not a good idea either (would take a long time) One idea is to build several partial indices in main memory, one at a time, saving them to disk and then merging all of them to obtain a single index Dictionary Trie Inverted Files - construction This procedure takes O(n log (n/M)) time plus O(n) to build the partial indices – where