Web信息体系结构 File organization.pptVIP

File organization File Organizations (Indexes) Choices for accessing data during query evaluation Scan the entire collection Typical in early (batch) retrieval systems Computational and I/O costs are O(characters in collection) Practical for only “small” text collections Large memory systems make scanning feasible Use indexes for direct access Evaluation time O(query term occurrences in collection) Practical for “large” collections Many opportunities for optimization Hybrids: Use small index, then scan a subset of the collection Indexes What should the index contain? Database systems index primary and secondarykeys This is the hybrid approach Index provides fast access to a subset of database records Scan subset to find solution set IR Problem: Cannot predict keys that people will use in queries Every word in a document is a potential search term IR Solution: Index by all keys (words) ?full text indexes Indexes Index is accessed by the atoms of a query language, The atoms are called “features” or “keys” or “terms” Most common feature types: – Words in text, punctuation – Manually assigned terms (controlled and uncontrolled vocabulary) – Document structure (sentence and paragraph boundaries) – Inter- or intra-document links (e.g., citations) Composed features – Feature sequences (phrases, names, dates, monetary amounts) – Feature sets (e.g., synonym classes) Indexing and retrieval models drive choices – Must be able to construct all components of those models Indexing choices (there is no “right” answer) Tokenization,Case folding (e.g., New vs new, Apple vs apple), Stopwords (e.g., the, a, its), Morphology (e.g., computer, computers, computing, computed) Index granularity has a large impact on speed and effectiveness Index Contents The contents depend upon the retrieval model Feature presence/absence Boolean Statistical (tf, df, ctf, doclen, maxtf) Often about 10% the size of the raw data, compressed Positional Feature location within document Granularities incl