搜索引擎技术闫宏飞北京大学计算机系网络实验室.ppt

搜索引擎技术 闫宏飞，yhf@ 北京大学计算机系网络实验室 2004年12月24日@CERNET2004 内容提要 搜索引擎工作原理 信息检索相关研究和机构 搜索引擎 — Web Search Engines 定义：允许用户递交查询，检索出与查询相关的网页结果列表，并且排序输出。 创建索引的方法 手工索引 自动索引 系统结构 集中式体系结构 分布式体系结构 搜索引擎三段式工作流程 搜集 批量搜集，增量式搜集；搜集目标，搜集策略 预处理 关键词提取；重复网页消除；链接分析；索引 服务 查询方式和匹配；结果排序；文档摘要 搜索引擎系统流程 天网搜索引擎系统流程 分布式Web搜集系统结构 天网存储格式 File Organizations (Indexes) Choices for accessing data during query evaluation Scan the entire collection Typical in early (batch) retrieval systems Computational and I/O costs are O(characters in collection) Practical for only “small” text collections Large memory systems make scanning feasible Use indexes for direct access Evaluation time O(query term occurrences in collection) Practical for “large” collections Many opportunities for optimization Hybrids: Use small index, then scan a subset of the collection Indexes What should the index contain? Database systems index primary and secondarykeys This is the hybrid approach Index provides fast access to a subset of database records Scan subset to find solution set IR Problem: Cannot predict keys that people will use in queries Every word in a document is a potential search term IR Solution: Index by all keys (words) ?full text indexes Index Contents The contents depend upon the retrieval model Feature presence/absence Boolean Statistical (tf, df, ctf, doclen, maxtf) Often about 10% the size of the raw data, compressed Positional Feature location within document Granularities include word, sentence, paragraph, etc Coarse granularities are less precise, but take less space Word-level granularity about 20-30% the size of the raw data,compressed Indexes: Implementation Common implementations of indexes Bitmaps Signature files Inverted files Common index components Dictionary (lexicon) Postings document ids word positions Inverted Files Inverted Files Word-Level Inverted File Inverted Search Algorithm Find query elements (terms) in the lexicon Retrieve postings for each lexicon entry Manipulate postings according t