Combining Systems and Databases A Search Engine Retrospective结合系统和数据库的搜索引擎回顾.ppt

Combining Systems and Databases: A Search Engine Retrospective Reviewed By: Rooma Rathore Rohini Prinja Overview: Problem Problem Statement Why is the problem important Why is the problem hard Approaches Contributions of the paper Assumptions Validations Rewrite Problem Statement: Given: Current Search Engines and DBMS Find An efficient search engine design including A schema to store data A query language Implementation of query mechanism Ways to leverage the Database principles in designing Data-Intensive (DI) applications without necessarily using the same semantics. Objectives Make use of database principles Efficient (Fast) Cost effective Scalable Constraints Highly available Why is the problem important? Widespread use of Search Engines (SEs). Amount of data to be searched is increasing every second Hence requires scalable design Glimpse of real-data (2005) # of documents - 3Billion Data – 10TB Queries/day – 150 Million Why is the problem hard? As mentioned in [2], very little research is done in the area of search engines. The documents/items to be searched are of the order of several billions. “Search Items” changes over the time from plain text to multimedia these days. Contributions: Discusses the challenges of designing a SE. Ranking Documents Ranking Query Results Availability Freshness of Data What principles of DBMS can be (should be) applied when designing DI applications like search engines: Top-Down Design Data Independence Declarative Query Language Contributions (contd.): Why SE’s can not be implemented as DBMS in true sense: Key Concepts: Ranking and scoring of Documents Word vs. Property Matching Query Q = {w1, w2, w3…. wk} Score(Q,d) ? Quality(d) + ? Score(wi, d) Where Quality(d) is the quality of document independent of query words Key Concepts: Proposed Design for SE’s Overview of SE design: Crawl, Index, Serve Query (read-only) Scoring of documents and words Making a Query Plan Query Implementation Access Methods and Physica