5-搜索引擎.pptVIP

LOGO LOGO Web Mining(5) 杨光飞 系统工程研究所 大连理工大学 gfyang@ Search Engine 搜索引擎概况 The search engine rankings for January 2012, according to comScore, were: Google?grew to 66.2 percent (up from 65.9 percent in December 2011). Bing?grew to 15.2 percent (up from 15.1 percent). Yahoo?fell to 14.1 percent (down from 14.5 percent). Ask?grew to 3 percent (up from 2.9 percent). AOL?remained unchanged at 1.6 percent. Search Engine Characteristics Unedited – anyone can enter content Quality issues; Spam Varied information types Phone book, brochures, catalogs, dissertations, news reports, weather, all in one place! Different kinds of users Lexis-Nexis: Paying, professional searchers Online catalogs: Scholars searching scholarly literature Web: Every type of person with every type of goal Scale Hundreds of millions of searches/day; billions of docs Web Search Queries Web search queries are short: ~2.4 words on average (Aug 2000) Has increased, was 1.7 (~1997) User Expectations: Many say “The first item shown should be what I want to see!” This works if the user has the most popular/common notion in mind, not otherwise. Standard Web Search Engine Architecture crawl the web create an inverted index Check for duplicates, store the documents Inverted index Search engine servers user query Show results To user DocIds Brin Page 98 Inverted Indexes Inverted indexes are still used, even though the web is so huge Some systems partition the indexes across different machines; each machine handles different parts of the data Other systems duplicate the data across many machines; queries are distributed among the machines Most do a combination of these In this example, the data for the pages is partitioned across machines. Additionally, each partition is allocated multiple machines to handle the queries. Each row can handle 120 queries per second Each column can handle 7M pages To handle more queries, add another row. How Inverted Files Are Created Periodically rebuilt, static otherw