云计算概述及Google搜索引擎系统结构解析.ppt

* * * * * * * * * Enter Title of Presentation Here Google Confidential * * * * * * * The result is split into a Dictionary file and a Postings file. Where do we pay in storage? Pointers Terms Query processing: AND Consider processing the query: Brutus AND Caesar Locate Brutus in the Dictionary; Retrieve its postings. Locate Caesar in the Dictionary; Retrieve its postings. “Merge” the two postings: 128 34 2 4 8 16 32 64 1 2 3 5 8 13 21 Brutus Caesar * 34 128 2 4 8 16 32 64 1 2 3 5 8 13 21 The merge Walk through the two postings simultaneously, in time linear in the total number of postings entries 128 34 2 4 8 16 32 64 1 2 3 5 8 13 21 Brutus Caesar 2 8 If the list lengths are x and y, the merge takes O(x+y) operations. Crucial: postings sorted by docID. 主要内容 云计算概述 文档的倒排索引 Google搜索引擎系统分析 Google Architecture Overview Google Architecture Overview URL Server URLserver sends lists of URLs to be fetched to the crawlers. Crawler Downloads Web Page Store Sever Web pages compress and stores into a repository Repository Stores the compressed web pages Google Architecture Overview Indexer(docID) Reads the repository, uncompresses the documents, and parses them. Converts each document into a set of word and Store them in barrels.?Create forward index. Parse out all the links in every web page and store in an anchors file. URL Resorlver reads the anchors file and converts relative URLs Into absolute URLs and in turn into docIDs. generates a database of links which are pairs of docIDs. The links database is used to compute PageRanks for all the documents.?Pagerank Generates the Doc Index Google Architecture Overview Doc index keeps information about each document. It is a fixed width ISAM (Index sequential access mode) index, ordered by docID. The information includes the current document status, a pointer into the repository, a document checksum, and various statistics. the document has been crawled, a pointer called docinfo which contains its URL and title. Otherwise th