大规模数据处理云计算Introduction.pptVIP

Katta Architecture(Distributed Lucene) / * QA?Thanks you! * * Why is IR hard? Because language is hard! Assume document is relevant if it has a lot of query terms Replace relevance(q, di) with sim(q, di) Compute similarity of vector representations * * * 22 * 22 [2,4] [3] [2,4] 2 2 1 Inverted Index: Positional Information 1 2 blue fish one fish, two fish Doc 1 red fish, blue fish Doc 2 cat in the hat Doc 3 green eggs and ham Doc 4 1 2 2 * 请写出”blue”和”fish”两个词在inverted index中的表示 Ranked Retrieval 请在为这个文档集建好的inverted index上执行ranked retrieval：“ blue fish”，描述retrieval的计算的过程 * one fish, two fish Doc 1 red fish, blue fish Doc 2 cat in the hat Doc 3 green eggs and ham Doc 4 MapReduce it? The indexing problem Scalability is critical Must be relatively fast, but need not be real time Fundamentally a batch operation Incremental updates may or may not be important For the web, crawling is a challenge in itself The retrieval problem Must have sub-second response time For the web, only need relatively few results Perfect for MapReduce! Uh… not so good… * Indexing: Performance Analysis How is it done on a single machine? Fundamentally, a large sorting problem Terms usually fit in memory Postings usually don’t How can it be done with MapReduce? First, let’s characterize the problem size: Size of vocabulary Size of postings * Vocabulary Size: Heaps’ Law Heaps’ Law: linear in log-log space Vocabulary size grows unbounded! M is vocabulary size T is collection size (number of tokens) k and b are constants Typically, k is between 30 and 100, b is between 0.4 and 0.6 * Heaps’ Law for RCV1 Reuters-RCV1 collection: 806,791 newswire documents (Aug 20, 1996-August 19, 1997) k = 44 b = 0.49 First 1,000,020 terms: Predicted = 38,323 Actual = 38,365 Manning, Raghavan, Schütze, Introduction to Information Retrieval (2008) * Postings Size: Zipf’s Law Zipf’s Law: (also) linear in log-log space Specific case of Power Law distributions In other words: A few elements occur ve