Applications of Map-Reduce.ppt

Feature absence and tool incompatibility Transaction updates Changing data and maintaining data integrity Data mining and replication tools Database design and construction tools Map Reduce Inapplicability * * * * Worcester Polytechnic Institute Applications of Map-Reduce Team 3 CS 4513 – D08 Very popular example to explain how Map-Reduce works Demo program comes with Nutch (where Hadoop originated) * Distributed Grep For Unix guru: ??? grep -Eh regex inDir/* | sort | uniq -c | sort -nr- counts lines in all files in inDir that match regex and displays the counts in descending order ????????????? ????????????  ?????????????????????????  - grep -Eh A|C in/* | sort | uniq -c | sort -nr- Analyzing web server access logs to find the top requested pages that match a given pattern Distributed Grep C B B C C A 3 C 1 A Result File 2 File 1 Map function in this case:?- ? input is (file offset, line) ?- ? output is either: 1. an empty list [] (the line does not match)  2. a key-value pair [(line, 1)] (if it matches)Reduce function in this case:?- input is (line, [1, 1, ...]) ?- output is (line, n) where n is the number of 1s in the list. Distributed Grep Map tasks:(0, C) - [(C, 1)](2, B) - [](4, B) - [](6, C) - [(C, 1)](0, C) - [(C, 1)](2, A) - [(A, 1)] Distributed Grep C B B C C A 3 C 1 A Result File 2 File 1 Reduce tasks:(A, [1])?????? - (A, 1)(C, [1, 1, 1]) - (C, 3) Large-Scale PDF Generation The New York Times needed to generate PDF files for 11,000,000 articles (every article from 1851-1980) in the form of images scanned from the original paper Each article is composed of numerous TIFF images which are scaled and glued together Code for generating a PDF is relatively straightforward The Problem Technologies Used Amazon Simple Storage Service (S3) Scalable, inexpensive internet storage which can store and retrieve any amount of data at any time from anywhere on the web Asynchronous, decentralized system which aims to reduce scaling bottlen