Take a Close Look at MapReduce.ppt

Take a Close Look at MapReduce Xuanhua Shi Acknowledgement Most of the slides are from Dr. Bing Chen, /chengbin/ Some slides are from SHADI IBRAHIM, /shadi/ What is MapReduce Origin from Google, [OSDI’04] A simple programming model Functional model For large-scale data processing Exploits large set of commodity computers Executes process in distributed manner Offers high availability Motivation Lots of demands for very large scale data processing A certain common themes for these demands Lots of machines needed (scaling) Two basic operations on the input Map Reduce Distributed Grep Distributed Word Count Map+Reduce Map: Accepts input key/value pair Emits intermediate key/value pair Reduce : Accepts intermediate key/value* pair Emits output key/value pair The design and how it works Architecture overview GFS: underlying storage system Goal global view make huge files available in the face of node failures Master Node (meta server) Centralized, index all chunks on data servers Chunk server (data server) File is split into contiguous chunks, typically 16-64MB. Each chunk replicated (usually 2x or 3x). Try to keep replicas in different racks. GFS architecture Functions in the Model Map Process a key/value pair to generate intermediate key/value pairs Reduce Merge all intermediate values associated with the same key Partition By default : hash(key) mod R Well balanced Diagram (1) Diagram (2) A Simple Example Counting words in a large set of documents map(string value)? //key: document name //value: document contents for each word w in value EmitIntermediate(w, “1”); reduce(string key, iterator values)? //key: word //values: list of counts int results = 0; for each v in values result += ParseInt(v); Emit(AsString(result)); How does it work? Locality issue Master scheduling policy Asks GFS for locations of replicas of input file blocks Map tasks typically split into 64MB (== GFS block size) Map tasks scheduled so GFS input block replica are on sa