lecture 2 – theoretical underpinnings of mapreduce - ubc ece.ppt

Key idea 3:Scale out, not up! For data-intensive workloads, a large number of commodity servers is preferred over a small number of high-end servers cost of super-computers is not linear Some numbers Processing data is quick, I/O is very slow: 1 HDD = 75 MB/sec; 1000 HDDs = 75 GB/sec Data volume processed: 80 PB/day at Google; 60TB/day at Facebook (~2012) Key idea 4“Shared-nothing” infrastructure(both hard- and soft-ware) Sharing vs. Shared nothing: Sharing: manage a common/global state Shared nothing: independent entities, no common state Functional programming as key enabler No side effects Recovery from failures much easier map/reduce – as subset of functional programming More examples Distributed Grep: The map function emits a line if it matches a supplied pattern. The reduce function is an identity function that just copies the supplied intermediate data to the output. Count of URL Access Frequency: The map function processes logs of web page requests and outputs URL; 1. The reduce function adds together all values for the same URL and emits a URL; total count pair. ReverseWeb-Link Graph: The map function outputs target; source pairs for each link to a target URL found in a page named source. The reduce function concatenates the list of all source URLs associated with a given target URL and emits the pair: target; list(source) Term-Vector per Host: … More info MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat, /papers/mapreduce.html The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-TakLeung, /papers/gfs.html * * * * * * * * * * * * * * * MapReduce: Acknowledgements: Some slides form Google University (licensed under the Creative Commons Attribution 2.5 License) others from Jure Leskovik MapReduce Concept from functional programming Applied to large number of problems Java: int fooA(String[] list) { return bar1(list) + bar2(list); } int fooB(String[] list) {