lecture 1 – introduction.ppt

Lecture 2 – MapReduce CPE 458 – Parallel Programming, Spring 2009 Outline MapReduce: Programming Model MapReduce Examples A Brief History MapReduce Execution Overview Hadoop MapReduce Resources MapReduce “A simple and powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves high performance on large clusters of commodity PCs.” MapReduce More simply, MapReduce is: A parallel programming model and associated implementation. Programming Model Description The mental model the programmer has about the detailed execution of their application. Purpose Improve programmer productivity Evaluation Expressibility Simplicity Performance Programming Models von Neumann model Execute a stream of instructions (machine code) Instructions can specify Arithmetic operations Data addresses Next instruction to execute Complexity Track billions of data locations and millions of instructions Manage with: Modular design High-level programming languages (isomorphic) Programming Models Parallel Programming Models Message passing Independent tasks encapsulating local data Tasks interact by exchanging messages Shared memory Tasks share a common address space Tasks interact by reading and writing this space asynchronously Data parallelization Tasks execute a sequence of independent operations Data usually evenly partitioned across tasks Also referred to as “Embarrassingly parallel” MapReduce:Programming Model Process data using special map() and reduce() functions The map() function is called on every item in the input and emits a series of intermediate key/value pairs All values associated with a given key are grouped together The reduce() function is called on every unique key, and its value list, and emits a value that is added to the output MapReduce:Programming Model MapReduce:Programming Model More formally, Map(k1,v1) -- list(k2,v2) Reduce(k2, list(v2)) -- list(v2) Map