Module4经典的MapReduce.ppt

Google Cluster Computing Faculty Training Workshop Module IV: MapReduce Theory, Implementation, and Algorithms Overview Functional Programming Recap MapReduce Theory Implementation MapReduce Algorithms Functional Programming Review Functional operations do not modify data structures: They always create new ones Original data still exists in unmodified form Data flows are implicit in program design Order of operations does not matter Functional Programming Review fun foo(l: int list) = sum(l) + mul(l) + length(l) Order of sum() and mul(), etc does not matter – they do not modify l Functional Updates Do Not Modify Structures fun append(x, lst) = let lst = reverse lst in reverse ( x :: lst ) Functions Can Be Used As Arguments fun DoDouble(f, x) = f (f x) Map map f lst: (’a-’b) - (’a list) - (’b list) Creates a new list by applying f to each element of the input list; returns output in order. Fold fold f x0 lst: (a*b-b)-b-(a list)-b Moves across a list, applying f to each element plus an accumulator. f returns the next accumulator value, which is combined with the next element of the list fold left vs. fold right Order of list elements can be significant Fold left moves left-to-right across the list Fold right moves from right-to-left Example fun foo(l: int list) = sum(l) + mul(l) + length(l) How can we implement this? Example (Solved) fun foo(l: int list) = sum(l) + mul(l) + length(l) fun sum(lst) = foldl (fn (x,a)=x+a) 0 lst fun mul(lst) = foldl (fn (x,a)=x*a) 1 lst fun length(lst) = foldl (fn (x,a)=1+a) 0 lst map Implementation This implementation moves left-to-right across the list, mapping elements one at a time … But does it need to? Implicit Parallelism In map In a purely functional setting, elements of a list being computed by map cannot see the effects of the computations on other elements If order of application of f to elements in list is commutative, we can reorder or parallelize execution This is the “secret” that MapRe