09-云计算_spark_intro.pdfVIP

Distributed Computing with Spark Problem Data growing faster than processing speeds Only solution is to parallelize on large clusters » Wide use in both enterprises and web industry How do we program these things? Outline • Data flow vs. traditional network programming • Limitations of MapReduce • Spark computing engine • Current State of Spark Ecosystem • Built-in Libraries Data flow vs. traditional network programming Traditional Network Programming Message-passing between nodes (e.g. MPI) Very difficult to do at scale: »How to split problem across nodes? • Must consider network data locality »How to deal with failures? (inevitable at scale) »Even worse: stragglers (node not failed, but slow) »Ethernet networking not fast »Have to write programs for each machine Rarely used in commodity datacenters Data Flow Models Restrict the programming interface so that the system can do more automatically Express jobs as graphs of high-level operators » System picks how to split each operator into tasks and where to run each task » Run parts twice fault recovery Map Reduce Biggest example: MapReduce Map Reduce Map Example MapReduce Algorithms Matrix-vector multiplication Power iteration (e.g. PageRank) Gradient descent methods Stochastic SVD Tall skinny QR Many others! Why Use a Data Flow Engine? Ease of programming » High-level functions instead of message passing Wide deployment » More common than MPI, especially “near” data Scalability to very largest clusters » Even HPC world is now concerned about resilience Examples: Pig, Hive, Scalding, Storm Limitations o