教程大数据平台与编程spark.pptxVIP

Apache SparkKishore Pusukuri, Spring 2018HTTP://WWW.CS.CORNELL.EDU/COURSES/CS5412/2018SPRecapMapReduceFor easily writing applications to process vast amounts of data in-parallel on large clusters in a reliable, fault-tolerant mannerTakes care of scheduling tasks, monitoring them and re-executes the failed tasksHDFS MapReduce: Running on the same set of nodes ? compute nodes and storage nodes same (keeping data close to the computation) ? very high throughputYARN MapReduce: A single master resource manager, one slave node manager per node, and AppMaster per applicationToday’s TopicsMotivation