ppt课件-cloudcomputingwithmapreduceandhadoop-mitcsail.pptVIP

Elastic MapReduce Workflow Elastic MapReduce Workflow Conclusions MapReduce programming model hides the complexity of work distribution and fault tolerance Principal design philosophies: Make it scalable, so you can throw hardware at problems Make it cheap, lowering hardware, programming and admin costs MapReduce is not suitable for all problems, but when it works, it may save you quite a bit of time Cloud computing makes it straightforward to start using Hadoop (or other parallel software) at scale Resources Hadoop: /core/ Pig: /pig Hive: /hive Video tutorials: /hadoop-training Amazon Web Services: / Amazon Elastic MapReduce guide:/ElasticMapReduce/latest/GettingStartedGuide/ My email: matei@ * * * * * * * * * My point in putting in the java code isn’t too actually walk through it. It’s just to show that you have to hand code a fair amount of java. That way I can just point and say “look how much java you’d have to write to accomplish this” * Rather than using capital letters, which makes Pig Latin look like SQL, I added Eclipse style highlighting instead. Hopefully this makes clear what are the key words without making it look like a Matisse painting. * No need to think about how many map reduce jobs this decomposes into, or connecting data flows between map reduces jobs, or even that this is taking place on top of map reduce at all. * No need to think about how many map reduce jobs this decomposes into, or connecting data flows between map reduces jobs, or even that this is taking place on top of map reduce at all. * Word Count with Combiner Input Map Combine Shuffle Sort Reduce Output the quick brown fox the fox ate the mouse how now brown cow Map Map Map Reduce Reduce brown, 2 fox, 2 how, 1 now, 1 the, 3 ate, 1 cow, 1 mouse, 1 quick, 1 the, 1 brown, 1 fox, 1 quick, 1 the, 2 fox, 1 how, 1 now, 1 brown, 1 ate, 1 mouse, 1 cow, 1 Fault Tolerance in MapReduce 1. If a task crashes: Retry on another node OK for a map because it has no dependencies OK for