ppt课件-parallelanddistributedcomputingmapreduce.pptVIP

Map reduce with Hadoop streaming and/or Hadoop Hadoop Job Hadoop Mapper Hadoop Reducer Partitioner Hadoop FileSystem Combiner Shuffle Sort Java, Unix File System, … Streaming Hadoop Job Hadoop FileSystem HS Mapper HS Reducer Adaptor Adaptor … API: text lines in, lines out Stream Sort Unix file system SS Mapper SS Reducer cat input | mapper | sort | reducer output API: text lines in, lines out mapper = java [-cp myJar.jar] MyMainClass1 reducer = java [-cp myJar.jar] MyMainClass2 Hadoop Job Hadoop Mapper Hadoop Reducer Partitioner Hadoop FileSystem Combiner Shuffle Sort Java, Unix File System, … Streaming Hadoop Job Hadoop FileSystem HS Mapper HS Reducer Adaptor Adaptor … API: text lines in, lines out Breaking this down… What actually is a key-value pair? How do you interface with Hadoop? One very simple way: Hadoop’s streaming interface. Mapper outputs key-value pairs as: One pair per line, key and value tab-separated Reduced reads in data in the same format Lines are sorted so lines with the same key are adjacent. An example: SmallStreamNB.java and StreamSumReducer.java: the code you just wrote. To run locally: To train with streaming Hadoop: First, you need to prepare the corpus by splitting it into shards … and distributing the shards to different machines: To train with streaming Hadoop: One way to shard text: hadoop fs -put LocalFileName HDFSName then run a streaming job with ‘cat’ as mapper and reducer and specify the number of shards you want with -numReduceTasks To train with streaming Hadoop: Next, prepare your code for upload and distribution to the machines cluster To train with streaming Hadoop: Next, prepare your code for upload and distribution to the machines cluster Now you can run streaming Hadoop: on EC2, what’s different?(aside from interface) s3://…./myjar.jar s3://…./myjar.jar “Real” Hadoop Streaming is simple but There’s no typechecking of inputs/outputs You need to parse strings a lot You can’t use compact binary encodings … basicall