普开数据大数据课程解析：如何使用开源软件为-Hadoop-构建大数据管道.ppt

Relationship with other Spring projects * Spring Hadoop – Core Functionality * Declarative configuration Create, configure, and parameterize Hadoop connectivity and all job types Environment profiles – easily move from dev to qa to prod Developer productivity Create well-formed applications, not spaghetti script applications Simplify HDFS and FsShell API with support for JVM scripting Runner classes for MR/Pig/Hive/Cascading for small workflows Helper “Template” classes for Pig/Hive/HBase Capabilities: Spring + Hadoop * Core Map Reduce idea * Standard Hadoop APIs Counting Words – Configuring M/R * Configuration conf = new Configuration(); Job job = new Job(conf, wordcount); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); Standard Hadoop API - Mapper Counting Words –M/R Code * public class TokenizerMapper extends MapperObject, Text, Text, IntWritable { private final static IntWritable one = new IntWritable(1); Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken());context.write(word, one); } } } Standard Hadoop API - Reducer Counting Words –M/R Code * public class IntSumReducer extends ReducerText,IntWritable,Text,IntWritable { private IntWritable result = new IntWritable(); public void reduce(Text key, IterableIntWritable values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum