Hadoop与数据分析-淘宝网.ppt

* * * * * * * * * * * * * * * * * * * * * * Hadoop与数据分析 淘宝数据平台及产品部基础研发组 周敏 日期：2010-05-26 Outline Hadoop基本概念 Hadoop的应用范围 Hadoop底层实现原理 Hive与数据分析 Hadoop集群管理 典型的Hadoop离线分析系统架构 常见问题及解决方案 关于打扑克的哲学 打扑克与MapReduce Input split shuffle output 分牌 各自齐牌 交换 再次理牌 搞定 统计单词数 The weather is good This guy is a good man Today is good Good man is good the 1 weather 1 is 1 good 1 today 1 is 1 good 1 this 1 guy 1 is 1 a 1 good 1 man 1 good 1 man 1 is 1 good 1 a 1 good 1 good 1 good 1 good 1 good 1 man 1 man 1 the 1 weather 1 today 1 guy 1 is 1 is 1 is 1 is 1 this 1 a 1 good 5 guy 1 is 4 man 2 the 1 this 1 today 1 weather 1 流量计算 * 趋势分析 * /截图 用户推荐 * 分布式索引 * * Hadoop 核心 Hadoop Common 分布式文件系统HDFS MapReduce框架 并行数据分析语言Pig 列存储NoSQL数据库 Hbase 分布式协调器Zookeeper 数据仓库Hive(使用SQL) Hadoop日志分析工具Chukwa Hadoop生态系统 * Data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Data data data data data Results Data data data data Data data data data Data data data data Data data data data Data data data data Data data data data Data data data data Data data data data Data data data data Hadoop Cluster DFS Block 1 DFS Block 1 DFS Block 2 DFS Block 2 DFS Block 2 DFS Block 1 DFS Block 3 DFS Block 3 DFS Block 3 MAP MAP MAP Reduce Hadoop实现 作业执行流程 // MapClass1中的map方法 public void map(LongWritable Key, Text value, OutputCollectorText, Text output, Reporter reporter) throws IOException { String strLine = value.toString(); String[] strList = strLine.split(\); String mid = strList[3]; String sid = strList[4]; String timestr = strList[0]; try{ timestr = timestr.substring(0,10); }catch(Exception e){return;} timestr += 0000; // 省略数十行 output.collect(new Text(mid + “\”” + “sid\”” + timestr , ...); } Hadoop案例(1) public static cl