(高级数据库)06-Hadoop.pptx

高级数据库及大规模存储技术;Agenda;Hadoop At Scale;;Hadoop Workflow;Who Uses Hadoop ?;Hadoop: Architecture Principles ;Hadoop Core ;Hadoop Cluster Components ; 1 HDFS Design Principles ;HDFS;NameNode Transient State ;NameNode持久状态;DataNodes ;HDFS Client ;HDFS Client;HDFS Read ;HDFS Read ;Replica Location Awareness ;HDFS Write ;HDFS Write ;Write Leases ;Append to a File ;Block Placement Policy ;System Integrity ;Cluster Startup ;Block Management ;HDFS快照：软件升级;Administration ;Hadoop Size ;2 Hadoop in action;2.1 Setting Up;2.2 Running Hadoop;2.2.1 Hadoop Running Mode;2.3 Working with files in HDFS;2.3.1 Basic file commands;2.3.2 Reading and writing to HDFS;2.4 Anatomy of a MapReduce program;Data Flow in a MapReduce Program in Hadoop;2.4.1 Hadoop data types;2.4.2 MapReduceBase;2.4.3 Mapper;2.4.4 Reducer;2.4.5 MapReduce数据流;2.4.7 Word counting;2.5 Reading and writing;2.5.1 InputFormat;2.5.2 OutputFormat;2.6 MapReduce Programming;2.6.1 Data ;2.6.2 Template for a typical Hadoop program;2.6.2 Template;2.7 Streaming;2.7.1 Streaming Python;2.7.2 Streaming PHP;2.8 Improving performance with combiners;reducer;combiner;;2.9 基本数据流;2.9.1 单Reduce任务数据流;2.9.2 多Reduce任务数据流;2.9.3 无Reduce任务数据流;2.9.4 Shuffle Sort;3 Advanced MapReduce;3.1 MapReduce flow;3.1.1 a MapReduce job;3.1.1 Job;3.1.1 Job;3.2 Joining;Joining;Joining;Joining;3.3 Reduce-side joining;3.3.1 DataJoinMapperBase;3.3.1 DataJoinMapperBase;3.3.1 DataJoinMapperBase;3.3.2 DataJoinReducerBase,;3.3.2 combine();3.3.3 extend TaggedMapOutput ;3.4 Replicated joins;3.4.1 Template;3.4.2 config;3.4.3 map;3.5 Semijoin : reduce-side join with map-side filtering;3.6 Bloom Filter;3.6.1 基本原理;3.6.2 类框架;3.6.3 add;3.6.4 contains;3.6.5 union;3.6.6 getHashIndexes;3.6.7 write;3.6.8 readFields ;3.6.9 mapper;3.6.10 Reducer;3.6.11 Run;4 Hadoop MapReduce;4.1 Hadoop;4.1.1 MapReduce DataFlow;4.1.2 MapReduce Componets;4.1.3 Job Submission;4.1.4 Initialization;4.1.5 Scheduling;4.1.6 Execution;4.1.7 Map Task;4.1.8 Sort Buffer;4.1.9 Reduce Task;4.2 YARN;4.2.1 YARN Archite