Thinking in Hadoop.pptx

Thinking in HadoopLiu Chaopopsuper1982AgendaHadoop ArchitectureHadoop File SystemMap-ReduceBig Data CasesDebugging and Profiling HadoopQAHadoop ArchitectureJobTrackerClientClientSecondary Name NodeJobClientDFSClientName NodeTaskTrackerTaskTrackerTaskTrackerDataNodeDataNodeDataNodeNameNodeHadoop File SystemFSNamesystemData StructuresFSDirectoryFSImageFSEditLog DistributedFileSystemrpcINodeDirectory DataStreamerDFSClientINodeFileBlockInfo[] Block - { INode, datanodes, self ref }BlocksMapFSDataInput/OutputStream Data NodeDataXceiverServerDataXceiverBlockReceiverHadoop File SystemTriangle Structure : Separate Metadata and data flowHadoop File SystemReplication and Load BalancerHadoop File SystemCheckpoint and logHadoop File SystemPipelineout.write( DataTransferProtocol.OP_WRITE_BLOCK );for (int i = 1; i nodes.length; i++) { nodes[i].write(out);}Name Node64MaddBlockData NodeData NodeClientDataXceiverServerDataXceiverServerpacketDatanodeInfo[] nodespacketdataQueuepacketDataXceiverDataXceiverpacketconnectconnectcommandcommandDataStreamerBlockReceiverBlockReceivermirrorOutmirrorOutinout inmirrorIn replyOutmirrorIn replyOutblockReplyStream Map-ReduceConcept 1: Divide and ConcurDivideConcurMergeMap-ReduceJobTrackerConcept 2: Bind data to processorClientClientJobSecondary Name NodeJobClientDFSClientName NodeTaskTrackerFile321TaskTrackerTaskTaskTrackerDataNodeTaskTaskBlock 3DataNodeDataNodeBlock 1Block 2HDFS block size is 64 MBMap-ReduceStep 1 : Identify the problem and logic process假设我们需要处理一批有关天气的数据，其格式如下：? 按照ASCII码存储，每行一条记录? 每一行字符从0开始计数，第15个到第18个字符为年? 第25个到第29个字符为温度，其中第25位是符号+/-Map-ReduceStep 2 :Logical DesignMap:k1,v1 -listk2,v2Reduce:k2,listv2 -k3,v3(0, 0067011990999991950051507+0000+)(33, 0043011990999991950051512+0022+)(66, 0043011990999991950051518-0011+)(99, 0043012650999991949032412+0111+)(132, 0043012650999991949032418+0078+)(165, 0067011990999991937051507+0001+)(198, 0043011990999991937051512-0002+)(231, 0043011990999991945051518+0001+)(264, 004301265099999194503241