ppt课件-hands-onhadooptutorial.pptVIP

Jian Wang Based on “Meet Hadoop! Open Source Grid Computing” by Devaraj Das Yahoo! Inc. Bangalore Apache Software Foundation Need to process 10TB datasets On 1 node: scanning @ 50MB/s = 2.3 days On 1000 node cluster: scanning @ 50MB/s = 3.3 min Need Efficient, Reliable and Usable framework Google File System?(GFS) paper Googles?MapReduce paper Hadoop uses HDFS, a distributed file system based on GFS, as its shared file system Files are divided into large blocks and distributed across the cluster (64MB) Blocks replicated to handle hardware failure Current block replication is 3 (configurable) It cannot be directly mounted by an existing operating system. Once you use the DFS (put something in it), relative paths are from /user/{your usr id}. E.G. if your id is jwang30 … your “home dir” is /user/jwang30 Master-Slave Architecture HDFS Master “Namenode” (irkm-1) Accepts MR jobs submitted by users Assigns Map and Reduce tasks to Tasktrackers Monitors task and tasktracker status, re-executes tasks upon failure HDFS Slaves “Datanodes” (irkm-1 to irkm-6) Run Map and Reduce tasks upon instruction from the Jobtracker Manage storage and transmission of intermediate output Hadoop is locally “installed” on each machine Version 0.19.2 Installed location is in /home/tmp/hadoop Slave nodes store their data in /tmp/hadoop-${} (configurable) If it is the first time that you use it, you need to format the namenode: - log to irkm-1 - cd /home/tmp/hadoop - bin/hadoop namenode –format Basically we see most commands look similar bin/hadoop “some command” options If you just type hadoop you get all possible commands (including undocumented) hadoop dfs [-ls path] [-du path] [-cp src dst] [-rm path] [-put localsrc dst] [-copyFromLocal localsrc dst] [-moveFromLocal localsrc dst] [-get [-crc] src localdst] [-cat src] [-copyToLocal [-crc] src localdst] [-m