Hadoop添加节点的方法.docVIP

Hadoop添加节点的方法自己实际添加节点过程：1. 先在slave上配置好环境，包括ssh，jdk，相关config，lib，bin等的拷贝；2. 将新的datanode的host加到集群namenode及其他datanode中去；3. 将新的datanode的ip加到master的conf/slaves中；4. 重启cluster,在cluster中看到新的datanode节点；5. 运行bin/start-balancer.sh，这个会很耗时间备注：1. 如果不balance，那么cluster会把新的数据都存放在新的node上，这样会降低mr的工作效率；2. 也可调用bin/start-balancer.sh 命令执行，也可加参数 -threshold 5? ?threshold 是平衡阈值，默认是10%，值越低各节点越平衡，但消耗时间也更长。3. balancer也可以在有mr job的cluster上运行，默认dfs.balance.bandwidthPerSec很低，为1M/s。在没有mr job时，可以提高该设置加快负载均衡时间。其他备注：1. 必须确保slave的firewall已关闭;2. 确保新的slave的ip已经添加到master及其他slaves的/etc/hosts中，反之也要将master及其他slave的ip添加到新的slave的/etc/hosts中mapper及reducer个数url地址： /hadoop/HowManyMapsAndReducesHowManyMapsAndReducesPartitioning your job into maps and reducesPicking the appropriate size for the tasks for your job can radically change the performance of Hadoop. Increasing the number of tasks increases the framework overhead, but increases load balancing and lowers the cost of failures. At one extreme is the 1 map/1 reduce case where nothing is distributed. The other extreme is to have 1,000,000 maps/ 1,000,000 reduces where the framework runs out of resources for the overhead.Number of MapsThe number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapr