异构资源环境下的MapReduce性能优化.pdfVIP

2013 年2 月第2 期 小型微型计算机系统 Journal of Chinese Computer Systems Vol. 34 No.2 2013 异构资源、环境下的 MapReduce 性能优化 陶永才，石磊 (郑州大学信息工程学院，郑州 45αlOl) E-mail: ieyctao@ 摘 要:针对现有Had∞p 难以适应异构资源环境的不足，提出一种自适应 MapReduce 调度器:CloudMR.基于数据局部性， CloudMR将同一机架内的key , value 对进行本地归约合并，减少中间结果中key , value 对的数目，从而减少机架间的 数据传送.根据资源性能和任务特征，CloudMR动态确定节点任务槽数和数据分配量.对于计算性能高的节点，CloudMR分配 较多的任务和数据量，而对于计算性能低的节点，相应地减轻任务和数据量负载.实验表明，在异构环境下，较之现有 Hadoop ， CloudMR减少了节点问数据传输和备份任务运行，缩短了作业完成时间. 关键词:云计算:异构资源;MapReduω;Had∞p 中固分类号:τ凹09 文献标识码:A 文章编号:1 ∞0-1220 (2013 ) 02-0287 -06 Optimization of MapReduce Perfol回ance in Heterogeneous Resource Environments TAO Yong-cai , SHI Lei (Sch∞，1 of Infonnation Engineering , Zhengzhou University , Zhengzhou 45侃阳， αIina) Abstract :An adaptive MapReduce scheduler , CloudMR, is presented and aimed at the current Had∞IP model deficiency to adapt to heterogeneous resource environments. ßased on data 1∞ality ， CloudMR starts 吐le 1∞al reduce tasks after all 由e map tasks on 由e same rack 缸e successfully executed to reduce the amount of intermediate data transfer across racks. CloudMR can adaptively deter- mine the number of slot and the amount of data placement for each node in light of node 严:rforman臼 and job charac阳. CloudMR as- signs more tasks and data pla，ωment for high - performance nodes and little for low - performance nodes. The experiment results show that in heterogeneous reso町ce environments , compared with c山rent Had∞Ip ， CloudMR reduces the amount of data transfer , the spec- ulative tasks , and deg:rades the job finish 也ne. Key words: cloud computing; heterogeneous reso阻四; MapReduce; Hadoop 1 引言 的大规模海量数据并行编程模式，它的思想是移动计算任 务要比移动数据容易 [5]. MapReduce 包含 Map