基于数据局部性的推测式Hadoop任务调度算法研究.docVIP

基于数据局部性的推测式Hadoop任务调度算法研究 刘 奎1，刘向东1，马宝来2，王翠荣1 (1.东北大学秦皇岛分校 计算机与通信学院,河北省省秦皇岛 066004；2. 东北大学 信息与工程学院，沈阳市 110000) 摘 要: 本文针对Hadoop平台现有任务调度算法优化程度不高的问题，提出了一种基于数据局部性的推测式任务调度算法。该算法通过计算节点上Map和Reduce任务时长比例，结合不同节点上数据的局部特性，采用了比现有算法更精确的任务进度探测方式找出快慢节点，在快节点上启动剩余时间最长的落后任务的备份任务，用移动计算代替移动数据。在Hadoop环境中进行了实验，结果表明该算法比现有算法缩短了任务平均运行时间，加快了任务的执行效率。 关键词: Hadoop；任务调度；异构环境；数据局部性 中图分类号:TP311　　　 文献标志码: A 文章编号： A speculative task scheduling algorithm based on locality of data in Hadoop LIU Kui1, LIU Xiang-dong1, MA Bao-lai2, WANG Cui-rong1 (1. College of Computer and Communication,Northeastern University at QinHuangdao, QinHuangdao Hebei 066004, China；2. College of Information and engineering, Northeastern University , Shenyang 110000, China) Abstract: For the reason that the existing algorithm on Hadoop doesnt have a high level of optimization, this paper presents a novel task scheduling algorithm based on data locality speculation. By calculating the time duration ratio of Map and Reduce task on each node combined with the local characteristics of tasks and data on different nodes, the algorithm introduces a more accurate task detection mechanism, and then launches backup tasks of slow tasks on fast nodes. For using computing migration instead of data migration, the algorithm can obtain higher efficiency. Experimental results in Hadoop show that compared with the existing scheduling algorithm, the algorithm proposed in this paper can shorten the task average operation time and reduce the network congestion caused by data exchange between cluster racks. It also can speed up the task execution efficiency. Key words: Hadoop; job scheduling; heterogeneous environments; locality of data 0引言 近年来，云计算作为一种新兴的商业计算模型，已经成为了学术界和各大IT商业巨头的研究热点，发展前景广阔。迄今为止，Google、IBM、Microsoft、Amazon、Yahoo!等IT公司都推出了自己的云计算服务平台，并把发展云计算技术作为未来的重要战略道路之一[1-2]。云计算数据中心的资源调度技术对于计算效率至关重要。当前，大多数开源云计算系统都是基于Hadoop资源管理平台，尤其在研究领域使用更为广泛。 Hadoop是基于谷歌的GFS(分布式文件系统)以及MapReduce编程模型的核心技术开发出来的开源