一种基于资源预取的Hadoop 作业调度算法Hadoop job - 云计算论坛.PDF

网络出版时间：2014-04-16 17:12 网络出版地址：/kcms/detail/51.1196.TP1712.086.html 一种基于资源预取的Hadoop 作业调度算法* 万 兵，黄梦醒，段 茜 （海南大学信息科学技术学院，海口 570228） 摘 要: Hadoop 原有的作业调度算法在任务级调度阶段均采用简单考虑数据本地性的任务分配策略，并不能保证良好的数据 本地性，而相应的改进算法改进目标不一，也不完全适用于所有作业调度场合。针对以上问题，结合数据预取理念提出一种 基于资源预取的作业调度算法。通过估算节点上正在运行任务的剩余执行时间，并与传输一个数据块所需时间进行比较，预 选出待预取的候选节点；并从当前正在运行作业的未分配任务中选取非本地 map 任务作为待预取任务；最后，为该任务在 候选节点上预取其所需数据。实验结果表明该改进算法有效地提高了作业的数据本地性，并在一定程度上减少了作业执行时 间。 关键词: 作业调度；数据本地性；资源预取；Hadoop ；MapReduce ；云计算 中图分类号: TP393 文献标志码: A 文章编号： Hadoop job scheduling algorithm based on resource prefetching WAN Bing, HUANG Meng-xing, DUAN Xi (College of Information Science Technology, Hainan University, Haikou 570228, China) Abstract: The existing hadoop job scheduling algorithms can’t guarantee good data locality by using a similar task allocation policy in task-level scheduling phase, and the corresponding improved algorithms vary widely in optimization goal, which are inapplicable to all hadoop job scheduling situations. To address problems above, combining with the concept of Data Prefetching, a job scheduling algorithm based on resource prefetching is proposed. First of all, by comparing the estimated remaining execution time of running map tasks on all TaskTrackers with the time of transmitting an input data block between TaskTrackers, a candidate TaskTracker to accomplish the prefetching operation is selected. Secondly, a non-local map task is selected from all unassigned map tasks of the current running job to prefetch. Finally, the required input data block of this candidate map task is prefetched into the candidate