基于Spark的异构集群调度策略研究.pdf

Computer Science and Application 计算机科学与应用, 2016, 6(11), 692-704 Published Online November 2016 in Hans. /journal/csa /10.12677/csa.2016.611084 Adaptive Scheduling Strategy for Heterogeneous Spark Cluster Jiajun Xu, Gongshen Liu, Bo Su, Kui Meng Shanghai Jiao Tong University, Shanghai rd st th Received: Nov. 3 , 2016; accepted: Nov. 21 , 2016; published: Nov. 24 , 2016 Copyright © 2016 by authors and Hans Publishers Inc. This work is licensed under the Creative Commons Attribution International License (CC BY). /licenses/by/4.0/ Open Access Abstract The scheduling strategy of Spark assumes that cluster is homogenized. However, as the change or update of hardware in cluster, it becomes more and more heterogeneous. Thus, the original sche- duling strategy cannot meet the performance requirement anymore and short board effect gradu- ally emerges. The paper proposes a new strategy to solve this problem. The new strategy refers the idea of hierarchical scheduling. It combines the task complexity, worker performance and worker CPU usage as its scheduling factors to improve the scheduling performance. And experi- ments show that the new strategy is absolutely effective. Keywords Spark, Heterogeneous Cluster, Scheduling Strategy 基于Spark的异构集群调度 策略研究 徐佳俊，刘功申，苏 波，孟 魁 上海交通大学，上海 收稿日期：2016年11月3 日；录用日期：2016年11月21 日；发布日期：2016年11月24 日 文章引用: 徐佳俊, 刘功申, 苏波, 孟魁. 基于 Spark 的异构集群调度策略研究[J]. 计算机科学与应用, 2016, 6(11): 692-704. /10.12677/csa.2016.611084 徐佳俊 等 摘 要 Spark 的原生调度策略建立在集群同质化的基本假设上。然而随着硬件的更迭以及高性能硬件的引入， 集群异质化现象日趋显著。因此现有的调度策略在异构集群环境下并不高效，短板效应严重。针对这个 问题，本