一种提高虚拟化Hadoop系统数据本地性的资源调度方法.PDF

第 43 卷 增刊 计 算 机 研 究 与 发 展 Vol.43, Suppl. 2014 年 11 月 JOURNAL OF COMPUTER RESEARCH AND DEVELOPMENT Nov. 2014 一种提高虚拟化Hadoop 系统数据本地性的资源调度方法 1,2 2 2 1,3 孙瑞琦 杨杰 高瞻 贺志强 1 (北京航空航天大学计算机学院，北京 100191) 2 (联想研究院云与智能计算实验室，北京 100085) 3 (联想集团，北京 100085) (sunruiqi2012@buaa.edu.cn) A Resource Scheduling Approach to Improving Data Locality for Virtualized Ha- doop Cluster on YARN 1,2 2 2 1,3 Sun Ruiqi , Yang Jie , Gao Zhan , and He Zhiqiang 1 (School of Computer Science and Engineering, Beihang University, Beijing 100191) 2 (Cloud and Intelligent Computing Lab, Lenovo Research Technology, Beijing 100085) 3 (Lenovo Group, Beijing 100085) Abstract In Hadoop system, the data locality is a critical factor impacting on performance of large-scale data analy- sis applications. However, most of existing resources scheduling approaches to improving data locality study on the traditional Hadoop cluster which is built using physical machines. These approaches are not effective in virtualized Hadoop clusters because of two levels distribution of data: virtual machines and physical servers. In this paper, we deploy virtualized Hadoop cluster in which computing node and storage node are placed in respective virtual ma- chines to improve flexibility. Moreover, we propose a novel resource scheduling approach which aims to improve data locality for virtualized Hadoop cluster through two key methods. One is adjusting the computational capability of virtual machine acted as computing node whose input data stored in a virtual machine acted as storage node run- ning in the same physical server, and the other is migra