基于机器学习的MapReduce资源调度算法.PDFVIP

网络出版时间：2015-07-07 17:05 网络出版地址：/kcms/detail/51.1196.TP1705.056.html 优先出版 计 算 机 应 用 研 究 第33 卷 基于机器学习的MapReduce 资源调度算法 1 1 2 1 于 倩 ，蔚承建 ，王 开 ，朱林军 (1．南京工业大学 电子与信息工程学院，南京 210009 ；2 ．东南大学 信息科学与工程学院，南京 210018) 摘 要：针对MapReduce 中允许map 和shuffle 阶段重叠的优化模型需要自适应性的问题，提出了基于此模型的机器学 习的资源调度算法，利用贝叶斯分类器依据作业对系统资源的需求和系统环境的匹配程度对作业进行调度，并不断更新 分类器，使其具有自适应性，考虑了map 和shuffle 的重叠阶段。通过模拟实验验证，改进后的算法能够提高MapReduce 系统的性能，获得更好的平均响应时间。 关键词：MapReduce ；重叠阶段；自适应性；机器学习；贝叶斯分类器 中图分类号： TP301 Resource scheduling algorithm for MapReduce based on machine learning 1 1 2 1 Yu Qian , Wei Chengjian , Wang Kai , Zhu Linjun (1 College of Electronics Information Engineering, Nanjing Tech University, Nanjing 210009, China; 2 Dept. of Information Science Engineering, Southeast University, Nanjing 210018, China) Abstract: To solve the problem that a map and shuffle optimization model allows overlapping phases in MapReduce is lack of adaptability. a resource scheduling algorithm based on this model of machine learning was proposed. bayesian classifier based on the degree of matching the operating system resource requirements and system environment is used for job scheduling. and then the classification is continuously updated. so that it has adaptability. considering the overlap phase map and shuffle. Verified through simulation experiments. the solution can improve the performance of MapReduce system. get a better average response tim