基于 hadoop 平台的作业调度技术研究-计算机应用技术专业论文.docxVIP

西安建筑科技大学硕士学位论文 西安建筑科技大学硕士学位论文 基于 Hadoop 平台的作业调度技术研究 专 业：计算机应用技术 硕 士 生：王 皎 指导教师：董丽丽 摘 要 随着数据规模的急速增大，应用种类的极大丰富，个人用户和企业对信息的 需求已经远远超过了原有平台的能力。越来越多的应用和平台，不论对企业级还 是个人级用户都不堪重负，Hadoop 平台作为一个开源的，且运行在集群上的大型 数据库处理框架得到了广泛应用，然而由于 Hadoop 还比较年轻，在很多地方还有 改进和修改的可能。 本文通过广泛的调研，介绍了云计算的定义和关键技术，对基于云计算的 Hadoop 平台的两大核心 Mapreduce 和 HDFS 进行了深入的研究。论文重点分析了 现有的 3 种调度算法：先入先出调度算法（FIFO）、公平调度算法(Fair-Scheduler) 和计算能力调度算法(Capacity Scheduler)的设计思路和优缺点，并针对 Hadoop 的 参数过多难以设置和调度算法过多难以选择的问题，提出了一种海量数据的抽样 算法，设计了基于该算法的改进 Hadoop 框架，即在原有框架基础上加入了策略选 择层，从而解决上述两个问题。最后对 Hadoop 平台重新打包并进行实验验证，实 验结果表明，和以前版本相比，其处理海量数据问题的性能有了明显的改进。 关 键 词：Hadoop；Mapreduce；HDFS；调度算法；策略选择层 论文类型：应用研究 西安建筑科技大学硕士学位论文 西安建筑科技大学硕士学位论文 Research of Job Scheduling technology in Hadoop Platform Major: Computer Application and Technology Name: Wang Jiao Instructor: Prof.Donng Lili Abstract With the rapid increase of data size and the great wealth of the application types,the demand of information to consumer and corporate far exceeds the ability of the original platform. enterprise-level or individual-level is overwhelmed by more and more applications and platfom.the Hadoop platform as an open source framework which handle large database running on the cluster becomes more and more popular.however,Hadoop is still young,There are many places have the possibility to make changes and improvements. Through extensive research,this paper introduces the definition of cloud copmuting and key technologies,In-depth researchs on the Mapreduce and the HDFS of Hadoop core..then,this paper analyze the design ideas and the advantages and disadvantages of existing three scheduling algorithms:FIFO,Fair-Scheduler,Capacity Scheduler. For the problem that Hadoop parameters too much to set up and scheduling algorithm too much difficult to choose,propose a sampling algorithm based on huge amount of data.and design the improved Hadoop framework which join the strategic choice layer to solve the two problem.finally, it repackage and experiment the Hadoop platform.Compared