（毕业论文）Hadoop云计算平台作业调度算法的研究.docVIP

摘 要 数据的指数级增长向世界互联网巨头Google、Yahoo、Amazon和Microsoft等这些处于市场领导地位的公司提出了挑战，它们需要分析TB级和PB级的海量数据以发现有价值的信息推荐给那些有潜在需求的人群。现有的工具正变得无力处理如此大的数据集，Google率先推出了MapReduce编程模型，这是Google公司为了在廉价的计算机集群上来存储并处理PB级的数据而提出的一种解决方案。这一解决方案引起了学术界和工业界的广泛关注，因为许多企业同样面临数据膨胀的挑战，一些开源软件诸如Hadoop等给这些企业带来了希望，使它们能够在廉价的计算机集群上存储海量数据，并能够运用MapReduce的思想并行处理这些海量数据，为它们节约了大量的计算以及存储成本。 由于现有的Hadoop集群中包含各种类型的作业，有些作业对完成时间没有要求，而某些作业的完成时间过长时有可能给企业带来损失，为此本文着重研究了集群中各种作业的调度执行流程，分析了MapReduce现有的调度机制，本文针对目前调度算法不能支持对时间紧迫性要求较高的作业的需求，对作业调度算法进行了相应的改进，设计并编写了一个双队列的作业调度器，并研究了集群中慢节点的判定方法、推测执行任务的选定方法，以将推测执行的任务尽量分配给快节点执行，最大化的减少集群中资源的浪费，更好地满足用户的需求。 最后，我们通过搭建Hadoop集群对编写的双队列调度器的性能进行测试，验证了改进后调度算法的正确性，在集群中有各种类型的作业时，该调度器能够优先执行一些紧迫性要求较高的作业，以使它们尽早地完成，提高了集群资源的使用率，从而满足各种用户的需求。 关键词：Hadoop；作业调度；云计算；推测执行任务 Abstract The giant companies of the Internet, such as Google, Yahoo, Amazon and Microsoft and so on, have large amounts of data. The exponential data bring out many problems, so they have to discover the new technologies to anaylyze TB and PB level mass data to achieve useful information. The message is useful to those companies to find the popular books and musics, and recommend the popular news and books to the potential customers. But, the existing tools are becoming unable to handle such large data sets. Google, the first company who provide the MapReduce programming model and the model is able to process the data of PB levels in inexpensive computer clusters parallelly. This solution has attracted many companies in academical and the industrial, because many companies face the same challenge of data expansion problems. The problem was that many companies didn’t have the ablity to develop theirs own tools. Some opensource software, such as Hadoop, OpenStack and so on, offer promising hope those companies, and the companies can store huge amounts of data on inexpensive computer clusters, and be able to use the idea of MapReduce parallel processing of these massive data , as they save a lot of computing and storage costs. As the existing Hadoo