面向迭代作业的分布式并行处理优化与实现-计算机技术专业毕业论文.docxVIP

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II万方数据 II 万方数据 Abstract Google MapReduce is a parallel programming model for large-scale data sets, the current application has been very extensive. In data mining, machine learning, and other areas of scientific computing, iterative calculation has a very wide range of applications. However, the existing MapReduce implementation such as Hadoop, can not effectively support the iterative data processing, thus improving the performance of traditional MapReduce processing iterative problem is very important. This paper analyzed parallel programming on MapReduce, iterative jobs and iterative improvement of existing MapReduce model deeply, summed up the problems of the existing methods. The traditional MapReduce when dealing with iterative jobs without distinction between static and dynamic data, and intermediate results of the iterative process write into distributed file system. Iteration termination condition judge also need to consume a MapReduce job. MapReduce haven’t improved the iterative process in In terms of dependency, iterative process strictly in accordance with the general operation executed in serial order of execution. An iterative parallel processing model based on temporary jobs is proposed. For the issues of the different treatment for dynamic data and static data and the Iterative intermediate results written into distributed file system, a local cache structures is designed to store static data and the intermediate results of iteration. For the original MapReduce processing iterative process in strictly dependence, a new iterative control loop structure is designed, using temporary jobs to join two iterations to achieve parallel execution of successive iterations. For the issues of the additional overhead in termination condition judgement, a new iteration termination condition detector is designed, the judgment automatically determined by the internal iterative process, and simultaneously with the next iteration.