面向集合的MapPartition计算模型的研究与实现-计算机软件与理论专业毕业论文.docxVIP

摘要 随着互联网技术的发展和云计算技术的出现，越来越多的公司企业开始面对海 量数据的处理。传统的集中式处理方式和分布式处理方式很难满足海量数据的计 算需求，而云计算技术的发展则为大规模数据处理提供了很好的技术平台。然而， 在云计算环境下的数据处理具有内在的高度并行性，并且是分布式的，这与传统 的顺序程序存在巨大差异。如何为普通用户提供一个简洁而又功能强大的编程模 型与接口，是当今云计算技术研究的重点。现有云计算环境下的计算模型往往对 迭代式编程的支持不够，并通常要求用户实现特定的自定义函数，而用户对程序 的逻辑缺少控制，不能像顺序程序一样简单的编写代码。 本文针对以上问题，通过对现有计算模型的研究，设计出了面向集合的 MapPartition 计算模型（Set-oriented Map Partition Computational Model，SOMP）。 该模型由一个 Main 函数和一些用户自定义的操作组成。在 Main 函数中，用户可 以像顺序程序一样进行迭代式编程，然后通过模型提供的一些原语将自定义的一 些操作作用在一些键值对集合上。通过操作的延迟执行和优化，SOMP 模型能够 很好的利用数据缓存提高计算效率。经过实际测试，该计算模型在处理迭代式数 据计算时具有较高的效率。 关键词：云计算模型 键值对集合 MapPartition 模型数据缓存 Abstract With the development of Internet technology and the emergence of cloud computing technology, a large growing number of companies begin to face the massive data processing. The traditional centralized or distributed approach is difficult to meet the computing needs of large amounts of data, but the cloud computing technology provides a good platform for the large-scale data processing. However there is a huge difference between the sequential program and the data processing under cloud computing environment, which has a high degree of parallelism, and is distributed. How to provide users with a simple yet powerful programming model and interface, is the focus of the study of cloud computing technology. Without powerful support for iterative programming models, existing cloud computing environments often require users to define some specific custom functions, at the cost of lack of control logic. In this paper, a Set-oriented Map Partition Computational Model (SOMP) is designed based on the research of existing computational models. The model consists of a main function and some user-defined operations. In the main function, users can perform iterative programming as easy as sequential program, and user-defined operations can be executed on each tuple in a set of key-value pairs by certain primitives. Due to the delay execution and optimization of operations, SOMP model can make ful