适应大规模数据处理的动态服务私有云系统.docVIP

适应大规模数据处理的动态服务私有云系统 摘要:为适应私有云环境下数据量大、计算密集、流程复杂的计算任务需求，借鉴公有云计算的相关理论与技术，结合私有云环境的特点，提出了一种适应大规模数据处理的动态服务私有云系统实现方案。该方案使用作业文件描述计算任务，以作业逻辑结构动态构建处理工作流程；通过数据流驱动服务请求，引入mapreduce并行框架进行大规模数据处理。实验结果表明：该方案能够正确有效地处理数据量大、计算密集、流程复杂的计算任务，显著提升处理效率，具有很高的实用性。 关键词:云计算；私有云计算；数据流驱动；动态服务；并行处理  private cloud computing system based on dynamic service adaptable to large.scale data processing  wang zhu1*, mei lin2, li lei2, zhao tai.yin1, hu guang.min1 1. key laboratory of optical fiber sensing and communication ministry of education,university of electronic science and technology of china, chengdu sichuan 611731, china; 2. geophysical exploration company, chuanqing drilling engineering company limited, chengdu sichuan 610213, china abstract: in order to deal with problem in private cloud environment caused by computing tasks with large amount of data, intensive computing and complex processing, an implementation of private cloud system based on dynamic service was proposed on basis of public cloud computing and the characteristic features of private cloud environment, which was able to adapt large-scale data processing. in this implementation, computing tasks were described by job files, processing workflows were constructed dynamically by job logic, service requests were driven by data streams and the large-scale data processing could be reflected more efficiently in mapreduce parallel framework. experiment results shows that this implementation offers a high practical value, can deal with computing tasks with large amount of data, intensive computing and complex processing correctly and efficiently. in order to deal with problem in private cloud environment caused by computing tasks with large amount of data, intensive computing and complex processing, an implementation of private cloud system based on dynamic service was proposed on the basis of public cloud computing and the characteristics of private cloud environment, which was able to adapt large.scale data processing. in th