基于hadoop云平台的智能电网MapReduce.doc

基于hadoop云平台的智能电网MapReduce 数据计算技术研究* 孟祥萍1，3，周来2，王晖1，3，纪秀1，3 （1．长春工程学院 电气与信息工程学院， 长春 130012； 2．东北电力大学 信息工程学院， 吉林 132012； 3. 吉林省配电自动化工程研究中心， 长春 130012） 摘要：为应对未来智能电网海量数据信息带来的实时计算、分析等难题，本文首先在Hadoop云计算平台基础上搭建MapReduce框架，论证了其良好的数据计算性能，并通过实验发现MapReduce在进一步提高计算效率方面的诸多问题--任务调度不均、数据偏移、异构环境下适应性差等。随后考虑MapReduce原始调度方式的弊端并给出均衡数据映射、评估节点性能的MapReduce 架构改进方案，并提出了动态匹配的调度算法（DMSA--Dynamic Matching Scheduling Algorithm），最后通过在仿真平台上的集群实验, 减少了系统计算资源的消耗，缩短了运行时间，显著地提高了集群性能，同时增强了数据本地性，证明了该策略提高MapReduce计算效率的可行性。 关键词：智能电网；数据计算；hadoop；mapreduce 中图分类号： 文献标识码：B 文章编号：1001-1390（2015）00-0000-00 Research on data computing technologies of Mmapreduce for smart grid based on hadoop cloud platform MengENG Xiangping1,3,ZhouHOU Lai2,WangANG Hui1,3,JiI Xiu1,3 (1.Institute of Electrical and Information Engineering,ChangchunEngineering, Changchun Institute of Technology,ChangchunTechnology, Changchun 130012,China, China.; 2.School2. School of Information Engineering,NortheastEngineering, Northeast Dianli University,JilinUniversity, Jilin 132012,China, China.; 3.Jilin3. Jilin Province Distribution Automation Engineering Research Center,ChangchunCenter, Changchun 130012, China) AbstractBSTRACT: In response to the problems of real-time calculation and analysis broughtedbrought by massive data of future smart grid,wegrid, we firstlyfirst built MapReduce frame based on a Hadoop cloud computing platform, demonstrated its powerful performance of data computing, and found many problems by an experiment in further improving computing efficiency for MapReduce, such as - uneven scheduling task, offsetting data and the poor adaptability in heterogeneous environments, etc. Then we considered the drawbacks of the original scheduling program and gave an improved program for MapReduce architecture of mapping equilibrium data,assessingdata, assessing node performance,andperformance, and proposed a dynamic matching scheduling algorithm（DMSA--Dynamic Matching Scheduling Algorithm）. Finally, we reduced