2基于Hadoop的风力发电监测大数据存储优化-电测与仪表.doc

第44卷 第504期 电测与仪表 Vol.44 No.504 2007年 第12期 Electrical Measurement Instrumentation Dec.2007 PAGE 1 - 基于Hadoop的风力发电监测大数据存储优化及并行查询方法 王林童1, 赵 腾1，张 焰1，苏 运2 （1.上海交通大学 电气工程系，上海 200240；2.国网上海市电力公司电力科学研究院，上海 200437） 摘要：随着风力发电的广泛发展以及智能化监测技术的推广应用，风力发电监测数据呈现出体量大、类型多、增长快的大数据特征。针对风力发电监测大数据高效存储和快速查询两方面核心问题，本文基于Hadoop平台进行大数据存储优化方法研究，提出考虑风力发电监测数据关联性的哈希分桶存储算法，实现了相关联数据的集中存储，从而提升后期数据查询及处理的效率。在数据存储优化的基础上，实现基于MapReduce的多源风力发电监测大数据并行关联查询。通过在Hadoop平台上进行测试表明，经过哈希分桶存储优化后的多源数据并行关联查询相比传统Hadoop方法查询时间显著缩短。 关键词：大数据；风力发电监测；Hadoop；哈希分桶算法 中图分类号：TM721 文献标识码：A 文章编号：1001-1390（2018）00-0000-00 Storage optimization and parallel query method for big data of wind power monitoring based on Hadoop Wang Lintong1, Zhao Teng1,Zhang, Zhang Yan1,Su, Su Yun2 (1. Department of Electrical Engineering, Shanghai Jiaotong Jiao Tong University, Shanghai 200240, China; . 2.Electric Power Research Institute, SMEPC, Shanghai 200437, China) Abstract：: With the extensive development of wind power generation and the generalized application of intelligent monitoring technology, wind power monitoring data shows the big data characteristics of large volume, multi types and fast growth. In order to solve the two major problems of big data with efficient storage and quick query, in this paper, the optimization method of big data storage is studied based on Hadoop platform. A Hash bucket algorithm considering wind power monitoring data correlation is proposed. The algorithm, which realizes the centralized storage of related data, so as to enhance the efficiency of data query and processing. On the basis of data storage optimization, the parallel association query for multi-source big data of wind power monitoring based on MapReduce is realized. Tests on a Hadoop platform show that, after optimization of hash bucket storage, the time of the multi-source data parallel associatio