云平台下输变电设备状态监测大数据存储优化和并行处理.pdfVIP

第 35 卷 第 2 期 中 国 电 机 工 程 学 报 Vol.35 No.2 Jan.20, 2015 2015 年 1 月 20 日 Proceedings of the CSEE ©2015 Chin.Soc.for Elec.Eng. 255 DOI ：10.13334/j.0258-8013.pcsee.2015.02.001 文章编号：0258-8013 (2015) 02-0255-13 中图分类号：TM 764 云平台下输变电设备状态监测大数据存储优化与 并行处理 宋亚奇，周国亮，朱永利，李莉，王刘旺，王德文 (华北电力大学控制与计算机工程学院，河北省 保定市 071003) Storage Optimization and Parallel Processing of Condition Monitoring Big Data of Transmission and Transforming Equipment Based on Cloud Platform SONG Yaqi, ZHOU Guoliang, ZHU Yongli, LI Li, WANG Liuwang, WANG Dewen (School of Control and Computer Engineering, North China Electric Power University, Baoding 071003, Hebei Province, China) ABSTRACT: Applying big data technology for improving the 行了数据分布策略、数据块尺寸调优、集群网络拓扑规划三 condition evaluation of power transmission and transforming 方面的存储优化研究和大数据并行分析的研究。提出计及数 equipment and solving its practical problems becomes a new 据相关性的多副本一致哈希数据存储算法，能将具有相关性 challenge in power industry. For high reliable storage and rapid 的数据在集群中聚集，提升数据处理执行效率。基于数据相 access of data, the data distribution strategy, data block size 关性多副本一致哈希数据分布，应用 MapReduce 并行编程 adjustment and the cluster network topology are studied based 模型设计实现了多数据源并行连接查询算法和多通道数据 on hadoop. A multi-copy consistency Hash algorithm based on 融合并行特征提取算法。将两种算法在实验室搭建的集群上 data correlation (CMCH) is proposed. The algorithm makes the 测试运行，结果表明，多数据源并行连接查询的执行时间仅 relevant data gathering in the cluster and improves the data 为标准 Hadoop 方案的 32%，多通道数据融合并行特征提取 processing speed. Based on the CMCH algorithm