基于MapReduce的K_means并行算法及改进①.PDF

计 算 机 系 统 应 用 2015 年 第 24 卷 第 6 期 ① 基于MapReduce 的K_means 并行算法及改进 衣治安, 王 月 (东北石油大学 计算机与信息技术学院, 大庆 163318) 摘 要: 针对传统 k_means 聚类算法在处理海量数据时所面临的内存不足、运算速度慢等问题, 提出了一种基于 MapReduce 的 K_means 并行算法, 同时为了改善k_means 算法在初始值确定方面的盲目性, 采用 canopy 算法进 行改进. 实验结果表明, 基于MapReduce 的K_means 并行算法和改进后的算法均能产生良好的聚类效果, 不仅提 高了聚类质量, 而且在处理大数据集方面, 改进后的算法的还能够得到趋近于线性的加速比. 关键词: MapReduce; k-means 算法; canopy 算法; 并行计算; 聚类 Parallel K-Means Algorithm and Improved Based on MapReduce YI Zhi-An, WANG Yue (Northeast Petroleum University, College of Computer and Information Technology, Daqing 163318, China) Abstract: In view of the problems that traditional k-means clustering algorithm faces in dealing with mass data, such as running out of memory, the operating in slow speed and so on, this paper proposes a parallel k-means algorithm based on MapReduce. At the same time, in order to overcome the blindness of the k-means algorithm in terms of determining the initial value, we use the canopy algorithm to improve the insufficient. The experimental results show that the parallel k-means algorithm based on MapReduce has an effect on clustering before and after the improvement, not only the quality of the clustering has been increased, but in terms of processing large datasets. The speed-up ratio of the improved algorithm can get closer to the linear. Key words: MapReduce; k-means algorithm; canopy algorithm; parallel computation; cluster 随着社会信息化的不断发展, 人们日常生活中所 数据集的分布式计算框架, 它主要通过 Map( 映射)和 产生的信息也越来越多, 网络上的数据正以成倍的速 Reduce(化简)两个步骤实现对大规模数据的并行处 度日益增长, 如何从海量数据中检索到有价值的信息 理[2,3]. 简单来说, Map 是一个切分和解析的过程, 负 已成为人们最迫切的需求. 聚类分析作为数据挖掘的 责将一个大的数据集切分成多个小的数据块, 并映射 重要组成部分,