基于hadoop平台的k-means聚类算法参考.pdfVIP

计 算 机 系 统 应 用 2017 年 第 26 卷 第 6 期 ① 基于Hadoop 平台的K-means 聚类算法 刘宝龙, 苏 金 (西安工业大学 计算机科学与工程学院, 西安 710021) 摘 要: 传统的 K-means 算法虽然具有很多优点, 但聚类准则函数对簇密度不均的数据集分类效果较差. 文中在 加权标准差准则函数的基础之上, 增加了收敛性判定, 并在 Hadoop 平台上提出了一种基于 MapReduce 编程思想 设计与优化的 K-means 并行算法. 与传统的 K-means 算法相比, 设计的并行算法在聚类结果的准确性、加速比、 扩展性、收敛性等方面都有显著的提高, 降低了因簇密度不均引起误分的概率, 提高了算法的聚类精度, 并且数 据规模越大、节点越多, 优化的效果就越明显. 关键词: K-means; 簇密度; 聚类精度; MapReduce; Hadoop K-Means Clustering Algorithm Based on Hadoop LIU Bao-Long, SU Jin (School of Computer Science and Engineering, Xi’an Technological University, Xi’an 710021, China) Abstract : Although there are many advantages in traditional K-means algorithm, the clustering criterion function has poor efficiency on classification of the data set with uneven cluster density. On the basis of weighted standard deviation criterion function, this paper proposes a K-means parallel algorithm which is designed and optimized based on MapReduce progra ing. And it also increases the convergence judgment. Compared with the traditional K-means algorithm, the designed parallel algorithm has a significant improvement in the aspects of accuracy, speedup ratio, scalability and the convergence of clustering results. It also reduces the probability of misclassification caused by the uneven cluster density, and improves the clustering accuracy of the algorithm. What’s more, the optimization effect will be ore obvious when it deals with lager data size and more nodes. Key words: K-means; cluster density; clustering accuracy; MapReduce; Hadoop 随着电