基于Hadoop的仿射传播大数据聚类分析方法.pdfVIP

Computer Engineering and Applications 计算机工程与应用 2015 ，51（4 ） 29 基于Hadoop 的仿射传播大数据聚类分析方法 唐东明 TANG Dongming 西南交通大学 信息化研究院，成都 610031 Institute of Information Research, Southwest Jiaotong University, Chengdu 610031, China TANG Dongming. Affinity propagation clustering for big data based on Hadoop. Computer Engineering and Appli- cations, 2015, 51 （4 ）：29-34. Abstract ：Affinity Propagation Clustering （APC ）is a new clustering algorithm. APC has been applied in various fields recently. However, AP can ’t be applied for analyzing large-scale data sets. To overcome this limitation, an improved Affinity Propagation cluster analysis algorithm （Affinity Propagation Clustering for Big Data Based on Hadoop, APCH ）is proposed in the Hadoop distributed computing framework. After redesigning algorithm flow based on Hadoop framework, APCH becomes parallelized cluster analysis method for large-scale data. Moreover, APCH can efficiently operate big dada, and directly determine the number of clusters. To verify the provided method, we experiment its performance on many data sets. The experimental results show that APCH provides good scalability and flexibility on big data analysis. In addition, APCH is open-source software and can be freely downloaded. APCH can be deployed on your Hadoop clusters, or Amazon Elastic Compute Cloud （Amazon EC2 ）, etc. All compiled execution binary package, user manual, including some test data sets can be downloaded from https ：///HelloWorldCN/MapReduceAPC. Key words ：affinity propagation clustering; MapReduce; Hadoop; key-value store; big data 摘 要：仿射传播聚类算法（AP ）是一个新的聚类分析方法，已经被广泛应用于各种领域。APC 算法不能用于大型 数据的分析。为了克服这个限制，在Hadoop 分布式框架的基础上提出一种改进的放射传播聚类分析方法（基于Ha- doop 的仿射传播大数据聚类分析方法，简称APCH ）。通过在Hadoop 环境下重新设计算法流程，APCH 算法成为了 一个并行化的大数据聚类分析方法。此外APCH 算法能够高效操作大数据，并能够直接决定聚类的个数。为了验 证方法的性能，在多个数据集上进行了实验。实验结果表明APCH 对大数据处理有很好的适应性和延展性。APCH 采用开源的方式提供可执行软件程序和源代码，用户可以下载后部署在自己的分布式集群中或者是部署在亚马逊 EC2 等云计算环境中。所有编译后的执行程序，源代码，用户手册