刘志强,etal.基于SparkR的.PDFVIP

基于SparkR 的分类算法并行化研究 刘志强, 顾荣, 袁春风, 黄宜华 (南京大学计算机软件新技术国家重点实验室 南京 210046) (江苏省软件新技术与产业化协同创新中心 南京 210046) (MF 1433027@) The Parallelization of Classification Algorithms Based on SparkR Liu Zhiqiang, Gu Rong, Huang Yihua and Yuan Chunfeng (State Key Laboratory for Novel Software Technology at Nanjing University, Nanjing 210046) (Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing 210046) Abstract In recent years, the parallelization of algorithms for big data machine learning and data mining has become an important research issue in the field of big data. A few years ago, research community and industry around the world more concern about the parallel algorithms designed on the Hadoop platform. However, due to the overhead of network and disk read and write, the Hadoop MapReduce parallel computing framework cannot efficiently support machine learning algorithms which usually involve many iterative computations. With the advent and development of Spark, a new generation of big data parallel processing platform launched by UC Berkeley AMPLab, recently the research community and industry pay more attentions on the design and implementation of parallel algorithms of machine learning and data mining on the Spark platform. To support data analysts who are familiar with the R language in the general application areas to conduct the data analysis on the Spark platform, Spark provides data analysts with a programming interface, SparkR, to leverage the Spark parallel programming interfaces and powerful computing ability in the R environment. In this paper, we proposed the design and implementation of several widely-used parallel classification algorithms including Multinomial NaiveBayes, SVM and Logistic Regression based on SparkR. We also presented how we optimize the SVM and LR algorithms to i