基于Spark的并行频繁模式挖掘算法.PDF

86 2016 ，52（20 ） Computer Engineering and Applications 计算机工程与应用 基于Spark 的并行频繁模式挖掘算法 1 2 1 1 1 曹 博 ，倪建成 ，李淋淋 ，于苹苹 ，姚彬修 1 2 1 1 1 CAO Bo , NI Jiancheng , LI Linlin , YU Pingping , YAO Binxiu 1.曲阜师范大学 信息科学与工程学院，山东 日照 276800 2.曲阜师范大学 软件学院，山东 曲阜 273100 1.College of Information Science and Engineering, Qufu Normal University, Rizhao, Shandong 276800, China 2.College of Software, Qufu Normal University, Qufu, Shandong 273100, China CAO Bo, NI Jiancheng, LI Linlin, et al. Parallel frequence pattern mining algorithm based on Spark. Computer Engineering and Applications, 2016, 52 （20 ）：86-91. Abstract ：Under the environment of big data, the frequent pattern mining algorithm Apriori has some defects, including presetting minimum threshold and high time complexity when in data processing process. Therefore, the multistage mining strategy is adopted to realize the parallel frequent pattern mining algorithm （PTFP-Apriori ）. Firstly, the preprocessed data is stored in a pattern tree, and the optimal threshold is got by the most frequent K model. Subsequently, according to the threshold, the frequent pattern that can ’t grow up to be frequent patterns could be removed to reduce the computing scale. The RDD is used to accomplish the task of itemsets support counting and candidate itemsets generating. The experimental results show that the algorithm has higher effectivity and scalability than the traditional algorithm. Key words ：big data; frequent pattern mining; Top-k; pattern tree; parallel computing 摘 要：在大数据环境下Apriori 频繁模式挖掘算法在数据处理过程具有预先设定最小阈值、时间复杂度高等缺陷, 为此采用多阶段挖掘策略实现并行化频繁模式挖掘算法PTFP-Apriori 。首先将预处理数据以模式树的形式存储，通 k 过最为频繁的 个模式得到最优阈值。然后根据该值删除预期不能成长为频繁的模式以降低计算规模，并利用弹 性分布式数据集RDD 完成统计项集支持度计数、候选项集生成的工作。实验分析表明相比于传统的频繁模式挖掘 算法，该算法具有更高的效率以及可扩展性。 关键词：大数据；频繁模式挖掘；Top-k ；模式树；并行计算 文献标志码：A 中图分类号：TP301.6 doi ：10.37