均衡的基于MapReduce并行的FP-Growth算法BalancedParallelFP.PDF

均衡的基于MapReduce并行的FP-Growth算法 均衡的基于MapReduce并行的 FP-Growth算法 Balanced Parallel FP-Growth with MapReduce 周乐 李俊杰 黄哲学 冯圣中 摘 要  频繁项目集挖掘在关联规则挖掘，相关挖掘等许多重要的数据挖掘任务中起到非常关键的作用。但是，随着数据 集规模日益增大，已有的频繁项目集挖掘算法或是由于资源需求过大，或是由于通信消耗过多而变得低效。在这篇文章 中，我们提出了基于PFP算法的BPFP算法，PFP算法通过MapReduce的方式实现了FP-Growth算法的并行化。BPFP算法 通过在PFP算法中加入负载均衡特性而提高并行度的方法实现了算法整体性能的提升。实验结果表明，引入负载均衡分组 策略的BPFP算法性能高于采用简单分组策略的PFP算法。 关键词  FP-Growth; MapReduce; 负载均衡 ABSTRACT　Frequent itemset mining (FIM) plays an essential role in mining associations, correlations and many other important data mining tasks. Unfortunately, as the volume of dataset gets larger day by day, most of the FIM algorithms in literature become ineffective due to either too huge resource requirement or too much communication cost. In this paper, we propose a balanced parallel FP-Growth algorithm BPFP, based on the PFP algorithm [1], which parallelizes FP-Growth in the MapReduce approach. BPFP adds into PFP load balance feature, which improves parallelization and thereby improves performance. Through empirical study, BPFP outperformed the PFP which uses some simple grouping strategy. KEYWORDS FP-Growth; MapReduce; Load Balance 1 Introduction each mining unit, and then fairly divides these units into several groups in order to balance load among all groups. requent itemset mining plays an essential role With load balance feature added into the PFP algorithm, Fin mining associations, correlations, causality, BPFP partitions the whole mining task into more even sequential patterns, episodes, multidimensional patterns, subtasks, and hence achieves higher parallelization. max-patterns, partial periodicity, emerging patterns, and