并行挖掘频繁项目集新算法.docVIP

并行挖掘频繁项目集新算法 　　摘要：基于列存储的Eclat算法比基于行存储的频繁项目集挖掘算法如Apriori和FPGrowth等算法挖掘效率更高。 　　针对Eclat算法在挖掘海量数据中的频繁项目集时存在的内存和计算资源不足等问题，提出了基于Map/Reduce计算模型的并行挖掘算法――MREclat。首先，将水平型数据库转换成垂直型数据库；然后，将转换后的数据按2项集的前缀分发到各个计算节点上，且在分发数据时引入了均衡策略；接着，在各个计算节点上求出以某一前缀开头的所有频繁项目集；最后，合并各个节点的结果得到所有频繁项目集。介绍了MREclat的设计思想，研究了算法的运行性能。实验结果表明，MREclat算法效率大约是PEclat算法的2倍，加速比性能比PEclat算法提高了64%。 　　关键词：频繁项目集；并行挖掘算法；列存储；Map/Reduce；Eclat算法 　　中图分类号： TP311.13 　　文献标志码：A 　　Abstract： Eclat algorithm mines frequent itemsets from vertical layout databases. It is more efficient than those algorithms based on horizontal layout databases， such as algorithm Apriori and FPGrowth. 　　Aiming at the problem that the memory and computational capability is insufficient while using Eclat algorithm to mine frequent itemsets from massive dataset， a parallel mining algorithm based on Map/Reduce framework， called MREclat（MapReduce Eclat）， was proposed. Firstly， MREclat algorithm converted the horizontal database into a vertical one. Secondly， it redistributed the converted dataset according to the first item of each frequent 2itemset and loadbalance was taken into consideration while distributing datasets. Then， all the frequent itemsets prefixed by the same item were computed in each computing node. Finally， MREclat algorithm collected the result of each computing node and generated the whole frequent itemsets. In this paper， the idea of MREclat was introduced and the performance of the algorithm was studied. The experimental results show that MREclat algorithm is twice as efficient as PEclat algorithm， and the speedup performance of MREclat algorithm is 64% higher than that of PEclat. 　　Key words： frequent itemset； parallel mining algorithm； columnstore； Map/Reduce； Eclat algorithm 　　0引言 　　频繁项目集挖掘是数据挖掘领域的重要研究内容，它同时广泛应用于如购物篮分析、网络入侵检测和疾病诊断等领域。频繁项目集挖掘算法根据其所挖掘的数据源呈现形式分为两类：1）基于水平型（即事务型）数据库的算法，所谓水平型是指每条记录在磁盘中按行存储。这类算法中最典型的就是Apriori算法[1]和FPGrowth（Frequent Pattern Growth）算法[2]，这类算法常需要多次扫描数据库。2）基于垂直型数据的算法，垂直型的本质是记录在磁盘上以列存储方式存储。Eclat算