基于垂直频繁模式树带有负载均衡的分布关联规则挖掘算法.docVIP

基于垂直频繁模式树带有负载均衡的分布关联规则挖掘算法 　　摘 要： 　　大数据时代，开展面向海量、分布数据的知识发现研究成为学界和业界关注的热点，而负载均衡问题是开发分布式挖掘算法必须考虑的重要因素之一。为此，提出了一种基于垂直频繁模式树带有负载均衡的分布关联规则挖掘算法，算法采用垂直频繁模式树存储项及其关联而无需对局部挖掘结果进行合并，减少了通信量，简化了处理流程。同时所提出的算法采用混合体系结构即中心站点按照局部站点的处理能力分配任务，实现了负载均衡，提升了算法的性能。实验结果表明所提算法切实可行并具有较高效率。 　　关键词：关联规则挖掘；分布式；垂直频繁模式；负载均衡；序列化 　　中图分类号： TP311.1 　　文献标志码：A 　　Distributed rules mining algorithm with load balance based on vertical FP-tree 　　Abstract： 　　In mass data era， the research on knowledge discovery of massive and distributed data has become the hot spot in both academic field and industry. The problem of load balance is one of the important factors that must be considered in developing a distributed mining algorithm. Therefore， a distributed association rules mining algorithm with load balance based on vertical FP-tree （VFP-LBDM） was proposed in this paper. Vertical frequent pattern tree was used in this algorithm to store items and their associations， and there was no need to combine the local mining results. Therefore， the communication cost was reduced and the processing procedure was also simplified. At the same time， the algorithm used the hybrid architecture in which the central site assigned tasks according to the processing capacity of each local site. It realized the load balance and improved the performance of the algorithm. The experiment shows that the algorithm given in this paper is feasible and has higher efficiency. 　　Key words： 　　association rules mining； distribution； Vertical Frequent Pattern （VFP）； load balance； serialization 　　0 引言 　　大数据时代，企业数据量激增且分布存储于不同站点，这些数据蕴含着巨大的商业价值，企业亟需利用数据挖掘工具从中获取竞争优势。分布关联规则挖掘因其过程简捷、结果易于理解等优势，已成为应用最为广泛的挖掘方法[1-2]，其适应性改进与完善一直为学界与业界所关注。 　　目前，主流分布关联规则挖掘算法大多是为适应分布式数据环境对经典Apriori算法[3]和频繁模式增长（Frequent Pattern growth， FP-growth）算法[4]的改进，如：Tseng等[5]提出将大型数据库按照相关性进行分块后再分配给各站点进行挖掘，只在任务分配和结果返回时产生通信；何波[6]提出的基于频繁模式树的分布式关联规则挖掘算法（Distributed algorithm for Mining Association Rules based on FP-tree， DMARF）采用顶部和底部剪枝策略，有效压缩了候选项集，降低了通信量。以上算法在合并局部结果时都存在通信量开销较