分布式大数据函数依赖发现【精选】.pdfVIP

计算 机研 究与发展 DOI：10．7544／issn1000—1239。2015 JournalofComputerResearchandDevelopment 52(2)j282—294，2015 分布式大数据函数依赖发现 李卫榜 李战}；i= 陈 群 姜 涛 刘海龙 潘 巍 (西北工业大学计算机科学学院 西安 710072) (wbli2003@163．com) FunctionalDependenciesDiscoveringinDistributedBigData LiW eibang，LiZhanhuai，ChenQun，JiangTao，LiuHailong，andPanW ei (CollegeofComputerScience，NorthwesternPolytechnicalUniversity ·X ’an 710072) Abstract Discovering functionaldependencies (FDs) from relationa1databases iS an important databaseanalysistechnique，whichhasawiderangeofapplicationsinknowledgediscovery，database semanticsanalysis，data quality assessmentand databasedesign．Existing functionaldependencies discoveryalgorithmsaremainlyappliedincentralizeddata，andaresuitabletothecaseofsmalldata sizeonly．However，itisfarmorechallengingtodiscoverfunctionaldependenciesindistributed databases，especially with big data． In this paper，we proposea novelfunctionaldependencies discoveringapproach in distributedbigdata．Firstlyweexecutefunctionaldependenciesdiscovering algorithm inparallelin each node，then prunethecandidatesetoffunctionaldependenciesbasedOn theregultsofdisc0very．Secondlywegroupthecandidatesetoffunctionaldependenciesaccordingto thefeaturesofcandidatefunctionaldependencies’lefthandside，and executefunctionaldependencies discoveryalgorithm basedon each candidate setin parallel，and getallthe functionaldependency eventually．W eanalyzethenumberofcandidatefunctionswith regardtodifferentgroups，and data shipment and load balance are taken into account when discovering functionaldependencies． Experiments on real～world big datasets demonstrate that compared with previous discovering methods，ourapproachiSmoreeffectiveinefficiency． Key words discovering functional dependencies；functionaldependencies；big data；knowledge discovery；parallelcomputing 摘 要 在关 系数据库 中，函数依赖发现是一种十分重要 的数据库分析技术，在知识发现、