模集群上基于预散列的连接处理和优化.pdf

ISSN 1000—1239／CN11一I 计算机研究与发展 777／TP of Research Journal and 53．2010 Computer Development 47(Suppl．)：146—1 大规模集群上基于预散列的连接处理和优化 常洞霞 钱卫宁 周傲英 (华东师范大学软件学院上海200062) (wnqian@sei．ecnu．edu．cn) Join and BasedonPre—Hashover Clusters ProcessingOptimizing Large Zhou ChangDongxia，QJanWeining，andAoying ChinaNormal 200062) (SoftⅡ)areEngineeringInstitute，East University，Shanghai Withthe ofdatasizeinmodern andthe of Abstract increasing applicationsemergencecomplicated traditionalcentralizedanddistributeddata can’twork needs，the processing processingtechnologies well．Inthissituation，thescaleclustershavebeenused indata-intensive large widely applications becauseoftheir andfault have tolerance．Inthis three scalability，highavailability paper，we contributionsto and over thebottleneck large joinprocessingoptimizing clusters．Firstly，weanalyze ofHash basedon amethodwhichuses joinprocessingMap／Reduceframework．Secondly，wepropose to in researchthe ofstar pre-HashoptimizejoinMap／Reduce．Thirdly，weoptimizationtechnology fromthecostmodelestimationand methodscan the join．And experiments，our improvejoin