大规模数据密集型系统中去重查询优化.pdfVIP

ISSNN 111777TP Journal of Computer Research and Development 7( ) : 581588, 2010 1, 2 1, 2 1, 2 3 1 宋怀明 安明远 王 洋 袁春阳 孙凝晖 1 ( 100190) 2 ( 1000 9) 3 ( 100029) ( shm@ ncic. ac. cn) Duplication Elimination in Large Scale Data Intensive Systems 1, 2 1, 2 1, 2 3 1 Song Huaiming , An Mingyuan , Wang Yang , Yuan Chunyang , and Sun Ninghui 1 (K ey L aboratory of Comp uter System and Architecture, I nstituteof Comp uting Technology , ChineseA cademy of Sciences, Beij ing 100190) 2 ( rad uate University of Chinese A cademy of Sciences, Beij ing 1000 9) 3 (N ational Comp uter N etw ork E mergency Resp onse Technical TeamCoord ination Center of China, Beij ing 100029) Abstract As the emerging data intensive applications have received more and more attentions from researchers, its a severe challenge for duplication elimination for large volume data in a shared nothing environment. The authors propose an effective and adaptive data placement method w hich is a combination of hash partition and histogram, as w ell as a design of an asynchronous parallel query engine ( APQE) for duplication elimination. Hash partition divides data into nonrelevant subsets in order to reduce data migration in duplication elimination, w hile histogram method keeps balance in data size in different nodes. Furthermore, adaptive approach can make data size rebalanced w hile data skew occurs. The parallel query engine develops maximum degree of pipeline parallelism for large scale data processing by employing coarsegrained pipelining, and the asynchronous method makes further efforts to eliminate synchronous overhead of multiple nodes parallelism. APQE launches data m