海量多维数据的存储与查询研究-计算机工程与应用.PDF

Computer Engineering and Applications 计算机工程与应用 2016 ，52（13） 25 海量多维数据的存储与查询研究 宋爱波，万雨桐，贡 欢，薛荧荧 SONG Aibo, WAN Yutong, GONG Huan, XUE Yingying 东南大学 计算机科学与工程学院，南京 211189 School of Computer Science and Engineering, Southeast University, Nanjing 211189, China SONG Aibo, WAN Yutong, GONG Huan, et al. Research on storage and query of large-scale multidimensional data. Computer Engineering and Applications, 2016, 52 （13）：25-31. Abstract ：The OLAP （Online Analytical Processing ）system built on warehouse is the most popular tool to analyze large-scale multidimensional data. With the development of information technology, data volume grows rapidly and data structure becomes more and more complicated, so the performance of OLAP system has dropped severely, failing to meet daily data analysis needs. This paper proposes new methods to store large-scale multidimensional data and perform aggre- gation query with Hadoop, a parallel computing system. The paper implements a new column-store format HCFile （HDFS column file ）, and proposals a new storage solution based on it. This project can improve the efficiency of aggregation, with a good scalability. Meanwhile, this paper leverages the hierarchy schema to build dimension hierarchy index, and uses MapReduce to perform efficiency aggregation query. Through comparison experiments with Hive, it proves that the proposed storage solution and aggregation query can effectively improve the efficiency of large-scale multidimensional data analysis. Key words ：large-scale multidimensional data; Hadoop; data index; aggregation query 摘 要：基于数据仓库的OLAP 系统是当前海量多维数据分析的主要工具。随着信息技术的发展，海量多维数据的 规模急剧增长，结构日益复杂，OLAP 系统的性能严重下降，已经无法满足人们的数据分析需求。基于分布式计算系 统Hadoop 给出了新的海量多维数据的存储方法和查询方法。设计了HDFS 上的列存储文件格式HCFile ，基于 HCFile 给出了海量多维数据存储方案，该方案能够提高聚集计算效率，并有很好的可扩展性。同时，利用多维数据 的层次性语义特征，设计了维层次索引，并给出了利用维层次索引和MapReduce 进行聚集计算的方法。通过和Hive 的对比实验，表明了数据存储方案和查询方法能够有效提高海量多维数据分析的性能。 关键词：海量多维数据；Ha