大数据基准测试程序包构建方法研究.pdfVIP

第 3 卷 第4期 集 成 技 术 Vol. 3 No. 4 2014年7月 JOURNAL OF INTEGRATION TECHNOLOGY Jul. 2014 大数据基准测试程序包构建方法研究 熊 文 喻之斌  须成忠 (中国科学院深圳先进技术研究院云计算技术研究中心 深圳 518055) 摘  要  基准测试程序是评估计算机系统的关键测试工具。然而，大数据时代的到来使得开发大数据系统基准测 试程序面临着更加严峻的挑战，当前学术界和产业界还不存在得到广泛认可的大数据基准测试程序包。文章利 用实际的交通大数据系统构建了一个基于 Hadoop 平台的交通大数据基准测试程序包 SIAT-Bench 。通过选取多个 层次属性量化了程序行为特征，采用聚类算法分析了不同程序-输入数据集对的相似性。根据聚类结果，为 SIAT- Bench 选取了有代表性的程序和输入数据集。实验结果表明，SIAT-Bench 在满足程序行为多样性的同时消除了基 准测试集中的冗余。 关键词  大数据基准测试程序；输入数据集；程序相似性；城市交通系统；GPS 轨迹数据 中图分类号 TP 39 文献标志码  A An Approach to Build a Big Data Benchmark Suite XIONG Wen YU Zhibin XU Chengzhong ( Cloud Computing Technology Research Center, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China ) Abstract Benchmarks are important tools to evaluate the performance of a variety of computing systems. However, benchmarks for big data systems are lacking as big data is relatively new and researchers are interested in understanding how big data systems including hardware and software work but do not have data. In this paper, an approach to develop big data benchmarks was devised at first. Then a big data benchmark suite named SIAT-Bench, which contains five representative workloads from Shenzhen urban transportation system, was presented. To this end, the program behavior was characterized and the impact of input data sets was qualified by observing metrics from multiple levels such as micro- architecture, OS and application layer. Then statistical techniques such as Principal Component Analysis (PCA) and Cluste