基于spark的大数据热图可视化方法-计算机辅助设计与图形学学报.pdfVIP

第28 卷 第 11 期 计算机辅助设计与图形学学报 Vol. 28 No.11 2016 年 11 月 Journal of Computer-Aided Design Computer Graphics Nov. 2016 基于Spark 的大数据热图可视化方法 1,2) 1) 1) 1) 2)* 张 繁 , 袁兆康 , 肖凡平 , 尤 堃 , 王章野 1) (浙江工业大学计算机科学与技术学院 杭州 310023) 2) (浙江大学CADCG 国家重点实验室 杭州 310058) (zywang@) : 针对普通客户端浏览和分析大数据困难的问题, 结合 Spark 和 LOD 技术, 以热图为例提出一种面向大数据 可视化技术框架. 首先利用Spark 平台分层并以瓦片为单位并行计算, 然后将结果分布式存储在HDFS 上, 最后通过 web 服务器应用 Ajax 技术结合地理信息提供各种时空分析服务. 文中重点解决了数据点位置和地图之间的映射, 以 及由于并行计算导致的热图瓦片之间边缘偏差这 2 个问题. 实验结果表明, 该方法将数据交互操作与数据绘制和计 算任务分离, 为浏览器端大数据可视化提供了一个新的思路. ：热图; 并行计算; 大数据; 细节层次 ：TP391.41 Research on Heatmap for Big Data Based on Spark Zhang Fan1,2), Yuan Zhaokang1), Xiao Fanping1), You Kun1), and Wang Zhangye2)* 1) (College of Computer Science Technology, Zhejiang University of Technology, Hangzhou 310023) 2) (State Key Laboratory of CADCG, Zhejiang University, Hangzhou 310058) Abstract: It is important to provide data analysts with effective and efficient exploratory tools via web browsers. However, due to the characteristics of big data, current data visualization approaches can hardly display the whole datasets. This paper proposes a general-purpose visualization framework based on Spark and LOD. Firstly, we implement a tile-based parallel computing algorithm for layered datasets with Spark. Secondly, we store the tem- porary results on HDFS. Finally, with Ajax and geographic information, we provide all kinds of spatial-temporal analysis services via web. This paper resolves two problems: one is the mapping of data points from heat map to geographic map; the other is the correction of the marginal error in overlapping areas caused by parallel comput- ing. The experiment results suggest that by separating the data displ