基于压缩-字对齐位图的天文海量数据实时索引.pdfVIP

Computer Engineering and Applications 计算机工程与应用 2016 ，52（1） 37 ◎大数据与云计算◎ 基于压缩-字对齐位图的天文海量数据实时索引 1，3 1，2 ，3 2 2 1，2 ，3 2 刘应波 ，王 锋 ，季凯帆 ，邓 辉 ，戴 伟 ，梁 波 ，3 ，2 ，3 2 2 ，2 ，3 2 LIU Yingbo , WANG Feng , JI Kaifan , DENG Hui , DAI Wei , LIANG Bo 1.中国科学院 云南天文台，昆明 650011 2.昆明理工大学 云南省计算机技术应用重点实验室，昆明 650500 3. 中国科学院大学，北京 100049 1.Yunnan Observatories, Chinese Academy of Sciences, Kunming 650011, China 2.Computer Technology Application Key Laboratory of Yunnan Province, Kunming University of Science and Technology, Kunming 650500, China 3.University of Chinese Academy of Sciences, Beijing 100049, China LIU Yingbo, WANG Feng, JI Kaifan, et al. Massive astronomical real-time data indexing based on compressed word-aligned bitmap. Computer Engineering and Applications, 2016, 52 （1）：37-41. Abstract ：At present, New Vacuum Solar Telescope （NVST ）is generating data more than 2 TB per day, and due to the characteristic of massive non-structure data, it is beyond traditional database systems to search efficiently for a subset of these extremely large with low latency and response time and time overtime is increasing by the way of off-line index building. Aiming at these problems, combined with the strategy of on-line real data indexing in observation data capture system, using an approach of the compressed word-aligned bitmap index for observation data indexing and querying is proposed. This approach cannot only save the time cost of accessing FITS file, reading FITS file header and parsing header keyword in off-line indexing mode, but also can solve the problem of massive solar data retrieval. Experiments show that time overhead can be reduced by using real-time data indexing, and the effectiveness and feasibility of the metho