Parallel HDF5Introductory Tutorial课件.ppt

Writing Dataset by Column P1 P0 File Memory block[1] Block[0] P0 offset[1] P1 offset[1] stride[1] dimsm[0] dimsm[1] 精品文档 96 /* Create property list for collective dataset write. */ 98 plist_id = H5Pcreate(H5P_DATASET_XFER); -99 H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE); 100 101 status = H5Dwrite(dset_id, H5T_NATIVE_INT, 102 memspace, filespace, plist_id, data); 104 H5Dclose(dset_id); Dataset collective Write 精品文档 My PHDF5 Application I/O is slow If my application I/O performance is slow, what can I do? Use larger I/O data sizes Independent vs. Collective I/O Specific I/O system hints Increase I/O bandwidth 精品文档 Independent Vs Collective Access User reported Independent data transfer mode was much slower than the Collective data transfer mode Data array was tall and thin: 230,000 rows by 6 columns : : : 230,000 rows : : : 精品文档 Parallel HDF5Introductory Tutorial May 19, 2008 Kent Yang The HDF Group help@hdfgroup.org 精品文档 Outline Overview of Basic HDF5 concept Overview of Parallel HDF5 design MPI-IO vs. Parallel HDF5 Overview of Parallel HDF5 programming model The benefits of using Parallel HDF5 Situations where parallel HDF5 may not work well 精品文档 Overview of Basic HDF5 Concept 精品文档 What is HDF5? File format for managing any kind of data Software (library and tools) for accessing data in that format Especially suited for large and/or complex data collections Platform independent C, F90, C++, Java APIs 精品文档 Example HDF5 file “/” (root) “/foo” Raster image palette 3-D array 2-D array Raster image lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Table 精品文档 compressed extendable Metadata for Fred Dataset “Fred” File A File B Data for Fred Special Storage Options chunked compressed extendable Split file Better subsetting Access time; extendable Improves storage efficiency, Transmission speed Arrays can be extended in any direction Metadata in one file, raw data in another 精