NetCDF-4SoftwareImplementinganEnhancedDataModel.pptVIP

NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder 2006-01-31 Acknowledgments This work was supported by the NASA Earth Science Technology Office under NASA award AIST-02-0071. Unidata’s work is primarily supported by the National Science Foundation. We appreciate the collaboration and development efforts of the NCSA HDF Group (now The HDF Group, Inc.). Many netCDF users have made analysis, visualization, and data management software available and have made useful suggestions for enhancements to netCDF-3: /software/netcdf/credits.html History of netCDF NetCDF’s Niche Simple data model for scientific datasets Portable, self-describing data Direct access (unlike XML) Simple language interfaces, lots of applications: C, Fortran, Java, C++, Python, Ruby, Perl NCO, ncbrowse, ncview, IDV, ArcGIS, IDL, MATLAB, … Appendable, sharable, archivable NetCDF-3 Data Model Some NetCDF-3 Limitations Only one shared unlimited dimension No structures, just scalars and multidimensional arrays No strings, just arrays of characters Limited numeric types No ragged arrays or nested structures Only ASCII characters in names Changes to file schema can be expensive Efficient access requires reads in same order as writes No built-in compression Only serial I/O Flat name space limits scalability No querying by value or indexing for fast queries NetCDF-4 Features Address Limitations Multiple unlimited dimensions Portable structured types String type Additional numeric types Variable-length types for ragged arrays Unicode names Efficient dynamic schema changes Multidimensional tiling (chunking) Per variable compression Parallel I/O Nested scopes using Groups NetCDF-4 Data Model NetCDF-4 Architecture NetCDF-4 uses HDF5 for storage, high performance Parallel I/O Chunking for efficient access in different orders Conversion using “reader makes right” approach Provides simple netCDF interface to subset