NetCDF
From Wikipedia, the free encyclopedia
| Network Common Data Form | |
|---|---|
| File name extension | .nc |
| Internet media type | application/netcdf |
| Magic number | CDF\001 |
| Developed by | UCAR |
| Type of format | scientific binary data |
| Extended from | CDF |
NetCDF (Network Common Data Form) is a machine-independent, self-describing, binary data format standard for exchanging scientific data. The project homepage is hosted by the Unidata program at the University Corporation for Atmospheric Research (UCAR). They are also the chief source of netCDF software, standards development, updates etc. The format is an open standard.
The project is actively supported, currently at version 3, and with plans underway for a version 4. It is planned to merge the project in version 4 with the HDF5 data file format, giving an alternative netCDF interface to HDF5 files.
The format was originally based on the conceptual model of the NASA CDF but has since diverged and is not compatible with it.
Contents |
[edit] Format description
The data format is "self-describing". This means that there is a header which describes the layout of the rest of the file, in particular the data arrays, as well as arbitrary file metadata in the form of name/value attributes. The format is platform independent, with issues such as endianness being addressed in the software libraries. The data arrays are rectangular, not ragged, and stored in a simple and regular fashion that allows efficient subsetting.
The new 4.0 version of the netCDF API, which is in development now, will be extended and implemented on top of the HDF5 data format. NetCDF users will be able to create HDF5 files with benefits not available with the netCDF format, such as much larger files and multiple unlimited dimensions. Backward compatibility in accessing old netCDF files will be supported. The combined library will preserve the desirable common characteristics of netCDF and HDF5 while taking advantage of their separate strengths: the widespread use and simplicity of netCDF and the generality and performance of HDF5.
[edit] Software
[edit] Access libraries
The software libraries supplied by UCAR provide read-write access to netCDF files, encoding and decoding the necessary arrays and metadata. The core library is written in C, and provides an API for C, C++ and Fortran applications. An independent implementation, also developed and maintained by Unidata, is written in 100% Java, which extends the core data model and adds additional functionality. Interfaces to netCDF based on the C library are also available in other languages including R (ncdf and ncvar packages), Perl, Python, Ruby, Matlab, IDL, and Octave. The specification of the API calls is very similar across the different languages, apart from inevitable differences of syntax. The API calls for version 2 were rather different from those in version 3, but are also supported by version 3 for backward compatibility. Application programmers using supported languages need not normally be concerned with the file structure itself, even though it is available as an open format.
[edit] Applications
A wide range of application software has been written which makes use of netCDF files. These range from command line utilities to graphical visualization packages.
- A commonly used set of Unix command line utilities for netCDF files is the NetCDF Operators (NCO) suite, which provide a range of commands for manipulation and analysis of netCDF files including basic record concatenating, slicing and averaging.
- NcBrowse is a generic netCDF file viewer that includes Java graphics, animations and 3D visualizations for a wide range of netCDF file conventions.
- ncview is a visual browser for netCDF format files. Typically you would use ncview to get a quick and easy, push-button look at your netCDF files. You can view simple movies of the data, view along various dimensions, take a look at the actual data values, change color maps, invert the data, etc.
- Panoply is a netCDF file viewer developed at the NASA Goddard Institute for Space Studies which focuses on presentation of geo-gridded data. It is written in Java and thus platform independent. Although its feature set overlaps with ncBrowse and ncview, Panoply is distinguished by offering a wide variety of map projections and ability to work with different scale color tables.
- The NCAR Command Language is used to analyze and visualize data in netCDF files (among other formats).
- PyNIO is a Python programming language module that allows read and/or write access to a variety of data formats, including netCDF.
- Ferret is an interactive computer visualization and analysis environment designed to meet the needs of oceanographers and meteorologists analyzing large and complex gridded data sets. Ferret offers a Mathematica-like approach to analysis; new variables may be defined interactively as mathematical expressions involving data set variables. Calculations may be applied over arbitrarily shaped regions. Fully documented graphics are produced with a single command.
- nCDF_Browser is a visual nCDF browser, written in the IDL programming language. Variables, attributes, and dimensions can be immediately downloaded to the IDL command line for further processing. All the Coyote Library files necessary to run nCDF_Browser are available in the zip file.
[edit] Common uses
It is commonly used in climatology and meteorology applications (e.g., weather forecasting, climate change) and GIS applications.
It is an input/output format for many GIS applications, and for general scientific data exchange. To quote from their site "NetCDF (network Common Data Form) is an interface for array-oriented data access and a library that provides an implementation of the interface. The netCDF library also defines a machine-independent format for representing scientific data."
[edit] Parallel-NetCDF
An extension of netCDF for parallel computing called Parallel-NetCDF (or PnetCDF) has been developed by Argonne National Laboratory and Northwestern University [1]. This is built upon MPI-IO, the I/O extension to MPI communications, and using the high-level netCDF data structures, the PnetCDF libraries can make use of optimizations to efficiently distribute the file read and write applications between multiple processors.
[edit] NetCDF-Java Common Data Model
The NetCDF-Java library currently reads the following file formats and remote access protocols:
- BUFR (ongoing development)
- CINRAD level II (Chinese Radar format)
- DMSP (Defense Meteorological Satellite Program)
- DORADE radar file format
- GINI (GOES Ingest and NOAAPORT Interface ) image format
- GEMPAK gridded data
- GRIB version 1 and version 2 (ongoing work on tables)
- GTOPO 30-sec elevation dataset (USGS)
- Hierarchical Data Format (HDF4, HDF-EOS, HDF5, HDF5-EOS)
- NetCDF (classic and large format)
- NetCDF-4(built on HDF5)
- NEXRAD Radar level 2 and level 3.
- OPeNDAP client/server protocol
There are a number of other formats in development. Since each of these is accessed transparently through the NetCDF API, the NetCDF-Java library is said to implement a Common Data Model for scientific datasets.
The Common Data Model has three layers, which build on top of each other to add successively richer semantics:
- The data access layer, also know as the syntactic layer, handles data reading.
- The coordinate system layer identifies the coordinates of the data arrays. Coordinates are a completely general concept for scientific data; specialized georeferencing coordinate systems, important to the Earth Science community, are specially annotated.
- The scientific data type layer identifies specific types of data, such as grids, images, and point data, and adds specialized methods for each kind of data.
The Data model of the data access layer is a generalization of the NetCDF-3 data model, and substantially the same as the NetCDF-4 data model. The coordinate system layer implements and extends the concepts in the CF Metadata Conventions. The scientific data type layer allows data to be manipulated in coordinate space, analogous to the Open Geospatial Consortium specifications. The identification of coordinate systems and data typing is ongoing, but users can plug in their own classes at runtime for specialized processing.
[edit] See also
- Common Data Format (CDF)
- GRIB (GRIdded Binary)
- Hierarchical Data Format (HDF)
- FITS
[edit] External links
- netCDF project page at the University Corporation for Atmospheric Research
- netCDF-Java project
- netCDF4 project page
- ncBrowse - generic netCDF file viewer - includes Java graphics, animations and 3D visualizations for wide range of netCDF file conventions
- "An Introduction to Distributed Visualization"; section 4.2 contains a comparison of CDF, HDF, and netCDF.
- NCO a suite of programs known as operators, which facilitate manipulation and analysis of netCDF files
- dapper data server and DChart web client for OPeNDAP in-situ data in netCDF format
This article was originally based on material from the Free On-line Dictionary of Computing, which is licensed under the GFDL.
