Important Note: when running this notebook interactively in a browser, you probably will not be able to execute individual cells out of order without getting an error. Instead, choose "Run All" from the Cell menu after you modify a cell.
import netCDF4 # Note: python is case-sensitive!
import numpy as np
Let's create a new, empty netCDF file named 'data/new.nc', opened for writing.
Be careful, opening a file with 'w' will clobber any existing data (unless clobber=False
is used, in which case an exception is raised if the file already exists).
mode='r'
is the default.mode='a'
opens an existing file and allows for appending (does not clobber existing data)format
can be one of NETCDF3_CLASSIC
, NETCDF3_64BIT
, NETCDF4_CLASSIC
or NETCDF4
(default). NETCDF4_CLASSIC
uses HDF5 for the underlying storage layer (as does NETCDF4
) but enforces the classic netCDF 3 data model so data can be read with older clients.try: ncfile.close() # just to be safe, make sure dataset is not already open.
except: pass
ncfile = netCDF4.Dataset('data/new.nc',mode='w',format='NETCDF4_CLASSIC')
print(ncfile)
<type 'netCDF4._netCDF4.Dataset'> root group (NETCDF4_CLASSIC data model, file format HDF5): dimensions(sizes): variables(dimensions): groups:
The ncfile object we created is a container for dimensions, variables, and attributes. First, let's create some dimensions using the createDimension
method.
ncfile.dimensions
dictionary.Setting the dimension length to 0
or None
makes it unlimited, so it can grow.
NETCDF4
files, any variable's dimension can be unlimited.NETCDF4_CLASSIC
and NETCDF3*
files, only one per variable can be unlimited, and it must be the leftmost (fastest varying) dimension.lat_dim = ncfile.createDimension('lat', 73) # latitude axis
lon_dim = ncfile.createDimension('lon', 144) # longitude axis
time_dim = ncfile.createDimension('time', None) # unlimited axis (can be appended to).
for dim in ncfile.dimensions.items():
print(dim)
('lat', <type 'netCDF4._netCDF4.Dimension'>: name = 'lat', size = 73 ) ('lon', <type 'netCDF4._netCDF4.Dimension'>: name = 'lon', size = 144 ) ('time', <type 'netCDF4._netCDF4.Dimension'> (unlimited): name = 'time', size = 0 )
ncfile.title='My model data'
print(ncfile.title)
My model data
Try adding some more attributes...
Now let's add some variables and store some data in them.
The createVariable
method takes 3 mandatory args.
variables
dictionary.NETCDF4
file, any unlimited dimension must be the leftmost one.format='NETCDF4'
) to control compression, chunking, fill_value, etc.# Define two variables with the same names as dimensions,
# a conventional way to define "coordinate variables".
lat = ncfile.createVariable('lat', np.float32, ('lat',))
lat.units = 'degrees_north'
lat.long_name = 'latitude'
lon = ncfile.createVariable('lon', np.float32, ('lon',))
lon.units = 'degrees_east'
lon.long_name = 'longitude'
time = ncfile.createVariable('time', np.float64, ('time',))
time.units = 'hours since 1800-01-01'
time.long_name = 'time'
# Define a 3D variable to hold the data
temp = ncfile.createVariable('temp',np.float64,('time','lat','lon')) # note: unlimited dimension is leftmost
temp.units = 'K' # degrees Kelvin
temp.standard_name = 'air_temperature' # this is a CF standard name
print(temp)
<type 'netCDF4._netCDF4.Variable'> float64 temp(time, lat, lon) units: K standard_name: air_temperature unlimited dimensions: time current shape = (0, 73, 144) filling on, default _FillValue of 9.96920996839e+36 used
The netCDF4 module provides some useful pre-defined Python attributes for netCDF variables, such as dimensions, shape, dtype, ndim.
Note: since no data has been written yet, the length of the 'time' dimension is 0.
print("-- Some pre-defined attributes for variable temp:")
print("temp.dimensions:", temp.dimensions)
print("temp.shape:", temp.shape)
print("temp.dtype:", temp.dtype)
print("temp.ndim:", temp.ndim)
-- Some pre-defined attributes for variable temp: ('temp.dimensions:', (u'time', u'lat', u'lon')) ('temp.shape:', (0, 73, 144)) ('temp.dtype:', dtype('float64')) ('temp.ndim:', 3)
To write data a netCDF variable object, just treat it like a numpy array and assign values to a slice.
nlats = len(lat_dim); nlons = len(lon_dim); ntimes = 3
# Write latitudes, longitudes.
# Note: the ":" is necessary in these "write" statements
lat[:] = -90. + (180./nlats)*np.arange(nlats) # south pole to north pole
lon[:] = (180./nlats)*np.arange(nlons) # Greenwich meridian eastward
# create a 3D array of random numbers
data_arr = np.random.uniform(low=280,high=330,size=(ntimes,nlats,nlons))
# Write the data. This writes the whole 3D netCDF variable all at once.
temp[:,:,:] = data_arr # Appends data along unlimited dimension
print("-- Wrote data, temp.shape is now ", temp.shape)
# read data back from variable (by slicing it), print min and max
print("-- Min/Max values:", temp[:,:,:].min(), temp[:,:,:].max())
('-- Wrote data, temp.shape is now ', (3, 73, 144)) ('-- Min/Max values:', 280.00283562143028, 329.99987991477548)
Let's add another time slice....
# create a 2D array of random numbers
data_slice = np.random.uniform(low=280,high=330,size=(nlats,nlons))
temp[3,:,:] = data_slice # Appends the 4th time slice
print("-- Wrote more data, temp.shape is now ", temp.shape)
('-- Wrote more data, temp.shape is now ', (4, 73, 144))
Note that we have not yet written any data to the time variable. It automatically grew as we appended data along the time dimension to the variable temp
, but the data is missing.
print(time)
times_arr = time[:]
print(type(times_arr),times_arr) # dashes indicate masked values (where data has not yet been written)
<type 'netCDF4._netCDF4.Variable'> float64 time(time) units: hours since 1800-01-01 long_name: time unlimited dimensions: time current shape = (4,) filling on, default _FillValue of 9.96920996839e+36 used (<class 'numpy.ma.core.MaskedArray'>, masked_array(data = [-- -- -- --], mask = [ True True True True], fill_value = 9.96920996839e+36) )
Let's add write some data into the time variable.
from datetime import datetime
from netCDF4 import date2num,num2date
# 1st 4 days of October.
dates = [datetime(2014,10,1,0),datetime(2014,10,2,0),datetime(2014,10,3,0),datetime(2014,10,4,0)]
print(dates)
times = date2num(dates, time.units)
print(times, time.units) # numeric values
time[:] = times
# read time data back, convert to datetime instances, check values.
print(num2date(time[:],time.units))
[datetime.datetime(2014, 10, 1, 0, 0), datetime.datetime(2014, 10, 2, 0, 0), datetime.datetime(2014, 10, 3, 0, 0), datetime.datetime(2014, 10, 4, 0, 0)] (array([ 1882440., 1882464., 1882488., 1882512.]), u'hours since 1800-01-01') [datetime.datetime(2014, 10, 1, 0, 0) datetime.datetime(2014, 10, 2, 0, 0) datetime.datetime(2014, 10, 3, 0, 0) datetime.datetime(2014, 10, 4, 0, 0)]
It's important to close a netCDF file you opened for writing:
# first print the Dataset object to see what we've got
print(ncfile)
# close the Dataset.
ncfile.close(); print('Dataset is closed!')
<type 'netCDF4._netCDF4.Dataset'> root group (NETCDF4_CLASSIC data model, file format HDF5): title: My model data dimensions(sizes): lat(73), lon(144), time(4) variables(dimensions): float32 lat(lat), float32 lon(lon), float64 time(time), float64 temp(time,lat,lon) groups: Dataset is closed!
So far we've only exercised features associated with the old netCDF version 3 data model. netCDF version 4 adds a lot of new functionality that comes with the more flexible HDF5 storage layer.
Let's create a new file with format='NETCDF4'
so we can try out some of these features.
ncfile = netCDF4.Dataset('data/new2.nc','w',format='NETCDF4')
print(ncfile)
<type 'netCDF4._netCDF4.Dataset'> root group (NETCDF4 data model, file format HDF5): dimensions(sizes): variables(dimensions): groups:
netCDF version 4 added support for organizing data in hierarchical groups.
analagous to directories in a filesystem.
Groups serve as containers for variables, dimensions and attributes, as well as other groups.
A netCDF4.Dataset
creates a special group, called the 'root group', which is similar to the root directory in a unix filesystem.
groups are created using the createGroup
method.
takes a single argument (a string, which is the name of the Group instance). This string is used as a key to access the group instances in the groups
dictionary.
Here we create two groups to hold data for two different model runs.
grp1 = ncfile.createGroup('model_run1')
grp2 = ncfile.createGroup('model_run2')
for grp in ncfile.groups.items():
print(grp)
('model_run1', <type 'netCDF4._netCDF4.Group'> group /model_run1: dimensions(sizes): variables(dimensions): groups: ) ('model_run2', <type 'netCDF4._netCDF4.Group'> group /model_run2: dimensions(sizes): variables(dimensions): groups: )
Create some dimensions in the root group.
lat_dim = ncfile.createDimension('lat', 73) # latitude axis
lon_dim = ncfile.createDimension('lon', 144) # longitude axis
time_dim = ncfile.createDimension('time', None) # unlimited axis (can be appended to).
Now create a variable in grp1 and grp2. The library will search recursively upwards in the group tree to find the dimensions (which in this case are defined one level up).
temp1 = grp1.createVariable('temp',np.float64,('time','lat','lon'),zlib=True)
temp2 = grp2.createVariable('temp',np.float64,('time','lat','lon'),zlib=True)
for grp in ncfile.groups.items(): # shows that each group now contains 1 variable
print(grp)
('model_run1', <type 'netCDF4._netCDF4.Group'> group /model_run1: dimensions(sizes): variables(dimensions): float64 temp(time,lat,lon) groups: ) ('model_run2', <type 'netCDF4._netCDF4.Group'> group /model_run2: dimensions(sizes): variables(dimensions): float64 temp(time,lat,lon) groups: )
Here we create a variable with a compound data type to represent complex data (there is no native complex data type in netCDF).
createCompoundType
method.# create complex128 numpy structured data type
complex128 = np.dtype([('real',np.float64),('imag',np.float64)])
# using this numpy dtype, create a netCDF compound data type object
# the string name can be used as a key to access the datatype from the cmptypes dictionary.
complex128_t = ncfile.createCompoundType(complex128,'complex128')
# create a variable with this data type, write some data to it.
cmplxvar = grp1.createVariable('cmplx_var',complex128_t,('time','lat','lon'))
# write some data to this variable
# first create some complex random data
nlats = len(lat_dim); nlons = len(lon_dim)
data_arr_cmplx = np.random.uniform(size=(nlats,nlons))+1.j*np.random.uniform(size=(nlats,nlons))
# write this complex data to a numpy complex128 structured array
data_arr = np.empty((nlats,nlons),complex128)
data_arr['real'] = data_arr_cmplx.real; data_arr['imag'] = data_arr_cmplx.imag
cmplxvar[0] = data_arr # write the data to the variable (appending to time dimension)
print(cmplxvar)
data_out = cmplxvar[0] # read one value of data back from variable
print(data_out.dtype, data_out.shape, data_out[0,0])
<type 'netCDF4._netCDF4.Variable'> compound cmplx_var(time, lat, lon) compound data type: [('real', '<f8'), ('imag', '<f8')] path = /model_run1 unlimited dimensions: time current shape = (1, 73, 144) (dtype([('real', '<f8'), ('imag', '<f8')]), (73, 144), (0.578177705604801, 0.18086070805676357))
netCDF 4 has support for variable-length or "ragged" arrays. These are arrays of variable length sequences having the same type.
createVLType
method.vlen_t = ncfile.createVLType(np.int64, 'phony_vlen')
A new variable can then be created using this datatype.
vlvar = grp2.createVariable('phony_vlen_var', vlen_t, ('time','lat','lon'))
Since there is no native vlen datatype in numpy, vlen arrays are represented in python as object arrays (arrays of dtype object
).
vlen_data = np.empty((nlats,nlons),object)
for i in range(nlons):
for j in range(nlats):
size = np.random.randint(1,10,size=1) # random length of sequence
vlen_data[j,i] = np.random.randint(0,10,size=size)# generate random sequence
vlvar[0] = vlen_data # append along unlimited dimension (time)
print(vlvar)
print('data =\n',vlvar[:])
<type 'netCDF4._netCDF4.Variable'> vlen phony_vlen_var(time, lat, lon) vlen data type: int64 path = /model_run2 unlimited dimensions: time current shape = (1, 73, 144) ('data =\n', array([[[array([0, 4, 0, 9, 2, 2, 2, 4, 2]), array([7, 5, 4, 4, 9, 8, 0]), array([3, 6, 6, 8, 2, 7]), ..., array([5, 0, 0, 8, 8, 1, 5, 3]), array([4, 2, 7]), array([0])], [array([5, 6, 6, 6, 1, 0, 7]), array([7]), array([7, 5, 8, 9, 6, 9, 3]), ..., array([0, 6, 5, 4]), array([7, 1, 9, 7, 7, 2]), array([1, 4, 0])], [array([4, 3, 1]), array([6, 3, 9, 7, 8]), array([8]), ..., array([6, 5, 8, 0]), array([0]), array([0, 9, 6, 2, 4])], ..., [array([8, 4, 4]), array([4, 1, 6]), array([1, 4, 2, 3, 9]), ..., array([9, 1]), array([7, 2, 5, 1, 5, 8, 2]), array([2, 9, 9, 1, 4, 6, 3, 5, 2])], [array([4, 7, 9, 8, 2, 3, 6, 6]), array([1, 4, 1, 6, 1, 1, 2, 3, 9]), array([9, 5, 6, 2, 4, 3, 8, 2, 9]), ..., array([9, 5, 7]), array([3, 9]), array([4, 2, 6, 9])], [array([8, 9, 9, 2, 2, 8, 8, 5]), array([3]), array([8, 8, 0, 2, 9, 2, 3, 0, 9]), ..., array([7]), array([5, 1, 0, 6, 8, 6]), array([8, 6, 3, 6, 9, 8, 4, 2, 5])]]], dtype=object))
Close the Dataset and examine the contents with ncdump.
ncfile.close()
!ncdump -h data/new2.nc
netcdf new2 { types: compound complex128 { double real ; double imag ; }; // complex128 int64(*) phony_vlen ; dimensions: lat = 73 ; lon = 144 ; time = UNLIMITED ; // (1 currently) group: model_run1 { variables: double temp(time, lat, lon) ; complex128 cmplx_var(time, lat, lon) ; } // group model_run1 group: model_run2 { variables: double temp(time, lat, lon) ; phony_vlen phony_vlen_var(time, lat, lon) ; } // group model_run2 }