Lazy loading a big (25GB) NetCDF4 file on S3 with Xarray¶

Just as when opening from a filesystem, when opening from s3, xarray loads:

metadata (including coordinate variables) eagerly
data variables lazily

We demonstrate this here opening a 25GB NetCDF file using S3 in a matter of seconds with memory usage below 1GB.

In [1]:

import fsspec
import xarray as xr

Inspect the big NetCDF4 file on S3 using fsspec¶

In [2]:

fs = fsspec.filesystem('s3', anon=True, client_kwargs=dict(endpoint_url='https://ncsa.osn.xsede.org'))

In [3]:

ncfile_on_s3 = 's3://esip/examples/adcirc/adcirc_01.nc'

In [4]:

fs.size(ncfile_on_s3)/1e9  # GB

Out[4]:

26.140007264

Open the file with Xarray, loading only the metadata and coordinate vars¶

In [5]:

%%time
ds = xr.open_dataset(fs.open(ncfile_on_s3), chunks={'time':10, 'node':141973})

CPU times: user 3.27 s, sys: 312 ms, total: 3.58 s
Wall time: 9.4 s

In [6]:

ds.zeta

Out[6]:

<xarray.DataArray 'zeta' (time: 720, node: 9228245)>
dask.array<open_dataset-247d281a3fb8e3c3d9504cca1b0c84eczeta, shape=(720, 9228245), dtype=float64, chunksize=(10, 141973), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2031-08-03T02:10:00 ... 2031-08-08T02:00:00
    x        (node) float64 dask.array<chunksize=(141973,), meta=np.ndarray>
    y        (node) float64 dask.array<chunksize=(141973,), meta=np.ndarray>
Dimensions without coordinates: node
Attributes:
    location:       node
    long_name:      water surface elevation above geoid
    mesh:           adcirc_mesh
    standard_name:  sea_surface_height_above_geoid
    units:          m

Compute the mean of the first 30 time steps (about 2GB of data)¶

Data is loaded lazily in chunks and in parallel by Dask using available cores

In [7]:

da = ds.zeta[:30,:]
da

Out[7]:

<xarray.DataArray 'zeta' (time: 30, node: 9228245)>
dask.array<getitem, shape=(30, 9228245), dtype=float64, chunksize=(10, 141973), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2031-08-03T02:10:00 ... 2031-08-03T07:00:00
    x        (node) float64 dask.array<chunksize=(141973,), meta=np.ndarray>
    y        (node) float64 dask.array<chunksize=(141973,), meta=np.ndarray>
Dimensions without coordinates: node
Attributes:
    location:       node
    long_name:      water surface elevation above geoid
    mesh:           adcirc_mesh
    standard_name:  sea_surface_height_above_geoid
    units:          m

In [8]:

%%time
da.mean(dim='time').compute()

CPU times: user 18.7 s, sys: 2.71 s, total: 21.5 s
Wall time: 1min 4s

Out[8]:

<xarray.DataArray 'zeta' (node: 9228245)>
array([5.01569067, 5.01422469, 5.01385336, ...,        nan,        nan,
              nan])
Coordinates:
    x        (node) float64 -91.79 -91.8 -91.8 -91.8 ... -88.19 -88.19 -88.19
    y        (node) float64 30.98 30.98 30.98 30.98 ... 30.58 30.58 30.58 30.58
Dimensions without coordinates: node