Sparse backed array example for EO data¶

Based on discussions here and here. Can DataArrays backed by sparse.COO improve performance and dask graph size over numpy.ndarray when working with with large EO time series data cubes.

In [1]:

import xarray as xr
import pandas as pd
import numpy as np
from dask.distributed import Client
import dask
import sparse

Generate a synthetic dataarray which represents the sparseness of an EO platform in a sun-synchronous orbit collecting data.¶

In [2]:

times = pd.date_range(start='2000-01-01',freq='1D',periods=200)
time_da = xr.DataArray(times, [('time', times)])

In [21]:

size = 2000
data_value = 100.0
da = xr.DataArray(np.zeros((size, size)), dims=["x", "y"], coords=[range(size), range(size)])
da = da.expand_dims(time=time_da).copy()

strip = 0
for idx, time in enumerate(da.time):
    da.data[idx, strip, 0:size] = data_value
    if strip == size - 1:
        strip = 0
    else:
        strip = strip + int(size / len(da.time))
da

Out[21]:

<xarray.DataArray (time: 200, x: 2000, y: 2000)>
array([[[100., 100., 100., ..., 100., 100., 100.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
...
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]]])
Coordinates:
  * time     (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-07-18
  * x        (x) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * y        (y) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999

xarray.DataArray

time: 200
x: 2000
y: 2000

100.0 100.0 100.0 100.0 100.0 100.0 100.0 ... 0.0 0.0 0.0 0.0 0.0 0.0

array([[[100., 100., 100., ..., 100., 100., 100.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
...
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]],

       [[  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        ...,
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.],
        [  0.,   0.,   0., ...,   0.,   0.,   0.]]])

Coordinates: (3)

time

(time)

datetime64[ns]

2000-01-01 ... 2000-07-18

array(['2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000',
       '2000-01-03T00:00:00.000000000', '2000-01-04T00:00:00.000000000',
       '2000-01-05T00:00:00.000000000', '2000-01-06T00:00:00.000000000',
       '2000-01-07T00:00:00.000000000', '2000-01-08T00:00:00.000000000',
       '2000-01-09T00:00:00.000000000', '2000-01-10T00:00:00.000000000',
       '2000-01-11T00:00:00.000000000', '2000-01-12T00:00:00.000000000',
       '2000-01-13T00:00:00.000000000', '2000-01-14T00:00:00.000000000',
       '2000-01-15T00:00:00.000000000', '2000-01-16T00:00:00.000000000',
       '2000-01-17T00:00:00.000000000', '2000-01-18T00:00:00.000000000',
       '2000-01-19T00:00:00.000000000', '2000-01-20T00:00:00.000000000',
       '2000-01-21T00:00:00.000000000', '2000-01-22T00:00:00.000000000',
       '2000-01-23T00:00:00.000000000', '2000-01-24T00:00:00.000000000',
       '2000-01-25T00:00:00.000000000', '2000-01-26T00:00:00.000000000',
       '2000-01-27T00:00:00.000000000', '2000-01-28T00:00:00.000000000',
       '2000-01-29T00:00:00.000000000', '2000-01-30T00:00:00.000000000',
       '2000-01-31T00:00:00.000000000', '2000-02-01T00:00:00.000000000',
       '2000-02-02T00:00:00.000000000', '2000-02-03T00:00:00.000000000',
       '2000-02-04T00:00:00.000000000', '2000-02-05T00:00:00.000000000',
       '2000-02-06T00:00:00.000000000', '2000-02-07T00:00:00.000000000',
       '2000-02-08T00:00:00.000000000', '2000-02-09T00:00:00.000000000',
       '2000-02-10T00:00:00.000000000', '2000-02-11T00:00:00.000000000',
       '2000-02-12T00:00:00.000000000', '2000-02-13T00:00:00.000000000',
       '2000-02-14T00:00:00.000000000', '2000-02-15T00:00:00.000000000',
       '2000-02-16T00:00:00.000000000', '2000-02-17T00:00:00.000000000',
       '2000-02-18T00:00:00.000000000', '2000-02-19T00:00:00.000000000',
       '2000-02-20T00:00:00.000000000', '2000-02-21T00:00:00.000000000',
       '2000-02-22T00:00:00.000000000', '2000-02-23T00:00:00.000000000',
       '2000-02-24T00:00:00.000000000', '2000-02-25T00:00:00.000000000',
       '2000-02-26T00:00:00.000000000', '2000-02-27T00:00:00.000000000',
       '2000-02-28T00:00:00.000000000', '2000-02-29T00:00:00.000000000',
       '2000-03-01T00:00:00.000000000', '2000-03-02T00:00:00.000000000',
       '2000-03-03T00:00:00.000000000', '2000-03-04T00:00:00.000000000',
       '2000-03-05T00:00:00.000000000', '2000-03-06T00:00:00.000000000',
       '2000-03-07T00:00:00.000000000', '2000-03-08T00:00:00.000000000',
       '2000-03-09T00:00:00.000000000', '2000-03-10T00:00:00.000000000',
       '2000-03-11T00:00:00.000000000', '2000-03-12T00:00:00.000000000',
       '2000-03-13T00:00:00.000000000', '2000-03-14T00:00:00.000000000',
       '2000-03-15T00:00:00.000000000', '2000-03-16T00:00:00.000000000',
       '2000-03-17T00:00:00.000000000', '2000-03-18T00:00:00.000000000',
       '2000-03-19T00:00:00.000000000', '2000-03-20T00:00:00.000000000',
       '2000-03-21T00:00:00.000000000', '2000-03-22T00:00:00.000000000',
       '2000-03-23T00:00:00.000000000', '2000-03-24T00:00:00.000000000',
       '2000-03-25T00:00:00.000000000', '2000-03-26T00:00:00.000000000',
       '2000-03-27T00:00:00.000000000', '2000-03-28T00:00:00.000000000',
       '2000-03-29T00:00:00.000000000', '2000-03-30T00:00:00.000000000',
       '2000-03-31T00:00:00.000000000', '2000-04-01T00:00:00.000000000',
       '2000-04-02T00:00:00.000000000', '2000-04-03T00:00:00.000000000',
       '2000-04-04T00:00:00.000000000', '2000-04-05T00:00:00.000000000',
       '2000-04-06T00:00:00.000000000', '2000-04-07T00:00:00.000000000',
       '2000-04-08T00:00:00.000000000', '2000-04-09T00:00:00.000000000',
       '2000-04-10T00:00:00.000000000', '2000-04-11T00:00:00.000000000',
       '2000-04-12T00:00:00.000000000', '2000-04-13T00:00:00.000000000',
       '2000-04-14T00:00:00.000000000', '2000-04-15T00:00:00.000000000',
       '2000-04-16T00:00:00.000000000', '2000-04-17T00:00:00.000000000',
       '2000-04-18T00:00:00.000000000', '2000-04-19T00:00:00.000000000',
       '2000-04-20T00:00:00.000000000', '2000-04-21T00:00:00.000000000',
       '2000-04-22T00:00:00.000000000', '2000-04-23T00:00:00.000000000',
       '2000-04-24T00:00:00.000000000', '2000-04-25T00:00:00.000000000',
       '2000-04-26T00:00:00.000000000', '2000-04-27T00:00:00.000000000',
       '2000-04-28T00:00:00.000000000', '2000-04-29T00:00:00.000000000',
       '2000-04-30T00:00:00.000000000', '2000-05-01T00:00:00.000000000',
       '2000-05-02T00:00:00.000000000', '2000-05-03T00:00:00.000000000',
       '2000-05-04T00:00:00.000000000', '2000-05-05T00:00:00.000000000',
       '2000-05-06T00:00:00.000000000', '2000-05-07T00:00:00.000000000',
       '2000-05-08T00:00:00.000000000', '2000-05-09T00:00:00.000000000',
       '2000-05-10T00:00:00.000000000', '2000-05-11T00:00:00.000000000',
       '2000-05-12T00:00:00.000000000', '2000-05-13T00:00:00.000000000',
       '2000-05-14T00:00:00.000000000', '2000-05-15T00:00:00.000000000',
       '2000-05-16T00:00:00.000000000', '2000-05-17T00:00:00.000000000',
       '2000-05-18T00:00:00.000000000', '2000-05-19T00:00:00.000000000',
       '2000-05-20T00:00:00.000000000', '2000-05-21T00:00:00.000000000',
       '2000-05-22T00:00:00.000000000', '2000-05-23T00:00:00.000000000',
       '2000-05-24T00:00:00.000000000', '2000-05-25T00:00:00.000000000',
       '2000-05-26T00:00:00.000000000', '2000-05-27T00:00:00.000000000',
       '2000-05-28T00:00:00.000000000', '2000-05-29T00:00:00.000000000',
       '2000-05-30T00:00:00.000000000', '2000-05-31T00:00:00.000000000',
       '2000-06-01T00:00:00.000000000', '2000-06-02T00:00:00.000000000',
       '2000-06-03T00:00:00.000000000', '2000-06-04T00:00:00.000000000',
       '2000-06-05T00:00:00.000000000', '2000-06-06T00:00:00.000000000',
       '2000-06-07T00:00:00.000000000', '2000-06-08T00:00:00.000000000',
       '2000-06-09T00:00:00.000000000', '2000-06-10T00:00:00.000000000',
       '2000-06-11T00:00:00.000000000', '2000-06-12T00:00:00.000000000',
       '2000-06-13T00:00:00.000000000', '2000-06-14T00:00:00.000000000',
       '2000-06-15T00:00:00.000000000', '2000-06-16T00:00:00.000000000',
       '2000-06-17T00:00:00.000000000', '2000-06-18T00:00:00.000000000',
       '2000-06-19T00:00:00.000000000', '2000-06-20T00:00:00.000000000',
       '2000-06-21T00:00:00.000000000', '2000-06-22T00:00:00.000000000',
       '2000-06-23T00:00:00.000000000', '2000-06-24T00:00:00.000000000',
       '2000-06-25T00:00:00.000000000', '2000-06-26T00:00:00.000000000',
       '2000-06-27T00:00:00.000000000', '2000-06-28T00:00:00.000000000',
       '2000-06-29T00:00:00.000000000', '2000-06-30T00:00:00.000000000',
       '2000-07-01T00:00:00.000000000', '2000-07-02T00:00:00.000000000',
       '2000-07-03T00:00:00.000000000', '2000-07-04T00:00:00.000000000',
       '2000-07-05T00:00:00.000000000', '2000-07-06T00:00:00.000000000',
       '2000-07-07T00:00:00.000000000', '2000-07-08T00:00:00.000000000',
       '2000-07-09T00:00:00.000000000', '2000-07-10T00:00:00.000000000',
       '2000-07-11T00:00:00.000000000', '2000-07-12T00:00:00.000000000',
       '2000-07-13T00:00:00.000000000', '2000-07-14T00:00:00.000000000',
       '2000-07-15T00:00:00.000000000', '2000-07-16T00:00:00.000000000',
       '2000-07-17T00:00:00.000000000', '2000-07-18T00:00:00.000000000'],
      dtype='datetime64[ns]')

x

(x)

int64

0 1 2 3 4 ... 1996 1997 1998 1999

array([   0,    1,    2, ..., 1997, 1998, 1999])

y

(y)

int64

0 1 2 3 4 ... 1996 1997 1998 1999

array([   0,    1,    2, ..., 1997, 1998, 1999])

Attributes: (0)

In [22]:

chunked = da.chunk(chunks={'x': 255, 'y': 255, 'time': -1})
chunked

Out[22]:

<xarray.DataArray (time: 200, x: 2000, y: 2000)>
dask.array<xarray-<this-array>, shape=(200, 2000, 2000), dtype=float64, chunksize=(200, 255, 255), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-07-18
  * x        (x) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * y        (y) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999

xarray.DataArray

time: 200
x: 2000
y: 2000

dask.array<chunksize=(200, 255, 255), meta=np.ndarray>

	Array	Chunk
Bytes	5.96 GiB	99.22 MiB
Shape	(200, 2000, 2000)	(200, 255, 255)
Count	1 Graph Layer	64 Chunks
Type	float64	numpy.ndarray

Coordinates: (3)

time

(time)

datetime64[ns]

2000-01-01 ... 2000-07-18

array(['2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000',
       '2000-01-03T00:00:00.000000000', '2000-01-04T00:00:00.000000000',
       '2000-01-05T00:00:00.000000000', '2000-01-06T00:00:00.000000000',
       '2000-01-07T00:00:00.000000000', '2000-01-08T00:00:00.000000000',
       '2000-01-09T00:00:00.000000000', '2000-01-10T00:00:00.000000000',
       '2000-01-11T00:00:00.000000000', '2000-01-12T00:00:00.000000000',
       '2000-01-13T00:00:00.000000000', '2000-01-14T00:00:00.000000000',
       '2000-01-15T00:00:00.000000000', '2000-01-16T00:00:00.000000000',
       '2000-01-17T00:00:00.000000000', '2000-01-18T00:00:00.000000000',
       '2000-01-19T00:00:00.000000000', '2000-01-20T00:00:00.000000000',
       '2000-01-21T00:00:00.000000000', '2000-01-22T00:00:00.000000000',
       '2000-01-23T00:00:00.000000000', '2000-01-24T00:00:00.000000000',
       '2000-01-25T00:00:00.000000000', '2000-01-26T00:00:00.000000000',
       '2000-01-27T00:00:00.000000000', '2000-01-28T00:00:00.000000000',
       '2000-01-29T00:00:00.000000000', '2000-01-30T00:00:00.000000000',
       '2000-01-31T00:00:00.000000000', '2000-02-01T00:00:00.000000000',
       '2000-02-02T00:00:00.000000000', '2000-02-03T00:00:00.000000000',
       '2000-02-04T00:00:00.000000000', '2000-02-05T00:00:00.000000000',
       '2000-02-06T00:00:00.000000000', '2000-02-07T00:00:00.000000000',
       '2000-02-08T00:00:00.000000000', '2000-02-09T00:00:00.000000000',
       '2000-02-10T00:00:00.000000000', '2000-02-11T00:00:00.000000000',
       '2000-02-12T00:00:00.000000000', '2000-02-13T00:00:00.000000000',
       '2000-02-14T00:00:00.000000000', '2000-02-15T00:00:00.000000000',
       '2000-02-16T00:00:00.000000000', '2000-02-17T00:00:00.000000000',
       '2000-02-18T00:00:00.000000000', '2000-02-19T00:00:00.000000000',
       '2000-02-20T00:00:00.000000000', '2000-02-21T00:00:00.000000000',
       '2000-02-22T00:00:00.000000000', '2000-02-23T00:00:00.000000000',
       '2000-02-24T00:00:00.000000000', '2000-02-25T00:00:00.000000000',
       '2000-02-26T00:00:00.000000000', '2000-02-27T00:00:00.000000000',
       '2000-02-28T00:00:00.000000000', '2000-02-29T00:00:00.000000000',
       '2000-03-01T00:00:00.000000000', '2000-03-02T00:00:00.000000000',
       '2000-03-03T00:00:00.000000000', '2000-03-04T00:00:00.000000000',
       '2000-03-05T00:00:00.000000000', '2000-03-06T00:00:00.000000000',
       '2000-03-07T00:00:00.000000000', '2000-03-08T00:00:00.000000000',
       '2000-03-09T00:00:00.000000000', '2000-03-10T00:00:00.000000000',
       '2000-03-11T00:00:00.000000000', '2000-03-12T00:00:00.000000000',
       '2000-03-13T00:00:00.000000000', '2000-03-14T00:00:00.000000000',
       '2000-03-15T00:00:00.000000000', '2000-03-16T00:00:00.000000000',
       '2000-03-17T00:00:00.000000000', '2000-03-18T00:00:00.000000000',
       '2000-03-19T00:00:00.000000000', '2000-03-20T00:00:00.000000000',
       '2000-03-21T00:00:00.000000000', '2000-03-22T00:00:00.000000000',
       '2000-03-23T00:00:00.000000000', '2000-03-24T00:00:00.000000000',
       '2000-03-25T00:00:00.000000000', '2000-03-26T00:00:00.000000000',
       '2000-03-27T00:00:00.000000000', '2000-03-28T00:00:00.000000000',
       '2000-03-29T00:00:00.000000000', '2000-03-30T00:00:00.000000000',
       '2000-03-31T00:00:00.000000000', '2000-04-01T00:00:00.000000000',
       '2000-04-02T00:00:00.000000000', '2000-04-03T00:00:00.000000000',
       '2000-04-04T00:00:00.000000000', '2000-04-05T00:00:00.000000000',
       '2000-04-06T00:00:00.000000000', '2000-04-07T00:00:00.000000000',
       '2000-04-08T00:00:00.000000000', '2000-04-09T00:00:00.000000000',
       '2000-04-10T00:00:00.000000000', '2000-04-11T00:00:00.000000000',
       '2000-04-12T00:00:00.000000000', '2000-04-13T00:00:00.000000000',
       '2000-04-14T00:00:00.000000000', '2000-04-15T00:00:00.000000000',
       '2000-04-16T00:00:00.000000000', '2000-04-17T00:00:00.000000000',
       '2000-04-18T00:00:00.000000000', '2000-04-19T00:00:00.000000000',
       '2000-04-20T00:00:00.000000000', '2000-04-21T00:00:00.000000000',
       '2000-04-22T00:00:00.000000000', '2000-04-23T00:00:00.000000000',
       '2000-04-24T00:00:00.000000000', '2000-04-25T00:00:00.000000000',
       '2000-04-26T00:00:00.000000000', '2000-04-27T00:00:00.000000000',
       '2000-04-28T00:00:00.000000000', '2000-04-29T00:00:00.000000000',
       '2000-04-30T00:00:00.000000000', '2000-05-01T00:00:00.000000000',
       '2000-05-02T00:00:00.000000000', '2000-05-03T00:00:00.000000000',
       '2000-05-04T00:00:00.000000000', '2000-05-05T00:00:00.000000000',
       '2000-05-06T00:00:00.000000000', '2000-05-07T00:00:00.000000000',
       '2000-05-08T00:00:00.000000000', '2000-05-09T00:00:00.000000000',
       '2000-05-10T00:00:00.000000000', '2000-05-11T00:00:00.000000000',
       '2000-05-12T00:00:00.000000000', '2000-05-13T00:00:00.000000000',
       '2000-05-14T00:00:00.000000000', '2000-05-15T00:00:00.000000000',
       '2000-05-16T00:00:00.000000000', '2000-05-17T00:00:00.000000000',
       '2000-05-18T00:00:00.000000000', '2000-05-19T00:00:00.000000000',
       '2000-05-20T00:00:00.000000000', '2000-05-21T00:00:00.000000000',
       '2000-05-22T00:00:00.000000000', '2000-05-23T00:00:00.000000000',
       '2000-05-24T00:00:00.000000000', '2000-05-25T00:00:00.000000000',
       '2000-05-26T00:00:00.000000000', '2000-05-27T00:00:00.000000000',
       '2000-05-28T00:00:00.000000000', '2000-05-29T00:00:00.000000000',
       '2000-05-30T00:00:00.000000000', '2000-05-31T00:00:00.000000000',
       '2000-06-01T00:00:00.000000000', '2000-06-02T00:00:00.000000000',
       '2000-06-03T00:00:00.000000000', '2000-06-04T00:00:00.000000000',
       '2000-06-05T00:00:00.000000000', '2000-06-06T00:00:00.000000000',
       '2000-06-07T00:00:00.000000000', '2000-06-08T00:00:00.000000000',
       '2000-06-09T00:00:00.000000000', '2000-06-10T00:00:00.000000000',
       '2000-06-11T00:00:00.000000000', '2000-06-12T00:00:00.000000000',
       '2000-06-13T00:00:00.000000000', '2000-06-14T00:00:00.000000000',
       '2000-06-15T00:00:00.000000000', '2000-06-16T00:00:00.000000000',
       '2000-06-17T00:00:00.000000000', '2000-06-18T00:00:00.000000000',
       '2000-06-19T00:00:00.000000000', '2000-06-20T00:00:00.000000000',
       '2000-06-21T00:00:00.000000000', '2000-06-22T00:00:00.000000000',
       '2000-06-23T00:00:00.000000000', '2000-06-24T00:00:00.000000000',
       '2000-06-25T00:00:00.000000000', '2000-06-26T00:00:00.000000000',
       '2000-06-27T00:00:00.000000000', '2000-06-28T00:00:00.000000000',
       '2000-06-29T00:00:00.000000000', '2000-06-30T00:00:00.000000000',
       '2000-07-01T00:00:00.000000000', '2000-07-02T00:00:00.000000000',
       '2000-07-03T00:00:00.000000000', '2000-07-04T00:00:00.000000000',
       '2000-07-05T00:00:00.000000000', '2000-07-06T00:00:00.000000000',
       '2000-07-07T00:00:00.000000000', '2000-07-08T00:00:00.000000000',
       '2000-07-09T00:00:00.000000000', '2000-07-10T00:00:00.000000000',
       '2000-07-11T00:00:00.000000000', '2000-07-12T00:00:00.000000000',
       '2000-07-13T00:00:00.000000000', '2000-07-14T00:00:00.000000000',
       '2000-07-15T00:00:00.000000000', '2000-07-16T00:00:00.000000000',
       '2000-07-17T00:00:00.000000000', '2000-07-18T00:00:00.000000000'],
      dtype='datetime64[ns]')

x

(x)

int64

0 1 2 3 4 ... 1996 1997 1998 1999

array([   0,    1,    2, ..., 1997, 1998, 1999])

y

(y)

int64

0 1 2 3 4 ... 1996 1997 1998 1999

array([   0,    1,    2, ..., 1997, 1998, 1999])

Attributes: (0)

Apply an aggregation operation commonly used in our ML workflows.¶

In [23]:

chunked.groupby('time.month').mean(dim='time').data.dask

Out[23]:

HighLevelGraph

HighLevelGraph with 23 layers and 1856 keys from all layers.

Layer1: xarray

xarray--b3c0e6ae34b37db5826b24abcc5826ea

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(200, 2000, 2000)
dtype	float64
chunksize	(200, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray

Layer2: getitem

getitem-beb6ae6dd5a88e92dfdaa16a6767068a

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(18, 2000, 2000)
dtype	float64
chunksize	(18, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer3: mean_chunk

mean_chunk-df94dc3f1a558e46ed07075fe7d33781

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(18, 2000, 2000)
dtype	float64
chunksize	(18, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-beb6ae6dd5a88e92dfdaa16a6767068a

Layer4: mean_agg-aggregate

mean_agg-aggregate-ae108abbf3df28e3ee0217736770c020

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-df94dc3f1a558e46ed07075fe7d33781

Layer5: getitem

getitem-f36a7c6cf37b4c467817c6ead03e95a6

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer6: mean_chunk

mean_chunk-c68709673a249cbde32aa3474398d410

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-f36a7c6cf37b4c467817c6ead03e95a6

Layer7: mean_agg-aggregate

mean_agg-aggregate-5abe3718deb4ae3c3e7926ffae8796eb

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-c68709673a249cbde32aa3474398d410

Layer8: getitem

getitem-02940c4c4e0cca74318e395a908f6f33

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer9: mean_chunk

mean_chunk-7359ba29f5579b8e526884a9817b163e

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-02940c4c4e0cca74318e395a908f6f33

Layer10: mean_agg-aggregate

mean_agg-aggregate-ae2e928bc74f1519fa55d210e2a59219

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-7359ba29f5579b8e526884a9817b163e

Layer11: getitem

getitem-0612c3d856f14432b1d422b65680e50d

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer12: mean_chunk

mean_chunk-62b58636235a4c1ebe19bdba79f20b10

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-0612c3d856f14432b1d422b65680e50d

Layer13: mean_agg-aggregate

mean_agg-aggregate-ede38176677a96ffe784dae1af470f55

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-62b58636235a4c1ebe19bdba79f20b10

Layer14: getitem

getitem-95ba3ef8e6d9207db3c42e21c48d7e10

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer15: mean_chunk

mean_chunk-687b5e52876ede0a60b67ce92161353f

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-95ba3ef8e6d9207db3c42e21c48d7e10

Layer16: mean_agg-aggregate

mean_agg-aggregate-2e5eb29a406dd32cc5741d07eb704a53

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-687b5e52876ede0a60b67ce92161353f

Layer17: getitem

getitem-e70f80cc9e4391076eb2354721fcd078

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(29, 2000, 2000)
dtype	float64
chunksize	(29, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer18: mean_chunk

mean_chunk-da20435460e769bc16ed90def3e5cc16

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(29, 2000, 2000)
dtype	float64
chunksize	(29, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-e70f80cc9e4391076eb2354721fcd078

Layer19: mean_agg-aggregate

mean_agg-aggregate-e1bf5e66b16eba8ea063cd043b8322ad

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-da20435460e769bc16ed90def3e5cc16

Layer20: getitem

getitem-5e665f1d8f4cab085eeaa7b8e8de57eb

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	xarray--b3c0e6ae34b37db5826b24abcc5826ea

Layer21: mean_chunk

mean_chunk-f745dcf8e35c8d9402cf7f7e98b6cf97

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	getitem-5e665f1d8f4cab085eeaa7b8e8de57eb

Layer22: mean_agg-aggregate

mean_agg-aggregate-59cf3b39c147c571a33b1cb17ed9a1f4

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_chunk-f745dcf8e35c8d9402cf7f7e98b6cf97

Layer23: stack

stack-d00a9f58487060aa308e00fbf55e95df

layer_type	MaterializedLayer
is_materialized	True
number of outputs	448
shape	(7, 2000, 2000)
dtype	float64
chunksize	(1, 255, 255)
type	dask.array.core.Array
chunk_type	numpy.ndarray
depends on	mean_agg-aggregate-59cf3b39c147c571a33b1cb17ed9a1f4
	mean_agg-aggregate-ae2e928bc74f1519fa55d210e2a59219
	mean_agg-aggregate-ae108abbf3df28e3ee0217736770c020
	mean_agg-aggregate-2e5eb29a406dd32cc5741d07eb704a53
	mean_agg-aggregate-e1bf5e66b16eba8ea063cd043b8322ad
	mean_agg-aggregate-ede38176677a96ffe784dae1af470f55
	mean_agg-aggregate-5abe3718deb4ae3c3e7926ffae8796eb

In [26]:

%%time
mean = chunked.groupby('time.month').mean(dim='time').compute()
mean

CPU times: user 16.3 s, sys: 17.9 s, total: 34.2 s
Wall time: 35.4 s

Out[26]:

<xarray.DataArray (month: 7, x: 2000, y: 2000)>
array([[[3.22580645, 3.22580645, 3.22580645, ..., 3.22580645,
         3.22580645, 3.22580645],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]],

       [[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
...
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]],

       [[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]]])
Coordinates:
  * x        (x) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * y        (y) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * month    (month) int64 1 2 3 4 5 6 7

xarray.DataArray

month: 7
x: 2000
y: 2000

3.226 3.226 3.226 3.226 3.226 3.226 3.226 ... 0.0 0.0 0.0 0.0 0.0 0.0

array([[[3.22580645, 3.22580645, 3.22580645, ..., 3.22580645,
         3.22580645, 3.22580645],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]],

       [[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
...
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]],

       [[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]]])

Coordinates: (3)
- x
  (x)
  int64
  0 1 2 3 4 ... 1996 1997 1998 1999
```
array([   0,    1,    2, ..., 1997, 1998, 1999])
```
- y
  (y)
  int64
  0 1 2 3 4 ... 1996 1997 1998 1999
```
array([   0,    1,    2, ..., 1997, 1998, 1999])
```
- month
  (month)
  int64
  1 2 3 4 5 6 7
```
array([1, 2, 3, 4, 5, 6, 7])
```
Attributes: (0)

Create a sparse.COO representation of the sample dataarray.¶

Use the non-zero values to determine the coordinate locations.¶

In [27]:

coords = np.transpose(np.argwhere(da.to_numpy()))
coords.shape

Out[27]:

(3, 400000)

In [28]:

s = sparse.COO(coords, data_value, shape=(len(time_da), size, size))
s

Out[28]:

Format	coo
Data Type	float64
Shape	(200, 2000, 2000)
nnz	400000
Density	0.0005
Read-only	True
Size	12.2M
Storage ratio	0.0

In [29]:

dasky_array = dask.array.from_array(s, chunks=255)
dasky_array

Out[29]:

	Array	Chunk
Shape	(200, 2000, 2000)	(200, 255, 255)
Count	2 Graph Layers	64 Chunks
Type	float64	sparse._coo.core.COO

In [30]:

da2 = xr.DataArray(dasky_array, dims=["time","x", "y"], coords=[time_da, range(size), range(size)])
da2

Out[30]:

<xarray.DataArray 'array-40493e374c37fb35762453c355a1a3c0' (time: 200, x: 2000,
                                                            y: 2000)>
dask.array<array, shape=(200, 2000, 2000), dtype=float64, chunksize=(200, 255, 255), chunktype=sparse.COO>
Coordinates:
  * time     (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-07-18
  * x        (x) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * y        (y) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999

xarray.DataArray

'array-40493e374c37fb35762453c355a1a3c0'

time: 200
x: 2000
y: 2000

dask.array<chunksize=(200, 255, 255), meta=sparse.COO>

	Array	Chunk
Shape	(200, 2000, 2000)	(200, 255, 255)
Count	2 Graph Layers	64 Chunks
Type	float64	sparse._coo.core.COO

Coordinates: (3)

time

(time)

datetime64[ns]

2000-01-01 ... 2000-07-18

array(['2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000',
       '2000-01-03T00:00:00.000000000', '2000-01-04T00:00:00.000000000',
       '2000-01-05T00:00:00.000000000', '2000-01-06T00:00:00.000000000',
       '2000-01-07T00:00:00.000000000', '2000-01-08T00:00:00.000000000',
       '2000-01-09T00:00:00.000000000', '2000-01-10T00:00:00.000000000',
       '2000-01-11T00:00:00.000000000', '2000-01-12T00:00:00.000000000',
       '2000-01-13T00:00:00.000000000', '2000-01-14T00:00:00.000000000',
       '2000-01-15T00:00:00.000000000', '2000-01-16T00:00:00.000000000',
       '2000-01-17T00:00:00.000000000', '2000-01-18T00:00:00.000000000',
       '2000-01-19T00:00:00.000000000', '2000-01-20T00:00:00.000000000',
       '2000-01-21T00:00:00.000000000', '2000-01-22T00:00:00.000000000',
       '2000-01-23T00:00:00.000000000', '2000-01-24T00:00:00.000000000',
       '2000-01-25T00:00:00.000000000', '2000-01-26T00:00:00.000000000',
       '2000-01-27T00:00:00.000000000', '2000-01-28T00:00:00.000000000',
       '2000-01-29T00:00:00.000000000', '2000-01-30T00:00:00.000000000',
       '2000-01-31T00:00:00.000000000', '2000-02-01T00:00:00.000000000',
       '2000-02-02T00:00:00.000000000', '2000-02-03T00:00:00.000000000',
       '2000-02-04T00:00:00.000000000', '2000-02-05T00:00:00.000000000',
       '2000-02-06T00:00:00.000000000', '2000-02-07T00:00:00.000000000',
       '2000-02-08T00:00:00.000000000', '2000-02-09T00:00:00.000000000',
       '2000-02-10T00:00:00.000000000', '2000-02-11T00:00:00.000000000',
       '2000-02-12T00:00:00.000000000', '2000-02-13T00:00:00.000000000',
       '2000-02-14T00:00:00.000000000', '2000-02-15T00:00:00.000000000',
       '2000-02-16T00:00:00.000000000', '2000-02-17T00:00:00.000000000',
       '2000-02-18T00:00:00.000000000', '2000-02-19T00:00:00.000000000',
       '2000-02-20T00:00:00.000000000', '2000-02-21T00:00:00.000000000',
       '2000-02-22T00:00:00.000000000', '2000-02-23T00:00:00.000000000',
       '2000-02-24T00:00:00.000000000', '2000-02-25T00:00:00.000000000',
       '2000-02-26T00:00:00.000000000', '2000-02-27T00:00:00.000000000',
       '2000-02-28T00:00:00.000000000', '2000-02-29T00:00:00.000000000',
       '2000-03-01T00:00:00.000000000', '2000-03-02T00:00:00.000000000',
       '2000-03-03T00:00:00.000000000', '2000-03-04T00:00:00.000000000',
       '2000-03-05T00:00:00.000000000', '2000-03-06T00:00:00.000000000',
       '2000-03-07T00:00:00.000000000', '2000-03-08T00:00:00.000000000',
       '2000-03-09T00:00:00.000000000', '2000-03-10T00:00:00.000000000',
       '2000-03-11T00:00:00.000000000', '2000-03-12T00:00:00.000000000',
       '2000-03-13T00:00:00.000000000', '2000-03-14T00:00:00.000000000',
       '2000-03-15T00:00:00.000000000', '2000-03-16T00:00:00.000000000',
       '2000-03-17T00:00:00.000000000', '2000-03-18T00:00:00.000000000',
       '2000-03-19T00:00:00.000000000', '2000-03-20T00:00:00.000000000',
       '2000-03-21T00:00:00.000000000', '2000-03-22T00:00:00.000000000',
       '2000-03-23T00:00:00.000000000', '2000-03-24T00:00:00.000000000',
       '2000-03-25T00:00:00.000000000', '2000-03-26T00:00:00.000000000',
       '2000-03-27T00:00:00.000000000', '2000-03-28T00:00:00.000000000',
       '2000-03-29T00:00:00.000000000', '2000-03-30T00:00:00.000000000',
       '2000-03-31T00:00:00.000000000', '2000-04-01T00:00:00.000000000',
       '2000-04-02T00:00:00.000000000', '2000-04-03T00:00:00.000000000',
       '2000-04-04T00:00:00.000000000', '2000-04-05T00:00:00.000000000',
       '2000-04-06T00:00:00.000000000', '2000-04-07T00:00:00.000000000',
       '2000-04-08T00:00:00.000000000', '2000-04-09T00:00:00.000000000',
       '2000-04-10T00:00:00.000000000', '2000-04-11T00:00:00.000000000',
       '2000-04-12T00:00:00.000000000', '2000-04-13T00:00:00.000000000',
       '2000-04-14T00:00:00.000000000', '2000-04-15T00:00:00.000000000',
       '2000-04-16T00:00:00.000000000', '2000-04-17T00:00:00.000000000',
       '2000-04-18T00:00:00.000000000', '2000-04-19T00:00:00.000000000',
       '2000-04-20T00:00:00.000000000', '2000-04-21T00:00:00.000000000',
       '2000-04-22T00:00:00.000000000', '2000-04-23T00:00:00.000000000',
       '2000-04-24T00:00:00.000000000', '2000-04-25T00:00:00.000000000',
       '2000-04-26T00:00:00.000000000', '2000-04-27T00:00:00.000000000',
       '2000-04-28T00:00:00.000000000', '2000-04-29T00:00:00.000000000',
       '2000-04-30T00:00:00.000000000', '2000-05-01T00:00:00.000000000',
       '2000-05-02T00:00:00.000000000', '2000-05-03T00:00:00.000000000',
       '2000-05-04T00:00:00.000000000', '2000-05-05T00:00:00.000000000',
       '2000-05-06T00:00:00.000000000', '2000-05-07T00:00:00.000000000',
       '2000-05-08T00:00:00.000000000', '2000-05-09T00:00:00.000000000',
       '2000-05-10T00:00:00.000000000', '2000-05-11T00:00:00.000000000',
       '2000-05-12T00:00:00.000000000', '2000-05-13T00:00:00.000000000',
       '2000-05-14T00:00:00.000000000', '2000-05-15T00:00:00.000000000',
       '2000-05-16T00:00:00.000000000', '2000-05-17T00:00:00.000000000',
       '2000-05-18T00:00:00.000000000', '2000-05-19T00:00:00.000000000',
       '2000-05-20T00:00:00.000000000', '2000-05-21T00:00:00.000000000',
       '2000-05-22T00:00:00.000000000', '2000-05-23T00:00:00.000000000',
       '2000-05-24T00:00:00.000000000', '2000-05-25T00:00:00.000000000',
       '2000-05-26T00:00:00.000000000', '2000-05-27T00:00:00.000000000',
       '2000-05-28T00:00:00.000000000', '2000-05-29T00:00:00.000000000',
       '2000-05-30T00:00:00.000000000', '2000-05-31T00:00:00.000000000',
       '2000-06-01T00:00:00.000000000', '2000-06-02T00:00:00.000000000',
       '2000-06-03T00:00:00.000000000', '2000-06-04T00:00:00.000000000',
       '2000-06-05T00:00:00.000000000', '2000-06-06T00:00:00.000000000',
       '2000-06-07T00:00:00.000000000', '2000-06-08T00:00:00.000000000',
       '2000-06-09T00:00:00.000000000', '2000-06-10T00:00:00.000000000',
       '2000-06-11T00:00:00.000000000', '2000-06-12T00:00:00.000000000',
       '2000-06-13T00:00:00.000000000', '2000-06-14T00:00:00.000000000',
       '2000-06-15T00:00:00.000000000', '2000-06-16T00:00:00.000000000',
       '2000-06-17T00:00:00.000000000', '2000-06-18T00:00:00.000000000',
       '2000-06-19T00:00:00.000000000', '2000-06-20T00:00:00.000000000',
       '2000-06-21T00:00:00.000000000', '2000-06-22T00:00:00.000000000',
       '2000-06-23T00:00:00.000000000', '2000-06-24T00:00:00.000000000',
       '2000-06-25T00:00:00.000000000', '2000-06-26T00:00:00.000000000',
       '2000-06-27T00:00:00.000000000', '2000-06-28T00:00:00.000000000',
       '2000-06-29T00:00:00.000000000', '2000-06-30T00:00:00.000000000',
       '2000-07-01T00:00:00.000000000', '2000-07-02T00:00:00.000000000',
       '2000-07-03T00:00:00.000000000', '2000-07-04T00:00:00.000000000',
       '2000-07-05T00:00:00.000000000', '2000-07-06T00:00:00.000000000',
       '2000-07-07T00:00:00.000000000', '2000-07-08T00:00:00.000000000',
       '2000-07-09T00:00:00.000000000', '2000-07-10T00:00:00.000000000',
       '2000-07-11T00:00:00.000000000', '2000-07-12T00:00:00.000000000',
       '2000-07-13T00:00:00.000000000', '2000-07-14T00:00:00.000000000',
       '2000-07-15T00:00:00.000000000', '2000-07-16T00:00:00.000000000',
       '2000-07-17T00:00:00.000000000', '2000-07-18T00:00:00.000000000'],
      dtype='datetime64[ns]')

x

(x)

int64

0 1 2 3 4 ... 1996 1997 1998 1999

array([   0,    1,    2, ..., 1997, 1998, 1999])

y

(y)

int64

0 1 2 3 4 ... 1996 1997 1998 1999

array([   0,    1,    2, ..., 1997, 1998, 1999])

Attributes: (0)

In [31]:

da2.groupby('time.month').mean(dim='time').data.dask

Out[31]:

HighLevelGraph

HighLevelGraph with 24 layers and 1857 keys from all layers.

Layer1: original-array

original-array-40493e374c37fb35762453c355a1a3c0

layer_type	MaterializedLayer
is_materialized	True
number of outputs	1

Layer2: array

array-40493e374c37fb35762453c355a1a3c0

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(200, 2000, 2000)
dtype	float64
chunksize	(200, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	original-array-40493e374c37fb35762453c355a1a3c0

Layer3: getitem

getitem-ae850b0168330aea9d97c44d9b58213c

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(18, 2000, 2000)
dtype	float64
chunksize	(18, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer4: mean_chunk

mean_chunk-5d8f77e7e5e4cddf7a06705e981ab72e

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(18, 2000, 2000)
dtype	float64
chunksize	(18, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-ae850b0168330aea9d97c44d9b58213c

Layer5: mean_agg-aggregate

mean_agg-aggregate-9969e6088c494b66321bea285ea12e1c

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-5d8f77e7e5e4cddf7a06705e981ab72e

Layer6: getitem

getitem-a23612e05f4d4532b3c789e285660bed

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer7: mean_chunk

mean_chunk-6d3971045a49c029027d479d5339b539

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-a23612e05f4d4532b3c789e285660bed

Layer8: mean_agg-aggregate

mean_agg-aggregate-cd4c1bbfc15e761e72ec060d54582724

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-6d3971045a49c029027d479d5339b539

Layer9: getitem

getitem-107980703abd81b36e6e1f5d38e61b1d

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer10: mean_chunk

mean_chunk-96d55f2040db33380698a0fd2aa5221e

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-107980703abd81b36e6e1f5d38e61b1d

Layer11: mean_agg-aggregate

mean_agg-aggregate-1e296b266deb73e84e154d330a8e0a70

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-96d55f2040db33380698a0fd2aa5221e

Layer12: getitem

getitem-a49c0b736cf54f6298bcdee3da3196ed

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer13: mean_chunk

mean_chunk-a35b9db1e508eb9d2510c80bf61ddfbc

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(30, 2000, 2000)
dtype	float64
chunksize	(30, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-a49c0b736cf54f6298bcdee3da3196ed

Layer14: mean_agg-aggregate

mean_agg-aggregate-0b6e6ff95fd34f4bc18d685833bbf0d5

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-a35b9db1e508eb9d2510c80bf61ddfbc

Layer15: getitem

getitem-ae5909dee7d0458a1476ed98b3d215af

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer16: mean_chunk

mean_chunk-67d1324649a0317ebf65e6d15ea77ac4

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-ae5909dee7d0458a1476ed98b3d215af

Layer17: mean_agg-aggregate

mean_agg-aggregate-dc5dd318ed810b36ac9261168bbc5afd

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-67d1324649a0317ebf65e6d15ea77ac4

Layer18: getitem

getitem-afc75ca982744287acbe541fd8f28699

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(29, 2000, 2000)
dtype	float64
chunksize	(29, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer19: mean_chunk

mean_chunk-b5853d7be310b4c59f4093a033de7da4

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(29, 2000, 2000)
dtype	float64
chunksize	(29, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-afc75ca982744287acbe541fd8f28699

Layer20: mean_agg-aggregate

mean_agg-aggregate-79ebd4289decb5115886d50a6cd1c8e6

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-b5853d7be310b4c59f4093a033de7da4

Layer21: getitem

getitem-d6813dfe508e4e68bf49ff6213cf11f0

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	array-40493e374c37fb35762453c355a1a3c0

Layer22: mean_chunk

mean_chunk-323124bdcfce49d1e163b3e6a625aa87

layer_type	Blockwise
is_materialized	False
number of outputs	64
shape	(31, 2000, 2000)
dtype	float64
chunksize	(31, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	getitem-d6813dfe508e4e68bf49ff6213cf11f0

Layer23: mean_agg-aggregate

mean_agg-aggregate-3135e663b8b93006e16eae7c4208b369

layer_type	MaterializedLayer
is_materialized	True
number of outputs	64
shape	(2000, 2000)
dtype	float64
chunksize	(255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_chunk-323124bdcfce49d1e163b3e6a625aa87

Layer24: stack

stack-d832ae4e2a9ee03058ad2f0aa0e66224

layer_type	MaterializedLayer
is_materialized	True
number of outputs	448
shape	(7, 2000, 2000)
dtype	float64
chunksize	(1, 255, 255)
type	dask.array.core.Array
chunk_type	sparse._coo.core.COO
depends on	mean_agg-aggregate-79ebd4289decb5115886d50a6cd1c8e6
	mean_agg-aggregate-3135e663b8b93006e16eae7c4208b369
	mean_agg-aggregate-1e296b266deb73e84e154d330a8e0a70
	mean_agg-aggregate-dc5dd318ed810b36ac9261168bbc5afd
	mean_agg-aggregate-0b6e6ff95fd34f4bc18d685833bbf0d5
	mean_agg-aggregate-cd4c1bbfc15e761e72ec060d54582724
	mean_agg-aggregate-9969e6088c494b66321bea285ea12e1c

In [32]:

%%time
sparse_mean = da2.groupby('time.month').mean(dim='time').compute()
sparse_mean

CPU times: user 1.59 s, sys: 266 ms, total: 1.86 s
Wall time: 7.86 s

Out[32]:

<xarray.DataArray 'array-40493e374c37fb35762453c355a1a3c0' (month: 7, x: 2000,
                                                            y: 2000)>
<COO: shape=(7, 2000, 2000), dtype=float64, nnz=400000, fill_value=0.0>
Coordinates:
  * x        (x) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * y        (y) int64 0 1 2 3 4 5 6 7 ... 1993 1994 1995 1996 1997 1998 1999
  * month    (month) int64 1 2 3 4 5 6 7

In [ ]: