Processing ERA5 data in NetCDF Format

This notebook demonstrates how to work with the ECMWF ERA5 reanalysis available as part of the AWS Public Dataset Program (https://registry.opendata.aws/ecmwf-era5/).

This notebook utilizes Amazon SageMaker & AWS Fargate for providing an environment with a Jupyter notebook and Dask cluster. There is an example AWS CloudFormation template available at https://github.com/awslabs/amazon-asdi/tree/main/examples/dask for quickly creating this environment in your own AWS account to run this notebook.

Python Imports

In [1]:
%matplotlib inline
import boto3
import botocore
import datetime
import matplotlib.pyplot as plt
import matplotlib
import xarray as xr
import numpy as np
import s3fs
import fsspec
import dask
from dask.distributed import performance_report, Client, progress

font = {'family' : 'sans-serif',
        'weight' : 'normal',
        'size'   : 18}
matplotlib.rc('font', **font)

Scale out Dask Workers

In [2]:
ecs = boto3.client('ecs')
resp = ecs.list_clusters()
clusters = resp['clusterArns']
if len(clusters) > 1:
    print("Please manually select your cluster")
cluster = clusters[0]
cluster
Out[2]:
'arn:aws:ecs:us-east-1:816257832715:cluster/dask-DaskEnvironmentStack-KGPO2QLZ9KQO-Fargate-Dask-Cluster'

You will need to update the --cluster option in the comamnd below to make your cluster name from above

In [13]:
numWorkers=70
ecs.update_service(cluster=cluster, service='Dask-Worker', desiredCount=numWorkers)
ecs.get_waiter('services_stable').wait(cluster=cluster, services=['Dask-Worker'])

Set up the Dask Client to talk to our Fargate Dask Distributed Cluster

In [14]:
client = Client('Dask-Scheduler.local-dask:8786')
client
/home/ec2-user/anaconda3/envs/daskpy3/lib/python3.7/site-packages/distributed/client.py:1105: VersionMismatchWarning: Mismatched versions found

+---------+----------------+---------------+---------------+
| Package | client         | scheduler     | workers       |
+---------+----------------+---------------+---------------+
| python  | 3.7.10.final.0 | 3.7.9.final.0 | 3.7.9.final.0 |
+---------+----------------+---------------+---------------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Out[14]:

Client

Client-0d8e5176-26bc-11ec-b033-12f573b2e3a7

Connection method: Direct
Dashboard: http://Dask-Scheduler.local-dask:8787/status

Scheduler Info

Scheduler

Scheduler-c00d6c21-3b25-4b8d-8708-e5d7deb0b6ce

Comm: tcp://10.10.9.203:8786 Workers: 70
Dashboard: http://10.10.9.203:8787/status Total threads: 140
Started: 19 hours ago Total memory: 456.35 GiB

Workers

Worker: tcp://10.10.102.223:9000

Comm: tcp://10.10.102.223:9000 Total threads: 2
Dashboard: http://10.10.102.223:39689/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-2_cw8mpt
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.00 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.102.37:9000

Comm: tcp://10.10.102.37:9000 Total threads: 2
Dashboard: http://10.10.102.37:45065/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-jw3k3zzs
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.05 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.104.248:9000

Comm: tcp://10.10.104.248:9000 Total threads: 2
Dashboard: http://10.10.104.248:34261/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-awcbwl1j
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.05 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.106.112:9000

Comm: tcp://10.10.106.112:9000 Total threads: 2
Dashboard: http://10.10.106.112:35235/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-njqpzw4j
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.20 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.106.17:9000

Comm: tcp://10.10.106.17:9000 Total threads: 2
Dashboard: http://10.10.106.17:45743/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-gd8l_71u
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.27 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.107.10:9000

Comm: tcp://10.10.107.10:9000 Total threads: 2
Dashboard: http://10.10.107.10:35693/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-9af9s6cx
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 88.79 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.107.152:9000

Comm: tcp://10.10.107.152:9000 Total threads: 2
Dashboard: http://10.10.107.152:35279/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-y1e3dwrc
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.10 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.110.213:9000

Comm: tcp://10.10.110.213:9000 Total threads: 2
Dashboard: http://10.10.110.213:41041/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-wru_f58_
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.12 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.110.246:9000

Comm: tcp://10.10.110.246:9000 Total threads: 2
Dashboard: http://10.10.110.246:45957/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-y81tv2hw
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.38 MiB Spilled bytes: 0 B
Read bytes: 0.91 kiB Write bytes: 4.89 kiB

Worker: tcp://10.10.111.128:9000

Comm: tcp://10.10.111.128:9000 Total threads: 2
Dashboard: http://10.10.111.128:45299/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-rzpr8_yw
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.11 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.111.39:9000

Comm: tcp://10.10.111.39:9000 Total threads: 2
Dashboard: http://10.10.111.39:41305/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-2kp655dg
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 88.77 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.111.57:9000

Comm: tcp://10.10.111.57:9000 Total threads: 2
Dashboard: http://10.10.111.57:46833/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-y06rdndo
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.63 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.112.187:9000

Comm: tcp://10.10.112.187:9000 Total threads: 2
Dashboard: http://10.10.112.187:36585/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-p5xjsurh
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.12 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.114.179:9000

Comm: tcp://10.10.114.179:9000 Total threads: 2
Dashboard: http://10.10.114.179:39143/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ff4z50s7
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.28 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.117.122:9000

Comm: tcp://10.10.117.122:9000 Total threads: 2
Dashboard: http://10.10.117.122:38535/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-kmput86h
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.29 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.117.175:9000

Comm: tcp://10.10.117.175:9000 Total threads: 2
Dashboard: http://10.10.117.175:38695/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-0jtcevap
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.92 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.119.178:9000

Comm: tcp://10.10.119.178:9000 Total threads: 2
Dashboard: http://10.10.119.178:45445/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-anq6j_gz
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.53 MiB Spilled bytes: 0 B
Read bytes: 861.9187219581622 B Write bytes: 4.79 kiB

Worker: tcp://10.10.119.225:9000

Comm: tcp://10.10.119.225:9000 Total threads: 2
Dashboard: http://10.10.119.225:35235/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-un4j_mvx
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.36 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.119.93:9000

Comm: tcp://10.10.119.93:9000 Total threads: 2
Dashboard: http://10.10.119.93:45367/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ktfsl4l7
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.07 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.121.252:9000

Comm: tcp://10.10.121.252:9000 Total threads: 2
Dashboard: http://10.10.121.252:45735/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-pj8k20ec
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.79 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.122.155:9000

Comm: tcp://10.10.122.155:9000 Total threads: 2
Dashboard: http://10.10.122.155:46161/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-bw0j9f_f
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.47 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.123.156:9000

Comm: tcp://10.10.123.156:9000 Total threads: 2
Dashboard: http://10.10.123.156:46049/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-f7e9jmib
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.02 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.123.172:9000

Comm: tcp://10.10.123.172:9000 Total threads: 2
Dashboard: http://10.10.123.172:41387/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-c787a_a3
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.10 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.124.153:9000

Comm: tcp://10.10.124.153:9000 Total threads: 2
Dashboard: http://10.10.124.153:34823/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-xq9wm_43
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.98 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.124.158:9000

Comm: tcp://10.10.124.158:9000 Total threads: 2
Dashboard: http://10.10.124.158:45455/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-62x0law7
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.03 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.125.201:9000

Comm: tcp://10.10.125.201:9000 Total threads: 2
Dashboard: http://10.10.125.201:34381/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-so6gwrcj
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.05 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.126.210:9000

Comm: tcp://10.10.126.210:9000 Total threads: 2
Dashboard: http://10.10.126.210:40881/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-wvyj1y5b
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 88.76 MiB Spilled bytes: 0 B
Read bytes: 863.8274118015552 B Write bytes: 4.80 kiB

Worker: tcp://10.10.127.63:9000

Comm: tcp://10.10.127.63:9000 Total threads: 2
Dashboard: http://10.10.127.63:35229/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-v0iok51s
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.43 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.65.171:9000

Comm: tcp://10.10.65.171:9000 Total threads: 2
Dashboard: http://10.10.65.171:34259/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-rtuo7aw3
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.33 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.66.89:9000

Comm: tcp://10.10.66.89:9000 Total threads: 2
Dashboard: http://10.10.66.89:39881/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-wf_i1ixx
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.69 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.67.160:9000

Comm: tcp://10.10.67.160:9000 Total threads: 2
Dashboard: http://10.10.67.160:34121/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-0t1kdy4y
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.32 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.67.192:9000

Comm: tcp://10.10.67.192:9000 Total threads: 2
Dashboard: http://10.10.67.192:38447/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-9xpnewdo
Tasks executing: 2 Tasks in memory: 41
Tasks ready: 2312 Tasks in flight: 0
CPU usage: 8.0% Last seen: Just now
Memory usage: 2.82 GiB Spilled bytes: 0 B
Read bytes: 14.65 MiB Write bytes: 42.11 kiB

Worker: tcp://10.10.67.199:9000

Comm: tcp://10.10.67.199:9000 Total threads: 2
Dashboard: http://10.10.67.199:34453/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-x40y1_pu
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.14 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.68.149:9000

Comm: tcp://10.10.68.149:9000 Total threads: 2
Dashboard: http://10.10.68.149:33695/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ugugpfzf
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.04 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.68.57:9000

Comm: tcp://10.10.68.57:9000 Total threads: 2
Dashboard: http://10.10.68.57:42267/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-nfdq2lj2
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.21 MiB Spilled bytes: 0 B
Read bytes: 863.3618913023383 B Write bytes: 4.79 kiB

Worker: tcp://10.10.69.153:9000

Comm: tcp://10.10.69.153:9000 Total threads: 2
Dashboard: http://10.10.69.153:35835/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-zm2xuke8
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.38 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.69.252:9000

Comm: tcp://10.10.69.252:9000 Total threads: 2
Dashboard: http://10.10.69.252:33557/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-e68g78xr
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 88.64 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.69.60:9000

Comm: tcp://10.10.69.60:9000 Total threads: 2
Dashboard: http://10.10.69.60:41145/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-2drakph2
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.21 MiB Spilled bytes: 0 B
Read bytes: 0.92 kiB Write bytes: 4.80 kiB

Worker: tcp://10.10.71.203:9000

Comm: tcp://10.10.71.203:9000 Total threads: 2
Dashboard: http://10.10.71.203:44529/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-s3f0r10s
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.97 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.73.76:9000

Comm: tcp://10.10.73.76:9000 Total threads: 2
Dashboard: http://10.10.73.76:35151/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-wjvgu8xa
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.40 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.74.121:9000

Comm: tcp://10.10.74.121:9000 Total threads: 2
Dashboard: http://10.10.74.121:43243/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ujpiyf3o
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.20 MiB Spilled bytes: 0 B
Read bytes: 140.3187554124761 B Write bytes: 0.0 B

Worker: tcp://10.10.75.206:9000

Comm: tcp://10.10.75.206:9000 Total threads: 2
Dashboard: http://10.10.75.206:34537/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-vnacacxr
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 88.62 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.77.140:9000

Comm: tcp://10.10.77.140:9000 Total threads: 2
Dashboard: http://10.10.77.140:46169/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ny9f1wsq
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.08 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.77.246:9000

Comm: tcp://10.10.77.246:9000 Total threads: 2
Dashboard: http://10.10.77.246:42161/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-tleavps2
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.08 MiB Spilled bytes: 0 B
Read bytes: 208.1894118298394 B Write bytes: 264.24040732248847 B

Worker: tcp://10.10.78.55:9000

Comm: tcp://10.10.78.55:9000 Total threads: 2
Dashboard: http://10.10.78.55:42191/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-27njkd3x
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.32 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.78.83:9000

Comm: tcp://10.10.78.83:9000 Total threads: 2
Dashboard: http://10.10.78.83:42625/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-peb17kcn
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.93 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.79.182:9000

Comm: tcp://10.10.79.182:9000 Total threads: 2
Dashboard: http://10.10.79.182:46575/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ojnhisj9
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.47 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.80.237:9000

Comm: tcp://10.10.80.237:9000 Total threads: 2
Dashboard: http://10.10.80.237:39757/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-d7se3vdw
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.43 MiB Spilled bytes: 0 B
Read bytes: 208.4732805675421 B Write bytes: 264.60070225880344 B

Worker: tcp://10.10.81.68:9000

Comm: tcp://10.10.81.68:9000 Total threads: 2
Dashboard: http://10.10.81.68:41673/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-0q8xbngp
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.51 MiB Spilled bytes: 0 B
Read bytes: 208.0332313215698 B Write bytes: 264.0421782158386 B

Worker: tcp://10.10.82.13:9000

Comm: tcp://10.10.82.13:9000 Total threads: 2
Dashboard: http://10.10.82.13:42267/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-yiqgjo88
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.55 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.83.237:9000

Comm: tcp://10.10.83.237:9000 Total threads: 2
Dashboard: http://10.10.83.237:39047/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-yq7d61m6
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.60 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.84.167:9000

Comm: tcp://10.10.84.167:9000 Total threads: 2
Dashboard: http://10.10.84.167:40809/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-4r3svhqd
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.38 MiB Spilled bytes: 0 B
Read bytes: 347.42944192981423 B Write bytes: 263.5671628433073 B

Worker: tcp://10.10.84.47:9000

Comm: tcp://10.10.84.47:9000 Total threads: 2
Dashboard: http://10.10.84.47:46479/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-7a8kcvgf
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.73 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.84.67:9000

Comm: tcp://10.10.84.67:9000 Total threads: 2
Dashboard: http://10.10.84.67:33851/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-yke72gi_
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.12 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.84.68:9000

Comm: tcp://10.10.84.68:9000 Total threads: 2
Dashboard: http://10.10.84.68:38665/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-em__q4cd
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.19 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.85.4:9000

Comm: tcp://10.10.85.4:9000 Total threads: 2
Dashboard: http://10.10.85.4:33859/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-1k21fkr2
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.09 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.87.172:9000

Comm: tcp://10.10.87.172:9000 Total threads: 2
Dashboard: http://10.10.87.172:36205/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-8sn2jmct
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.60 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.88.115:9000

Comm: tcp://10.10.88.115:9000 Total threads: 2
Dashboard: http://10.10.88.115:35989/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-p2pteiz7
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 88.93 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.88.154:9000

Comm: tcp://10.10.88.154:9000 Total threads: 2
Dashboard: http://10.10.88.154:37489/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-gwj8bsic
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.03 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.89.219:9000

Comm: tcp://10.10.89.219:9000 Total threads: 2
Dashboard: http://10.10.89.219:35839/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-rhitnh6q
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.22 MiB Spilled bytes: 0 B
Read bytes: 863.1517278141856 B Write bytes: 4.79 kiB

Worker: tcp://10.10.89.43:9000

Comm: tcp://10.10.89.43:9000 Total threads: 2
Dashboard: http://10.10.89.43:43309/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-tl0d8kns
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.41 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.93.99:9000

Comm: tcp://10.10.93.99:9000 Total threads: 2
Dashboard: http://10.10.93.99:43551/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-5_26igfv
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 88.78 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.93.9:9000

Comm: tcp://10.10.93.9:9000 Total threads: 2
Dashboard: http://10.10.93.9:33991/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-7hktyk6z
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.18 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.94.168:9000

Comm: tcp://10.10.94.168:9000 Total threads: 2
Dashboard: http://10.10.94.168:41449/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-4pz9_hy1
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 4.0% Last seen: Just now
Memory usage: 88.74 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.94.17:9000

Comm: tcp://10.10.94.17:9000 Total threads: 2
Dashboard: http://10.10.94.17:44551/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-cf_pt58n
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.22 MiB Spilled bytes: 0 B
Read bytes: 208.1611966053424 B Write bytes: 264.20459569139615 B

Worker: tcp://10.10.94.209:9000

Comm: tcp://10.10.94.209:9000 Total threads: 2
Dashboard: http://10.10.94.209:37027/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-big84fu6
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.17 MiB Spilled bytes: 0 B
Read bytes: 862.0454320766809 B Write bytes: 4.78 kiB

Worker: tcp://10.10.94.95:9000

Comm: tcp://10.10.94.95:9000 Total threads: 2
Dashboard: http://10.10.94.95:45687/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-ag3dsyve
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.32 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.96.64:9000

Comm: tcp://10.10.96.64:9000 Total threads: 2
Dashboard: http://10.10.96.64:33481/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-w_8uyfq8
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 0.0% Last seen: Just now
Memory usage: 89.13 MiB Spilled bytes: 0 B
Read bytes: 0.91 kiB Write bytes: 4.88 kiB

Worker: tcp://10.10.98.200:9000

Comm: tcp://10.10.98.200:9000 Total threads: 2
Dashboard: http://10.10.98.200:35895/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-zq2b4vaw
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.71 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B

Worker: tcp://10.10.99.8:9000

Comm: tcp://10.10.99.8:9000 Total threads: 2
Dashboard: http://10.10.99.8:34003/status Memory: 6.52 GiB
Nanny: None
Local directory: /dask-worker-space/worker-2zqnpwme
Tasks executing: 0 Tasks in memory: 0
Tasks ready: 0 Tasks in flight: 0
CPU usage: 2.0% Last seen: Just now
Memory usage: 89.18 MiB Spilled bytes: 0 B
Read bytes: 0.0 B Write bytes: 0.0 B
In [15]:
# Make garbage collection explicit to prevent deadlocks
import gc
from distributed import WorkerPlugin

class WorkerExplicitGC(WorkerPlugin):
    def setup(self, worker):
        self.worker = worker
        gc.disable()             

    def transition(self, key, start, finish, *args, **kwargs):
       if finish == 'executing':
          self.worker._throttled_gc.collect()
          if gc.isenabled():
             gc.disable()

plugin = WorkerExplicitGC()
client.register_worker_plugin(plugin)
Out[15]:
{'tcp://10.10.102.223:9000': {'status': 'OK'},
 'tcp://10.10.102.37:9000': {'status': 'OK'},
 'tcp://10.10.104.248:9000': {'status': 'OK'},
 'tcp://10.10.106.112:9000': {'status': 'OK'},
 'tcp://10.10.106.17:9000': {'status': 'OK'},
 'tcp://10.10.107.10:9000': {'status': 'OK'},
 'tcp://10.10.107.152:9000': {'status': 'OK'},
 'tcp://10.10.110.213:9000': {'status': 'OK'},
 'tcp://10.10.110.246:9000': {'status': 'OK'},
 'tcp://10.10.111.128:9000': {'status': 'OK'},
 'tcp://10.10.111.39:9000': {'status': 'OK'},
 'tcp://10.10.111.57:9000': {'status': 'OK'},
 'tcp://10.10.112.187:9000': {'status': 'OK'},
 'tcp://10.10.114.179:9000': {'status': 'OK'},
 'tcp://10.10.117.122:9000': {'status': 'OK'},
 'tcp://10.10.117.175:9000': {'status': 'OK'},
 'tcp://10.10.119.178:9000': {'status': 'OK'},
 'tcp://10.10.119.225:9000': {'status': 'OK'},
 'tcp://10.10.119.93:9000': {'status': 'OK'},
 'tcp://10.10.121.252:9000': {'status': 'OK'},
 'tcp://10.10.122.155:9000': {'status': 'OK'},
 'tcp://10.10.123.156:9000': {'status': 'OK'},
 'tcp://10.10.123.172:9000': {'status': 'OK'},
 'tcp://10.10.124.153:9000': {'status': 'OK'},
 'tcp://10.10.124.158:9000': {'status': 'OK'},
 'tcp://10.10.125.201:9000': {'status': 'OK'},
 'tcp://10.10.126.210:9000': {'status': 'OK'},
 'tcp://10.10.127.63:9000': {'status': 'OK'},
 'tcp://10.10.65.171:9000': {'status': 'OK'},
 'tcp://10.10.66.89:9000': {'status': 'OK'},
 'tcp://10.10.67.160:9000': {'status': 'OK'},
 'tcp://10.10.67.192:9000': {'status': 'OK'},
 'tcp://10.10.67.199:9000': {'status': 'OK'},
 'tcp://10.10.68.149:9000': {'status': 'OK'},
 'tcp://10.10.68.57:9000': {'status': 'OK'},
 'tcp://10.10.69.153:9000': {'status': 'OK'},
 'tcp://10.10.69.252:9000': {'status': 'OK'},
 'tcp://10.10.69.60:9000': {'status': 'OK'},
 'tcp://10.10.71.203:9000': {'status': 'OK'},
 'tcp://10.10.73.76:9000': {'status': 'OK'},
 'tcp://10.10.74.121:9000': {'status': 'OK'},
 'tcp://10.10.75.206:9000': {'status': 'OK'},
 'tcp://10.10.77.140:9000': {'status': 'OK'},
 'tcp://10.10.77.246:9000': {'status': 'OK'},
 'tcp://10.10.78.55:9000': {'status': 'OK'},
 'tcp://10.10.78.83:9000': {'status': 'OK'},
 'tcp://10.10.79.182:9000': {'status': 'OK'},
 'tcp://10.10.80.237:9000': {'status': 'OK'},
 'tcp://10.10.81.68:9000': {'status': 'OK'},
 'tcp://10.10.82.13:9000': {'status': 'OK'},
 'tcp://10.10.83.237:9000': {'status': 'OK'},
 'tcp://10.10.84.167:9000': {'status': 'OK'},
 'tcp://10.10.84.47:9000': {'status': 'OK'},
 'tcp://10.10.84.67:9000': {'status': 'OK'},
 'tcp://10.10.84.68:9000': {'status': 'OK'},
 'tcp://10.10.85.4:9000': {'status': 'OK'},
 'tcp://10.10.87.172:9000': {'status': 'OK'},
 'tcp://10.10.88.115:9000': {'status': 'OK'},
 'tcp://10.10.88.154:9000': {'status': 'OK'},
 'tcp://10.10.89.219:9000': {'status': 'OK'},
 'tcp://10.10.89.43:9000': {'status': 'OK'},
 'tcp://10.10.93.99:9000': {'status': 'OK'},
 'tcp://10.10.93.9:9000': {'status': 'OK'},
 'tcp://10.10.94.168:9000': {'status': 'OK'},
 'tcp://10.10.94.17:9000': {'status': 'OK'},
 'tcp://10.10.94.209:9000': {'status': 'OK'},
 'tcp://10.10.94.95:9000': {'status': 'OK'},
 'tcp://10.10.96.64:9000': {'status': 'OK'},
 'tcp://10.10.98.200:9000': {'status': 'OK'},
 'tcp://10.10.99.8:9000': {'status': 'OK'}}

Open an Example File and Check the Native Chunking

We want to chunk in a similar way for maximum performance

In [16]:
url = 's3://era5-pds/2010/01/data/air_temperature_at_2_metres.nc'
ncfile = fsspec.open(url)
ds = xr.open_dataset(ncfile.open())

ds.air_temperature_at_2_metres.encoding
Out[16]:
{'chunksizes': (24, 100, 100),
 'fletcher32': False,
 'shuffle': True,
 'zlib': True,
 'complevel': 4,
 'source': '<File-like object S3FileSystem, era5-pds/2010/01/data/air_temperature_at_2_metres.nc>',
 'original_shape': (744, 721, 1440),
 'dtype': dtype('<f4'),
 '_FillValue': array([9.96921e+36], dtype=float32)}

Open 2-m air temperature as a single dataset

In [17]:
start_year = 1979
end_year = 2020
years = list(np.arange(start_year, end_year+1, 1))
months = ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"]
file_pattern = 's3://era5-pds/{year}/{month}/data/air_temperature_at_2_metres.nc'

@dask.delayed
def s3open(path):
    fs = s3fs.S3FileSystem(anon=True, default_fill_cache=False)
    return fs.open(path)

files_mapper = [s3open(file_pattern.format(year=year,month=month)) for year in years for month in months]
In [18]:
%%time

ds = xr.open_mfdataset(files_mapper, engine='h5netcdf', chunks={'lon':200,'lat':200,'time0':720}, concat_dim='time0', combine='nested', coords='minimal', compat='override', parallel=True)
CPU times: user 6.89 s, sys: 8.61 s, total: 15.5 s
Wall time: 37.9 s
In [19]:
print('ds size in GB {:0.2f}\n'.format(ds.nbytes / 1e9))
ds.info
ds size in GB 1529.06

Out[19]:
<bound method Dataset.info of <xarray.Dataset>
Dimensions:                      (lon: 1440, lat: 721, time0: 368184)
Coordinates:
  * lon                          (lon) float32 0.0 0.25 0.5 ... 359.5 359.8
  * lat                          (lat) float32 90.0 89.75 89.5 ... -89.75 -90.0
  * time0                        (time0) datetime64[ns] 1979-01-01 ... 2020-1...
Data variables:
    air_temperature_at_2_metres  (time0, lat, lon) float32 dask.array<chunksize=(720, 200, 200), meta=np.ndarray>
Attributes:
    source:       Reanalysis
    institution:  ECMWF
    title:        ERA5 forecasts>

The ds.info output above shows us that there are four dimensions to the data: lat, lon, and time0; and two data variables: air_temperature_at_2_metres, and air_pressure_at_mean_sea_level.

Convert units to F from K

In [20]:
ds['air_temperature_at_2_metres'] = (ds.air_temperature_at_2_metres - 273.15) * 9.0 / 5.0 + 32.0
ds.air_temperature_at_2_metres.attrs['units'] = 'F'

Calculate the mean 2-m air temperature for all times

In [21]:
# calculates the mean along the time dimension
temp_mean = ds['air_temperature_at_2_metres'].mean(dim='time0')

The expressions above didn’t actually compute anything. They just build the dask task graph. To do the computations, we call the persist method:

In [22]:
temp_mean = temp_mean.persist()
progress(temp_mean)

Plot Average Surface Temperature

In [23]:
temp_mean.compute()
temp_mean.plot(figsize=(20, 10))
plt.title('1979-2020 Mean 2-m Air Temperature')
Out[23]:
Text(0.5, 1.0, '1979-2020 Mean 2-m Air Temperature')

Repeat for standard deviation

In [24]:
temp_std = ds['air_temperature_at_2_metres'].std(dim='time0')
In [25]:
temp_std = temp_std.persist()
progress(temp_std)
In [15]:
temp_std.compute()
temp_std.plot(figsize=(20, 10))
plt.title('1979-2020 Standard Deviation 2-m Air Temperature')
Out[15]:
Text(0.5, 1.0, '1979-2020 Standard Deviation 2-m Air Temperature')

Plot temperature time series for points

In [26]:
# location coordinates
locs = [
    {'name': 'Santa Barbara', 'lon': -119.70, 'lat': 34.42},
    {'name': 'Colorado Springs', 'lon': -104.82, 'lat': 38.83},
    {'name': 'Honolulu', 'lon': -157.84, 'lat': 21.29},
    {'name': 'Seattle', 'lon': -122.33, 'lat': 47.61},
]

# convert westward longitudes to degrees east
for l in locs:
    if l['lon'] < 0:
        l['lon'] = 360 + l['lon']
locs
Out[26]:
[{'name': 'Santa Barbara', 'lon': 240.3, 'lat': 34.42},
 {'name': 'Colorado Springs', 'lon': 255.18, 'lat': 38.83},
 {'name': 'Honolulu', 'lon': 202.16, 'lat': 21.29},
 {'name': 'Seattle', 'lon': 237.67000000000002, 'lat': 47.61}]
In [27]:
ds_locs = xr.Dataset()
air_temp_ds = ds

# interate through the locations and create a dataset
# containing the temperature values for each location
for l in locs:
    name = l['name']
    lon = l['lon']
    lat = l['lat']
    var_name = name

    ds2 = air_temp_ds.sel(lon=lon, lat=lat, method='nearest')

    lon_attr = '%s_lon' % name
    lat_attr = '%s_lat' % name

    ds2.attrs[lon_attr] = ds2.lon.values.tolist()
    ds2.attrs[lat_attr] = ds2.lat.values.tolist()
    ds2 = ds2.rename({'air_temperature_at_2_metres' : var_name}).drop(('lat', 'lon'))

    ds_locs = xr.merge([ds_locs, ds2])

ds_locs.data_vars
Out[27]:
Data variables:
    Santa Barbara     (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
    Colorado Springs  (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
    Honolulu          (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
    Seattle           (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>

Convert to dataframe

In [28]:
df_f = ds_locs.to_dataframe()
df_f.describe()
Out[28]:
Santa Barbara Colorado Springs Honolulu Seattle
count 368184.000000 368184.000000 368184.000000 368184.000000
mean 60.281300 46.359184 75.154930 52.309616
std 9.913921 19.551779 2.568616 10.984550
min 25.767511 -25.982491 61.992512 5.742512
25% 52.655010 31.167511 73.355011 44.330009
50% 60.080009 46.355011 75.380013 51.417511
75% 67.505013 60.755013 77.067513 59.742512
max 97.092514 98.330009 83.817513 98.330009

Plot temperature timeseries

In [29]:
ax = df_f.plot(figsize=(20, 10), title="ERA5", grid=1)
ax.set(xlabel='Date', ylabel='2-m Air Temperature (deg F)')
plt.show()