Processing ERA5 data in NetCDF Format¶
This notebook demonstrates how to work with the ECMWF ERA5 reanalysis available as part of the AWS Public Dataset Program (https://registry.opendata.aws/ecmwf-era5/).
This notebook utilizes Amazon SageMaker & AWS Fargate for providing an environment with a Jupyter notebook and Dask cluster. There is an example AWS CloudFormation template available at https://github.com/awslabs/amazon-asdi/tree/main/examples/dask for quickly creating this environment in your own AWS account to run this notebook.
Python Imports¶
In [1]:
%matplotlib inline
import boto3
import botocore
import datetime
import matplotlib.pyplot as plt
import matplotlib
import xarray as xr
import numpy as np
import s3fs
import fsspec
import dask
from dask.distributed import performance_report, Client, progress
font = {'family' : 'sans-serif',
'weight' : 'normal',
'size' : 18}
matplotlib.rc('font', **font)
Scale out Dask Workers¶
In [2]:
ecs = boto3.client('ecs')
resp = ecs.list_clusters()
clusters = resp['clusterArns']
if len(clusters) > 1:
print("Please manually select your cluster")
cluster = clusters[0]
cluster
Out[2]:
'arn:aws:ecs:us-east-1:816257832715:cluster/dask-DaskEnvironmentStack-KGPO2QLZ9KQO-Fargate-Dask-Cluster'
You will need to update the --cluster option in the comamnd below to make your cluster name from above
In [13]:
numWorkers=70
ecs.update_service(cluster=cluster, service='Dask-Worker', desiredCount=numWorkers)
ecs.get_waiter('services_stable').wait(cluster=cluster, services=['Dask-Worker'])
Set up the Dask Client to talk to our Fargate Dask Distributed Cluster¶
In [14]:
client = Client('Dask-Scheduler.local-dask:8786')
client
/home/ec2-user/anaconda3/envs/daskpy3/lib/python3.7/site-packages/distributed/client.py:1105: VersionMismatchWarning: Mismatched versions found +---------+----------------+---------------+---------------+ | Package | client | scheduler | workers | +---------+----------------+---------------+---------------+ | python | 3.7.10.final.0 | 3.7.9.final.0 | 3.7.9.final.0 | +---------+----------------+---------------+---------------+ warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Out[14]:
Client
Client-0d8e5176-26bc-11ec-b033-12f573b2e3a7
| Connection method: Direct | |
| Dashboard: http://Dask-Scheduler.local-dask:8787/status |
Scheduler Info
Scheduler
Scheduler-c00d6c21-3b25-4b8d-8708-e5d7deb0b6ce
| Comm: tcp://10.10.9.203:8786 | Workers: 70 |
| Dashboard: http://10.10.9.203:8787/status | Total threads: 140 |
| Started: 19 hours ago | Total memory: 456.35 GiB |
Workers
Worker: tcp://10.10.102.223:9000
| Comm: tcp://10.10.102.223:9000 | Total threads: 2 |
| Dashboard: http://10.10.102.223:39689/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-2_cw8mpt | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.00 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.102.37:9000
| Comm: tcp://10.10.102.37:9000 | Total threads: 2 |
| Dashboard: http://10.10.102.37:45065/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-jw3k3zzs | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.05 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.104.248:9000
| Comm: tcp://10.10.104.248:9000 | Total threads: 2 |
| Dashboard: http://10.10.104.248:34261/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-awcbwl1j | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.05 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.106.112:9000
| Comm: tcp://10.10.106.112:9000 | Total threads: 2 |
| Dashboard: http://10.10.106.112:35235/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-njqpzw4j | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.20 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.106.17:9000
| Comm: tcp://10.10.106.17:9000 | Total threads: 2 |
| Dashboard: http://10.10.106.17:45743/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-gd8l_71u | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.27 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.107.10:9000
| Comm: tcp://10.10.107.10:9000 | Total threads: 2 |
| Dashboard: http://10.10.107.10:35693/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-9af9s6cx | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 88.79 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.107.152:9000
| Comm: tcp://10.10.107.152:9000 | Total threads: 2 |
| Dashboard: http://10.10.107.152:35279/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-y1e3dwrc | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.10 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.110.213:9000
| Comm: tcp://10.10.110.213:9000 | Total threads: 2 |
| Dashboard: http://10.10.110.213:41041/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-wru_f58_ | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.12 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.110.246:9000
| Comm: tcp://10.10.110.246:9000 | Total threads: 2 |
| Dashboard: http://10.10.110.246:45957/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-y81tv2hw | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.38 MiB | Spilled bytes: 0 B |
| Read bytes: 0.91 kiB | Write bytes: 4.89 kiB |
Worker: tcp://10.10.111.128:9000
| Comm: tcp://10.10.111.128:9000 | Total threads: 2 |
| Dashboard: http://10.10.111.128:45299/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-rzpr8_yw | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.11 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.111.39:9000
| Comm: tcp://10.10.111.39:9000 | Total threads: 2 |
| Dashboard: http://10.10.111.39:41305/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-2kp655dg | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 88.77 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.111.57:9000
| Comm: tcp://10.10.111.57:9000 | Total threads: 2 |
| Dashboard: http://10.10.111.57:46833/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-y06rdndo | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.63 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.112.187:9000
| Comm: tcp://10.10.112.187:9000 | Total threads: 2 |
| Dashboard: http://10.10.112.187:36585/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-p5xjsurh | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.12 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.114.179:9000
| Comm: tcp://10.10.114.179:9000 | Total threads: 2 |
| Dashboard: http://10.10.114.179:39143/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ff4z50s7 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.28 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.117.122:9000
| Comm: tcp://10.10.117.122:9000 | Total threads: 2 |
| Dashboard: http://10.10.117.122:38535/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-kmput86h | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.29 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.117.175:9000
| Comm: tcp://10.10.117.175:9000 | Total threads: 2 |
| Dashboard: http://10.10.117.175:38695/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-0jtcevap | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.92 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.119.178:9000
| Comm: tcp://10.10.119.178:9000 | Total threads: 2 |
| Dashboard: http://10.10.119.178:45445/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-anq6j_gz | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.53 MiB | Spilled bytes: 0 B |
| Read bytes: 861.9187219581622 B | Write bytes: 4.79 kiB |
Worker: tcp://10.10.119.225:9000
| Comm: tcp://10.10.119.225:9000 | Total threads: 2 |
| Dashboard: http://10.10.119.225:35235/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-un4j_mvx | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.36 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.119.93:9000
| Comm: tcp://10.10.119.93:9000 | Total threads: 2 |
| Dashboard: http://10.10.119.93:45367/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ktfsl4l7 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.07 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.121.252:9000
| Comm: tcp://10.10.121.252:9000 | Total threads: 2 |
| Dashboard: http://10.10.121.252:45735/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-pj8k20ec | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.79 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.122.155:9000
| Comm: tcp://10.10.122.155:9000 | Total threads: 2 |
| Dashboard: http://10.10.122.155:46161/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-bw0j9f_f | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.47 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.123.156:9000
| Comm: tcp://10.10.123.156:9000 | Total threads: 2 |
| Dashboard: http://10.10.123.156:46049/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-f7e9jmib | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.02 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.123.172:9000
| Comm: tcp://10.10.123.172:9000 | Total threads: 2 |
| Dashboard: http://10.10.123.172:41387/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-c787a_a3 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.10 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.124.153:9000
| Comm: tcp://10.10.124.153:9000 | Total threads: 2 |
| Dashboard: http://10.10.124.153:34823/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-xq9wm_43 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.98 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.124.158:9000
| Comm: tcp://10.10.124.158:9000 | Total threads: 2 |
| Dashboard: http://10.10.124.158:45455/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-62x0law7 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.03 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.125.201:9000
| Comm: tcp://10.10.125.201:9000 | Total threads: 2 |
| Dashboard: http://10.10.125.201:34381/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-so6gwrcj | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.05 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.126.210:9000
| Comm: tcp://10.10.126.210:9000 | Total threads: 2 |
| Dashboard: http://10.10.126.210:40881/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-wvyj1y5b | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 88.76 MiB | Spilled bytes: 0 B |
| Read bytes: 863.8274118015552 B | Write bytes: 4.80 kiB |
Worker: tcp://10.10.127.63:9000
| Comm: tcp://10.10.127.63:9000 | Total threads: 2 |
| Dashboard: http://10.10.127.63:35229/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-v0iok51s | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.43 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.65.171:9000
| Comm: tcp://10.10.65.171:9000 | Total threads: 2 |
| Dashboard: http://10.10.65.171:34259/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-rtuo7aw3 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.33 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.66.89:9000
| Comm: tcp://10.10.66.89:9000 | Total threads: 2 |
| Dashboard: http://10.10.66.89:39881/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-wf_i1ixx | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.69 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.67.160:9000
| Comm: tcp://10.10.67.160:9000 | Total threads: 2 |
| Dashboard: http://10.10.67.160:34121/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-0t1kdy4y | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.32 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.67.192:9000
| Comm: tcp://10.10.67.192:9000 | Total threads: 2 |
| Dashboard: http://10.10.67.192:38447/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-9xpnewdo | |
| Tasks executing: 2 | Tasks in memory: 41 |
| Tasks ready: 2312 | Tasks in flight: 0 |
| CPU usage: 8.0% | Last seen: Just now |
| Memory usage: 2.82 GiB | Spilled bytes: 0 B |
| Read bytes: 14.65 MiB | Write bytes: 42.11 kiB |
Worker: tcp://10.10.67.199:9000
| Comm: tcp://10.10.67.199:9000 | Total threads: 2 |
| Dashboard: http://10.10.67.199:34453/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-x40y1_pu | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.14 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.68.149:9000
| Comm: tcp://10.10.68.149:9000 | Total threads: 2 |
| Dashboard: http://10.10.68.149:33695/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ugugpfzf | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.04 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.68.57:9000
| Comm: tcp://10.10.68.57:9000 | Total threads: 2 |
| Dashboard: http://10.10.68.57:42267/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-nfdq2lj2 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.21 MiB | Spilled bytes: 0 B |
| Read bytes: 863.3618913023383 B | Write bytes: 4.79 kiB |
Worker: tcp://10.10.69.153:9000
| Comm: tcp://10.10.69.153:9000 | Total threads: 2 |
| Dashboard: http://10.10.69.153:35835/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-zm2xuke8 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.38 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.69.252:9000
| Comm: tcp://10.10.69.252:9000 | Total threads: 2 |
| Dashboard: http://10.10.69.252:33557/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-e68g78xr | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 88.64 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.69.60:9000
| Comm: tcp://10.10.69.60:9000 | Total threads: 2 |
| Dashboard: http://10.10.69.60:41145/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-2drakph2 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.21 MiB | Spilled bytes: 0 B |
| Read bytes: 0.92 kiB | Write bytes: 4.80 kiB |
Worker: tcp://10.10.71.203:9000
| Comm: tcp://10.10.71.203:9000 | Total threads: 2 |
| Dashboard: http://10.10.71.203:44529/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-s3f0r10s | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.97 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.73.76:9000
| Comm: tcp://10.10.73.76:9000 | Total threads: 2 |
| Dashboard: http://10.10.73.76:35151/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-wjvgu8xa | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.40 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.74.121:9000
| Comm: tcp://10.10.74.121:9000 | Total threads: 2 |
| Dashboard: http://10.10.74.121:43243/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ujpiyf3o | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.20 MiB | Spilled bytes: 0 B |
| Read bytes: 140.3187554124761 B | Write bytes: 0.0 B |
Worker: tcp://10.10.75.206:9000
| Comm: tcp://10.10.75.206:9000 | Total threads: 2 |
| Dashboard: http://10.10.75.206:34537/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-vnacacxr | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 88.62 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.77.140:9000
| Comm: tcp://10.10.77.140:9000 | Total threads: 2 |
| Dashboard: http://10.10.77.140:46169/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ny9f1wsq | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.08 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.77.246:9000
| Comm: tcp://10.10.77.246:9000 | Total threads: 2 |
| Dashboard: http://10.10.77.246:42161/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-tleavps2 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.08 MiB | Spilled bytes: 0 B |
| Read bytes: 208.1894118298394 B | Write bytes: 264.24040732248847 B |
Worker: tcp://10.10.78.55:9000
| Comm: tcp://10.10.78.55:9000 | Total threads: 2 |
| Dashboard: http://10.10.78.55:42191/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-27njkd3x | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.32 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.78.83:9000
| Comm: tcp://10.10.78.83:9000 | Total threads: 2 |
| Dashboard: http://10.10.78.83:42625/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-peb17kcn | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.93 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.79.182:9000
| Comm: tcp://10.10.79.182:9000 | Total threads: 2 |
| Dashboard: http://10.10.79.182:46575/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ojnhisj9 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.47 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.80.237:9000
| Comm: tcp://10.10.80.237:9000 | Total threads: 2 |
| Dashboard: http://10.10.80.237:39757/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-d7se3vdw | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.43 MiB | Spilled bytes: 0 B |
| Read bytes: 208.4732805675421 B | Write bytes: 264.60070225880344 B |
Worker: tcp://10.10.81.68:9000
| Comm: tcp://10.10.81.68:9000 | Total threads: 2 |
| Dashboard: http://10.10.81.68:41673/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-0q8xbngp | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.51 MiB | Spilled bytes: 0 B |
| Read bytes: 208.0332313215698 B | Write bytes: 264.0421782158386 B |
Worker: tcp://10.10.82.13:9000
| Comm: tcp://10.10.82.13:9000 | Total threads: 2 |
| Dashboard: http://10.10.82.13:42267/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-yiqgjo88 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.55 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.83.237:9000
| Comm: tcp://10.10.83.237:9000 | Total threads: 2 |
| Dashboard: http://10.10.83.237:39047/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-yq7d61m6 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.60 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.84.167:9000
| Comm: tcp://10.10.84.167:9000 | Total threads: 2 |
| Dashboard: http://10.10.84.167:40809/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-4r3svhqd | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.38 MiB | Spilled bytes: 0 B |
| Read bytes: 347.42944192981423 B | Write bytes: 263.5671628433073 B |
Worker: tcp://10.10.84.47:9000
| Comm: tcp://10.10.84.47:9000 | Total threads: 2 |
| Dashboard: http://10.10.84.47:46479/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-7a8kcvgf | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.73 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.84.67:9000
| Comm: tcp://10.10.84.67:9000 | Total threads: 2 |
| Dashboard: http://10.10.84.67:33851/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-yke72gi_ | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.12 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.84.68:9000
| Comm: tcp://10.10.84.68:9000 | Total threads: 2 |
| Dashboard: http://10.10.84.68:38665/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-em__q4cd | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.19 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.85.4:9000
| Comm: tcp://10.10.85.4:9000 | Total threads: 2 |
| Dashboard: http://10.10.85.4:33859/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-1k21fkr2 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.09 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.87.172:9000
| Comm: tcp://10.10.87.172:9000 | Total threads: 2 |
| Dashboard: http://10.10.87.172:36205/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-8sn2jmct | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.60 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.88.115:9000
| Comm: tcp://10.10.88.115:9000 | Total threads: 2 |
| Dashboard: http://10.10.88.115:35989/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-p2pteiz7 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 88.93 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.88.154:9000
| Comm: tcp://10.10.88.154:9000 | Total threads: 2 |
| Dashboard: http://10.10.88.154:37489/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-gwj8bsic | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.03 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.89.219:9000
| Comm: tcp://10.10.89.219:9000 | Total threads: 2 |
| Dashboard: http://10.10.89.219:35839/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-rhitnh6q | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.22 MiB | Spilled bytes: 0 B |
| Read bytes: 863.1517278141856 B | Write bytes: 4.79 kiB |
Worker: tcp://10.10.89.43:9000
| Comm: tcp://10.10.89.43:9000 | Total threads: 2 |
| Dashboard: http://10.10.89.43:43309/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-tl0d8kns | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.41 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.93.99:9000
| Comm: tcp://10.10.93.99:9000 | Total threads: 2 |
| Dashboard: http://10.10.93.99:43551/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-5_26igfv | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 88.78 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.93.9:9000
| Comm: tcp://10.10.93.9:9000 | Total threads: 2 |
| Dashboard: http://10.10.93.9:33991/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-7hktyk6z | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.18 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.94.168:9000
| Comm: tcp://10.10.94.168:9000 | Total threads: 2 |
| Dashboard: http://10.10.94.168:41449/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-4pz9_hy1 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 4.0% | Last seen: Just now |
| Memory usage: 88.74 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.94.17:9000
| Comm: tcp://10.10.94.17:9000 | Total threads: 2 |
| Dashboard: http://10.10.94.17:44551/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-cf_pt58n | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.22 MiB | Spilled bytes: 0 B |
| Read bytes: 208.1611966053424 B | Write bytes: 264.20459569139615 B |
Worker: tcp://10.10.94.209:9000
| Comm: tcp://10.10.94.209:9000 | Total threads: 2 |
| Dashboard: http://10.10.94.209:37027/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-big84fu6 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.17 MiB | Spilled bytes: 0 B |
| Read bytes: 862.0454320766809 B | Write bytes: 4.78 kiB |
Worker: tcp://10.10.94.95:9000
| Comm: tcp://10.10.94.95:9000 | Total threads: 2 |
| Dashboard: http://10.10.94.95:45687/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-ag3dsyve | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.32 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.96.64:9000
| Comm: tcp://10.10.96.64:9000 | Total threads: 2 |
| Dashboard: http://10.10.96.64:33481/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-w_8uyfq8 | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 0.0% | Last seen: Just now |
| Memory usage: 89.13 MiB | Spilled bytes: 0 B |
| Read bytes: 0.91 kiB | Write bytes: 4.88 kiB |
Worker: tcp://10.10.98.200:9000
| Comm: tcp://10.10.98.200:9000 | Total threads: 2 |
| Dashboard: http://10.10.98.200:35895/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-zq2b4vaw | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.71 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
Worker: tcp://10.10.99.8:9000
| Comm: tcp://10.10.99.8:9000 | Total threads: 2 |
| Dashboard: http://10.10.99.8:34003/status | Memory: 6.52 GiB |
| Nanny: None | |
| Local directory: /dask-worker-space/worker-2zqnpwme | |
| Tasks executing: 0 | Tasks in memory: 0 |
| Tasks ready: 0 | Tasks in flight: 0 |
| CPU usage: 2.0% | Last seen: Just now |
| Memory usage: 89.18 MiB | Spilled bytes: 0 B |
| Read bytes: 0.0 B | Write bytes: 0.0 B |
In [15]:
# Make garbage collection explicit to prevent deadlocks
import gc
from distributed import WorkerPlugin
class WorkerExplicitGC(WorkerPlugin):
def setup(self, worker):
self.worker = worker
gc.disable()
def transition(self, key, start, finish, *args, **kwargs):
if finish == 'executing':
self.worker._throttled_gc.collect()
if gc.isenabled():
gc.disable()
plugin = WorkerExplicitGC()
client.register_worker_plugin(plugin)
Out[15]:
{'tcp://10.10.102.223:9000': {'status': 'OK'},
'tcp://10.10.102.37:9000': {'status': 'OK'},
'tcp://10.10.104.248:9000': {'status': 'OK'},
'tcp://10.10.106.112:9000': {'status': 'OK'},
'tcp://10.10.106.17:9000': {'status': 'OK'},
'tcp://10.10.107.10:9000': {'status': 'OK'},
'tcp://10.10.107.152:9000': {'status': 'OK'},
'tcp://10.10.110.213:9000': {'status': 'OK'},
'tcp://10.10.110.246:9000': {'status': 'OK'},
'tcp://10.10.111.128:9000': {'status': 'OK'},
'tcp://10.10.111.39:9000': {'status': 'OK'},
'tcp://10.10.111.57:9000': {'status': 'OK'},
'tcp://10.10.112.187:9000': {'status': 'OK'},
'tcp://10.10.114.179:9000': {'status': 'OK'},
'tcp://10.10.117.122:9000': {'status': 'OK'},
'tcp://10.10.117.175:9000': {'status': 'OK'},
'tcp://10.10.119.178:9000': {'status': 'OK'},
'tcp://10.10.119.225:9000': {'status': 'OK'},
'tcp://10.10.119.93:9000': {'status': 'OK'},
'tcp://10.10.121.252:9000': {'status': 'OK'},
'tcp://10.10.122.155:9000': {'status': 'OK'},
'tcp://10.10.123.156:9000': {'status': 'OK'},
'tcp://10.10.123.172:9000': {'status': 'OK'},
'tcp://10.10.124.153:9000': {'status': 'OK'},
'tcp://10.10.124.158:9000': {'status': 'OK'},
'tcp://10.10.125.201:9000': {'status': 'OK'},
'tcp://10.10.126.210:9000': {'status': 'OK'},
'tcp://10.10.127.63:9000': {'status': 'OK'},
'tcp://10.10.65.171:9000': {'status': 'OK'},
'tcp://10.10.66.89:9000': {'status': 'OK'},
'tcp://10.10.67.160:9000': {'status': 'OK'},
'tcp://10.10.67.192:9000': {'status': 'OK'},
'tcp://10.10.67.199:9000': {'status': 'OK'},
'tcp://10.10.68.149:9000': {'status': 'OK'},
'tcp://10.10.68.57:9000': {'status': 'OK'},
'tcp://10.10.69.153:9000': {'status': 'OK'},
'tcp://10.10.69.252:9000': {'status': 'OK'},
'tcp://10.10.69.60:9000': {'status': 'OK'},
'tcp://10.10.71.203:9000': {'status': 'OK'},
'tcp://10.10.73.76:9000': {'status': 'OK'},
'tcp://10.10.74.121:9000': {'status': 'OK'},
'tcp://10.10.75.206:9000': {'status': 'OK'},
'tcp://10.10.77.140:9000': {'status': 'OK'},
'tcp://10.10.77.246:9000': {'status': 'OK'},
'tcp://10.10.78.55:9000': {'status': 'OK'},
'tcp://10.10.78.83:9000': {'status': 'OK'},
'tcp://10.10.79.182:9000': {'status': 'OK'},
'tcp://10.10.80.237:9000': {'status': 'OK'},
'tcp://10.10.81.68:9000': {'status': 'OK'},
'tcp://10.10.82.13:9000': {'status': 'OK'},
'tcp://10.10.83.237:9000': {'status': 'OK'},
'tcp://10.10.84.167:9000': {'status': 'OK'},
'tcp://10.10.84.47:9000': {'status': 'OK'},
'tcp://10.10.84.67:9000': {'status': 'OK'},
'tcp://10.10.84.68:9000': {'status': 'OK'},
'tcp://10.10.85.4:9000': {'status': 'OK'},
'tcp://10.10.87.172:9000': {'status': 'OK'},
'tcp://10.10.88.115:9000': {'status': 'OK'},
'tcp://10.10.88.154:9000': {'status': 'OK'},
'tcp://10.10.89.219:9000': {'status': 'OK'},
'tcp://10.10.89.43:9000': {'status': 'OK'},
'tcp://10.10.93.99:9000': {'status': 'OK'},
'tcp://10.10.93.9:9000': {'status': 'OK'},
'tcp://10.10.94.168:9000': {'status': 'OK'},
'tcp://10.10.94.17:9000': {'status': 'OK'},
'tcp://10.10.94.209:9000': {'status': 'OK'},
'tcp://10.10.94.95:9000': {'status': 'OK'},
'tcp://10.10.96.64:9000': {'status': 'OK'},
'tcp://10.10.98.200:9000': {'status': 'OK'},
'tcp://10.10.99.8:9000': {'status': 'OK'}}
Open an Example File and Check the Native Chunking¶
We want to chunk in a similar way for maximum performance
In [16]:
url = 's3://era5-pds/2010/01/data/air_temperature_at_2_metres.nc'
ncfile = fsspec.open(url)
ds = xr.open_dataset(ncfile.open())
ds.air_temperature_at_2_metres.encoding
Out[16]:
{'chunksizes': (24, 100, 100),
'fletcher32': False,
'shuffle': True,
'zlib': True,
'complevel': 4,
'source': '<File-like object S3FileSystem, era5-pds/2010/01/data/air_temperature_at_2_metres.nc>',
'original_shape': (744, 721, 1440),
'dtype': dtype('<f4'),
'_FillValue': array([9.96921e+36], dtype=float32)}
Open 2-m air temperature as a single dataset¶
In [17]:
start_year = 1979
end_year = 2020
years = list(np.arange(start_year, end_year+1, 1))
months = ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"]
file_pattern = 's3://era5-pds/{year}/{month}/data/air_temperature_at_2_metres.nc'
@dask.delayed
def s3open(path):
fs = s3fs.S3FileSystem(anon=True, default_fill_cache=False)
return fs.open(path)
files_mapper = [s3open(file_pattern.format(year=year,month=month)) for year in years for month in months]
In [18]:
%%time
ds = xr.open_mfdataset(files_mapper, engine='h5netcdf', chunks={'lon':200,'lat':200,'time0':720}, concat_dim='time0', combine='nested', coords='minimal', compat='override', parallel=True)
CPU times: user 6.89 s, sys: 8.61 s, total: 15.5 s Wall time: 37.9 s
In [19]:
print('ds size in GB {:0.2f}\n'.format(ds.nbytes / 1e9))
ds.info
ds size in GB 1529.06
Out[19]:
<bound method Dataset.info of <xarray.Dataset>
Dimensions: (lon: 1440, lat: 721, time0: 368184)
Coordinates:
* lon (lon) float32 0.0 0.25 0.5 ... 359.5 359.8
* lat (lat) float32 90.0 89.75 89.5 ... -89.75 -90.0
* time0 (time0) datetime64[ns] 1979-01-01 ... 2020-1...
Data variables:
air_temperature_at_2_metres (time0, lat, lon) float32 dask.array<chunksize=(720, 200, 200), meta=np.ndarray>
Attributes:
source: Reanalysis
institution: ECMWF
title: ERA5 forecasts>
The ds.info output above shows us that there are four dimensions to the data: lat, lon, and time0; and two data variables: air_temperature_at_2_metres, and air_pressure_at_mean_sea_level.
Convert units to F from K¶
In [20]:
ds['air_temperature_at_2_metres'] = (ds.air_temperature_at_2_metres - 273.15) * 9.0 / 5.0 + 32.0
ds.air_temperature_at_2_metres.attrs['units'] = 'F'
Calculate the mean 2-m air temperature for all times¶
In [21]:
# calculates the mean along the time dimension
temp_mean = ds['air_temperature_at_2_metres'].mean(dim='time0')
The expressions above didn’t actually compute anything. They just build the dask task graph. To do the computations, we call the persist method:
In [22]:
temp_mean = temp_mean.persist()
progress(temp_mean)
Plot Average Surface Temperature¶
In [23]:
temp_mean.compute()
temp_mean.plot(figsize=(20, 10))
plt.title('1979-2020 Mean 2-m Air Temperature')
Out[23]:
Text(0.5, 1.0, '1979-2020 Mean 2-m Air Temperature')
Repeat for standard deviation¶
In [24]:
temp_std = ds['air_temperature_at_2_metres'].std(dim='time0')
In [25]:
temp_std = temp_std.persist()
progress(temp_std)
In [15]:
temp_std.compute()
temp_std.plot(figsize=(20, 10))
plt.title('1979-2020 Standard Deviation 2-m Air Temperature')
Out[15]:
Text(0.5, 1.0, '1979-2020 Standard Deviation 2-m Air Temperature')
Plot temperature time series for points¶
In [26]:
# location coordinates
locs = [
{'name': 'Santa Barbara', 'lon': -119.70, 'lat': 34.42},
{'name': 'Colorado Springs', 'lon': -104.82, 'lat': 38.83},
{'name': 'Honolulu', 'lon': -157.84, 'lat': 21.29},
{'name': 'Seattle', 'lon': -122.33, 'lat': 47.61},
]
# convert westward longitudes to degrees east
for l in locs:
if l['lon'] < 0:
l['lon'] = 360 + l['lon']
locs
Out[26]:
[{'name': 'Santa Barbara', 'lon': 240.3, 'lat': 34.42},
{'name': 'Colorado Springs', 'lon': 255.18, 'lat': 38.83},
{'name': 'Honolulu', 'lon': 202.16, 'lat': 21.29},
{'name': 'Seattle', 'lon': 237.67000000000002, 'lat': 47.61}]
In [27]:
ds_locs = xr.Dataset()
air_temp_ds = ds
# interate through the locations and create a dataset
# containing the temperature values for each location
for l in locs:
name = l['name']
lon = l['lon']
lat = l['lat']
var_name = name
ds2 = air_temp_ds.sel(lon=lon, lat=lat, method='nearest')
lon_attr = '%s_lon' % name
lat_attr = '%s_lat' % name
ds2.attrs[lon_attr] = ds2.lon.values.tolist()
ds2.attrs[lat_attr] = ds2.lat.values.tolist()
ds2 = ds2.rename({'air_temperature_at_2_metres' : var_name}).drop(('lat', 'lon'))
ds_locs = xr.merge([ds_locs, ds2])
ds_locs.data_vars
Out[27]:
Data variables:
Santa Barbara (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
Colorado Springs (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
Honolulu (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
Seattle (time0) float32 dask.array<chunksize=(720,), meta=np.ndarray>
Convert to dataframe¶
In [28]:
df_f = ds_locs.to_dataframe()
df_f.describe()
Out[28]:
| Santa Barbara | Colorado Springs | Honolulu | Seattle | |
|---|---|---|---|---|
| count | 368184.000000 | 368184.000000 | 368184.000000 | 368184.000000 |
| mean | 60.281300 | 46.359184 | 75.154930 | 52.309616 |
| std | 9.913921 | 19.551779 | 2.568616 | 10.984550 |
| min | 25.767511 | -25.982491 | 61.992512 | 5.742512 |
| 25% | 52.655010 | 31.167511 | 73.355011 | 44.330009 |
| 50% | 60.080009 | 46.355011 | 75.380013 | 51.417511 |
| 75% | 67.505013 | 60.755013 | 77.067513 | 59.742512 |
| max | 97.092514 | 98.330009 | 83.817513 | 98.330009 |
Plot temperature timeseries¶
In [29]:
ax = df_f.plot(figsize=(20, 10), title="ERA5", grid=1)
ax.set(xlabel='Date', ylabel='2-m Air Temperature (deg F)')
plt.show()