Demo with esgf search for CMIP5 at DKRZ site (subset only)¶

ESGF Node at DKRZ: https://esgf-data.dkrz.de/search/cmip5-dkrz/

Use esgf search at DKRZ ... no distributed search¶

Using esgf-pyclient: https://esgf-pyclient.readthedocs.io/en/latest/notebooks/examples/search.html

In [ ]:

from pyesgf.search import SearchConnection
conn = SearchConnection('http://esgf-data.dkrz.de/esg-search',
                        distrib=False)

Search only CMIP5 files locally available at DKRZ

In [ ]:

ctx = conn.new_context(project='CMIP5', data_node='esgf1.dkrz.de,esgf3.dkrz.de', latest=True, replica=False)
ctx.hit_count

Select only one dataset ... use variable later for filtering.

In [ ]:

variable = 'tas'

results = ctx.search(
    institute='MPI-M',
    model='MPI-ESM-MR',
    experiment='historical',
    realm='atmos',
    variable=variable, 
    time_frequency='day',
    ensemble='r1i1p1'
)
len(results)

In [ ]:

ds = results[0]
ds.json

Get a dataset identifier used by rook

In [ ]:

dataset_id = ds.json['instance_id']
dataset_id

Time range

In [ ]:

f"{ds.json['datetime_start']}/{ds.json['datetime_stop']})"

Bounding Box: (West, Sout, East, North)

In [ ]:

f"({ds.json['west_degrees']}, {ds.json['south_degrees']},{ds.json['east_degrees']}, {ds.json['west_degrees']}, {ds.json['north_degrees']})"

Size in GB

In [ ]:

f"{ds.json['size'] / 1024 / 1024 / 1024} GB"

Use Rook to run subset¶

In [ ]:

import os
os.environ['ROOK_URL'] = 'http://rook.dkrz.de/wps'
os.environ['ROOK_MODE'] = 'async'

from rooki import operators as ops

Run subset workflow

http://bboxfinder.com/

In [ ]:

bbox_africa = "-23.906250,-35.746512,63.632813,37.996163"

wf = ops.Subset(
        ops.Input(
            f'{variable}', [f"{dataset_id}.{variable}"]
        ),
        time="1850-01-01/1850-12-31",
        area=bbox_africa,
    
)
resp = wf.orchestrate()
resp.ok

The outputs are available as a Metalink document¶

https://github.com/metalink-dev

Metalink URL

In [ ]:

resp.url

Number of files

In [ ]:

resp.num_files

Total size in MB

In [ ]:

resp.size_in_mb

Download URLs

In [ ]:

resp.download_urls()

Download and open with xarray

In [ ]:

ds_0 = resp.datasets()[0]
ds_0

Provenance¶

Provenance information is given using the PROV standard. https://pypi.org/project/prov/

Provenance: URL to json document

In [ ]:

resp.provenance()

Provenance Plot

In [ ]:

from IPython.display import Image
Image(resp.provenance_image())