Most IOOS data are available for access online via user applications and data services for machine-to-machine access. ERDDAP (see here and here) is one of those applicattions. "ERDDAP is a data server that gives you a simple, consistent way to download data in the format and the spatial and temporal coverage that you want. ERDDAP is a web application with an interface for people to use. It is also a RESTful web service that allows data access directly from any computer program (e.g. Matlab, R, or webpages)."
ERDDAP has RESTful API that is very convenient for creating web apps, data portals, etc. However, writing those URLs manually can be tedious and error prone. This notebook walks through an easy to set up ERDDAP RESTful URL by using the python client, erddapy, https://pyoceans.github.io/erddapy/
A typical ERDDAP RESTful URL looks like:
Let's break it down to smaller parts:
Note: This Jupyter notebook originated from an ERDDAPY notebook from the IOOS gallery
import pandas as pd
from erddapy import ERDDAP
from erddapy.utilities import urlopen
import hvplot.xarray
import holoviews as hv
# New Axiom ERDDAP for OOI
server = 'http://erddap.dataexplorer.oceanobservatories.org/erddap'
protocol = 'tabledap'
e = ERDDAP(server=server, protocol=protocol)
A search for everything looks like this. The only effective filtering parameters being passed are protocol=tabledap and response=csv.
df = pd.read_csv(urlopen(e.get_search_url(response='csv', search_for='all')))
len(df)
540
Now we'll refine our search by adding temporal, bounding box and variable constraints.
min_time = '2018-07-01T00:00:00Z'
max_time = '2018-07-15T00:00:00Z'
min_lon, max_lon = -127, -123.75
min_lat, max_lat = 43, 48
standard_name = 'sea_water_practical_salinity'
kw = {
'standard_name': standard_name,
'min_lon': min_lon,'max_lon': max_lon,'min_lat': min_lat,'max_lat': max_lat,
'min_time': min_time,'max_time': max_time,
}
search_url = e.get_search_url(response='csv', **kw)
search_df = pd.read_csv(urlopen(search_url))
search_df = search_df[['Institution', 'Dataset ID','tabledap']]
search_df
| Institution | Dataset ID | tabledap | |
|---|---|---|---|
| 0 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-02-flortd000 | http://erddap.dataexplorer.oceanobservatories.... |
| 1 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 2 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-03-dostad000 | http://erddap.dataexplorer.oceanobservatories.... |
| 3 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-07-nutnrb000 | http://erddap.dataexplorer.oceanobservatories.... |
| 4 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-06-phsend000 | http://erddap.dataexplorer.oceanobservatories.... |
| ... | ... | ... | ... |
| 84 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-pc01a-4b-phsena102 | http://erddap.dataexplorer.oceanobservatories.... |
| 85 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-sf01a-3a-flortd101 | http://erddap.dataexplorer.oceanobservatories.... |
| 86 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-sf01a-2a-ctdpfa102 | http://erddap.dataexplorer.oceanobservatories.... |
| 87 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-sf01a-4a-nutnra101 | http://erddap.dataexplorer.oceanobservatories.... |
| 88 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-sf01a-2d-phsena101 | http://erddap.dataexplorer.oceanobservatories.... |
89 rows × 3 columns
Let's inspect a specific dataset_id.
dataset_id = 'ooi-ce06issm-sbd17-06-ctdbpc000'
Construct the ERDDAP URL to get the data
e.dataset_id = dataset_id
e.constraints = {'time>=': min_time,'time<=': max_time}
e.response='csv'
e.variables = [ 'time', e.get_var_by_attr(dataset_id=dataset_id, standard_name=standard_name)[0]]
print(e.get_download_url())
http://erddap.dataexplorer.oceanobservatories.org/erddap/tabledap/ooi-ce06issm-sbd17-06-ctdbpc000.csv?time,sea_water_practical_salinity&time>=1530403200.0&time<=1531612800.0
Read the data into Xarray
ds = e.to_xarray(decode_times=True)
#ds = ds.swap_dims({'row':'time'})
#[ds[var].plot() for var in ds.data_vars];
ds
<xarray.Dataset>
Dimensions: (obs: 673, timeseries: 1)
Coordinates:
longitude (timeseries) float64 -124.3
latitude (timeseries) float64 47.13
time (obs) datetime64[ns] 2018-07-01 ... 2018-07-15
Dimensions without coordinates: obs, timeseries
Data variables:
station (timeseries) object ''
rowSize (timeseries) int32 673
sea_water_practical_salinity (obs) float64 32.03 32.01 ... 32.58 32.56
Attributes:
cdm_data_type: TimeSeries
cdm_timeseries_variables: station,longitude,latitude
contributor_email: feedback@axiomdatascience.com
contributor_name: Axiom Data Science
contributor_role: processor
contributor_role_vocabulary: NERC
contributor_url: https://www.axiomdatascience.com
Conventions: IOOS-1.2, CF-1.6, ACDD-1.3
creator_institution: Ocean Observatories Initiative (OOI)
creator_name: Ocean Observatories Initiative (OOI)
creator_sector: gov_federal
creator_type: institution
creator_url: https://oceanobservatories.org/
Easternmost_Easting: -124.27087
featureType: TimeSeries
geospatial_lat_max: 47.13445
geospatial_lat_min: 47.13445
geospatial_lat_units: degrees_north
geospatial_lon_max: -124.27087
geospatial_lon_min: -124.27087
geospatial_lon_units: degrees_east
geospatial_vertical_positive: up
geospatial_vertical_units: m
history: Downloaded from Ocean Observatories Initia...
id: 103806
infoUrl: https://sensors.ioos.us/#metadata/103806/s...
institution: Ocean Observatories Initiative (OOI)
license: The data may be used and redistributed for...
naming_authority: com.axiomdatascience
Northernmost_Northing: 47.13445
platform: buoy
platform_name: Coastal Endurance: Washington Inshore Surf...
platform_vocabulary: http://mmisw.org/ont/ioos/platform
processing_level: Level 2
publisher_institution: Ocean Observatories Initiative (OOI)
publisher_name: Ocean Observatories Initiative (OOI)
publisher_sector: gov_federal
publisher_type: institution
publisher_url: https://oceanobservatories.org/
references: https://ooinet.oceanobservatories.org/data...
sourceUrl: https://ooinet.oceanobservatories.org/data...
Southernmost_Northing: 47.13445
standard_name_vocabulary: CF Standard Name Table v72
station_id: 103806.0
summary: Timeseries data from 'Coastal Endurance: W...
time_coverage_end: 2018-07-15T00:00:00Z
time_coverage_start: 2018-07-01T00:00:00Z
title: Coastal Endurance: Washington Inshore Surf...
Westernmost_Easting: -124.27087array([-124.27087])
array([47.13445])
array(['2018-07-01T00:00:00.000000000', '2018-07-01T00:30:00.000000000',
'2018-07-01T01:00:00.000000000', ..., '2018-07-14T23:00:00.000000000',
'2018-07-14T23:30:00.000000000', '2018-07-15T00:00:00.000000000'],
dtype='datetime64[ns]')array([''], dtype=object)
array([673], dtype=int32)
array([32.027097, 32.013399, 31.989715, ..., 32.550299, 32.579715, 32.555465])
Let's narrow this down by only taking the "CTDBP" data
ctdbp = search_df[search_df['Dataset ID'].str.contains("ctdbp")].reset_index()
ctdbp
| index | Institution | Dataset ID | tabledap | |
|---|---|---|---|---|
| 0 | 1 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 1 | 5 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-mfd37-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 2 | 9 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-sbd17-06-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 3 | 14 | Ocean Observatories Initiative (OOI) | ooi-ce04osbp-lj01c-06-ctdbpo108 | http://erddap.dataexplorer.oceanobservatories.... |
| 4 | 26 | Ocean Observatories Initiative (OOI) | ooi-ce04ossm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 5 | 32 | Ocean Observatories Initiative (OOI) | ooi-ce02shbp-lj01d-06-ctdbpn106 | http://erddap.dataexplorer.oceanobservatories.... |
| 6 | 35 | Ocean Observatories Initiative (OOI) | ooi-ce02shsm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 7 | 46 | Ocean Observatories Initiative (OOI) | ooi-ce06issm-rid16-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 8 | 50 | Ocean Observatories Initiative (OOI) | ooi-ce06issm-mfd37-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 9 | 54 | Ocean Observatories Initiative (OOI) | ooi-ce06issm-sbd17-06-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 10 | 59 | Ocean Observatories Initiative (OOI) | ooi-ce09ossm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 11 | 63 | Ocean Observatories Initiative (OOI) | ooi-ce09ossm-mfd37-03-ctdbpe000 | http://erddap.dataexplorer.oceanobservatories.... |
| 12 | 68 | Ocean Observatories Initiative (OOI) | ooi-ce07shsm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 13 | 72 | Ocean Observatories Initiative (OOI) | ooi-ce07shsm-mfd37-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
Let's just take five datasets to speed up demo
ndatasets = 5
df_list = []
hv_list = []
for dataset_id in ctdbp['Dataset ID'].values:
e.dataset_id = dataset_id
e.variables = [
'time',
e.get_var_by_attr(dataset_id=dataset_id, standard_name=standard_name)[0]
]
try:
ds = e.to_xarray(decode_times=True)
# ds = ds.swap_dims({'row':'time'})
df_list.append(ds)
print(dataset_id)
#[ds[var].plot() for var in ds.data_vars];
hv_list.append(ds[e.variables[-1]].hvplot(label=dataset_id))
except:
pass
if len(df_list)==ndatasets: break
ooi-ce01issm-rid16-03-ctdbpc000 ooi-ce01issm-mfd37-03-ctdbpc000 ooi-ce01issm-sbd17-06-ctdbpc000 ooi-ce04osbp-lj01c-06-ctdbpo108 ooi-ce04ossm-rid27-03-ctdbpc000
hv.Overlay(hv_list)