Discover, Customize and Access NSIDC DAAC Data¶

This notebook is based off of the NSIDC-Data-Access-Notebook provided through NSIDC's Github organization.

Now that we've visualized our study areas, we will first explore data coverage, size, and customization (subsetting, reformatting, reprojection) service availability, and then access those associated files.

___A note on data access options:___ We will be pursuing data discovery and access "programmatically" using Application Programming Interfaces, or APIs.

What is an API? You can think of an API as a middle man between an application or end-use (in this case, us) and a data provider. In this case the data provider is both the Common Metadata Repository (CMR) housing data information, and NSIDC as the data distributor. These APIs are generally structured as a URL with a base plus individual key-value-pairs separated by ‘&’.

There are other discovery and access methods available from NSIDC listed on the data set landing page (e.g. ATL07 Data Access) and NASA Earthdata Search. Programmatic API access is beneficial for those of you who want to incorporate data access into your visualization and analysis workflow. This method is also reproducible and documented to ensure data provenance.

Here are the steps you will learn in this customize and access notebook:

1. Search for data programmatically using the Common Metadata Repository API by time and area of interest.
2. Determine subsetting, reformatting, and reprojection capabilities for our data of interest.
3. Access and customize data using NSIDC's data access and service API.

Import packages¶

Explore data availability using the Common Metadata Repository¶

The Common Metadata Repository (CMR) is a high-performance, high-quality, continuously evolving metadata system that catalogs Earth Science data and associated service metadata records. These metadata records are registered, modified, discovered, and accessed through programmatic interfaces leveraging standard protocols and APIs. Note that not all NSIDC data can be searched at the file level using CMR, particularly those outside of the NASA DAAC program.

CMR API documentation: https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html

Select data sets and determine version number¶

Data sets are selected by data set IDs (e.g. ATL07, ATL10, and MOD29). In the CMR API documentation, a data set id is referred to as a "short name". These short names are located at the top of each NSIDC data set landing page in white underneath the full title, after 'DATA SET:'.

We are using the Python Requests package to access the CMR. Data are then converted to JSON format; a language independant human-readable open-standard file format. More than one version can exist for a given data set:

Select time and area of interest¶

We will add spatial and temporal filters to the data dictionary. The bounding box coordinates cover our region of interest over the East Siberian Sea and the temporal range covers March 23, 2019.

Determine how many files exist over this time and area of interest, as well as the average size and total volume of those granules¶

We will use the granule_info function to query the CMR granule API. The function prints the number of granules, average size, and total volume of those granules. It returns the granule number value.

Note that subsetting, reformatting, or reprojecting can alter the size of the granules if those services are applied to your request.

Determine the subsetting, reformatting, and reprojection services enabled for your data set of interest.¶

The NSIDC DAAC supports customization (subsetting, reformatting, reprojection) services on many of our NASA Earthdata mission collections. Let's discover whether or not our data set has these services available using the print_service_options function. If services are available, we will also determine the specific service options supported for this data set, which we will then add to our data dictionary.

Input Earthdata Login credentials¶

An Earthdata Login account is required to query data services and to access data from the NSIDC DAAC. If you do not already have an Earthdata Login account, visit http://urs.earthdata.nasa.gov to register. We are going to use the earthaccess library to authenticate with our Earthdata Login credentials. We recommend using a netrc file to store your Earthdata Login credentials for authentication. Below are instructions for creating a netrc file. If we don't want to use a netrc file, skip these instructions and instead we will be prompted to enter our credentials below.

Creating a netrc file¶

1. In your home directory create a netrc file using a text editor. On a Mac or Linux Operating System the file should be named ".netrc" and if on a Windows Operating System it should be named "_netrc".
2. In the file add the following content:

machine urs.earthdata.nasa.gov login <USERNAME> password <PASSWORD>

where <USERNAME> and <PASSWORD> are replaced by your Earthdata Login username and password.

We now need to create an HTTP session in order to store cookies and pass our credentials to the data service URLs. The capability URL below is what we will query to determine service information.

Populate data dictionary with services of interest¶

We already added our CMR search keywords to our data dictionary, so now we need to add the service options we want to request. A list of all available service keywords for use with NSIDC's access and service API are available in our Key-Value-Pair table, as a part of our Programmatic access guide. For our ATL10 and ATL07 request, we are interested in bounding box, temporal subsetting. For MOD29 we are interested in spatial subsetting. These options crop the data values to the specified ranges and variables of interest. We will enter those values into our data dictionary below.

Spatial subsetting: Output files are cropped to the specified bounding box extent.

Temporal subsetting: Output files are cropped to the specified temporal range extent.

For ATL07, we are also interested in variable subsetting.

Variable subsetting: Subsets the data set variable or group of variables. For hierarchical data, all lower level variables are returned if a variable group or subgroup is specified.

For ATL07, we will use only strong beams since these groups contain higher coverage and resolution due to higher surface returns. According to the user guide, the spacecraft was in the backwards orientation during our day of interest, setting the gt*l beams as the strong beams.

We'll use these primary geolocation, height and quality variables of interest for each of the three strong beams. The following descriptions are provided in the ATL07 Data Dictionary, with additional information on the algorithm and variable descriptions in the ATBD (Algorithm Theoretical Basis Document).

delta_time: Number of GPS seconds since the ATLAS SDP epoch.

latitude: Latitude, WGS84, North=+, Lat of segment center

longitude: Longitude, WGS84, East=+,Lon of segment center

height_segment_height: Mean height from along-track segment fit determined by the sea ice algorithm

height_segment_confidence: Confidence level in the surface height estimate based on the number of photons; the background noise rate; and the error analysis

height_segment_quality: Height segment quality flag, 1 is good quality, 0 is bad

height_segment_surface_error_est: Error estimate of the surface height (reported in meters)

height_segment_length_seg: along-track length of segment containing n_photons_actual

We will enter these variables into the data dictionary below

Select data access configurations¶

The data request can be accessed asynchronously or synchronously. The asynchronous option will allow concurrent requests to be queued and processed as orders. Those requested orders will be delivered to the specified email address, or they can be accessed programmatically as shown below. Synchronous requests will automatically download the data as soon as processing is complete. For this tutorial, we will be selecting the asynchronous method.

Create the data request API endpoint¶

Programmatic API requests are formatted as HTTPS URLs that contain key-value-pairs specifying the service operations that we specified above. We will first create a string of key-value-pairs from our data dictionary and we'll feed those into our API endpoint. This API endpoint can be executed via command line, a web browser, or in Python below.

Request data and clean up Output folder¶

We will now download data using the request_data function, which utilizes the Python requests library. Our param_dict and HTTP session will be passed to the function to allow Earthdata Login access. The data will be downloaded directly to this notebook directory in a new Outputs folder. The progress of the order will be reported. The data are returned in separate files, so we'll use the clean_folder function to remove those individual folders.

To review, we have explored data availability and volume over a region and time of interest, discovered and selected data customization options, constructed API endpoints for our requests, and downloaded data. Let's move on to the analysis portion of the tutorial.