#!/usr/bin/env python
# coding: utf-8
#
#
#
# # Data Access - Preparatory Material
# ## Introduction
# This notebook will introduce you to the data sources that we will be working on during this training school. You will learn about:
# * [Where to get data](#sources)
# * [Where to register for accounts](#register)
# * [Different ways to access data](#access)
# * [Citing data](#citing)
#
# ## Where to get data
# You will learn about **ten sources of data** during this training school and practice accessing data from them. Ten sources sounds like a lot but don't worry, we will also include links to access the data in all the example notebooks. We have also pre-downloaded the data for you in case you have limited internet access and it takes a long time for you to download the data.
#
# Here are the ten data sources grouped by type of data:
#
# ### Satellite data
#
# * **[EUMETSAT Earth Observation (EO) Portal](https://eoportal.eumetsat.int/)** is where you can access numerous data products from EUMETSAT.
#
# * **[NASA LAADS DAAC](https://ladsweb.modaps.eosdis.nasa.gov/)** archives and distributes data on clouds, water vapor, and aerosols in Earth’s atmosphere as well as key instrument data for NASA, NOAA and European Space Administration (ESA) missions.
#
# * **[Sentinel-5P Pre-Operations Data Hub](https://s5phub.copernicus.eu/dhus/#/home)** is a web portal to download data products from the Sentinel-5P TROPOMI instrument.
#
# * **[Tropospheric Emission Monitoring Internet Service (TEMIS)](https://www.temis.nl/)** is the source for tropospheric data products produced by ESA. We will use this source to access datat from the GOME-2 instrument onboard the MetOp-A/B/C satellites.
#
# * **[Copernicus Climate Data Store (CDS)](https://cds.climate.copernicus.eu/#!/home)** provides access to climate datasets produced by ECMWF. While there are also model-based data available at CDS, we will use this source to access data products from the IASI instrument onboard the MetOp-A/B/C satellites.
#
# ### Ground-based data
#
# * **[Aerosol Robotic Network (AERONET)](https://aeronet.gsfc.nasa.gov/)** is a global network of ground-based measurement stations managed by NASA.
#
# * **[European Aerosol Research Lidar Network (EARLINET)](https://www.earlinet.org/index.php?id=125)** provides ground-based data for aerosol vertical distribution over Europe under the ACTRIS framework.
#
# * **[European Environmental Agency (EEA) AirBase](https://www.eea.europa.eu/data-and-maps/data/airbase-the-european-air-quality-database-8)** provides data from a Europe-wide network of air quality stations.
#
# ### Model-based data
#
# * **[Copernicus Atmosphere Data Store (ADS)](https://ads.atmosphere.copernicus.eu/#!/home)** provides access to atmospheric composition datasets produced by ECMWF. We will use this source to access model forecast and reanalysis data from the Copernicus Atmosphere Monitoring Service (CAMS).
#
# * **[WMO Sand and Dust Storm Warning Advisory and Assessment System (SDS-WAS)](https://dust.aemet.es/products)** is an international framework linking institutions involved in Sand and Dust Storm (SDS) research, operations and delivery of services. We will use this source to access regional dust forecasts from the MONARCH model.
#
# ## Where to register for accounts
# Five of these data sources require you to sign-up for an account to access the data. You are advised to register for an account before the training school so you are ready ahead of time.
#
# Recommended: Sign-up for an account at each of the following websites:
#
# 1. EUMETSAT Earth Observation (EO) Portal - [Register for an EO Portal account.](https://eoportal.eumetsat.int/userMgmt/register.faces)
# 2. NASA LAADS DAAC - [Register for an Earthdata account.](https://urs.earthdata.nasa.gov/users/new)
# 3. European Aerosol Research Lidar Network (EARLINET) - [Register for an EARLINET account.](https://data.earlinet.org/earlinet/userRegistration.zul)
# 4. Copernicus Atmosphere Data Store (ADS) - [Register for an ADS account.](https://ads.atmosphere.copernicus.eu/user/register)
# 5. Copernicus Climate Data Store (CDS) - [Register for an CDS account.](https://cds.climate.copernicus.eu/user/register)
#
# ## Different ways to access data
# There may be more than one way to access the data from a source. You could access the data **manually** or **programmatically**. Some data sources allow you to do both.
#
# You could go to a web portal to **manually** select and download data using a map or a dropdown menu for filtering a date and time. There may be a file system that allows you to see a directory of data files and click on individual files to download them. You can also use the web portal to view and browse available data without downloading anything.
#
# Or you could access the data **programmatically** using some Python code via an Application Programme Interface (API) or a software package such as [wget](https://www.gnu.org/software/wget/).
# ### Accessing data manually
# #### Web portal
#
# An example of a web portal, the [Sentinel-5P Pre-Operations Data Hub](https://s5phub.copernicus.eu/dhus/#/home), is shown below. The left panel contains dropdown menus that allow you to select the product type and processing level, as well as the period of interest. You can draw a bounding box on the map around your area of interest to filter the data geographically.
#
#
#
#
#
# Manually accessing data via a web portal using dropdown menus (left panel) and drawing a bounding box around your area of interest (orange box)
#
#
# #### File system on a Web Server
#
# One example of a file system is on the [SDS-WAS Data Download page](https://dust.aemet.es/products/data-download), also shown in the screenshot below. After clicking on folders to navigate through the directory structure, you will end up inside a folder where you can click on individual data files to download them.
#
#
#
#
#
# Manually accessing data via a file system by navigating through folder directories
#
#
# Note that many file systems also allow you to access data programmatically as well. In the case of the SDS-WAS, this file system is based on a THREDDS Data Server (TDS) which "is a web server that provides metadata and data access for scientific datasets, using OPeNDAP, OGC WMS and WCS, HTTP, and other remote data access protocols." ([Source](https://docs.unidata.ucar.edu/tds/current/userguide/index.html)) You can also use the software package *wget* (will be introduced in the next section) to access the data programmatically.
# ### Accessing Data Programmatically
# If you need to download larger amounts of data, using a few lines of code can make things much easier. You can specify the filters to use in order to download specific data products over an geographic area or time period you want. Accessing data programmatically via an API or a software package are common ways to do this.
#
# #### Application Programme Interface (API)
# Simply put, an API allows two computer programmes to communicate and transfer data between each other. The screenshot below shows some code allowing you to use an API to request and download data from the [Copernicus Atmosphere Data Store (ADS)](https://ads.atmosphere.copernicus.eu/). Both the ADS and Climate Data Store use the same API, called the [CDS API](https://cds.climate.copernicus.eu/api-how-to).
#
# To see an example of how to download data programmatically from the ADS using an API, click on [this link](../04_data_workflows/43_model_data/431_CAMS_global_forecast_duaod_load_browse.ipynb). You will need to use an API key that is unique to you in order to confirm your identity (or authenticate) your access to the API.
#
#
#
#
#
# Programmatically accessing data via an API with some Python code
#
#
# #### Software packages
#
# Another way to programmatically download data is using a software package such as *[wget](https://www.gnu.org/software/wget/)*. This software package helps you retrieve files using common Internet protocols. For an example of using *wget* to download data from AERONET, click on [this link](../04_data_workflows/42_groundbased_observations/421_AERONET.ipynb). You could use *wget* to access data from many different sources as it is a general software package.
#
# There are also some software packages that specialise in accessing data from just one source. For example, the Python package *[airbase](https://airbase.readthedocs.io/en/latest/)* helps you download data from the EEA air quality database. To see how to use *airbase* to download data, click on [this link](../04_data_workflows/42_groundbased_observations/423_EEA_Air_Quality.ipynb).
# ### How do I choose which method to use?
#
# For downloading small amounts of data over a specific area and short period of time, it is often easier to download data manually via a web portal. For longer-term studies and creating time-series from large amounts of data, using a programmatic approach is usually more efficient.
#
# ## Citing data
# If you use data you downloaded to produce research, figures, maps, articles etc., you should always state the source of the data you used and check if there are license terms you need to follow. Data sources often have a recommended way to cite or acknowledge this data. Usually either the data source or an associated research paper is cited. For each data source below, here are the suggested citations or acknowledgements.
#
# ### Satellite data
#
# * **[EUMETSAT Earth Observation (EO) Portal](https://eoportal.eumetsat.int/)**
# Read the [EUMETSAT Data Policy](https://www.eumetsat.int/data-policy/eumetsat-data-policy.pdf) and the [EUMETSAT data licensing page](https://www.eumetsat.int/eumetsat-data-licensing). You have to quote EUMETSAT as the data source.
#
# * **[NASA LAADS DAAC](https://ladsweb.modaps.eosdis.nasa.gov/)**
# * Levy, R., Hsu, C., et al., 2015. MODIS Atmosphere L2 Aerosol Product. NASA MODIS Adaptive Processing System, Goddard Space Flight Center, USA: http://dx.doi.org/10.5067/MODIS/MOD04_L2.061
# * VNP02IMG - VIIRS/NPP Imagery Resolution 6-Min L1B Swath 375m distributed from NASA LAADS DAAC. Available on-line https://ladsweb.modaps.eosdis.nasa.gov/search/order/2/VNP02IMG--5200. doi:10.5067/VIIRS/VNP02IMG.002
# * VNP02DNB - VIIRS/NPP Day/Night Band 6-Min L1B Swath 750m distributed from NASA LAADS DAAC. Available on-line https://ladsweb.modaps.eosdis.nasa.gov/search/order/2/VNP02DNB--5200. doi:10.5067/VIIRS/VNP02IMG.002
# * VNP02MOD - VIIRS/NPP Moderate Resolution 6-Min L1B Swath 750m distributed from NASA LAADS DAAC. Available on-line https://ladsweb.modaps.eosdis.nasa.gov/search/order/2/VNP02MOD--5200. doi:10.5067/VIIRS/VNP02MOD.002
# * VNP03IMG - VIIRS/NPP Imagery Resolution Terrain-Corrected Geolocation 6-Min L1 Swath 375m distributed from NASA LAADS DAAC. Available on-line https://ladsweb.modaps.eosdis.nasa.gov/search/order/2/VNP03IMG--5200. doi:10.5067/VIIRS/VNP03IMG.002
# * VNP03DNB - VIIRS/NPP Day/Night Band Moderate Resolution Terrain-Corrected Geolocation 6-Min L1 Swath 750m distributed from NASA LAADS DAAC. Available on-line https://ladsweb.modaps.eosdis.nasa.gov/search/order/2/VNP02DNB--5200. doi:10.5067/VIIRS/VNP02DNB.002
# * VNP03MOD - VIIRS/NPP Moderate Resolution Terrain-Corrected Geolocation 6-Min L1 Swath 750m distributed from NASA LAADS DAAC. Available on-line https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/products/VNP03MOD. doi:10.5067/VIIRS/VNP03MOD.002
#
# * **[Sentinel-5P Pre-Operations Data Hub](https://s5phub.copernicus.eu/dhus/#/home)**
# From the [Legal notice on the use of Copernicus Sentinel Data and Service Information](https://sentinels.copernicus.eu/documents/247904/690755/Sentinel_Data_Legal_Notice):
#
# ```
# Where the user communicates to the public or distributes Copernicus Sentinel Data and
# Service Information, he/she shall inform the recipients of the source of that Data and
# Information by using the following notice:
#
# (1) 'Copernicus Sentinel data [Year]' for Sentinel data; and/or
# (2) 'Copernicus Service information [Year]' for Copernicus Service Information.
#
# Where the Copernicus Sentinel Data and Service Information have been adapted or
# modified, the user shall provide the following notice:
#
# (1) 'Contains modified Copernicus Sentinel data [Year]' for Sentinel data;
# and/or
# (2) 'Contains modified Copernicus Service information [Year]' for Copernicus
# Service Information.
# ```
#
# * **[Copernicus Climate Data Store (CDS)](https://cds.climate.copernicus.eu/#!/home)**
# From the [License to use Copernicus Products](https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-aerosol-properties?tab=overview:~:text=Licence%20to%20use%20Copernicus%20Products):
#
# ```
# 5.1. All users of Copernicus Products must provide clear and visible attribution to the Copernicus programme. The Licensee will communicate to the public the source of the Copernicus Products by crediting the Copernicus Climate Change and Atmosphere Monitoring Services:
#
# 5.1.1. Where the Licensee communicates or distributes Copernicus Products to the public, the Licensee shall inform the recipients of the source by using the following or any similar notice:
# • 'Generated using Copernicus Climate Change Service information [Year]' and/or
# • 'Generated using Copernicus Atmosphere Monitoring Service information [Year]'.
#
# 5.1.2. Where the Licensee makes or contributes to a publication or distribution containing adapted or modified Copernicus Products, the Licensee shall provide the following or any similar notice:
# • 'Contains modified Copernicus Climate Change Service information [Year]'; and/or
# • 'Contains modified Copernicus Atmosphere Monitoring Service information [Year]'
#
# 5.1.3. Any such publication or distribution covered by clauses 5.1.1 and 5.1.2 shall state that neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus information or data it contains.
# ```
#
# * **[Tropospheric Emission Monitoring Internet Service (TEMIS)](https://www.temis.nl/)**
# * **GOME-2B Absorbing Aerosol Index citation:**
# AC SAF (2017): GOME-2B Absorbing Aerosol Index PMD (R1) - Metop, EUMETSAT SAF on Atmospheric Composition Monitoring, DOI: 10.15770/EUM_SAF_O3M_0006. http://dx.doi.org/10.15770/EUM_SAF_O3M_0006
#
# ### Ground-based data
#
# * **[European Aerosol Research Lidar Network (EARLINET)](https://www.earlinet.org/index.php?id=125)**
# Read the [EARLINET data policy](https://www.earlinet.org/index.php?id=127).
#
# * **[European Environmental Agency (EEA) AirBase](https://www.eea.europa.eu/data-and-maps/data/airbase-the-european-air-quality-database-8)**
# European Commission [YEAR]
#
# * **[Aerosol Robotic Network (AERONET)](https://aeronet.gsfc.nasa.gov/)**
# * **Aerosol Optical Depth v3 Product citation:**
# Giles, D. M., Sinyuk, A., Sorokin, M. G., Schafer, J. S., Smirnov, A., Slutsker, I., Eck, T. F., Holben, B. N., Lewis, J. R., Campbell, J. R., Welton, E. J., Korkin, S. V., and Lyapustin, A. I. (2019): Advancements in the Aerosol Robotic Network (AERONET) Version 3 database – automated near-real-time quality control algorithm with improved cloud screening for Sun photometer aerosol optical depth (AOD) measurements, Atmos. Meas. Tech., 12, 169-209, https://doi.org/10.5194/amt-12-169-2019.
# ### Model-based data
#
# * **[Copernicus Atmosphere Data Store (ADS)](https://ads.atmosphere.copernicus.eu/#!/home)**
# From the [License to use Copernicus Products](https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-aerosol-properties?tab=overview:~:text=Licence%20to%20use%20Copernicus%20Products):
#
# ```
# 5.1. All users of Copernicus Products must provide clear and visible attribution to the Copernicus programme. The Licensee will communicate to the public the source of the Copernicus Products by crediting the Copernicus Climate Change and Atmosphere Monitoring Services:
#
# 5.1.1. Where the Licensee communicates or distributes Copernicus Products to the public, the Licensee shall inform the recipients of the source by using the following or any similar notice:
# • 'Generated using Copernicus Climate Change Service information [Year]' and/or
# • 'Generated using Copernicus Atmosphere Monitoring Service information [Year]'.
#
# 5.1.2. Where the Licensee makes or contributes to a publication or distribution containing adapted or modified Copernicus Products, the Licensee shall provide the following or any similar notice:
# • 'Contains modified Copernicus Climate Change Service information [Year]'; and/or
# • 'Contains modified Copernicus Atmosphere Monitoring Service information [Year]'
#
# 5.1.3. Any such publication or distribution covered by clauses 5.1.1 and 5.1.2 shall state that neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus information or data it contains.
# ```
#
# * **[WMO Sand and Dust Storm Warning Advisory and Assessment System (SDS-WAS)](https://dust.aemet.es/products)**
# Read the [WMO SDS-WAS Data Policy](https://dust.aemet.es/products/daily-dust-products#data-policy).
#
#
# 
# 
# 
# 