The Solar Forecast Arbiter allows users to upload and download metadata and data using an HTTP API. The HTTP API documentation is available here and contains examples for each type of request. This Jupyter notebook is designed to introduce you to the solarforecatarbiter-core package's Python wrapper of the API.
Click the ">| Run" button in the toolbar above or type shift-enter to run the code in each cell. The help menu contains a brief User Interface Tour.
import datetime
import os
import pandas as pd
import requests
from bokeh.io import output_notebook
from bokeh.plotting import show
TOOLS = "pan,box_zoom,xwheel_zoom,reset,save"
output_notebook()
from solarforecastarbiter import datamodel
There are two important objects in the solarforecastarbiter
API wrapper:
request_cli_access_token
APISession
See the documentation here.
from solarforecastarbiter.io.api import APISession, request_cli_access_token
To access data in the Solar Forecast Arbiter, a user must use a valid username and password to obtain a token. Read more about authentication here. The request_cli_access_token
is a convenient function for obtaining a token within Python. Your token will be different from the one printed below when you opened the notebook. To get started, we will use a testing account that has read-only permissions.
# don't store your real passwords or tokens in plain text like this! only for demonstration purposes!
token = request_cli_access_token('testing@solarforecastarbiter.org', 'Thepassword123!')
token
'eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik5UZENSRGRFTlVNMk9FTTJNVGhCTWtRelFUSXpNRFF6TUVRd1JUZ3dNekV3T1VWR1FrRXpSUSJ9.eyJpc3MiOiJodHRwczovL3NvbGFyZm9yZWNhc3RhcmJpdGVyLmF1dGgwLmNvbS8iLCJzdWIiOiJhdXRoMHw1YmUzNDNkZjcwMjU0MDYyMzc4MjBiODUiLCJhdWQiOlsiaHR0cHM6Ly9hcGkuc29sYXJmb3JlY2FzdGFyYml0ZXIub3JnIiwiaHR0cHM6Ly9zb2xhcmZvcmVjYXN0YXJiaXRlci5hdXRoMC5jb20vdXNlcmluZm8iXSwiaWF0IjoxNjA0OTc2MDE0LCJleHAiOjE2MDQ5ODY4MTQsImF6cCI6ImMxNkVKbzQ4bGJUQ1FFaHFTenRHR2xteHh4bVo0elg3Iiwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImd0eSI6InBhc3N3b3JkIn0.TMLxZcXRmzf0QFCYb8XfoC-6GkYEQKraat31sYYiu6BxjaGTzd9oEA9HNh2tlaI8fdwtr8OwfH9jefCuR0RaIXdi2ZgMfLPfubZ9OYDSklBs70d_6ou9wA-CxOv5f6ZGEpd3IyXqJPimb_6KlGaGwTD_ucupUCcBEpUz-LWzMwlm1VzwPQQpN7aGCalbi2XUk-3vBOuTUEtjplLvD_ngn8Hf4ZMTPFzXwZYC-atj83PixwwH-VkzwW1-ZcpDP5ThTf98qEPTPZiOH422yeFNkkUcsKX7DGn7sPYgzGJlU8SARVwdo07oTAhtdGaEJgJ_WDxXfAzJzpS3tQll0QEWsw'
The APISession
uses the valid token to communicate with the API.
session = APISession(token)
The APISession.list_sites
method returns a list of all sites that the user has access to. Most of these are reference data sites.
sites = session.list_sites()
Let's see how many sites we have access to.
len(sites)
226
# print every 30th
for site in sites[::30]:
print(site, '\n')
SolarPowerPlant(name='PSEL Reference System', latitude=35.05, longitude=-106.54, elevation=1657.0, timezone='America/Denver', site_id='af1e53ca-7e3d-11e9-ab45-52540015d5ce', provider='Reference', extra_parameters='{"network": "Sandia RTC"}', climate_zones=('Reference Region 3',), modeling_parameters=FixedTiltModelingParameters(ac_capacity=0.003, dc_capacity=0.003, temperature_coefficient=-0.4, dc_loss_factor=0.0, ac_loss_factor=0.0, surface_tilt=35.0, surface_azimuth=180.0, tracking_type='fixed')) Site(name='NREL MIDC National Wind Technology Center', latitude=39.9106, longitude=-105.2347, elevation=1855.0, timezone='Etc/GMT+7', site_id='9ecbbad2-7e49-11e9-b20a-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "NWTC", "network_api_abbreviation": "NWTC M2", "observation_interval_length": 1}', climate_zones=('Reference Region 4',)) Site(name='NOAA USCRN Elkins WV', latitude=39.01, longitude=-79.47, elevation=1033.0, timezone='America/New_York', site_id='c41990fe-7e49-11e9-b7a7-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "WV_Elkins_21_ENE", "network_api_abbreviation": "WV_Elkins_21_ENE", "observation_interval_length": 5}', climate_zones=('Reference Region 5',)) Site(name='NOAA USCRN Joplin MO', latitude=37.42, longitude=-94.58, elevation=290.0, timezone='America/Chicago', site_id='c6e6aca6-7e49-11e9-8ac1-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "MO_Joplin_24_N", "network_api_abbreviation": "MO_Joplin_24_N", "observation_interval_length": 5}', climate_zones=('Reference Region 5',)) Site(name='NOAA USCRN Monroe LA', latitude=32.88, longitude=-92.11, elevation=27.0, timezone='America/Chicago', site_id='c9d6549e-7e49-11e9-9a83-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "LA_Monroe_26_N", "network_api_abbreviation": "LA_Monroe_26_N", "observation_interval_length": 5}', climate_zones=('Reference Region 6',)) Site(name='NOAA USCRN Titusville FL', latitude=28.61, longitude=-80.69, elevation=1.0, timezone='America/New_York', site_id='cccb75b8-7e49-11e9-a14d-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "FL_Titusville_7_E", "network_api_abbreviation": "FL_Titusville_7_E", "observation_interval_length": 5}', climate_zones=('Reference Region 8',)) Site(name='DOE ARM E11 Byron, Oklahoma', latitude=36.881001, longitude=-98.285004, elevation=367.5, timezone='America/Chicago', site_id='cfb5a324-88c4-11ea-bdba-0a580a820092', provider='Reference', extra_parameters='{"observation_interval_length": 1.0, "network_api_abbreviation": "sgp", "attribution": "https://www.arm.gov/capabilities/vaps/qcrad", "network": "DOE ARM", "arm_site_id": "sgp", "network_api_id": "E11", "datastreams": {"met": "sgpmetE11.b1", "qcrad": {"sgpqcrad1longE11.c2": "1995-06-30/2019-08-02", "sgpqcrad1longE11.c1": "2019-08-02/now"}}}', climate_zones=('Reference Region 4',)) SolarPowerPlant(name='NREL PVDAQ Andre Agassi Preparatory Academy Building C', latitude=36.1952, longitude=-115.1582, elevation=625.33, timezone='Etc/GMT+8', site_id='6d0fccec-d2b5-11ea-b670-0a580a820081', provider='Reference', extra_parameters='{"network": "NREL PVDAQ", "network_api_id": 1277, "network_api_abbreviation": "pvdaq", "observation_interval_length": 15, "attribution": "https://developer.nrel.gov/docs/solar/pvdaq-v3/", "inverter_mfg": "SatCon Technology", "inverter_model": "30kW", "module_mfg": "Sharp ", "module_model": "NU-U240F1", "module_tech": "1"}', climate_zones=('Reference Region 3',), modeling_parameters=FixedTiltModelingParameters(ac_capacity=0.04056, dc_capacity=0.04056, temperature_coefficient=0.3, dc_loss_factor=0.0, ac_loss_factor=0.0, surface_tilt=10.0, surface_azimuth=180.0, tracking_type='fixed'))
We'd like to find the site that represents the NREL MIDC observing station located on the University of Arizona campus.
sites_filtered = list(filter(lambda x: 'NREL MIDC' in x.name and 'Arizona' in x.name, sites))
sites_filtered
[Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',))]
The filtered list has just one site, so pick it out for future queries.
oasis = sites_filtered[0]
Now we repeat the process for observations.
observations = session.list_observations()
There are 3-6 observations per site, so the list is quite long.
len(observations)
913
# print every 200th
for observation in observations[::200]:
print(observation, '\n')
Observation(name='PSEL Reference POA Irradiance', variable='poa_global', interval_value_type='instantaneous', interval_length=Timedelta('0 days 00:01:00'), interval_label='beginning', site=SolarPowerPlant(name='PSEL Reference System', latitude=35.05, longitude=-106.54, elevation=1657.0, timezone='America/Denver', site_id='af1e53ca-7e3d-11e9-ab45-52540015d5ce', provider='Reference', extra_parameters='{"network": "Sandia RTC"}', climate_zones=('Reference Region 3',), modeling_parameters=FixedTiltModelingParameters(ac_capacity=0.003, dc_capacity=0.003, temperature_coefficient=-0.4, dc_loss_factor=0.0, ac_loss_factor=0.0, surface_tilt=35.0, surface_azimuth=180.0, tracking_type='fixed')), uncertainty=0.1, observation_id='af1ec4ff-7e3d-11e9-ab45-52540015d5ce', provider='Reference', extra_parameters='{"network": "Sandia RTC"}', units='W/m^2') Observation(name='Madison Wisconsin dni', variable='dni', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NOAA SOLRAD Madison Wisconsin', latitude=43.0725, longitude=-89.41133, elevation=271.0, timezone='America/Chicago', site_id='c2a4ec4a-7e49-11e9-b4d5-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA SOLRAD", "network_api_id": "MSN", "network_api_abbreviation": "msn", "observation_interval_length": 1}', climate_zones=('Reference Region 5',)), uncertainty=0.0, observation_id='c2aeae8a-7e49-11e9-a920-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA SOLRAD", "network_api_id": "MSN", "network_api_abbreviation": "msn", "observation_interval_length": 1}', units='W/m^2') Observation(name='King Salmon AK ghi', variable='ghi', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:05:00'), interval_label='ending', site=Site(name='NOAA USCRN King Salmon AK', latitude=58.2, longitude=-155.92, elevation=201.0, timezone='America/Anchorage', site_id='c7712218-7e49-11e9-8a01-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "AK_King_Salmon_42_SE", "network_api_abbreviation": "AK_King_Salmon_42_SE", "observation_interval_length": 5}', climate_zones=()), uncertainty=0.0, observation_id='c7748ab0-7e49-11e9-877f-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "AK_King_Salmon_42_SE", "network_api_abbreviation": "AK_King_Salmon_42_SE", "observation_interval_length": 5}', units='W/m^2') Observation(name='Fairhope AL ghi', variable='ghi', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:05:00'), interval_label='ending', site=Site(name='NOAA USCRN Fairhope AL', latitude=30.54, longitude=-87.87, elevation=29.0, timezone='America/Chicago', site_id='cc753eec-7e49-11e9-82c4-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "AL_Fairhope_3_NE", "network_api_abbreviation": "AL_Fairhope_3_NE", "observation_interval_length": 5}', climate_zones=('Reference Region 8',)), uncertainty=0.0, observation_id='cc78be2e-7e49-11e9-9f5d-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NOAA USCRN", "network_api_id": "AL_Fairhope_3_NE", "network_api_abbreviation": "AL_Fairhope_3_NE", "observation_interval_length": 5}', units='W/m^2') Observation(name='Portland OR PV 15 deg tilt dhi_1', variable='dhi', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='beginning', site=SolarPowerPlant(name='UO SRML Portland OR PV 15 deg tilt', latitude=45.51, longitude=-122.69, elevation=70.0, timezone='Etc/GMT+8', site_id='c2e91bfa-a4fd-11ea-b630-0a580a80039b', provider='Reference', extra_parameters='{"network": "UO SRML", "network_api_id": "94808.0(15)", "network_api_abbreviation": "PS", "observation_interval_length": 1.0, "attribution": "Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02"}', climate_zones=('Reference Region 1',), modeling_parameters=FixedTiltModelingParameters(ac_capacity=0.000816, dc_capacity=0.000816, temperature_coefficient=0.0, dc_loss_factor=0.0, ac_loss_factor=0.0, surface_tilt=15.0, surface_azimuth=180.0, tracking_type='fixed')), uncertainty=0.0, observation_id='c42073fa-a4fd-11ea-8d06-0a580a80039b', provider='Reference', extra_parameters='{"network": "UO SRML", "network_api_id": "94808.0(15)", "network_api_abbreviation": "PS", "observation_interval_length": 1.0, "attribution": "Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02", "network_data_label": "dhi_1"}', units='W/m^2')
Notice that each observation object contains metadata about the observation type (e.g. variable, interval length) and the site that it is associated with. We can extract the observations from the site of interest using another filter statement.
observations_oasis = list(filter(lambda x: x.site == oasis, observations))
for observation in observations_oasis:
print(observation, '\n')
Observation(name='University of Arizona OASIS ghi', variable='ghi', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='9f657636-7e49-11e9-b77f-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1, "network_data_label": "Global Horiz (platform) [W/m^2]"}', units='W/m^2') Observation(name='University of Arizona OASIS dni', variable='dni', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='9f69e78a-7e49-11e9-a29d-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1, "network_data_label": "Direct Normal [W/m^2]"}', units='W/m^2') Observation(name='University of Arizona OASIS dhi', variable='dhi', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='9f6e651c-7e49-11e9-a723-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1, "network_data_label": "Diffuse Horiz [W/m^2]"}', units='W/m^2') Observation(name='University of Arizona OASIS air_temperature', variable='air_temperature', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='9f738abe-7e49-11e9-b653-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1, "network_data_label": "Air Temperature [deg C]"}', units='degC') Observation(name='University of Arizona OASIS relative_humidity', variable='relative_humidity', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='9f7882ac-7e49-11e9-9c99-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1, "network_data_label": "Rel Humidity [%]"}', units='%') Observation(name='University of Arizona OASIS wind_speed', variable='wind_speed', interval_value_type='interval_mean', interval_length=Timedelta('0 days 00:01:00'), interval_label='ending', site=Site(name='NREL MIDC University of Arizona OASIS', latitude=32.22969, longitude=-110.95534, elevation=786.0, timezone='Etc/GMT+7', site_id='9f61b880-7e49-11e9-9624-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1}', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='9f7cdf58-7e49-11e9-97e7-0a580a8003e9', provider='Reference', extra_parameters='{"network": "NREL MIDC", "network_api_id": "UAT", "network_api_abbreviation": "UA OASIS", "observation_interval_length": 1, "network_data_label": "Avg Wind Speed @ 3m [m/s]"}', units='m/s')
Now we're ready to get data from the API using session.get_observation_values
.
start = pd.Timestamp('20190520 0000Z')
end = pd.Timestamp('20190525 0000Z')
The currently method requires an observation_id
string, so extract that from an observation.
oasis_ghi = observations_oasis[0]
oasis_ghi_id = oasis_ghi.observation_id
oasis_ghi_id
'9f657636-7e49-11e9-b77f-0a580a8003e9'
oasis_ghi_values = session.get_observation_values(oasis_ghi_id, start, end)
oasis_ghi_values.head()
value | quality_flag | |
---|---|---|
timestamp | ||
2019-05-20 00:00:00+00:00 | 200.669 | 2 |
2019-05-20 00:01:00+00:00 | 199.503 | 2 |
2019-05-20 00:02:00+00:00 | 195.973 | 2 |
2019-05-20 00:03:00+00:00 | 192.769 | 2 |
2019-05-20 00:04:00+00:00 | 190.638 | 2 |
oasis_ghi_values.tail()
value | quality_flag | |
---|---|---|
timestamp | ||
2019-05-24 23:56:00+00:00 | 470.606 | 2 |
2019-05-24 23:57:00+00:00 | 467.037 | 2 |
2019-05-24 23:58:00+00:00 | 464.077 | 2 |
2019-05-24 23:59:00+00:00 | 460.442 | 2 |
2019-05-25 00:00:00+00:00 | 456.825 | 2 |
from solarforecastarbiter.plotting import timeseries
fig = timeseries.generate_observation_figure(oasis_ghi, oasis_ghi_values)
show(fig)
New sites, observations, and forecasts may be created using the python API wrappers. Our testing account is not allowed to post, so we have to use real credentials to demonstrate this. To maintain security, we've set our username and password in environment variables, and the code below will extract those and use them to obtain a token. The code will not work unless you export your username/password to the correct environment variables. You will also need permissons to create new sites, observations, and data -- check with your organization administrator if you get an error.
token = request_cli_access_token(os.environ['SFA_API_USERNAME'], os.environ['SFA_API_PASSWORD'])
session = APISession(token)
To do so, we:
Site
.The observation and forecast need to be associated with the unique site_id
assigned by the API.
site = datamodel.Site(
name='Tucson AZ',
latitude=32.2,
longitude=-110.9,
elevation=700,
timezone='America/Phoenix'
)
The API returns a new site object that has the same attributes but includes:
site_id
provider
field that is automatically determined by the user's affiliationsite_returned = session.create_site(site)
site_returned
Site(name='Tucson AZ', latitude=32.2, longitude=-110.9, elevation=700.0, timezone='America/Phoenix', site_id='1185ad1c-22fe-11eb-a505-0a580a8201a5', provider='University of Arizona', extra_parameters='', climate_zones=('Reference Region 3',))
Now we can use the site returned by the API to create the Observation and Forecast objects.
observation = datamodel.Observation(
name='sample observation',
interval_length=pd.Timedelta('1hr'),
interval_label='ending',
interval_value_type='interval_mean',
variable='ghi',
uncertainty=0,
site=site_returned
)
forecast = datamodel.Forecast(
name='sample forecast',
issue_time_of_day=datetime.time(0),
lead_time_to_start=pd.Timedelta('1h'),
interval_length=pd.Timedelta('1h'),
run_length=pd.Timedelta('1h'),
interval_label='ending',
interval_value_type='interval_mean',
variable='ghi',
site=site_returned
)
observation_returned = session.create_observation(observation)
observation_returned
Observation(name='sample observation', variable='ghi', interval_value_type='interval_mean', interval_length=Timedelta('0 days 01:00:00'), interval_label='ending', site=Site(name='Tucson AZ', latitude=32.2, longitude=-110.9, elevation=700.0, timezone='America/Phoenix', site_id='1185ad1c-22fe-11eb-a505-0a580a8201a5', provider='University of Arizona', extra_parameters='', climate_zones=('Reference Region 3',)), uncertainty=0.0, observation_id='1193ad22-22fe-11eb-83a9-0a580a8201a5', provider='University of Arizona', extra_parameters='', units='W/m^2')
forecast_returned = session.create_forecast(forecast)
forecast_returned
Forecast(name='sample forecast', issue_time_of_day=datetime.time(0, 0), lead_time_to_start=Timedelta('0 days 01:00:00'), interval_length=Timedelta('0 days 01:00:00'), run_length=Timedelta('0 days 01:00:00'), interval_label='ending', interval_value_type='interval_mean', variable='ghi', site=Site(name='Tucson AZ', latitude=32.2, longitude=-110.9, elevation=700.0, timezone='America/Phoenix', site_id='1185ad1c-22fe-11eb-a505-0a580a8201a5', provider='University of Arizona', extra_parameters='', climate_zones=('Reference Region 3',)), aggregate=None, forecast_id='11a3e2f0-22fe-11eb-9632-0a580a8201a5', provider='University of Arizona', extra_parameters='', units='W/m^2')
You can visit the dashboard to confirm that the site, observation, and forecast all exist. Browse the full site list for the name of your new site, or go directly to https://dashboard.solarforecastarbiter.org/sites/<<copy/paste-site-id-here>>
. The site_id
is printed below for reference. Once on the site page, click on "Observations" or "Forecasts" to see those metadata.
print(site_returned.site_id)
1185ad1c-22fe-11eb-a505-0a580a8201a5
As a simple example, let's use the NREL MIDC OASIS data we previously downloaded as the upload for the new observation.
# quality_flag=0 indicates no problems with the data. quality_flag=1 indicates user flagged problem.
data_to_upload = pd.DataFrame({'value': oasis_ghi_values['value'], 'quality_flag': 0})
new_obs_id = observation_returned.observation_id
session.post_observation_values(new_obs_id, data_to_upload)
--------------------------------------------------------------------------- HTTPError Traceback (most recent call last) <ipython-input-31-1fa095677b0d> in <module> ----> 1 session.post_observation_values(new_obs_id, data_to_upload) ~/miniconda3/envs/sfawork/lib/python3.8/site-packages/solarforecastarbiter/io/api.py in post_observation_values(self, observation_id, observation_df, params) 856 """ # NOQA 857 json_vals = observation_df_to_json_payload(observation_df) --> 858 self.post(f'/observations/{observation_id}/values', 859 data=json_vals, params=params, 860 headers={'Content-Type': 'application/json'}) ~/miniconda3/envs/sfawork/lib/python3.8/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs) 579 """ 580 --> 581 return self.request('POST', url, data=data, json=json, **kwargs) 582 583 def put(self, url, data=None, **kwargs): ~/miniconda3/envs/sfawork/lib/python3.8/site-packages/solarforecastarbiter/io/api.py in request(self, method, url, *args, **kwargs) 138 result = super().request(method, url, *args, **kwargs) 139 if result.status_code >= 400: --> 140 raise requests.exceptions.HTTPError( 141 f'{result.status_code} API Request Error: {result.reason} for ' 142 f'url: {result.url} and text: {result.text}', HTTPError: 400 API Request Error: BAD REQUEST for url: https://api.solarforecastarbiter.org/observations/1193ad22-22fe-11eb-83a9-0a580a8201a5/values and text: {"errors":{"timestamp":["7080 extra times present in index. First extra time is 2019-05-20 00:01:00+00:00. Uploads must have equally spaced timestamps from 2019-05-20 00:00:00+00:00 to 2019-05-25 00:00:00+00:00 with 60 minutes between each timestamp."]}}
The API rejected the upload because the interval length of the data does not match the interval length we specified for the metadata. Recall that the metadata described data with hourly mean and interval ending label.
resampled_data = data_to_upload.resample('1h', label='right').mean()
fig = timeseries.generate_observation_figure(observation, resampled_data)
show(fig)
Now we post the data to the API.
session.post_observation_values(new_obs_id, resampled_data)
It (should have) worked. Let's confirm that the data exists.
oasis_ghi_values_1h = session.get_observation_values(new_obs_id, start, end)
fig = timeseries.generate_observation_figure(observation, oasis_ghi_values_1h)
show(fig)
The quality flag returned by the API may be "NOT VALIDATED" or it may show at least "NIGHTTIME". The API only checks that the data format is valid. The validation step occurs only after the API sends an "OK" response to the data post function. The validation typically takes at least a few seconds to complete. So, if you're reading along as you execute the code you may have given the server enough time to validate the data. If you're quickly executing the code cells or used the "Run all" command then you may need to wait a few seconds and request the data again.
The cells below wait for 5 seconds to insure that the API has time to process the data and then request the data once again.
import time
time.sleep(5)
oasis_ghi_values_1h = session.get_observation_values(new_obs_id, start, end)
fig = timeseries.generate_observation_figure(observation, oasis_ghi_values_1h)
show(fig)
Let's finish by deleting the data and metadata that we created. The solarforecastarbiter-core
python library does wrap the API's delete methods, so we'll have to use the SFA HTTP API directly.
Here we use the requests
library to make the HTTP API calls. First we set up the authorization header.
base_url = 'https://api.solarforecastarbiter.org'
headers = {'Authorization': f'Bearer {token}'}
Sites can only be deleted once all associated observations and forecasts are deleted, so we'll first delete the observations/forecasts and then the site.
This is the API address for the observation we created.
url = f'{base_url}/observations/{observation_returned.observation_id}'
url
'https://api.solarforecastarbiter.org/observations/1193ad22-22fe-11eb-83a9-0a580a8201a5'
We can add /metadata
to that url and confirm that this is the observation we're looking for.
r = requests.request('GET', f'{url}/metadata', headers=headers)
r.json()
{'_links': {'site': 'https://api.solarforecastarbiter.org/sites/1185ad1c-22fe-11eb-a505-0a580a8201a5'}, 'created_at': '2020-11-10T02:40:17+00:00', 'extra_parameters': '', 'interval_label': 'ending', 'interval_length': 60, 'interval_value_type': 'interval_mean', 'modified_at': '2020-11-10T02:40:17+00:00', 'name': 'sample observation', 'observation_id': '1193ad22-22fe-11eb-83a9-0a580a8201a5', 'provider': 'University of Arizona', 'site_id': '1185ad1c-22fe-11eb-a505-0a580a8201a5', 'uncertainty': 0.0, 'variable': 'ghi'}
r = requests.request('DELETE', url, headers=headers)
r
<Response [204]>
A response of 204 indicates a successful deletion. Now repeat the process for the forecast and finally the site.
url = f'{base_url}/forecasts/single/{forecast_returned.forecast_id}'
r = requests.request('DELETE', url, headers=headers)
r
<Response [204]>
url = f'{base_url}/sites/{site_returned.site_id}'
r = requests.request('DELETE', url, headers=headers)
r
<Response [204]>