This section gives an overview over the most commonly askes questions and the best practice when it comes to plugin development.
Freva from jupyter notebooks?¶The easiest way to use freva in a jupyter notebook is to log on the HPC system where freva is installed and load the freva module. The name of the specific freva module should be
communicated by your admin team. If the name of the freva module is freva then simply load it with:
module load freva
After that you can install a new jupyter kernel via:
jupyter-kernel-install python --name freva --display-name FrevaKernel
After you have installed the kernel you can use freva in your jupyter notebook:
import freva
# Get an overview of the availalbe freva plugins
freva.get_tools_list()
# Get the documentation of a plugin
freva.plugin_doc("animator")
If you don't want to create a new jupyter kernel that points to the python environment where the freva system is installed you can also install freva into your own python environment. You can either install freva via pip:
python3 -m pip install freva
or conda:
conda install -c conda-forge freva
Afterwards you can use freva in your own environment.
The only difference to the above approach of using the freva python environment is that you will have to tell freva to use the correct configuration.
import freva
# In order to use freva we have to tell where to get the configuration
freva.config("/path/to/the/freva/config/file.conf")
# Talk to your admins to get the location of the config file.
Please talk to your admins on where the central freva configuration is located.
Freva in my analysis workflows?¶You can use Freva without creating or applying data analysis plugins. One example would be using the databrowser command in you data analysis workflow:
# Use the databrowser search and pipe the output to ncremap
freva databrowser project=observations experiment=cmorph time_frequency=30min | ncremap -m map.nc -O drc_rgr
Below you can find a python example, which you could use in a notebook
import freva
import xarray as xr
# Open the data with xarray
dset = xr.open_mfdataset(freva.databrowser(project="obs*", time_frequency='30min'), combine='by_coords')['pr']
dset
Freva module in a plugin¶Like above you can use the Freva python module within your wrapper API code of a plugin
import json
from evaluation_system.api import plugin, parameters
import freva
class MyPlugin(plugin.PluginAbstract):
"""An analysis plugin that uses the databrowser search."""
__parameters__ = parameters.ParameterDictionary(
parameters.SolrField(name="project", facet="project", help="Set the project name"),
parameters.SolrField(name="variable", facet="variable", help="Set the variable name"),
)
def run_tool(self, config_dict):
"""Main plugin method that makes Freva calls."""
# Search for files
files = list(freva.databrowser(**config_dict))
# Save the files to a json file
with open("/tmp/some_json_file.json", "w") as f:
json.dumps(files, f)
self.call("external_command /tmp/some_json_file.json")
If you have a plugin that makes calls to a command line interface (like a shell script) you should avoid long command line argument calls like this:
import json
from evaluation_system.api import plugin, parameters
import freva
class MyPlugin(plugin.PluginAbstract):
def run_tool(self, config_dict):
result = self.call('%s/main.py %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s'
% (self.class_basedir, config_dict['inputdir'], config_dict['project'],
config_dict['product'], config_dict['institute'], config_dict['model'],
config_dict['experiment'], config_dict['time_frequency'],
config_dict['variable'], config_dict['ensemble'], config_dict['cmor_table'],
config_dict['outputdir'], config_dict['cache'], config_dict['cacheclear'],
config_dict['seldate'], config_dict['latlon'], config_dict['area_weight'],
config_dict['percentile_threshold'], config_dict['threshold'],
config_dict['persistence'], config_dict['sel_extr'], config_dict['dryrun'])
)
Such call are very hard to read and should be avoided. Instead you can use Freva as much as possible within the wrapper code and save relevant results into a json file that is passed and a single argument to the tool. Using the above example the code could be simplified as follows:
import json
from tempfile import NamedTemporaryFile
from evaluation_system.api import plugin, parameters
import freva
class MyPlugin(plugin.PluginAbstract):
def run_tool(self, config_dict):
config_dict['input_files'] = list(
freva.databrowser(
product=config_dict.pop('product'), project=config_dict.pop('project'),
institute=config_dict.pop('institute'), model=config_dict.pop('model'),
experiment=config_dict.pop('experiment'), variable=config_dict.pop('variable'),
ensemble=config_dict.pop('ensemble'), cmor_table=config_dict.pop('cmor_table')
)
)
with NamedTemporaryFile(suffix='.json') as tf:
with open(tf.name, 'w') as f:
json.dump(config_dict, f)
self.call(f"{self.class_basedir}/main.py {tf.name}")
Here we use json because most scripting and programming languages have a json parser functionality which can be conveniently used.