Incense demo¶

This notebook demonstrates how incense can be used to retrieve experiments stored in a mongoDB by sacred. It demonstrates the most of the capabilities of incense and should be enough to get you started.

If you want to run the notebook locally you will have to

start up the mongoDB service using docker. Go to infrastructure/sacred_setup and run docker compose-up
execute the example experiment. Got to example/experiment and run python conduct.py

In [1]:

%load_ext autoreload
%autoreload 2

In [2]:

import sys
sys.path.append('.')
from datetime import datetime
import numpy as np
import matplotlib.pyplot as plt

import incense
from incense import ExperimentLoader

Finding experiments¶

To use incense we first have to instantiate an experiment loader that will enable us to query the database for specific runs.

In [3]:

loader = ExperimentLoader(
    mongo_uri=None,    
    db_name='incense_test'
)

It is easiest to retrieve experiments by their id.

In [4]:

exp = loader.find_by_id(2)
exp

Out[4]:

Experiment(id=2, name=example)

It is also possible to find a set of experiments based on their configuration values. Multiple experiments are returned as a QuerySet that just acts as a list, but exposes some custom methods.

In [5]:

loader.find_by_config_key('optimizer', 'sgd')

Out[5]:

QuerySet([Experiment(id=1, name=example), Experiment(id=2, name=example)])

In [6]:

loader.find_by_config_key('epochs', 3.0)

Out[6]:

QuerySet([Experiment(id=2, name=example)])

For more complex queries we can rely on the full power of mongoDB queries using find.

In [7]:

query = {"$and": [
            {"config.optimizer": "sgd"},
            {"config.epochs": 3},
        ]}
loader.find(query)

Out[7]:

QuerySet([Experiment(id=2, name=example)])

Using mongoDB queries und can also request experiments in certain time ranges.

In [8]:

query = {"start_time": {"$gt": datetime(2019, 4, 1)}}
loader.find(query)

Out[8]:

QuerySet([Experiment(id=1, name=example), Experiment(id=2, name=example), Experiment(id=3, name=example)])

Caching¶

By default, the experiment loader will cache the returned experiments. When you want to see updates in your in the database you have to explicitly clear the cache. Caching right now only works for find-methods that take immutable arguments.

In [9]:

loader.cache_clear()

Accessing experiment attributes¶

The experiment object exposes all fields from the sacred data model. To see which keys and values are available we can use the to_dict method.

In [10]:

exp.to_dict()

Out[10]:

{'_id': 2,
 'experiment': {'name': 'example',
  'base_dir': '/home/jarno/projects/incense/example_experiment',
  'sources': [['conduct.py', ObjectId('5caf94233bd29849e342a7e2')]],
  'dependencies': ['matplotlib==3.0.2',
   'numpy==1.16.2',
   'pandas==0.24.1',
   'sacred==0.7.4-onurgu',
   'scikit-learn==0.20.3',
   'seaborn==0.9.0',
   'tensorflow==1.13.1'],
  'repositories': [],
  'mainfile': 'conduct.py'},
 'format': 'MongoObserver-0.7.0',
 'command': 'conduct',
 'host': {'hostname': 'work',
  'os': ['Linux', 'Linux-4.18.0-17-generic-x86_64-with-debian-buster-sid'],
  'python_version': '3.6.8',
  'cpu': 'Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz',
  'ENV': {}},
 'start_time': datetime.datetime(2019, 4, 11, 19, 23, 23, 890000),
 'config': {'epochs': 3, 'optimizer': 'sgd', 'seed': 0},
 'meta': {'command': 'conduct',
  'options': {'--sql': None,
   '--mongo_db': None,
   '--name': None,
   '--file_storage': None,
   '--capture': None,
   '--loglevel': None,
   '--queue': False,
   '--enforce_clean': False,
   '--pdb': False,
   '--beat_interval': None,
   '--comment': None,
   '--print_config': False,
   '--priority': None,
   '--tiny_db': None,
   '--force': False,
   '--unobserved': False,
   '--debug': False,
   '--help': False}},
 'status': 'COMPLETED',
 'resources': [],
 'artifacts': [{'name': 'predictions_df',
   'file_id': ObjectId('5caf943d3bd29849e342a7fe')},
  {'name': 'predictions', 'file_id': ObjectId('5caf943d3bd29849e342a800')},
  {'name': 'confusion_matrix',
   'file_id': ObjectId('5caf943d3bd29849e342a802')},
  {'name': 'confusion_matrix.pdf',
   'file_id': ObjectId('5caf943d3bd29849e342a804')},
  {'name': 'accuracy_movie', 'file_id': ObjectId('5caf943e3bd29849e342a806')},
  {'name': 'history', 'file_id': ObjectId('5caf943e3bd29849e342a808')},
  {'name': 'model.hdf5', 'file_id': ObjectId('5caf943e3bd29849e342a80a')}],
 'captured_out': 'INFO - example - Running command \'conduct\'\nINFO - example - Started run with ID "2"\nFailed to detect content-type automatically for artifact /home/jarno/projects/incense/predictions_df.pickle.\nAdded text/csv as content-type of artifact /home/jarno/projects/incense/predictions.csv.\nAdded image/png as content-type of artifact /home/jarno/projects/incense/confusion_matrix.png.\nAdded application/pdf as content-type of artifact /home/jarno/projects/incense/confusion_matrix.pdf.\nINFO - matplotlib.animation - MovieWriter.run: running command: [\'ffmpeg\', \'-f\', \'rawvideo\', \'-vcodec\', \'rawvideo\', \'-s\', \'3840x2880\', \'-pix_fmt\', \'rgba\', \'-r\', \'1\', \'-loglevel\', \'quiet\', \'-i\', \'pipe:\', \'-vcodec\', \'h264\', \'-pix_fmt\', \'yuv420p\', \'-y\', \'accuracy_movie.mp4\']\nAdded video/mp4 as content-type of artifact /home/jarno/projects/incense/accuracy_movie.mp4.\nAdded text/plain as content-type of artifact /home/jarno/projects/incense/history.txt.\nFailed to detect content-type automatically for artifact /home/jarno/projects/incense/model.hdf5.\nINFO - example - Result: 0.9315000176429749\nINFO - example - Completed after 0:00:19\n',
 'info': {'metrics': [{'id': '5caf9435eeb8baa519c5a8af',
    'name': 'training_loss'},
   {'id': '5caf9435eeb8baa519c5a8b1', 'name': 'training_acc'},
   {'id': '5caf943feeb8baa519c5a8e8', 'name': 'test_loss'},
   {'id': '5caf943feeb8baa519c5a8ea', 'name': 'test_acc'}]},
 'heartbeat': datetime.datetime(2019, 4, 11, 19, 23, 43, 148000),
 'result': 0.9315000176429749,
 'stop_time': datetime.datetime(2019, 4, 11, 19, 23, 43, 146000)}

However, the experiment object exposes all keys as attributes, so they can be conveniently accessed using dot notation.

In [11]:

exp.status

Out[11]:

'COMPLETED'

In [12]:

exp.start_time

Out[12]:

datetime.datetime(2019, 4, 11, 19, 23, 23, 890000)

In [13]:

exp.result

Out[13]:

0.9315000176429749

In [14]:

print(exp.captured_out)

INFO - example - Running command 'conduct'
INFO - example - Started run with ID "2"
Failed to detect content-type automatically for artifact /home/jarno/projects/incense/predictions_df.pickle.
Added text/csv as content-type of artifact /home/jarno/projects/incense/predictions.csv.
Added image/png as content-type of artifact /home/jarno/projects/incense/confusion_matrix.png.
Added application/pdf as content-type of artifact /home/jarno/projects/incense/confusion_matrix.pdf.
INFO - matplotlib.animation - MovieWriter.run: running command: ['ffmpeg', '-f', 'rawvideo', '-vcodec', 'rawvideo', '-s', '3840x2880', '-pix_fmt', 'rgba', '-r', '1', '-loglevel', 'quiet', '-i', 'pipe:', '-vcodec', 'h264', '-pix_fmt', 'yuv420p', '-y', 'accuracy_movie.mp4']
Added video/mp4 as content-type of artifact /home/jarno/projects/incense/accuracy_movie.mp4.
Added text/plain as content-type of artifact /home/jarno/projects/incense/history.txt.
Failed to detect content-type automatically for artifact /home/jarno/projects/incense/model.hdf5.
INFO - example - Result: 0.9315000176429749
INFO - example - Completed after 0:00:19

In [15]:

exp.config

Out[15]:

{'epochs': 3, 'optimizer': 'sgd', 'seed': 0}

This works down to deeper levels of the data model.

In [16]:

exp.config.epochs

Out[16]:

Alternatitvely, the classic dictionary access notation can still be used. This is useful, if the the keys of the data model are not valid python identifiers.

In [17]:

exp.meta.options['--unobserved']

Out[17]:

False

Artifacts¶

.artifacts is a dict that maps from artifact names to artifact objects. The artifacts can rendered according to their type by calling .render() on them. They can be saved locally by calling .save() on them. The artifact dict might be empty if the run was just restarted and did not yet finish an epoch.

In [18]:

exp.artifacts

Out[18]:

{'predictions_df': Artifact(name=predictions_df),
 'predictions': CSVArtifact(name=predictions),
 'confusion_matrix': ImageArtifact(name=confusion_matrix),
 'confusion_matrix.pdf': PDFArtifact(name=confusion_matrix.pdf),
 'accuracy_movie': MP4Artifact(name=accuracy_movie),
 'history': Artifact(name=history),
 'model.hdf5': Artifact(name=model.hdf5)}

PNG artifacts will be shown as figures by default.

In [19]:

exp.artifacts['confusion_matrix'].render()

Out[19]:

In [20]:

exp.artifacts['confusion_matrix'].save()

While CSV artifacts will be converted into pandas.DataFrames.

In [21]:

exp.artifacts['predictions'].render().head()

Out[21]:

	predictions	targets
0	7	7
1	2	2
2	1	1
3	0	0
4	4	4

In [22]:

exp.artifacts['predictions'].show().head()

/home/jarno/.miniconda/envs/incense-dev/lib/python3.6/site-packages/ipykernel_launcher.py:1: DeprecationWarning: `show` is deprecated in favor of `render` and will removed in a future release.
  """Entry point for launching an IPython kernel.

Out[22]:

	predictions	targets
0	7	7
1	2	2
2	1	1
3	0	0
4	4	4

MP4 artifacts will be downloaded and embedded as an HTML element in the notebook. This can be useful for visualizing dynamics over time.

In [23]:

exp.artifacts['accuracy_movie'].render()

Out[23]:

Finally pickle artifacts will the restored to the Python object they originally represented. However, since pickle does not have a proper detectable content-type they will be only recognized as Artifacts without any more specific type. We can use the as_type method to interpret an artifact as an artifact of a more specific or just different type. In our example we just saved the data frame we already have as CSV as a pickle file as well.

In [24]:

pickle_artifact = exp.artifacts['predictions_df'].as_type(incense.artifact.PickleArtifact)
pickle_artifact.render().head()

Out[24]:

	predictions	targets
0	7	7
1	2	2
2	1	1
3	0	0
4	4	4

Metrics¶

.metrics works similiar to .artifacts, but maps from metrics names to pandas.Series. Therefore, metrics can easily be plotted.

In [25]:

exp.metrics.keys()

Out[25]:

dict_keys(['training_loss', 'training_acc', 'test_loss', 'test_acc'])

In [26]:

exp.metrics['training_loss'].plot()

Out[26]:

<matplotlib.axes._subplots.AxesSubplot at 0x7f93c14c6748>

In [27]:

exp.metrics['training_acc'].plot()
exp.metrics['training_loss'].plot()
plt.legend()

Out[27]:

<matplotlib.legend.Legend at 0x7f93c0418668>

Projecting onto DataFrames¶

Often you want to pull experiment attributes and metrics into a dataframe. Either just to get and overview or do a custom analysis. You can easily transform a QuerySet of experiments by calling project on it. Pass a list of dot separated paths that point to some value in the experiement model to the on parameter. By default the columns will be named as the last element in the path.

In [28]:

exps = loader.find_by_ids([1,2,3])
exps.project(on=["experiment.name", "config.optimizer", "config.epochs"])

Out[28]:

	name	optimizer	epochs
exp_id
1	example	sgd	1
2	example	sgd	3
3	example	adam	1

If a path points to a value that is non-scalar, e.g. a metric, you can pass a dict of the path mapping to a function that reduces the the values to a single value.

In [29]:

exps.project(on=["experiment.name", "config.optimizer", {"metrics.training_loss": np.median}])

Out[29]:

	name	optimizer	training_loss_median
exp_id
1	example	sgd	0.637839
2	example	sgd	0.345012
3	example	adam	0.218707

Saving mulitple artifacts at once¶

QuerySets mimick the API of single artifacts, so you can also get the artifacts and save all of them. This has the advantage that the download will happen in a multithreaded fashion, which should make things faster for large number of bigger artifacts.

In [30]:

loader.find_all().artifacts["confusion_matrix"].save(to_dir="artifacts")

To match more than one artifact per experiment you can use globbing patterns to filter the artifacts. However, you will not get an error if no artifacts are matched for different experiments.

In [31]:

(loader
 .find_all()
 .artifacts
 .filter("*matrix*")
 .save(to_dir="artifacts"))

Utils¶

The utils module contains realted functionality, that might be useful during the manual interpretation of experiments.

In [32]:

from incense import utils

The find_differing_config_keys function returns the set of config values that differ in a set of experiments.

In [33]:

exps = loader.find_by_ids([1, 2])
utils.find_differing_config_keys(exps)

Out[33]:

{'epochs'}

Deleting experiments¶

It is possible to completely delete experiments including their associated metrics and artifacts. Per default the method will ask for confirmation, so we do not accidentially delete our experiments. This can be skipped by passing confirmed=True.

In [34]:

exp = loader.find_by_id(2)
exp.delete()

---------------------------------------------------------------------------
StdinNotImplementedError                  Traceback (most recent call last)
<ipython-input-34-1005ee59ea02> in <module>
      1 exp = loader.find_by_id(2)
----> 2 exp.delete()

~/projects/incense/incense/experiment.py in delete(self, confirmed)
     76         """
     77         if not confirmed:
---> 78             confirmed = input(f"Are you sure you want to delete {self}? [y/N]") == "y"
     79         if confirmed:
     80             self._delete()

~/.miniconda/envs/incense-dev/lib/python3.6/site-packages/ipykernel/kernelbase.py in raw_input(self, prompt)
    846         if not self._allow_stdin:
    847             raise StdinNotImplementedError(
--> 848                 "raw_input was called, but this frontend does not support input requests."
    849             )
    850         return self._input_request(str(prompt),

StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

In [ ]: