CSV Analysis Example¶

This is a demo notebook showing how to use the orion.analysis.analyze function on a CSV signal.

1. Load the data¶

In the first step, we setup the environment and load the CSV that we want to process.

To do so, we need to import the orion.data.load_signal function and call it passing the path to the CSV file.

In this case, we will be loading the S-1.csv file from inside the data folder.

In [1]:

import logging;

logging.basicConfig(level=logging.ERROR)
logging.getLogger().setLevel(level=logging.ERROR)

import warnings
warnings.simplefilter("ignore")

In [2]:

from orion.data import load_signal

signal_path = 'S-1'

data = load_signal(signal_path)
data.head()

Out[2]:

	timestamp	value
0	1222819200	-0.366359
1	1222840800	-0.394108
2	1222862400	0.403625
3	1222884000	-0.362759
4	1222905600	-0.370746

2. Detect anomalies using a pipeline¶

Once we have the data, let us try to use the LSTM pipeline to analyze it and search for anomalies.

In order to do so, we will have import the orion.analysis.analyze function and pass it the loaded data and the path to the pipeline JSON that we want to use.

In this case, we will be using the lstm_dynamic_threshold.json pipeline from inside the orion folder.

The output will be a pandas.DataFrame containing a table with the detected anomalies.

In [3]:

from orion.analysis import analyze

hyperparameters = {
    'keras.Sequential.LSTMTimeSeriesRegressor#1': {
        'epochs': 5,
        'verbose': True
    }
}

pipeline = 'lstm_dynamic_threshold'

anomalies = analyze(pipeline, data, hyperparams=hyperparameters)
anomalies

Using TensorFlow backend.

Train on 7919 samples, validate on 1980 samples
Epoch 1/5
7919/7919 [==============================] - 35s 4ms/step - loss: 0.1959 - mse: 0.1959 - val_loss: 0.3439 - val_mse: 0.3439
Epoch 2/5
7919/7919 [==============================] - 34s 4ms/step - loss: 0.1915 - mse: 0.1915 - val_loss: 0.3147 - val_mse: 0.3147
Epoch 3/5
7919/7919 [==============================] - 34s 4ms/step - loss: 0.1909 - mse: 0.1909 - val_loss: 0.3534 - val_mse: 0.3534
Epoch 4/5
7919/7919 [==============================] - 34s 4ms/step - loss: 0.1873 - mse: 0.1873 - val_loss: 0.2893 - val_mse: 0.2893
Epoch 5/5
7919/7919 [==============================] - 34s 4ms/step - loss: 0.1891 - mse: 0.1891 - val_loss: 0.3037 - val_mse: 0.3037
9899/9899 [==============================] - 13s 1ms/step
9899/9899 [==============================] - 12s 1ms/step

Out[3]:

	start	end	score
0	1228219200	1229472000	0.656399
1	1400587200	1404172800	0.166991