Orion Tulog¶

This is a tutorial for anomaly detection using Orion. Orion is a python package for time series anoamly detection. It provides a suite of both statistical and machine learning models that enable efficient anomaly detection.

In this tutorial, we will learn how to set up Orion, train a machine learning model, and perform anomaly detection. We will delve into each part seperately and then run the evaluation pipeline from beginning to end in order to compare multiple models against each other.

Part 1¶

In part one of the series, we explore a time series data, particularly the NYC taxi data. You can find the raw data on the TLC or the processed version maintained by Numenta here. We also explore what reasons could possibily be contributing to producing such anomalies.

Data Loading¶

There is a collection of data already available in Orion, to load them, we use the load_signal function and pass the name of the signal we wish to obtain. Similarly, since this data is labeled, we use the load_anomalies function to get the corresponding anomaly of the signal

Part 2¶

In part two of the series, we look at anomaly detection through time series reconstruction, particularly using a GAN model. We go through a sequence of transformations and data preparation, as well as model training and prediction.

Orion API¶

We will use Orion to perform these sequence of actions. We will be emphasizing the usage of the TadGAN model which is a time series anomaly detection using GANs model. The model is specified in a json format accompanied with this notebook named tadgan.json. There are more pipelines defined within the repository including: ARIMA, LSTM, etc.

The Orion API is a simple interface that allows you to interact with anomaly detection pipeline. To train the model on the data, we simply use the fit method; to do anomaly detection, we use the detect method. In our case, we want to fit the data and then perform detection; therefore we use the fit_detect method. This might take some time to run. Once it’s done, we can visualize the results.

Note: the model might take some time to train. For experimentation purposes, you can reduce the number of epochs in the tadgan.json file such that you reduce the number of training iterations.

Let's visualize the results.

The red intervals depict the detected anomalies, the green intervals show the ground truth. Cool! the model was able to detect some anomalies. We also see that it detected some other intervals that were not included in the ground truth labels. It is clear though, they are falling out of shape with respect to the remaining signal. Note: the results might differ between runs.

We might have jumped straight to the results but let's trace back and look at what the model actually did.

There is a series of transformations happening to the data in order to obtain the result you have just seen. From data preprocessing, model training, to post-processing functionalities. We specify these functions, which we refer to as primitives, within the model’s .json file. What are these primitives? If we were to look at the tadgan.json model, we find these sequential primitives:

"primitives": [ 
    "mlprimitives.custom.timeseries_preprocessing.time_segments_aggregate”,
    "sklearn.impute.SimpleImputer",
    "sklearn.preprocessing.MinMaxScaler",
    "mlprimitives.custom.timeseries_preprocessing.rolling_window_sequences",
    "orion.primitives.tadgan.TadGAN",
    "orion.primitives.tadgan.score_anomalies",
    "orion.primitives.timeseries_anomalies.find_anomalies"
]

Each primitive is responsible for a single task. We describe the procedure of each primitive in the remainder of this notebook

Data Preparation¶

A. data frequency¶

Adjust signal spacing to be of equal width across all times. There are two important parameters in this process:

• interval: an interger that refers to the time span to compute aggregation of.
• method: what aggregation method should be used to compute the value, by default this set to the mean.

In addition to, passing the array of values and which column holds the values we wish to alter the frequency of.

If we go back to the source of the NYC Taxi data, we find that it records a value each 30 minutes. In the timestamp world, this is equivalent to 1800 seconds, therefore we set the interval to be 1800. We also opt for the default aggregation method which is taking the mean value of each interval.

Technically speaking, in our example the data is perfectly spaced, so we can skip this preprocessing step. However, that is not always the case and so we include it as a preprocessing primitive in the general pipeline as you will see later on.

B. data imputation¶

impute missing values that appear within the signal using scikit-learn's SimpleImputer which fills missing values by the mean value.

C. data normalization¶

normalize the data between a specific range, we use scikit-learn's MinMaxScaler to scale data between [-1, 1].

Notice how the y-axis changed after normalizing the data between [-1, 1]

D. slice the data to rolling window¶

to prepare the data, we need to transform it into a sequence that is ingestable by the machine learning model. We take the signal we're interested in analyzing and we generate training examples. These training examples are mere snapshots of signal at different times.

In order to do that, we adopt the sliding window approach of choosing a window of a pre-specified width and a particular step size. Once that's been decided we divide the signal indo segments, similar to what is depicited in the illustration below.

We create a rolling_window_sequence function that slices the data into parts, each part contains:

• the target value; the value at time t.
• previous observed values, this is determined by the window width.

Where X represents the input used to train the model. In the previous example, we see X has 10222 training data points. Notice that 100 represents the window size. On the other hand, y is the real signal after processing, which we will use later on to calculate the error between the reconstructed and real signal.

Pipeline Training and Detection¶

The architecture of the model requires four neural networks:

• encoder: maps X to its latent representation Z.
• generator: maps the latent variable Z back to X, which we will denote later on as X_hat.
• criticX: discriminates between X and generator(Z) or X_hat.
• criticZ: discriminates between Z and encoder(X).

we detail the composition of each network in model.py.

To use the TadGAN model, we specify a number of parameters including the model layers (structure of the previously mentioned neural networks). We also specify the input dimensions, the number of epochs, the learning rate, etc. All the parameters are listed below.

To reassemble or “unroll” the predicted signal X_hat we can choose different aggregation methods (e.g., mean, max, etc). In our implementation, we chose it to as the median value.

We can see that the GAN model did really well in trying to reconstruct the signal. We also see how it expected the signal to be, in comparison to what it actually is. The discrepancies between the two signals will be used to calculate the error. The higher the error, the more likely it is an anomaly

Error Computation¶

In the TadGAN pipeline, we use tadgan.score_anomalies to perform error calculation for us. It is a smoothed error function that uses a window based method to smooth the curve then uses either: area, point difference, or dtw as a measure of discrepancy.

Area¶

This method captures the general shape of the orignal and reconstructed signal and then compares them together.

Point¶

This method applies a point-to-point comparison between the original and reconstructed signal. It is considered a strict approach that does not allow for many mistakes.

DTW¶

A more lenient method yet very effective is Dynamic Time Warping (DTW). It compares two signals together using any pair-wise distance measure but it allows for one signal to be lagging behind another.

Now we can visually see where the error reaches a substantially high value. But how should we decide if the error value determines a potential anomaly? We could use a fixed threshold that says if error > 10 then let’s classify the datapoint as anomalous.

While a fixed threshold raised some correct anomalies, it missed out on others. If we were to look back at the error plot, we notice that some deviations are abnormal within its local region. So how can we incorporate this information in our thresholding technique? We can use window based methods to detect anomalies with respect to their context.

We first define the window of errors, that we want to analyze. We then find the anomalous sequences in that window by looking at the mean and standard deviation of the errors in the window. We store the start/stop index pairs that correspond to each sequence, along with its score. We then move the window and repeat the procedure. Lastly, we combine overlapping or consecutive sequences.

Cool! We now have the same result as we saw previously. The red intervals depict the detected anomalies, the green intervals show the ground truth. We also see that it detected some other intervals that were not included in the ground truth labels.

Using the Orion API and pipelines, we simplified this process yet allowed flexibility for pipeline configuration.

End-to-End Pipeline Configuration¶

To configure a pipeline, we adjust the parameters of the primitive of interest within the pipeline.json file or directly by passing the dictionary to the API.

In the following example, I changed the aggregation level as well as the number of epochs for training. These changes will override the parameters specified in the .json file. To know more about the API usage and primitive designs, please refer to the documentation.

The anomalies detected in this run are a bit different from the earlier example. Although, it still succeeded in detecting anomalies. Maybe a 1 hour aggregate is not the appropriate value? Maybe we did not train the model enough times, or maybe too many times... How can we tell? One way is to look at the result of the model like we have done previously.

You can use the visualization parameter within detect to return intermediate outputs (primitive outputs) that we are interested in. For example in the tadgan.json file, use visualization to return the following variables:

• X: this is the output of the preprocessing steps from averaging, imputing, and scaling. These steps were showcased previously as steps (A, B, and C).
• X_hat: this is the "predicted" output by the TadGAN model without any processing. It represents the reconstructed window at each timepoint.
• es: this is the error calculated by capturing the discrepancies between original and reconstructed signal.

then we use anomalies, viz = orion.detect(df, visualization=True) where viz will be a dictionary of intermediate outputs.

Note: we will talk more about how to evaluate the detected anommalies with respect to the ground truth in part 3 of the tutorial.

Part 3¶

In part three of the series, we look at evaluation and end-to-end anomaly detection evaluation.

Evaluation¶

We compare the anomalies given to us as ground truth labels to the detected anomalies. But first, we look at some of the mechanisms we have for evaluation, namely:

• weighted segment: Assessing every datapoint in the detected anomalies with its counterpart in the ground truth.
• overlap segment: Assesses the detected anomaly segment by seeing if we caught an overlap with the correct anomalies.

We will look at both approaches, but first let's construct a dummy dataset.

Let's assume that the signal starts at timestamp 1, and ends at timestamp 20. We can then see that the ground truth contains three anomalies, namely (5, 8), (12, 13), and (17, 18), where (i, j) expresses the starting timestamp i and ending timestamp j.

We can also see that, we detected two anomalies, namely (5, 8) and (12, 15). So how can we compare both sets?

There are two approaches for comparing anomaly sets, as expressed earlier.

(1) Weighted Segment, a stricter method, it is valuable to use when you want to equalize the importance of detecting anomalies, and normal instances.

• It first segments the signal into partitions based on the ground truth and detected sequences.
• Then it makes a segment to segment comparison and records TP/FP/FN/TN accordingly.
• The overall score is weighted by the duration (size) of each segment.

Visually, this operation is summarized by the illustration below.

we can use orion.evaluation subpackage to compute multiple metrics using the weighted segment approach. For example to compute the accuracy, we use contextual_accuracy(..., weighted=True). There are other metrics available, for reference checkout the orion.evaluation documentation.

(2) Overlap Segment, a more lenient approach of evaluation. It takes the perspective of rewarding the system if it manages to alarm the user of a subset of an anomaly. More particularly, it records:

1. TP, if a ground truth segment overlaps with the detected segment.

2. FN, If the ground truth segment does not overlap any detected segments.

3. FP, If a detected segment does not overlap any labeled anomalous region.

This can be summarized by the illustration below.

Similarly, we can use the same metric functions, but this time we use the parameter weighted=False. Note: overlap segment approach, does not account for true negatives. Reason being, anomalies in time series data are rare and so "normal" instances will skew the value of the computed metric. Therefore, using this approach we cannot compute metrics such as the accuracy.

Pipeline evaluation end-to-end¶

We integrate the evaluation suite into the Orion API, such that you can evaluate the pipeline on a dataset (with its labels) end-to-end.

Following part 2 introduction to the Orion API, we can create an orion instance the use its evaluate functionality. We support the method with the following arguments:

• data, a pandas.DataFrame containing two columns: timestamp and value.
• truth, a pandas.DataFrame containing two columns: start timestamp and end timestamp of ground truth labels
• fit, a flag denoting whether to train the pipeline before evaluating it.

train_data, a pandas.DataFrame containing two columns: timestamp and value used to train the pipeline, if not given, the pipeline will be trained on data.

• metrics, a list of metrics used to evaluate the pipeline.

In the previous part we went through how to train a pipeline and use it for anomaly detection, the focus now is on defining metrics and evaluating the performance of the pipeline.

metrics is list of function names that compares a ground truth labels against detected labels and returns a metric value. We have seen some functions of that sort, such as contextual_accuracy and contextual_f1_score. To construct our metrics list we select some of the predefined metrics in Orion, such as:

• Accuracy
• F1 Score
• Precision
• Recall

By default, we use the weighted segment approach, you can override metrics defined by specifying aa new metrics dictitonary.