Find anomalies with spike detection and ML.NET
Tags: dotnet, csharp, machinelearning, mlnet
Published: 2021-04-21 11:20:25

---
title: Find anomalies with spike detection and ML.NET
published: true
description: A real-life example of how to find anomalies in time-series data using spike detection available in ML.NET. Learn how to utilize the Microsoft.ML package.
tags: dotnet, csharp, machinelearning, mlnet
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/77obar9o5qtkcjfonj9b.png
canonical_url: https://blog.elmah.io/find-anomalies-with-spike-detection-and-ml-net/
---

I recently started experimenting with machine learning on [elmah.io](https://elmah.io). I chose ML.NET as the framework and I'm pretty happy with the results so far. The amount of documentation is good but mostly limited to flower detection samples (using the frequently used Iris data set). In this post, I'll share how we implemented anomaly/spike detection and which problems we ran into.

> This is not an introduction post to machine learning. There are a lot of good resources out there, but I'll try to make sure that everyone can follow along.

There is a range of sub-areas within machine learning. For the rest of this post, I'll focus on anomaly/spike detection exclusively. So what is anomaly/spike detection all about?

Being able to log errors to a service like elmah.io is a great first step for all .NET developers. The next step is to get a good overview of the errors your applications are generating. This can be quite challenging. When you have a few errors generated every day, it's easy. But the complexity typically increases by the number of applications and lines of code you configure to log errors. anomaly/spike detection is a great way to help you with this process.

By letting machine learning crunch through your data, you can get a visual indication of a spike. A spike being something out of the ordinary (an anomaly). Looking at anomalies is a great way to identify when new errors are introduced and to help you classify the severity of different errors. Let's look at a real-life example from one of our logs:

![](https://blog.elmah.io/content/images/2019/12/anomaly-detection.png)

As seen on the chart, two bars are marked with a red background color which is our way to indicate an anomaly visually. If you look at the error count before 10/19/2019 they are limited to only a few instances per day, while we see a sudden spike on that day. We did introduce a bug that day which was fixed pretty quickly. The system then ran error-free for two days and a new bug was introduced (dough!). Again, the bug is indicated by a new spike on 10/22/2019.

By looking at the chart and the description above, I hope that it is clear for everyone what a spike is and why it can be called an anomaly.

You may have heard about model training in machine learning and ML.NET. When I started diving into ML.NET, training was the first step in most of the blog posts I read. Luckily, spike detection doesn't require any training, why the code is extremely simple. In short, you hand a time series to ML.NET and it outputs a list of predictions.

No more explaining. Let's look at some code! Start by installing the following NuGet packages:

```
Install-Package Microsoft.ML
Install-Package Microsoft.ML.TimeSeries
```

The `Microsoft.ML` package contains all of the core code and interfaces to use ML.NET. `Microsoft.ML.TimeSeries` contains the prediction engine for time-series data. To use ML.NET, you will need to create a new instance of `MLContext`. The context is similar to contexts known from Entity Framework.

```csharp
var mlContext = new MLContext();
```

As a quick test, we'll generate an input that contains a spike:

```csharp
var counts = new[] { 0, 1, 1, 0, 2, 1, 0, 0, 1, 1, 50, 0, 1, 0, 2, 1, 0, 1 };
```

See the spike? The value `50` stands out. Let's see if ML.NET agrees. We will need a strongly typed object as input to ML.NET:

```csharp
class Input
{
    public float Count { get; set; }
}
```

Like the input, ML.NET needs a strongly typed output class:

```csharp
class Output
{
    [VectorType(3)]
    public double[] Prediction { get; set; }
}
```

The `Output` will include the prediction in a `double` array. Looking at the output for the first time, the code looks weird. How can an array of doubles with a `VectorType` attribute of 3 elements contain a prediction from a machine learning framework? Keep reading and we'll see in a minute.

Next, we will need an estimator. An estimator is an object that can be used to train a machine learning model (not interesting in this sample) and get a transformer needed to identify spikes:

```csharp
var estimator = mlContext.Transforms.DetectIidSpike(
    nameof(Output.Prediction),
    nameof(Input.Count),
    confidence:99,
    pvalueHistoryLength:counts.Length / 4);
```

We call the `DetectIidSpike`-method part of the `Microsoft.ML.TimeSeries` package. The method takes 4 parameters. The names of the input and output properties that I already showed you, as well as two integers. `confidence` needs to be an integer in the range from `0` to `100`. The higher the value, the more confident ML.NET needs to be if a value is a spike or not to output it as one. `pvalueHistoryLength` is somewhat of a magic value to me and I probably need to study machine learning more to fully understand this parameter (hey, at least I'm honest here 😂). Most spike detection tutorials set the value of `pvalueHistoryLength` to one-fourth of the input length, why I'll do the same for now.

Finally, time to get the transformer:

```csharp
ITransformer transformer = estimator.Fit(mlContext.Data.LoadFromEnumerable(new List<Input>()));
```

The transformer is acquired by calling the `Fit`-method on the estimator that we fetched in a previous step. Notice how we send an empty list of `Input` objects to the method? The parameter for the `Fit`-method is used for training, which we don't need for spike detection.

With the transformer in hand, we can transform the input data to get the predictions from ML.NET:

```csharp
var input = counts.Select(x => new Input { Count = x });
IDataView transformedData = transformer.Transform(mlContext.Data.LoadFromEnumerable(input));
```

The `Transform`-method expects an enumerable of objects conforming to the "schema" that we already gave ML.NET (`nameof(Input.Count)`). I'm using the `LoadFromEnumerable`-method once again, but this time with the list of integers instead of an empty list. If you're coding along, this is the input data you want a prediction on.

Only a single step left to get the predictions in a readable format:

```csharp
var predictions = mlContext.Data.CreateEnumerable<Output>(transformedData, false);
```

In this step, I convert the output from `Transform` to an enumerable of `Output` objects as declared in a previous step. With this in hand, we can output the predictions:

```csharp
foreach (var p in predictions)
{
    Console.WriteLine($"{p.Prediction[0]}\t{p.Prediction[1]}\t{p.Prediction[2]}");
}
```

Remember the double array and the `VectorType` on the `Output` class? Here you see the output of that. Each entry in the predictions enumerable is an array of three doubles:

```
0       0       0,5
0       1       0,15865526383236372
0       1       0,2613750311936282
0       0       0,17796491136567605
0       2       0,011390874461660316
0       1       0,49999999975
0       0       0,16490956024045023
0       0       0,2804576814507901
0       1       0,4034750154990393
0       2       0,011390874461660316
1       50      1E-08
0       0       0,36323554577142725
0       1       0,377624053546577
0       0       0,36323554577142725
0       2       0,4013610122501119
0       1       0,4034750154990393
0       0       0,16490956024045023
0       1       0,4034750154990393
```

The first column is a bit indicating if the entry is a spike or not. `0` means no and `1` means yes. The second column is the original input. And the third and final column is a calculated confidence value. The lower the number, the more likeable the input is a spike. When looking at the output we see the value of `50` being the only spike.

Here's the full code so far:

```csharp
var counts = new[] { 0, 1, 1, 0, 2, 1, 0, 0, 1, 1, 50, 0, 1, 0, 2, 1, 0, 1 };

var mlContext = new MLContext();
var estimator = mlContext.Transforms.DetectIidSpike(nameof(Output.Prediction), nameof(Input.Count), confidence:99, pvalueHistoryLength:counts.Length / 4);
ITransformer transformer = estimator.Fit(mlContext.Data.LoadFromEnumerable(new List<Input>()));
var input = counts.Select(x => new Input { Count = x });
IDataView transformedData = transformer.Transform(mlContext.Data.LoadFromEnumerable(input));
var predictions = mlContext.Data.CreateEnumerable<Output>(transformedData, false);

foreach (var p in predictions)
{
    Console.WriteLine($"{p.Prediction[0]}\t{p.Prediction[1]}\t{p.Prediction[2]}");
}
```

At first glance, the code may look a bit over-engineered. But remember that ML.NET can be used for multiple purposes and the flow is pretty much the same if you train a model or not.

Let's play with the settings. Remember the `confidence` parameter with the value of `99`? By specifying such a high value, ML.NET needs a very high confidence when making the judgement call whether an input is a spike or not. Let's change it to `95` and see what happens:

```csharp
var estimator = mlContext.Transforms.DetectIidSpike(
    nameof(Output.Prediction),
    nameof(Input.Count),
    confidence:95,
    pvalueHistoryLength:counts.Length / 4);
```

With the same input, the predictions now looks like this:

```
0       0       0,5
0       1       0,15865526383236372
0       1       0,2613750311936282
0       0       0,17796491136567605
1       2       0,011390874461660316
0       1       0,49999999975
0       0       0,16490956024045023
0       0       0,2804576814507901
0       1       0,4034750154990393
0       1       0,2613750311936282
1       50      1E-08
0       0       0,368076688458975
0       1       0,38239457274913025
0       0       0,368076688458975
0       2       0,4013610122501119
0       1       0,4034750154990393
0       0       0,16490956024045023
0       1       0,4034750154990393
```

Notice the difference? The fifth element (value of `2`) is now marked as a spike too.

Which value to use for `confidence` is totally up to your case and input value. Play around with different values and be the judge of what produces the best results for you.

One thing I came across while implementing spike detection for elmah.io is what may or may not be a bug. Let's change the input values to the following:

```csharp
var counts = new[] { 1, 3, 0, 4, 5, 5, 4, 3, 3, 0, 13, 8, 1, 61, 21, 40, 7, 7, 5, 6, 8, 33, 11, 5,
    2, 10, 11, 18, 14, 23, 8, 17, 15, 13, 24, 29, 15, 20, 29, 19, 18, 17, 23, 47, 7, 14, 26, 28,
    5, 22, 47, 22, 20, 9, 40, 6, 8, 4, 10, 10, 1, 4, 27, 3, 3, 7, 6, 12, 8, 3, 1, 2, 0, 0, 2, 0,
    2, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 2 };
```

Spotting anomalies manually and without the help of ML.NET is now a whole lot harder. Here's part of the result:

```
0       1       0,5
0       3       0,022750062887256395
0       0       0,08000261555683919
0       4       0,07336260731420347
0       5       0,09324626262574676
0       5       0,19087154278882695
0       4       0,37936589390701053
0       3       0,4522491125423572
0       3       0,4543375551070543
0       0       0,09656461688070728
0       13      7,231836548493575E-07
0       8       0,16279622550878553
0       1       0,26866445055948984
0       61      1E-08
0       21      0,19532181235194518
0       40      0,07628985932177507
0       7       0,4814184562065965
0       7       0,4835626729623246
0       5       0,44158539201429914
0       6       0,467300428950856
0       8       0,48429199889094554
0       33      0,11185621221616127
0       11      0,4415187627738487
0       5       0,4219793790334213
0       2       0,34845080975673626
0       10      0,4668402175020555
0       11      0,45435592220182563
0       18      0,3164680969641882
0       14      0,4109081873786849
0       23      0,25204845788437136
0       8       0,4222073383577716
0       17      0,38281677839584366
0       15      0,4397732642572547
0       13      0,49932909372608614
0       24      0,26998351731163683
0       29      0,2086753725973
0       15      0,4965672805299245
0       20      0,32702008169487357
0       29      0,14736951748986427
0       19      0,3341649373312337
0       18      0,3817593360276714
0       17      0,4308978322639091
0       23      0,25778090864274994
1       47      0,0025041522676845784
```

The last line with the value of `47` is marked as a spike. When looking at the previous counts and the pattern in the data, `47` definitely looks like an anomaly. But take a look at line number 14. With value from `0` to `13` before it, `61` definitely looks like an anomaly as well. ML.NET seems to have a problem with confidence values lower than `0.05`. StackOverflow suggested this quickfix:

```csharp
if (p.Prediction[2] < (1 - 0.95))
{
    p.Prediction[0] = 1;
}
```

It's a bit of a hack, but it gets the job done:

```
0       1       0,5
1       3       0,022750062887256395
0       0       0,08000261555683919
0       4       0,07336260731420347
0       5       0,09324626262574676
0       5       0,19087154278882695
0       4       0,37936589390701053
0       3       0,4522491125423572
0       3       0,4543375551070543
0       0       0,09656461688070728
1       13      7,231836548493575E-07
0       8       0,16279622550878553
0       1       0,26866445055948984
1       61      1E-08
```

The downside here is that more input values are marked as spikes. I hope to be able to come up with a better fix or for Microsoft to make this work properly.

## Would your users appreciate fewer errors?

elmah.io is the easy error logging and uptime monitoring service for .NET. Take back control of your errors with support for all .NET web and logging frameworks.

➡️ [Error Monitoring for .NET Web Applications](https://elmah.io/?utm_source=devto&utm_medium=social&utm_campaign=devtoposts) ⬅️

This article first appeared on the elmah.io blog at https://blog.elmah.io/find-anomalies-with-spike-detection-and-ml-net/