Anomaly Detection with Azure Stream Analytics
Published: 2020-02-09 13:36:56

In this post I’ll show you how to implement anomaly detection in your live data stream and how to visualize this in Power BI.
I used a MXChip as sensor and let it send temperature and humidity data, but you’re free to use any other device to send the data.

![](https://cdn-images-1.medium.com/max/2288/1*NSb4AaVOyMKQEAMG13jMuw.png)

## Prerequisites

• Azure Subscription
• Device with sensors to send temperature data to Azure
• Power BI account

## Setup Azure resources

### IoT Hub

Azure IoT Hub provides a cloud-hosted solution backend to connect virtually any device. In this case we use the IoT Hub for receiving the sensor data from the MXChip.

I’ll use Azure Cloud Shell to create the IoT Hub, but you can also use the [Azure portal](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal).
To create the IoT Hub use the follow CLI commands.

    az group create -l westeurope -n rg-anonmalydetection

    az iot hub create --name anonmalydetection-hub --resource-group rg-anonmalydetection --sku F1 --partition-count 2

The first command creates a resource group (in this case I gave it the name rg-anonmalydetection) and the second command creates the IoT Hub (free pricing tier).

*If you want more information on creating an IoT Hub using the Cloud Shell you can look [here](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-create-using-cli).*

### Stream Analytics

I’ll create the Stream Analytics job using the portal. This gives the best overview of what happens.

Follow these steps to create the Stream Analytics job:

1. Go to the [Azure portal](https://portal.azure.com) and login

1. Open the same resource group as where you created the IoT Hub

1. Use the ‘Add’ button to create a new resource. Search for ‘Stream Analytics job’. Click ‘Create’.

1. Give the job a name and again click ‘Create’ (the other fields can keep their default value).

Now we need to prepare the stream analytics job to process the data and do the anomaly detection.

First create an input, so we receive the data from our IoT Hub. To do this:

1. Open the stream analytics job and navigate to ‘Inputs’.

1. Add a stream input using. Select IoT Hub.

1. Give the input an alias eg ‘mxchip’, select the IoT Hub you created earlier and select ‘service’ as policy.

1. Click ‘Save’ to create the input.

<img src="https://cdn-images-1.medium.com/max/2000/1*4-TH75PpUkk7i3CcUZddvA.png" width="400" />

Now the input is created we’ll create an output. This is the destination of the data processed by the analytics job. In this case we’re going to stream the data to Power BI. To do this:

1. Open the stream analytics job and navigate to ‘outputs’.

1. Add an output and select Power BI.

1. Authorize with Power BI and configure the output settings.

<img src="https://cdn-images-1.medium.com/max/2000/1*dT5c5qoIBwAS7dGG5hlxJQ.png" width="400" />

*Stream Analytics will create a streaming dataset using this name in your Power BI workspace. The job must be running and have processed at least 1 event for the dataset to be created.*

Next and final step in configuring the analytics job is to write a query which selects the input data, does the anomaly detection and outputs it to Power BI. To do this follow these steps:

1. Open the stream analytics job and navigate to ‘Query’.

1. Edit the query and use the following query:
```
    WITH AnomalyDetectionStep AS
    (
    SELECT
    EVENTENQUEUEDUTCTIME AS time,
    CAST(temperature AS float) AS temp,
    AnomalyDetection_SpikeAndDip(CAST(temperature AS float), 99, 120, 'spikesanddips') OVER(LIMIT DURATION(second, 120)) AS SpikeAndDipScores
    FROM mxchip
    where temperature is not null
    )

    SELECT
    time,
    temp,
    CAST(GetRecordPropertyValue(SpikeAndDipScores, 'Score') AS float) AS Score,
    CAST(GetRecordPropertyValue(SpikeAndDipScores, 'IsAnomaly') AS bigint) AS IsAnomaly
    INTO powerbi
    FROM AnomalyDetectionStep
```
This query uses the input data send by the mxchip to the IoT Hub and finally sends it to Power BI. This also does the anomaly detection using the build in function ‘[AnomalyDetection_SpikeAndDip](https://docs.microsoft.com/en-us/stream-analytics-query/anomalydetection-spikeanddip-azure-stream-analytics)’.

This completes the Stream Analytics job configuration, so start the job.

***To avoid unnecessary costs stop the job when you’re done playing around.***

## Connect the device

Follow the ‘Getting started’ tutorial to setup your MXChip:
[https://microsoft.github.io/azure-iot-developer-kit/docs/get-started/](https://microsoft.github.io/azure-iot-developer-kit/docs/get-started/)

The ‘Get Started’ sample project used in this tutorial sends the data we want (temperature). Make sure to connect the device to the IoT Hub you created in the previous step.

If you go to your IoT Hub in the Azure portal you should now see device to cloud messages.

## Power BI

Now the data stream is setup we’re going to visualize the result in Power BI.

Logon to [Power BI](https://app.powerbi.com/). Use the same credentials as provided in the output of the Stream Analytics job.
Go to configured workspace and under datasets you should see your dataset containing the streaming data.

![](https://cdn-images-1.medium.com/max/4450/1*9Ri3Wug8V1hO3IYJso9foQ.png)

To visualize real-time data we need to create a Dashboard and add ‘Custom Streaming Data’ tiles to the dashboard. To show the result of the sensor data and also the result of the anomaly detection we’ll add 3 tiles (add them one by one) to the dashboard.

![](https://cdn-images-1.medium.com/max/4412/1*5ybxUa4yIO9rp_QV_Tf_ag.png)

Go to the next step select your streaming dataset en proceed to the next step. In this step you can configure the visualization (see image below).

  <img src="https://cdn-images-1.medium.com/max/2092/1*YdJI1ZayweDsIAFZsRMgqw.png" width="300" />
  <img src="https://cdn-images-1.medium.com/max/2170/1*ueidWdekYOxCG4WnR4i1vQ.png" width="300" /> 
  <img src="https://cdn-images-1.medium.com/max/2162/1*RoJOoQd91xp35q3PRwcAjg.png" width="300" />

If your data is still streaming you should now see the result. Try to trigger an anomaly by heating the sensor for a short time. When you did this the anomaly will be detected and a spike appears in the anomaly line chart.
The last chart will show the score of the anomaly function used in the query in the Stream Analytics job.

![](https://cdn-images-1.medium.com/max/3088/1*5dn4e2bU-7njMc3v7PP14Q.png)