Streaming Azure Data Pipelines into real-time dashboards
Published: 2023-02-01 15:56:47

Sometimes organizations are required to respond to events in  real-time, allowing them to detect and respond to changes such as security threats, customer-impacting operational issues, business KPIs, system monitoring, etc. Using real-time dashboards can help them make data-driven decisions fast.

**What is a real-time dashboard?**
A real-time dashboard is a type of visualization that is automatically updated with the most current data available.

**Which data sources can be used for real-time dashboards?**
The sources of streaming data can be IOT sensors, social media posts, monitoring metrics, or any other time-sensitive data being collected.

**Which technology can process streaming events?**
There are many technologies that allow producing and consuming streams of events, such as Apache Kafka, Spark and others. 
We will be using Azure technologies that will help us hit the road running, in no-time.

In this post I will show you how to create synthetic data, stream it using Azure Events hub, process it with Azure Stream Analytics and showcase it in Power BI in a live dashboard using "streaming datasets".

## **Let's start**
**What are we building?**

We will:
- Run a python script creating synthetic events and sending them to Azure Events hub
- Use Azure Stream Analytics to consume the events from Events hub and sending them to Power BI streaming dataset
- Visualize in Power BI the events in real time with a dashboard.

**Here is the architecture** 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mgqcg3ebw5ylfw7ktbdd.png)

**The recipe**
I assume you already have an azure account [(click here if you don't)](https://azure.microsoft.com/en-us/free/search/?&ef_id=CjwKCAiAuOieBhAIEiwAgjCvcjllhUzymVpLAwj0UYrCWrW7qTfm1MbvvwujlSC0FqAHaY5qkQxTWRoCDU0QAvD_BwE:G:s&OCID=AIDcmm81syc84i_SEM_CjwKCAiAuOieBhAIEiwAgjCvcjllhUzymVpLAwj0UYrCWrW7qTfm1MbvvwujlSC0FqAHaY5qkQxTWRoCDU0QAvD_BwE:G:s&gclid=CjwKCAiAuOieBhAIEiwAgjCvcjllhUzymVpLAwj0UYrCWrW7qTfm1MbvvwujlSC0FqAHaY5qkQxTWRoCDU0QAvD_BwE)

**Create an events hub**
In the azure portal create an Events Hub 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ikwjrhr9wp4j4wi48nhz.png)

**Click on Create**
**Fill all the relevant fields**
**Click "review and create"**
**Click "create"**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/be2uwq8aduhvy8w6lrnj.png)

**Go to your newly created events hub**
**Create a Shared Access Policy**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pmca6kypdy43bope3xqr.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/13ou4ceul6m7dt5bfu41.png)

**Copy the Shared Access Policy primary key and the events hub name and paste it in the python script you downloaded from [github](https://github.com/denisa-ms/public-snippets.git)**

```
import datetime
import time
import random
import socket
import struct
import json
from azure.eventhub import EventHubProducerClient, EventData
import psutil
import pyodbc
import pandas as pd

eventHubConnString = "<paste here the events hub shared access signature primary key>"
eventHubName = "<paste here the events hub name>"
producer = EventHubProducerClient.from_connection_string(conn_str=eventHubConnString, eventhub_name=eventHubName)
hostname = socket.gethostname()

def generateRandomIP():
    ip = socket.inet_ntoa(struct.pack('>I', random.randint(1, 0xffffffff)))
    return ip

def getRandomBrowser():
    browsersList = ["Edge", "Chrome", "Safari", "Firefox"]
    browser = random.choice(browsersList)
    return browser

def getRandomMethod():
    methods = [ "GET", "POST", "PUT", "DELETE" ]
    method = random.choice(methods)
    return method

def getRandomURL():
    urls = ["/page1.html", "/page2.html", "/page3.html", "/page4.html", "/page5.html" ,"/page6.html"  ]
    url = random.choice(urls)
    return url

def getCurrentDate():
    today = datetime.datetime.now()
    current = today.strftime("%Y-%m-%d")
    return current

def getCurrentTime():
    today = datetime.datetime.now()
    current = today.strftime("%H:%M:%S")
    return current

def getRandomTemperature():
    return random.randint(0,99)

def generateClickEvent():
    click_event = {
        "tableName" : "clicks",
        "date" : getCurrentDate(),
        "time": getCurrentTime(),
        "ip": generateRandomIP(),
        "browser": getRandomBrowser(),
        "method": getRandomMethod(),
        "url" : getRandomURL(),
        "productID": getRandomProductId()
    }
    return click_event

def getRandomProductId():
    return str(random.randint(0,99999))

def sendToEventsHub(jsonEvent):
    eventString = json.dumps(jsonEvent)
    print(eventString) 

    event_data_batch = producer.create_batch() 
    event_data_batch.add(EventData(eventString)) 
    producer.send_batch(event_data_batch)

def generateClicks():
    try:
        while True:
            clickEvent = generateClickEvent()    
            sendToEventsHub(clickEvent)
            # time.sleep(10)

    except KeyboardInterrupt:
        producer.close()
        conn.close()


if __name__ == "__main__":
    generateClicks()
```
**Create a stream analytics job**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/si5g3bzcxb7z2o9vrqjo.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qwufu7mqvg33vsq1ak83.png)

**Fill the details and click review and create and then Click create**
**Click on Inputs - Add stream input**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gr6t1ibx9s57rhfm3nen.png)

**Choose your subscription, the events hub we created above, and settings as shown below**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2n85gygz7zvxe642i15r.png)

**Click Outputs and add a Power BI output**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/udxy4quyvhz7faw2wnis.png)

**Fill all the details as shown below**
Please notice that this will authorize the stream analytics job to access your power BI account (I recommend for demo purposes to use the same account as the one logged in into the Azure Portal.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6wx00j2nw560om9vncod.png)


**Click on Query**
Copy and paste the query snippet below
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/736xfkkjxkdnasib8oh3.png)

```
SELECT 
    DateAdd(s,-3,System.TimeStamp) AS windowstart,
    System.TimeStamp AS windowend,
    browser,
COUNT(*) AS entries
INTO
[powerbi]
FROM
[bddemo-clicks] TIMESTAMP BY EventProcessedUtcTime
GROUP BY browser, TumblingWindow(s, 3)
```

**Save the query**
**Create a real time dashboard in Power BI**
**Login to power BI with the same credentials you provided the stream analytics job above**
**Create a dashboard**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mj632l4e1u72xev98htf.png)
**Add a tile to your dashboard**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5en4maricdwjm9tqqymc.png)

**Click on Custom Streaming Data**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2bbnsowpsku1oua8j9y0.png)

**Choose the stream we created in the stream analytics job**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cwsynsvukvvnepx4k7wy.png)

**Choose the visualization type**
We will choose the Line chart and fill the details 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1btvt3ybscteeyi59sxr.png)

**Choose a tile name and click Apply**

## We are done preparing everything!

## Let's run our demo
**Run the python script provided**
Make sure you are authenticated to run your local code and connect to Azure [(if not click here and install powershell)](https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.3.0)

Now our code is running and sending events to the events hub 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ojltla897kp1x91js1xn.png)

**Start the stream analytics job**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7tyeswwptwq7hdzxggjr.png)

**After 1-2 minutes the job status will be "running" and the events will start flowing**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kzmd6pw3l6bsqfxe996j.png)

**In Power BI you will see the events flowing in your dashboard**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x6qn0393nkxqrju03cyy.png)

**MAKE SURE to shut down your script AND STOP the stream analytics job to avoid unnecessary costs.**


That's all folks!

I hope you enjoyed this tutorial.
Cheers!
Denise


Photo by <a href="https://unsplash.com/@jachan_devol?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Jachan DeVol</a> on <a href="https://unsplash.com/s/photos/stream?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>