Serverless Pub/Sub using Azure EventGrid and Functions in Python
Tags: azure, python, eventgrid, serverless
Published: 2019-09-28 19:13:18

---
title: Serverless Pub/Sub using Azure EventGrid and Functions in Python
published: true
description: Pub/Sub using Azure Functions in Python with Event Grid
tags: Azure, Python, EventGrid, Serverless
cover_image: https://thepracticaldev.s3.amazonaws.com/i/0gl1xevb816fhd7i5xho.png
---

An event is the starting point of any ETL pipeline. So then the question becomes, how do we handle the event such that the proper orchestration and handling produce the desired outcome we are looking for?

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/zcax3xdr7m19p2q2igyh.png)

Let's walk through a "Hello, world!" example of a pub/sub pipeline using some of the tools the Azure platform provides. We will create two Functions using VS Code + the Azure plugin. The first will be a webhook that publishes an event to the Event Grid. The second Function will receive the message from Event Grid and log a response to the logs.

## 0. Prerequisites
- [Microsoft Azure Account](https://azure.microsoft.com)
- [Azure Functions Core Tools](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local)
- [Python 3.6.x](https://www.python.org/)
- [Visual Studio Code](https://code.visualstudio.com/)
- [Postman](https://www.getpostman.com/)

## 1. Hello World HTTP Azure Function

Install the Visual Studio Code plugin "Azure Functions". Once completed, an Azure icon will appear on your left-hand menu, click on it. Once in Azure Functions menu, click "Sign in to Azure...". After signing in, click the "Create New Project..." icon from the top menu.

Once you choose a location, make the following selections:
- _Select a language for your function project_
 - Python
- _Select a template for your project's first function_
 - HTTP trigger
- _Provide a function name_
 - publish-event
- _Authorization level_
 - Anonymous
- _Select how you would like to open your project_
 - Open in current window
- _Enter the alias or full path of the python "3.6x" executable to use_
 - python3

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/nwpvnk45jpnexy51jhvx.png)

You should be able to run the function locally pressing F5. Once it is up and running, in a web browser you should be able to navigate to:
- `http://localhost:7071/api/publish-event`
- `http://localhost:7071/api/publish-event?name=dev3l`

**NOTE**: At this point, I had some issues with Python 3.7 where the plugin was looking for 3.6.x. Refer to this [Stack Overflow](https://stackoverflow.com/questions/51726203/installing-python3-6-alongside-python3-7-on-mac) if you run into a similar issue, then recreate the function.

## 2. Hello World Event Grid Azure Function

From the Azure Functions menu, click the thunderbolt icon from the top menu to create a new function.

Make the following selections:
- _Select a template for your project's first function_
 - Azure Event Grid trigger
- _Provide a function name_
 - subscribe-event

We have two functions that run locally. I was trying to find an easy way to test the subscription event locally, but could not find any straightforward solutions. Let's have faith that it is running.

## 3. Push Up to Azure

From the Azure Functions menu, click the up arrow icon from the top menu.

Make the following selections:
- _Select Function App in Azure_
 - Create new Function App in Azure...
 - pub-sub-function
- _Select a location for new resources._
 - US East

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/qjtzpunj5o150bi2fpbx.png)

## 4. Test in Azure

Log in to [Azure Portal](https://portal.azure.com/). From "All Resources" click on the newly created "pub-sub-function" resource.

For the webhook, it can be easily hit through its web URL. For my function, it is: `https://pub-sub-function.azurewebsites.net/api/publish-event`.

Clicking on each shows the code and ability to "Run". Run each and validate that each completes successfully. For the Event Grid function, after running wait a few seconds and view the "Monitor" section to validate the function was called.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/lseo35a1ifdmhgjoj0c1.png)

My Event Grid function threw an exception about JSON serialization. We will be able to test this functionality better using Postman in a few steps.

## 5. Create Event Grid Topic

Click the + icon to create a new resource. Search for "Event Grid Topic" and fill out the details like the image below with a name of "pub-sub-grid".

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/zjean3xstgrddtz5fqf3.png)

Navigate to the pub-sub-grid resource and be sure to copy the **Subscription ID** and the **Resource group** to be used later with Postman.

## 6. Create Function Event Grid subscription

Head back to the "pub-sub-function" Function resource. From the "subscribe-event" function, click the "Add Event Grid subscription" link.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/ah8bbuzegluqd6hc41ra.png)

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/wnuvyhc3f0lf2xz7z7i7.png)

Be sure to select the newly created "pub-sub-grid" from the Event Grid Topics selection.

## 7. Validate pub-sub-function is called from pub-sub-grid topic

For some more in-depth detail about calling Event Grid, refer to the bottom of [this](https://medium.com/@arjitss/tutorial-to-setup-azure-eventgrid-topic-to-call-azure-function-that-triggers-a-sendgrid-email-735f9aa87ba3) blog post.

In Azure Portal navigate to the pub-sub-grid resource. From here we will need to retrieve the Event Grid Topic API URL and an Access Key.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/c9xawyyw5b2bqqeh6q0v.png)

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/3342zy4jxwi6pfep0iof.png)

Using postman set the method to POST and the URL to the Event Grid Topic API URL, `https://pub-sub-grid.eastus-1.eventgrid.azure.net/api/events` in my case. Select Headers and add a key `aeg-sas-key` with a value of your Access Key. In the Body of the post select raw and paste the following:

```
[
   {   "id": "1",
       "eventType": "pub-sub",
       "subject": "pubsub/event",
       "eventTime": "2019-09-28T12:00:00+00:00",
       "data": {
                  "name": "dev3l"   
                },  
        "dataVersion": "1.0",   
        "metadataVersion": "1",
        "topic": "/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.EventGrid/topics/pub-sub-grid"
   }
]
```

Find and replace the following:
- **subscription_id**: Subscription ID copied above from pub-sub-grid
- **resource_group**: Resource group copied above from pub-sub-grid

Validate that you receive a 200 response and then view the Monitor of the pub-sub-function subscribe-event function.

## 8. Publish Event in publish-event Function

Back to Visual Studio Code to make some changes to our functions. First, add "requests" to the `requirements.txt` file. Click the "Terminal" to start a new Terminal session inside your virtual environment. Type `pip install requests` to get the library installed in your local environment for testing.

Update the contents of **publish-event/__init__.py** to be:

```
import logging
import requests

import azure.functions as func

url = "{event_grid_topic_url}"

payload = """[{
   "id": "1",
    "eventType": "pub-sub",
    "subject": "pubsub/event",
    "eventTime": "2019-09-28T12:00:00+00:00",
    "data": {
      "name": "|NAME|"
    },
    "dataVersion": "1.0",
    "metadataVersion": "1",
    "topic": "/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.EventGrid/topics/pub-sub-grid\"
}]"""

headers = { 'aeg-sas-key': "{access_key}" }

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        # PUBLISH EVENT
        requests.request("POST", url, data=payload.replace("|NAME|", name), headers=headers)
        return func.HttpResponse(f"Hello {name}!")
    else:
        return func.HttpResponse(
             "Please pass a name on the query string or in the request body",
             status_code=400
        )
```

Find and replace the following:
- **event_grid_topic_url**: Event Grid Topic URL for pub-sub-grid
- **subscription_id**: Subscription ID copied above from pub-sub-grid
- **resource_group**: Resource group copied above from pub-sub-grid
- **acces_key**: Event Grid Topic Access Key for pub-sub-grid

Granted we are hardcoding the URL, but this opens the door to being fancy later. Hit F5 to run the project and navigate to the URL `http://localhost:7071/api/publish-event?name=pubsub`.

By doing so, you should receive a webpage response and the publish event should be sent to the event grid. You can once again validate this by viewing the Monitor logs of the subscribe-event function. Push up the changes and let's next test it works in Azure!

## 9. Bringing It All Together in Azure

Once you have pushed the updated Azure Functions, navigate back to your web function URL. For me, the address is: `https://pub-sub-function.azurewebsites.net/api/publish-event?name=dev3lsolutions`. Where **name** is query string parameter that is displayed on the next page AND sent as a publish event to the pub-sub-grid Resource, which then invokes the subscribe-event Function!

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/8wapp7yg7kfbs9yir8vu.png)

The source code for this tutorial can be found [here in GitHub](https://github.com/DEV3L/azure-functions-python)!
# Conclusion

Microsoft Azure has allowed us to create a **serverless pub/sub** architecture using Python. In this example of what can be done, we explored several tools and resource types of the Azure of the ecosystem. Functions can publish events in a variety of ways to be consumed by any number of subscriber Functions. The availability and ability to scale in this cloud solution make it a clear choice of traditional in house message queuing and processing systems used by business ETL processes.