Data Ingestion with Azure Event Hubs using Python
Tags: azure, python, eventhubs, etl
Published: 2019-08-25 20:28:51

---
title: Data Ingestion with Azure Event Hubs using Python
published: true
description: "Hello World" example of Azure Event Hubs using Python
tags: azure, python, eventhubs, etl
cover_image: https://thepracticaldev.s3.amazonaws.com/i/2pclx0bl7dkl78s2zo3u.jpg
---

[Extract Transform Load (ETL)](https://www.webopedia.com/TERM/E/ETL.html) is a data integration pattern I have used throughout my career. Decoupling each step is easier than ever with Microsoft Azure. Using [Azure Event Hubs](https://azure.microsoft.com/en-us/services/event-hubs/) we should be able to begin to scaffolding an ephemeral pipeline by creating a mechanism to ingest data however it is extracted.

I have been exposed to many flavors of the ETL pattern throughout my career. From manually inserting data (typically when something goes wrong) through complex snowflake message queue architectures (RabbitMQ + Workers). Ease was never the goal. We believed the problem needed to be solved _RIGHT NOW_ (manual) to it can _NEVER FAIL_ (eventual consistency).

Microsoft has turned its ship around with "Any Developer, Any App, Any Platform". Enablement and reduction of complications is a hallmark of the ecosystem, convention over configuration. My favorite of the twelve principles in the [Manifesto for Agile Software Development](https://agilemanifesto.org/) is *Simplicity*; Azure has nailed it.

```
Simplicity--the art of maximizing the amount of work not done--is essential.
```

# 1. Setup in Azure

First, we need a [Microsoft Azure account](https://azure.microsoft.com/en-us/free/). I was able to connect my GitHub account to their free tier with very little effort.

![GitHub Account Link](https://thepracticaldev.s3.amazonaws.com/i/qu5f8a7tjptjnebs07f6.png)

# 2. Create Event Hub namespace and topic

![Event Hub Architecture](https://thepracticaldev.s3.amazonaws.com/i/bj5vjs7f1lw5natn3fr8.png)

Follow the [Quickstart: Create an event hub using Azure portal](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create). After following the guide, be sure to copy your Event Hub name and primary connection string. We will need these in a bit!

# 3. Send and Receive Events in Python

Now that we have an Azure Event Hub, let's send some messages. As a starting point, I followed along here: [Send events to or receive events from Event Hubs using Python](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send)

### Example

Feel free to use [python-azure-eventhub](https://github.com/DEV3L/python-azure-eventhub) in GitHub as a starting template!

![GitHub README](https://thepracticaldev.s3.amazonaws.com/i/xfrfuym1c1k3km6l1xiq.png)

### Output
![Example Output](https://thepracticaldev.s3.amazonaws.com/i/ij8fiorscqlmmvfxd43a.png)


# Conclusion

Cloud architecture lets us focus on our application and not the management of the machines. We ride the robustness of the Microsoft Azure platform, for a total cost of ownership that is significantly cheaper than internally managed infrastructure. With the scaffolding of Event Hub in place, our first step to a robust ETL pipeline is complete. Next steps are to hook into [Azure Event Grid](https://azure.microsoft.com/en-us/services/event-grid/) to chain messages sent to our queue using the pub-sub design pattern.