Tutorial: Using Azure Event Hubs and Blob Storage with the Dapr framework
Tags: azure, showdev, opensource, tutorial
Published: 2019-11-14 11:53:36

The [previous blog](https://dev.to/azure/learn-how-to-use-azure-event-hubs-with-the-dapr-framework-333h) demonstrated an example of how to use [**Azure Event Hubs**](https://azure.microsoft.com/services/event-hubs/?WT.mc_id=devto-blog-abhishgu) integration with [**Dapr**](https://dapr.io/). Azure Event Hubs was used as a "binding" within the `Dapr` runtime to allow the application to communicate with Azure Event Hubs without actually knowing about it or being coupled to it directly (via SDK, library etc.), using a simple model defined by the `Dapr` runtime.

This blog post will showcase how to stitch together multiple `Dapr` bindings. Although this applies to any supported binding, the example will combine [Azure Event Hubs](https://azure.microsoft.com/services/event-hubs/?WT.mc_id=devto-blog-abhishgu) and [Azure Blob Storage](https://azure.microsoft.com/services/storage/blobs/?WT.mc_id=devto-blog-abhishgu), where they will be used as Input and Output bindings respectively. All this is done in a simplified manner with no direct reference to Event Hubs or Blob storage in the application code, thanks to the binding based integration.

We will cover:

- Setup for `Dapr` and related Azure services
- Run our application and see it in action, sending data from Azure Event Hubs to Azure Blog Storage
- Walkthrough of how it works behind the scenes

> **Azure Event Hubs** is a fully managed Platform-as-a-Service (PaaS) for streaming and event ingestion and **Azure Blob Storage** is an object storage solution for the cloud, optimized for storing massive amounts of unstructured data.

## Hello Dapr!

**Dapr** stands for *Distributed Application Runtime*. It is an open source, portable runtime to help developers build resilient, microservice stateless and stateful applications by codifying the best practices for building such applications into independent components.

If you're new to Dapr, I would recommend starting off with the [overview](https://github.com/dapr/docs/blob/master/overview.md) and [concepts](https://github.com/dapr/docs/tree/master/concepts). Try the [getting-started guide](https://github.com/dapr/docs/tree/master/getting-started) and then move on to [samples](https://github.com/dapr/samples/blob/master/README.md) and [how-to guides](https://github.com/dapr/docs/tree/master/howto). As you advance further, you can dig into the [Dapr runtime API reference](https://github.com/dapr/docs/tree/master/reference/api) and [individual components](https://github.com/dapr/components-contrib)


### Dapr Bindings

![](https://github.com/dapr/docs/blob/master/images/overview_standalone.png?raw=true)

At the time of writing, `Dapr` is in alpha state and supports the following distributed systems building blocks which you can plug into your applications - Service invocation, State Management, Pub/Sub messaging, Resource Bindings, Distributed tracing and Actor pattern. Bindings provide a common way to trigger an application with events from external systems or invoke an external system with optional data payloads. These "external systems" could be anything: a queue, messaging pipeline, cloud-service, filesystem, etc.

> [Currently supported bindings](https://github.com/dapr/components-contrib/blob/master/bindings/Readme.md) include Kafka, Rabbit MQ, Azure Event Hubs etc.

In a nutshell, `Dapr` bindings allow you to focus on business logic rather than integrating with individual services such as databases, pub/sub systems, blob storage etc.

Let's get started...

## Setup: Dapr, Azure Event Hubs and Blob Storage

This section will guide you through the setup process for Dapr, Azure Event Hubs, and Blob Storage.

To begin with, you will need a [Microsoft Azure account](https://docs.microsoft.com/azure/?WT.mc_id=devto-blog-abhishgu). If you don't have one already, please go ahead and [sign up for a free one!](https://azure.microsoft.com/free/?WT.mc_id=devto-blog-abhishgu)

### Setup Dapr

For `Dapr`, you will require:

- [Docker](https://docs.docker.com/install/)
- [Go 1.12 or above](https://golang.org/)

To keep things simple, we'll run `Dapr` locally as a standalone component.

> If you're itching to run `Dapr` on Kubernetes, check out [this getting started guide](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md#installing-dapr-on-a-kubernetes-cluster)!

Start by installing the [Dapr CLI](https://github.com/dapr/cli) which allows you to setup Dapr on your local dev machine or on a Kubernetes cluster, provides debugging support, launches and manages Dapr instances.

For e.g. on your Mac, you can simply use this to install `Dapr` to `/usr/local/bin`

```
curl -fsSL https://raw.githubusercontent.com/dapr/cli/master/install/install.sh | /bin/bash
```

> Refer to the [documentation for details](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md#installing-dapr-cli)

You can use the CLI to [install Dapr in standalone mode](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md#installing-dapr-in-standalone-mode). All you need is a single command

```
dapr init
```

.. and that's it!

### Setup Azure Event Hubs

You can quickly set up Azure Event Hubs using either of the following quickstarts:

- Using Azure portal - [here is a step-by-step guide](https://docs.microsoft.com/azure/event-hubs/event-hubs-create?WT.mc_id=devto-blog-abhishgu)
- Using Azure CLI or Azure Cloud shell (in your browser!) - [here is a step-by-step guide](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli?WT.mc_id=devto-blog-abhishgu)

You should now have an Event Hub instance with a [namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features?WT.mc_id=devto-blog-abhishgu#namespace) and associated Event Hub (topic). As a final step, you need to get the connection string in order to [authenticate to Event Hubs](https://docs.microsoft.com/azure/event-hubs/authenticate-shared-access-signature?WT.mc_id=devto-blog-abhishgu) - [use this guide](https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string?WT.mc_id=devto-blog-abhishgu) to finish this step.

### Setup Azure Blob storage

If you'd like to use the Azure portal:

- [Create a storage account](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal&WT.mc_id=devto-blog-abhishgu), and
- [Create a container](https://docs.microsoft.com/azure/storage/blobs/storage-quickstart-blobs-portal?WT.mc_id=devto-blog-abhishgu#create-a-container)

If you want to use the Azure CLI or Azure Cloud shell, [use this step-by-step guide](https://docs.microsoft.com/azure/storage/blobs/storage-quickstart-blobs-cli?WT.mc_id=devto-blog-abhishgu)

Now that you have everything set up, let's move ahead and try out the application

## Run the app with Dapr

Start by cloning the repo and change into the correct directory

```
git clone https://github.com/abhirockzz/dapr-eventhubs-blobstore
```

Update `components/eventhubs_binding.yaml` to include Azure Event Hubs connection string in the `spec.metadata.value` section. 

> Please note that you will have to append the name of the Event Hub to end the connection string i.e. `;EntityPath=<EVENT_HUBS_NAME>`. 

This is what the value for `connectionString` attribute should look like:

```
	Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<EVENT_HUBS_KEY>;EntityPath=<EVENT_HUBS_NAME>
```

Update `components/blobstorage.yaml` to include Azure Blob storage details - `storageAccount`, `storageAccessKey` and `container`

Start the Go app

```
    export APP_PORT=8080
    dapr run --app-port $APP_PORT go run app.go
```

You should see the logs:

```
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="STARTING Dapr Runtime -- version edge -- commit ff7815d-dirty"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="log level set to: info"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="standalone mode configured"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="dapr id: Seekergrass-Shaker"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="loaded component messagebus (pubsub.redis)"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="loaded component statestore (state.redis)"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="loaded component eventhubs-input (bindings.azure.eventhubs)"
	== DAPR == time="2019-11-14T15:33:14+05:30" level=info msg="loaded component storage (bindings.azure.blobstorage)"
	......
```

## Run Azure Event Hubs producer application

This app uses the [Azure Event Hubs native Go client](https://github.com/Azure/azure-event-hubs-go) to send messages.

Set the required environment variables:

```
	export EVENT_HUBS_NAMESPACE="<EVENT_HUBS_NAMESPACE>"
	export EVENT_HUBS_KEY="<EVENT_HUBS_KEY>"
	export EVENT_HUB_NAME="<EVENT_HUB_NAME>"
```

> Please ensure that the name of the Event Hub is the same as what you configured for the connection string in the input binding configuration

Run the producer app

```
	export GO111MODULE=on
	go run eventhubs-producer/producer.go
```

It will send five messages to Event Hubs and exit. you should see logs like:

```
	Sent message {"time":"Thu Nov 14 15:35:13 2019"}
	Sent message {"time":"Thu Nov 14 15:35:18 2019"}
	Sent message {"time":"Thu Nov 14 15:35:20 2019"}
	Sent message {"time":"Thu Nov 14 15:35:23 2019"}
	Sent message {"time":"Thu Nov 14 15:35:25 2019"}
```

## Confirm

Check Dapr application logs, you should see the messages received from Event Hubs.

```
	== APP == time from Event Hubs 'Thu Nov 14 15:35:13 2019'
	== APP == time from Event Hubs 'Thu Nov 14 15:35:18 2019'
	== APP == time from Event Hubs 'Thu Nov 14 15:35:20 2019'
	== APP == time from Event Hubs 'Thu Nov 14 15:35:23 2019'
	== APP == time from Event Hubs 'Thu Nov 14 15:35:25 2019'
```

Check Azure Blob storage. First, the Azure CLI needs your storage account credentials. Get your storage account keys by using the `az storage account keys list` command

```
	export AZURE_SUBSCRIPTION_ID=<to be filled>
	export AZURE_STORAGE_ACCOUNT=<to be filled>
	export AZURE_STORAGE_ACCOUNT_RESOURCE_GROUP=<to be filled>

	az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT --resource-group $AZURE_STORAGE_ACCOUNT_RESOURCE_GROUP --subscription $AZURE_SUBSCRIPTION_ID --output table
```

Use either of the two keys and export it in form of environment variable

```
	export AZURE_STORAGE_KEY=<to be filled>
```

List the blobs in the container (use the name of the container which you created while setting up Azure Blob Storage in the previous section)

```
	export CONTAINER_NAME=<to be filled>
	az storage blob list --container-name $CONTAINER_NAME --subscription $AZURE_SUBSCRIPTION_ID --output table
```

You will should five blobs in the container. This is because five messages were pushed to Event Hubs and then saved to Azure Blob storage by Dapr. You can confirm their contents as well by downloading the blob

```
	export BLOB_NAME=<to be filled>

	az storage blob download --container-name $CONTAINER_NAME --subscription $AZURE_SUBSCRIPTION_ID --name $BLOB_NAME --file $BLOB_NAME
```

This will download the contents to a file (with same name as the blob) in your current directory. To peek inside, simply

```
cat $BLOB_NAME
```

You should see the message sent to Azure Event Hubs

```
	{"time":"Thu Nov 14 15:35:20 2019"}
```

> repeate the same with other blobs

## Behind the scenes

Here is a summary of how it works:

### Input Binding

The `eventhub_binding.yaml` config file captures the connection string for Azure Event Hubs. 

```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: eventhubs-input
spec:
  type: bindings.azure.eventhubs
  metadata:
  - name: connectionString
    value: Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<KEY>;EntityPath=<EVENT_HUBS_NAME>
```

The key attributes are:

- `metadata.name` - name of the Input binding component
- `spec.metadata.name` - Event Hubs connection string

> Notice that the connection string contains the information for the broker URL (`<EVENT_HUBS_NAMESPACE>.servicebus.windows.net`), primary key (for authentication) and also the name of the topic or Event Hub to which your app will be bound and receive events from.

### Output Binding

The `blobstorage.yaml` config file captures the connection string for Azure Blob storage

```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: storage
spec:
  type: bindings.azure.blobstorage
  metadata:
  - name: storageAccount
    value: [BLOB_STORAGE_ACCOUNT]
  - name: storageAccessKey
    value: [BLOB_STORAGE_ACCESS_KEY]
  - name: container
    value: [BLOB_STORAGE_CONTAINER_NAME]
```

### Using the bindings in the app

The Go app exposes a REST endpoint at `/eventhubs-input` - this is the same as the name of the Input Binding component (not a coincidence!)

```
	func main() {
		http.HandleFunc("/eventhubs-input", func(rw http.ResponseWriter, req *http.Request) {
			var _time TheTime
			err := json.NewDecoder(req.Body).Decode(&_time)
			if err != nil {
				fmt.Println("error reading message from event hub binding", err)
				rw.WriteHeader(500)
				return
			}
			fmt.Printf("time from Event Hubs '%s'\n", _time.Time)
			rw.WriteHeader(200)
			err = json.NewEncoder(rw).Encode(Response{To: []string{"storage"}, Data: _time})
			if err != nil {
				fmt.Printf("unable to respond'%s'\n", err)
			}
		})
		http.ListenAndServe(":"+port, nil)
	}
```

Dapr runtime does the heavy lifting of consuming from Event Hubs and making sure that it invokes the Go application with a `POST` request at the `/eventhubs-input` endpoint with the event payload. 

The application responds by returning a JSON output which contains the payload received from Event Hub and the name of the output binding (`storage` in this case). It is represented by the following `struct`

```
	type Response struct {
		To   []string    `json:"to"`
		Data interface{} `json:"data"`
	}
```

And that's where the magic happens! Dapr gathers this response and sends the payload to the output binding which is Azure Blob Storage in this case.

## Summary

In this blog post, you saw how to stitch (or compose) together different services by using them as Dapr bindings. All your application did was interact with the Dapr runtime (just a sidecar) using the Dapr HTTP API!

> It is also possible to do it using gRPC or language specific SDKs

As the time of writing, `Dapr` is in alpha state (`v0.1.0`) and gladly accepting community contributions 😃 Vist https://github.com/dapr/dapr to dive in!

If you found this article helpful, please like and follow 🙌 Happy to get feedback via [Twitter](https://twitter.com/abhi_tweeter) or just drop a comment.