Learn how to use Azure Event Hubs with the Dapr framework
Tags: tutorial, azure, microservices, showdev
Published: 2019-11-04 13:09:23

This blog uses an example to walk you through how to use [**Azure Event Hubs**](https://azure.microsoft.com/services/event-hubs/?WT.mc_id=devto-blog-abhishgu) integration with [**Dapr**](https://dapr.io/), a set of distributed system building blocks for microservices development. 

Azure Event Hubs will be used as a "binding" within the `Dapr` runtime. This will allow services to communicate with Azure Event Hubs without actually knowing about it or being coupled to it directly (via SDK, library etc.), using a simple model defined by the `Dapr` runtime.

Sounds too good to be true? 😉 Read on...

You will:

- Learn the basics of `Dapr` and how to set it up for local development
- Deploy your first Dapr app(s)
- Setup Event Hubs on Azure and try out an example to see how to communicate with it transparently using Dapr integration

> [Azure Event Hubs](https://docs.microsoft.com/azure/event-hubs/?WT.mc_id=devto-blog-abhishgu) is a fully managed Platform-as-a-Service (PaaS) for streaming and event ingestion. It also provides Apache Kafka support enabling clients and applications to talk to Event Hubs without need to set up, configure, and manage your own Kafka clusters!

## Dapr? What's that?

`Dapr` stands for Distributed Application Runtime. You can get all the scoop in [this announcement blog](https://cloudblogs.microsoft.com/opensource/2019/10/16/announcing-dapr-open-source-project-build-microservice-applications/), but here is a (buzzword compliant!) gist to get you started.

It is an open source, portable runtime to help developers build resilient, microservice stateless and stateful applications. It does so by codifying the best practices for building such applications into independent components. The capabilities exposed by the runtime components can be accessed over HTTP or gRPC, making it completely agnostic and allows it to work with any language and/or framework.

![](https://github.com/dapr/docs/blob/master/images/overview.png?raw=true)

### Dapr: modus operandi

Dapr adopts a sidecar architecture and runs either as a separate container or as a process. This means that the application itself does not need to include Dapr runtime as a dependency. This allows for easy integration as well as providing separation of concerns.

![](https://github.com/dapr/docs/blob/master/images/overview-sidecar.png?raw=true)

Dapr also includes language specific SDKs for [Go](https://github.com/dapr/go-sdk), [Java](https://github.com/dapr/java-sdk), [JavaScript](https://github.com/dapr/js-sdk), [.NET](https://github.com/dapr/dotnet-sdk) and [Python](https://github.com/dapr/python-sdk). These SDKs expose the functionality in the Dapr building blocks, such as saving state, publishing an event or creating an actor, through a typed, language API rather than calling the `http/gRPC` API.

> Dapr is platform agnostic. You can run your applications locally, on any Kubernetes cluster and other hosting environments that Dapr integrates with.

### Dapr components

At the time of writing, `Dapr` is in alpha state and supports the following distributed systems building blocks which you can plug into your applications - Service invocation, State Management, Pub/Sub messaging, Resource Bindings, Distributed tracing and Actor pattern

> For a comprehensive overview, please refer [to the Dapr documentation](https://github.com/dapr/docs/blob/master/overview.md)

As mentioned earlier, the example application in this post makes use of the Dapr bindings for Azure Event Hubs. Here is a peek at what "bindings" are.

#### Dapr Resource Bindings

Bindings provide a common way to trigger an application with events from external systems or invoke an external system with optional data payloads. These "external systems" could be anything: a queue, messaging pipeline, cloud-service, filesystem, etc.

> [Currently supported bindings](https://github.com/dapr/components-contrib/blob/master/bindings/Readme.md) include Kafka, Rabbit MQ, Azure Event Hubs etc.

In a nutshell, Dapr bindings allow you to focus on business logic rather than integrating with individual services such as databases, pub/sub systems, blob storage etc.

- you can leverage off the shelf bindings to integrate with different systems without depending on or including specific SDKs or libraries
- you can also swap/replace bindings without changing your code

Alright, enough theory!

![](https://media.giphy.com/media/3oKIPnAiaMCws8nOsE/200w_d.gif)

## Pre-requisites

You can run `Dapr` anywhere, including [Kubernetes](https://kubernetes.io/), thanks to its first class [Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) based support. Since this is an introductory blog post, let's focus on the end to end flow, keep things simple and run `Dapr` locally as a standalone component.

> If you're itching to run `Dapr` on Kubernetes, check out [this getting started guide](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md#installing-dapr-on-a-kubernetes-cluster)!

For the sample app, you will need:

- [Docker](https://docs.docker.com/install/)
- [Go 1.12 or above](https://golang.org/)
- [Azure Event Hubs](https://azure.microsoft.com/services/event-hubs/?WT.mc_id=devto-blog-abhishgu)

### Setup Dapr

Start by installing the [Dapr CLI](https://github.com/dapr/cli) which allows you to setup Dapr on your local dev machine or on a Kubernetes cluster, provides debugging support, launches and manages Dapr instances.

For e.g. on your Mac, you can simply use this to install `Dapr` to `/usr/local/bin`

    curl -fsSL https://raw.githubusercontent.com/dapr/cli/master/install/install.sh | /bin/bash

> Refer to the [docs for details](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md#installing-dapr-cli)

You can use the CLI to [install Dapr in standalone mode](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md#installing-dapr-in-standalone-mode). All you need is a single command

    dapr init

.. and that's it!

### Setup Azure Event Hubs

If you don't already have a [Microsoft Azure account](https://docs.microsoft.com/azure/?WT.mc_id=devto-blog-abhishgu), go ahead and [sign up for a free one!](https://azure.microsoft.com/free/?WT.mc_id=devto-blog-abhishgu) Once you're done you can quickly set up Azure Event Hubs using either of the following quickstarts:

- Azure portal - [here is a step-by-step guide](https://docs.microsoft.com/azure/event-hubs/event-hubs-create?WT.mc_id=devto-blog-abhishgu)
- Azure CLI or Azure Cloud shell (in your browser!) - [here is a step-by-step guide](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli?WT.mc_id=devto-blog-abhishgu)

You should now have an Event Hub instance with a [namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features?WT.mc_id=devto-blog-abhishgu#namespace) and associated Event Hub (topic). As a final step you need to get the connection string in order to [authenticate to Event Hubs](https://docs.microsoft.com/azure/event-hubs/authenticate-shared-access-signature?WT.mc_id=devto-blog-abhishgu) - [use this guide](https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string?WT.mc_id=devto-blog-abhishgu) to finish this step.

## Receive Event Hubs data using Input Bindings

An Input Binding in `Dapr` represents an event resource that `Dapr` uses to read events from and push to your application. Let's run an app that uses this to receive data from Azure Event Hubs.

### Run the app with Dapr

Start by cloning the repo and change into the correct directory

    git clone https://github.com/abhirockzz/dapr-eventhubs-bindings
    cd input-binding-app

Update `components/eventhubs_binding.yaml` to include Azure Event Hubs connection string in the `spec.metadata.value` section. Please note that you will have to append the name of the Event Hub to end the connection string i.e. `;EntityPath=<EVENT_HUBS_NAME>`. This is what the value for `connectionString` attribute should look like:

    Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<EVENT_HUBS_KEY>;EntityPath=<EVENT_HUBS_NAME>

Start the Go app which uses the Azure Event Hubs Input Bindings

    export APP_PORT=8080
    dapr run --app-port $APP_PORT go run app.go

You should see the logs

    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="starting Dapr Runtime -- version 0.1.0 -- commit 4358565-dirty"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="log level set to: info"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="standalone mode configured"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="dapr id: foobar"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="loaded component messagebus (pubsub.redis)"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="loaded component timebound (bindings.azure.eventhubs)"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="loaded component statestore (state.redis)"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="application protocol: http. waiting on port 8080"
    == DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="application discovered on port 8080"
    .....

### Run Azure Event Hubs producer application

This app uses the [Azure Event Hubs native Go client](https://github.com/Azure/azure-event-hubs-go) to send messages.

Set the required environment variables:

    export EVENT_HUBS_NAMESPACE="<EVENT_HUBS_NAMESPACE>"
    export EVENT_HUBS_KEY="<EVENT_HUBS_KEY>"
    export EVENT_HUB_NAME="<EVENT_HUB_NAME>"

> Please ensure that the name of the Event Hub is the same as what you configured for the connection string in the input binding configuration

Run the producer app - it will send five messages to Event Hubs and exit

    cd eventhubs-producer
    go run producer.go

### Confirm

Check Dapr application logs, you should see the messages received from Event Hubs.

    == APP == time from Event Hubs 'Thu Oct 31 16:57:45 2019'
    == APP == time from Event Hubs 'Thu Oct 31 16:57:49 2019'
    == APP == time from Event Hubs 'Thu Oct 31 16:57:52 2019'
    == APP == time from Event Hubs 'Thu Oct 31 16:57:54 2019'
    == APP == time from Event Hubs 'Thu Oct 31 16:57:56 2019'

### Behind the scenes

Here is a summary of how it works:

**Input Binding**

The `eventhub_binding.yaml` config file captures the connection string for Azure Event Hubs. 

```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: eventhubs-input
spec:
  type: bindings.azure.eventhubs
  metadata:
  - name: connectionString
    value: Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<KEY>;EntityPath=<EVENT_HUBS_NAME>
```

The key attributes are:

- `metadata.name` - name of the Input binding component
- `spec.metadata.name` - Event Hubs connection string

> Notice that the connection string contains the information for the broker URL (`<EVENT_HUBS_NAMESPACE>.servicebus.windows.net`), primary key (for authentication) and also the name of the topic or Event Hub to which your app will be bound and receive events from.

**Using the binding in the app**

The Go app exposes a REST endpoint at `/eventhubs-input` - this is the same as the name of the Input Binding component (not a coincidence!)


    func main() {
        http.HandleFunc("/eventhubs-input", func(rw http.ResponseWriter, req *http.Request) {
            var _time TheTime
            err := json.NewDecoder(req.Body).Decode(&_time)
            if err != nil {
                fmt.Println("error reading message from event hub binding", err)
                rw.WriteHeader(500)
                return
            }
            fmt.Printf("time from Event Hubs '%s'\n", _time.Time)
            rw.WriteHeader(200)
        })
        http.ListenAndServe(":"+port, nil)
    }

Dapr runtime does the heavy lifting of consuming from Event Hubs and making sure that it invokes the Go application with a `POST` request at the `/eventhubs-input` endpoint with the event payload. The app logic is then executed, which in this case is simply logging to standard output.

## Send data to Event Hubs data with Output Bindings

An output binding represents a resource that `Dapr` will use invoke and send messages to. Let's use an output binding to send data to Event Hubs.

## Run the app with Dapr

Change to the correct directory

    cd output-binding-app

Update `components/eventhubs_binding.yaml` to include Azure Event Hubs connection string in the `spec.metadata.value` section. Please note that you will have to append the name of the Event Hub to end the connection string i.e. `;EntityPath=<EVENT_HUBS_NAME>`. This is what the value for `connectionString` attribute should look like:

    Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<EVENT_HUBS_KEY>;EntityPath=<EVENT_HUBS_NAME>

Start the Go app which uses the Azure Event Hubs Output Bindings. It will send five messages to the Dapr binding HTTP endpoint in quick succession and exit.

    dapr run --port 3500 go run app.go

In the app, you will see logs:

    == APP == sent message {"data": {"time":"2019-10-31 17:18:07.602377 +0530 IST m=+0.000569729"}}
    == APP == response 200 OK
    == APP == sent message {"data": {"time":"2019-10-31 17:18:12.375237 +0530 IST m=+4.773393815"}}
    == APP == response 200 OK
    == APP == sent message {"data": {"time":"2019-10-31 17:18:14.504914 +0530 IST m=+6.903055209"}}
    == APP == response 200 OK
    == APP == sent message {"data": {"time":"2019-10-31 17:18:16.637992 +0530 IST m=+9.036116365"}}
    == APP == response 200 OK
    == APP == sent message {"data": {"time":"2019-10-31 17:18:18.763691 +0530 IST m=+11.161800122"}}
    == APP == response 200 OK
    == APP == finished sending messages... use ctrl+c to exit


This means that the messages were sent to Azure Event Hub via the Dapr output binding

In the receiver (input bindings) app, you should see

    == APP == time from Event Hubs '2019-10-31 17:18:07.602377 +0530 IST m=+0.000569729'
    == APP == time from Event Hubs '2019-10-31 17:18:12.375237 +0530 IST m=+4.773393815'
    == APP == time from Event Hubs '2019-10-31 17:18:14.504914 +0530 IST m=+6.903055209'
    == APP == time from Event Hubs '2019-10-31 17:18:16.637992 +0530 IST m=+9.036116365'
    == APP == time from Event Hubs '2019-10-31 17:18:18.763691 +0530 IST m=+11.161800122'

> To keep things simple, we used the same Event Hub topic for input and output binding example. In reality these can be used independently i.e. an application which needs to be triggered by Event Hubs in an event-driven manner can subscribe to a topic (via Dapr) and another app which needs to push data to Event Hubs (to a different topic) can use the Dapr binding HTTP endpoint to `POST` event data.

### Behind the scenes

The output binding configuration remains the same as Input binding. 

The Go app sends a message to Event Hubs via the output binding. It does so by sending a `POST` request to the Dapr HTTP endpoint `http://localhost:<dapr_port>/v1.0/bindings/<output_binding_name>`. 
    
    ....
    const daprURL = "http://localhost:3500/v1.0/bindings/eventhubs-output"
	go func() {
        ....
		for i := 1; i <= 5; i++ {
			body := fmt.Sprintf(format, time.Now().String())
			resp, err := http.Post(daprURL, contentType, strings.NewReader(body))
            .....
			fmt.Println("sent message", body)
			fmt.Println("response", resp.Status)
			time.Sleep(2 * time.Second)
		}
		done <- true
	}()
    ....

In this example, the name of the output binding is `eventhubs-output` and the Dapr HTTP port is `3500`, the endpoint URL is:

    http://localhost:3500/v1.0/bindings/eventhubs-output

As was the case with Input Binding, the `Dapr` runtime takes care of sending the event to Event Hubs. Since we have the Input binding app running, it receives the payload and it shows up in the logs.

## Summary

In this blog post, you saw how to use `Dapr` "bindings" to connect integrate your applications via Azure Event Hubs without having to know about it! Instead, you connected through the Dapr runtime (just a sidecar) using the Dapr HTTP API.

> It is also possible to do it using gRPC or language specific SDKs (as previously mentioned)

As the time of writing, `Dapr` is in alpha state (`v0.1.0`) and gladly accepting community contributions 😃 Vist https://github.com/dapr/dapr to dive in!

If you found this article helpful, please like and follow 🙌 Happy to get feedback via [Twitter](https://twitter.com/abhi_tweeter) or just drop a comment.