Manage Azure Event Hubs with Azure Service Operator on Kubernetes
Tags: kubernetes, azure, tutorial
Published: 2020-07-13 07:43:09

[Azure Service Operator](https://github.com/Azure/azure-service-operator) is an open source project to help you provision and manage Azure services using Kubernetes. Developers can use it to provision Azure services from any environment, be it Azure, any other cloud provider or on-premises - Kubernetes is the only common denominator! 

It can also be included as a part of CI/CD pipelines to create, use and tear down Azure resources on-demand. Behind the scenes, all the heavy lifting is taken care of by a combination of [Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) which define Azure resources and the corresponding [Kubernetes Operator(s)](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) which ensure that the state defined by the Custom Resource Definition is reflected in Azure as well.

![](https://dev-to-uploads.s3.amazonaws.com/i/2x8pc7at7sog3f1xuktb.jpg)

> Read more in the recent announcement here - https://cloudblogs.microsoft.com/opensource/2020/06/25/announcing-azure-service-operator-kubernetes/

In this blog post:

- You will get a high level overview of Azure Service Operator (sometimes referred to as `ASO` in this blog)
- How to set it up and use it to provision [Azure Event Hubs](https://docs.microsoft.com/azure/event-hubs/?WT.mc_id=devto-blog-abhishgu)
- Deploy apps to Kubernetes which use the Azure Event Hubs cluster

> All the artefacts are available on this GitHub repo https://github.com/abhirockzz/eventhubs-using-aso-on-k8s

## Getting started....

[Azure Service Operator supports many Azure services](https://github.com/Azure/azure-service-operator#supported-azure-services) including databases ([Azure Cosmos DB](https://docs.microsoft.com/azure/cosmos-db/introduction?WT.mc_id=devto-blog-abhishgu), [PostgreSQL](https://docs.microsoft.com/azure/postgresql/overview?WT.mc_id=devto-blog-abhishgu), [MySQL](https://docs.microsoft.com/azure/mysql/overview?WT.mc_id=devto-blog-abhishgu), [Azure SQL](https://docs.microsoft.com/azure/azure-sql/?WT.mc_id=devto-blog-abhishgu) etc.), core infrastructure components ([Virtual Machines](https://docs.microsoft.com/azure/virtual-machines/linux/overview?WT.mc_id=devto-blog-abhishgu), [VM Scale sets](https://docs.microsoft.com/azure/virtual-machine-scale-sets/overview?WT.mc_id=devto-blog-abhishgu), Virtual Networks etc.) and others as well.

It also supports Azure Event Hubs which is a fully managed data streaming platform and event ingestion service [with support for Apache Kafka](https://docs.microsoft.com/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview?WT.mc_id=devto-blog-abhishgu) and other tools in the Kafka ecosystem. With Azure Service Operator you can provision and manage Azure Event Hubs namespaces, Event Hub and Consumer Groups.

So, let's dive in without further ado! Before we do that, please note that you will need the following in order to try out this tutorial:

### Pre-requisites

Start by getting an Azure account if you don't have one already - you can [get for FREE!](https://azure.microsoft.com/free/?WT.mc_id=devto-blog-abhishgu) Please make sure you've [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) and [Helm 3](https://helm.sh/docs/intro/install/) installed as well. 

Although the steps outlined in this blog should work with any Kubernetes cluster (including `minikube` etc.), I used [Azure Kubernetes Service (AKS)](https://docs.microsoft.com/azure/aks/intro-kubernetes?WT.mc_id=devto-blog-abhishgu). You can setup a cluster using [Azure CLI](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough?WT.mc_id=devto-blog-abhishgu), [Azure portal](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough-portal?WT.mc_id=devto-blog-abhishgu) or even an [ARM template](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough-rm-template?WT.mc_id=devto-blog-abhishgu). Once that's done, simply [configure `kubectl` to point to it](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough?WT.mc_id=devto-blog-abhishgu#connect-to-the-cluster)

```bash
az aks get-credentials --resource-group <CLUSTER_RESOURCE_GROUP> --name <CLUSTER_NAME>
```

Ok, you're now ready to...

## ... Install Azure Service Operator

![](https://media.giphy.com/media/lYZjoIy0UOEJa/giphy.gif)

Nothing too fancy about it... just following the steps to install it using `Helm`

Start by installing [cert-manager](https://cert-manager.io/docs/installation/kubernetes/)

### Setup `cert-manager`

```bash
kubectl create namespace cert-manager

kubectl label namespace cert-manager cert-manager.io/disable-validation=true

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml

//make sure cert manager is up and running
kubectl rollout status -n cert-manager deploy/cert-manager-webhook
```

### Authentication...

Since the operator will create resource on Azure, we need to authorize it to do so by providing the appropriate credentials. Currently, you can use [Managed Identity](https://docs.microsoft.com/azure/active-directory/managed-identities-azure-resources/overview?WT.mc_id=devto-blog-abhishgu) or [Service Principal](https://docs.microsoft.com/azure/active-directory/develop/app-objects-and-service-principals?WT.mc_id=devto-blog-abhishgu#service-principal-object)

I will be using a Service Principal, so let's start by creating one (with [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu)) using the [az ad sp create-for-rbac](https://docs.microsoft.com/cli/azure/ad/sp?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu#az-ad-sp-create-for-rbac) command

```azurecli
az ad sp create-for-rbac -n "aso-rbac-sp"

//JSON output
{
  "appId": "eb4280db-4242-4ed0-a7d2-42424242f0d0",
  "displayName": "aso-rbac-sp",
  "name": "http://aso-rbac-sp",
  "password": "7d69a422-428d-42d4-a242-cd1d425424b2",
  "tenant": "42f988bf-42f1-42af-42ab-2d7cd421db42"
}
```

### Install



Setup required environment variables:

```bash
export AZURE_SUBSCRIPTION_ID=<enter Azure subscription ID>
export AZURE_TENANT_ID=<enter value from the "tenant" attribute in the JSON payload above>
export AZURE_CLIENT_ID=<enter value from the "appId" attribute in the JSON payload above>
export AZURE_CLIENT_SECRET=<enter value from the "password" attribute in the JSON payload above>
export AZURE_SERVICE_OPERATOR_NAMESPACE=<name of the namespace into which ASO will be installed>
```

Add the repo, create namespace

```bash
helm repo add azureserviceoperator https://raw.githubusercontent.com/Azure/azure-service-operator/master/charts

kubectl create namespace $AZURE_SERVICE_OPERATOR_NAMESPACE
```

Use `helm upgrade` to initiate setup:


```bash
helm upgrade --install aso azureserviceoperator/azure-service-operator \
-n $AZURE_SERVICE_OPERATOR_NAMESPACE \
--set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
--set azureTenantID=$AZURE_TENANT_ID \
--set azureClientID=$AZURE_CLIENT_ID \
--set azureClientSecret=$AZURE_CLIENT_SECRET
```

Before you proceed, wait for the Azure Service Operator `Pod` to startup

```bash
kubectl get pods -n $AZURE_SERVICE_OPERATOR_NAMESPACE

NAME                                              READY   STATUS    RESTARTS   AGE
azureoperator-controller-manager-68f44fd4-cm6wl   2/2     Running   0          6m
```

## Setup Azure Event Hubs components...

Start by cloning the repo:

```bash
git clone https://github.com/abhirockzz/eventhubs-using-aso-on-k8s
cd eventhubs-using-aso-on-k8s
```

Create an [Azure Resource Group](https://docs.microsoft.com/azure/azure-resource-manager/management/overview?WT.mc_id=devto-blog-abhishgu#resource-groups)

> I have used the `southeastasia` location. Please update `eh-resource-group.yaml` if you need to use a different one

```bash
kubectl apply -f deploy/eh-resource-group.yaml

//confirm that its created
kubectl get resourcegroups/eh-aso-rg
```

Create [Event Hubs namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features?WT.mc_id=devto-blog-abhishgu#namespace)

> I have used the `southeastasia` location. Please update `eh-namespace.yaml` if you need to use a different one


```bash
kubectl apply -f deploy/eh-namespace.yaml

//wait for creation
kubectl get eventhubnamespaces -w
```

Once done, you should see this:

```bash
NAME        PROVISIONED   MESSAGE
eh-aso-ns   true          successfully provisioned
```

You can get details with `kubectl describe eventhubnamespaces` and also double-check using [az eventhubs namespace show](https://docs.microsoft.com/cli/azure/eventhubs/namespace?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu#az-eventhubs-namespace-show)

The namespace is ready, we can now create an Event Hub

```
kubectl apply -f deploy/eh-hub.yaml

kubectl get eventhubs/eh-aso-hub

//once done...
NAME        PROVISIONED   MESSAGE
eh-aso-hub  true          successfully provisioned
```

You can get details with `kubectl describe eventhub` and also double-check using [az eventhubs eventhub show](https://docs.microsoft.com/cli/azure/eventhubs/eventhub?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu#az-eventhubs-eventhub-show)

As a final step, create the consumer group

> This is addition to the default consumer group (appropriately named `$Default`)

```
kubectl apply -f deploy/eh-consumer-group.yaml

kubectl get consumergroups/eh-aso-cg

NAME        PROVISIONED   MESSAGE
eh-aso-cg  true          successfully provisioned
```

You can get details with `kubectl describe consumergroup` and also double-check using [ eazventhubs eventhub consumer-group show](https://docs.microsoft.com/cli/azure/eventhubs/eventhub/consumer-group?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu#az-eventhubs-eventhub-consumer-group-show)


## What's next?

Let's make use of what we just setup! We'll deploy a pair of producer and consumer apps to Kubernetes that will send and receive messages from Event Hubs respectively. Both these client apps are written in [Go](https://golang.org/) and use the [Sarama library for Kafka](https://github.com/Shopify/sarama/). I am not going to dive into the details since they are relatively straightforward

Deploy the consumer app:

```bash
kubectl apply -f deploy/consumer.yaml

//wait for it to start
kubectl get pods -l=app=eh-consumer -w
```

Keep a track of the logs for the consumer app:

```bash
kubectl logs -f $(kubectl get pods -l=app=eh-consumer --output=jsonpath={.items..metadata.name})
```

You should see something similar to:

```bash
Event Hubs broker [eh-aso-ns.servicebus.windows.net:9093]
Sarama client consumer group ID eh-aso-cg
new consumer group created
Event Hubs topic eh-aso-hub
Waiting for program to exit
Partition allocation - map[eh-aso-hub:[0 1 2]]
```

Using another terminal, deploy the producer app:

```bash
kubectl apply -f deploy/producer.yaml
```

Once the producer app is up and running, the consumer should kick in, start consumer the messages and print them to the console. So you'll see logs similar to this:

```bash
...
Message topic:"eh-aso-hub" partition:0 offset:6
Message content value-2020-07-06 15:37:06.116674866 +0000 UTC m=+67.450171692
Message topic:"eh-aso-hub" partition:0 offset:7
Message content value-2020-07-06 15:37:09.133115988 +0000 UTC m=+70.466612714
Message topic:"eh-aso-hub" partition:0 offset:8
Message content value-2020-07-06 15:37:12.149068005 +0000 UTC m=+73.482564831
...
```

> In case you want to check producer logs as well: `kubectl logs -f $(kubectl get pods -l=app=eh-producer --output=jsonpath={.items..metadata.name})`

Alright, it worked! 

- We created an Event Hubs namespace, Event Hub along and a consumer group.. all using `kubectl` (and YAMLs of course)
- Deployed a simple producer and consumer for testing

## But, what just happened ...?

... how did the consumer and producer apps connect to Event Hubs without connection info, credentials etc.?

![](https://media.giphy.com/media/xTiQygY6HW1GjoYKFq/giphy.gif)

Notice this part of Event Hub manifest (`eh-hub.yaml` file):


```yaml
...
spec:
  secretName: eh-secret
  location: southeastasia
  resourceGroup: eh-aso-rg
...
```

`secretName: eh-secret` ensured that a Kubernetes `Secret` was created with the required connectivity details including  connection strings (primary, secondary), keys (primary, secondary), along with the basic info such as Event Hubs namespace and hub name.

The producer and consumer `Deployment`s were simply able to refer to this. Take a look at this snippet from the consumer app `Deployment`


```yaml
...
      containers:
        - name: eh-consumer
          image: abhirockzz/eh-kafka-consumer
          env:
            - name: EVENTHUBS_CONNECTION_STRING
              valueFrom:
                secretKeyRef:
                  name: eh-secret
                  key: primaryConnectionString
            - name: EVENTHUBS_NAMESPACE
              valueFrom:
                secretKeyRef:
                  name: eh-secret
                  key: eventhubNamespace
            - name: EVENTHUBS_BROKER
              value: $(EVENTHUBS_NAMESPACE).servicebus.windows.net:9093
            - name: EVENTHUBS_TOPIC
              valueFrom:
                secretKeyRef:
                  name: eh-secret
                  key: eventhubName
            - name: EVENTHUBS_CONSUMER_GROUPID
              value: eh-aso-cg
...
```

The app uses env vars `EVENTHUBS_CONNECTION_STRING`, `EVENTHUBS_NAMESPACE` and `EVENTHUBS_TOPIC` whose values were sourced from the Secret (`eh-secret`). The value for `EVENTHUBS_CONSUMER_GROUPID` is hardcoded to `eh-aso-cg` which was the name of the consumer group specified in `eh-consumer-group.yaml`.

## Clean up

To remove all the resources including Event Hubs and the client apps, simply use `kubectl delete -f deploy`

## Conclusion

Azure Service Operator provides a layer of abstraction on top Azure specific primitives. It allows you to manage Azure resources and also provide ways to connect to them using other applications deployed in the same Kubernetes cluster.

I covered Azure Event Hubs as an example, but as I mentioned earlier, Azure Service Operator also supports other services too. Head over to [the GitHub repo](https://github.com/Azure/azure-service-operator) and give them a try!