Install and Maintain Apache Kafka® with a GitOps Approach
Tags: kafka, gitops, devops, argocd
Published: 2023-05-16 13:57:00

Previously, we showed you how you can install and manage Apache Kafka® on Kubernetes with Koperator. While a one-time installation is great, you can unlock more value by automating this process to replicate Kafka deployments as needed for different environments. In this blog post, I will walk you through automating Apache Kafka deployments with ArgoCD.

## Setup

Before you get started, you’ll need to install the necessary tools:

- a Kubernetes cluster on AWS
- `eksctl`
- [Helm](https://helm.sh)
- [ArgoCD](https://argo-cd.readthedocs.io/en/stable/)

### Provision a new cluster

Next, we’ll use `eksctl` to provision new clusters. If you don't have `eksctl` on your machine yet, install it by following [these instructions.](https://eksctl.io/introduction/#installation)

`eksctl` either accepts command-line arguments or a configuration file. Today, we'll use the following parameters in this yaml configuration to setup our cluster:

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: koperator-demo 
  region: us-east-2 
  version: "1.24"

managedNodeGroups:
  - name: ng-1
    labels: 
    instanceType: m5.xlarge
    desiredCapacity: 2
    volumeSize: 80
    privateNetworking: true
```

With the configuration in place, let's create a new cluster:

```bash
eksctl create cluster -f workstation.yaml
```

This step might take a while since it needs to provision a lot of resources.

Once the cluster is ready, you can continue installing ArgoCD. Argo CD is an open-source GitOps continuous delivery tool. It monitors your cluster and your declaratively defined infrastructure stored in a Git repository and resolves differences between the two.

### Install ArgoCD

Run the following commands:

```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```

This will install ArgoCD on your new cluster. Note that ArgoCD can deploy applications within the same cluster or remote clusters. For the sake of scope, we will deploy within the same cluster.

Next you’ll install the ArgoCD CLI:

```bash
brew install argocd 
```

(If you're using a different operating system, please visit [this link](https://argo-cd.readthedocs.io/en/stable/cli_installation/) for install instructions.)

By default, the Argo CD API server is not exposed with an external IP. To access the API server, run the following command:

```bash
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
```

Next, you’ll configure access credentials:

```bash
argocd admin initial-password -n argocd
```

We won't expose ArgoCD directly, but rather leverage port forwarding to access the API server and the web interface:

```bash
kubectl port-forward svc/argocd-server -n argocd 8080:443
```

Next, you'll log in with username `admin` and the password you configured:

```bash
argocd login localhost:8080
```

After you’re logged in, you’ll also need to configure a deployment target. In this case, we'll use the same cluster ArgoCD runs on:

```bash
argocd cluster add <your clustername>.us-east-2.eksctl.io
```

Now you’re good to go!

## Continuous deployment and GitOps

Per my previous blog post, I showed you how to set up a new Kafka cluster on Kubernetes in minutes. While that process will get you up and running quickly, the real value comes with an automated infrastructure setup. For that, we want to leverage a [GitOps](https://www.gitops.tech/)-based approach to deploy Apache ZooKeeper<sup>TM</sup> and Kafka, using ArgoCD.

We will use [*https://github.com/banzaicloud/koperator*](https://github.com/banzaicloud/koperator) as a starting point and turn its instructions into Helm Charts, which we will deploy with ArgoCD.

## Helm Charts

A Helm Chart allows you to make application installations on Kubernetes reproducible. Usually, to install an application on Kubernetes, you'd run a series of `kubectl apply` commands to create various resources, such as deployments, pods, `services`, etc. A Helm Chart bundles these resources and allows us to parameterize them. A Helm Chart can be installed several times within a Kubernetes cluster, under different names, so-called `release`. We'll use a single repository that will hold two Helm Charts--one for ZooKeeper and one for Koperator/Kafka.

### ZooKeeper

To install and run ZooKeeper, we'll use the ZooKeeper from `charts.pravega.io`. This operator does the heavy lifting.

The ZooKeeper Helm Chart lists it as dependency in `Chart.yaml`:

```yaml
#Chart.yaml
apiVersion: v2
name: -setup
description: Installs  aplication
version: 0.1.0
appVersion: "1.16.0"
dependencies:
  - name: zookeeper-operatorsitory: https://charts.pravega.io
    version: "0.2.15"
```

Next, let's take a look at `templates/zookeeper-cluster.yml`:

```yaml
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
    name: {{.Values.zookeeper.name }} 
    namespace: {{.Values.zookeeper.namespace }} 
spec:
    replicas: {{.Values.zookeeper.replicas }}
    persistence:
        reclaimPolicy: {{.Values.zookeeper.reclaimPolicy }}
```


When executed, this yaml code creates a new ZooKeeper cluster. The template file does not contain specific values, only template variables. Since Helm allows us to install a chart more than once, this yaml code is parameterized, so the user can override values such as the cluster name or namespace.

Helm allows us to provide values in different ways. For now, we stick with a few defaults defined in `values.yaml`:

```yaml
# Default values for koperator-setup.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

zookeeper:
  replicas: 1
  name: zookeeper
  namespace: zookeeper
  reclaimPolicy: Delete
```

As you can see here, all variables have sensible default values, allowing you to get up and running without edits. It's possible to override certain values on the command line when executing Helm. 

With this Helm chart in place, let's move on to configuring ArgoCD. ArgoCD can deploy from various sources, such as Helm repositories or Git repositories. In this case, we're going with the latter. While we're choosing to create everything on the command line for consiseness, you can also perform the same steps in the web interface.

Let's apply what we have:

```bash
argocd app create zookeeper --repo https://github.com/schultyy/koperator-install.git --path zookeeper --dest-namespace zookeeper --dest-server https://kubernetes.default.svc --sync-option Server-Side-Apply=true --sync-option createnamespace=true
argocd app sync zookeeper
```

First, we create a new application and provide some configuration options, such as the git repository, namespace and server.

Next, we `sync` the application, so it starts a deployment run. While the first step completes quickly, the second might take a few minutes to complete succesfully. If you want to confirm successful rollout of all resources inspect [visit the dashboard](https://localhost:8080/applications/argocd/zookeeper?view=tree&resource=).

Now, whenever we make any changes to our ZooKeeper Helm chart, we sync the app and ArgoCD takes care of rolling out the changes automatically.

### Koperator and Kafka

Next on our list is Koperator and Kafka. We'll use [Koperator](https://github.com/banzaicloud/koperator) to spin up and manage Kafka instances.

This Helm Chart, similar to ZooKeeper, makes use of several Custom Resource Definitions (CRDs). These CRDs will allow us later to:

- Define and configure new Kafka clusters
- Define Kafka topics

For Koperator itself, the project's README suggests to install CRDs in a separate step. Therefore, we add all CRDs from <https://github.com/banzaicloud/koperator/releases/download/v0.24.1/kafka-operator.crds.yaml> to this chart into a separate `crd` directory. We need to add the CRDs to the `Chart.yml`:

```yaml
apiVersion: v2
name: koperator
description: A Helm chart to install Koperator
type: application
version: 0.1.0
appVersion: "1.16.0"
dependencies:
  - name: kafka-operator
    repository: https://kubernetes-charts.banzaicloud.com
    version: "0.24.1"
# CRDs are defined here
crds:
  - kafkatopics.crds.yml
  - cruisecontrol.crds.yml
  - kafkauser.crds.yml
  - kafkacluster.crds.yml
```

We also define our dependency to the Koperator Helm Chart.

When the Helm chart gets installed, Helm will automatically install these CRDs for us.

To get Kafka up and running, the main aspect will be the `simplekafkacluster.yml` file in the templates directory. This file also comes from the Koperator README. The only change made to it is replacing some of the values with template variables, so we can parameterize it.

Additionally, this chart also contains a template to create one or more Kafka topics:

```yaml
{{- range .Values.topics }}
---
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: {{ .name }} 
spec:
    clusterRef:
        name: {{ $.Values.kafka.name }}
    name: {{ .name }}
    partitions: {{ .partitions }}
    replicationFactor: {{ .replicationFactor }}
{{- end }}
```

`values.yml` has everything we need to start creating Kafka clusters:

```bash
# Install the Kafka cluster
kafka:
  name: kafka
  readOnlyConfig:
    autoCreateTopics: false

#Setup one or more topics
topics:
  - name: my-topic
    partitions: 1
    replicationFactor: 1
```

With this in place, we are all set to create a new Kafka cluster and the first topic:

```bash
argocd app create koperator --repo https://github.com/schultyy/koperator-install.git --path koperator --dest-namespace kafka --dest-server https://kubernetes.default.svc --sync-option Server-Side-Apply=true --sync-option validate=false --sync-option createnamespace=true
argocd app sync koperator
```

## What's next?

With this workflow in place, from now on you can declaratively deploy and manage your Kafka cluster. With a few modifications to the Helm charts, you can create as many Kafka cluster as you like or need.

Check out the Helm charts used for these examples [here](https://github.com/schultyy/koperator-install/) and make sure to also star the [koperator repository!](https://github.com/banzaicloud/koperator)