Running Kafka on Kubernetes for local development with Helm
Tags: kafka, kubernetes, helm, devops
Published: 2022-01-30 09:41:43

Now that I have posted about running [Kafka](https://kafka.apache.org/documentation/#introduction) on Kubernetes using [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)(PV) and [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims)(PVC), and another post about running it using [Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) it's time to grab that knowledge and have a version of it running with [Helm Charts](https://helm.sh/).

Helm is a package manager for Kubernetes, Helm helps you manage, define, install, and upgrade Kubernetes applications. It promotes re-usability and standardization. 

In this post we will use Helm to convert the Local Kafka setup using Storage Classes I explained in a previous post. So if you haven't seen that post yet I suggest that you read and play with that setup first as it will help you to understand and extend the approach explained here. 

> Helm has many more features than the ones covered in this simple post.   

As usual you can find the source code with instructions in my [Github Repository](https://github.com/mmaia/kafka-local-kubernetes). To clone it: `git clone git@github.com:mmaia/kafka-local-kubernetes.git`, have fun. 


### Pre-reqs, install:

- [Docker](https://docs.docker.com/get-docker/)
- [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
- [Kind](https://kind.sigs.k8s.io/)
- [Helm](https://helm.sh/)
 
## The setup using Helm

In this next section I will just cover the relevant information covering the helm setup and the actual `Kubernetes` files used are basically the same as the ones covered within the [Running Kafka for local development with Storage Class](https://dev.to/thegroo/running-kafka-on-kubernetes-for-local-development-with-storage-class-4oa9) post, so please refer to that article for further details on the Kubernetes setup.

After you clone the repo as described above if you expand the folder `helm` you will find the following structure: 

![Helm kafka local setup tree](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s7mhspqdcb4t98wa4rkp.png)
 
As seen in my previous posts the `kind-config.yaml` file is used to setup the local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). 

`kind.config.yaml` file:

```yaml
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
  - role: worker
    extraPortMappings:
      - containerPort: 30092 # internal kafka nodeport
        hostPort: 9092 # port exposed on "host" machine for kafka
      - containerPort: 30081 # internal schema-registry nodeport
        hostPort: 8081 # port exposed on "host" machine for schema-registry
    extraMounts:
      - hostPath: ./tmp
        containerPath: /var/local-path-provisioner
        readOnly: false
        selinuxRelabel: false
        propagation: Bidirectional
``` 

Then inside the `local-kafka-dev` folder you'll find the main Helm Chart file: `Chart.yaml` and a file to set variables: `values.yaml`.

`Chart.yaml` file: 

```yaml
apiVersion: v2 #helm 3
name: local-kafka-dev
appVersion: "7.0.1"
description: A Helm chart for a local, single node Confluent Kafka, Schema Registry and Zookeeper for development.
version: 1.0.0
type: application
```

`values.yaml` file: 

```yaml
confluent:
  version: 7.0.1
  kafka: confluentinc/cp-kafka
  schemaRegistry: confluentinc/cp-schema-registry
  zookeeper: confluentinc/cp-zookeeper
```

Then inside the `templates` folder you will find all the same Kubernetes files used in the previous post as mentioned before with same minimal changes to use a few of the Helm variables defined in the `values.yaml` file. The only new file in there is the `NOTES.txt` which is used to display a nice message after you run the Helm chart as you will be able to see next while running it. 


## Running it

1. Open a terminal and cd to helm folder
2. Start Kind running: `kind create cluster --config=kind-config.yml`
3. Run `helm install ${pick_a_name} local-kafka-dev`. i.e - `helm install local-kafka local-kafka-dev`
4. When done stop kafka setup using `helm uninstall ${name_picked_step_3}`, you may also stop Kind if you want: `kind delete cluster`

That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes and managed with Helm Charts which you can reach from your application running on your developer machine or IDE.

Enjoy! 


Photo by <a href="https://unsplash.com/@thisisengineering?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">ThisisEngineering RAEng</a> on <a href="https://unsplash.com/s/photos/developer?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>