How to Set Up and Run Kafka on K8s
Tags: kubernetes, kafka, tutorial, devops
Published: 2020-05-14 16:29:23

Apache Kafka is a leading open-source distributed streaming platform first developed at LinkedIn. It consists of several APIs such as the Producer, the Consumer, the Connector and the Streams. Together, those systems act as high-throughput, low-latency platforms for handling real-time data. This is why Kafka is preferred among several of the top-tier tech companies such as Uber, Zalando and AirBnB.

Quite often, we would like to deploy a fully-fledged Kafka cluster in Kubernetes, just because we have a collection of microservices and we need a resilient message broker in the center. We also want to spread the Kafka instances across nodes, to minimize the impact of a failure.

In the following tutorial, the Platform9 technical team presents an example Kafka deployment within the Platform9 Free Tier Kubernetes platform, backed up by some DigitalOcean droplets.

Let’s get started.

##Setting Up the Platform9 Free Tier Cluster##
Below are the brief instructions to get you up and running with a working Kubernetes Cluster from Platform9:

 1. Signup with [Platform9](https://platform9.com/signup/)
 2. Click the Create Cluster button and inspect the instructions. We need a server to host the Cluster.
 3. Create a few Droplets with at least 3gb Ram and 2vCPU’s. Follow instructions to install the **pf9cli tool** and prepping the nodes.

<code class="language-plaintext highlighter-rouge">$ bash <(curl -sL http://pf9.io/get_cli)</code>
<code class="language-plaintext highlighter-rouge">$ pf9ctl cluster prep-node -i <nodeIP>  </code>

 4. Switch to the Platform9 UI and click the refresh button. You should see the new nodes in the list. Assign the first node as a master and the other ones as workers.
 5. Leave the default values in the next steps. Then create the cluster.
 6. Wait until the cluster becomes healthy. It will take at least 20 minutes to finish.
 7. Click on the **API Access** tab and select to download the kubeconfig button:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/i7pcubs4qse5o7xfipak.png)

 8. Once downloaded, export the config and test the cluster health:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ export KUBECONFIG=/Users/itspare/Theo/Projects/platform9/example.yaml
$ kubectl cluster-info

Kubernetes master is running at https://134.122.106.235
CoreDNS is running at https://134.122.106.235/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://134.122.106.235/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

</code></pre></div>    </div>

##Creating Persistent Volumes##
Before we install Helm and the Kafka chart, we need to create some persistent volumes for storing Kafka replication message files.

This step is crucial to be able to enable persistence in our cluster because without that, the topics and messages would disappear after we shutdown any of the servers, as they live in memory.

In our example, we are going to use a local file system, Persistent Volume (PV), and we need one persistent volume for each Kafka instance; so if we plan to deploy three instances, we need three PV’s.

Create and apply first the Kafka namespace and the PV specs:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ cat namespace.yml
---
apiVersion: v1
kind: Namespace
metadata:
  name: kafka

$ kubectl apply -f namespace.yml
namespace/kafka created


$ cat pv.yml
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-pv-volume-2
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-pv-volume-3
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

$ kubectl apply -f pv.yml

</code></pre></div>    </div>

If you are using the Kubernetes UI, you should be able to see the PV volumes on standby:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8hag1ucjnsicvnyycb6d.png)

##Installing Helm##
We begin by installing Helm on our computer and installing it in Kubernetes, as it’s not bundled by default.

First, we download the install script:

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh
</code></pre></div>    </div>

Make the script executable with chmod:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ chmod u+x install-helm.sh
</code></pre></div>    </div>

Create the tiller service account:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ kubectl -n kube-system create serviceaccount tiller
</code></pre></div>    </div>

Next, bind the tiller serviceaccount to the cluster-admin role:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
</code></pre></div>    </div>

Now we can run helm init:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ helm init --service-account tiller
</code></pre></div>    </div>

Now we are ready to install the Kafka chart.

##Deploying the Helm Chart##
In the past, trying to deploy Kafka on Kubernetes was a good exercise. You had to deploy a working Zookeeper Cluster, role bindings, persistent volume claims and apply correct configuration.

Hopefully for us, with the use of the [Kafka Incubator Chart](https://github.com/helm/charts/tree/0ca37cc106467190bd705aff647d2c7361e1d6f1/incubator/kafka), the whole process is mostly automated (with a few quirks here and there).

We add the Helm chart:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
</code></pre></div>    </div>

Export the chart values in a file:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ curl https://raw.githubusercontent.com/helm/charts/master/incubator/kafka/values.yaml > config.yml
</code></pre></div>    </div>

Carefully inspect the configuration values, particularly around the parts about persistence and about the number of Kafka stateful sets to deploy.

Then install the chart:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ helm install --name kafka-demo --namespace kafka incubator/kafka -f values.yml --debug
</code></pre></div>    </div>

Check the status of the deployment
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ helm status kafka-demo
LAST DEPLOYED: Sun Apr 19 14:05:15 2020
NAMESPACE: kafka
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                  DATA  AGE
kafka-demo-zookeeper  3     5m29s

==> v1/Pod(related)
NAME                    READY  STATUS   RESTARTS  AGE
kafka-demo-zookeeper-0  1/1    Running  0         5m28s
kafka-demo-zookeeper-1  1/1    Running  0         4m50s
kafka-demo-zookeeper-2  1/1    Running  0         4m12s
kafka-demo-zookeeper-0  1/1    Running  0         5m28s
kafka-demo-zookeeper-1  1/1    Running  0         4m50s
kafka-demo-zookeeper-2  1/1    Running  0         4m12s

==> v1/Service
NAME                           TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
kafka-demo                     ClusterIP  10.21.255.214  <none>       9092/TCP                    5m29s
kafka-demo-headless            ClusterIP  None           <none>       9092/TCP                    5m29s
kafka-demo-zookeeper           ClusterIP  10.21.13.232   <none>       2181/TCP                    5m29s
kafka-demo-zookeeper-headless  ClusterIP  None           <none>       2181/TCP,3888/TCP,2888/TCP  5m29s

==> v1/StatefulSet
NAME                  READY  AGE
kafka-demo            3/3    5m28s
kafka-demo-zookeeper  3/3    5m28s

==> v1beta1/PodDisruptionBudget
NAME                  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
kafka-demo-zookeeper  N/A            1                1                    5m29s
</code></pre></div>    </div>

During this phase, you may want to navigate to the Kubernetes UI and inspect the dashboard for any issues. Once everything is complete, then the pods and Persistent Volume Claims should be bound and green.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/11opfsstpiohr2hea5lh.png)

Now we can test the Kafka cluster.

##Testing the Kafka Cluster##
We are going to deploy a test client that will execute scripts against the Kafka cluster.

Create and apply the following deployment:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ cat testclient.yml

apiVersion: v1
kind: Pod
metadata:
  name: testclient
  namespace: kafka
spec:
  containers:
  - name: kafka
    image: solsson/kafka:0.11.0.0
    command:
      - sh
      - -c
      - "exec tail -f /dev/null"

$ kubectl apply -f testclient
</code></pre></div>    </div>

Then, using the testclient, we create the first topic, which we are going to use to post messages:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-topics.sh --zookeeper kafka-demo-zookeeper:2181 --topic messages --create --partitions 1 --replication-factor 1
Created topic "messages".
</code></pre></div>    </div>
Here we need to use the correct hostname for zookeeper cluster and the topic configuration.

Next, verify that the topic exists:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-topics.sh --zookeeper kafka-demo-zookeeper:2181 --list
Messages
</code></pre></div>    </div>

Now we can create one consumer and one producer instance so that we can send and consume messages.

First create one or two listeners, each on its own shell:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka-demo:9092 --topic messages --from-beginning
</code></pre></div>    </div>

Then create the producer session and type some messages. You will be able to see them propagate to the consumer sessions:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-producer.sh --broker-list kafka-demo:9092 --topic messages
>Hi
>How are you?
>Hope you're well
>

<switching on each consumer you will see>

Hi
How are you?
Hope you're well
</code></pre></div>    </div>

##Destroying the Helm Chart##
To clean up our resources, we just destroy the Helm Chart and delete the PVs we created earlier:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
$ helm delete kafka-demo --purge
$ kubectl delete -f pv.yml -n kafka
</code></pre></div>    </div>

##Next Steps##
Stay put for more tutorials showcasing common deployment scenarios within Platform9’s fully-managed Kubernetes platform.