Setting up a local Apache Kafka instance for testing
Tags: tutorial, kafka, docker
Published: 2022-09-01 00:13:43

---
title: Setting up a local Apache Kafka instance for testing
published: true
date: 2022-09-01 00:00:00 UTC
tags: tutorial, kafka, docker
canonical_url: https://sahansera.dev/setting-up-kafka-locally-for-testing/
cover_image: https://sahansera.dev/static/e7cca1a99fb3be015cc50af41d132c01/dcbba/setting-up-kafka-locally-for-testing.webp
---

I recently started digging into Apache Kafka for a project I’m working on at work. The more I started looking into it I realized I need to get myself a local Kafka cluster to play around with. This article walks you through that process and hopefully save your some time 😄.

If you don’t know what Kafka is, [here’s a great video](https://www.youtube.com/watch?v=FKgi3n-FyNU&ab_channel=Confluent) from Confluent.

> 💡 You can follow along by cloning the repo I have created over [here](https://github.com/sahansera/kafka-docker).

## Installing the binaries

There’s a quickstart [guide](https://kafka.apache.org/quickstart) over at the official docs. To be honest, I didn’t want to install anything locally in order to keep the Kafka instance fairly separate my dev environment, hence went with the Docker way which you will find below.

## The Docker way

This is by far the easiest I have found and saves you ton of time. Once you have cloned the repo just spin it up!

```sh
docker-compose up -d
```

If you want to know what’s going on in here let’s take a look at the docker-compose.yaml file.

[docker-compose.yaml](https://github.com/sahansera/kafka-docker/blob/main/docker-compose.yaml)

```yaml
---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.0.1
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
```

1. We spin up a [Zookeeper](https://zookeeper.apache.org/) instance at port `2181` internally within the Docker network (i.e. not accessible from the host)
2. Next we spin up our broker which is the Kafka instance which has a dependency on Zookeeper. This is exposed at port `9092` and accessible from the host.

Everything else is set to default. If you want to learn what each environment variable does, head over [here](https://docs.confluent.io/platform/current/kafka/multi-node.html). You can add multiple brokers (Kafka instances) if you want in the same docker-compose file.

This approach is quite beneficial if you want to stand up an instance for testing on the go or even in a CI environment.

## Interacting with the cluster

Before we publish anything, let’s spin up a consumer so that we know our setup is working. To do that, run the following command from the CLI.

```sh
docker exec --interactive --tty broker \
kafka-console-consumer --bootstrap-server localhost:9092 \
                       --topic example-topic \
                       --from-beginning
```

Now let’s create some messages and see them in action!

```
docker exec --interactive --tty broker \
kafka-console-producer --bootstrap-server localhost:9092 \
                       --topic example-topic
```

This is what it will look like from the terminal. To your left is the `producer` and `consumer` to your right hand side.

![setting-up-kafka-locally-for-testing-1.png](https://sahansera.dev/static/018146d435429a3497576c80172e22e3/5a190/setting-up-kafka-locally-for-testing-1.png "setting-up-kafka-locally-for-testing-1.png")

## A GUI perhaps?

Not a fan of the CLI? There’s many tools out there. I wanted to find a nice, lightweight, OSS solution for this. I decided to go ahead with [Tools for Apache Kafka for VS Code](https://marketplace.visualstudio.com/items?itemName=jeppeandersen.vscode-kafka).

Once you have that going you could interact with it from the side bar. There will be a new icon with Kafka’s logo on it!

![setting-up-kafka-locally-for-testing-2.png](https://sahansera.dev/static/5d851500362a638d86358d395c19a699/5a190/setting-up-kafka-locally-for-testing-2.png "setting-up-kafka-locally-for-testing-2.png")

Here’s what it looks like when it finds your instance.

![setting-up-kafka-locally-for-testing-3.png](https://sahansera.dev/static/1de368b211cb45a6fe43f6591f4dedd7/084e2/setting-up-kafka-locally-for-testing-3.png "setting-up-kafka-locally-for-testing-3.png")

[producer.kafka](https://github.com/sahansera/kafka-docker/blob/main/producer.kafka)

```json
PRODUCER keyed-message
topic: example-topic
key: mykeyq
record content

###

PRODUCER non-keyed-json-message
topic: example-topic
{
    "type": "my_test_event-{{random.number}}"
}
```

[consumer.kafka](https://github.com/sahansera/kafka-docker/blob/main/consumer.kafka)

```json
CONSUMER consumer-group-id
topic: example-topic
partitions: 0
from: 1
```

You can invoke the producers and consumers right within VS Code, which is pretty easy and cool!

![setting-up-kafka-locally-for-testing-4](https://sahansera.dev/static/e4fd10918691b2d4db95f2f18502b26f/5a190/setting-up-kafka-locally-for-testing-4.png "setting-up-kafka-locally-for-testing-4")

## Conclusion

Welll that’s it! Standing up a docker container certainly did save me some time just for testing purposes. Hope you enjoyed this post and see you next time 👋

## References

1. [https://developer.confluent.io/quickstart/kafka-docker/](https://developer.confluent.io/quickstart/kafka-docker/)
2. [https://marketplace.visualstudio.com/items?itemName=jeppeandersen.vscode-kafka](https://marketplace.visualstudio.com/items?itemName=jeppeandersen.vscode-kafka)
3. [https://docs.confluent.io/platform/current/kafka/multi-node.html](https://docs.confluent.io/platform/current/kafka/multi-node.html)