Setup Kafka for development
Tags: tutorial, java, docker
Published: 2019-07-04 09:46:59

# Introduction
When developing applications using Apache Kafka, I had the need to make quick tests like publishing/consuming messages, check its format or just observe what was inside a topic. 

In this article I will show how I setup a local productive environment for Apache Kafka development using Docker. There's a demo application in [Github](https://github.com/fabiothiroki/spring-boot-kafka-docker).

# Install dependencies
To run our local Kafka and its tools locally we need to install:
- [Docker CE](https://docs.docker.com/install/)
- [Docker Compose](https://docs.docker.com/compose/install/)

If you are on Mac OS or Windows you need to install also:
- [Virtual Box](https://www.virtualbox.org/wiki/Downloads)
- [Docker Machine](https://docs.docker.com/machine/install-machine/)


# Docker image
For running Kafka, its dependencies and tools, I've chosen the [fast-data-dev](https://hub.docker.com/r/landoop/fast-data-dev/) image, because it already contains Kafka, Zookeeper and Kafka Topics UI. 


# Exposing Kafka through Docker
Our Kafka instance will run outside Docker internal network, so we need to expose an IP address for our application connection. If you're on Linux you can use the `localhost` address directly.

The chosen Docker image mentionend above already contains the steps necessary to create the `docker-machine` and set the environment variables:
```
docker-machine create --driver virtualbox --virtualbox-memory 4096 landoop
```
```
eval $(docker-machine env landoop)
```

By running `docker-machine ip` you can see the machine's IP (usually 192.168.99.100).

# Docker compose
I like to use docker-compose files to register `docker` commands on a `git` repository so I don't need to type and remember what to type everytime I want to run a docker image.

For example, instead of running:
```
docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 \
       -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=192.168.99.100 \
       landoop/fast-data-dev:latest
```

I just need to create a file called `docker-compose.yml` containing:
```
version: '3'
services:
  kafka:
    image: landoop/fast-data-dev:latest
    ports:
      - "9092:9092"
      - "8081:8081"
      - "8082:8082"
      - "8083:8083"
      - "2181:2181"
      - "3030:3030"
      - "9581-9585:9581-9585"
    environment:
      - ADV_HOST=192.168.99.100 // Use your 'docker-machine ip' or 'localhost' if linux
 ```

And run it by:
```
docker-compose up
```

# Test
I've provided a [sample application](https://github.com/fabiothiroki/spring-boot-kafka-docker) to test the Kafka connection using the same docker configuration above. It's written in Java using [Spring Kafka](https://spring.io/projects/spring-kafka) project, but it should work for any language or library. You can also use the [command line interface](https://kafka.apache.org/quickstart) to publish simple messages.  

After running the demo application provided, you just need to access the endpoint http://localhost:8080/publish?message=sample_message to publish a message with content `sample_message` in `test` topic.

Now to check your topic messages, you can access the address where [Kafka Topics UI](https://github.com/Landoop/kafka-topics-ui) is running on address `<docker-ip>:3030/kafka-topics-ui` (i.e.: http://192.168.99.100:3030/kafka-topics-ui/):

![](https://thepracticaldev.s3.amazonaws.com/i/zqjm27mk2jargez25053.png)

# Conclusion
With this simple setup and a single `docker-compose.yml` file, you are ready to publish and consume Kafka messages and start developing your application.