Installing Apache Kafka on Windows (For Development)
Tags: kafka, java, windows, zookeeper
Published: 2021-06-04 15:33:27

## Prerequisites

Have `Java 8` or greater installed on your machine. Open the command prompt and execute this to confirm it is installed:

```
java -version
```

Meanwhile I'm using `Java 11`

## Download The Apache Kafka Binary

Go to https://kafka.apache.org/downloads, if you're using `Scala` on your machine then find the matched version, if not download any latest available version Kafka binary. 

![kafka-download-page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vpj0p4p6pod0vr14dmdh.png)

Let's say I got `kafka_2.12-2.8.0.tgz`. From here on, I will use my home directory `C:\Users\dendi` so you can use yours to follow along. I extracted it into `C:\Users\dendi` and renamed the folder from `kafka_2.12-2.8.0` into `kafka`. So the complete path is `C:\Users\dendi\kafka`.

Here is the structure inside the kafka folder:

```
kafka
    |__ bin
    |__ config
    |__ libs
    |__ licenses
    |__ logs
    |__ site-docs
    |__ LICENSE (file)
    |__ NOTICE (file)

```

## Provide Data Folder For Zookeeper and Kafka

In the kafka directory, let's create another folder named `data` and create two folders named `zookeeper` and `kafka` inside the `data` folder.

```
kafka
    |__ bin
    |__ config
    |__ data
        |__ kafka
        |__ zookeeper
    |__ libs
    |__ licenses
    |__ logs
    |__ site-docs
    |__ LICENSE (file)
    |__ NOTICE (file)

```

## Configure Zookeeper and Kafka Server

Find the `config\zookeeper.properties` file, open it with code editor and find the `dataDir` variable and fill it with:

```
...

dataDir=data/zookeeper

...
```

And then find the `config\server.properties` file, open it with code editor and find the `log.dirs` variable and fill it with:

```
...

log.dirs=data/kafka

...
```

> If you're using app like Windows Terminal, Terminus or Hyper, it's very useful for managing consoles because we will open 4-5 consoles later.

## Running Zookeeper

Now let's open our 1st command prompt, go to kafka directory, and execute this command:

```
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
```

if the result similar like this, then it's already running:

```
[2021-06-06 11:09:01,739] INFO Server environment:java.io.tmpdir=C:\Users\dendi\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,739] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,741] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,742] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,743] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,743] INFO Server environment:user.name=dendi (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,743] INFO Server environment:user.home=C:\Users\dendi (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,744] INFO Server environment:user.dir=C:\Users\dendi\kafka (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,744] INFO Server environment:os.memory.free=492MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,745] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,745] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,747] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,748] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,755] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir data\zookeeper\version-2 snapdir data\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,802] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-06-06 11:09:01,806] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-06-06 11:09:01,810] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-06-06 11:09:01,834] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-06-06 11:09:01,841] INFO Reading snapshot data\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2021-06-06 11:09:01,891] INFO Snapshotting: 0x8d to data\zookeeper\version-2\snapshot.8d (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-06-06 11:09:01,919] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2021-06-06 11:09:01,931] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
```

## Running Kafka

Let's open our 2nd command prompt, go to kafka directory and execute this command:

```
.\bin\windows\kafka-server-start.bat .\config\server.properties
```

Then your kafka server should be running, hopefully there is no error log in console result. But we will test if it actually running well or not by creating a topic in next step.

## Creating Topic

Let's open our 3rd command prompt, go to kafka directory, we will create a simple topic named `test` with this command:

```
.\bin\windows\kafka-topics.bat --create --topic test --bootstrap-server localhost:9092
```

We could tell if it was successfuly created when we see this result:

```
Created topic test.
```

## Running Consumer

We could use the same 3rd command prompt after we creating topic, now we will use the built-in kafka consumer. We already inside kafka folder, then execute this command to start the consumer for our created `test` topic:

```
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
```

After executing it, you will see absolutely nothing because the consumer is running and streaming the `test` topic and there is no data available at the moment. Don't close this command prompt, we will see if it's actually working if we produce a message.

## Running Producer

Let's open our 4th command prompt, go to kafka directory, and let's run the built-in producer for the `test` topic by this command:

```
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
```

It will open interactive session that we can type anything in it and hit enter to produce the message to the topic. Try type anything and hit enter, see the typed message is showed in the consumer (3rd command prompt). If you see it, then the kafka is running perfectly and you're ready to use it for your project development.

_______________________________________

```
version used:

java version "11.0.10" 2021-01-19 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)

kafka 2.12-2.8.0

```