Installing Apache Kafka (Confluent edition)
Tags: tutorial, machinelearning, beginners
Published: 2022-02-04 17:28:52


## Table of Contents
  - [Download](#download)
  - [1. Start zookeeper](#1-start-zookeeper)
  - [2. Create a Kafka Broker](#2-create-a-kafka-broker)
  - [3. Create a Kafka topic](#3-create-a-kafka-topic)
    - [Listing the topics](#listing-the-topics)
  - [4. Create a Kafka Producer](#4-create-a-kafka-producer)
  - [5. Create a Kafka Consumer](#5-create-a-kafka-consumer)
  - [LOGS (DEBUG)](#logs-debug)

## Download
```bash
curl -O http://packages.confluent.io/archive/5.4/confluent-community-5.4.0-2.12.zip
```


## 1. Start zookeeper
Navigate to the folder where confluent is installed. For me it was `/home/divyaksh/Downloads/software/confluent-5.4.0`

Linux
```bash
bin/zookeeper-server-start etc/kafka/zookeeper.properties
```
Windows
```batch
bin\windows\zookeeper-server-start.bat etc\kafka\zookeeper.properties
```
**Error**

Occurs if something is missing. I don't know what do do if this happens in linux.
```
Classpath is empty. Please build the project first using 'gradlew jarAll'
```
**Solution**
- Go to `bin/kafka-run-class` 
- Add some code. Go to a line above this script (**do not delete anything**)
```batch
rem Classpath addition for core
```
add the below code
```batch
rem Classpath addition for LSB style path
if exist %BASE_DIR%\share\java\kafka\* (
    call :concat %BASE_DIR%\share\java\kafka\*
)
```

## 2. Create a Kafka Broker
Linux
```bash
bin/kafka-server-start etc/kafka/server.properties
```
Windows
```batch
bin\windows\kafka-server-start.bat etc\kafka\server.properties
```
## 3. Create a Kafka topic
Linux
```bash
bin/kafka-topics --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
```
Windows
```batch
bin\windows\kafka-topics.bat --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
```
**Warning**
```
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both
```
Thus, create topics with names not haveing `.` or `_`.


`--partition` n:
- Number of partitions
  - How much storage is needed (how big my message is)
  - Nature of parallel processing

Here we have a simple workload of a single producer and a single consumer (n = 1)

`--replication-factor` f : number of copies of each partition in the cluster. You create multiple brokers and a in each you make a copy of the messages. Higher the replication factor, more number of copies. No sense of keeping f=10 when there is only 1 broker. in our example f = 1

`--bootstrap-server`: it takes 2 arguments `ip-address` and `port-number`. In our example we are running the zookeeper on local machine and the default port number of zookeeper is `9092`.

### Listing the topics
```
$ bin/kafka-topics --list --bootstrap-server localhost:9092
__confluent.support.metrics
sampleTopic
sample_topic
```

## 4. Create a Kafka Producer
Linux
```bash
bin/kafka-console-producer --topic sampleTopic --broker-list localhost:9092  < <path to some file>
```
Windows
```batch
bin\windows\kafka-console-producer.bat --topic sampleTopic --broker-list localhost:9092 < <path to some file>
```

`--broker-list`: same as `--bootstrap-server`.


**NOTICE**
```
$ bin/kafka-console-producer --help
This tool helps to read data from standard input and publish it to Kafka.
```
So you have to input the file using `<` operator before giving `<path to some file>`

## 5. Create a Kafka Consumer
Linux
```bash
bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
```
Windows
```batch
bin\windows\kafka-console-consumer.bat --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
```

**OUTPUT**
```
$ bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
To: divyaksh.shukla@gmail.com
From: ideapad.divyaksh@gmail.com
Subject: This is a test email from msmtp

MSMTP says hello
```
I had given a test email file as an input to the producer.


## LOGS (DEBUG)
You can find the logs at `/tmp/kafka-logs/` to debug kafka. Windows `C:\tmp\kafka-logs\`