Kafka Quickstart
Tags: kafka
Published: 2024-03-13 13:01:05

# Getting Started on Kafka (Windows)

### 1. Start up zookeeper server
Open a command prompt in C:\kafka and enter this command:

```bash
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ljwvjylp2hf13201kvz.png)

Zookeeper is running on Port 2181
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4b1gy4zt0uaas4fe78ch.png)

### 2. Start the Kafka server
Open a new command prompt and start the Kafka server:
```bash
.\bin\windows\kafka-server-start.bat .\config\server.properties
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ew7glew99toc12292de.png)

### 3. Create a topic
Open a new command prompt in C:\kafka\bin\windows:
```bash
kafka-topics.bat --create --bootstrap-server localhost:9092 --topic mytopic
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1bmgxsjo2xy4getoqxop.png)

### 4. Create a Producer Console
```bash
kafka-console-producer.sh --broker-list <BROKER_LIST> --topic <TOPIC_NAME>
```
Open another new command prompt in C:\kafka\bin\windows:
```bash
kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
```
You should see a > prompt when the producer is ready. 

### 5. Create a Consumer Console
```bash
kafka-console-consumer.sh --bootstrap-server <BROKER_LIST> --topic <TOPIC_NAME>
```
Open another new command prompt in C:\kafka\bin\windows:
```bash
kafka-console-consumer.bat --topic mytopic --bootstrap-server localhost:9092 --from-beginning
```

### 6. Paste Data to Producer and Observe Consumer

Sample data for use:
```bash
{"Name":"Kenneth","Age":"35","Gender":"Male"}
{"Name":"Gemma","Age":"25","Gender":"Female"}
{"Name":"Amy","Age":"15","Gender":"Male"}
```
Referencing the image below, once you paste a row of data into the producer terminal (left), the data appears on the consumer terminal (right) almost instantaneously.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0cf0tbkewtznvu2mbt2b.png)

# In Git Bash/ Linux
Similar commands collated from [kafka quickstart](https://kafka.apache.org/quickstart)

```bash
# Download and extract
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0

# Start the ZooKeeper service
bin/zookeeper-server-start.sh config/zookeeper.properties
# Start and hide logs on terminal
bin/zookeeper-server-start.sh config/zookeeper.properties > /dev/null 2>&1 &
# verify that Zookeeper is indeed running
ps -aef

# Start the Kafka broker service in another terminal
bin/kafka-server-start.sh config/server.properties

# Create a topic in another terminal
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

# Show details of a topic
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic: quickstart-events        TopicId: z_zqwwkiT6ueXVJxi3EAdA PartitionCount: 1       ReplicationFactor: 1     Configs:
        Topic: quickstart-events        Partition: 0    Leader: 0       Replicas: 0     Isr: 0

# Run Producer Console in another terminal
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092

# Run Consumer Console in another terminal
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

# Write event to topic at Producer console
This is my first message
This is my second message
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9x4lgoxa49kmpac7e2ph.png)

## Demonstrating 3 Brokers Replica
```bash
# Create 2 other brokers
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
ll config/serv*
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9l1ptjdw0s3ttbyhf5ly.png)

Update these 3 areas in server-1 and server-2 file:
```bash
# SERVER-1
# id of the broker
broker.id=1
# broker port
listeners=PLAINTEXT://:9093
# log files directories 
log.dirs=/tmp/kafka-logs-1

# SERVER-2
# id of the broker
broker.id=2
# broker port
listeners=PLAINTEXT://:9094
# log files directories 
log.dirs=/tmp/kafka-logs-2
```
Now start all 3 brokers
```bash
bin/kafka-server-start.sh config/server.properties
bin/kafka-server-start.sh config/server-1.properties
bin/kafka-server-start.sh config/server-2.properties
```
Now Create a topic with 2 partitions and 3 replication factor across 3 brokers
```bash
bin/kafka-topics.sh --create --topic TestTopicXYZ --partitions 2 --replication-factor 3 --bootstrap-server localhost:9092, localhost:9093, localhost:9094
```
Lastly describe the topic to examine:
```bash
bin/kafka-topics.sh --describe --topic TestTopicXYZ --bootstrap-server localhost:9092,localhost:9093, localhost:9094

Topic: TestTopicXYZ     TopicId: 65thxHOeTIGSJJ5L_jl-XQ PartitionCount: 2       ReplicationFactor: 3    Configs:
        Topic: TestTopicXYZ     Partition: 0    Leader: 0       Replicas: 0,2,1 Isr: 0,2,1
        Topic: TestTopicXYZ     Partition: 1    Leader: 2       Replicas: 2,1,0 Isr: 2,1,0

```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/01dme63wwlbnldskfsoi.png)


## What is Kafka?
1. Publish/Subscribe systems

With Publish/Subscribe systems, each event can be processed by multiple consumers who are listening to the topic. So multiple consumers can all get the same messages from the producer.

Pub/Sub systems introduce the concept of a topic. We can have multiple topics (to categorize your messages) and multiple producers writing to each topic and multiple consumers reading from a topic.

2. Distributed
Kafka is like a messaging system. Kafka Cluster is made up of **more than 1** Kafka servers. Each Kafka server is referred to as a **Broker**. A Kafka cluster will have multiple brokers, therefore it is distributed application.

3. Fault-Tolerant
In Kafka cluster, messages are replicated in multiple brokers. A message published in Broker 1 will also be published in Broker 3. Therefore Kafka is fault-tolerant as message is replicated and not lost even if Broker 1 is down.

4. Language Agnostic
Data transferred (called Messages) are byte arrays so you can use JSON or a data format like Avro. Messages are immutable and have a timestamp, a value and optional key/headers

## Kafka Architecture
Every Kafka ecosystem can have multiple producers/publishers.
A Kafka system consists of several components:

- **Producer** - Any service that pushes messages to Kafka is a producer.

- **Consumer** - Any service that consumes (pulls) messages from Kafka is a consumer. There can be multiple Consumer groups in the Kafka ecosystem. Consumers must be associated in a Consumer groups.

- **Kafka Brokers** - the kafka cluster is composed of a network of nodes called brokers. Each machine/instance/container running a Kafka process is called a broker. 

In a Kafka cluster, the individual servers are referred to as brokers. Brokers will handle:

1. Receiving Messages - accepting messages from producers
2. Storing Messages - storing them on disk organized by topic. Each message has a unique offset identifying it.
3. Serving Messages - sending messages to consumer services when the consumer requests it

- **Topics** - A Kafka server has multiple topics. Every topics can have multiple partitions. Messages in Kafka are stored in topics, which are the core structure for organizing and storing your messages. You can think of a topic as a log file that stores all the messages and maintains their order. New message are appended to the end of the topic.

## [Offset Explorer](https://www.kafkatool.com/download.html)

A UI tool for Kafka to help us visualize kafka clusters, topics and offset etc.

Thank you!