Basic CLI Implementation of Producer & Consumer using - Apache Kafka Broker & Zookeeper
Tags: apachekafka, zookeeper
Published: 2021-06-27 07:07:02


Signed in as @priyankanandula

What's up, programmers :question: Nothing is ever too late to learn :sunglasses:

In today's `blog`, we'll look at how to use <u>_Apache Kafka Broker and  Zookeeper_</u> to send and consume messages using **Producer and Consumer**.

_Isn't it fascinating :question:_

Let's start with a brief overview of `Kafka` before moving on to _implementation_.

##Apache Kafka:

<strong>`Apache Kafka`</strong> is a <u>publish-subscribe</u> based durable messaging system. A messaging system sends messages between _processes, applications, and servers_.

######These are four main parts in a Kafka system:

<dl>
  <dt><b>Broker</b></dt>
  <dd> - Handles all requests from <em>clients (produce, consume, and metadata)</em> and keeps data replicated within the cluster. There can be one or more brokers in a <u><em>cluster</em></u>.</dd>

  <dt><b>Zookeeper</b></dt>
  <dd> - Keeps the state of the <em>cluster (brokers, topics, users)</em>.</dd>

<dt><b>Producer</b></dt>
  <dd> - Sends records to a <em>broker</em>.</dd>

<dt><b>Consumer</b></dt>
  <dd> - Consumes batches of records from the <em>broker</em>.</dd>
</dl>


</br>
In this `blog`, I will not delve further into the _theoretical aspects_. So, let's get started with the implementation :point_down:

###_Implementation steps_ : 



> NOTE : Java 8 or higher must be installed on your computer.


</br>
1. Download and Install _Apache Kafka(Windows)_.

2. Set up the _kafka environment path variable_.

3. Inside the Kafka folder, creating a data folder.

4. Modify the _properties file's_ configuration parameters.

5. Getting the Zookeeper and Kafka instances running.

6. Creating a _kafka topic_.

7. Instantiate  a _Console Producer_.

8. Instantiate  a _Console Consumer_.

-----

#### 1. Download and Install Apache Kafka(Windows):

</br>
 *  Go through the link --> [https://kafka.apache.org/downloads].

 * Click on the link in the image below.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u2zgath7h2zplwawmkb8.png)
</br>


* Download the <b>kafka_2.13-2.8.0.tgz</b> file. 

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xdotlxtgt3oq7ziwdkze.png)

* After you've downloaded the file, unzip it with a tool like _7-zip or winRAR_.

-----

#### 2. Set up the kafka environment path variable :

Copy and paste the _bin/windowd folder path_ into the `System Environment variables`.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/clgnrydo3q9st4vwxzuj.png)
 

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kzvstag1xhq3272jyf78.png)

-----

#### 3. Inside the Kafka folder, creating a data folder :

* Create a new folder <u>data</u> inside the _kafka_2.13-2.8.0_ folder.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k3b29v19evmpd7tay9mj.png)

* Inside data folder --> Create a two new folders as <u>kafka</u> & <u>zookeeper</u>

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sl2cgjcq76wq66bpzam6.png)

-----

####4. Modify the properties file's configuration parameters: 

* Inside config folder --> Open the <u>**zookeeper.properties**</u> file with notepad++ or any editor.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dqy61lk3l2xk2kjoi990.png)

* In that file change the value for `dataDir` property to the <u>_zookeeper folder path_</u>  which we have create in earlier step and _Save it_.
 

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9k6eg0c97ov5gjo8s2h9.png)
 
* Again inside config folder --> Open the <u>**server.properties**</u> file with notepad++ or any editor.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7se1ihkr9901qgekt249.png)

*  In that file change the value for `dataDir` property to the <u>_kafka folder path_</u>  which we have create in earlier step and _Save it_.
 
######Also add the following two properties in the same file.

```
listeners = PLAINTEXT://localhost:9092
auto.create.topics.enable=false
```

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/81ph7jf9lk2vonspubbx.png)

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l3hd17egc8dliorfmf7s.png)

> NOTE : Remember to use <b>'/'</b> _forward slash_ for `path` otherwise it will not work.

> C:/kafka_2.13-2.8.0/data/kafka

-----

####5. Getting the _Zookeeper and Kafka_ instances running : 

###### Start up the Zookeeper 

> NOTE : Please make sure you're in the _kafka folder_ in `CMD` before proceeding.

To start the _Zookeeper instance_, use the **command** below.

```
zookeeper-server-start.bat config\zookeeper.properties
```

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cyg9kcryvlcjoar54ks2.png)

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fjp5lshm9gf1rt2u1fue.png)

> If you can see these logs, it signifies that the Zookeeper instance was running successfully in `Port No - 2181`.

###### Start up the Kafka

Now open the new command terminal.
</br>

To start the _Kafka server_, use the **command** below.

```
kafka-server-start config\server.properties
```
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1i3s9bikbgl76x9vd5oh.png)

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lhbzomffjmlhn0hjrmo7.png)

> If you can see these logs, it signifies that the `Kafka` instance was running successfully in `Port No - 9092`.

> `kafka` _id_ started with _id zero_ and might have noticed some logs ran in the _zookeeper instance_ so thats a signal that <u>broker registered itself with the zookeeper</u>.

-----

#### 6. Creating a _kafka topic_ : 
</br>

The very _first step_ is we are going to create the <u>Kafka topic</u>.

> NOTE : Please make sure that both **Zookeeper & Kafka** instances are _up and running_ in your local.

###### Open new `CMD terminal` and use the command below to create a _Kafka Topic_ 

```
kafka-topics.bat --create --topic test-topic -zookeeper localhost:2181 --replication-factor 1 --partitions 4
```

* <b>_kafka-topics.bat_</b> is the _command_ that we'll use. This is a batch file for just creating the `topic`, and then we'll pass the _required arguments_ to that **batch file**.

* _-- create_ followed by that we provide the <u>_topic name_</u> and then will provide the <u>_zookeeper instance_</u>.

* After that we have a new _attribute_ called `Replication factor` as '1' because we just have have only _one broker running in our machine_.

* After that we are passing the number of <u>_paritions_</u>. 

* Normally when you provide a `_create topic command _`,the command will be _directed to the zookeeper_,from the _zookeeper_ it takes care of _redirecting  the request to the Kafka Broker_.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ye1i1obsugw798fspg27.png)

> once you see this message the **topic got created successfully**

</br>

Next steps we are going to do start _Producing_  and _Consuming_ `messages` :point_down:.

-----

####7.Instantiate  a _Console Producer_ :

######Instantiate console producer without key.Open new `CMD terminal` and use the command below to create a Producer. 

```
kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic
```

* we have `batch file` called <u>_kafka-console-producer_</u>, for that we need to provide a <b>_broker-list_</b> as a parameter and followed by that we are going to provide the _<b>topic name_</b>.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/os3mx15stz8k7pfvfgrv.png)

> If you able to see <b>'>'</b> then start _posting messages_.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qfmarxgg3k8xwqv6wwnd.png)

* And now we have _four messages_ successfully posted to the <u>test topic</u> and it is going to be _consumed_.

-----

####8. Instantiate  a _Console Consumer_ :

The next step we are going to _instantiate a console consumer_.
</br>

######Open new `CMD terminal` and use the command below to create a Consumer.

```
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning
```

* Here we provide the _boostrap server_ in which our `kafka broker` is _up and running_ after that we provide the <u>topic</u> and we are  going to give the options **from the beginning** .

* <b>from the beginning</b> - The meaning of this one is that I _read all the messages from the beginning_.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t4d91qenmgkntq9k0zgb.png)
 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z7ikx23313dyv28oqz4c.png)

> As you produce messages , these are consumed by the _consumer_ , so we have **producer and consumer successfully**.

![download (4)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qei862xrq6xon8nvpocz.jpg)


###If we  do not provide this from beginning ,what would happen?

Basically , it is not going to read the messages that are already there but as you **post the new messages** ,it's going to _read that message_.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/orxgfayvsqa77tux85jv.png)

> Here I didn't provide the option to start <u>from the beginning</u>, so it simply read the most recently posted message, "This is a message produced by the producer." 

> NOTE : Sometimes you may get the `messages` in **different order** because of the _message_ is getting posted to <u>different partitions</u>. 

> So if you want ordering , you have to make sure you're going to post to the same partition.

> So for _ordering_ you need to pass the **same key** , if you pass the same key , then the messages will be retrieved in the same order.

<center>"<b>We will discuss creating **Producer & consumer** with Key and also about Consumer Offsets in my next blog</b>"</center>


-----

That's all for now, guys, I hope you _learnt something new today_ :metal: .



![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m6ckqe94no6qwiwtrdoi.png)

Have a great day....

This is _priyanka nandula_ signing off...