Kafka Getting Started - Kafka Series - Part 2
Tags: kafka, bigdata
Published: 2019-04-26 06:30:14

---
title: Kafka Getting Started - Kafka Series - Part 2
published: true
description: Kafka Getting Started - Kafka Series - Part 2
tags: Kafka,bigdata
canonical_url : https://cloudnweb.dev/2019/04/kafka-getting-started-kafka-series-part-2/
series: Kafka
---

<!-- wp:paragraph -->
<p>This is part two of Kafka series. if you want to learn how Kafka works and Kafka architecture. Read here <a href="https://cloudnweb.dev/2019/04/how-apache-kafka-works-kafka-getting-started-part-1/">Kafka Architecture</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In this article, we will see how to configure Kafka locally and run the Kafka server.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Kafka node Setup</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Firstly, On a single machine, a <strong>3 broker</strong> Kafka instance is at best the minimum, for a hassle-free working. Also, the <strong>replication factor is set to 2</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let's say A, B and C are our Kafka brokers. With replication factor 2, the data in A will be copied to both B &amp; C, the data in B will be copied to A &amp; C and the data of C is copied to A &amp; B.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Prerequisites</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>java &gt;= 1.8 installed in the machine</li><li>Set java environment in the machine. <a href="https://hackernoon.com/setting-up-java-on-ubuntu-linux-and-windows-f60c910d39e3">Environment Setup </a></li><li>download binary distribution of Kafka <a href="https://kafka.apache.org/downloads">here</a></li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Setup</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Meanwhile, extract the Kafka archive in the convenient place and <em><strong>cd</strong></em>  into it. Use the terminal to run the Kafka ecosystem</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First we need to run the Kafka ZooKeeper in the Terminal.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>What is ZooKeeper?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>ZooKeeper used to manage the service discovery and to do the leadership election Kafka Brokers. &nbsp;it sends changes of the topology to Kafka, so each node in the cluster knows when a new broker joined, a Broker died, a topic was removed or a topic was added, etc. it provides an in-sync view of Kafka Cluster configuration.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":328} -->
<figure class="wp-block-image"><img src="https://cloudnweb.dev/wp-content/uploads/2019/04/kafka_zookeeper.png" alt="" class="wp-image-328"/><figcaption>            ZooKeeper Overview</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>start the ZooKeeper instace with </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>bin/zookeeper-server-start.sh config/zookeeper.properties</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Kafka Broker</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>In the <strong>Config</strong> folder, there would be a <strong>server.properties</strong> file. This is the kafka server's config file. We need 3 instances of kafka brokers.</li><li>Make a copy.&nbsp;<code>$ cp config/server.properties config/server.bk1.properties</code></li><li>In the copied file, make the following changes</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code>broker.id=1  #unique id for our broker instance
listeners=PLAINTEXT://:9093 #port where it listens 
log.dirs=/home/neoito/kafka-logs-1  #to a place thats not volatile </code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Further, make 2 more copies of it with the same process. change the above line with a consecutive number like broker.id=2, broker.id=3</li><li>Finally, Run the individual brokers like<br><br></li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$  bin/kafka-server-start.sh config/server.bk1.properties
$  bin/kafka-server-start.sh config/server.bk2.properties
$  bin/kafka-server-start.sh config/server.bk3.properties</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>After that, we need to create a topic where Producer can push the records and consumer can subscribe/listen to it.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Above command will create a topic in the Kafka Broker with a replication factor of 3 with one  partition </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-replicated-topic</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p> the command will start the producer from the command cli where we can push the records to Kafka Brokers.  After that, we need to start the consumer from the cli.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-replicated-topic</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>it will start the consumer in the port 9093.<strong> --from-beginning</strong> command will read the records in the topic from the beginning.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>That is to say, if we type anything in the producer cli, we can read those records from the consumers command line.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":349} -->
<figure class="wp-block-image"><img src="https://cloudnweb.dev/wp-content/uploads/2019/04/kafka_demo_gif.gif" alt="" class="wp-image-349"/><figcaption>Kafka Demo</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In conclusion, we can now subscribe to a topic and listen to the records without losing any data. it will be useful in several scenarios.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>we will see how to use Apache Kafka with web application in next blog.Stay tuned!!!!! :-)</p>
<!-- /wp:paragraph -->