How Apache Kafka works? Kafka Series - Part 1
Tags: kafka, microservices, bigdata
Published: 2019-04-25 07:27:25

---
title: How Apache Kafka works? Kafka Series - Part 1
published: true
description: How Kafka Works Internally.
tags: kafka,microservices,big data
canonical_url : https://cloudnweb.dev/2019/04/how-apache-kafka-works-kafka-getting-started-part-1/
series:  Kafka 
---
<!-- wp:paragraph -->
<p>To Configure the Apache Kafka in local machine and Run the server. <a href="https://cloudnweb.dev/2019/04/kafka-getting-started-kafka-series-part-2/">Kafka Series Part 2</a></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>what is kafka?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Firstly, Kafka is a streaming platform which follows the pub/sub pattern for sending the messages/monitor the events</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Usecase of kafka</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kafka is generally used for two broad classes of applications:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Building real-time streaming data pipelines that reliably get data between systems or applications<br><br></li><li>Building real-time streaming applications that transform or react to the streams of data<br><br></li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":4} -->
<h4>Kafka Core API'S :</h4>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><a href="https://kafka.apache.org/documentation.html#producerapi"><strong>Producer API</strong> </a> : it allows an application to publish a stream of records to a particular topic. we will see what topic is in the later part of this article.</li><li><strong><a href="https://kafka.apache.org/documentation.html#consumerapi"> Consumer API</a></strong>  :  it allows an application to subscribe to one or more topic and process the stream of records produced to them</li><li><strong><a href="https://kafka.apache.org/documentation/streams/">Streams API</a></strong>  :  allows an application to act as a&nbsp;<em>stream processor</em>, consuming an input stream from one or more topics and producing an output stream to one or more output topics.</li><li><strong><a href="https://kafka.apache.org/documentation.html#connect">Connector API</a></strong>  : allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems.</li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>How kafka works ?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In layman terms, Producer pushes the records into Kafka broker where consumer take those records and take action according to that message/event.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":316} -->
<figure class="wp-block-image"><img src="https://cloudnweb.dev/wp-content/uploads/2019/04/kafka_architecture.png" alt="" class="wp-image-316"/><figcaption>Kafka Architectural Overview</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>what is Topic and Partition ?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Topic in Kafka is nothing but a feed where a producer can push the records and consumer can get the records.&nbsp;Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":321} -->
<figure class="wp-block-image"><img src="https://cloudnweb.dev/wp-content/uploads/2019/04/log_anatomy.png" alt="" class="wp-image-321"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To make a topic fault tolerant and to handle the throughput, topic is maintained in different partitions. partitions can be in the same disk or in different disk.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>What is Kafka Broker ?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kafka Broker contains the cluster. Each Kafka Broker contains the topic log partitions. Connecting to one broker starts a client to the entire Kafka cluster. For failover, you want to start with at least three to five brokers. A Kafka cluster can have, 10, 100, or 1,000 brokers in a cluster if needed.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":328} -->
<figure class="wp-block-image"><img src="https://cloudnweb.dev/wp-content/uploads/2019/04/kafka_zookeeper.png" alt="" class="wp-image-328"/><figcaption>           Kafka Broker Overview</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To Configure the Apache Kafka in local machine and Run the server. <a href="https://cloudnweb.dev/2019/04/kafka-getting-started-kafka-series-part-2/">Kafka Series Part 2</a></p>
<!-- /wp:paragraph -->