Notes on Kafka: Kafka Architecture
Tags: kafka, microservices, eventdriven, 100daysofcode
Published: 2021-06-15 11:44:20


As mentioned in the previous article, Apache Kafka is a messaging system, specifically a **publish-subscribe** messaging system where there are *publishers* of messages and *subscribers* of messages.

![kafka-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j6elhoq2bp3siinmdcey.JPG)

Publishers creates the data and sends them to a specific location. They are called **Producers.**

On the other end, subscribers can listen in and retrieve the messages. These are called the **Consumers**.

At the middle lies the **Topics** which is basically a grouping of messages. This is where the producers send their data. Topics have specific name which can defined during creation or modified on-demand.

The main rule here is that the producers must know what is the topic name that they want to send their message to and they must also be authorized to do so. The same goes with subscribers - they must also be authorized to retrieved the message from the topic.

---

## Brokers as containers of data

Kafka keeps and maintains all their topics in the **broker**. This is an executable or daemon service running on a machine, which can either be physical or virtual.

The broker has access to the *file system* of the machine. It uses the file system to store the messages and categorize as topics. Brokers receive the messages, assigns offsets, and commits them to storage on disks.

As with any executable, machines can have more than one broker but each must be unique so they won't conflict with one another. To do this, each broker will have its own *broker id*.

![kafka-12](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c4cru8kia8024qgv20pl.JPG)

Brokers retain all published messages regardless of whether it is consumed or not. Because of this, Kafka's ability to handle the topics and messages gives it the main edge when it comes to achieving high-throughput.

----

## Achieving High Throughput

To achieve high throughput, a system must be able to distribute its load and efficiently process it in multiple nodes and in parallel. Kafka achieves this by scaling out its brokers to accomodate the load, all this being done without affecting existing producers and consumers.

![kafka-13](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1x714nkqvm15ugo140mx.JPG)

----

## Clusters ##

Recall that a machine can have one or more Kafka brokers running on it. A **Kafka Cluster** is a grouping of multiple Kafka brokers on a single machine or brokers on different machines.

Additionally, a cluster's access is through any of the broker within the cluster. This means when you connect to one broker, you also get access to all other brokers in that cluster.

![kafka-14](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/24mk213sd6x3prqfsnvu.JPG)

The **cluster size** is the number of brokers that are within the cluster, regardless if these brokers are running on the same machine or separate machines.

Knowing about the cluster size is important since this is the mechanism which allows Kafka to scale to thousands of brokers. Now to enable the scaling out activity, the clusters will need metadata to operate at scale and reliably. This is where the **Zookeeper** comes in.

----

The succeeding notes will dive in deeper into Kafka as a distributed system and how Apache Zookeeper fits in the equation. If you'd like to know more, please proceed to the next note in the series.

Similarly, you can check out the following resources:

- [Getting Started with Apache Kafka](https://www.pluralsight.com/courses/apache-kafka-getting-started) by Ryan Plant

- [Apache Kafka Series - Learn Apache Kafka for Beginners v2](https://www.udemy.com/course/apache-kafka/) by Stephane Maarek

- [Apache Kafka A-Z with Hands on Learning](https://www.udemy.com/course/apache-kafka-learnkarts/) by Learnkart Technology Private Limited

- [The Complete Apache Kafka Practical Guide](https://www.udemy.com/course/apache_kafka/) by Bogdan Stashchuk

---------

If you've enjoyed this short but concise article, I'll be glad to connect with you on [Twitter!](https://twitter.com/eden_noel08). You can also hit the ***Follow*** below to stay updated when there's new awesome contents! 😃

{% user jeden %}

---------------------------------