Notes on Kafka: Partition Count and Replication Factor
Tags: kafka, microservices, eventdriven, 100daysofcode
Published: 2021-09-29 05:21:26

This article has four sections, and can jump ahead to the ones you're interested in:

- [Why they're set during topic creation](#why-theyre-set-during-topic-creation)
- [Partition Count](#partition-count)
- [Replication Factor](#replication-factor)
- [Cluster Guidelines](#cluster-guidelines)

---

## Why they're set during topic creation

As we've previously see in the **Kafka CLI commands** section, there are two important parameters to define when creating a topic. These two impacts the performance and durability of your overall system.
- Partition Count
- Replication Factor

Let's use this diagram as an example.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u7vndh0czfx59yvtdncc.png)*<center><small>(Photo courtesy of Stephane Maarek's Apache for Beginner's course)</small></center>*

Currently, we have three brokers and a topic with 2 partitions and 2 in-sync replicas. Now if we were change either of the two parameters,
- if we change the partition count to 3, the **keys ordering guarantees will break**.
- if we change the replication factor to 3, there will be **pressure on your cluster** which can lead to instability and performance decrease.

Increasing the replication factor maximizes the available brokers, which means each brokers will have to put in more work.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mb47td3znicnic53do9l.png)*<center><small>(Photo courtesy of Stephane Maarek's Apache for Beginner's course)</small></center>*

---

## Partition Count

Here are a few points to remember:
- Each partition can handle a throughput of a few MB/sec
- Be sure to measure for your setup
- more partitions, more throughput; better parallelism
- more partitions means more consumers can run at a scale
- recall that no consumer will have the same topic partition
- more partitions are ideal in a large cluster

But:
- more partitions means more election to be done by Zookeeper
- more files are opened on Kafka

Thus, some guidelines are:
- for small cluster(less than 6 brokers), partitions should be twice the number of brokers
- for big cluster(more than 12 brokers), safe to have 1:1 ratio for partition and broker
- adjust for the number of consumers in parallel at peak throughput
- adjust for producer throughput (increase if super high throughput)

**TEST!** Every Kafka cluster will have different performance.

---

## Replication Factor

Higher replication factor means:
- better resilience
- more replication

But:
- higher latencies if ack=all
- more disk space on the system

Thus, some guidelines are:
- should be at least 2, usually 3, max of 4
- Never set ISR=1 in production
- safe start with ISR=3
- if replication performance persists, get better hardware

---

## Cluster Guidelines

- A cluster should not hold more than 2000 to 4000 partitions
- Maximum of 20,000 partitions across all brokers
- This is because when brokers crash, the Zookeeper performs leader elections for each partition
- Thousand of partitions means lot of elections
- If more partitions is needed, increase the number of brokers
- if more than 20,000 partitions is needed, follow the **Netflix model** and adapt **Kafka clusters** 

Start at a reasonable partition count, test the performance, and increase accordingly.

---

If you've enjoyed this write-ups or if they somehow brought some value, I'll be glad to connect with you on [Twitter!](https://twitter.com/eden_noel08). 😃

---------------------------------