New Winner of Kafka Consumers: Scala to Go Journey 🚀
Tags: go, scala, kafka, concurrency
Published: 2023-04-18 14:42:34

_350 Million Messages per Day_

Co-Authors: [Abdulsamet Ileri](https://www.linkedin.com/in/abdulsamet-ileri-839222121/), [Nihat Alim](https://www.linkedin.com/in/nihat-alim-288072b0/)

In this article, you will read a story about how we decreased memory usage by ~50%, increased CPU Efficiency by ~%80 and TP by ~20 by migrating from Scala to Go.

---

### Introduction
Trendyol has more than 170 million sellable and 200 million ready-to-sell products right now. At any time, there may be a change, or invalidation, as we call it, on these products via different events like promotion, stock, and price. By changes, we mean millions of them, approximately 350 million a day if we need to be precise. As the Trendyol Indexing Team, we need to apply these changes almost near real-time because any latency could cause product data to be displayed incorrectly. We don’t even want to think about writing the price of a product wrong.

In the middle of all these changes is our application, Perses. In Greek mythology, Perses was the Greek Titan God of Destruction. Like the Destruction God, our application Perses destroys the old product data and replaces it with its new version.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/btfvx6mi1kwiuq82qkjc.png"><figcaption>Perses Design</figcaption>
</figure>

As we can see from the daily throughput graphic of Perses below, Perses does millions of I/O operations daily. It is crucial to perform them correctly and without any latency to be able to show the users the correct product data. To achieve our purpose of having a high-performant application, Perses is designed as a multi-deployment application so that every deployment can scale independently and does not block each other invalidation process.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lwdhhhxb5qik4io4t7z0.png"><figcaption>Perses’ Daily Throughput</figcaption>
</figure>

From all of these, we can easily say that Perses plays an important role in [Trendyol’s invalidation process](https://medium.com/trendyol-tech/evolution-of-our-invalidation-system-bdfee456498b).

---

There were several reasons why we made this migration decision.

- Previously, getting better results in terms of resource usage and performance by migrating our other smaller consumer projects.
- Learning and maintaining Go was easier than maintaining old Scala projects for our team.

### Our Implementation Steps

We will explain our re-platforming journey in 5 steps.

#### 1. Without monitoring, you cannot solve murders.

In Scala Perses, the consuming operation was made in batches using Akka Stream (v2.12). You can see the implementation in the code block below.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n8xu584mz711nncby9lc.png">
</figure>

We tested and compared these two as soon as we implemented batch consuming in Go Perses ([via kafka-go](https://github.com/segmentio/kafka-go), but the results shocked us. While Scala Perses was processing 13k messages in a minute per pod, Go could only process 4k. 😢

After seeing the results, we monitored the application and saw the bottleneck in the [Kafka producer part](https://github.com/segmentio/kafka-go#writer). When we dived into the codebase of the producer [in the Kafka library](https://github.com/segmentio/kafka-go), we realized that the producing operation was done synchronously. It means that we were going to the broker for every message, which was harmful to the performance.

#### 2. Caught the killer! Wait, there are more!?

We decided to change the producer implementation from synchronous to asynchronous and to in batches. The message queue contained messages sent to two different topics at that time.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0eh5arvy10triqmsl97k.png">
</figure>

After completing the implementation, we ran a load test immediately, but the results were still disappointing. Go Perses processed 9k messages in a minute per pod, but we still didn’t reach Scala Perses 13k. 😞

#### 3. Following the breadcrumbs

We separated the message channel for our topics. We had only two topics that we needed to produce back then, so we created two message channels and one Goroutine per channel.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fqt6aq1zk9ufvse6244k.png">
</figure>

After this change, we tried different “batch size” and “batch duration” parameters to achieve the best performance. After several tries, we saw that the optimal values were 500 for batch size and 500ms for the batch duration. When we ran a load test again with the final parameters with 36 partitions and 12 pods, we processed 1.094.800 messages in 10 minutes with 136k throughput. Scala was still more performant with its 156k throughput, it could process the same number of messages in 8 minutes.🤔

#### 4. It’s not a dead end

We are using [Uber’s automaxprocs](https://github.com/uber-go/automaxprocs) library for one of our other Go projects and saw performance enhancement, so we also wanted to try it for Go Perses. Unfortunately, throughput was not changed because Perses is more i/o bound than CPU bound. 😢

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jpcs42ww2k5hfme5zjuf.png">
</figure>

### 5. The sweet smell of victory

Because we still couldn’t get the desired results, we decided to focus on our architectural design to figure out what to do.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7jzdim421vuamixpg4di.png">
</figure>

In the first part of the architecture, a goroutine continuously listens to a topic and fetches new messages using an internal message queue. We think we could try [tuning the queue size here](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.39#ReaderConfig.QueueCapacity), so we decided to configure it.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e9b4vntu9famkb454nhz.png">
</figure>

In the second part of the architecture, after the fetched messages come to the channel, the goroutines process them. Here, we thought we could tune the [channel buffer size](https://stackoverflow.com/questions/11943841/what-is-channel-buffer-size) and the number of worker goroutines.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qmu9lm8fjez06gsefbp0.png">
</figure>

We were surprised by the results after changing the configurations in the tuning process 💃💃. You can see our trials and their results in the table below.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/70qd84bpe4i6hbynhpeo.png">
</figure>

You can see the comparison of the most performant Scala Perses and Go Perses results below.

<figure>
  <img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r1ax4418gkme6r9a013o.png">
</figure>

<figure><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6yvyrdk985r2n9vte8wv.png"><figcaption>Scala Perses Final Throughput</figcaption>
</figure>
<figure><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ob2lwl517fhzb3chrxu4.png"><figcaption>Go Perses Final Throughput</figcaption>
</figure>

### Conclusion

As a result of the migration process, we optimized the following resources

- Memory usage: ~50% Decrease (from 1.127 GB to 0.622 GB)
- CPU: ~80% Efficient (from 1.72 to 0.379)
- TP: ~20% Increase (from 156k to 189k)

We are very pleased to make this optimization on a multiple-deployment codebase that works under high load.

### What we did learn!

- We realized how important monitoring is when it helps us to address the kafka producer problem.
- By diving into our project design, we realized some critical parameters still needed to be tuned to improve performance.

Thanks for reading 🎈🎈🎈

We open-sourced some of our experience [with a built-in retry manager](https://github.com/Trendyol/kafka-cronsumer) at [kafka-konsumer](https://github.com/Trendyol/kafka-konsumer). We are excited to get your feedback.

Thank you to [Kutlu Araslı](https://www.linkedin.com/in/kutluarasli/), [Emre Odabas](https://www.linkedin.com/in/emreodabas/), and [Mert Bulut](https://www.linkedin.com/in/mertbulut95/) for their support 💚

If you’re interested in joining our team, you can apply for the backend developer role or any of our current [open positions](https://jobs.lever.co/trendyol?lever-via=OUSPhixG90).