Open sourcing Kafka Monitoring
Tags: kafka, opensource, monitoring
Published: 2021-11-16 18:45:52

# How it all started

As a Kafka guy, when I'm helping teams learning about how to setup Kafka, there's always one recurring element: how do you monitor Kafka ?

Kafka is a wonderful piece of machinery, but it's complex to manage. You have to monitor (at minimum) 2 products, Zookeeper (3 or 5 instances, in typical installation) and the Kafka brokers (3 or 4 instances, usually, but as an architect would say: "it depends of the context").

For Zookeeper, to start you can just say: is it alive ? Does it respond to requests ? And It's usually fine. But for the brokers, knowing they are running doesn't tell you the whole story. You have to understand how they are running: is your cluster healthy ? Is it serving requests ? How full is your request queue ? etc.

Once you monitor the brokers, you should also monitor the client application. How are they performing in their interaction with Kafka ?

And once you have done that, you have to consider monitoring all other parts of the Kafka ecosystem : Kafka connect, Kafka streams ecosystem, Schema registry, etc.

The amount of things you have to understand about Kafka to monitor it can be _overwhelming_. For us, at Zenika, the best thing is to start with some dashboards that can tell you a story.

What we want as Zenika's Kafka guys is to provide a way for teams to easily bootstrap a generic monitoring setup, learn from it, and make their own dashboard and alerting once they understand how all of this works.

# Our open source project

With a colleague, we decided it was time to make an open source project that would show default dashboards for Kafka. We started the project [kafka-monitoring](https://github.com/Zenika/kafka-monitoring) as a draft of what it would look like.

We worked together on the goal: providing Grafana dashboards that teams could easily copy and setup in their context. We wanted to address all Kafka ecosystem products: producer, consumers, connectors, stream apps (and if we had the time, KSQL, Schema Registry, Rest Proxy ?).

A [docker-compose file](https://github.com/Zenika/kafka-monitoring/blob/main/docker-compose.yaml) was the way to go, which would explain some of the best practices in Kafka configuration (like a default [replication.factor](https://kafka.apache.org/documentation/#brokerconfigs_default.replication.factor) and [min.insync.replicas](https://kafka.apache.org/documentation/#topicconfigs_min.insync.replicas)).

As big fan of agile, we started with the most valuable thing: the metrics of the Kafka brokers. We worked on a [first Grafana dashboard](https://github.com/Zenika/kafka-monitoring/blob/main/config/grafana_dashboard.json).

# StreamThoughts already thought of this

A huge ecosystem like Kafka has dozen of open source project. Surely some [Kakfa dashboards already existed!](https://grafana.com/search/?term=kafka&type=dashboard) That's true, but how do you navigate this sea of projects ? How do you know which dashboards are reliable, and which dashboards are up to date ?

It's when we encountered [StreamThoughts' project](https://github.com/streamthoughts/), that we knew that instead of doing our own work, we would contribute to existing work. We knew of StreamThoughts because it's co-founder, [Florian Hussonnois](https://www.linkedin.com/in/florian-hussonnois/), is an ex-colleague of ours.

That's the magic of open source: finding people that want to solve the same problem, and making them work together!

# An exciting opportunity

We forked StreamThoughts' project and are starting to work on some contribution! I'm excited to see where this will bring us.