Kafka lag exporter Standalone
Tags: kafka, monitoring, troubleshooting, devops
Published: 2020-07-23 19:42:56

To research and diagnose issues of [Apache Kafka](https://kafka.apache.org/) consumer groups I usually use a docker compose made up of [Kafka Lag Exporter], [Prometheus], [Grafana] and a [Dashboard](https://github.com/lightbend/kafka-lag-exporter/tree/master/grafana).

Here is the repo:

{% github cspinetta/kafka-lag-exporter-standalone no-readme %}

In this post I want to tell you what it is about and how it can be useful for troubleshooting in case you don't have much visibility of your Kafka consumer groups.

### Why this docker compose?

The main idea was to have a docker compose with [Kafka Lag Exporter], [Prometheus] and [Grafana] together, so that it can be **quick** and **easy** to get a dashboard for analyzing the consumer groups of a Kafka deployment. This is particularly useful when you don't have enough monitoring on your Kafka yet. If we had not made a docker compose, we would have to install and configure each part separately, which could be a bit cumbersome when faced with an incident that needs to be resolved quickly.

It aims to provide a quick installation for troubleshooting and not a final installation for permanent monitoring. It's perfect if you are facing an issue in production and need more visibility about what is happening internally in Kafka.

### Example use cases

#### Consumer group lag in seconds

![Consumer Groups Time Lag](https://dev-to-uploads.s3.amazonaws.com/i/jax18tlj4zesw7jsgkz2.png)

In this graph we can see the time between last commit and current time, this is also known as lag in seconds, and we can see it by consumer group.

This example was taken from a `Kafka-Connect` that commits every 30 minutes if everything goes well, if we analyze it we can say that some consumer groups by some reason fail on commit.

#### Consumers group lag in seconds and offsets

![Consumer Group Time and Events Lag](https://dev-to-uploads.s3.amazonaws.com/i/dig6tfkebbngs1le7ynb.png)

In this other example, we can see a particular consumer group status. It shows the consumer group lag in seconds and messages/events (the difference between current message and last commited)

### Getting started

Next, the steps we do when we have to use it:

1. Download and unpack it:

```bash
curl -fsSL -o kafka-lag-exporter-standalone.tar.gz https://github.com/cspinetta/kafka-lag-exporter-standalone/releases/download/0.0.1/kafka-lag-exporter-standalone-0.0.1.tar
tar -xf kafka-lag-exporter-standalone.tar.gz
```

2. Specify kafka nodes on [kafka-exporter-standalone/kafka-lag-exporter/application.conf].

3. Run with docker compose: `docker-compose -f kafka-exporter-standalone/docker-compose.yaml up`.

Then you can open the Grafana webapp exposed at port `3000` and navigate to the dashboard **Kafka Lag Exporter**.

On the first time you enter Grafana ask you to login. Type `admin` for the username and password. Then Grafana will ask you to choose a new password.

That's all!

[Kafka Lag Exporter]: https://github.com/lightbend/kafka-lag-exporter
[Prometheus]: https://prometheus.io/
[Grafana]: https://grafana.com/
[kafka-exporter-standalone/kafka-lag-exporter/application.conf]: kafka-exporter-standalone/kafka-lag-exporter/application.conf
[kafka-exporter-standalone/docker-compose.yaml]: kafka-exporter-standalone/docker-compose.yaml