Monitoring Kafka brokers using Jolokia, Metricbeat and ElasticSearch
Tags: kafka, jolokia, metricbeat, elasticsearch
Published: 2020-05-29 13:43:32

If you don't have commercial installation of Kafka that comes packed with monitoring solution, you have to build it on your own. You always need to have at least some tools to find what is going on in your service. Visualizing available metrics is indeed valuable option.

In this text, I will describe how to get these important metrics from Kafka and ship them to ElasticSearch where they can be visualized using Kibana. I will focus mainly on Kafka and Metricbeat configuration (how to get the metrics) rather than on visualization (make figures to your own taste).

**Component list**
- [Kafka](https://kafka.apache.org/)
- [Jolokia](https://jolokia.org)
- [Metricbeat](https://www.elastic.co/beats/metricbeat)
- [ElasticSearch](https://www.elastic.co/elasticsearch/)
- [Kibana](https://www.elastic.co/kibana)

# Data pipeline

Kafka advertises runtime metrics using dedicated MBeans that can be made available using JMX interface. This is exactly the channel we will use here. One of the tools that is developed to collect metrics from various systems is Metricbeat that comes with prepared Kafka module that simply "knows what metrics Kafka provides". Behind the scenes, Metricbeat makes use of Jolokia that serves as bridge for JMX and provides metrics using HTTP/JSON.

![pipeline](https://www.plantuml.com/plantuml/svg/NSun3i8m44JHtgVOaXI-048e53H4L9HMYebPo2gEZUmDI3azXWxQhwVHj4MHTPc3CQx3RG9jNg8ZdL98au0ETuxQzBIpvCwiMVp0q9wsL30_0fkVQlVaZW55nLyOsOzVg2bNzzTj7UnaGCJ7FAKi2BAoCIt7Q_tp1W00)

# Setting parts up

### Kafka with Jolokia

> Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies.

In this article, I will not describe installation of Kafka. There is plenty of up-to-date documentation available. I am using rpm packaged Kafka distribution from [Confluent.io](https://www.confluent.io). Let's focus on Jolokia.

We will be running Jolokia in agent mode. This is, we'll put Jolokia agent library on Kafka's class path, configure Jolokia in Kafka's configuration files and run it. This way, Jolokia will connect to Kafka monitoring MBeans and will provide metrics on HTTP interface.

+ Download **Jolokia JVM Agent** - https://jolokia.org/download.html 
+ Save it into Kafka's lib folder `/usr/share/java/kafka`
+ For easier configuration, either rename or make symlink `/usr/share/java/kafka/jolokia-jvm-agent.jar`

Now, we have the agent library in place and need to configure it

+ Open `/usr/bin/kafka-server-start` and add `KAFKA_JMX_OPTS`

```
export KAFKA_JMX_OPTS="
-javaagent:/usr/share/java/kafka/jolokia-jvm-agent.jar=port=8778,host=localhost \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=localhost \
-Dcom.sun.management.jmxremote.host=localhost \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Djava.net.preferIPv4Stack=true"

```

+ Start Kafka

At this point, you shall be able to see listening process on port 8778. You can try to get some numbers from it.

```bash
sudo netstat -ltnp
curl -s http://localhost:8778/jolokia/version | jq
```

### Metricbeat

After installing Metricbeat, you find default configuration in `/etc/metricbeat` with Kafka module disabled. In Metricbeat, we need to do 2 things:
+ Configure Kafka module for sampling, and
+ Configure Metricbeat to send data to ElasticSearch and create Kibana artifacts.

To configure Kafka sampling

+ Rename `/etc/metricbeat/modules.d/kafka.yml.disabled` to `/etc/metricbeat/modules.d/kafka.yml` to activate it.
+ Open `/etc/metricbeat/modules.d/kafka.yml`. To read all available metrics, the configuration shall have this contents

```yaml
# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-kafka.html

# Kafka metrics collected using the Kafka protocol
- module: kafka
  metricsets:
    - partition
    - consumergroup
  period: 10s
  hosts: ["localhost:9092"]

  client_id: metricbeat
  
  # List of Topics to query metadata for. If empty, all topics will be queried.
  #topics: []

# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:8778"]
```
For detailed description of Metricbeat's Kafka module see [documentation](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kafka.html). 

To configure proper forwarding of metrics to ElasticSearch

+ Open `/etc/metricbeat/metricbeat.yml`, and configure `elasticsearch` output

```yaml
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "metricbeat-kafka-%{[agent.version]}-%{+yyyy-MM-dd}"
```

+ Start Metricbeat service.

```bash
sudo systemctl start metricbeat
```

At this moment, Metricbeat shall start to sample Kafka metrics with 10 seconds interval and send them to ElasticSearch.

If you have Kibana installed, you can command Metricbeat to create visualizations and dashboard for you. For this, you need to configure [Kibana endpoint](https://www.elastic.co/guide/en/beats/metricbeat/current/setup-kibana-endpoint.html) parameters - where is your Kibana - and [Kibana dashboard](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-dashboards.html) parameters - where in Kibana you want the artifacts to be created.

To create Kibana artifacts, run `metricbeat setup` command.

**Version information**

For this post, I used following component versions:
- Kafka - Confluent Community 5.4.1
- Jolokia 1.6.2
- Metricbeat 7.6.2
- ElasticSearch 7.6.0
- Kibana 7.6.0