Apache Kafka without Zookeeper and Dedicated Controllers
Published: 2023-01-26 04:17:33

---
title: "Apache Kafka without Zookeeper and Dedicated Controllers"
description: "The steps performed to build an Apache Kafka Cluster using KRaft, separate Controller nodes, and Confluent Docker Images."
published: true
---

Are you interested in setting up Kafka without Zookeeper and with a dedicated controller quorum? Here are the steps and reference project showcasing how to do this using the Confluent community-licensed container images. A Grafana dashboard to observe the new metrics is also provided.

## Introduction

Kafka Raft brings the consensus protocol into the Controller Plane of Apache Kafka from Zookeeper. With this change, the role of a Kafka instance can be that of a controller, broker, or both. Getting this configuration stood up requires some tweaks to the confluent [cp-kafka](https://hub.docker.com/r/confluentinc/cp-kafka) image.

If you want a deeper understanding of the design and implementation details, check out Jun Rao's course on Kafka Internals. Specifically, the [control-plane](https://developer.confluent.io/learn-kafka/architecture/control-plane/) section.

## Configuration

There are many configuration parameters with Apache Kafka; highlighted here are the ones necessary to build out the cluster with KRaft.

* [`node.id`](https://kafka.apache.org/documentation/#brokerconfigs_node.id)

The property `node.id` replaces `broker.id`. Be sure that all identifiers in the cluster are unique across brokers and controllers.

* [`process.roles`](https://kafka.apache.org/documentation/#brokerconfigs_process.roles)

A node can be both a `broker` or a `controller`. Set to `broker,controller` to enable both the controller and data planes on a node.

* [`controller.listener.names`](https://kafka.apache.org/documentation/#brokerconfigs_controller.listener.names)

List out the listener names used for the controller. This indicates to a node the listener to use for that communication. While the property is a list, just like `advertised.listeners`, the first one is what is used for controller communication.

* [`controller.quorum.voters`](https://kafka.apache.org/documentation/#brokerconfigs_controller.quorum.voters)

A comma delimited list of voters in the control plane, where a controller is noted as: `node_id@hostname:port`.

example:
```yaml
KAFKA_CONTROLLER_QUORUM_VOTERS: 10@controller-0:9093,11@controller-1:9093,12@controller-2:9093
```

### Additional Configuration

There are other [tuning parameters](https://kafka.apache.org/documentation/#brokerconfigs) for the controller plan, see the documentation for details.

### Lesson Learned

Do not remove cluster settings from the dedicated controllers, since a controller is the node that performs administration operations, such as creating a topic.

Incorrectly removing these from the controllers caused topics to be created without Apache Kafka defaults.

```yaml
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_NUM_PARTITIONS: 4
```


### Storage

Another change to setting up Apache Kafka with Raft is the storage. The storage on each node must be configured, before starting the JVM. This can be done with a `kafka-storage` command provided as part of Apache Kafka.

* Generate a unique UUID for the cluster, you can use `kafka-storage random-uuid` or another means.

* Before starting the cluster, format the metadata storage with `kafka-storage format`. 

```shell
kafka-storage format -t $KAFKA_CLUSTER_ID -c <server.properties>
```

### Container Images

With these details in mind, applying them to Confluent's [cp-kafka](https://hub.docker.com/r/confluentinc/cp-kafka) image takes a little finesse, at least with version `7.3.0`. The `cp-kafka` container's entry point, `/etc/confluent/docker/run`, builds the configuration for Apache Kafka from environment variables following conventions. In addition, there are validation steps to catch misconfiguration. These validations, however, need to change, since certain assumptions no longer apply in a zookeeper-less setup. In addition, a node that is only for a `controller` does not define `advertised.listeners` so validation for that needs to be removed.

#### Script Modifications

The following tasks need to be done to start up these images with raft. 

* Remove `KAFKA_ZOOKEEPER_CONNECT` validation for all nodes.
* Remove checking for zookeeper ready state for all nodes.
* Remove `KAFKA_ADVERTISED_LISTENERS` validation for dedicated controller nodes.
* Create Metadata store for all nodes by running `kafka-storage format`. 

### Command

The `cp-kafka` image's command is `/etc/confluent/docker/run`, and the scripts and docker-compose command setting, allow
these containers to start with raft consensus protocol.

* [broker.sh](https://github.com/kineticedge/dev-local/blob/main/kafka-raft/broker.sh)

```yaml
volumes:
  - ./broker.sh:/tmp/broker.sh
command: bash -c '/tmp/broker.sh && /etc/confluent/docker/run'
```

* [controller.sh](https://github.com/kineticedge/dev-local/blob/main/kafka-raft/controller.sh)

```yaml
volumes:
  - ./controller.sh:/tmp/controller.sh
command: bash -c '/tmp/controller.sh && /etc/confluent/docker/run'
```

### Seeing It In Action

If you are interested in seeing all this in action, check out the [kafka-raft](https://github.com/kineticedge/dev-local/tree/main/kafka-raft) docker-compose
setup in the [dev-local](https://github.com/kineticedge/dev-local) project. It is a fully working examples with 3 controllers and 4 brokers.

### Metrics

If you are going to deploy Kafka with Raft to production, having visibility to metrics is important. Adding that visibility is just as important (if not more so) than having dedicated controllers. The key to dashboards, ensure that they report correctly on data and control planes are separated or combined.

#### Grafana

A Grafana Dashboard is a multi-step process of extracting the metrics and storing them in a time-series database (e.g. Prometheus) and then visualizing those collected metrics in a Grafana Dashboard.

##### JMX Prometheus Exporter

The [KRaft Monitor](https://kafka.apache.org/documentation/#kraft_monitoring) metrics are defined in the documentation, with an MBean name, such as `kafka.server:type=raft-metrics,name=current-state`. Using a JMX client, such as `jmxterm`, shows the MBean is name is just `kafka.server:type=raft-metrics` with each metric an attribute in that bean. This is different from the current documentation.

If you deploy Java applications with JMX Metrics in containers, I highly recommend [jmxterm](https://github.com/jiaqi/jmxterm/releases).

```shell
java -jar jmxterm.jar
```

In the cp containers, the java process is process `1`, but use the command `jvms` to see all available processes; and verify the JVM's process-id is indeed `1`.

```shell
$>open 1
```

Show all the MBeans in the JVM, with `beans`.

```shell
$>beans
...
kafka.server:type=raft-metrics
...
```

Select a bean and use `info` to explore the attributes on a bean and `get` to get the current value of an attribute.

```shell
$>bean kafka.server:type=raft-metrics
#bean is set to kafka.server:type=raft-metrics
```

```shell
$>info
# attributes
%0   - append-records-rate (double, r)
%1   - commit-latency-avg (double, r)
%2   - commit-latency-max (double, r)
%3   - current-epoch (double, r)
%4   - current-leader (double, r)
%5   - current-state (double, r)
%6   - current-vote (double, r)
%7   - election-latency-avg (double, r)
%8   - election-latency-max (double, r)
%9   - fetch-records-rate (double, r)
%10  - high-watermark (double, r)
%11  - log-end-epoch (double, r)
%12  - log-end-offset (double, r)
%13  - number-unknown-voter-connections (double, r)
%14  - poll-idle-ratio-avg (double, r)
```

```shell
$>get current-leader
current-leader = 10.0;
```

Leveraging the above, the following properly exposes these from JMX Prometheus Exporter. Since `current-state` attribute value is a string, its value needs to be associated with a label to capture it in Prometheus.

```yaml
rules:
- pattern: "kafka.server<type=raft-metrics><>(current-state): (.+)"
  name: kafka_server_raft_metrics
  labels:
  name: $1
  state: $2
  value: 1
- pattern: "kafka.server<type=raft-metrics><>(.+): (.+)"
  name: kafka_server_raft_metrics
  labels:
  name: $1
```

##### Grafana Dashboard

Grafana is great for custom configuration, but that means time and effort are needed to build them. Here is a dashboard around some of those raft metrics; it is not a complete dashboard.

###### __Node Information__

So with metrics emitting, put them into a Grafana dashboard.

By using the `current-state` we can see who is the leader, in addition to capturing `node.id`. In addition, the dashboard component joins in other data to provide additional information on each node.

![node info](https://www.kineticedge.io/images/blog/kafka-raft/node-info_hu2d6d4c439fe0020661a8ea561be89976_260827_1600x0_resize_box_3.png)

###### __Node Counts__

Counts of nodes and active controller are always re-assuring, and this leverages an existing metric, `kafka_controller_kafkacontroller_value{name="ActiveControllerCount",}`. This metric only is emitted from a controller, so by counting the existence of this metric you see the number of controllers in the cluster, and by summing the value of the metric you get the actual number of active controllers; alert if this is ever not equal to 1.

Check out the dashboard to see how the other values are calculated, as it is the same as in the zookeeper-based installations.

![nodes](https://www.kineticedge.io/images/blog/kafka-raft/nodes_huf1deff26c8139713de2df050e44f19c1_91796_1600x0_resize_box_3.png)

###### __Active Controller__

To get the `node.id` of the quorum leader, just find `max(kafka_server_raft_metrics{name="current-leader",})`. Because scraping of each node is from slightly different times, different values are possible at the time of a change; max is used to make the single value display easy to build.

If a new leader is being voted, that will show up in the voted leader metric. In a single value dashboard, I do not expect this to be that useful, but in a time-series history of values, more value would be in having this metric recorded.

![leader](https://www.kineticedge.io/images/blog/kafka-raft/leader_hue7d9dd872581b91b852092ddaee702f1_25835_1600x0_resize_box_3.png)

##### __Full Dashboard__

Here is an example dashboard that also captures the fetch rate of the controller metadata.

![full](https://www.kineticedge.io/images/blog/kafka-raft/full_hu98b119011914a8d9ffa92f80b5651829_1116307_1600x0_resize_box_3.png)

#### Open-Source Tools

I have checked a variety of open-source tools out there and have been unsuccessful in seeing raft metrics. Active controller information is incorrectly displayed. None of the tools tried came to support raft, but it is important that before you upgrade, you have a proper monitoring and alerting strategy in place.

#### Summary

Be sure you properly test your monitoring and Apache Kafka support infrastructure as part of your move to Kafka Raft Consensus Protocol. Also, validate that kafka-raft metrics are captured and ensure that dashboards and tools work when brokers and controllers are on separate nodes.