How to Upgrade Kafka from 1.1.1 with Zero-Downtime: An Applicable Approach
Tags: kafka, devops, upgrade, dataplatform
Published: 2024-03-29 08:25:54

As a data engineer or, more specifically, data platform engineer, a service with high dependency may be handed over to you. Upgrading such a service is a terrifying process. Suppose that service is Kafka, and it's the main component of your data stack at the company. However, the solution isn't ignoring the complexity because every bug fix or new feature can save you from downtime and help you increase the performance of the services. So, what is the solution? How can we ensure all services that depend on Kafka work fine after the upgrade? In this post, I will share my experience through this process.

## Main concerns

When we talk about services like Kafka, we know many producers and consumers are in between. So, what happens to them after an upgrade? Do they continue to produce/consume? What about the schema registry and other components that depend on Kafka? So, one of the main concerns is the healthiness of the dependent element.
Also, we want to upgrade Kafka for two significant versions; how should we check deprecated configs? Should I read all the changelogs one by one? There is a better approach that minimizes the time spent and the probability of downtime. 

## Proposed approach

Honestly, every time I think about Docker, I wonder what a beautiful tool this is :D You know? Amazingly, you can independently set up a whole stack in a separate network with tools like docker-compose. 

A better approach is to use Docker to simulate production services in a safe environment. We can set up a whole stack with the same configs but fewer resources, simulate upgrades, and check each component's behavior.

## Applied approach for Kafka
To simulate the upgrade process for Kafka, I am supposed to create a stack including these components:
1. Zookeeper Instances -> Coordinator for Kafka Cluster
2. Kafka Instances -> Main component
4. Schema Registry -> Persist schema of produced messages
5. Kafka UI -> Monitor Kafka cluster and see incoming messages in topics
6. Producers -> Python code to produce data into Kafka topic in Avro format.
7. Consumer -> Python code to consume data produced by `Producer`.
8. Clickhouse -> Analytical database to store data coming from Kafka
9. Postgres -> OLTP database stores transactional data
10. Postgres Producer -> Python code, which Inserts one record every 0.1 seconds into the `Postgres` database
11. Debezium -> Capture each change in `Postgres` and send it to the corresponding Kafka topic in Avro format.

Now, it's time to prepare the appropriate `docker-compose.yaml`

### Implement detail


- **Zookeeper**
    - Image: [Official Image](https://hub.docker.com/_/zookeeper)
    - Configs:
        - Mount `zoo.cfg` into the container
        - [`myid` and data directory](https://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_dataFileManagement) created using [`zookeeper_conf_creator.py`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/zookeeper/zookeeper_conf_creator.py)
- **Kafka**
    - Image: [Bitnami Image](https://hub.docker.com/r/bitnami/kafka)
        - Image customized by [`Dockerfile-Kafka`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/kafka/Dockerfile-Kafka) ([https://hub.docker.com/r/bitnami/kafka](https://hub.docker.com/r/bitnami/kafka))
    - Configs:
        - Set as an environment variable
        - `server.properties` converted to `server.env` using [`kafka_env_creator.py`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/kafka/kafka_env_creator.py)
        - This image didn't support `SCRAM-SHA` for authentication. So [`libkafka.sh`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/kafka/libkafka.sh) (which is bitnami's Kafka library), rewritten.
- **Schema Registry**:
    - Image: [Official Image](https://hub.docker.com/r/confluentinc/cp-schema-registry)
    - Configs:
        - Set as an environment variable
        - `schema-registry.properties` converted to `schema-registry.env` using [`schema_registry_config_creator.py`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/schema-registry/schema_registry_config_creator.py)
- **Kafka UI**
    - Image: [Official Image](https://hub.docker.com/r/provectuslabs/kafka-ui)
    - Configs:
        - Set as an environment variable
        - Directly in `docker-compose.yaml`
- *_Producer and Consumer *_
    - Image: [Official Python Image](https://hub.docker.com/_/python)
        - Image customized by [`Dockerfile-Python`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/python-producer-consumer/Dockerfile-Python)
    - Code: [`producer.py`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/python-producer-consumer/producer.py) and [`consumer.py`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/python-producer-consumer/consumer.py)
    - Configs: Set as environment variables directly in `docker-compose.yaml`
- **Clickhouse**
    - Image: [Official Image](https://hub.docker.com/r/clickhouse/clickhouse-server)
    - Tables: Tables DDL defined [here](https://github.com/snapp-incubator/kafka-upgrade-simulation/tree/main/clickhouse/schemas) and then mounted into `/docker-entrypoint-initdb.d`
        - For each table in `Postgres` three tables are defined here:
            - Base table -> data persist here
            - Kafka table -> read data from kafka
            - Materialize view -> ship data from the Kafka table into the base table.
    - Configs: Default configs used only [`kafka.xml`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/clickhouse/configs/kafka.xml) mounted into `/etc/clickhouse-server/config.d/kafka.xml`
    - Logs: For debug purposes, logs mounted into [local directory](https://github.com/snapp-incubator/kafka-upgrade-simulation/tree/main/clickhouse/log)
- **Postgres**
    - Image: [Debezium Example Image](https://quay.io/debezium/example-postgres)
        - This Postgres contains sample sale data.
- **Postgres Producer**
    - Same as `Producer` and `Consumer` but [this code](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/python-producer-consumer/postgres-producer.py) used
- **Debezium**
    - Image: [Official Image](https://hub.docker.com/r/debezium/connect)
    - Configs:
        - Set as an environment variable
        - `kafka-connect.properties` converted to `kafka-connect.env` using [`kafka_connect_config_generator.py`](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/debezium/kafka_connect_config_generator.py)

**Some Extra Containers**

- **kafka-setup-user**
    
    - It uses the same image as `Kafka`; it runs after `kafka1` becomes healthy. Some users are created after this container runs (exit with status 0). See them [here](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/kafka-setup-user/entrypoint.sh)
    - It needs one Kafka broker and also a Zookeeper cluster because `SCRAM-SHA` needs to persist on Zookeeper.
- **kafka-setup-topic**
    
    - It uses the same image as `Kafka` and creates some topics. See the list [here](https://github.com/snapp-incubator/kafka-upgrade-simulation/tree/main/kafka-setup-topic)
- **submit-connector**
    
    - It use [curl image](https://hub.docker.com/r/curlimages/curl) to submit [this connector](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/submit-connector/register-postgres.json) into `Debezium`. The connector captures the changes in `Postgres`, sends events to `Kafka`, and then `Clickhouse` consumes the data into appropriate tables.

**Some Extra Notes:**

- The version of all containers defined in the [`.env` file](https://github.com/snapp-incubator/kafka-upgrade-simulation/blob/main/.env). You can change them from this file.
    
- Container dependencies are defined accurately. So, if one container depends on another to come up, appropriate `healthcheck` and `depends_on` conditions are defined for it.
    
- If you take a look at the `healthcheck` of containers, for example, kafka, you see this command:  
    

```bash
   (echo > /dev/tcp/kafka1/9092) &>/dev/null && exit 0 || exit 1
```

This shell script helps check the TCP port in a container without `telnet`.

### Simulation Process
To run the simulation, you can follow [these steps](https://github.com/snapp-incubator/kafka-upgrade-simulation?tab=readme-ov-file#test-process).


### Result

All tests were successful. By successful, I mean the producer can still produce messages without errors, and consumers can consume messages without errors. No other criteria were investigated; you can define your metrics for this simulation. Only one problem was seen in this process.

**Problems:**
1. In `Setup Kafka User`: `java.lang.ClassNotFoundException: kafka.security.auth.SimpleAclAuthorizer` occurred
	1. It deprecated after 2.4.0. See [here](https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface)
	2. Doc recommends to use `kafka.security.authorizer.AclAuthorizer` instead. It's fully compatible with deprecated class, so it was replaced in docker-compose and it worked

**Conclusion**:
- As there is the official document for upgrading from any version to 3.6.1 (and another previous version), there is no obstacle in this process. Also, our test shows this process works, and we can upgrade our Kafka to whatever version we want.

## Conclusion

This article is a suggestion for the best approach for upgrading highly dependent services. We talked about the details of implementing this process, and then, as we saw in the Result section, one problem was found before upgrading so we can upgrade our Kafka cluster seamlessly, with zero-downtime :)