Empowering Your Kafka Connectors: A Guide to Connector Guardian
Tags: kafkaconnect, python, opensource, devops
Published: 2024-02-27 18:15:06

Hi there :)

In this post, I want to introduce you to Connector Guardian. If you've ever found yourself grappling with the management of Kafka Connect connectors, you're in for a treat. Connector Guardian is tailor-made to simplify your life as a developer or operator, providing efficient tools for the seamless management and maintenance of your Kafka Connectors.

## How It Works

Connector Guardian smoothly interacts with your Kafka Connect cluster using its [REST API](https://docs.confluent.io/platform/current/connect/references/restapi.html). In its initial release, version [0.1.0](https://github.com/snapp-incubator/connector-guardian/releases/tag/0.1.0), it harnesses the power of [jq](https://github.com/jqlang/jq) for JSON parsing. Later, transitioning to version [0.2.0](https://github.com/snapp-incubator/connector-guardian/releases/tag/0.2.0), Connector Guardian adopts Python's built-in JSON library.

## Features

* **Auto Connector Restart**: Starting from [V0.1.0](https://github.com/snapp-incubator/connector-guardian/releases/tag/0.1.0), Connector Guardian monitors the status of connectors and tasks, automatically restarting them if they fail.

* **Restart Back Off**: Introduced in [V0.3.0](https://github.com/snapp-incubator/connector-guardian/releases/tag/0.3.0), this feature ensures that restarts occur at increasing time intervals. The initial restart happens immediately, and subsequent restarts are delayed exponentially. This approach allows for efficient issue resolution, even in the face of prolonged network outages. After a configurable number of restarts (`MAX_RESTART`), the Guardian stops automatic restarting, leaving it to you for manual intervention.

## How to Add Guardian to Kafka Connect Cluster

### Container Image

You can easily pull the Connector Guardian image from [Docker Hub](https://hub.docker.com/r/anvaari/connector-guardian), then run it with `docker run` command:

```bash
docker run -d \
  -e KAFKA_CONNECT_HOST=localhost \
  -e KAFKA_CONNECT_PORT=8083 \
  -e KAFKA_CONNECT_PROTO=http \
  -e KAFKA_CONNECT_USER='' \
  -e KAFKA_CONNECT_PASS='' \
  -e ENABLE_BACKOFF=1 \
  -e MAX_RESTART=7 \
  -e EXPONENTIAL_RATIO=2 \
  anvaari/connector-guardian
```

### Non-Cloud Environments

For deployment on your server, use the provided [docker-compose](https://github.com/snapp-incubator/connector-guardian/blob/main/deploy/docker-compose.yaml) file. Before deploying the image, ensure that you set the appropriate environment variables in [docker-compose.yaml](https://github.com/snapp-incubator/connector-guardian/blob/main/deploy/docker-compose.yaml).

```bash
cd deploy
docker compose up -d
```

### Kubernetes or OpenShift

Utilize the provided Helm chart for deployment. Make sure to set the required environment variables in [values.yaml](https://github.com/snapp-incubator/connector-guardian/blob/main/deploy/chart/values.yaml).

```bash
helm upgrade connector-guardian --install -n {your_namespace_name} -f deploy/chart/values.yaml deploy/chart
```

Once deployed, Connector Guardian runs as a pod, executing `connector_guardian.py` every 5 minutes.

### Environment Variables

To use the Docker image, [docker-compose](https://github.com/snapp-incubator/connector-guardian/blob/main/deploy/docker-compose.yaml), or [Helm chart](https://github.com/snapp-incubator/connector-guardian/blob/main/deploy/chart/), set the following environment variables:

* `KAFKA_CONNECT_HOST`: Default = `localhost`
* `KAFKA_CONNECT_PORT`: Default = `8083`
* `KAFKA_CONNECT_PROTO`: Default = `http`
* `KAFKA_CONNECT_USER`: Default = `''`
* `KAFKA_CONNECT_PASS`: Default = `''`
* `ENABLE_BACKOFF`: Default = `1`
* `MAX_RESTART`: Default = `7`
* `EXPONENTIAL_RATIO`: Default = `2`

## In The End ...

Connector Guardian is your steadfast companion in the realm of Kafka Connect connectors. Whether you are a seasoned developer or an operations expert, this tool streamlines the management of your connectors, offering an automated approach to restarts and intelligent back-off mechanisms.

As I continue to evolve Connector Guardian, I invite you to be part of this journey. Your feedback, suggestions, and contributions are not only valued but crucial in shaping the future of this open-source project. Let's work together to make Kafka Connector maintenance a seamless experience for all.

**Get Involved!**

* Contribute to the project on [GitHub](https://github.com/snapp-incubator/connector-guardian).
* Share your experiences and ideas in the [issues](https://github.com/snapp-incubator/connector-guardian/issues) section.
* Spread the word - let others in your network know about Connector Guardian.
* Stay tuned for updates and new features!