An Overview About the Different Kafka Connect Plugins
Tags: apachekafka, kafkaconnect
Published: 2021-02-18 22:54:36

---
title: An Overview About the Different Kafka Connect Plugins
published: true
date: 2021-02-18 22:20:00 UTC
tags: apachekafka, kafkaconnect 
canonical_url: https://rueedlinger.ch/posts/2021/kafka_connect_plugins/
---

Kafka Connect is a great framework for connecting Kafka with external systems. In the best case you can use Connect right away. But in some special cases you might have to write your own plugins to add missing functionality to the framework. In this blog post I give a short overview about the different plugin types which can be used to add new functionality to Connect.

In the last part of this blog post I give you a short introduction to my GitHub project _[Ready, Steady, Connect - A Kafka Connect Quickstart](https://github.com/rueedlinger/kafka-connect-quickstart)_. This project contains example Java code you can use to extend Connect with your own plugins.

> The name of the project came from the blog post [(Ready, Steady, Connect. Help Your Organization to Appreciate Kafka)](https://medium.com/swlh/ready-steady-connect-help-your-organization-to-appreciate-kafka-8ed6cfcfd6d8)I wrote about the experience we had with Connect.

## Plugin Types

There are two main plugin categories that can be used to add new functionality to Kafka Connect:

- **Connect Plugins** are part of the Connect API and can be used to extend the functionality of Connect.
- **Kafka Client Plugins** are part of the Kafka Client (Consumer / Producer API). Kafka Connect is build on top of the _Kafka Consumer / Producer API_, so we have the possibility to write plugins which are part of these “lower API’s” and use them with Connect.

![plugin types](https://rueedlinger.ch/posts/2021/kafka_connect_plugins/plugin_types.png)

### Connect Plugins (Connect API)

| Plugin | Description | API |
| --- | --- | --- |
| Sink Connector | A `SinkConnector` can load data from Kafka and store it into an external system (eg. database). It’s quite easy to write your own sink connector or take an existing open source version and modify it to your needs. | [SinkConnector](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/sink/SinkConnector.html), [SinkTask](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/sink/SinkTask.html) |
| Source Connector | A `SourceConnector` can load data from an external system and store it into Kafka. A source connector is a bit more complicated to write than a sink connector. But with some inspiration from other open source connectors this should not be to hard. | [SourceConnector](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/source/SourceConnector.html), [SourceTask](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/source/SourceTask.html) |
| Single Message Transforms (SMTs) | With a `Transformation` (SMT) you can transform Kafka messages when they are processed by a connector. For example you could write a SMT which appends a UUID to every message that passes trough. | [Transformation](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/transforms/Transformation.html) |
| Predicates | A SMT can be configured with a `Predicate` ([KIP-585](https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs)). The SMT is only applied when the condition of the predicate was true. | [Predicate](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/transforms/predicates/Predicate.html) |
| Config Providers | A `ConfigProvider` loads configuration values from external resources. These configuration values can then be referenced in the connector configuration. You could write a `ConfigProvider` which loads configuration values from a database, rest endpoint or from environment variables. | [ConfigProvider](https://kafka.apache.org/27/javadoc/org/apache/kafka/common/config/provider/ConfigProvider.html) |
| Rest Extensions | With a `RestExtension` ([KIP-285](https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin)) you can extend the existing Kafka Connect Rest API. You could write an authorization filter or liveness/readiness endpoints for k8s. | [ConnectRestExtension](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html) |
| Converter | The `Converter` provides support for translating between Kafka Connect’s runtime data format and the raw payload of the Kafka messages (JSON, Avro, …). | [Converter](https://kafka.apache.org/27/javadoc/org/apache/kafka/connect/storage/Converter.html) |

### Kafka Client Plugins (Kafka Producer / Consumer API)

| Plugin | Description | API |
| --- | --- | --- |
| Kafka Consumer Interceptor | The `ConsumerInterceptor` ([KIP-42](https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors)) can be used to intercept Kafka messages before they are processed by the consumer. | [ConsumerInterceptor](https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerInterceptor.html) |
| Kafka Producer Interceptor | The `ProducerInterceptor` ([KIP-42](https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors)) is a neat way to intercept Kafka messages before they are published to Kafka. | [ProducerInterceptor](https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/producer/ProducerInterceptor.html) |
| Kafka Metrics Reporter | The `MetricsReporter` can be used to listen to Kafka client metrics and process them. | [MetricsReporter](https://kafka.apache.org/27/javadoc/org/apache/kafka/common/metrics/MetricsReporter.html) |

## Create Your Own Connect Plugins

The Docker image and the source code for all plugin examples can be found in the _[Ready, Steady, Connect - A Kafka Connect Quickstart (rueedlinger/kafka-connect-quickstart)](https://github.com/rueedlinger/kafka-connect-quickstart)_ repository.

The first step is to clone the project.

```
git clone https://github.com/rueedlinger/kafka-connect-quickstart
```

The main components of the project are the **Docker Image** , **Java source code** and the **Docker Compose** file.

- The custom Connect **Docker image** has two parts.
  - The **builder part** to build the example plugins from the Java source code.
  - The **main part** to run the Kafka Connect container with all the Kafka Connect plugins.
- The **Java source code** contains all the Kafka Connect plugin examples (connectors, transforms, etc.).
- The **Docker Compose** file can be used to run the whole infrastructure (Kafka broker, zookeeper, etc).

The next step is to build all the plugins (Java) and start the containers with Docker.

```
docker-compose up --build
```

When all containers are started you can access the following services:

- **Kafka Connect Rest API** => http://localhost:8083/
- **Kafdrop** from [Obsidian Dynamics (GitHub)](https://github.com/obsidiandynamics/kafdrop) => http://localhost:8082/
- **Schema Registry** from [Confluent (GitHub)](https://github.com/confluentinc/schema-registry) => http://localhost:8081/
- **Kafka UI** from [Provectus (GitHub)](https://github.com/provectus/kafka-ui) => http://localhost:8080/
- **Kafka Connect UI** from [Lenses.io (GitHub)](https://github.com/lensesio/kafka-connect-ui) => http://localhost:8000/

## Happy Coding

Now that everything is up and running. You can start to play around with Kafka Connect. I hope the _kafka-connect-quickstart_ project is useful and gives you an easy start into the world of Kafka Connect plugins.