Kafka Connect - Deep Dive into Single Message Transforms
Tags: apachekafka, kafkaconnect, dataengineering, streamprocessing
Published: 2021-01-04 16:34:52

---
title: Kafka Connect - Deep Dive into Single Message Transforms
published: true
date: 2021-01-04 00:00:00 UTC
tags: apachekafka,kafkaconnect,dataengineering,streamprocessing
canonical_url: https://rmoff.net/2021/01/04/kafka-connect-deep-dive-into-single-message-transforms/
---

[KIP-66](https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect) was added in Apache Kafka 0.10.2 and brought new functionality called **Single Message Transforms** (SMT). Using SMT you can modify the data and its characteristics as it passes through Kafka Connect pipeline, without needing additional stream processors. For things like manipulating fields, changing topic names, conditionally dropping messages, and more, SMT are a perfect solution. If you get to things like aggregation, joining streams, and lookups then SMT may not be the best for you and you should head over to [Kafka Streams](https://kafka.apache.org/documentation/streams/) or [ksqlDB](https://ksqldb.io) instead.

I recently completed a twelve-day exercise of digging into many of the Single Message Transform that are available - almost all of them ship with Apache Kafka itself. For each one I recorded a video, wrote up a blog detailing the SMT, and built a test environment in Docker so that **you** can go and try it out too :-)

## ✨ The Highlights ✨

SMT as a concept are a highlight of Kafka Connect in themselves, but here are a handful of the ones that thought were _particularly_ neat:

- [Add the timestamp of a field to the topic name](https://rmoff.net/2020/12/23/twelve-days-of-smt-day-12-community-transformations#_add_the_timestamp_of_a_field_to_the_topic_name)

- [Filtering out null records](https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/#_filtering_out_null_records)

- [Conditionally renaming fields based on the topic name](https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter/#_conditionally_renaming_fields_based_on_the_topic_name)

- [Changing the topic name to which a source connector writes](https://rmoff.net/2020/12/11/twelve-days-of-smt-day-4-regexrouter/#_changing_the_topic_name_to_which_a_source_connector_writes)

- [Changing the data type of fields as they pass through Kafka Connect](https://rmoff.net/2020/12/18/twelve-days-of-smt-day-9-cast/#_changing_the_data_type_of_fields_as_they_pass_through_kafka_connect)

## 🎥 Videos Playlist

[![smtplaylist](https://rmoff.net/images/2021/01/smtplaylist.png)](https://www.youtube.com/watch?v=3Gj_SoyuTYk&list=PL5T99fPsK7pq7LiaaL-S6b7wQqzxyjgya)

## 👾 Code

You can grab the Docker Compose and tutorial files on [GitHub](https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-single-message-transforms)

## 📝 The Complete List

Here are links to the blogs and videos of each Single Message Transform:

- [Community Transformations](https://rmoff.net/2020/12/23/twelve-days-of-smt-day-12-community-transformations)

- [Predicate and Filter](https://rmoff.net/2020/12/22/twelve-days-of-smt-day-11-predicate-and-filter)

- [ReplaceField](https://rmoff.net/2020/12/21/twelve-days-of-smt-day-10-replacefield)

- [Cast](https://rmoff.net/2020/12/18/twelve-days-of-smt-day-9-cast)

- [TimestampConverter](https://rmoff.net/2020/12/17/twelve-days-of-smt-day-8-timestampconverter)

- [TimestampRouter](https://rmoff.net/2020/12/16/twelve-days-of-smt-day-7-timestamprouter)

- [InsertField II](https://rmoff.net/2020/12/15/twelve-days-of-smt-day-6-insertfield-ii)

- [MaskField](https://rmoff.net/2020/12/14/twelve-days-of-smt-day-5-maskfield)

- [RegExRouter](https://rmoff.net/2020/12/11/twelve-days-of-smt-day-4-regexrouter)

- [Flatten](https://rmoff.net/2020/12/10/twelve-days-of-smt-day-3-flatten)

- [ValueToKey and ExtractField](https://rmoff.net/2020/12/09/twelve-days-of-smt-day-2-valuetokey-and-extractfield)

- [InsertField (timestamp)](https://rmoff.net/2020/12/08/twelve-days-of-smt-day-1-insertfield-timestamp)