Learn how to use Kafkacat – the most versatile Kafka CLI client
Tags: apachekafka, kafkacat, tutorial
Published: 2020-05-05 15:59:54

---
title: Learn how to use Kafkacat – the most versatile Kafka CLI client
published: true
date: 2020-05-04 18:47:42 UTC
tags: ApacheKafka, kafkacat, tutorial
canonical_url: https://codingharbour.com/apache-kafka/learn-how-to-use-kafkacat-the-most-versatile-cli-client/?utm_source=rss&utm_medium=rss&utm_campaign=learn-how-to-use-kafkacat-the-most-versatile-cli-client
---

Kafkacat is an awesome tool and today I want to show you how easy it is to use it and what are some of the cool things you can do with it.

All the features explained below are available in version 1.5.0.


> Looking for a quick Kafkacat reference? Download the **[Kafkacat 1-page cheatsheet](https://codingharbour.com/kafkacat-cheatsheet/)**

## Installing Kafkacat

Kafkacat is available from Homebrew (latest version) and some Linux repositories, but it is possible that Linux repos don’t contain the latest version. If that’s the case, you can always run the latest kafkacat from [docker](https://hub.docker.com/r/edenhill/kafkacat/tags).

## The basics

Kafkacat is a command-line tool for producing and consuming Kafka messages. In addition, you can view metadata about the cluster or topics.

Kafkacat has quite a few parameters and it might look scary learning them all, yet (most of) the parameters make sense and are easy to remember. Let’s start with the most important: **modes**. When making a call to Kafkacat, you’ll always use it in one of the four modes it has. All the modes use capital letter:

- -P = **P**roduce data
- -C = **C**onsume data
- -L = **L**ist metadata
- -Q = **Q**uery 

The next most important option is the **b**roker list (-b) and after that, it’s usually **t**opic (-t).

So you can almost write your command like a story. The following command:

```bash
kafkacat -C -b localhost:9092 -t topic1 -o beginning
```

could be read as: I want to **C**onsume from **b**roker localhost:9092 and **t**opic topic1 with **o**ffset set to the beginning.

Ok, now that I have hopefully convinced you that all those cryptical parameters make sense, let’s look at how to use Kafkacat to achieve some common tasks.

## Producing data (-P)

What do we need so we could produce data? At a minimum, you need a broker and a topic you want to write to.

**Produce values**

```bash
kafkacat -P -b localhost:9092 -t topic1
```

Default message separator is Enter. Type your messages, and separate them with Enter.

**Producing keys and values**

If you want to produce messages with key, you need to specify the **K**ey delimiter (-K). Let’s use a colon to separate the key and the message in the input:

```bash
kafkacat -P -b localhost:9092 -t topic1 -K :
key3:message3
key4:message4
```

Notice that parameter uses capital K.

**Produce messages with headers**

If you want to add **h**eaders to the messages, add them using -H parameter, in a key=value format:

```bash
kafkacat -P -b localhost:9092 \
-t topic1 \
-H appName=kafkacat -H appId=1
```

As you see, additional headers are added by repeating -H flag. Note that all the messages produced will have the two headers specified with -H flag.

**Produce data from a file**

If you want to produce data using a file, use the option -l (as in: fi**l**e)… I did say that _most of the parameters_ are easy to remember :). Let’s say we have a file called data.txt containing key-value pairs, separated by a colon:

```
key1:message1
key2:message2
key3:message3
```

So the command would be:

```bash
kafkacat -P -b localhost:9092 -t topic1 -K: -l data.txt
```

Produce message with compression

Using a (-z) parameter you can specify message compression:

```bash
kafkacat -P -b localhost:9092 -t topic1 -z snappy
```

Supported values are: snappy, gzip and lz4.

## Consuming data (-C)

### Simple consumer

**Consume all the messages from a topic**

```bash
kafkacat -C -b localhost:9092 -t topic1
```

Note that, unlike kafka-console-consumer, kafkacat will consume the messages from the beginning of the topic by default. This approach makes more sense to me, but YMMV.

**Consume X messages**

You can control how many messages will be consumed using the **c**ount parameter (-c, lowercase).

```bash
kafkacat -C -b localhost:9092 -t topic1 -c 5
```



### Consuming from an offset

If you want to read data from a particular **o**ffset, you can use the -o parameter. The offset parameter is very versatile. You can:

**Consume messages from the beginning or end**

```bash
kafkacat -C -b localhost:9092 -t topic1 -o beginning
```

Use constants **beginning** or **end** to tell kafkacat where to begin the consumption.

**Consume from a given offset**

```bash
kafkacat -C -b localhost:9092 -t topic1 -o 123
```

Use an absolute value for the offset and Kafkacat will start consuming from the given offset. If you don’t specify the partition to consume, Kafkacat will consume all the partitions from the given offset.

**Consume last X messages in a partition(s)**

```bash
kafkacat -C -b localhost:9092 -t topic1 -o -10
```

We do this by using a negative offset value.

### **Consume based on a timestamp**

It is possible to start consuming after a given timestamp in milliseconds using the format -o s@start\_timestamp. Technically this is consuming based on an offset, the difference is that kafkacat figures out the offset for you based on the provided timestamp(s).

```bash
kafkacat -C -b localhost:9092 -t topic1 -o s@start_timestamp
```

You can also stop consuming when a given timestamp is reached using:

```bash
kafkacat -C -b localhost:9092 -t topic1 -o e@end_timestamp
```

This is very useful when you are debugging an error that occurred, you have the timestamp of the error, but you want to check how the message looked. Then, combining the start and end offset, you can narrow down your search:

```bash
kafkacat -C -b localhost:9092 -t topic1 -o s@start_timestamp -o e@end_timestamp
```

### Formatting the output

By default, Kafkacat will print out only the message payload (value of the Kafka record), but you can print anything you’re interested in. To define the custom output, specify (-f) flag, as in **f**ormat, followed by a format string. Here’s an example that prints a string with key and value of the message:

```bash
kafkacat -C -b localhost:9092 -t topic1 \
 -f 'Key is %k, and message payload is: %s \n'
```

%k and %s are format string tokens. The output might be something like this:

```bash
Key is key3, and message payload is: message3
Key is key4, and message payload is: message4
```

So what can you print out using format string?

- Topic (%t), 
- partition (%p)
- offset (%o)
- timestamp (%T)
- message key (%k)
- message value (%s)
- message headers (%h)
- key length (%K)
- value length (%S)

As you’ve seen above, you can use newline (\n \r) or tab characters(\t) in the format string as well.

### Serdes

If messages are not written as strings, you need to configure a proper **s**erde for keys and values using -s parameter.

For example, if both key and value are 32-bit integers, you would read it using:

```bash
kafkacat -C -b localhost:9092 -t topic1 -s i
```

You can specify separately serde for the key and value using:

```bash
kafkacat -C -b localhost:9092 -t topic1 -s key=i -s value=s
```

You will find the list of all the serdes in a kafkacat help (kafkacat -h).

### Avro serde

Avro messages are a bit special since they require a schema registry. But Kafkacat has you covered there as well. Use (-r) to specify the schema registry URL:

```bash
kafkacat -C -b localhost:9092 \
-t avro-topic \
-s key=s -s value=avro \
-r http://localhost:8081
```

In the example above, we’re are reading messages from a topic where keys are strings, but values are Avro.

## List metadata (-L)

Listing metadata gives you info about topics: how many partitions it has, which broker is a leader for a partition as well as the list of in-sync replicas (isr).

**Metadata for all topics**

```bash
kafkacat -L -b localhost:9092 
```

Simply calling -L with no other parameters will display the metadata for all the topics in the cluster.

**Metadata for a given topic**

If you want to see metadata for just one topic, specify it using (-t) parameter:

```bash
kafkacat -L -b localhost:9092 -t topic1
```

## Query mode (-Q)

If you want to find an offset of a Kafka record based on a timestamp, Query mode can help with that. Just specify the topic, partition and a timestamp:

```bash
kafkacat -b localhost:9092 -Q -t topic1:1:1588534509794
```

## Is this all?

I'm glad that you asked because it's not :) I have created a **1-page Kafkacat cheatsheet** for you to download. Grab it [here](https://codingharbour.com/kafkacat-cheatsheet/).