What’s in a name? Making Sense of Apache Kafka’s auto.offset.reset
Tags: kafka, opensource, backend, programming
Published: 2024-05-22 15:17:16

Users of Apache Kafka® have varying opinions on so many things&mdash;it’s what makes the community diverse and interesting. But regardless of what your policy on schemas is (they’re obviously good) or whether you believe that adding partitions to a Kafka Topic is okay (it’s definitely not), I’ll bet you can still find some common ground with your fellow users. 

Case in point: pretty much everyone that works with Kafka can agree that `auto.offset.reset` is quite possibly the worst named Kafka configuration out there.

Driven by my frustration as a user and my own curiosity as a developer, I decided to dive into this configuration and try to understand it and its awful name, once and for all. 

But first…

## Some background on `auto.offset.reset`
I strongly believe that this particular origin story will be helpful for anyone, regardless of whether they’ve been using Kafka for years or days. So let’s start by getting everyone on the same page.

Kafka Consumers. You know them, right? They’re pretty useful in helping you to consume messages from Kafka Topics. As you read messages from a Topic, every so often, the Consumer will record the last message that it saw and processed using the offset of that message. This is really helpful so that, if the Consumer goes down for any reason, you can rest assured that, when it comes back online, the Consumer can use that stored offset to pick back up from its last committed position. 

Once our Consumers are up and running, offsets and Consumers go hand-in-hand. But what about _before_ the Consumer has started running? What about brand new Consumers that have never seen a single message from the Kafka Topic? That’s where `auto.offset.reset` comes in. 

Consumers that don’t have an offset available to use still need to know where to start reading from in the Topic. To put it simply, `auto.offset.reset` is a [Kafka Consumer configuration](https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset), which can be set to either `earliest` or `latest` (or `None`), that defines where a Consumer should begin reading from in the Kafka Topic _when it doesn’t have any other valid offsets to start from_. 

That last bit is really important, and it’s what trips folks up when they first try to use and understand this configuration parameter.

## So, what _is_ in a name?
The biggest issue that folks appear to have with `auto.offset.reset` is its name, which is a little disappointing because one would think that a three-word configuration would have the potential to be short, sweet, and to-the-point. Alas, `sasl.oauthbearer.jwks.endpoint.retry.backoff.ms` appears to both be more descriptive and far less problematic than `auto.offset.reset`.
 
So what does `auto.offset.reset` _imply_? What does someone using this configuration for the first time assume, erroneously, that it does? I’ve heard a number of variations, but generally folks assume that it automatically resets the stored Consumer offset to `earliest` or `latest`. 

That… would make sense, right? Except that we all know the frustration, the pain, the anguish, when we set `auto.offset.reset=earliest` and realize that our Kafka Consumer is not, in fact, automatically reading from the beginning of the Topic, nor did it reset our offsets. The reality is that, if there's a valid offset, `auto.offset.reset` doesn't do a single thing.

So what gives? Where is this disconnect with `auto` and `reset` coming from?

## Auto vs. Manual
Let’s start with `auto`. Take a look at the rest of the Kafka Consumer configurations that have `auto` in them; they all, truly, have to do with handling something automatically for the Consumer. 

What does `auto.offset.reset` do for us automatically? It certainly doesn't appear to be to overriding Consumer offsets automatically. But to really make sense of what `auto.offset.reset` does for us behind the scenes, we have to think about the manual version of this offset-setting process.

### A Consumer without an offset
Consider what would happen if  you didn’t want to set `auto.offset.reset`, but you had a brand new Consumer (or a Consumer whose offsets you've deleted).
 
If you read the `auto.offset.reset` [documentation](https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset) closely, you’ll find that you don’t actually have to set it. But, when `auto.offset.reset=None` and you have no stored offsets, when your Kafka Consumer tries to poll the Topic, it will throw an exception. This implies that, oops, our Kafka Consumers need initial offsets to actually read from the Topic.

### Setting offsets manually
A brand new Consumer or a Consumer without offsets doesn’t have to rely on `auto.offset.reset`, though. You have the option to manually set your offsets using [`Consumer.seek()`](https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seek(org.apache.kafka.common.TopicPartition,long))&mdash;as well as the related methods [`Consumer.seekToBeginning()`](https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seekToBeginning(java.util.Collection)) and [`Consumer.seekToEnd()`](https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seekToEnd(java.util.Collection)). 

`seek()` is an in-memory operation that takes in a TopicPartition and an offset and manually overrides the offset that the Consumer has for that partition&mdash;even if it currently has no offset for that partition. 

Never heard of `Consumer.seek()`? That’s probably because it’s mostly used in manual one-off scripts&mdash;at least, that’s been my experience with it.

Now that you know about the existence of a way to manually set Consumer offsets, doesn’t the `auto` in `auto.offset.reset` make just a little bit more sense? That is, until we think a bit more about the fact that it doesn’t reset anything…

## Reset, but only just sometimes

Okay, bear with me, here.

Let’s go back to the official Apache Kafka `auto.offset.reset` [documentation](https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset) and focus on a bit that I’ll bet you missed the first time around:

> What to do when there is no initial offset in Kafka _or if the current offset does not exist any more on the server (e.g. because that data has been deleted)_

Bingo.

Unless you have infinite retention set up, data doesn’t live in a Kafka Topic forever. Eventually, it’ll be deleted or compacted. If, for some reason, our Consumer is inactive and the message with the last offset that that Consumer has seen is deleted, then suddenly we have an invalid offset for that Topic&mdash;which, when you think about it, is just as bad as having no offset at all because having no offset is just a special case of an invalid offset...

Gee, it sure would be nice if our Consumer knew where to start from when it comes back online. If we have `auto.offset.reset` set up, automatically the Consumer would reset its invalid offset to one of `earliest` or `latest`, meaning that we don’t have to worry about the Consumer throwing an exception. 

## Automatic and resetting

So where are we with `auto.offset.reset` now?

In the event that there aren’t any valid offsets available for a Kafka Consumer&mdash;whether that means no offsets at all or a stale offset from a deleted message&mdash;`auto.offset.reset` defines how to _automatically reset_ the invalid offsets and allow the Consumer to continue operating.

This probably isn’t what you wanted to hear, but, after all of that, the name `auto.offset.reset` does a pretty decent job describing exactly what it does. Kafka users everywhere just needed a little bit more context to understand why this frustrating configuration is called what it’s called.

So what do you think? Is this explanation enough to make you just a little bit less frustrated at `auto.offset.reset`? If not, what would you call it?