# Redis Streams

## Summary


Redis has data types that could be used for events or message sequences but with different tradeoffs. Sorted sets are memory hungry. Clients can't block for new messages. It's also not a good choice for time series data since entries can be moved around. Lists don't offer fan-out: a message is delivered to a single client. List entries don't have fixed identifiers. For 1-to-n workloads, there's Pub/Sub but this is a "fire-and-forget" mechanism. Sometimes we wish to keep history, make range queries, or re-fetch messages after a reconnection. Pub/Sub lacks these properties. 

Redis Streams addresses these limitations. Stream data type can be seen as similar to logging, except that Stream is an abstraction that's more performant due to logical offsets. It's built using radix trees and listpacks, making it space efficient while also permitting random access by IDs.

## Discussion

### What are some use cases for Redis Streams?

Redis Streams is useful for building chat systems, message brokers, queuing systems, event sourcing, etc. Any system that needs to implement unified logging can use Streams.  Queuing apps such as Celery and Sidekiq could use Streams. Slack-style chat apps with history can use Streams. 

For IoT applications, Streams can run on end devices. This is essentially time-series data that Streams timestamps for sequential ordering. Each IoT device will store data temporarily and asynchronously push these to the cloud via Streams. 

While we could use Pub/Sub along with lists and hashes to persist data, Stream is a better data type that's designed to be more performant. Also, if we use Pub/Sub and Redis server is restarted, then all clients have to resubscribe to the channel. 

Since Streams supports blocking, clients need not poll for new data. Blocking enables real-time applications, that is, clients can act on new messages as soon as possible. 


### Which are the new commands introduced by Redis Streams?

All commands of Redis Streams are documented online. We briefly mention them: 

    + **Adding**: `XADD` is the only command for adding data to a stream. Each entry has a unique ID that enables ordering.
    + **Reading**: `XREAD` and `XRANGE` read items in the order determined by the IDs. `XREVRANGE` returns items in reverse order. `XREAD` can read from multiple streams and can be called in a blocking manner.
    + **Deleting**: `XDEL` and `XTRIM` can remove data from the stream.
    + **Grouping**: `XGROUP` is for managing consumer groups. `XREADROUP` is a special version of `XREAD` with support for consumer groups. `XACK`, `XCLAIM` and `XPENDING` are other commands associated with consumer groups.
    + **Information**: `XINFO` shows details of streams and consumer groups. `XLEN` gives number of entries in a stream.

### What are main features of Redis Streams?

Streams is a first-class citizen of Redis. It benefits from the usual Redis capabilities of persistency, replication and clustering. It's stored in-memory and under a single key.  

The main features of Streams are: 

    + **Asynchronous**: Producers and consumers need not be simultaneously connected to the stream. Consumers can subscribe to streams (push) or read periodically (pull).
    + **Blocking**: Consumers need not keep polling for new messages.
    + **Capped Streams**: Streams can be truncated, keeping only the N most recent messages.
    + **At-Least Once Delivery**: This makes the system robust.
    + **Counter**: Every pending message has a counter of delivery attempts. We can use this for dead letter queuing.
    + **Deletion**: While events and logs don't usually have deletion as a feature, Streams supports this efficiently. Deletion allows us to address privacy or regulatory concerns.
    + **Persistent**: Unlike Pub/Sub, messages are persistent. Since history is saved, a consumer can look at previous messages.
    + **Lookback Queries**: This helps consumers analyse past data, such as, obtain temperature readings in a particular 10-second window.
    + **Scale-Out Options**: Via consumer groups, we can easily scale out. Consumers can share the load of processing a fast-incoming data stream.

### Could you explain consumer groups in Redis?

A consumer group allows consumers of that group to share the task of consuming messages from a stream. Thus, a message in a stream can be consumed by only one consumer in that consumer group. This relieves the burden on a consumer to process all messages. 

Command `XGROUP` creates a consumer group. A consumer is added to a group the first time it calls `XREADGROUP`. A consumer always has to identify itself with a unique consumer name. 

A stream can have multiple consumer groups. Each consumer group tracks the ID of the last consumed message. This ID is shared by all consumers of the group. Once a consumer reads a message, it's ID is added to a *Pending Entries List (PEL)*. The consumer must *acknowledge* that it has processed the message, using `XACK` command. Once acknowledged, the pending list is updated. Another consumer can *claim* a pending message using `XCLAIM` command and begin processing it. This helps in recovering from failures. However, a consumer can choose to use the `NOACK` subcommand of `XREADGROUP` if high reliability is not important. 


### Could you share more details about IDs in Redis Streams?

Entries within a stream are ordered using IDs. Each ID has two parts separated by hyphen: UNIX millisecond timestamp followed by sequence number to distinguish entries added at the same millisecond time. Each part is a 64-bit number. For example, `1526919030474-55` is a valid ID. 

IDs are autogenerated when `XADD` command is called. However, a client can specify its own ID but it should be an ID greater than all other IDs in the stream. 

Incomplete IDs are when the second part is omitted. With `XRANGE`, Redis will fill in a suitable second part for us. With `XREAD`, the second part is always `-0`. 

Some IDs are special:

    + `$`: Used with `XREAD` to block for new messages, ignoring messages already in the stream.
    + `-` & `+`: Used with `XRANGE`, to specify minimum and maximum IDs possible within the stream. For example, the following command will return every entry in the stream: `XRANGE mystream - +`
    + `>`: Used with `XREADGROUP`, to get new messages (never delivered to other clients). If this command uses any other ID, it has the effect of returning pending entries of that client.

### In what technical aspects does Redis Streams differ from other Redis data types?

Unlike other Redis blocking commands that specify timeouts in seconds, commands `XREAD` and `XREADGROUP` specify timeouts in milliseconds. Another difference is that when blocking on list pop operations, the first client will be served when new data arrives. With Stream `XREAD` command, every client blocking on the stream will get the new data. 

When an aggregate data type is emptied, its key is automatically destroyed. This is not the case with Stream data type. The reason for this is to preserve the state associated with consumer groups. Stream is not deleted even if there are no consumer groups but this behaviour may be changed in future versions. 


### How does Redis Streams compare against Kafka?

Apache Kafka is a well-known alternative for Redis Streams. In fact, some features of Streams such as consumer groups have been inspired by Kafka. 

However, Kafka is said to be difficult to configure and expensive to operate on typical public clouds. Streams is therefore a better option for small, inexpensive apps. 


### Could you share some performance numbers on Redis Streams?

In one test on a two-core machine with multiple producers and consumers, messages were generated at 10K per second. With `COUNT 10000` given to `XREADGROUP` command, every iteration processed 10K messages. It was seen that 99.9% requests had a latency of less than 2 ms. Real-world performance is expected to be better than this. 

When compared against traditional Pub/Sub messaging, Streams gives 100x better throughput. It's able to handle more than 1 million operations per second. If Pub/Sub messages are persisted to network storage, latency is about 5 ms. Streams has less than 1 ms latency. 


### Could you share some developer tips for using Redis Streams?

There are dozens of Redis clients in various languages. Many of these have support for Streams.

Use `XREAD` for 1-to-1 or 1-to-n messaging. Use `XRANGE` for windowing-based stream processing. Within a consumer group, if a client fails temporarily, it can reread messages from a specific ID. For permanent failures, other clients can claim pending messages. 

For real-time streaming analytics, one suggestion is to pair up Redis Streams with Apache Spark. The latter has the feature Structured Streaming that pairs up nicely with Streams. To scale out, multiple Spark jobs can belong to a single consumer group. Since Streams is persistent, even if a Spark job restarts, it won't miss any data since it will start consuming from where it left off. 

## Milestones

Sep  
2017

Salvatore Sanfilippo, creator of Redis, gives a demo of Redis Streams and explains the API. In October, he blogs about it. He explains that the idea occurred much earlier. He tinkered with implementing a generalization of sorted sets and lists but was not happy with the results. When Redis 4.0 came out with support for modules, Timothy Downs created a data type for logging transactions. Sanfilippo used this as an inspiration to create Redis Streams. 

May  
2018

The first release candidate RC1 of Redis 5.0 is released. This supports **Stream** data type. 

Jul  
2018

Beta version of Redis Enterprise Software (RS) 5.3 is released. This is based on Redis 5.0 RC3 with support for Stream data type. 

Oct  
2018

Redis 5.0.0 is released. 

May  
2019

At the Redis Conference 2019, Dan Pipe-Mazo talks about Atom, a microservices SDK powered by Redis Streams. Microservices interact with one another using Streams.