Redis is more than a Cache #1 - Delaying Jobs
Published: 2024-03-15 23:18:06

My current company - [Luma Health Inc](https://www.lumahealth.io/) - has an `Event-Driven Architecture` where all of our backend systems interact via async messaging/jobs. Thus our backbone is sustained by an AMQP broker - RabbitMQ - which routes the jobs to interested services.

Since our jobs are very critical - we cannot support failures AND should design to make the system more resilient - because well...we don't want a patient not being notified of their appointment, appointments not being created when they should, patients showing off into facilities where they were never notified the patient had something scheduled.

Besides the infra and product reliability - some use cases could need `postponing` - maybe reaching out to an external system who's offline/or not responding. Maybe some error which needs a retry - who knows? 

The fact is, delaying/retrying its a very frequent requirement into Event Driven Architectures. With this a service responsible for doing it was created - and it worked fine.

But - as the company sold bigger contracts and grew up in scale - this system was almost stressed out and not reliable.


## The Unreliable Design

Before giving the symptoms, let's talk about the organism itself - the service old design.

The design was really straightforward - if our service handlers asked for a postpone OR we failed to send the message to RabbitMQ - we just insert the JSON object from the Job into a Redis `Sorted Set` and using the `Score` as the timestamp which it was meant to be retried/published again.

To publish back into RabbitMQ the postponed messages, a job would be triggered each 5 seconds - doing the following:
1. Read from a `set` key containing all the existing `sorted set` keys - basically the queue name
2. Fetch run a `zrangebyscore` from 0 to current timestamp BUT `limit` to 5K jobs.
3. Publish the job and remove it from `sorted set`

## The Issues

This solution actually scaled up until 1-2 years ago when we started having issues with it - the main one's being:

1. It could not catch up to a huge backlog of delayed messages
2. It would eventually OOM or SPIKE up to 40GB of memory
	1. Due to things being fetched into memory AND some instability OR even internal logic could shovel too much data into Redis - the service just died :skull:
3. We could not scale horizontally - due to consuming and fetching objects into memory before deleting them.

## The solution

The solution was very simple: we implemented something that I liked to call `streaming approach`

Using the same data structure, we are now:
1. Running a `zcount` from 0 to current timestamp
	- Counting the amount of Jobs -> returning N
1. Creating an `Async Iterator` for N times - that used the `zpopmin` method from Redis
	- `zpopmin` basically returns AND removes the least score object - ie most recent timestamp

The `processor` for the SortedSet

![Processor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l5ntqt6ztmx2o4l68vzz.png)

The `Async Iterator`

![Async Iterator](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g8v24but6t63t1qhyvrs.png)

And that's _all_!

This simple algorithm change annihilated the need for:
1. Big In Memory fetches - makes our memory allocation big
2. Limit of 5K in fetches - makes our throughput lower


## Results

I think the screenshots can speak for themselves but:
- We processed the entire backlog of 40GB of pending jobs pretty quickly
- From a constant usage of ~8GB - we dropped down to ~200MB
- We are now - trying to be play safe and still oversize - safely allocating 1/4 of the resources.

Money-wise: We are talking at least of 1K USD/month AND more in the future if we can lower our Rediscache instance.

![Memory Usage Service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0wsttrpkjcwhuyzbpt03.png)

![Memory Usage Redis](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dbyqngwltyd7ekmz1z0n.png)

![Mem Alloc lowering PR](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/62khw4or8k8dzvubbivk.png)

### Note
We currently have more enhancements in the roadmap - such as making the job delaying via RPC, using different storages for different postpone amount (1milli, 1 second, 1 day, 1 week++) and making it more reliable overall.