Using Celery with RabbitMQ's Lazy Queue
Tags: celery, rabbitmq, lazyqueue
Published: 2020-04-15 01:57:15

---
title: Using Celery with RabbitMQ's Lazy Queue
published: true
tags: celery,rabbitmq,lazy queue
canonical_url: https://blog.whtsky.me/tech/2020/using-celery-with-rabbitmqs-lazy-queue/
---

## Celery & RabbitMQ basics

Skip this part if you know what Celery & RabbitMQ is.

[Celery](https://docs.celeryproject.org/en/stable/index.html) is a task queue system in Python. Celery can help you run something in the background, schedule cronjobs and distribute workloads across multiple servers.

[RabbitMQ](https://www.rabbitmq.com/) is a [message broker](https://en.wikipedia.org/wiki/Message_broker). A message broker is a program to help you send messages. Celery needs to use a message broker to send & receive messages, and they [highly recommend using RabbitMQ](https://docs.celeryproject.org/en/stable/getting-started/first-steps-with-celery.html#rabbitmq).

## What is Lazy Queue?

When you publish a message into a RabbitMQ's regular queue, the message is stored into the memory so they can be delivered to consumers pretty fast. RabbitMQ will try to keep as much as messages it can until it [hit a memory threshold](https://www.rabbitmq.com/memory.html#threshold). When it happens, RabbitMQ will save messages to disk(["page out"](https://www.rabbitmq.com/persistence-conf.html)), which will block the queue and prevent publishers from publishing any new messages.

As you may notice, if you have a lot of messages to publish, a regular queue will

1. use a lot of memory, depending on your message size and number
2. sometimes stop receiving new messages ( when paging out ), make the performance less predictive.

A [lazy queue](https://www.rabbitmq.com/lazy-queues.html) will try to save the message to disk as early as possible, therefore minimize the memory usage ( use constant space in most cases ) with the cost of throughput time, disk I/O and CPU usage.

## Should I use Lazy Queue?

I'd always use lazy queues unless there are some really high-performance requirements. Lazy queues [use significantly less memory](https://www.rabbitmq.com/lazy-queues.html#performance) and can hold a lot more messages than regular queues. [CloudAMQP recommends to enable lazy queues to get predictable performance.](https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#enable-lazy-queues-to-get-predictable-performance).

But before you do, I highly recommend you to carefully read the [Caveats and Limitations](https://www.rabbitmq.com/lazy-queues.html#caveats-limitations) section of lazy queues' documentation, and do some benchmarks yourself.

## How to Celery with RabbitMQ's Lazy Queue

### Specify queue arguments in Celery

Say you have a celery app

```python
from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_routes = ([
    ('account.tasks.*', {
        'queue': 'account'
    }),
    ('server.tasks.*', {
        'queue': 'server'
    })
])
```

You can specify `queue_arguments` to make queues lazy.

```python
from celery import Celery
from kombu import Queue

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_queues = [
    Queue('account', queue_arguments={'x-queue-mode': 'lazy'}),
    Queue('server', queue_arguments={'x-queue-mode': 'lazy'}),
]

app.conf.task_routes = ([
    ('account.tasks.*', {
        'queue': 'account'
    }),
    ('server.tasks.*', {
        'queue': 'server'
    })
])
```

Or, if you haven't specified any task routes

```python
from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y
```

In this case, Celery will route them into a default queue called [`celery` by default](https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-default-queue). You need to change the default queue into lazy mode.

```python
from celery import Celery
from kombu import Queue

app = Celery('tasks', broker='amqp://guest@localhost//')

app.conf.task_queues = [
    Queue('celery', queue_arguments={'x-queue-mode': 'lazy'}),
]


@app.task
def add(x, y):
    return x + y
```

If the queue is already created, you may encounter a `PRECONDITION_FAILED` error saying the queue was declared with `inequivalent arg 'x-queue-mode'`. You should consider deleting the queue ( WILL LOSE ALL MESSAGES INSIDE IT ) or changing queue mode via policy.

### Changing Queue Mode using RabbitMQ's policy

You can also change queues' mode using RabbitMQ's [policy](https://www.rabbitmq.com/parameters.html#policies). This allows you to change an existing queue's mode without removing & recreating it. You can set a policy in shell using `rabbitmqctl`:

```bash
rabbitmqctl set_policy Lazy "^celery$" '{"queue-mode":"default"}' --apply-to queues
```

The above command changes the queue `celery` into lazy mode.

I prefer to specify queue arguments in Celery since it can be version controlled and don't require extra operations on the production server.

Enjoy.

_Originally published at https://blog.whtsky.me/tech/2020/using-celery-with-rabbitmqs-lazy-queue/_