Deploy Celery & RabbitMQ with Compose
Tags: python, celery, tutorial
Published: 2021-04-07 13:42:48

Off late, I had been working a lot with FastAPI. Recently, however I had a requirement. I needed to run tasks in the background after the request was made. So something like, when the request is recieved, add it to the task list and return a response. The task will be done in the background.

I somewhat knew about [Celery](https://docs.celeryproject.org/), however never had the need to work with it. So I finally decided to use Celery and boy oh boy, was I surprised.

## What is Celery

So, Celery is a task scheduler. What I described above in the first paragraph, Celery does exactly that! We pass it a task and it runs that in the background.

It basically runs the task as a synchornous function which effectively makes the function run in the background. [You can read more about it here](https://docs.celeryproject.org/en/stable/getting-started/introduction.html).

However, in order for Celery to run properly, it needs a **broker**.

### What is a broker?

A broker will keep a list of all the tasks that are to be executed and will accordingly supply Celery with the tasks. Celery will then use the task and work on it.

Directly from Google:

> A broker is a person or firm who arranges transactions between a buyer and a seller for a commission when the deal is executed.

This pretty much sums it up. In our case, we are the **seller**, celery is the **buyer** and we will use a **broker** in between in order to handle all the tasks. Brokers are also called as message queues or task queues.

Some of the brokers that Celery works with are:

- [RabbitMQ](https://www.rabbitmq.com/)
- [Redis](https://redis.io/)

### RabbitMQ

In this article, I will primarily use **RabbitMQ** as the broker. You can read on how to [use Redis with Celery](https://docs.celeryproject.org/en/stable/getting-started/brokers/redis.html).

Easiest way to setup RabbitMQ is to use a docker file. Using the following command, a container with RabbitMQ can be deployed within seconds.

```console
docker run -d --rm -it --hostname my-rabbit -p 15672:15672 -p 5672:5672 rabbitmq:3-management
```

> In the above command, the `management` image is used. You can check [other available images here](https://hub.docker.com/_/rabbitmq)

Breaking down the above command:

- We are mapping the `15672` port of the container to our host
- We are mapping the `5672` port of the container to our host.

This is because the `15672` port serves the GUI for rabbitmq and `5672` is how **Celery** will communicate with it.

## Worker

Now that we have our broker in place, let's use a Dockerfile to deploy **Celery**. Celery, since it does tasks in the background, is referred to as **worker**.

We will build the `worker` with the following Dockerfile:

```dockerfile
FROM python:3.6

# copy contents of project into docker
COPY ./ /app/

# We will use internal functions of the API
# So install all dependencies of the API
RUN cd app && pip install -r requirements.txt

WORKDIR /app

ENTRYPOINT celery -A worker worker --loglevel=INFO
```

Using the above Dockerfile, we can deploy the worker.

## Using a compose file

Now that we have two of the services ready, we are ready to write our **docker compose** file. [Read more about docker compose here](https://docs.docker.com/compose/gettingstarted/).

> Usually, the worker is run along with an API and the API makes calls to the worker in order to run the worker tasks in the background.

In our case, we will be creating two containers:

- RabbitMQ container
- Worker container

We want our worker to access the rabbitMQ container through the network and accordingly use it as a broker.

> Most of the time, you'll probably also need an API container that will also interact with the worker using the network.

Following is the compose file:

```yaml
version: "3.7"

services:
  # Deploy the broker.
  rabbitmq_server:
    image: rabbitmq:3-management
    ports:
      # Expose the port for the worker to add/get tasks
      - 5672:5672
      # OPTIONAL: Expose the GUI port
      - 15672:15672

  # Deploy the worker
  worker:
    # Build using the worker Dockerfile
    build:
      context: .
      dockerfile: worker.Dockerfile
    # Need to access the database
    # OPTIONAL: If you worker needs to access your db that is deployed
    # locally, then make the network mode as host.
    network_mode: host
    # Pass the rabbitmq_uri as env varible in order to
    # connect to our service
    environment:
      # NOTE: Below we are using 127.0.0.1 because this container
      # will run on the host network, thus it will have access to the
      # host network.
      # If it would not have run locally, we would have had to
      # connect using the service name like following:
      # amqp:rabbitmq_server:5672
      rabbitmq_uri: amqp://127.0.0.1:5672
    # Make it wait for rabbitmq deployment
    depends_on: 
      - rabbitmq_server
```

With the above file, you can deploy it as follows:

```console
docker-compose -f docker-compose.yml up --detach --scale worker=2 --build
```

> In the above command, we are scaling the `worker` service to have 2 containers.

## Gotcha's to look out for

### Connection URI for RabbitMQ

Let's say we have RabbitMQ deployed in a container called `rabbitmq`. Now, from our `worker` container we need to access RabbitMQ in order to add tasks. In this case, we will have to connect to RabbitMQ using a connection URI. This URI will be something like:

```amqp://rabbitmq:5672```

> Note that we have name of the container in the URI. This will map the URI to the network of that container.

Typically this URI should be something like `amqp://localhost:5672`

However, now, let's say we need to run our container in the network. This can be easily done using the `network_mode: host` field in the compose file or the `--network=host` arguement to the deploy command.

In cases like this, our container will have the network of the host which means the RabbitMQ container will be accessible as it will be accessible to the network which will be:

```amqp://127.0.0.1:5672```

> Note that we exposed the port 5672 when deploying the rabbitmq container.

This article is posted on [my personal blogging site](https://blog.deepjyoti30.dev/celery_compose)