Trigger a task with FastAPI, Celery, Docker, and Docker Compose: A Step-by-Step Guide
Tags: fastapi, celery, docker
Published: 2023-06-21 18:30:08

[![Awesome](https://img.shields.io/badge/Awesome-Celery-informational?style=for-the-badge&logo=awesomelists&labelColor=17202A&color=1abc9c&logoColor=1abc9c)](https://docs.celeryq.dev/en/stable/getting-started/introduction.html)

## Table of Contents
- [Introduction](#introduction)
- [Project Structure](#project-structure)
- [Dockerization](#dockerization)
- [Up and Run](#up-and-run)
- [Trigger a task](#trigger-a-task)

### Introduction

In today's digital landscape, background tasks have become a crucial aspect of many applications. Whether it's automating repetitive processes or achieving instant responses, one popular solution that comes to the rescue is `Celery`. In this article, we'll guide you through the process of creating and managing background tasks using Celery. By seamlessly integrating it with `FastAPI`, we'll show you how to containerize your project. Get ready for a straightforward and exciting exploration!

**<u>FastAPI:</u>** It's is a modern web framework for building APIs with Python. Its utilization of asynchronous programming techniques combined with the power of the `Starlette` framework allows `FastAPI` to achieve remarkable performance levels.

**<u>Celery:</u>** A python-based distributed ask queue system with built-in support for task scheduling, result tracking and fault tolerance.

**<u>Docker:</u>** An open-source platform, empowers you to bundle applications and their dependencies into compact, portable containers. These containers guarantee a consistent runtime environment, enabling your application to run seamlessly on any host system. 

**<u>Docker Compose:</u>** A tool that simplifies the management of multi-container applications. It allows you to define and configure multiple Docker containers as a single application using a `YAML` file. It also specifies the dependencies and relationships between different services.

### Project Structure

```sh
celery-with-fastapi/
├── app/
│   ├── celery/
│   │   ├── app.py
│   │   ├── worker.py
│   │   └── __init__.py
│   ├── core/
│   │   ├── config.py
│   │   └── __init__.py
│   ├── __init__.py
│   └── main.py
├── .env
├── .gitignore
├── docker-compose.yml
├── Dockerfile
├── README.md
├── requirements.txt
└── run.sh
```

### Dockerization

Make sure you've installed `Docker` and `docker-compose` in your pc. Your Dockerfile and docker-compose.yml file would be like as follows

> `Dockerfile`
```Dockerfile
# Base image
FROM python:3.10-buster

# set work directory
WORKDIR /app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install dependencies
COPY [ "requirements.txt", "run.sh", "./"]
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt && chmod +x ./run.sh

# copy project
COPY . .

# Set entrypoint
ENTRYPOINT bash ./run.sh
```
> `run.sh`
```sh
celery -A app.celery.app worker  -l info --concurrency=2 &
celery -A  app.celery.app flower -l info & celery -A app.celery.app beat -l INFO &
uvicorn app.main:app --host 0.0.0.0 --reload --workers 2
```

> `docker-compose.yml`

```yml
version: "3.8"

services:
  web:
    build: .
    ports:
      - 8001:8000
      - 5556:5555
    volumes:
      - .:/app
    depends_on:
      - celerybackend

  celerybackend:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: [ "CMD", "redis-cli", "ping" ]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always
```

### Up and Run

[This is the final code](https://github.com/shahnawaz-pabon/celery-with-fastapi.git). Before getting started, clone the repository first.

Create a `.env` file and copy these lines after cloning the project:

```sh
# .env file
CELERY_BROKER_URL=redis://celerybackend:6379/0
CELERY_RESULT_BACKEND=redis://celerybackend:6379/0
```

```sh
git clone https://github.com/shahnawaz-pabon/celery-with-fastapi.git
cd celery-with-fastapi
docker-compose up --build
```

You'll be able to see the backend project running at [http://0.0.0.0:8001/](http://0.0.0.0:8001/) and the flower dashboard running at [http://0.0.0.0:5556/](http://0.0.0.0:5556/)

### Trigger a task

Open a terminal and trigger a new task

```sh
curl http://localhost:8001/tasks -H "Content-Type: application/json" --data '{"type": 0}'
```

To know more about celery, [please visit this](https://docs.celeryq.dev/en/stable/getting-started/introduction.html).

---

I would like to express my heartfelt gratitude to each and every one of you who took the time to read my articles. Your support and engagement mean the world to me. :pray::pray::pray: