My Docker stack to deploy a Django + Celery web app
Tags: docker, django, python, deploy
Published: 2024-04-18 09:48:50

{%- # TOC start -%}

- [Introduction](#introduction)
- [Project Structure](#project-structure)
- [Stack Diagram](#stack-diagram)
- [Docker images](#docker-images)
   * [Django image](#django-image)
   * [Nginx image](#nginx-image)
   * [Redis image](#redis-image)
- [Docker Compose](#docker-compose)
   * [Environment Variables](#environment-variables)
   * [The file](#the-file)
- [Deploying](#deploying)
   * [Building](#building)
   * [Deploying with Docker Compose](#deploying-with-docker-compose)

{%- # TOC end -%}
After many iterations, this is my current process to deploy a django application.

The cool thing is that I'm now deploying with only an `.env` file and nothing else.

Note that this is just a single instance of the stack, without Kubernetes or any kind of load balancer.

## Introduction

My stack consists on:
- A Postgres database
- A Redis database
- A django instance
- Celery beat and workers

I won't go into details of how to set up anything like this here, if you want to learn more about celery [in their docs](https://docs.celeryq.dev/en/stable/index.html).

---

## Project Structure

To understand the `docker-compose.yml` file below, it's important to see how I structure my django project:

```
MY-DJANGO-PROJECT/
├── core  # settings.py lives here
├── app1/
│   ├── migrations
│   ├── models.py
│   └── ...
├── app2/
│   ├── migrations
│   ├── models.py
│   └── ...
├── data  # A data directory where I store stuff like logs
├── nginx/
│   ├── certs/
│   │   ├── fullchain.pem
│   │   └── privkey.pem
│   ├── conf/
│   │   ├── default.conf
│   │   ├── prod.conf
│   │   └── staging.conf
│   └── Dockerfile
├── Dockerfile
├── entrypoint-django.sh
├── entrypoint-beat.sh
├── entrypoint-worker.sh
├── Pipfile
└── ...
```

---

## Stack Diagram

To visualize the flow better here's a diagram that describes how everything is interconnected:

![Docker Stack Diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ylnht5ckfpa13wgd6lj.png)

---

## Docker images

The stack, though it might seem complicated, is only composed of 3 images, 2 of which are custom:

### Django image

This is a custom image built from python.

This image will be used for django, celery workers and celery beat containers.

Here's the `Dockerfile` for it:

```Dockerfile
# Python as the base image
# I use bullseye because I'm more comfortable with it
# but you can use Alpine for a more lightweight container
FROM python:3.11-bullseye

# Exposes port 8000
# Make sure to change this to your used port
EXPOSE 8000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1

# Working directory
WORKDIR /app/backend

# Install pipenv
# This is not necessary if you use pip in your code
RUN pip install -U pipenv

# Install pipenv requirements
# Turns the Pipfile to a requirements.txt
# so it can be installed globally with pip
COPY Pipfile Pipfile.lock /app/backend/
RUN pipenv requirements > requirements.txt
RUN pip install -r requirements.txt
RUN rm -rf ./Pipfile ./Pipfile.lock

# Copy all the code over
COPY . .

# Create the media directory
RUN mkdir -p /app/backend/media

# Create a volume for the media directory
VOLUME /app/backend/media

# Create a volume for the static directory
VOLUME /app/backend/django_static

# Make the entrypoint scripts executable
# There's one entrypoint for each service that uses this image
RUN chmod +x /app/backend/entrypoint-django.sh
RUN chmod +x /app/backend/entrypoint-worker.sh
RUN chmod +x /app/backend/entrypoint-beat.sh

# Set the default entrypoint in case this Dockerfile is run
# by itself
ENTRYPOINT ["/app/backend/entrypoint-django.sh"]
```

These are the entry point files for each service:
#### django entry point
```sh
#!/bin/bash

# Migrate any new migrations to the database on deployment
echo "Migrating..."

python manage.py migrate --no-input

# Collect static files
echo "Collecting static files..."

python manage.py collectstatic --no-input

# Ensure the data directory exists
# I use the data directory to store files such as logs
mkdir -p data

# Start gunicorn
echo "Starting server..."

gunicorn core.wsgi:application --forwarded-allow-ips="*" --bind 0.0.0.0:8000
```

#### Worker entry point
```sh
#!/bin/sh

# Wait until the backend directory is created
until cd /app/backend
do
    echo "Waiting for server volume..."
done

# run a worker
# I like having only one task per worker but you can change it
# by increasing the concurrency
echo "Starting celery worker..."
celery -A core worker -l info --concurrency 1 -E
```

#### Beat entry point
```sh
#!/bin/sh

# Wait until the server volume is available
until cd /app/backend
do
    echo "Waiting for server volume..."
done

# run celery beat
echo "Starting celery beat..."
celery -A core beat -l info
```

### Nginx image

This container serves the application.

I create a custom nginx image that includes my certificates and configuration, so I don't have to copy them over to the server.

> Note: I don't use certbot, as I find it more straightfoward to generate the certificates from cloudflare and just store them in the custom image

This means that the image should be secure in a private registry with authentication, otherwise you risk security of your web app.

Here's the `Dockerfile` for it:
```Dockerfile
FROM nginx:stable-bullseye

# Export ports 80 and 443
EXPOSE 80
EXPOSE 443

# Copy the nginx configuration files to the image
COPY ./conf/default.conf /etc/nginx/conf.d/default.conf
COPY ./conf/prod.conf /etc/nginx/conf.d/prod.conf
COPY ./conf/staging.conf /etc/nginx/conf.d/staging.conf

# Copy the CloudFlare Origin CA certificate to the image
COPY ./certs/fullchain.pem /etc/nginx/certs/fullchain.pem
COPY ./certs/privkey.pem /etc/nginx/certs/privkey.pem
```


### Redis image

I just use the default Redis image for this.

Just want to note that, because this is a single instance deployment, I like deploying Redis directly here as I find it's enough.

It is recommended, though, to spin up a Redis database somewhere more centralized.

---

## Docker Compose

### Environment Variables

Before I get into the gist of the Docker Compose file here are some environment variables I put in my `.env` file for deployment:

- `DOCKER_REGISTRY`: My private, authentication enabled, docker registry where I upload the build images
- `DJANGO_DOCKER_IMAGE`: The name I give the django image
- `NGINX_DOCKER_IMAGE`: The name I give the NGINX image
- `DOCKER_TAG`: Usually the version I want to deploy, i.e.: `1.5` or `latest`

### The file

```yml
version: "3"

services:
  redis:
    container_name: redis
    restart: unless-stopped
    image: redis:7.2.0-alpine
    expose:
      - 6379

  backend:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
    env_file:
      - ./.env
    entrypoint: /app/backend/entrypoint-django.sh
    ports:
      - 8000:8000
    volumes:
      - ./data:/app/backend/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/healthcheck/"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s 

  worker:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
    env_file:
      - ./.env
    entrypoint: /app/backend/entrypoint-worker.sh
    volumes:
      - ./data:/app/backend/data
    depends_on:
      backend
        condition: service_healthy
      redis
        condition: service_started

  beat:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
    env_file:
      - ./.env
    entrypoint: /app/backend/entrypoint-beat.sh
    volumes:
      - ./data:/app/backend/data
    depends_on:
      backend
        condition: service_healthy
      redis
        condition: service_started
  
  nginx:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${NGINX_DOCKER_IMAGE}:${DOCKER_TAG}
    ports:
      - 80:80
      - 443:443
    depends_on:
      backend
        condition: service_healthy
```

As you can see, the compose file has the 5 services, Redis, django, celery worker, celery beat and NGINX.

---

## Deploying

### Building

First I build the images and push them to the registry. Before, I did this manually, now I use a GitHub action. You [can learn more about this automation here.](https://dev.to/onticdani/automatically-build-docker-images-with-github-actions-3n8e)

### Deploying with Docker Compose

Then I head to the server where I want to deploy this. Make sure that the `.env` file is updated and then just:

1. `docker compose down`: Spin the old instance down
2. `docker system prune -a -f`: This makes sure I remove the `latest` image to force the download of the new one from the registry.
3. `docker compose up --scale worker=5 -d`: Spin the new instance up
That's it!