Django Background Tasks
Tags: django, python
Published: 2023-10-05 20:03:31

Background tasks are processes that run in the background so your server is ready to take further requests. 

Some common use cases for background tasks:

- Web Scraping
- Heavy data processing
- Image processing
- Multiple 3rd party API calls
- Building Slack-bots (Slack requires a 3 second response)

Basically, if the task takes a long time to complete (e.g. >30 seconds) you probably want to offload it to a background task.

If you have your Django project set up it's dead easy to get a background tasks up and running.
 

## Step 1: Getting Started

We want to install [Django Q](https://django-q.readthedocs.io/en/latest/) via: 

```
pip install django-q
```

Django Q allows us to set up some queues to run in the background. Be sure to run the migrations as well to generate the job models (which you'll see in your admin panel)

```
python manage.py migrate
```

It requires a **broker** though...


## Step 2: Setting up Redis

We'll use [redis](https://redis.io/). This is basically the communication channel between your **django instance** (normal django) and your **django worker** (this is your background task/worker). 

You can check if your redis server is running by running:

```
redis-cli ping
```

If not you'll need to follow the [redis](https://redis.io/) instructions to set up on your system.


## Step 3: Setting up the Cluster

The standard django-q configuration will work for us locally but with a couple of changes so we can run this in production as well. Add the following to your base `settings.py` file:

```
REDIS_HOST = os.environ.get("REDIS_HOST", "localhost")
REDIS_PORT = os.environ.get("REDIS_PORT", 6379)
REDIS_DB = os.environ.get("REDIS_DB", 0)
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", None)

Q_CLUSTER = {
    "name": "hello-django",
    "workers": 8,
    "recycle": 500,
    "timeout": 60,
    "compress": True,
    "cpu_affinity": 1,
    "save_limit": 250,
    "queue_limit": 500,
    "label": "Django Q",
    "redis": {
        "host": REDIS_HOST,
        "port": REDIS_PORT,
        "db": REDIS_DB,
        "password": REDIS_PASSWORD,
    },
}

```

You can play around with the number of `workers` - the default is `8` and these are the number of instances that can run in parallel with each other. 

We can now actually start running the **cluster**. Have 2 terminal windows and in 1 have your standard `python manage.py runserver` running and in the other run:

```
python manage.py qcluster
```

You now have your cluster running. 


## Step 4: Creating our Task

Create a standard API to make a simple `POST` request and let's add the task:

```
from django_q.tasks import async_task
from rest_framework.views import APIView
import time

...

class SimpleTask(APIView):

    def task(self, s = 5):
        print("⏳ Running task")
        # We could call some long running code here
        time.sleep(s)
        print("⌛ Task finished")

    def post(self, request):
        # We can run the task in the background
        async_task(self.task, s=5)
        return Response({
            "message": "Task started"
        })
```

We use `async_task` to wrap the task. The code will essentially see this and send it to the redis server (which will send it to the cluster) and continue to the response. 

You can place anything in the `task` and the response will still be returned in the same amount of time. 


## Step 5: Deploying to Production

Having this run in a safe local environment is one thing. Deploying it to a production environment is another. 

We'll be using [Railway](https://railway.app/) (other than them I'd recommend [Fly.io](https://fly.io/) or [Render](https://render.com/) - the principles are the same).

We want to add a `railway.json` file in our Django App with the following:

```
{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "NIXPACKS"
  },
  "deploy": {
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}
```

In Railway you want to create **1 Redis database**, **2 Django Apps**. In the first Django App you want the **Deploy (Start Command)** to be:

```
python manage.py migrate && gunicorn NAMEOFYOURAPP.wsgi
```

And in the second:

```
python manage.py migrate && python manage.py qcluster
```

You'll want to update the environment variables for both applications to match the `REDIS` environment variables (see **Step 2**), for this I recommend using Railway's **shared variables**. The end result looks something like:




![Railway Setup](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p70i86dv6i3wyfammmsy.png)

Think of this mirroring your terminals (the standard Django instance and the new Django qcluster instance)


To verify this is all working, make a request to your `SimpleTask` (you can find the URL if you click on the `api` app) and take a look at the **Observability** tab and you should see your **background task running** 🎉