Asynchronous tasks in Django with Django Q
Tags: django, python, webdev, redis
Published: 2020-01-16 23:35:28

---
title: "Asynchronous tasks in Django with Django Q"
published: true
description: "Learn how to use Django Q for offloading asynchronous tasks in your Django applications."
tags: django, python, webdev, redis
canonical_url: https://www.valentinog.com/blog/django-q/
---

_Learn how to use Django Q, the task queue, with the Redis broker for offloading long running tasks in your Django applications._

## Requirements

To follow along you'll need:

- an Heroku account if you want to use their Redis add-on
- the Heroku CLI installed on your system
- a newer version of Python, ideally 3.6 or 3.7
- Git

**Deployment on Heroku is optional, and you can use your own Redis instance if you've already got one locally.**

## Setting up the project

And now let's get to work! To start off we're going to create a new Python virtual environment alongside with a Django installation:

```bash
mkdir django-q-django && cd $_
python3 -m venv venv
source venv/bin/activate
pip install django
```

Next up we're going to create a **new Django project from a template**:

```bash
django-admin startproject \
    --template https://github.com/valentinogagliardi/ponee/archive/master.zip \
    --name=Procfile \
    --extension=py,example django_q_django .
```

If you're wondering what I'm doing here, this is a template of mine. I've got a link in the resources with a tutorial for creating your own Django project template.

Now let's install the dependencies with pip:

```bash
pip install -r ./requirements/dev.txt
```

We need also to provide some environment variables for our project:

```bash
mv .env.example .env
```

and finally we're going to run Django migrations:

```bash
python manage.py makemigrations
python manage.py migrate
```

at this point you should be able to run the development server:

```bash
python manage.py runserver
```

Now before moving to **Django Q let's see what problem is it meant to solve**.

## Asynchronous tasks in Django with Django Q: the problem with synchronous code

The main issue for Python and Django is that they're **synchronous**. It's not a bad thing per se, and there are a lot of ways to circumvent it.

**Python, on which Django builds on, is single threaded by nature**. **Single threaded means that the language interpreter can only run your code in sequence**.

The **practical implication is that any view in a Django application can get stuck if one or more operations take too much to complete**.

To demonstrate the concept let's create a new Django application inside our project:

```bash
django-admin startapp demo_app
```

In this app we're going to define a view which returns a simple JSON response:

```python
# demo_app/views.py

from django.http import JsonResponse

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    return JsonResponse(json_payload)
```

And let's also create the corresponding url:
 
```python
# demo_app/urls.py

from django.urls import path
from .views import index

urlpatterns = [
    path("demo-app/", index)
]
```

Don't forget to wire up the url for the new app:

```python
# django_q_django/urls.py

from django.contrib import admin
from django.urls import path, include
from .settings.base import ADMIN_URL

urlpatterns = [
    path(f'{ADMIN_URL}/', admin.site.urls),
    # the new url
    path("", include("demo_app.urls"))
]
```

And finally activate the app:

```python
# django_q_django/settings/base.py

INSTALLED_APPS = [
    # omitted for brevity
    'demo_app.apps.DemoAppConfig'
]
```

Now to **simulate a blocking event in the view** we're going to use **sleep from the time module, part of the Python standard library**:

```python
from django.http import JsonResponse
from time import sleep

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    sleep(10)
    return JsonResponse(json_payload)

```

Run the development server, head over http://127.0.0.1:8000/demo-app/ and you can see the **view hanging for 10 seconds before returning to the user**.

Now, this is a delay created on purpose, but **in a real application the block could happen for a number of reasons**:

- I/O operations taking too long
- network delay
- interactions with the file system

Even if it's a contrived example you can see why **it's crucial to offload long running tasks in a web application**.

**Django Q** was born with this goal in mind. In the next sections we'll finally put our hands on it.

## Wait, how about asynchronous Django?

We're left with a Django project, a Django application, and a **view that remains stuck for 10 seconds**!

**It's not that Django and Python don't scale**. There are **a number of ways to get around single threading**.

For Python there is asyncio. Django instead moved to async only recently, the implementation is in its infancy and still there's **no support for async views**. 

Things are going to change in the future, I suggest staying tuned on [Andrew Godwin](https://twitter.com/andrewgodwin) because it's the lead person for the async story in Django.

But **the need for third party queues won't go away anytime soon even when Django will go 100% async**. Still a nice skill to have.

## Preparing the Heroku app and the Redis instance

In this section we'll prepare the Heroku project. I'm using Heroku here because you may want to deploy to production later, and also because they offer the **Redis add-on for free**.

If you're new to **Redis, it's an in-memory database, can be used as a cache and as a message broker**.

**A message broker** is more or less like a post office box: it takes messages, holds them in a queue, and folks from around the city can retrieve these messages later.

If you're interested in how Django Q uses brokers [check out this page](https://django-q.readthedocs.io/en/latest/architecture.html).

Still in the **project folder initialize a Git repo**:

```bash
git init
```

Then **create a new Heroku app**. I'm going to add two add-ons: 

- heroku-postgresql which is more robust than the default sqlite for production
- **heroku-redis** which will give us the Redis instance

If you haven't got the Heroku CLI and an Heroku account go create one, install the CLI and come back later.

Otherwise follow along with me and create the app:

```bash
heroku create --addons=heroku-postgresql,heroku-redis
```

Once done give Heroku a couple of minutes and then run:

```bash
heroku config:get REDIS_URL
```

This command will reveal **REDIS_URL**, an environment variable with the credentials for the Redis instance.

Take note of it and head over the next section!

## Asynchronous tasks in Django with Django Q: installing and running Django Q

Let's install Django Q and the Redis client library (the client is needed by the Redis broker for Django Q):

```bash
pip install django-q redis
```

Once done activate Django Q in the list of installed apps:

```python
INSTALLED_APPS = [
    # omit
    # add Django Q
    'django_q'
]
```

Now reveal the Redis Heroku credentials:

```bash
heroku config:get REDIS_URL
```

You should see a string like this:

```text
redis://h:p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf@ec2-52-18-11-1.eu-west-1.compute.amazonaws.com:9059
```

Before the @ you'll find the password:

```text
p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf
```

After the @ there's the host:

```text
ec2-52-18-11-1.eu-west-1.compute.amazonaws.com
```

And **9059** is the port. **Note that the credentials will be different for you, don't use mine!**

(Needless to say, by the time you read this article these credentials will be gone.)

Now configure Django Q in **django_q_django/settings/base.py**. Fill host, port, and password with your credentials:

```python
Q_CLUSTER = {
    'name': 'django_q_django',
    'workers': 8,
    'recycle': 500,
    'timeout': 60,
    'compress': True,
    'save_limit': 250,
    'queue_limit': 500,
    'cpu_affinity': 1,
    'label': 'Django Q',
    'redis': {
        'host': 'ec2-52-18-11-1.eu-west-1.compute.amazonaws.com',
        'port': 9059,
        'password': 'p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf',
        'db': 0, }
}
```

You might wonder **why I'm not using REDIS_URL as it is**. The reason is that Django Q wants credentials in a dictionary.

~~I didn't have time to check if is the Python Redis client imposing this limitation, maybe I'll write a patch for both in the future.~~ It was a limitation of Django Q, ~~hope I'll have time to open a PR~~ [I opened a pull request](https://github.com/Koed00/django-q/pull/403) which got merged, and now you can use a Redis url:

```python
Q_CLUSTER = {
    'name': 'django_q_django',
    # omitted for brevity  
    'label': 'Django Q',
    'redis': 'redis://h:asdfqwer1234asdf@ec2-111-1-1-1.compute-1.amazonaws.com:111'
}
```

(When **running the project in production you may want to switch to using environment variables**. See the base configuration for learning how to use env).

Once you're done run the migrations (Django Q needs to create its tables in the database):

```bash
python manage.py migrate
```

At this point you're ready to **run the Django Q cluster** with:

```bash
python manage.py qcluster
```

If everything goes well you should see this:

![Django Q cluster](https://thepracticaldev.s3.amazonaws.com/i/6jowrdcmvnrhluu8d6ch.png)

Well done! In the next section we'll create our first **asynchronous task**.

What is the Django Q cluster? [Check this out](https://django-q.readthedocs.io/en/latest/cluster.html).

## Asynchronous tasks in Django with Django Q: async_task

Worth doing a quick recap of what we covered so far:

- we created a Django project
- we created a Django application
- we installed Django Q and the Redis client
- we created an Heroku project and a Redis instance
- finally we configured Django Q

To test that Django Q could connect to Redis I launched:

```bash
python manage.py qcluster
```

With the project in place let's finally see an **example of Django Q in action**. Remember your view?

```python
# demo_app/views.py

from django.http import JsonResponse
from time import sleep

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    sleep(10)
    return JsonResponse(json_payload)

```

Remove the time import and create a new file in **demo_app/services.py** (the name of this file is totally up to you).

In this new module we're going to define a function, sleep_and_print:

```python
# demo_app/services.py

from time import sleep

def sleep_and_print(secs):
    sleep(secs)
    print("Task ran!")
```

In the view instead we'll borrow **async_task** from Django Q:

```python
from django.http import JsonResponse
from django_q.tasks import async_task


def index(request):
    json_payload = {
        "message": "hello world!"
    }
    """
    TODO
    """
    return JsonResponse(json_payload)
```

**async_task** is the principal function you'll use with Django Q. It takes at least one argument, the function's module that you want to enqueue:

```python
# example

async_task("demo_app.services.sleep_and_print")
```

The second group of arguments instead is any argument that the function is supposed to take. sleep_and_print in our example takes one argument, the seconds to wait before printing. That means for async_task:

```python
# example

async_task("demo_app.services.sleep_and_print", 10)
```

That's enough to enqueue a task. Let's now mix our **view with async_task**.

## Asynchronous tasks in Django with Django Q: enqueue your first task

Back to our view, with async_task imported, call it right after the return statement:

```python
from django.http import JsonResponse
from django_q.tasks import async_task


def index(request):
    json_payload = {"message": "hello world!"}
    # enqueue the task
    async_task("demo_app.services.sleep_and_print", 10)
    #
    return JsonResponse(json_payload)

```

Now run the cluster:

```bash
python manage.py qcluster
```

Run the Django server:

```bash
python manage.py runserver
```

And finally **make a call to your view**, either from http://127.0.0.1:8000/demo-app/ or from the terminal:

```bash
curl http://127.0.0.1:8000/demo-app/
```

Now **you should notice a couple of things**. The Django dev server should log:

```text
13:55:42 [Q] INFO Enqueued 1
```

The Django Q cluster should log something along these lines:

```text
13:55:42 [Q] INFO Process-1:1 processing [juliet-mountain-august-alaska]
```

And after that you should see:

```text
Task ran!
```

Here's my terminal:

![Enqueue your first task](https://thepracticaldev.s3.amazonaws.com/i/2dmorc28ahr0ixbq69ox.png)

What happened here is that:

1. the Django **view responded immediately to the request**
2. Django Q saved the task (just a reference) in Redis
3. Django Q ran the task

With this "architecture" the **view does not remain stuck anymore**. Brilliant.

Think about the use cases for this pattern. You can:

- safely interact with the I/O
- crunch data in the background
- safely move out API calls from your views

and much more.

## Asynchronous tasks in Django with Django Q: what's next?

In addition to async_task Django Q has the ability to **schedule** a task. A practical use case is **do X every X days**, much like a cron job. Check the [documentation to learn more](https://django-q.readthedocs.io/en/latest/schedules.html).

Django Q supports other brokers in addition to Redis. Again, the docs are your friend.

If you don't need other brokers than Redis, [django-rq](https://github.com/rq/django-rq) might be a lightweight alternative to Django Q.

## Asynchronous tasks in Django with Django Q: why not Celery?

Fun fact: Celery was created by a friend of mine. We were in high school together. Despite that I don't have much experience with Celery itself, but I always heard a lot of people complaining about it.

Check [this out for a better perspective](https://github.com/Koed00/django-q/issues/8).

Thanks for reading and stay tuned!

## Resources

- [Django Q](https://django-q.readthedocs.io/en/latest/)
- [Redis](https://redis.io/)
- [What is message broker?](https://en.wikipedia.org/wiki/Message_broker)
- [Background Jobs and Queueing](https://devcenter.heroku.com/articles/background-jobs-queueing)
- [How to create a Django project from a template](https://www.valentinog.com/blog/django-project/)

Originally published on [my blog](https://www.valentinog.com/blog/django-q/)