Multithreading in Django using threading and Queue
Tags: python, django
Published: 2022-08-08 07:54:00

In Django, if the process executed on a request is time-consuming, you may want to keep the process running behind and return a response anyway.

For example, sending a confirmation e-mail when a user registration is completed. If an application that takes 1 second to send an email receives 1,000 accesses per second, following things will be required.

- The application does not lock during network I/O for sending mail.
- The mail sending jobs are run behind and executed in the order in which they are accessed.

I had the opportunity to create an email sending function in Django, so here are my notes.

## Confirmation of basic operation
First, we modified the sample code in the [queue reference](https://docs.python.org/ja/3/library/queue.html) a little to check the process using queue and threading.

```python
import threading
import queue
import time

q = queue.Queue()

def worker():
    while True:
        item = q.get()
        print(f'Working on {item}')
        time.sleep(1)
        print(f'Finished {item}')
        q.task_done()

# turn-on the worker thread
threading.Thread(target=worker, daemon=True).start()

# send thirty task requests to the worker
for item in range(5):
    print("Put item", item)
    q.put(item)
print('All task requests sent\n', end='')

# block until all tasks are done
q.join()
print('All work completed')
```
First declare `q` as a `Queue` instance in a global variable. This queue will be filled with jobs to be executed. 

Next, we define the `worker` method as the job executor, which will retrieve the job, print it, and wait one second. Because of `while true`, it's always waiting, 
When `q.get()` returns a return value, the underlying process is executed.

Next, start a new thread with `threading.Thread()`, passing target the method you want to execute. Then add the item as a job to the queue and the worker starts running. `q.join()` stops further processing until `q.task_done()` is called for all `q`.

Outpus
```
Put item 0
Put item 1
Put item 2
Put item 3
Put item 4
All task requests sent
Working on 0
Finished 0
Working on 1
Finished 1
Working on 2
Finished 2
Working on 3
Finished 3
Working on 4
Finished 4
All work completed
```

The output shows that all job additions are completed first, and the jobs are executed in the order they were added, indicating that job management and execution are performed asynchronously. It seems that sending emails can also be implemented in this way. Try it.

## Asynchronous processing of tasks in Django
First, create a Django API for testing.
```
$ django-admin startproject mysite
$ cd mysite/
$ python manage.py startapp api
```
Then add the following code.

`mysite/settings.py`
```python
...
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'api',
    'rest_framework'
]
...
```

`mysite/urls.py`
```python
from django.contrib import admin
from django.urls import path
from api import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('index/', views.index)
]
```
`api/views.py`
```python
from django.http import HttpResponse
# Create your views here.
def index(request):
    return HttpResponse("Hello, world.")
```
Then, start server
```
$ python manage.py runserver
```
Access `http://127.0.0.1:8000/index/` and if `Hello, world.` is displayed, you are ready.

Now let's move on to the creation of the asynchronous mail function.

Create a new file `api/mail.py` and write the contents as follows.
```python
import threading
import queue
import time

q = queue.Queue()

def worker():
    while True:
        item = q.get()
        print(f'Working on {item}')
        time.sleep(3)  # Do any processing here.
        print(f'Finished {item}')
        q.task_done()

# Five threads are set up to parallelize the process.
for _ in range(5): 
    threading.Thread(target=worker, daemon=True).start()

def add(item):
    q.put(item)
```

Next, modify `views.py` as follows.
```python
from django.http import HttpResponse
from api.mail import add as mail_add

# Create your views here.
def index(request):
    item = request.GET.get("param")
    mail_add(item)
    return HttpResponse("Hello, world.")
```
Restart the server in this state and try sending a series of requests as follows.
```
curl http://127.0.0.1:8000/index/?param=1
curl http://127.0.0.1:8000/index/?param=2
curl http://127.0.0.1:8000/index/?param=3
curl http://127.0.0.1:8000/index/?param=4
curl http://127.0.0.1:8000/index/?param=5
```
Outputs
```
Working on 1
[15/May/2021 12:12:28] "GET /index/?param=1 HTTP/1.1" 200 13
Working on 2
[15/May/2021 12:12:28] "GET /index/?param=2 HTTP/1.1" 200 13
Working on 3
[15/May/2021 12:12:28] "GET /index/?param=3 HTTP/1.1" 200 13
Working on 4
[15/May/2021 12:12:28] "GET /index/?param=4 HTTP/1.1" 200 13
Working on 5
[15/May/2021 12:12:28] "GET /index/?param=5 HTTP/1.1" 200 13
Finished 1
Finished 2
Finished 3
Finished 4
Finished 5
```
Finished will be output almost simultaneously. If you rewrite the `time.sleep(3)` part as mail processing, an asynchronous mail sending function is completed.