Deploying Django with Celery and Redis on Ubuntu
Tags: devops, django, python, webdev
Published: 2019-11-05 15:17:08

---
title: Deploying Django with Celery and Redis on Ubuntu
published: true
description: Deploying production-ready applications is always a tricky job. In this post, I have tried to explain how Celery enabled Django applications can be deployed to Ubuntu with Redis, Nginx, and Gunicorn.
tags: devops, django, python, webdev
---
Most of the time we sort of think about project deployment difficulties before even implementing the project itself. My aim in writing this post is to show how easy it is to deploy a Django project on a Linux machine.

Django's primary deployment platform is [WSGI](https://wsgi.readthedocs.io/en/latest/), the Python standard for web servers and applications. The art of running Django includes tools such as Nginx, Gunicorn, virtualenv, Supervisor. Without much ado, let's get rolling!

## Environment
Before we even begin, let us understand what environment we will be using for the deployment.
- Operating System - Ubuntu 16.04.6 LTS  (AWS AMI)
- Python 3.7.3 (Check this [link](https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/) to install the latest version)

## Prerequisites
We will assume that you have `root` access to the Ubuntu system to install the required packages. We will be using `ubuntu` user (default that comes with AWS Ubuntu AMI) to perform all of the configurations below.

Secondly, we will also assume that you have a Django project available to clone via Git. Our Django project will require Celery and Redis for background task processing.

**:blue_book: NOTE**
- Your project might not require Celery and Redis and hence you can skip corresponding sections to focus only on Django deployment
- You might require to install `Database` or `RabbitMQ` as per your project which is pretty easy and does not require any server configuration to run apart from installation scripts
- All the commands will be executed under `ubuntu` user which has root access, depicted as `$` in the commands listed throughout this post. If it has a prefix `(myproj_env)$`, then the command has to be executed in the virtual environment

To make things simpler, we will divide the process into the following steps:
1. Installing required packages
2. Setting up Django project
3. Installing Redis for Celery
4. Setting up Gunicorn
5. Setting up Supervisor
6. Setting up Nginx

### 1. Installing required packages
Let's get started by making sure our system is up to date
```sh
$ sudo apt-get update
```
Next is to install some global packages for our application to run (based on your Django project dependencies, you may require additional packages)
```sh
$ sudo apt-get install libmysqlclient-dev python3-dev
```

### 2. Setting up Django project
- The first step in setting up any python project is to install a package manager (we will be using `pip`) and set up a virtual environment. We will be using `virutalenv` package for the same
```sh
# installing pip for Python 3.7
$ curl https://bootstrap.pypa.io/get-pip.py | sudo -H python3.7
# installing virtualenv
$ sudo pip3 install virtualenv
```
- Before we create a virtual environment for our project, let's understand our setup folder structure. By default, Ubuntu will land us to this base folder `/home/ubuntu/`. To correctly identify our projects on the machine, we will create a `webapps` folder and create all logs, project folders under it. So our project structure would look like this
```sh
# base folder
/home/ubuntu/webapps/
# under webapps
/webapps
│---projects                 # Base folder for our project
    │---myproj_env           # Django project virtual environment
    │---myproj               # Django project directory
    │   │---manage.py
    │   │---Myproj
    │   |---requirements.txt
    logs                     # All our logs would be inside this folder
    │---gunicorn
    │---redis
    │---nginx
    |---celery

```
   From above, please note that our project name is `myproj`(git repo) and virtual environment name is `myproj_env`

- Let's create virtual environment in `projects` folder as stated above
```sh
$ cd /home/ubuntu/webapps/projects # create folders if you haven't created
# virutal environment for python 3.7
$ virtualenv -p python3.7 myproj_env
```
- Clone our project `myproj` in `projects` folder
```sh
$ cd /home/ubuntu/webapps/projects
# clone the repository
$ git clone <myproj-git-link>
```
- Activate virtual envrionment and install all the dependencies of our project
```sh
$ cd /home/ubuntu/webapps/projects
# activate virtual envrionment
$ source myproj_env/bin/activate
# installing project dependencies
(myproj_env)$ cd /home/ubuntu/webapps/projects/myproj
(myproj_env)$ pip install -r requirements.txt
# optionally, check if django is running locally
(myproj_env)$ python manage.py runserver
```


### 3. Installing Redis for Celery
[Celery](https://docs.celeryproject.org/en/latest/getting-started/introduction.html) is an asynchronous task queue/job queue based on distributed message passing. Task queues are used as a mechanism to distribute work across threads or machines. Some of the use-cases of Celery are sending an email, scheduling tasks, asynchronous execution, etc. Celery requires a message transport to send and receive messages such as RabbitMQ, Redis, etc. For simplicity, we will only use Redis that serves our purpose here.

Installing Redis
```sh
$ sudo apt-get install redis-server
# check if Redis is working
$ redis-cli ping 
PONG
# Autostart Redis on server restart
$ sudo systemctl enable redis-server.service
```

Celery can be installed in virtual environment of our project which is generally included in project dependency file `requirements.txt`


### 4. Setting up Gunicorn
In production, we won't be using Django's single-threaded development server, but a dedicated application server called `gunicorn`.

Install gunicorn in your application's virtual environment
```sh
$ cd /home/ubuntu/webapps/projects
# activate virtual envrionment
$ source myproj_env/bin/activate
# installing gunicorn
$ cd /home/ubuntu/webapps/projects/myproj
(myproj_env):$ pip install gunicorn
# optionally, check if it is running
(myproj_env):$ gunicorn Myproj.wsgi:application --bind 8001
[2019-11-04 04:52:01 +0000] [2010] [INFO] Starting gunicorn 19.9.0
[2019-11-04 04:52:01 +0000] [2010] [INFO] Listening at: http://0.0.30.60:8000 (2010)
[2019-11-04 04:52:01 +0000] [2010] [INFO] Using worker: sync
[2019-11-04 04:52:01 +0000] [2013] [INFO] Booting worker with pid: 2013

# make sure you kill Gunicorn process after you have tested to move further
```
Gunicorn is ready to serve requests for our app. We should now be able to access the Gunicorn server from `http://<server-IP-address>:8001`. Let's create a `Bash` script to automatically start the gunicorn server with some custom configurations. We will save this bash script inside our `projects` folder as `gunicorn_start.bash` so our folder structure will look like (listing only required folders below)
```sh
/webapps
│---projects                     # Base folder for our project
    │---myproj_env               # Django project virtual environment
        │---run                  # create this folder to hold gunicorn sock file
            │---gunicorn.sock    # just create this empty file 
    │---myproj                   # Django project directory
    │---gunicorn_start.bash      # Gunicorn Bash script
```
Gunicorn Bash Script
{% gist https://gist.github.com/idris-rampurawala/6881f30ffa1ae08ebfe8d26f6cc72a9e file=django-gunicorn-bash.sh %}

Set the executable bit on the gunicorn_start script
```sh
# make sure you are in the projects folder
$ chmod u+x gunicorn_start.bash
```

Execute the gunicorn bash script and check if it is running
```sh
# make sure you are in the projects folder
$ ./gunicorn_start.bash
# make sure you kill Gunicorn process after you have tested to move further
```
Gunicorn Bash script insights
- You will need to set the paths and filenames to match your project setup
- Option `--workers (NUM_WORKERS)` specifies the worker processes Gunicorn should spawn to handle the traffic. General idea is to set it according to formula - `2 * CPUs + 1`
- If you get error `/run/gunicorn.sock: No such file or directory`, just create an empty `gunicorn.sock` in `run` folder (see our setup folder above)


### 5. Setting up Supervisor
One of the most important steps is to configure `Supervisor`. Supervisor is a system that allows its users to monitor and control a number of processes on UNIX-like operating systems. We need to make sure that our server (Gunicorn) starts automatically with the system and that it can automatically restart if for some reason it exits unexpectedly. These tasks can easily be handled by Supervisor. 

The server piece of Supervisor is named `supervisord` and the command-line piece as `supervisorctl`.

Let's start by installing it
```sh
$ sudo apt-get install supervisor
```

Supervisor works on configuration files (.conf) to manage processes. You will find `supervisord.conf` in `/etc/supervisor` in which global settings related to the supervisord process are set. For us to configure our project-related processes, we will create 3 different configuration files at `/etc/supervisor/conf.d` to manage processes namely:
1. `gunicorn.conf` - To manage Gunicorn server that we set up in the previous step to automatically run with help of its bash script
{% gist https://gist.github.com/idris-rampurawala/d34954a476141f7b7bdf00eeca6ee1dd file=gunicorn-supervisor.conf %}
2. `celery_beat.conf` - A scheduler that kicks off tasks at regular intervals
{% gist https://gist.github.com/idris-rampurawala/809a673261ef2713285797670c124c20 file=celery-beat-supervisor.conf %}
3. `celery_worker.conf` - A Celery worker that executes the task for celery
{% gist https://gist.github.com/idris-rampurawala/5c6027a9a875471997e8199778f55361 file=celery-worker-supervisor.conf %}

Once we save these configuration files, we need to notify supervisor of the change via command-line utility `supervisorctl`
```sh
# for every change in conf file, we need to execute the following commands
$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl start all # starting all our processes
```
Some useful `supervisorctl` commands
```sh
# to check the status of our processes
$ sudo supervisorctl status <program_name|all>
# example
$ sudo supervisorctl status all
celery                           RUNNING   pid 1413, uptime 7:59:29
celerybeat                       RUNNING   pid 1411, uptime 7:59:29
my_proj                          RUNNING   pid 1412, uptime 7:59:29

# to start, stop, restart all or some of the processes
$ sudo supervisorctl start <program_name|all>
$ sudo supervisorctl restart <program_name|all>
$ sudo supervisorctl stop <program_name|all>
```


### 6. Setting up Nginx
Already feeling the complexity of setting up Django? Hold on! This is the last step, I promise :grimacing: Setting up Nginx is easy.
```sh
# installing Nginx
$ sudo apt-get install nginx
# optionally execute following command to autostart nginx on system reboot
$ sudo update-rc.d nginx defaults
```
Lastly, create Nginx configuration file `myproj.conf` at `/etc/nginx/conf.d` with following to start serving requests from Nginx to Gunicorn
{% gist https://gist.github.com/idris-rampurawala/7fb3b78c2c43fe8e1919f1a9efd78ca4 file=nginx-gunicorn.conf %}

Restart Nginx and we are done for the day!
```sh
$ sudo service nginx restart 
```

We should be able to access our Django routes from `http://<server-IP-address>`

## Final Words
Congratulations! :clap: We have successfully set up the Django project on Ubuntu along with Celery and Redis. Do not forget to check the `Useful Links` section to find links to read more about the tools/libraries used in this post.


## Useful Links :star:
- [virtualenv](https://virtualenv.pypa.io/en/latest)
- [Gunicorn](https://gunicorn.org/)
- [Supervisor](http://supervisord.org/)
- [Celery](https://docs.celeryproject.org/en/latest/getting-started/introduction.html)
- [Redis](https://redis.io/documentation)

Check out my earlier post `Configure SSH for git` to get rid of entering passwords on every git pull :sunglasses:
{% post idrisrampurawala/configuring-ssh-for-git-2of1 %}

See ya! until my next post :yum: