Celery with Django in production
Tags: django, celery, ubuntu, python
Published: 2021-02-11 07:21:27


[0. Background](#0)
[1. Create the dedicated user and group](#1)
[2. Create the celery configuration file](#2)
[3. Create the systemd file](#3)
[4. Restart the server](#4) 
[5. References](#5)

### <a name="0"></a> 0. Background
Three months ago, I deployed my react, Django project on a ubuntu server. I used celery for a task queue. Today I tried to look celery environment and I got confused about the files and my celery configuration. Now In this post, I will try to explain all the things that you should know about the celery configuration on the server. 

> If you never heard about celery, check out the [celery official documentation](https://docs.celeryproject.org/en/stable/getting-started/introduction.html), [First steps with celery](https://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html) and [First steps with Django](https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html) before reading this post.

In the celery official documentation, you will find the three possible ways to [daemoniz your celery with python](https://docs.celeryproject.org/en/stable/userguide/daemonizing.html), i.e. [init-script](https://docs.celeryproject.org/en/stable/userguide/daemonizing.html#init-script-celeryd), [Init-script: celerybeat](https://docs.celeryproject.org/en/stable/userguide/daemonizing.html#init-script-celerybeat) and [using systemd](https://docs.celeryproject.org/en/stable/userguide/daemonizing.html#init-script-celerybeat). In this post, I will only explain about the **systemd demonization**.

### <a name="1"></a> 1. Create the dedicated user and group
First of all, you need to create the dedicated user and group as `celery`,

```bash
sudo groupadd celery
sudo useradd -g celery celery
```

### <a name="2"></a> 2. Create the celery configuration file
Now let's create the celery configuration file inside the `/etc/default/celeryd` directory, 

**Note:** Please replace the `user` with your ubuntu user, `CELERY_APP_NAME`, `CELERYD_CHDIR`, and `CELERY_BIN`. 

```bash
#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/user/.virtualenvs/venv/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="celery_app_name"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# Where to chdir at start.
CELERYD_CHDIR="/home/user/django-project/"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"

# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"

# %n will be replaced with the first part of the node name.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"
CELERYD_LOG_LEVEL="INFO"
# If enabled PID and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
```
Now lets change the owner of the celery log and PID file, 

```bash
chown -R celery:celery /var/log/celery/
chown -R celery:celery /var/run/celery/
```
### <a name="3"></a> 3. Create the systemd file
Now lets create the systemd file inside `/etc/systemd/system/celery.service` directory and write following content,

**Note:** Please replace the `user` with your ubuntu user and make sure to check the path to bin celery file (this file will available probably inside your virtual environment)

```bash
[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=celery
Group=celery

EnvironmentFile=/etc/default/celeryd
WorkingDirectory=/home/user/django-project
ExecStart=/home/user/.virtualenvs/venv/bin/celery multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}
ExecStop=/home/user/.virtualenvs/venv/bin/celery ${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}
ExecReload=/home/user/.virtualenvs/venv/bin/celery ${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}

[Install]
WantedBy=multi-user.target
```

### <a name="4"></a> 4. Restart the server

Now everything is setup. Now you just need to restart the service
```bash
sudo systemctl daemon-reload
sudo systemctl enable celery
sudo systemctl restart celery
```

Now your celery is ready to use in production. You can check the status of the celery manually by typing following code,

```bash
celery -A CELERY_APP_NAME worker -l INFO
```

### <a name="5"></a> 5. References
1. [Celery documentation](https://docs.celeryproject.org/en/stable/)
2. [Daemonization celery in production: stackoverflow](https://stackoverflow.com/questions/57266711/daemonization-celery-in-production)
3. [Celery not running (Permission Denied): stackoverflow](https://stackoverflow.com/questions/28825760/celery-not-running-permission-denied/28826090)