Django x Postgres x Selenium x Chromedriver with Docker
Published: 2022-09-08 16:23:55

disclaimer: I am new to Docker so please take this with a grain of salt. at the time of writing, this setup worked on my computer.

also please let me know in the comments if I got something wrong

## the files

These files assumes that your Dockerfile lives in the root of your Django project (i.e. in the same level as `manage.py`).

### Dockerfile

```Dockerfile
FROM python:3.9-slim-buster

# set environment variables
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
# don't write .pyc files
ENV PYTHONDONTWRITEBYTECODE 1 
# prevent Docker from buffering stdout
ENV PYTHONUNBUFFERED 1 

# set working directory
WORKDIR /code

RUN apt-get update && apt-get install -y \
  curl \
  gnupg2 \
  unzip \
  wget

# chrome
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' \
  && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
  && apt-get update \
  && apt-get install google-chrome-stable -y

# chromedriver
RUN CHROME_VERSION="$(google-chrome --version)" \
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
  && CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  && cd /tmp \
  && unzip chromedriver_linux64.zip \
  && rm -rf chromedriver_linux64.zip \
  && mv chromedriver /usr/local/bin/chromedriver \
  && chmod +x /usr/local/bin/chromedriver \
  && cd /code

COPY ./requirements.txt .
 
# install dependencies
RUN apt-get update &&  apt-get install -y \
	gcc \ 
  less \
  libmagickwand-dev \
  libpq-dev \
  vim \
	&& rm -rf /var/lib/apt/lists/* \
	&&  pip install -r requirements.txt

# copy project
COPY . .
```

### docker-compose.yml

```yaml
version: "3.9"
services:
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    environment:
      - CHROME_DRIVER_PATH=/usr/local/bin/chromedriver
    volumes:
      - .:/code  
    ports:
      - "8000:8000"
    depends_on:
      - db

  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"

  redis:
    image: redis:alpine

  celery:
    build:
      context: .
    command: celery -A rises_ops worker -l INFO
    environment:
      - CHROME_DRIVER_PATH=/usr/local/bin/chromedriver
    volumes:
      # same volume as web
      - .:/code
    depends_on:
      - db
      - redis
      - web
volumes:
  postgres_data:
```

## migration script

after running `docker compose up --build`, run the migration from your terminal

```bash
$ docker-compose exec web python manage.py migrate
```

(there is probably a smarter way to do this, let me know in the comments)

## references

blogs that actually helped

https://learndjango.com/tutorials/django-docker-and-postgresql-tutorial
https://learndjango.com/tutorials/django-docker-and-postgresql-tutorial

## debugging tips

```bash
# build image with no cache
$ docker build . --no-cache

# output build process to your terminal
$ docker build . --progress=plain

# re-build your services
$ docker compose up --build

# clean cache
$ docker system prune
$ docker image prune 
```

note that the `pg_data` volume is actually saved in **Host**, so the DB data will persist even after you delete your conatiners. to clear even the DB,

```bash
# identify your volume
# format is project_db-data e.g. myproject_db-data
$ docker volume ls

# remove
$ docker volume rm <VOLUME ID>
```

ref: https://torajirousan.hatenadiary.jp/entry/2020/07/08/094325

## `docker-compose.yml`

the `docker-compose.yml` is pretty straightforward. we have 4 services: `web` `db` `redis` `celery`. 

- web: the Django werver
- db: PostgreSQL DB
- redis: Redis used in the Celery workers
- celery: worker process for Django

the commands written in the `command:` section of the file should be similar to what you execute in your local terminal.

one thing to note is that, `web` and `celery` share the same volume, since both of them need access to the Django code

## `Dockerfile`

the bulk of this file is to install chrome and chromedriver. both of them are not in the default `apt-get` repository, so you have to

* add the repository for `google-chrome-stable`, and
* download the zip file for `chromedriver`


in my past experience, Ubuntu has `chromium` and `chromium-driver` in the default repository, so this might be a better option if image size is not a problem for you.

### chrome

to explain the process for installing Google Chrome,

```bash
# add the repository to the list
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' \
```

```bash
  # add key for signed packages
  && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
```

```bash
  # don't forget to update the list after adding the repo
  && apt-get update \
  # finally install the package
  && apt-get install google-chrome-stable -y
```

### chromedriver

getting chromedriver is even more complex. 

this is because
- chrome and chromedriver needs to have **compatible** versions
- BUT chrome and chromedriver versions are not always the same


to get the compatible versions, you have to first download the chrome, and get the compatible version from a web page.

let's go through this with an example.

at the time of writing (Sep. 2022), the latest version of `google-chrome` is `105.0.5195.102`.

```bash
$ google-chrome --version
# Google Chrome 105.0.5195.102
```

to get the corresponding chromedriver version, go to 
https://chromedriver.storage.googleapis.com/LATEST_RELEASE_105

(if your chrome version was `80.x.x.x`, you would instead go to https://chromedriver.storage.googleapis.com/LATEST_RELEASE_80 )

the page would show you that the corresponding chromedriver version is `105.0.5195.52`

`Dockerfile` does this in the script by first getting the `105` part from the chrome version `105.0.5195.102`.

```bash
# get chrome version
RUN CHROME_VERSION="$(google-chrome --version)" \
  # remove string "Google Chrome " from the output
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  # get the major version
  # `%%` removes the longest match of the following pattern `.*`
  # so it removes `.0.5195.52` and stores `105` in the variable
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
```

next, it accesses the `LATEST_RELEASE_` page, and stores the response in the variable `CHROMEDRIVER_VERSION`.

```bash
&& CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
# 105.0.5195.52
# coincindentally, chrome & chromedriver versions match
```

then, download the zip via `curl --output`, and unzip in the `/tmp` folder

```bash
  # specify output location to /tmp/chromedriver_linux64.zip 
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  # go to `/tmp`
  && cd /tmp \
  # unzip in `/tmp`
  && unzip chromedriver_linux64.zip \
``` 

finally, we move chromedriver to `/usr/local/bin` and make sure chromedriver is executable

```bash
  # remove zip file because we don't need it anymore
  && rm -rf chromedriver_linux64.zip \
  # move executable to /usr/local/bin
  && mv chromedriver /usr/local/bin/chromedriver \
  # add permission
  && chmod +x /usr/local/bin/chromedriver \
```

as a clean up step, we move back to the `WORKDIR`

```bash
  && cd /code
```

put together, we have 

```bash
RUN CHROME_VERSION="$(google-chrome --version)" \
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
  && CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  && cd /tmp \
  && unzip chromedriver_linux64.zip \
  && rm -rf chromedriver_linux64.zip \
  && mv chromedriver /usr/local/bin/chromedriver \
  && chmod +x /usr/local/bin/chromedriver \
  && cd /code
```

(same result as `docker-compose.yml` in the first section)

ref: https://qiita.com/from_host/items/a12d75368ece7bf5bcc1