Ruby on Whales: Dockerizing Ruby and Rails development
Tags: ruby, rails, docker
Published: 2019-07-24 19:27:13


_Originally posted in [Martian Chronicles](https://evilmartians.com/chronicles/ruby-on-whales-docker-for-ruby-rails-development)_.

_This post is a b-side of my recent RailsConf talk "Terraforming legacy Rails applications" ([video](https://www.youtube.com/watch?v=-NKpMn6XSjU), [slides](https://speakerdeck.com/palkan/railsconf-2019-terraforming-legacy-rails-applications))._

In this post, I am not going to convince you to switch to Docker for application development (though you can check the [RailsConf video](https://www.youtube.com/watch?v=-NKpMn6XSjU) for some arguments). My goal is to share the configuration I currently use for Rails projects, and which was born in ~~production~~ development at [Evil Martians](https://evilmartians.com/). Feel free to use it!

I've started using Docker in my development environment about three years ago (instead of Vagrant which was too heavy for my 4GB RAM laptop). It wasn't all roses since the start, of course—I spent two years trying to find a configuration that is _good enough_, suitable not only for myself but also for my team.

Let me present this config here and explain (almost) every line of it, because we've all had enough of cryptic tutorials that just assume you _know stuff_.

> The source code could be found in the [evilmartians/terraforming-rails](https://github.com/evilmartians/terraforming-rails/blob/master/examples/dockerdev) repository on GitHub.

We use the following stack in this example:
- Ruby 2.6.3
- PostgreSQL 11
- NodeJS 11 & Yarn (for Webpacker-backed assets compilation)

## [`Dockerfile`](https://github.com/evilmartians/terraforming-rails/blob/master/examples/dockerdev/.dockerdev/Dockerfile)

`Dockerfile` defines the _environment_ for our Ruby application: this is where we run servers, console (`rails c`), tests, Rake tasks, interact with our code in any way _as developers_:

```dockerfile
ARG RUBY_VERSION
# See explanation below
FROM ruby:$RUBY_VERSION

ARG PG_MAJOR
ARG NODE_MAJOR
ARG BUNDLER_VERSION
ARG YARN_VERSION

# Add PostgreSQL to sources list
RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
  && echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list

# Add NodeJS to sources list
RUN curl -sL https://deb.nodesource.com/setup_$NODE_MAJOR.x | bash -

# Add Yarn to the sources list
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo 'deb http://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list

# Install dependencies
# We use an external Aptfile for that, stay tuned
COPY .dockerdev/Aptfile /tmp/Aptfile
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    build-essential \
    postgresql-client-$PG_MAJOR \
    nodejs \
    yarn=$YARN_VERSION-1 \
    $(cat /tmp/Aptfile | xargs) && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    truncate -s 0 /var/log/*log

# Configure bundler and PATH
ENV LANG=C.UTF-8 \
  GEM_HOME=/bundle \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3
ENV BUNDLE_PATH $GEM_HOME
ENV BUNDLE_APP_CONFIG=$BUNDLE_PATH \
  BUNDLE_BIN=$BUNDLE_PATH/bin
ENV PATH /app/bin:$BUNDLE_BIN:$PATH

# Upgrade RubyGems and install required Bundler version
RUN gem update --system && \
    gem install bundler:$BUNDLER_VERSION

# Create a directory for the app code
RUN mkdir -p /app

WORKDIR /app
```

This configuration contains the essentials only and could be used as a starting point. Let me show what we are doing here.

The first two lines could look a bit strange:

```dockerfile
ARG RUBY_VERSION
FROM ruby:$RUBY_VERSION
```

Why not just `FROM ruby:2.6.3`, or whatever Ruby stable version du jour it is? We want to make our environment configurable from the outside using Dockerfile as a sort of a template:

- the exact versions of runtime dependencies are specified in the `docker-compose.yml` (see below);
- the list of `apt`-installable dependencies is stored in a separate file (also see below).

The following three lines define arguments for PostgreSQL, NodeJS, Yarn, and Bundler versions:

```dockerfile
ARG PG_MAJOR
ARG NODE_MAJOR
ARG BUNDLER_VERSION
ARG YARN_VERSION
```

Since we do not expect anyone to use this Dockerfile without [Docker Compose](https://docs.docker.com/compose/), we do not provide default values.

Installing PostgreSQL, NodeJS, Yarn via `apt` requires adding their deb packages repos to the sources list.

For PostgreSQL (based in the [official documentation](https://www.postgresql.org/download/linux/debian/)):

```dockerfile
RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
  && echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list
```

For NodeJS (from [NodeSource repo](https://github.com/nodesource/distributions/blob/master/README.md#debinstall)):

```dockerfile
RUN curl -sL https://deb.nodesource.com/setup_$NODE_MAJOR.x | bash -
```

For Yarn (from the [official website](https://yarnpkg.com/en/docs/install#debian-stable)):

```dockerfile
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo 'deb http://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list
```

Now it's time to install the dependencies, i.e. run `apt-get install`:

```dockerfile
COPY .dockerdev/Aptfile /tmp/Aptfile
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    build-essential \
    postgresql-client-$PG_MAJOR \
    nodejs \
    yarn \
    $(cat /tmp/Aptfile | xargs) && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    truncate -s 0 /var/log/*log
```

First, let's talk about the Aptfile trick:

```dockerfile
COPY .dockerdev/Aptfile /tmp/Aptfile
RUN apt-get install \
    $(cat /tmp/Aptfile | xargs)
```

I borrowed this idea from [heroku-buildpack-apt](https://github.com/heroku/heroku-buildpack-apt), which allows installing additional packages on Heroku. If you're using this buildpack, you can even re-use the same Aptfile for local and production environment (though the buildpack's one provides more functionality).

Our [default Aptfile](https://github.com/evilmartians/terraforming-rails/blob/master/examples/dockerdev/.dockerdev/Aptfile) contains only a single package (we use Vim to edit Rails Credentials):

```sh
vim
```


In one of the previous project I worked on, we generated PDFs using LaTeX and [TexLive](https://www.tug.org/texlive/). Our Aptfile might look like this (those days I didn't use this trick):

```sh
vim
texlive
texlive-latex-recommended
texlive-fonts-recommended
texlive-lang-cyrillic
```

This way, we keep the task-specific dependencies in a separate file, making our Dockerfile more universal.

With regards to `DEBIAN_FRONTEND=noninteractive`, I kindly ask you to take a look at [answer on Ask Ubuntu](https://askubuntu.com/a/972528).

The `--no-install-recommends` switch helps us to save some space (and make our image slimmer) by not installing recommended packages. See more [here](http://xubuntugeek.blogspot.com/2012/06/save-disk-space-with-apt-get-option-no.html).

The last part of this `RUN` (`apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && truncate -s 0 /var/log/*log`) also serves the same purpose—clears out the local repository of retrieved package files (we installed everything, we don't need them anymore) and all the temporary files and logs created during the installation. We need this cleanup to be in the same `RUN` statement to make sure this particular [Docker layer](https://docs.docker.com/storage/storagedriver/#images-and-layers) doesn't contain any garbage.

The final part is mostly devoted to Bundler:

```dockerfile
ENV LANG=C.UTF-8 \
  GEM_HOME=/bundle \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3
ENV BUNDLE_PATH $GEM_HOME
ENV BUNDLE_APP_CONFIG=$BUNDLE_PATH \
  BUNDLE_BIN=$BUNDLE_PATH/bin
ENV PATH /app/bin:$BUNDLE_BIN:$PATH

# Upgrade RubyGems and install required Bundler version
RUN gem update --system && \
    gem install bundler:$BUNDLER_VERSION
```

The `LANG=C.UTF-8` sets the default locale to UTF-8. Otherwise Ruby uses US-ASCII for strings and bye-bye those sweet sweet emojis 👋

We set the path for gem installations via `GEM_HOME=/bundle`. What is `/bundle`? That's the path where we're going to mount as a _volume_ later to persist the dependencies on the host system, i.e., your development machine (see below in `docker-compose.yml`).

The `BUNDLE_PATH` and `BUNDLE_BIN` variables tell Bundler where to look for gems and Ruby executables.

Finally, we expose Ruby and application binaries globally:

```dockerfile
ENV PATH /app/bin:$BUNDLE_BIN:$PATH
```

That allows us to run `rails`, `rake`, `rspec` and other _binstubbed_ commands without prefixing them with `bundle exec`.


## [`docker-compose.yml`](https://github.com/evilmartians/terraforming-rails/blob/master/examples/dockerdev/docker-compose.yml)

[Docker Compose](https://docs.docker.com/compose/) is a tool to orchestrate our containerized environment. It allows us to link containers to each other, define persistent volumes and services.

Below is the compose file for a typical Rails application development with PostgreSQL as a database, and Sidekiq background job processor:

```yml
version: '3.4'

services:
  app: &app
    build:
      context: .
      dockerfile: ./.dockerdev/Dockerfile
      args:
        RUBY_VERSION: '2.6.3'
        PG_MAJOR: '11'
        NODE_MAJOR: '11'
        YARN_VERSION: '1.13.0'
        BUNDLER_VERSION: '2.0.2'
    image: example-dev:1.0.0
    tmpfs:
      - /tmp

  backend: &backend
    <<: *app
    stdin_open: true
    tty: true
    volumes:
      - .:/app:cached
      - rails_cache:/app/tmp/cache
      - bundle:/bundle
      - node_modules:/app/node_modules
      - packs:/app/public/packs
      - .dockerdev/.psqlrc:/root/.psqlrc:ro
    environment:
      - NODE_ENV=development
      - RAILS_ENV=${RAILS_ENV:-development}
      - REDIS_URL=redis://redis:6379/
      - DATABASE_URL=postgres://postgres:postgres@postgres:5432
      - BOOTSNAP_CACHE_DIR=/bundle/bootsnap
      - WEBPACKER_DEV_SERVER_HOST=webpacker
      - WEB_CONCURRENCY=1
      - HISTFILE=/app/log/.bash_history
      - PSQL_HISTFILE=/app/log/.psql_history
      - EDITOR=vi
    depends_on:
      - postgres
      - redis

  runner:
    <<: *backend
    command: /bin/bash
    ports:
      - '3000:3000'
      - '3002:3002'

  rails:
    <<: *backend
    command: bundle exec rails server -b 0.0.0.0
    ports:
      - '3000:3000'

  sidekiq:
    <<: *backend
    command: bundle exec sidekiq -C config/sidekiq.yml

  postgres:
    image: postgres:11.1
    volumes:
      - .psqlrc:/root/.psqlrc:ro
      - postgres:/var/lib/postgresql/data
      - ./log:/root/log:cached
    environment:
      - PSQL_HISTFILE=/root/log/.psql_history
    ports:
      - 5432

  redis:
    image: redis:3.2-alpine
    volumes:
      - redis:/data
    ports:
      - 6379

  webpacker:
    <<: *app
    command: ./bin/webpack-dev-server
    ports:
      - '3035:3035'
    volumes:
      - .:/app:cached
      - bundle:/bundle
      - node_modules:/app/node_modules
      - packs:/app/public/packs
    environment:
      - NODE_ENV=${NODE_ENV:-development}
      - RAILS_ENV=${RAILS_ENV:-development}
      - WEBPACKER_DEV_SERVER_HOST=0.0.0.0

volumes:
  postgres:
  redis:
  bundle:
  node_modules:
  rails_cache:
  packs:
```

We define **eight** services. Why so many? Some of them only define shared configuration for others (_abstract_ services, e.g., `app` and `backend`), others are used to specific commands using the application container (e.g., `runner`).

With this approach, we do not use `docker-compose up` command to run our application, but always specify the exact service we want to run (e.g., `docker-compose up rails`). That makes sense: in development, you rarely need all of the services up and running (Webpacker, Sidekiq, etc.).

Let's take a thorough look at each service.

### `app`

The main purpose of this service is to provide all the required information to build our application container (the one defined in the `Dockerfile` above):

```yml
build:
  context: .
  dockerfile: ./.dockerdev/Dockerfile
  args:
    RUBY_VERSION: '2.6.3'
    PG_MAJOR: '11'
    NODE_MAJOR: '11'
    YARN_VERSION: '1.13.0'
    BUNDLER_VERSION: '2.0.2'
```

The `context` directory defines the [build context](https://docs.docker.com/compose/compose-file/#context) for Docker: this is something like a working directory for the build process, it's used by the `COPY` command, for example.

We explicitly specify the path to Dockerfile since we do not keep it in the project root, packing all Docker-related files inside a hidden `.dockerdev` directory.

And, as we mentioned earlier, we specify the exact version of dependencies using `args` declared in the Dockerfile.

One thing that we should pay attention to is the way we tag images:

```yml
image: example-dev:1.0.0
```

One of the benefits of using Docker for development is the ability to synchronize the configuration changes across the team automatically. You only need to upgrade the local image version every time you make changes to it (or to the arguments or files it relies on). The worst thing you can do is to use `example-dev:latest` as your build tag.

Keeping an image version also helps to work with two different environments without any additional hassle. For example, when you work on a long-running "chore/upgrade-to-ruby-3" branch, you can easily switch to `master` and use the older image with the older Ruby, no need to rebuild anything.

> The worst thing you can do is to use `latest` tags for images in your `docker-compose.yml`.

We also _tell_ Docker to [use tmpfs](https://docs.docker.com/v17.09/engine/admin/volumes/tmpfs/#choosing-the-tmpfs-or-mount-flag) for `/tmp` folder within a container to speed things up:

```yml
tmpfs:
  - /tmp
```

### `backend`

We reached the most interesting part of this post.

This service defines the shared behavior of all  Ruby services.

Let's talk about the volumes first:

```yml
volumes:
  - .:/app:cached
  - bundle:/bundle
  - rails_cache:/app/tmp/cache
  - node_modules:/app/node_modules
  - packs:/app/public/packs
  - .dockerdev/.psqlrc:/root/.psqlrc:ro
```

The first item in the volumes list mounts the current working directory (the project's root) to the `/app` folder within a container using the `cached` strategy. This `cached` modifier is the key to efficient Docker development on MacOS. We're not going to dig deeper in this post (we're working on a separate one on this subject 😉), but you can take a look at [the docs](https://docs.docker.com/docker-for-mac/osxfs-caching/).

The next line _tells_ our container to use a volume named `bundle` to store `/bundle` contents. This way we persist our gems data across runs: all the volumes defined in the `docker-compose.yml` stay put until we run `docker-compose down --volumes`.

The following three lines are also there to get rid of the "Docker is slow on Mac" curse. We put all the generated files into Docker volumes to avoid heavy disk operations on the host machine:

```yml
- rails_cache:/app/tmp/cache
- node_modules:/app/node_modules
- packs:/app/public/packs
```

> To make Docker fast enough on MacOS follow these two rules: use `:cached` to mount source files and use volumes for generated content (assets, bundle, etc.).

The last line adds a specific `psql` configuration to the container. We mostly need it to persist the commands history by storing it in the app's `log/.psql_history` file. Why `psql` in the Ruby container? It's used internally when you run `rails dbconsole`.

Our [`.psqlrc`](https://github.com/evilmartians/terraforming-rails/blob/master/examples/dockerdev/.dockerdev/.psqlrc) file contains the following trick to make it possible to specify the path to the history file via the env variable (allow specifying the path to history file via `PSQL_HISTFILE` env variable, and fallback to the defaukt `$HOME/.psql_history` otherwise):

```sql
\set HISTFILE `[[ -z $PSQL_HISTFILE ]] && echo $HOME/.psql_history || echo $PSQL_HISTFILE`
```

Let's talk about the environment variables:

```yml
environment:
  - NODE_ENV=${NODE_ENV:-development}
  - RAILS_ENV=${RAILS_ENV:-development}
  - REDIS_URL=redis://redis:6379/
  - DATABASE_URL=postgres://postgres:postgres@postgres:5432
  - WEBPACKER_DEV_SERVER_HOST=webpacker
  - BOOTSNAP_CACHE_DIR=/bundle/bootsnap
  - HISTFILE=/app/log/.bash_history
  - PSQL_HISTFILE=/app/log/.psql_history
  - EDITOR=vi
  - MALLOC_ARENA_MAX=2
  - WEB_CONCURRENCY=${WEB_CONCURRENCY:-1}
```

There are several things here, and I'd like to focus one.

First, the `X=${X:-smth}` syntax. It could be translated as "For X variable within the container use the host machine X env variable value if present and another value otherwise". _Thus, we make it possible to run a service in a different environment provided along with the command, e.g., `RAILS_ENV=test docker-compose up rails`_.

The `DATABASE_URL`, `REDIS_URL`, and `WEBPACKER_DEV_SERVER_HOST` variables _connect_ our Ruby application to other services. The `DATABASE_URL` and `WEBPACKER_DEV_SERVER_HOST` variables are supported by Rails (ActiveRecord and Webpacker respectively) out-of-the-box. Some libraries support `REDIS_URL` as well (Sidekiq) but not all of them (for instance, Action Cable must be configured explicitly).

We use [bootsnap](https://www.github.com/Shopify/bootsnap) to speed up the application load time. We store its cache in the same volume as the Bundler data because this cache mostly contains the gems data; thus, we should drop everything altogether in case we do another Ruby version upgrade, for instance.

The `HISTFILE=/app/log/.bash_history` is the significant setting from the developer's UX point of view: it tells Bash to store its history in the specified location, thus making it persistent.

The `EDITOR=vi` is used, for example, by `rails credentials:edit` command to manage credentials files.

Finally, the last two settings, `MALLOC_ARENA_MAX` and `WEB_CONCURRENCY`, are there to help you keep Rails memory handling in check.

The only lines in this service yet to cover are:

```yml
stdin_open: true
tty: true
```

They make this service _interactive_, i.e., provide a TTY. We need it, for example, to run Rails console or Bash within a container.

It is the same as running a Docker container with the `-it` options.

### `webpacker`

The only thing I want to mention here is the `WEBPACKER_DEV_SERVER_HOST=0.0.0.0` setting: it makes Webpack dev server accessible from the _outside_ (by default it runs on `localhost`).

### `runner`

To explain what is this service for, let me share the way I use Docker for development:

- I start a Docker daemon running a custom `docker-start` script:

```sh
#!/bin/sh

if ! $(docker info > /dev/null 2>&1); then
  echo "Opening Docker for Mac..."
  open -a /Applications/Docker.app
  while ! docker system info > /dev/null 2>&1; do sleep 1; done
  echo "Docker is ready to rock!"
else
  echo "Docker is up and running."
fi
```

- Then I run `dcr runner` (`dcr` is an alias for `docker-compose run`) in the project directory to log into the container's shell; this is an alias for:

```sh
$ docker-compose run --rm runner
```

- I run (almost) everything from within this container: tests, migrations, Rake tasks, whatever.

As you can see, I do not spin a new container every time I need to run a task, and I'm always using the same one.

Thus, I'm using `dcr runner` the same way I used `vagrant ssh` years ago.

The only reason why it's called `runner` and not `shell`, for example, is that it also could be used to _run_ arbitrary commands within a container.

**Note**: The `runner` service is a matter of taste, it doesn't bring anything new comparing to the `web` service, except from the default `command` (`/bin/bash`); thus, `docker-compose run runner` is exactly the same as `docker-compose run web /bin/bash` (but shorter 😉).

## Bonus: [dip.yml](https://github.com/evilmartians/terraforming-rails/blob/master/examples/dockerdev/dip.yml)

If you still think that the _Docker Compose_ way is too complicated, there is a tool called [Dip](https://github.com/bibendi/dip) developed by one of my colleages at Evil Martians that aims to make the developer experience smoother.

It is especially useful if you have multiple compose files or platform-dependent configurations because it could glue them together and provide a universal interface to manage the Docker development environment.

We're going to tell you more about it in the future. Stay tuned!

---

**P.S.** Special thanks to [Sergey Ponomarev](https://github.com/sponomarev) and [Mikhail Merkushin](https://github.com/bibendi) for sharing their tips on the subject. 🤘

---

Read more dev articles on [https://evilmartians.com/chronicles](https://evilmartians.com/chronicles)!