Controlling Service Readiness in Docker Compose
Tags: docker, networking
Published: 2020-03-24 22:43:56

TL;DR Add [dokku/wait] as a service, then run `wait -c service:port`

---

If you use [Docker Compose], you probably have already tried to request a service that isn't ready, despite the container is running.

Actually "running" and "ready" are two different concepts from [Docker's point of view](https://docs.docker.com/compose/startup-order/). "Running" refers to the container state. "Ready" refers to the service running inside it.

> You can control the order of service startup and shutdown with the depends_on option. (...) However, for startup Compose does not wait until a container is "ready" only until it’s running.
>
> https://docs.docker.com/compose/startup-order/

## Reproducing The Problem

To illustrate the problem, let's run some SQL command against a [PostgreSQL] instance just after starting it as a [Docker Compose].

```yaml
# docker-compose.yaml

version: '3'

services:

    postgres:
        image: postgres:12
        environment:
            POSTGRES_PASSWORD: s3cr3t
```

Using the [docker-compose.yaml] file above, if we run these commands sequentially:

```bash
docker-compose up -d
docker-compose exec postgres psql -U postgres -c "SELECT NOW()"
```

We'll probably get this output error:

```
psql: error: could not connect to server: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
```

This error is due to the fact we try to run a SQL before the [PostgreSQL] is ready.

We therefore need to find a way to be sure the service is ready before making requests to it. That's make sense, but how to fix it?

Note that this problem may also happen to other services, because they follow the same pattern: container is running but the service isn't ready. Here a list of outputs for some popular services:

- [Elasticsearch]
    - `Failed connect to elasticsearch:9200; Connection refused`
- [MySQL]
    - `ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)`
- [RabbitMQ]
    - `Error: unable to perform an operation on node 'rabbit@a1b2c3d4e5f6'`
- [Redis]
    - `Could not connect to Redis at redis:6379: Connection refused`

## The Solution

We can solve this by (1) adding [dokku/wait] as a service in the [docker-compose.yaml], then (2) asking it to watch the service we want.

> [dokku/wait] is a (...) [Docker] utility that blocks until another container is accepting TCP connections, and errors-out if it cannot connect within a given timeout.
>
> https://github.com/dokku/docker-wait

### Step 1: Add [dokku/wait] as a service

Let's change the [docker-compose.yaml] file by adding a new `wait` service.

```yaml
# docker-compose.yaml

version: '3'

services:

    postgres:
        image: postgres:12
        environment:
            POSTGRES_PASSWORD: s3cr3t

    wait:
        image: dokku/wait
```

### Step 2: Ask [dokku/wait] to watch the target service

The new `wait` service we added to the [docker-compose.yaml] file is able to reach any service in the stack by giving its name and its exposed port (e.g. `postgres:5432`) thanks to the [network set up by Docker Composer](https://docs.docker.com/compose/networking/).

We can then ask it to block until the database get ready.

```bash
docker-compose up -d
docker-compose run wait -c postgres:5432
docker-compose exec postgres psql -U postgres -c "SELECT NOW()"
```

### Result: SQL commands will work as expected

The `wait -c postgres:5432` command will block for some seconds the `psql -U postgres -c "SELECT NOW()"` execution. That's why the SQL will work as expected.

Furthermore, we can wait for multiples services at once:

```bash
docker-compose run wait -c mysql:3306,redis:6379,rabbitmq:5672,elasticsearch:9200
```

## Conclusion

There are other approaches<sup>1</sup> to solve this problem, but they assume we have to install some script into the host machine or into each dependent service.

Adopting [dokku/wait] as suggested here is less intrusive because it's not needed to install a script on our machine neither on docker services stack.

Tip: We can even apply this to an existing project via [docker-compose.override.yaml] without changing anything in the project itself.

<sup>1</sup> [wait-for-it], [dockerize], [wait-for]

[dokku/wait]: https://hub.docker.com/r/dokku/wait
[Docker Compose]: https://docs.docker.com/compose/
[Docker]: https://www.docker.com/
[docker-compose.yaml]: https://docs.docker.com/compose/compose-file/
[docker-compose.override.yaml]: https://docs.docker.com/compose/extends/
[PostgreSQL]: https://www.postgresql.org/
[wait-for-it]: https://github.com/vishnubob/wait-for-it
[dockerize]: https://github.com/jwilder/dockerize
[wait-for]: https://github.com/Eficode/wait-for
[Elasticsearch]: https://www.elastic.co/elasticsearch/
[MySQL]: https://www.mysql.com/
[Redis]: https://redis.io/
[RabbitMQ]: https://www.rabbitmq.com/