Docker Buildkit: the proper usage of --mount=type=cache
Tags: docker, python, cache
Published: 2022-02-17 21:56:08

> <sup>TL;DR</sup> The contents of directories mounted with `--mount=type=cache` are not stored in the docker image, so it makes sense to cache intermediate directories, rather than target ones.

In `dockerfile:1.3` there is a feature of mounting file system directories during the build process, that can be used for [caching](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#run---mounttypecache) downloaded packages or compilation artifacts.

For example, the `uwsgi` package must be compiled every time it is installed, and at first glance, build times can be reduced by making the entire Python package directory cacheable:

```dockerfile
# syntax=docker/dockerfile:1.3
FROM python:3.10

RUN mkdir /pip-packages

RUN --mount=type=cache,target=/pip-packages \
      pip install --target=/pip-packages uwsgi
```

```bash
> docker build -t pip-cache -f Dockerfile.pip .
# ...
[+] Building 14.6s (7/7) FINISHED
```

Looks like everything went well, but the target directory is empty:

```bash
> docker run -it --rm pip-cache ls -l /pip-packages
total 0
```

Something is definitely wrong. You can see that during the build uWSGI was compiled and installed. You can even check it adding `ls` in the build process:

```dockerfile
RUN --mount=type=cache,target=/pip-packages \
      pip install --target=/pip-packages uwsgi \
      && ls -1 /pip-packages
```

```
> docker build -t pip-cache --progress=plain -f Dockerfile.pip .
<...>
#6 12.48 Successfully installed uwsgi-2.0.20
<...>
#6 12.91 __pycache__
#6 12.91 bin
#6 12.91 uWSGI-2.0.20.dist-info
#6 12.91 uwsgidecorators.py
#6 DONE 13.0s
<...>
```

Everything is in its place. But the final image is empty again:

```
> docker run -it --rm pip-cache ls -l /pip-packages
total 0
```

The thing is, the `/pip-packages` catalog, that is inside the image, and the catalog, that is in `RUN --mount=type=cache,target=<dirname>`, are completely different. Let's try to put something inside this directory and track its contents during the build process:

```dockerfile
RUN mkdir /pip-packages \
    && touch /pip-packages/foo \
    && ls -1 /pip-packages

RUN --mount=type=cache,target=/pip-packages \
    ls -1 /pip-packages \
    && pip install --target=/pip-packages uwsgi \
    && ls -1 /pip-packages

RUN ls -1 /pip-packages
```

```
> docker build -t pip-cache --progress=plain -f Dockerfile.pip-track .
<...>
#5 [stage-0 2/4] RUN mkdir /pip-packages
      && touch /pip-packages/foo
      && ls -1 /pip-packages
#5 sha256:fb542<...>
#5 0.211 foo  👈1️⃣
#5 DONE 0.2s

#6 [stage-0 3/4] RUN --mount=type=cache,target=/pip-packages
      ls -1 /pip-packages
      && pip install --target=/pip-packages uwsgi
      && ls -1 /pip-packages
#6 sha256:10ed6<...>
#6 0.292 __pycache__            👈2️⃣
#6 0.292 bin
#6 0.292 uWSGI-2.0.20.dist-info
#6 0.292 uwsgidecorators.py
#6 2.802 Collecting uwsgi       🤔3️⃣
#6 3.189   Downloading uwsgi-2.0.20.tar.gz (804 kB)
#6 4.400 Building wheels for collected packages: uwsgi
<...>
#6 13.34 __pycache__            👈4️⃣
#6 13.34 bin
#6 13.34 uWSGI-2.0.20.dist-info
#6 13.34 uwsgidecorators.py
#6 DONE 13.4s

#7 [stage-0 4/4] RUN ls -1 /pip-packages
#7 sha256:fb6f4<...>
#7 0.227 foo  👈5️⃣
#7 DONE 0.2s
<...>
```

- 1️⃣ file `foo` created successfully
- 2️⃣ the directory with the results of the previous `docker build` was mounted, and there's no `foo` file
- 3️⃣ uWSGI is downloaded, compiled and installed again
- 4️⃣ an updated uWSGI package appeared in the catalog
- 5️⃣ only the file `foo` is left in the directory

This means that `--mount=type=cache` only works in the context of a single `RUN` instruction, replacing the directory created inside the image with `RUN mkdir /pip-packages` and then reverting it back. Also, caching turned out to be ineffective because `pip` reinstalled uWSGI with a full compilation.

In this case, it would be correct to cache not the target directory, but `/root/.cache`, where `pip` stores all the artifacts:

```dockerfile
RUN --mount=type=cache,target=/root/.cache \
    pip install --target=/pip-packages uwsgi
```

```
> docker build -t pip-cache -f Dockerfile.pip-right .
> docker run -it --rm pip-cache ls -1 /pip-packages
__pycache__
bin
uWSGI-2.0.20.dist-info
uwsgidecorators.py
```

Now everything is in place, the installed packages have not disappeared.

Let's check the effectiveness of caching by adding the `requests` package:

```dockerfile
RUN --mount=type=cache,target=/root/.cache \
    pip install --target=/pip-packages uwsgi requests
                                                👆
```

```
> docker build -t pip-cache --progress=plain -f Dockerfile.pip-right .
<...>
#6 6.297 Collecting uwsgi
#6 6.297   Using cached uWSGI-<...>.whl  👈
#6 6.561 Collecting requests
#6 6.980   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
<...>
```

`pip` used the pre-built wheel file from `/root/.cache` and installed a ready-to-use package from it.

_All sources are available on [GitHub](https://github.com/doroshev-com/buildkit-mount-cache)._