Ship smaller Docker images - Best practices
Tags: docker, devops, kubernetes
Published: 2023-07-21 13:08:40

As a follow-up to [__Building Secure Docker Images for Production__](https://dev.to/ciscoemerge/building-secure-docker-images-for-production-best-practices-5d0f), this blog post investigates what you can do to optimize building Docker images for build time and size.

Special thanks to Ed Warnicke for the suggestion and input leading to this blog post.

## The problem

You're working on a new microservice. The first production deployment is scheduled. Production deployments require a Docker image. 
Time to start building a simple `Dockerfile`:

```dockerfile
FROM rust:1.69.0

WORKDIR /usr/app
RUN USER=root cargo new --bin hello
WORKDIR /usr/app/hello
COPY ./Cargo.toml  ./Cargo.toml

RUN cargo build --release
RUN rm src/*.rs
COPY ./src ./src
RUN rm ./target/release/deps/hello*
RUN cargo build --release
CMD [ "./target/release/hello" ]
```

The main goal is to get the application to compile and run. We use one of the official Docker images with the toolchain we need already pre-installed. 

A quick `docker run` confirms this image works. But should you push this into a registry as is? Probably not.
A glance at `docker images` confirms it:

```dockerfile
$ docker images
REPOSITORY TAG    IMAGE ID     CREATED     SIZE
largeimage latest da536ad251e2 6 weeks ago 2.25GB
```

This image is over 2GB. That's a lot of bytes for a statically compiled application.

If we took inventory of what's in the image, we'd find we're shipping the entire Rust toolchain and any intermediate build files in `target/`.
Neither is needed in production.

## Large images are problematic

There are many reasons why we don't want to ship large images.

1. We waste storage space. Even if storage is generally affordable, it's unnecessary.
2. We're shipping unnecessary files, such as compilers or intermediate/temporary build files. A production image does not need the compiler present. If we need to recompile the application, we do so as part of our CI/CD pipeline.
3. A significant hit on **Pod Startup Time**. Large images take longer to pull, which takes a toll on scale-up and healing if replicas die.

## How Docker builds an image

Before diving into a practical example, we must understand how Docker approaches building images.
We could easily assume Docker images work the same way as virtual machine images. We install what we need, resulting in a large file we ship.

The reality couldn't be more different.
If you run `docker build` several times in a row, you'll notice the first invocation takes a while to finish, while all subsequent runs only take a fraction. 

Why is that?

When Docker builds a new image, it creates it layer by layer. 
Every line in a `Dockerfile` results in an additional layer. If a line hasn't changed, there's no need to rebuild the layer.

Consider this Dockerfile:

```dockerfile
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y vim
RUN rm -rf /var/apt/lists*
```

We use `ubuntu`, update package sources, install `vim`, and, once completed, clean up apt package caches, etc.

```bash
docker build -t layers .
[+] Building 23.1s (9/9) FINISHED
 => [internal] load .dockerignore                                                                                            0.1s
 => => transferring context: 2B                                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                                         0.1s
 => => transferring dockerfile: 168B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                             3.0s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                0.0s
 => [1/4] FROM docker.io/library/ubuntu:latest@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508       6.2s
 => [2/4] RUN apt-get update                                                                                                 5.6s
 => [3/4] RUN apt-get install -y vim                                                                                         7.4s
 => [4/4] RUN rm -rf /var/apt/lists*                                                                                         0.3s
 => exporting to image                                                                                                       0.6s
 => => exporting layers                                                                                                      0.6s
 => => writing image sha256:6dcadf381f8e3b7ee143e818884a9f2a773a23a11bdac78831671b8fcb10d233                                 0.0s
 => => naming to docker.io/library/layers                                                                                    0.0s
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
layers       latest    6dcadf381f8e   18 seconds ago   179MB
```

We want to confirm the number of layers present:

```bash
$ docker inspect 6dcadf381f8e | jq ".[0].RootFS.Layers"
```

`docker inspect` outputs a lot of information, but we're only interested in the number of layers. Therefore, we're using `jq` to reduce the output to the essential bits and pieces:

```bash
[
"sha256:59c56aee1fb4dbaeb334aef06088b49902105d1ea0c15a9e5a2a9ce560fa4c5d",  "sha256:c15e21155336e02611e896a2a73e93db8d27c903aa6fff59b1cc5956669b4119",  "sha256:811f5dabaddf01eb6d50d6d54da46f18acbeaba9925051ad2ac3d69b91af500f", "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
]
```

The result: Four layers. 

Let's retry this experiment with the following `Dockerfile`:
```dockerfile
FROM ubuntu:latest
RUN apt-get update && \ 
    apt-get install -y vim && \
    rm -rf /var/apt/lists*
```

Result:

```bash
docker inspect fc57434f273e | jq ".[0].RootFS.Layers"
[
  "sha256:59c56aee1fb4dbaeb334aef06088b49902105d1ea0c15a9e5a2a9ce560fa4c5d",
  "sha256:136403671b74fc503d5d4c2e08c8ae99ab461390f448983adb9f1e86197e80cf"
]
```

We still run the same commands, now only with two layers, though! 

Why would you consider using fewer layers, though? 

Comparing image sizes, both use the same amount of space. Fewer layers make a difference once the image needs to be (re-)built. 
Fewer layers mean faster builds. Also, if a higher-up layer changes, it invalidates all the following layers. 

## A Case Study

The following content is based on [this GitHub issue](https://github.com/edwarnicke/govpp/issues/16) and [Dockerfile:](https://github.com/edwarnicke/govpp/blob/main/Dockerfile)

```dockerfile
ARG VPP_VERSION=e416893a597959509c7f667c140c271c0bb78c14
ARG UBUNTU_VERSION=20.04
ARG GOVPP_VERSION=v0.3.5

FROM ubuntu:${UBUNTU_VERSION} as vppbuild
ARG VPP_VERSION
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive TZ=US/Central apt-get install -y git make python3 sudo asciidoc
RUN git clone https://github.com/FDio/vpp.git
WORKDIR /vpp
RUN git checkout ${VPP_VERSION}
COPY patch/ patch/
RUN test -x "patch/patch.sh" && ./patch/patch.sh || exit 1
RUN DEBIAN_FRONTEND=noninteractive TZ=US/Central UNATTENDED=y make install-dep
RUN make pkg-deb
RUN ./src/scripts/version > /vpp/VPP_VERSION

#------

FROM vppbuild as version
CMD cat /vpp/VPP_VERSION

#------

FROM ubuntu:${UBUNTU_VERSION} as vppinstall
COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/
COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"]
RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends /pkg/*.deb ca-certificates iputils-ping iproute2 tcpdump iptables; \
    rm -rf /var/lib/apt/lists/*; \
    rm -rf /pkg

#------

FROM ubuntu:${UBUNTU_VERSION} as vpp
COPY --from=vppinstall / /

#------

FROM vpp as vpp-dbg
WORKDIR /pkg/
COPY --from=vppbuild ["/vpp/build-root/libvppinfra-dev_*_amd64.deb", "/vpp/build-root/vpp-dbg_*_amd64.deb", "/vpp/build-root/vpp-dev_*_amd64.deb", "./" ]
RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends ./*.deb

#------

FROM golang:1.15.3-alpine3.12 as binapi-generator
ENV GO111MODULE=on
ENV CGO_ENABLED=0
ENV GOBIN=/bin
ARG GOVPP_VERSION
RUN go get git.fd.io/govpp.git/cmd/binapi-generator@${GOVPP_VERSION}

#------

FROM alpine:3.12 as gen
COPY --from=vpp /usr/share/vpp/api/ /usr/share/vpp/api/
COPY --from=binapi-generator /bin/binapi-generator /bin/binapi-generator
COPY --from=vppbuild /vpp/VPP_VERSION /VPP_VERSION
WORKDIR /gen
CMD VPP_VERSION=$(cat /VPP_VERSION) binapi-generator ${PKGPREFIX+-import-prefix ${PKGPREFIX}}
```

The following sections highlight some interesting aspects you can apply to your next Docker image.

Let's break it down.

### Multiple `FROM` statements

When perusing this file, several lines start with `FROM`. The `Dockerfile` syntax only allows one `FROM` at the beginning of the file - unless we're using a **multi-stage build.**
A multi-stage build pays off when the project has a complex build process, but we only want to ship a lean image containing the artifact plus runtime dependencies. 
Multi-stage builds allow us to orchestrate the build process over several stages while only shipping the output of the last step. 
The `Dockerfile` above heavily uses build stages to optimize and control what ends up in the final image.

Quoting [Ed Warnicke](https://github.com/edwarnicke):

> Building vpp means bloating an image  
up with a bunch of build dependencies, build artifacts, etc. Building vpp installation on top of that would  
lead to a multi-GB image, which is undesirable. So we isolate that work in the 'vppbuild' stage.

<small>Source: [govpp#16](https://github.com/edwarnicke/govpp/issues/16)</small>

Looking at the first stage in the `Dockerfile`, we see the installation of build dependencies and any other necessary steps of the build process.
Subsequent stages copy the files that are needed from their predecessors.

__The result__: The final Docker image does not contain any trace of the build environment or the installation steps.

### Reusing `/var/apt/lists`

Looking more closely at [line 22](https://github.com/edwarnicke/govpp/blob/main/Dockerfile#L22), we see `COPY --from`, which copies files and directories from the previous stage.
Example:
```dockerfile
FROM ubuntu:${UBUNTU_VERSION} as vppinstall
COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/ #(1)
COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"]
```

Instead of copying build artifacts, it copies apt indices (the result of `apt-get update`).

Copying these files saves some time as we can avoid to re-run `apt-get update`, which usually gets executed before `apt-get install`.

### Only install necessary packages when `apt-get install`'ing

A few things happen behind the scenes when you run  `apt-get install <package>`. To shed some more light on the behavior, open a terminal and run this:

```bash
$ docker run --rm -it ubuntu:jammy /bin/bash
root@1a3eebf01ee7:/# apt-get update
<...omitted...>
root@1a3eebf01ee7:/# apt-get install vim
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libexpat1 libgpm2 libmpdec3 libpython3.10 libpython3.10-minimal libpython3.10-stdlib libreadline8 libsodium23 libsqlite3-0
  media-types readline-common vim-common vim-runtime xxd
Suggested packages:
  gpm readline-doc ctags vim-doc vim-scripts
The following NEW packages will be installed:
  libexpat1 libgpm2 libmpdec3 libpython3.10 libpython3.10-minimal libpython3.10-stdlib libreadline8 libsodium23 libsqlite3-0
  media-types readline-common vim vim-common vim-runtime xxd
0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
```

In the output, you notice how `apt` lets us know about required dependencies and suggested packages. If we want `vim` to work correctly, there's no way around the required packages.

_Suggested_ packages, on the other hand, may not be needed.
Besides suggested packages, sometimes we also encounter _recommended_ packages.

Debian defines them as such:

> Recommends
        This declares a strong, but not absolute, dependency. 
        The Recommends field should list packages that would be found
        together with this one in all but unusual installations.
> Suggests
        This is used to declare that one package may be more useful with
        one or more others. Using this field tells the packaging system
        and the user that the listed packages are related to this one
        and can perhaps enhance its usefulness, but that installing this
        one without them is perfectly reasonable.

Source: https://lists.debian.org/debian-mentors/2007/08/msg00037.html

For instance, the package [`virtualbox`](https://packages.ubuntu.com/focal/virtualbox) comes with recommended packages.

While this behavior can benefit a desktop environment, it can bloat up Docker images. 
To make sure we only install what is needed, we can use `--no-install-recommends`:

```dockerfile
RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends /pkg/*.deb ca-certificates iputils-ping iproute2 tcpdump iptables; \
    rm -rf /var/lib/apt/lists/*; \
    rm -rf /pkg
```

This flag skips all recommended and suggested packages.

## What are your thoughts?

What other steps do you take to ship smaller Docker images?
Share your tips and tricks in the comments.