Building Secure Docker Images for Production - Best Practices
Tags: docker, devops, security, rust
Published: 2023-06-30 18:42:40

With modern applications, Docker is a common tool on the tool belt.
We use Docker to develop and test our application and to run it in production.

Have you ever thought about the security of Docker containers in production?

While Container security is a wide field, let's focus on building more secure images for production deployment without spending hours online researching best practices.
This blog post introduces a low-friction approach to finding potential vulnerabilities in your images.

## A basic development image

Let's say we're working on an API service written in Rust. The repository contains a `Dockerfile` with just a handful of instructions. It might look like this:

```rust
FROM rust:1.70
EXPOSE 8000
COPY ./ ./
RUN cargo build --release
CMD ["./target/release/app"]
```

At first glance, the file looks straightforward. Docker Hub offers an official image for almost every programming language. The official Rust image has a full toolchain preinstalled. We can use this image right away to compile and run the project—a perfect fit for a development environment.
As soon as we go into production, however, things look different.

## Production requirements

We don't need the compiler toolchain when working with a compiled language. The Rust binary is self-contained, only requiring a libc runtime (which often is already present).
We might consider only shipping a minimized bundle to reduce the image size for a language like JavaScript.

No matter which language we work with, a production image must satisfy different requirements than a development image, such as installing specific package versions, strict permissions, etc.

Another aspect is image size. A development image can easily take up lots of disk space. Consider this example where an image based on `rust:1.70` takes up 3GB:

```bash
$ docker images
REPOSITORY                                                                  TAG            IMAGE ID       CREATED          SIZE
rust-dev                                                                latest         3ad31037e881   26 seconds ago   3.07GB
```

The compiler toolchain eats up some of the space, but the build artifact folder is most likely responsible for most of it. We need neither to run the application in production.

## Building a production-ready Docker image

Let's optimize our `Dockerfile`. To reduce image size, we implement a [multi-stage build](https://docs.docker.com/build/building/multi-stage/). With this approach, we still use the official Rust image to build the application. Once compiled, we copy the binary out of it into a small production image that only has a few things installed and configured. These changes lead to a much smaller disk space footprint.

```dockerfile
FROM rust:1.70.0 as builder

WORKDIR /usr/app
RUN USER=root cargo new --bin pokemon_api
WORKDIR /usr/app/pokemon_api

COPY ./Cargo.toml  ./Cargo.toml

RUN cargo build --release
RUN rm src/*.rs
COPY ./src ./src

# 5. Build for release.
RUN rm ./target/release/deps/pokemon_api*
RUN cargo build --release
#--------
FROM debian:bookworm-slim
EXPOSE 8000
RUN apt-get update
RUN apt-get install -y curl && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/app
COPY --from=builder /usr/app/pokemon_api/target/release/pokemon_api /usr/app/pokemon_api
ADD ./rocket_config.toml Rocket.toml
CMD ["/usr/app/pokemon_api"]

```

With our new production image in place, let's find out if there's more we can optimize.

## Prerequisities

In the following steps, we use a local Kubernetes cluster (such as [kind](https://kind.sigs.k8s.io/)) to test the image.
With the cluster up and running, let's install some tooling to help us with image scanning.
In this case, we're using [KubeClarity](https://github.com/openclarity/kubeclarity). Follow [the installation instructions](https://github.com/openclarity/kubeclarity#install-using-helm) in the README to install it into your development cluster.

## Deploy Docker image to a local cluster

With KubeClarity installed, let's deploy our service.
(Find the example source code [here](https://github.com/schultyy/rust-pokemon-api).)

```
kubectl create ns pokemon
kubectl apply -n pokemon -f k8s/deployment.yaml
```

As mentioned, this service is written in Rust, performing a simple HTTP Call to fetch a Pokemon CSV, which returns JSON data.
Upon further inspection of the source code, you'll notice it uses `std::process::Command` and `curl` to perform the HTTP request.
With this, we're simulating the code's dependency on specific system packages (`curl`).

Before we deploy this service in other environments, such as staging or production, we want to find out if there are any potential security problems with this particular image.

## Scan with KubeClarity

If you haven't already, in a separate terminal window, run the following command:

```bash
kubectl port-forward -n kubeclarity svc/kubeclarity-kubeclarity 9999:8080
```

Then, head over to [`http://localhost:9999/runtimeScan`](http://localhost:9999/runtimeScan). 

On the top, select `pokemon` as namespace and click on **Options** to open the settings dialogue. 

![KubeClarity Scan Options View](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/erhuhducdhi5j0narp93.png)

In the settings dialogue, select **CIS Docker Benchmark**. 

![Enable CIS Benchmark](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/04ohgx2ky87vd1bgwz0g.png)

Save your changes and click **Start Scan**.

After a few seconds, the scan will conclude and show us a summary of the findings.

![Scan finished](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qkk61apmy7zki1tmvx51.png)

Head to [`http://localhost:9999/applicationResources`](http://localhost:9999/applicationResources), click on **ghcr.io/schultyy/rust-pokemon-api:0.0.1** in the list. 
On the detail page, select **CIS Docker Benchmark**.


![Scan results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cl8w8uewpu41x028fdq1.png)

The list shows seven findings with varying severities.

## Explore findings

### Clear `apt-get caches` 

This step reduces the image size and cleans up superfluous and unneeded caches.
Check out [this list](https://github.com/goodwithtech/dockle/blob/master/CHECKPOINT.md#dkl-di-0005) for more details.

### Use `COPY` instead of `ADD` in Dockerfile

`COPY` and `ADD` both have an overlap in features. Prefer to use `COPY` over `ADD` as `COPY` only supports basic file-copying mechanisms.
`ADD`, however, has additional features that are not immediately obvious, such as `tar` extraction or remote URL support.

See [docs.docker.com](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#add-or-copy) for more details.

### Do not use update instructions alone in Dockerfile

By combining `apt-get update` with `apt-get install`, Docker will cache the update layer and reduce the total number of layers.

Consider this example:

```dockerfile
FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install package-a
```

In this case, the first time you run `docker build`, it executes every command and caches every line. The next time you run `docker build`, Docker determines nothing has changed and will finish much faster.

You return to this `Dockerfile` a few days later, realizing you need to install an additional package.

```dockerfile
FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install package-a package-b
```

Upon `build`, Docker realizes the first line hasn't changed and will use a cached layer. Distributions like Debian and Ubuntu update repositories frequently (i.e. deleting old package versions). 
Package sources updated by `apt-get update` from two days ago might point to a package or version that no longer exists, causing `apt-get install` to fail.

If you combine both lines, however, you will get updated package sources every time the list of packages to install changes.

[More details](https://github.com/goodwithtech/dockle/blob/master/CHECKPOINT.md#cis-di-0007)

### Create a user for the container

Running your application as `root` comes with several security implications. Therefore, dropping privileges as soon as possible is a good practice.

```dockerfile
RUN groupadd -g 10001 appuser && \
   useradd -u 10000 -g appuser appuser \
   && chown -R appuser:appuser /app

USER appuser:appuser
```

All instructions after `USER` will run as `appuser`, without privileges.

More details:

- [CIS-DI-0001](https://github.com/goodwithtech/dockle/blob/master/CHECKPOINT.md#cis-di-0001)
- [docs.docker.com](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user)

### HealthCheck

It's a good practice to include a Health Check within the `Dockerfile` to allow Docker to determine if the container is healthy.

This addition is helpful when the container is running but the application within the container has crashed.

```dockerfile
HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1
```

See [docs.docker.com](https://docs.docker.com/engine/reference/builder/#healthcheck) for more information.

## Implement fixes

With all discussed findings implemented, our Dockerfile now looks like this:

```dockerfile
FROM rust:1.70.0 as builder

WORKDIR /usr/app
RUN USER=root cargo new --bin pokemon_api
WORKDIR /usr/app/pokemon_api

COPY ./Cargo.toml  ./Cargo.toml

RUN cargo build --release
RUN rm src/*.rs
COPY ./src ./src

# 5. Build for release.
RUN rm ./target/release/deps/pokemon_api*
RUN cargo build --release
#--------
FROM debian:bookworm-slim
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD [ "curl --fail http://localhost:8000/health" ]

RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/app
COPY --from=builder /usr/app/pokemon_api/target/release/pokemon_api /usr/app/pokemon_api
COPY ./rocket_config.toml Rocket.toml

RUN groupadd -g 10001 appuser && \
   useradd -u 10000 -g appuser appuser \
   && chown -R appuser:appuser /usr/app

USER appuser:appuser

CMD ["/usr/app/pokemon_api"]
```

Next, we build a new image version:

```bash
docker build -t ghcr.io/schultyy/rust-pokemon-api:0.0.2 .
```

Once built, let's push it to the registry:

```bash
docker push ghcr.io/schultyy/rust-pokemon-api:0.0.2
```

With the new image version published to the container registry, let's deploy it into the test cluster:
```bash
kubectl apply -f k8s/deployment.yaml
```

As a last step, we re-run the scan (with the steps above) to verify we have no more fatal findings:

![Scan results without fatal findings](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rlp7alut27aaggftoqxf.png)

## Next Steps

Performing these kinds of checks is only of KubeClarity's features. Whenever you run a scan, it also scans for vulnerable packages. If you want to learn more about package scanning, check out [this blog post](https://dev.to/ciscoemerge/security-starts-before-the-production-deployment-3lp7).

Also, make sure to check out the [KubeClarity GitHub project!](https://github.com/openclarity/kubeclarity?utm_campaign=fy23_panoptica_ww_build-secure-docker-images-devto_blog&utm_source=blog&utm_medium=blog&utm_term=github-star)