Building Docker Images Smaller, Rootless and Non-Shell for Kubernetes
Tags: docker, kubernetes, tutorials
Published: 2023-04-17 08:51:17

---
title: Building Docker Images Smaller, Rootless and Non-Shell for Kubernetes
published: true
date: 2023-04-14 13:43:46 UTC
tags: docker, kubernetes, tutorials
canonical_url: https://rnemet.dev/posts/docker/image_k8s/
---

After building [a Docker image faster](https://rnemet.dev/posts/docker/building_image_fast/), I wanted to build it for the K8s cluster. Running the container on
the local machine isn't the same as running it on a cluster. I'm packaging a Go application in my example. But the same principles apply to any other language.

## Starting Dockerfile

I'm starting with the following Dockerfile(Dockerfile_1):

```dockerfile
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN go build -o app

FROM debian:buster

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]
```

And I build it with the following command:

```bash
docker buildx build -t rnemet/echo:0.0.1 . -f Dockerfile_1 --cache-to type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:main --load
```

I'm using the `--cache-to` and `--cache-from` flags to cache the build process and the `--load` to load the image into the local Docker daemon. Also, I'm using BuildKit to build an image.

Let's see what I got:

```bash
❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
rnemet/echo   0.0.1     6fc43a3d85eb   4 minutes ago   133MB
```

If I run the container as a Pod in a K8s cluster or a Docker container on my local machine:

```bash
❯ docker exec -it  echo bash
root@30fa7aa78401:/app# whoami
root
```

I managed:

* to open a shell in the container
* found out that I'm running as root
* I run the `whoami` command, which is not a command I need when I run the container

The `whoami` is not the problem, but the fact that it is present. I don't need it, and it's not part of my application. There must be others as well. I need to
clean up the image.

Then I learned that my app is run as root. This isn't good. My app needs to run as a non-root user. Why? There are many reasons. For example, the app can access the 
whole system with root privileges. Do you remember? The container runtime isolates the process from the host system using [namespaces](https://www.linux.com/news/understanding-and-securing-linux-namespaces/). But the process can still access the host system.

As the cherry on top, I can open a shell in the container and run commands as root. I do not even want to discuss why this is not good.

## Smaller, Rootless, and Non-Shell Image

The first thing I have to do is to change the target image. I'm using `debian:buster` as a target image. That needs to be changed. I decided to use `scratch` as a
target image. It's a minimal image. It's not a Linux distribution. It's not a shell. It's not a root user. It's a blank slate. So my next Dockerfile(Dockerfile_2):

```dockerfile
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN go build -o app

FROM scratch

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]
```

And build it:

```bash
docker buildx build -t rnemet/echo:0.0.2 . -f Dockerfile_2 --cache-to type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:main --load
```

And the result:

```bash
❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
rnemet/echo   0.0.2     188f310d88e2   42 seconds ago   19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   59 minutes ago   133MB
```

Wow, the image is smaller. But it fails when I try to run it as a Pod in a K8s cluster or a Docker container on my local machine. Argh...

The problem is how I build the application. I'm building it on a particular Linux image with Go installed. That means all the libs and tools that Go needs to build and 
run the application are part of the image. So, I must build the app with all its dependencies packed together. When it comes to Go, it's easy(Dockerfile_3):

```dockerfile
...
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app
...
```

I'm using the `CGO_ENABLED=0` flag to disable using Cgo. I'm using the `-ldflags="-s -w"` flag to strip the debug information from the binary. And results are:

```bash
❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED             SIZE
rnemet/echo   0.0.3     81e4098f0e76   15 seconds ago      13MB
rnemet/echo   0.0.2     188f310d88e2   25 minutes ago      19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   About an hour ago   133MB
```

The image is smaller, and it works. But I'm still running it as root. I need to change that, too. But when working with `scratch` image, I need to create the user in 
the `builder` image and then copy the `/etc/password` file to the `scratch` image(Dockerfile_3):

```dockerfile
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app

RUN useradd -u 10001 appuser

FROM scratch

WORKDIR /app
COPY --from=builder /etc/passwd /etc/passwd

USER appuser
COPY --chown=appuser:appuser --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]
```

I'm running as a non-root user with no shell, and the image is smaller. And I should be happy, but no. Why? Let's go a few steps back.

### Why Is Image Smaller?

In this case, the image is smaller because I'm using a `scratch` image as a target image, which produces an image size of around 19MB. Plus, I'm using stripping
debugging information from the binary. That's why the image is so small. That is nice, right?

### Cgo Or No Cgo?

I'm using the `CGO_ENABLED=0` flag to disable using Cgo. Why? Because I'm using a `scratch` image as a target image.

If I use `debian:buster` as a target image, I don't need to disable Cgo. Why? Because `debian:buster` image has all the libs and tools that Go needs to build and run 
the application. So I don't need to pack all the libs and tools with the application. I can use the libs and tools that are part of the image.

The `cgo` tool is a tool that allows Go programs to call C code. It enables your Go app to call OS's native libraries dynamically. Using it leads to smaller and faster 
builds. But, as you can see, it is not always possible to use it. So, in this case, I have to disable it and create a static binary. The static binary will be bigger 
but will have all the libs the app needs to run. It will be more portable. It is common to disable `cgo` when creating multi-architecture images.

Unless your app depends on C libraries, you can generally disable `cgo`. Then you have to leave `cgo` enabled. It's always some trade-off.

### Do I Always Need To Scratch When It Itches?

While the `scratch` image is minimal, it uses the `root` user, and I can not use the `USER` instruction to change it. I had to do a workaround. See Dockerfile_3. I had 
to add the user to the `builder` image and then copy `/etc/password` file to the `scratch` image. Not a big deal, but it's a workaround. And I'm not too fond of
workarounds. I wonder if it is the only workaround I will do when using the `scratch` image.

Small detour. You need a base image that you can trust and use easily. When choosing a base image, you should trust the creator of the base image, and you don't want 
to make any changes or workarounds. In most cases, you'll see Alpine Linux because it is small, and people trust it. But it has its own [problems](https://martinheinz.dev/blog/92). My personal choice is [Distroless images](https://github.com/GoogleContainerTools/distroless). They are small, secure, and easy to use. It is not as 
small as a `scratch` image, but still small. And they are secure. They are based on Debian, and they use `non-root` user. So you don't need to do any workarounds.

Let's check it out. I'm using `distroless/static-debian11:nonroot` as a target image(Dockerfile_4):

```dockerfile
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app

FROM gcr.io/distroless/static-debian11:nonroot

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]
```

And results are:

```bash
❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED              SIZE
rnemet/echo   0.0.4     f04110959f3c   About a minute ago   15.5MB
rnemet/echo   0.0.3     0280e6cfb546   5 hours ago          13MB
rnemet/echo   0.0.2     188f310d88e2   6 hours ago          19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   7 hours ago          133MB
```

Using `distroless/static-debian11:nonroot` as a target image, I'm getting an image size of around 15.5MB. And I'm running as a non-root user. I do not have any 
workarounds. As well, I'm having `USER` instruction. My Dockerfile is clean and easy to read. I like it.

One last try to use `cgo` in Dockerfile_5:

```dockerfile
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN go build -ldflags="-s -w" -o app

FROM gcr.io/distroless/base-debian11:nonroot

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]
```

After building it, the results are:

```bash
❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
rnemet/echo   0.0.5     030cf87dd36e   3 minutes ago    33.5MB
rnemet/echo   0.0.4     f04110959f3c   15 minutes ago   15.5MB
rnemet/echo   0.0.3     0280e6cfb546   5 hours ago      13MB
rnemet/echo   0.0.2     188f310d88e2   6 hours ago      19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   7 hours ago      133MB
```

So, the smallest image is when we use a `scratch` image as a target image. But we have to do some workarounds. The second smallest image is when we use
`distroless/static-debian11:nonroot` as a target image. And we don't have to do any workarounds.

When you compare Distrolees images with a `scratch` image, you can see that the second is smaller. But, I trust Google and do not have to do any workarounds if I use 
Distrolees images. In the case of the Go app, I had only one workaround. If I need to pack another app based on Java or NodeJS, I could do more work using a `scratch` 
image. Distrolees images already come prepackaged with some tools. So I don't have to do any workarounds.

### Base Image Conclusion

Let's summarize the results:

| image:tag | base image | size | non-user | no-shell | comments |
|-----------|------------|------|------|----------|-------------|
| rnemet/echo:0.0.1 | debian:buster | 33.5MB | no | no | - |
| rnemet/echo:0.0.2 | scratch | 19.1MB | no | no | do not work |
| rnemet/echo:0.0.3 | scratch | 13MB | yes | yes | static build, strip debug information, requires workaround for non-root user |
| rnemet/echo:0.0.4 | distroless/static-debian11:nonroot | 15.5MB | yes | yes | static build, strip debug information |
| rnemet/echo:0.0.5 | distroless/base-debian11:nonroot | 33.5MB | no | no | - |

In the end, I got what I wanted, a fast build, a small image, a non-root user, no shell access, and no workarounds. I'm happy with the results.

I'm not telling you to use Distroless images without any doubts. Maybe for your case, [Alpine Linux](https://hub.docker.com/_/alpine)](https://hub.docker.com/_/alpine) is better as it is tiny.
Or you prefer [Wolfi](https://github.com/wolfi-dev/). Or start from `scratch`. As you can see, there will be some trade-offs. You need to consider your apps needs, the effort willing to put into your Dockerfile and your trust in the creator of the base image.

Ah, one more thing please try to avoid the latest tags. Use versioned base images whenever possible. You'll have much more control and less headache.

## Nice to Have: Labels

Add some metadata to your images. They can be handy later on. Like:

```dockerfile
LABEL version="0.0.1-beta"
LABEL vendor="rnemet.dev"
LABEL release-date="2024-02-12"
```

You could also use `ARG` to set labels. Like:

```dockerfile
ARG VERSION=0.0.1-beta
ARG VENDOR=rnemet.dev
ARG RELEASE_DATE=2024-02-12
LABEL version="0.0.1-beta"
LABEL vendor="rnemet.dev"
LABEL release-date="2024-02-12"
```

## Conclusion

I hope you enjoyed this post and learned something new. If you found this helpful post, please share it with your friends. You can subscribe to my newsletter at the 
bottom of this post and/or share this post. I would appreciate it.

If you like what I write check my [newsletter](https://rnemet.substack.com/) and [blog](https://rnemet.dev).

## References

* [BuildKit](https://docs.docker.com/build/buildkit/)
* [Linux namespaces](https://www.linux.com/news/understanding-and-securing-linux-namespaces/)
* [Docker user namespaces](https://docs.docker.com/engine/security/userns-remap/)
* [Scratch image](https://hub.docker.com/_/scratch)
* [Building image from scratch](https://docs.docker.com/build/building/base-images/#create-a-simple-parent-image-using-scratch)
* [Distroless images](https://github.com/GoogleContainerTools/distroless)
* [Wolfi](https://github.com/wolfi-dev/)
* [Alpine Linux](https://hub.docker.com/_/alpine)