Rethinking our ASP.NET Docker CI
Tags: dotnet, csharp, docker
Published: 2023-02-28 15:10:33

---
title: Rethinking our ASP.NET Docker CI
tags: [dotnet, csharp, docker]
canonical_url: https://keestalkstech.com/2023/02/rethinking-our-asp-net-docker-ci/
published: true
---

At Wehkamp we've been using ASP.NET for a long time. Containers brought .NET with us to our microservices architecture. We used a shell script in a special build container to build our service. Today we'll discuss how we turned that shell script into a [multi stage build Dockerfile](https://docs.docker.com/build/building/multi-stage/).

1. [Intro](#top)
2. [Goals](#goals)
3. [Project structure](#project-structure)
4. [Step 1: Configuration with global arguments](#step-1-configuration-with-global-arguments)
5. [Step 2: Build stage](#step-2-build-stage)  
   1. [(2.1) Restore in cacheable layers](#2-1-restore-in-cacheable-layers)  
   2. [(2.2) The build script](#2-2-the-build-script)  
   3. [(2.3) Optional testing and publishing](#2-3-optional-testing-and-publishing)
6. [Step 3: runtime stage](#step-3-runtime-stage)  
   1. [(3.1) Which runtime base image should we use?](#3-1-which-runtime-base-image-should-we-use)  
   2. [(3.2) Go self contained or not?](#3-2-go-self-contained-or-not)  
   3. [(3.3) The final runtime image](#3-3-the-final-runtime-image)
7. [All together](#all-together)
8. [Reading list](#reading-list)
9. [Changelog](#changelog)
10. [Comments](#comments)

## [Goals](#goals)

With the new setup we have some goals in mind:

* **Reproducibility** — _it should be on my machine as it is in the pipeline_. It would be great if I can just do a `docker build .` and everything should start building as it does in Jenkins.
* **Cacheability** — Docker uses [layers](https://vsupalov.com/docker-image-layers/), which greatly speeds up the system. When we do a _restore_ as a set of layers, we don't need to generate network traffic repeatedly.
* **Upgradability** — the current solution will focus on .NET 7.0, but we need things to be easily upgradable to a newer (or older) version of .NET.
* **Readability** — the Dockerfile should be treaded as a DevOps manifest, showing how to build and run our application. Let's make sure people understand what's going on. But also the _output_ that Docker shows in the terminal should be easy to read and understand.

If at some point we can add **security** to the Dockerfile, that would also be great.

## [Project structure](#project-structure)

Most of what we do is based on convention. Our project structure reflects those conventions:

```
.
├── src/
│   └── Blaze.Prime.Service.Api/
│       ├── Blaze.Prime.Service.Api.csproj
│       └── ...
├── test/
│   └── Blaze.Prime.Service.Api.UnitTests/
│       ├── Blaze.Prime.Service.Api.UnitTests.csproj
│       └── ...
├── .dockerignore
├── .gitignore
├── Blaze.Prime.sln
├── Dockerfile
└── nuget.config
```

It is important that the _name_ of the C# _project directory_ matches the name of the C# _project file_.

## [Step 1: Configuration with global arguments](#step-1-configuration-with-global-arguments)

Our Dockerfile may grow. Let's make it easy to upgrade and changing global settings by using [ARG](https://docs.docker.com/engine/reference/builder/#arg) instructions. We will use these arguments in the build and runtime images.

```docker
########################
# Configuration settings
########################

ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
   API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
   API_DLL="$API_NAME.dll" \
   PORT=5000 \
   ASPNET_VERSION="7.0" \
   # options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
   # minimal keeps your pipeline readable while inforing you what's going on
   VERBOSITY="minimal" \
   APP_DIR="/app" \
   TAG="" \
   EXECUTE_TESTS="true"
```

Let's construct the project and DLL names by convention. To make debugging easier, we set the `VERBOSITY` argument centrally. We find that the `minimal` settings is a great default, as it just gives enough output to understand what's going on.

Our CI fills the TAG with the tag of the Docker image that is being built.

## [Step 2: Build stage](#step-2-build-stage)

Now that we have the arguments in place, let's start with the beginning of our build image. We use the standard SDK provided by Microsoft. We don't feel the need to share [telemetry](https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry) and we would like to have our times in local Amsterdam time.

```docker
################################################
# Build image, uses ASPNET_VERSION, API_PROJECT, 
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################

FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build

ARG API_PROJECT VERBOSITY APP_DIR
ENV \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_NOLOGO=0

WORKDIR /build
```

Notice how we're _not_ defining `TAG` and `EXECUTE_TESTS` here. If we would, we would lose our caching ability when a new _tag_ is used or when testing is switched off.

### [2.1 Restore in cacheable layers](#2-1-restore-in-cacheable-layers)

In our old script we just executed a `dotnet restore` as part of the build script, which is fine, but nothing is cached by Docker. Our build script will take longer than necessary. [Andrew Lock did an excellent job of writing a solution that makes the restore cacheable](https://andrewlock.net/optimising-asp-net-core-apps-in-docker-avoiding-manually-copying-csproj-files-part-2/), so let's use his solution:

```docker
# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly

# copy global files to restore
COPY *.sln *.*config ./

# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
   echo "" \
   && echo "-------------" \
   && echo "1/4 RESTORING" \
   && echo "-------------" \
   && echo "" \
   && dotnet restore --verbosity "$VERBOSITY" || exit 1
```

This method restores the project structure and makes the `dotnet restore` cacheable.

Notice how we are using `echo` to make our output more readable?

[![Output of the docker build process. Notice how the title shines, so it makes it more readable what's going on.](https://keestalkstech.com/wp-content/uploads/2023/05/image-1024x997.png)](https://keestalkstech.com/wp-content/uploads/2023/05/image.png)

Output of the Dockerfile.

### [2.2 The build script](#2-2-the-build-script)

Now what should we copy? The base files like the solution and the NuGet config are already in there. Let's copy `src` and `test` only. This will help us cache builds that are triggered just because some documentation was changed (outside of the directory).

```docker
# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test

# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found

RUN echo "" \
   && echo "------------" \
   && echo "2/4 BUILDING" \
   && echo "------------" \
   && echo "" \
   && dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1
```

### [2.3 Optional testing and publishing](#2-3-optional-testing-and-publishing)

At the top of our Dockerfile, we've defined the `ARG` named `EXECUTE_TESTS`. Let's use it to test if we should do any testing. You might want to be able to skip testing to run faster locally.

```docker
# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
   && echo "-----------" \
   && echo "3/4 TESTING" \
   && echo "-----------" \
   && echo ""; \
   if [ "$EXECUTE_TESTS" = "true" ]; then \
      dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
   else \
      echo "Skipping unit tests"; \
   fi; \
   echo "" \
   && echo "--------------" \
   && echo "4/4 PUBLISHING" \
   && echo "--------------" \
   && echo "" \
   && dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1
```

## [Step 3: runtime stage](#step-3-runtime-stage)

The hard part is over: we have our application built. But there are still some choices to make!

### [3.1 Which runtime base image should we use?](#3-1-which-runtime-base-image-should-we-use)

So what runtime image are we going to use? We've built the entire project and checked the size of each resulting image:

| **Base image**                                             | **Total output size** |
| ---------------------------------------------------------- | --------------------- |
| mcr.microsoft.com/dotnet/aspnet:7.0                        | 218MB                 |
| mcr.microsoft.com/dotnet/nightly/aspnet:7.0-jammy-chiseled | 114MB                 |
| mcr.microsoft.com/dotnet/aspnet:7.0-alpine                 | 112MB                 |

Looks like the [chiseled project](https://canonical.com/blog/chiselled-containers-perfect-gift-cloud-applications) is very promising, as it promises a secure and minimal Debian image, but they have not released it for .NET 7\. Most of our projects will run fine on Alpine.

### [3.2 Go self contained or not?](#3-2-go-self-contained-or-not)

If you want to go even smaller, you can go to a [self contained (and trimmed) application](https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-self-contained). It is a great way to limit the attack surface even further and a will result in an even smaller image (which is also great). But there are some caveats: your application may take longer to boot and you can't cache your ASP.NET dependencies, so there might be some more network traffic involved. For now we just stick with the old adage as we try to read more on what the best way forward is.

### [3.3 The final runtime image](#3-3-the-final-runtime-image)

We've decided on Alpine for now, so our runtime image looks like this:

```docker
#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################

FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime

RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata

ARG PORT API_DLL APP_DIR

# create a new user and change directory ownership
RUN adduser --disabled-password \
   --home "$APP_DIR" \
   --gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"

# impersonate into the new user
USER dotnetuser

ENV \
   ASPNETCORE_URLS=http://*:$PORT \
   ASPNETCORE_ENVIRONMENT=Production \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

WORKDIR $APP_DIR
COPY --from=build $APP_DIR .

EXPOSE $PORT

ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"

ARG TAG
```

For security reasons, we create a new user, so we don't run as root.

## [All together](#all-together)

When we put it all together, we get the following `Dockerfile`:

```docker
########################
# Configuration settings
########################

ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
   API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
   API_DLL="$API_NAME.dll" \
   PORT=5000 \
   ASPNET_VERSION="7.0" \
   # options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
   # minimal keeps your pipeline readable while inforing you what's going on
   VERBOSITY="minimal" \
   APP_DIR="/app" \
   TAG="" \
   EXECUTE_TESTS="true"

################################################
# Build image, uses ASPNET_VERSION, API_PROJECT, 
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################

FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build

ARG API_PROJECT VERBOSITY APP_DIR
ENV \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_NOLOGO=0

WORKDIR /build

# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly

# copy global files to restore
COPY *.sln *.*config ./

# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
   echo "" \
   && echo "-------------" \
   && echo "1/4 RESTORING" \
   && echo "-------------" \
   && echo "" \
   && dotnet restore --verbosity "$VERBOSITY" || exit 1

# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test

# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found

RUN echo "" \
   && echo "------------" \
   && echo "2/4 BUILDING" \
   && echo "------------" \
   && echo "" \
   && dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1

# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
   && echo "-----------" \
   && echo "3/4 TESTING" \
   && echo "-----------" \
   && echo ""; \
   if [ "$EXECUTE_TESTS" = "true" ]; then \
      dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
   else \
      echo "Skipping unit tests"; \
   fi; \
   echo "" \
   && echo "--------------" \
   && echo "4/4 PUBLISHING" \
   && echo "--------------" \
   && echo "" \
   && dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1


#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################

FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime

RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata

ARG PORT API_DLL APP_DIR

# create a new user and change directory ownership
RUN adduser --disabled-password \
   --home "$APP_DIR" \
   --gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"

# impersonate into the new user
USER dotnetuser

ENV \
   ASPNETCORE_URLS=http://*:$PORT \
   ASPNETCORE_ENVIRONMENT=Production \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

WORKDIR $APP_DIR
COPY --from=build $APP_DIR .

EXPOSE $PORT

ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"

ARG TAG
```

Pretty cool, right?

## [Reading list](#reading-list)

While studying the subject we found some interesting blogs to read:

* On security: [How to build smaller and secure Docker Images for .NET](https://www.thorsten-hans.com/how-to-build-smaller-and-secure-docker-images-for-net5/)
* On `dotnet restore` & Docker optimizations: [Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2)](https://andrewlock.net/exploring-the-net-core-docker-files-dotnet-vs-aspnetcore-vs-aspnetcore-build/)
* On the new Chiselled project: [Chiselled Ubuntu: the perfect present for your containerised and cloud applications](https://ubuntu.com/blog/chiselled-containers-perfect-gift-cloud-applications)
* On Windows you might get the error "ERROR: failed to solve: error from sender: readdir: open src\\\*: The filename, directory name, or volume label syntax is incorrect." This is due to a [bug in the Docker Moby BuildKit](https://github.com/moby/buildkit/issues/1900), which cannot handle the wildcards. On PowerShell you can execute `$env:DOCKER_BUILDKIT=0` to disable BuildKit.

## [Changelog](#changelog)

* 2023-05-26: removed the section on compiling git info into the binary.
* 2023-05-26: installed the `tzdata` package to fix timezone support.
* 2023-05-26: notes on how to fix the Windows Docker error: "error from sender: readdir".
* 2023-05-26: nuget.config is now optional.
* 2023-05-26: make sure docker build exits on error.