Docker : Build complex Solution with multi-contexts and Bake
Tags: docker, dotnet, tooling, devops
Published: 2024-05-16 06:14:17

In the .NET world, Solutions serve as a structural mechanism to organize multiple projects, such as applications and libraries, under a single umbrella. Typically, a Solution includes all the projects that make up a full application, streamlining the process of sharing DLLs. Instead of publishing them to a NuGet feed, they are directly referenced in the .csproj files within the Solution.

Enabling Docker support for a project prompts VisualStudio to generate a Dockerfile in that project. However, the image must be built from the Solution directory, not the individual project directory. This approach ensures that all necessary code, including that which resides outside the project directory within the Solution, is accessible during the build.

```shell
docker build -t repo/image:1.0.0 -f Project/Dockerfile .
# or, build from the project folder
# docker build -t repo/image:1.0.0 ..
```

It's fine if you have only 2 projects on the solution (The domain, and the API for example).

What if you have a big solution and you want to build one project that depends only on one of these projects ?

Imagine building an IoT application. You need to develop two modules to retrieve data from different data sources, with one module dedicated to each source. These modules will convert the data into a format that the rest of your application can understand and then transmit it to a queue system. On the receiving end of the queue, there are two modules: one for aggregating and storing the data, and another for generating alerts and dispatching them to a notification channel. Additionally, there is a fifth module that exposes the aggregated data through an API.

For the data ingestion modules, you establish an external library that encompasses the domain of each data source. To facilitate data exchange between your ingestion and processing modules, you create another library that defines the object structures. Lastly, you develop a separate library for your domain, which is utilized by your API.

We have 5 applications, and 4 libraries


![describe the relations between each projects of the solution](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8to1utarsfqn5qvm2krg.png)

```mermaid
flowchart TB
  IngestionA(IngestionA)
  IngestionA.Domain{{IngestionA.Domain}}
  IngestionB(IngestionB)
  IngestionB.Domain{{IngestionB.Domain}}
  Exchange{{Exchange}}
  Data(Data)
  Alerts(Alerts)
  API(API)
  Domain{{Domain}}

  IngestionA --> IngestionA.Domain
  IngestionA --> Exchange
  IngestionB --> IngestionB.Domain
  IngestionB --> Exchange
  Data --> Exchange
  Alerts --> Exchange
  API --> Domain
  
```

Let's build our 5 applications

```shell
docker build -t repo/ingestionA:1.0.0 -f IngestionA/Dockerfile .
docker build -t repo/ingestionB:1.0.0 -f IngestionB/Dockerfile .
docker build -t repo/data:1.0.0 -f Data/Dockerfile .
docker build -t repo/alerts:1.0.0 -f Alerts/Dockerfile .
docker build -t repo/api:1.0.0 -f API/Dockerfile .
```

## Docker build context

When executing the `docker build` command, Docker uses the specified folder as the build context and transfers it to the Docker daemon. It omits files and folders specified in the `.dockerignore` file but includes all other files (the complete codebase).

For building our API, Docker transfers the entire solution to the build context, as denoted by the `.` in our command, despite the fact that we might only require the API and the domain code.

## Docker build mutli-contexts

To avoid sending the entire codebase, we can limit the context to only the essential parts of the code. However, it's crucial to define the context for the dependent code. 

This can be achieved with the [multi-contexts](https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/?utm_campaign=2022-05-03-brnd-dockerfilemultiplebuildcontexts&utm_medium=social&utm_source=twitter) build feature, which necessitates an update to the Dockerfile to accommodate this new approach.

This is our full Dockerfile base for the IngestionA module

```Dockerfile
# file: Dockerfile
FROM mcr.microsoft.com/dotnet/runtime:8.0 AS base
USER app
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY IngestionA.Domain/IngestionA.Domain.csproj ./IngestionA.Domain/
COPY Exchange/Exchange.csproj ./Exchange/
COPY IngestionA/IngestionA.csproj ./IngestionA/
RUN dotnet restore ./IngestionA/IngestionA.csproj
COPY . .
WORKDIR /src/IngestionA
RUN dotnet build ./IngestionA.csproj -c $BUILD_CONFIGURATION -o /app/build --no-restore

FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish ./IngestionA.csproj -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false --no-restore

FROM base AS final
EXPOSE 5000
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "IngestionA.dll"]
```

Let's focus on the `build` stage to use multi-contexts

```Dockerfile
# file: Dockerfile
...

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY --from=domain IngestionA.Domain.csproj ./IngestionA.Domain/
COPY --from=exchange Exchange.csproj ./Exchange/
COPY IngestionA.csproj ./IngestionA/
RUN dotnet restore ./IngestionA/IngestionA.csproj
COPY --from=domain . ./IngestionA.Domain/
COPY --from=exchange . ./Exchange/
COPY . ./IngestionA/
WORKDIR /src/IngestionA
RUN dotnet build ./IngestionA.csproj -c $BUILD_CONFIGURATION -o /app/build --no-restore

...
```

To build our image, we need to define each context

```shell
# from the IngestionA project folder
docker build -t repo/ingestionA:1.0.0 \
  --build-context domain=../IngestionA.Domain/ \
  --build-context exchange=../Exchange/ \
  -f Dockerfile .
```

> ⚠️ Each context must define it's own .dockerignore

Now, to build all 5 applications, we must write quite long commands

```shell
docker build -t repo/api:1.0.0 \
  --build-context domain=./Domain
  API/
docker build -t repo/ingestionA:1.0.0 \
  --build-context domain=./IngestionA.Domain/ \
  --build-context exchange=./Exchange/ \
  IngestionA/
docker build -t repo/ingestionB:1.0.0 \
  --build-context domain=./IngestionB.Domain/ \
  --build-context exchange=./Exchange/ \
  IngestionB/
docker build -t repo/data:1.0.0 \
  --build-context exchange=./Exchange/ \
  Data/
docker build -t repo/alerts:1.0.0 \
  --build-context exchange=./Exchange/ \
  Alerts/
```

## Improve developer experience with Bake

Docker buildx [bake](https://docs.docker.com/build/bake/reference/) is a new way to build our images while drawing parallelization and build's orchestration.

We need to add one configuration file named `docker-bake.hcl` and to change the way we build images.

```hcl
// file: docker-bake.hcl

variable "TAG" {
  default = "latest"
}

target "api" {
  context = ./API/
  contexts = {
    domain = ./Domain/
  }
  tags = [
    "repo/api:${TAG}"
  ]
}

target "ingestionA" {
  context = ./IngestionA/
  contexts = {
    domain = ./IngestionA.Domain/
    exchange = ./Exchange
  }
  tags = [
    "repo/ingestionA:${TAG}"
  ]
}

target "ingestionB" {
  context = ./IngestionB/
  contexts = {
    domain = ./IngestionB.Domain/
    exchange = ./Exchange
  }
  tags = [
    "repo/ingestionB:${TAG}"
  ]
}

target "data" {
  context = ./Data/
  contexts = {
    exchange = ./Exchange
  }
  tags = [
    "repo/data:${TAG}"
  ]
}

target "alerts" {
  context = ./Alerts/
  contexts = {
    exchange = ./Exchange
  }
  tags = [
    "repo/alerts:${TAG}"
  ]
}

group "default" {
  targets = [
    "data",
    "alerts",
    "api",
    "ingestionA",
    "ingestionB"
  ]
}

group "ingestions" {
  targets = [
    "ingestionA",
    "ingestionB"
  ]
}

group "data-pipeline" {
  targets = [
    "ingestionA",
    "ingestionB",
    "data",
    "alerts"
  ]
}

```

To build our 5 applications, we have to run a single command

```shell
docker buildx bake
```

By default, Bake will search for a group `default` (or target if group isn't defined). In this case, the group "default" contains all our targets

If we want to build only the ingestion and processing's modules, we must target `data-pipeline` group

```shell
docker buildx bake data-pipeline
```

To build only the API, we must call bake with the target `api`

```shell
docker buildx bake api
```

All the images we build will be tagged with `latest`, we can change this behavior in different ways.

1. Override the TAG value using ENV

```shell
TAG=1.0.0 docker buildx bake api
```

2. Override the tag's image using `--set` argument

```shell
docker buildx bake api --set api.tags=api:1.0.0
```

3. Define a variable for each target and override them in a parameter file

```hcl
// file: docker-bake.hcl
variable "API_TAG" = {
  default = "latest"
}

target "api" { 
 context = ./API/ 
 contexts = {
   domain = ./Domain/
 }
 tags = [ 
   "repo/api:${API_TAG}" 
 ] 
}
...
```

```hcl
// file: bake-parameters.hcl
API_TAG = "1.0.0"
```

```shell
docker buildx bake api -f docker-bake.hcl -f bake-parameters.hcl
```

## Conclusion

Docker's multi-context builds are particularly beneficial for large solutions with numerous projects, as they help to reduce build context's size. 
While they introduce complexity by requiring the definition of each context, this is mitigated when combined with Bake. 

Bake streamlines the building process of the entire solution, allowing for the simultaneous construction of all images with a single command.