Building docker images in docker, and dynamic sequential Jenkins stages
Tags: jenkins, docker, kubernetes
Published: 2022-02-17 09:51:29

A couple of things I thought would be super easy but turned out to require a few hours of research and trial-and-error...

At [GOALS](https://goals.co/) we run [Jenkins in Kubernetes](https://plugins.jenkins.io/kubernetes/) for various reasons.
Some build jobs generates Docker images as artefacts, for example when we build new versions of some backend service.
However, [Kubernetes is deprecating Docker](https://levelup.gitconnected.com/kubernetes-is-deprecating-docker-in-2021-fa8317f9f070), and we are running all our nodes on [containerd](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd).
So, how to build Docker images then?

Turns out that Google has a project for it, named [Kaniko](https://github.com/GoogleContainerTools/kaniko). And since we're running GKE, and has [Workload Identity](https://github.com/GoogleContainerTools/kaniko#pushing-to-gcr-using-workload-identity) properly configured, it should be super easy to build Docker images and push them to our GCP managed docker repo, right?

No. I get this error:

```
aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "europe-west1-docker.pkg.dev/XXX": creating push check transport for europe-west1-docker.pkg.dev failed: GET https://europe-west1-docker.pkg.dev/v2/token?YYY: UNAUTHORIZED: authentication failed
```
_AWS credentials_ - wth?

After some testing I found [this](https://github.com/GoogleContainerTools/kaniko/issues/1287#issuecomment-1036638533), which solved the issue :). For an example Jenkinsfile, see below.

Another issue I had was that I wanted to generate dynamic build stages in Jenkins, and _execute them sequentially_. There are a lot of examples of how to execute dynamically generated stages in parallel, but that's not what I wanted.
Turned out to be super simple in the end ofc, but it aint simple until you've learned.

Here's an example Jenkinsfile that demonstrates both Kaniko and dynamic sequential build steps:

```
def stageConfigs = [
  [name: "Sara", age: 20, color: "blue"],
  [name: "Mona", age: 10, color: "green"],
  [name: "Lotta", age: 8, color: "red"]
]

def generateBuildStage(stageConfig) {
  return {
    stage("Building ${stageConfig.name}") {
      container('build-container') {
        echo "Building ${stageConfig.name}"
        writeFile(
          file: "${stageConfig.name}.generated",
          text: "${stageConfig.name} is ${stageConfig.age} years old and loves ${stageConfig.color} things")
      }
    }
  }
}

def buildStages = stageConfigs.collectEntries {
    ["${it}" : generateBuildStage(it)]
}

podTemplate(
  inheritFrom: 'linux',
  containers: [
    containerTemplate(name: "build-container", image: "busybox", command: "sleep", args: "infinity"),
    containerTemplate(name: "kaniko", image: "gcr.io/kaniko-project/executor:343f78408c891ef7a85bab1ecbf2dd69367a58bc-debug", command: "sleep", args: "infinity", runAsUser: "0", ttyEnabled: true)])
  {
  node(POD_LABEL) {
    stage("Checkout") {
      checkout(scm)
    }

    stage("Build application") {
      // execute builds in parallel
      // parallel(buildStages)
      // execute builds sequentially
      for (stage in buildStages.values()) {
        stage.call()
      }
      container('build-container') {
        writeFile(
          file: "Dockerfile",
          text: '''
            FROM busybox
            COPY *.generated ./
          ''')
        sh "ls -al"
      }
    }

    stage('Build image') {
      container('kaniko') {
        sh "/kaniko/executor --context `pwd` --dockerfile `pwd`/Dockerfile --destination europe-west1-docker.pkg.dev/XXX"
      }
    }
  }
}
```