Jenkins: running workers in Kubernetes and Docker images build
Tags: docker, devops, kubernetes
Published: 2021-04-17 08:23:56

---
title: Jenkins: running workers in Kubernetes and Docker images build
published: true
date: 2021-02-27 15:22:32 UTC
tags: docker,devops,kubernetes
canonical_url: https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/
---

![](https://cdn-images-1.medium.com/max/883/1*NLnZANJaarm5DibG1pA3bg.png)

We have a Jenkins instance that is running jobs in Docker containers on its host.

Eventually, we’ve faced with an issue when the current AWS Ec2 instance t2.2xlarge (8 CPU, 32 RAM) during peak workload periods was too overloaded — not enough CPU time, not enough memory.

So, the first solution could be to proceed with its vertical scaling for example up to c5.9large, and proceed running builds on the master-host, or by moving some jobs to external workers.

At this time, we have three such workers — Android-builds are running on a PC in our office with Android studio, in the office we also have a bunch of MacMini for iOS builds, plus we have an additional EC2 in AWS for our QA team to run their UI autotests.

Now, let’s add an ability to run jobs on workers in a Kubernetes cluster.

We are using AWS Elastic Kubernetes Service and in this post will check how to use [Kubernetes Plugin](https://github.com/jenkinsci/kubernetes-plugin) for Jenkins.

- [Jenkins-master: spin up an instance](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins-master_spin_up_an_instance)
- [Docker install](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Docker_install)
- [Running Jenkins with Docker](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Running_Jenkins_with_Docker)
- [Jenkins Workers in Kubernetes](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins_Workers_in_Kubernetes)
- [Jenkins ServiceAccount](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins_ServiceAccount)
- [Jenkins ServiceAccount, and kubeconfig](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins_ServiceAccount_and_kubeconfig)
- [Jenkins Kubernetes Credentials](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins_Kubernetes_Credentials)
- [Jenkins Slaves Pod Template](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins_Slaves_Pod_Template)
- [Jenkins Job](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Jenkins_Job)
- [Docker in Docker via Docker in Kubernetes](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/#Docker_in_Docker_via_Docker_in_Kubernetes)

### Jenkins-master: spin up an instance

For the testing purpose, let’s create a dedicated AWS ЕС2 with Ubuntu 20.04, add a Docker there and set up a Jenkins master instance.

#### Docker install

Create an ЕС2, connect with SSH, install Docker:

```
root@ip-10–0–4–6:/home/ubuntu# apt update && apt -y upgrade
root@ip-10–0–4–6:/home/ubuntu# curl [https://get.docker.com/](https://get.docker.com/) | bash
```

And Docker Compose.

Find the [latest version](https://github.com/docker/compose/releases) on Github, at the moment of writing it was [1.28.4](https://github.com/docker/compose/releases/tag/1.28.4), download it:

```
root@ip-10–0–4–6:/home/ubuntu# curl -L “https://github.com/docker/compose/releases/download/1.28.4/docker-compose-$(uname -s)-$(uname -m)” -o /usr/local/bin/docker-compose]

root@ip-10–0–4–6:/home/ubuntu# chmod +x /usr/local/bin/docker-compose

root@ip-10–0–4–6:/home/ubuntu# docker-compose --version
docker-compose version 1.28.4, build cabd5cfb
```

#### Running Jenkins with Docker

Create a directory on the host to store Jenkins’ data:

```
root@ip-10–0–4–6:/home/ubuntu# mkdir jenkins_home
```

Write a Docker Compose file:

```
version: '3.5'

networks:
  jenkins:
    name: jenkins

services:

  jenkins:
    user: root
    image: jenkins/jenkins:2.249.3
     
    networks:
      - jenkins
    ports:
      - '8080:8080'
      - '50000:50000'
    volumes:
      - /home/ubuntu/jenkins_home/:/var/lib/jenkins
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/bin/docker
      - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7
    environment:
      - JENKINS_HOME=/var/lib/jenkins
      - JAVA_OPTS=-Duser.timezone=Europe/Kiev
    logging:
      driver: "journald"
```

Run the instance:

```
root@ip-10–0–4–6:/home/ubuntu# docker-compose -f jenkins-compose.yaml up
```

Open its URL in a browser:

![](https://cdn-images-1.medium.com/max/933/0*mpLYJYQBsn83BBdm.png)

_Administrator password_ can be found in the Docker Compose output during startup, or you can get it in a `/var/lib/jenkins/secrets/initialAdminPassword` file in the container:

```
root@ip-10–0–4–6:/home/ubuntu# docker exec -ti ubuntu_jenkins_1 cat /var/lib/jenkins/secrets/initialAdminPassword
15d***730
```

Log in, run the initial installation:

![](https://cdn-images-1.medium.com/max/886/0*MgaYdnBmiimL--54.png)

![](https://cdn-images-1.medium.com/max/989/0*u8lNNhCQRMJMC6HC.png)

Create a user:

![](https://cdn-images-1.medium.com/max/648/0*9xKSyYzbxvAzj8Up.png)

Finish it and go to the Plugins.

### Jenkins Workers in Kubernetes

Find the Kubernetes plugin:

![](https://cdn-images-1.medium.com/max/935/0*VJjMhEc86186aLcw.png)

Install it and go to the _Manage Nodes and Clouds_ > _Configure Clouds_:

![](https://cdn-images-1.medium.com/max/1024/0*jGWBvWfZgRbkg_0D.png)

Check the Kubernetes:

![](https://cdn-images-1.medium.com/max/691/0*ljFCzTGn1rT9rMp8.png)

Set the URL of your cluster’s API server and a Namespace:

![](https://cdn-images-1.medium.com/max/913/0*YQOUfxCTC8X9WXoN.png)

#### Jenkins ServiceAccount

In the cluster, create a Namespace and ServiceAccount which will be used by Jenkins for authorization.

On a Production setup, it’s better to configure access via an [EC2 Instance Profile](https://rtfm.co.ua/en/aws-iam-users-keys-rotation-ec2-iam-roles-and-jenkins/) with an IAM-role attached.

For now, add a [Kubernetes RoleBinding](https://rtfm.co.ua/en/kubernetes-part-5-rbac-authorization-with-a-role-and-rolebinding-example/) mapped to the [default admin role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) (or create your own — here the admin used just for simplicity as this is a PoC setup) in our _dev-1–18-devops-jenkins-slaves-ns_ namespace:

```
---
apiVersion: v1
kind: Namespace
metadata:
  name: dev-1-18-devops-jenkins-slaves-ns
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-slaves-service-account
  namespace: dev-1-18-devops-jenkins-slaves-ns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-slaves-rolebinding
  namespace: dev-1-18-devops-jenkins-slaves-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:                                                                                                                                                                     
- kind: ServiceAccount                                                                                                                                                        
  name: jenkins-slaves-service-account
  namespace: dev-1-18-devops-jenkins-slaves-ns
```

Deploy it:

```
$ kubectl apply -f jenkins-slaves-sa.yaml
namespace/dev-1–18-devops-jenkins-slaves-ns created
serviceaccount/jenkins-slaves-service-account created
rolebinding.rbac.authorization.k8s.io/jenkins-slaves-rolebinding created
```

#### Jenkins ServiceAccount, and kubeconfig

The next thing to do is to generate a [kubeconfig](https://rtfm.co.ua/kubernetes-kubectl-i-kubeconfig-obzor-fajla-dobavlenie-klastera-polzovatelya-i-konteksta/), which will use this ServiceAccount.

To do it, we need to get an EKS cluster ARN, its Certificate authority, API server’s address, and a [JWT token](https://rtfm.co.ua/en/kubernetes-serviceaccounts-jwt-tokens-authentication-and-rbac-authorization/) of the ServiceAccount.

Find a Secret cretaed for your ServiceAccount:

kubectl -n dev-1–18-devops-jenkins-slaves-ns get sa jenkins-slaves-service-account -o jsonpath=’{.secrets[0].name}’

jenkins-slaves-service-account-token-jsbb7

Certificate authority and Cluster ARN can be taken in the AWS Console:

![](https://cdn-images-1.medium.com/max/1024/0*59tDyOavIZIv_uV1.png)

Get the token from the secret, decrypt it with base64:

```
$ kubectl -n dev-1–18-devops-jenkins-slaves-ns get secret jenkins-slaves-service-account-token-jsbb7 -o jsonpath=’{.data.token}’ | base64 — decode
eyJ…s7w
```

Write a [kubeconfig](https://rtfm.co.ua/kubernetes-kubectl-i-kubeconfig-obzor-fajla-dobavlenie-klastera-polzovatelya-i-konteksta/), for example as a `jenkins-dev-1-18-kubeconfig.yaml` file:

```
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0...LQo=
    server: [https://676\*\*\*892.gr7.us-east-2.eks.amazonaws.com](https://676***892.gr7.us-east-2.eks.amazonaws.com)
  name: arn:aws:eks:us-east-2:534***385:cluster/bttrm-eks-dev-1-18
contexts:
- context:
    cluster: arn:aws:eks:us-east-2:534***385:cluster/bttrm-eks-dev-1-18
    user: jenkins-slaves-service-account
    namespace: dev-1-18-devops-jenkins-slaves-ns
  name: jenkins-slaves-service-account@bttrm-dev-1-18
current-context: jenkins-slaves-service-account@bttrm-dev-1-18
kind: Config
users:
- name: jenkins-slaves-service-account
  user:
    token: ZXl...N3c=
```

Check if it’s working:

```
$ kubectl -n dev --kubeconfig ../jenkins-dev-1–18-kubeconfig.yaml auth can-i get pod
yes
```

#### Jenkins Kubernetes Credentials

Go back to the Jenkins, add a new _Credential_:

![](https://cdn-images-1.medium.com/max/926/0*j-074xq1NAfTM_oS.png)

![](https://cdn-images-1.medium.com/max/615/0*Kv-AFWU2fDxzELnR.png)

Check its connection — click on the _Test Connection_, must see the _Connected to Kubernetes 1.18+ _message:

![](https://cdn-images-1.medium.com/max/1024/0*WkfqG9XxJYf_yDNg.png)

Save the settings.

#### Jenkins Slaves Pod Template

Go to the _Pod templates_:

![](https://cdn-images-1.medium.com/max/1024/0*wD_IJshyj-Qe73Gj.png)

Fill the fields of the pod and its default container, set the `jenkinsci/jnlp-slave` as the Docker image:

![](https://cdn-images-1.medium.com/max/1024/0*MZI9yAAKPYjo34Iv.png)

#### Jenkins Job

Create a test job with the _Pipeline_ type:

![](https://cdn-images-1.medium.com/max/614/0*GeUHIDBsyu7hY_n4.png)

Write a pipeline’s script:

```
podTemplate {
    node(POD_LABEL) {
        stage('Run shell') {
            sh 'echo hello world'
        }
    }
}
```

![](https://cdn-images-1.medium.com/max/821/0*DVSqSCLw1aI9g7a2.png)

Run the job and check Jenkins’ logs:

```
…
jenkins_1 | 2021–02–26 08:36:32.226+0000 [id=277] INFO hudson.slaves.NodeProvisioner#lambda$update$6: k8s-1-b5j7g-glscn-v0tfz provisioning successfully completed. We have now 2 computer(s)
jenkins_1 | 2021–02–26 08:36:32.522+0000 [id=276] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: dev-1–18-devops-jenkins-slaves-ns/k8s-1-b5j7g-glscn-v0tfz
…
```

And the pod is creating in the Kubernetes cluster:

```
$ kubectl --kubeconfig ../jenkins-dev-1–18-kubeconfig.yaml get pod
NAME READY STATUS RESTARTS AGE
k8s-1-b5j7g-glscn-v0tfz 0/1 ContainerCreating 0 12s
```

Job completion:

![](https://cdn-images-1.medium.com/max/1024/0*dfNWqSYmb6eaVMcM.png)

### Docker in Docker via Docker in Kubernetes

Our Jenkins is running as a Docker container, and it’s running its builds in Docker containers inside. It’s a good approach used by me for a long time as it gives us an opportunity not to install libraries on the host system more controlled way to setup build environments, and our developers can configure them as they wish.

So, now we need to realize the same — but in on the Kubernetes salves.

Install the [Docker pipeline plugin](https://plugins.jenkins.io/docker-workflow/) to the Jenkins:

![](https://cdn-images-1.medium.com/max/751/0*gMoDVh7e-OUOF_k7.png)

And create a new pipeline

Here we need another Docker image — [`docker.dind`](https://hub.docker.com/_/docker?tab=tags&page=1&ordering=last_updated&name=dind), and for it, we need another pod's template.

You can create it via Jenkins UI as we did it before, or we can describe it in the pipeline directly by using the [`podTemplate`](https://github.com/jenkinsci/kubernetes-plugin#pod-and-container-template-configuration).

For example, let’s build an NGINX image:

```
podTemplate(yaml: '''
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: docker
    image: docker:19.03.1-dind
    securityContext:
      privileged: true
    env:
      - name: DOCKER_TLS_CERTDIR
        value: ""
''') {
    node(POD_LABEL) {
        git 'https://github.com/nginxinc/docker-nginx.git'
        container('docker') {
            sh 'docker version && cd stable/alpine/ && docker build -t nginx-example .'
        }
    }
}
```

And run the job:

![](https://cdn-images-1.medium.com/max/685/0*YipmTTeLTYOL4_ZC.png)

![](https://cdn-images-1.medium.com/max/601/0*OfjWG7iqcXQSMz-E.png)

Done.

### Useful links

- [Kubernetes plugin Pipeline examples](https://github.com/jenkinsci/kubernetes-plugin/tree/master/examples)
- [Jenkins on Kubernetes — Building Docker Images](https://pushbuildtestdeploy.com/jenkins-on-kubernetes-building-docker-images/)

_Originally published at_ [_RTFM: Linux, DevOps и системное администрирование_](https://rtfm.co.ua/en/jenkins-running-workers-in-kubernetes-and-docker-images-build/)_._

* * *