Setting up Jenkins on MicroK8s
Tags: jenkins, kubernetes, microk8s, docker
Published: 2019-04-16 15:59:15

---
title: Setting up Jenkins on MicroK8s
published: true
description: How to run Jenkins on a local Kubernetes(MicroK8s) cluster
tags: jenkins, kubernetes, microk8s, docker
---

I recently bought a mini-pc (MSI Cubi 3 Silent) for home that I use as a server for various purposes and something that I wanted was to set up Jenkins for automatically building and deploying my side projects to my local MicroK8s cluster (Ubuntu Server 18.04).

It turns out you can install Jenkins on Kubernetes and thanks to [kubernetes-plugin](https://github.com/jenkinsci/kubernetes-plugin) Jenkins agents will run in ephimeral pods created just for building your task.

If you know what you have to do, it's actually pretty easy to configure but that's very often the case in hindsight, isn't it? I battled it out for a whole day and here are my findings, should probably take 30 mins to set up.

## Prerequisites
- [MicroK8s](https://microk8s.io/) cluster (should work with other Kubernetes clusters too), with **storage enabled**.
- Docker service running on the host (outside Kubernetes)
- [Helm](https://helm.sh) installed and deployed in Kubernetes

## Installing Jenkins on Kubernetes
```bash
# Download Jenkins chart values for customization. I'm linking the version I used, but the newest should work.
$ wget -O jenkins-chart.yml https://github.com/helm/charts/blob/b22774e2f2019c0e5d8182aab6111a84e95fa8ca/stable/jenkins/values.yaml

# Edit chart options to use NodePort instead of LoadBalancer, as MicroK8s doesn't
# ship with any: https://github.com/ubuntu/microk8s/issues/200#issuecomment-441180273
# 
# ServiceType: NodePort
# NodePort: 32000 (optional, if you want to have a fixed one for exposing it externally)
#
# Alternatively, if you have Ingress enabled on MicroK8s you could 
# use ServiceType: ClusterIP and add a rule in your Ingress configuration to
# direct traffic to Jenkins.
$ vim jenkins-chart.yml

$ helm install --name jenkins -f jenkins-chart.yaml stable/jenkins

# Follow the instructions to retrieve the generated password.
# If you need to uninstall Jenkins, then
$ helm del --purge jenkins
```
At this point, Jenkins should be accessible at http://your-server:32000. I then installed the Blue Ocean plugin as I find it more aesthetically pleasant plus it's designed for the [Jenkins Pipeline](https://jenkins.io/doc/book/pipeline/). Installing it is as easy as going to **Manage Plugins** and checking the **Blue Ocean** box. You don't need to check the other ones, they'll be added automatically as dependencies.

Now we're ready to create our first pipeline!

## Creating a Github pipeline
Let's assume we want to create a pipeline for a private repository we've got at Github. 

1. Follow the instructions [here](https://jenkins.io/doc/book/blueocean/creating-pipelines/) but skip the final pipeline creation wizard, as we're going to use a [Jenkinsfile](https://jenkins.io/doc/book/pipeline/jenkinsfile/) with the [declarative pipeline](https://jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline) syntax instead.
2. Create a **webhook** in your Github repo pointing to http://your-server:32000/github-webhook/ (customize with your domain/port/https, and keep the trailing slash) and verify that it's working (the secret doesn't matter). Theoretically this should be done automatically by Blue Ocean in step 1, but it didn't work for me. Note that if you're using a firewall or you're in a private network, you'll have to open the port and/or forward it to your server appropriately.
3. Create a file named **Jenkinsfile** at the root of the repository. I'm going to assume you've got a dockerized app and you've got the Kubernetes deployment files well configured.

```groovy
// Jenkinsfile

pipeline {
  agent {
    kubernetes {
      // this label will be the prefix of the generated pod's name
      label 'jenkins-agent-my-app'
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    component: ci
spec:
  containers:
    - name: docker
      image: docker
      command:
        - cat
      tty: true
      volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock
    - name: kubectl
      image: lachlanevenson/k8s-kubectl:v1.14.0 # use a version that matches your K8s version
      command:
        - cat
      tty: true
  volumes:
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock
"""
    }
  }

  stages {
    stage('Build image') {
      steps {
        container('docker') {
          sh "docker build -t your-registry/my-app:latest ."
          sh "docker push your-registry/my-app:latest"
        }
      }
    }

    stage('Deploy') {
      steps {
        container('kubectl') {
          sh "kubectl delete -f ./kubernetes/deployment.yaml"
          sh "kubectl apply -f ./kubernetes/deployment.yaml"
          sh "kubectl apply -f ./kubernetes/service.yaml"
        }
      }
    }
  }
}
```

Commit and push it to your repository. Jenkins should be notified and a build should be starting!

### A few things to note from the Jenkinsfile
- The `docker` and `kubectl` containers declared are so that we can run these commands in the pipeline without having them installed in the host, which is typically what you'd do with Jenkins outside Kubernetes.
- We're using the host's Docker daemon, which is why it was a prerequisite and the reason we defined `docker-sock` with `path: /var/run/docker.sock`. If you don't want to need to have Docker in your host, you could try [Docker in Docker](https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25).
- We're using [lachlanevenson/k8s-kubectl](https://github.com/lachie83/k8s-kubectl/blob/v1.14.0/Dockerfile), which is a very simple image with just `kubectl`. You could use any other image with `kubectl`, but make sure the image doesn't run with another user id or you'll have permission issues with Jenkins. I know it from the experience of trying to use [bitnami/kubectl](https://github.com/bitnami/bitnami-docker-kubectl/blob/ee7364cd186dfcf0bbb18dd0ad6b3e16ddadf564/1.14/debian-9/Dockerfile#L16).
- You might wonder why we've got `command: cat` and `tty: true`. They [avoid containers exiting early](https://github.com/jenkinsci/kubernetes-plugin#constraints). To be honest, I don't know exactly how it works.
- There's no specific step for `git`. If you use webhooks, the git step is automagically added when a push event is received.

### Other things of interest
- If you have submodules, make sure you check **Recursively update submodules** in your Jenkins repo build configuration, under **Advanced sub-modules behaviours**.
- You might notice 'zombie' processes (see `top` command) as builds are executed. I don't know why this happens but it could be a bug in Jenkins or Docker. It doesn't happen consistently though.

## Final words
Configuring Jenkins might seem a very daunting task and it's certainly not obvious. With Kubernetes and Helm I believe it's much simpler but still it's difficult to find easily reproducible instructions.

If you're setting a Jenkins environment without an existing Kubernetes cluster completely from scratch, you might want to have a look at [Jenkins X](https://jenkins-x.io/). Otherwise, I hope this guide works for you. 

See you next time!