The Serverless CI - Running Jenkins Slaves on AWS EKS Fargate
Tags: jenkins, kubernetes, aws, tutorial
Published: 2023-09-25 13:48:52

Jenkins requires no introduction, as it stands as the undisputed king of Continuous Integration. Over the years, it has adapted to all the technological disruptions in the industry, including Kubernetes. This blog post delves into an intriguing topic: how to execute on-demand slaves in a remote AWS Fargate cluster from a Jenkins master instance. For those wondering why such a capability is necessary, the following sections will elucidate not only the reasons but also the methods and associated advantages.

## Everything is remote!

Imagine this: you’re running cloud-native services on AWS EKS, and as the diligent engineer that you are, you establish two distinct clusters—one for production and another for all non-production purposes. You might be wondering why you would undertake such an approach. Here’s a hint: consider the blast radius. If you prioritize the security of your cloud-native services as fervently as we do at [NimbleWork](https://www.nimblework.com/), this decision makes sense. The dev cluster operates, among other things, all our Continuous Integration and Delivery tools. Speaking of Continuous Delivery, we run nightly pipelines that test the services for performance, security vulnerabilities, and regression before deploying them to the production cluster. In the production cluster, we adhere to a blue-green deployment model. This entails the Jenkins master, which operates on the dev EKS cluster, to run slaves on the production EKS cluster for various deployment, management, and general housekeeping tasks. Having outlined the reasons for this setup, let’s delve into the details of how it is accomplished.

## Prerequisites

This post assumes you have a running AWS EKS cluster, either on Fargate or Worker Nodes. You can refer to this [article](https://www.anadimisra.com/post/building-an-eks-fargate-cluster-with-terraform) for creating a Fargate cluster or [this](https://www.anadimisra.com/post/eks-karpenter) for Worker nodes if you don’t have them handy. The next step is to install Jenkins on Kubernetes, you can refer to [this](https://www.jenkins.io/doc/book/installing/kubernetes/) page in their official documentation for this. Since we’re configuring Jenkins slaves to run on AWS Fargate, install the [Kubernetes Plugin](https://plugins.jenkins.io/kubernetes/) in Jenkins.

## Serverless Jenkins Slaves

### Configuring Kubernetes Connection in Jenkins Master

Kubernetes cluster can be configured from the Manage Nodes and Clouds option on the Manage Jenkins Page. Navigate to `Manage Jenkins > Clouds > New Cloud` to open the Cloud configuration page

![Cloud configuration page in Jenkins LTS](https://www.anadimisra.com/assets/img/posts/jenkins-cloud.png)

Add a name for the cloud, choose Kubernetes in the Type section and click on “Create” to create the cloud configuration

![Create a Kubernetes cloud in Jenkins](https://www.anadimisra.com/assets/img/posts/new-cloud-jenkins.png)

Expand the Kubernetes Cloud details dropdown, this is where we will configure Jenkins master access to the AWS Fargate Cluster.

#### Kubernetes URL

Here we add the public API Server URL of the AWS Fargate Cluster. Log in to the AWS Management Console and select Elastic Kubernetes Service, click on the clusters link to list all clusters in your account and then click on the name of the cluster you want to connect Jenkins with to reach the overview page. Copy the API Server URL from the highlighted section in the image below and paste it to the Kubernetes URL field in the cloud configuration page in Jenkins.

![Cluster Info to get details of the API Server](https://www.anadimisra.com/assets/img/posts/eks-api-server.png)

Alternatively, you can get the same information by kubectl using the command line too. Point to your EKS cluster

```bash
export AWS_ACCESS_KEY_ID="KEY_ID_HERE"
export AWS_SECRET_ACCESS_KEY="ACCESS_KEY_HERE"
export AWS_SESSION_TOKEN="SESSION_TOKEN_HERE" 

aws eks update-kubeconfig --region us-east-1 --name mycluster

```

then run the `kubectl` commands as follows

```bash
kubectl cluster-info
Kubernetes control plane is running at https://XXXXXXXXXXXXX.gr7.us-east-1.eks.amazonaws.com

```

We’ll be using a Kubernetes Service Account to authenticate to the API Server. Perform the following steps on the AWS EKS cluster to enable Jenkins access.

- Create a namespace jenkins-jobs associated with the Fargate profile
- Create a service account named `jenkins-service-account` in the namespace
- Create a secret token named `jenkins-token` in the namespace associated with the service account
- Create a role-binding providing the service-account ClusterRole admin in this `jenkins-jobs` namespace

you can achieve this in multiple ways, via the Management Console or AWS CLI, we like to stick to IAC in NimbleWork so here’s a sample terraform snippet for the same:

<script src="https://gist.github.com/f958d70cce03d46e53f2bf5de0f69286.js?file=jenkins-rbac.tf"> </script>

Let’s retrieve the certificate key now, run the following command to get the service account certificate key and token

```bash
% kubectl get secret jenkins-token --namespace=jenkins-jobs -o yaml

```

The output contains ca.crt and token

```yaml
apiVersion: v1
data:
  ca.crt: XXXXXXXXXX
  namespace: XXXXXXXXXXXXXX
  token: XXXXXXXXXX
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: jenkins
    kubernetes.io/service-account.uid: XXXXXXXXXXXXXX
  creationTimestamp: "2023-05-20T17:25:16Z"
  labels:
    app.kubernetes.io/name: jenkins
  name: jenkins-token
  namespace: jenkins-jobs
  resourceVersion: "3388"
  uid: XXXXXXXXXXXXXX
type: kubernetes.io/service-account-token

```

the `ca.crt` value is `base64` encoded, decode it via the `base64 -d` command and paste the resultant value into the Kubernetes server certificate key field.

#### Kubernetes Namespace

Enter the value `jenkins-jobs` here

#### Credentials

Click on `Add > Jenkins` and choose the Secret text in the Kind dropdown list of the Credentials provider pop-up, then add base64 decoded value of the token from the output of the command kubectl get secret above to create the credentials.

![Adding Service Account token to Jenkins](https://www.anadimisra.com/assets/img/posts/jenkins-add-creds.png)

Click on the Test Connection button and you’ll see the message

`Connected to Kubernetes v1.28-eks-XXXXXXX`

When successfully connected!

#### Pod Template and Retention

Add the following values to Pod-Template and Retention settings

![Pod Settings](https://www.anadimisra.com/assets/img/posts/pod-settings.png)

Click Save to finish adding the cloud.

## Using the Kubernetes Cloud in Pipelines

Now that we have the Jenkins configuration in place let’s look at defining builds to run via Fargate Pods as slaves. We’re using the declarative pipeline syntax here. The pipeline job DSL Groovy should mention the configured cloud name as follows:

```groovy
  agent {
    hackernoonkube {
      yamlFile 'builder.yaml'
    }
  }

```

Add the labels defined in the POD template section above to run the job in a Jenkins slave running as a Fargate Pod.