Create Kubernetes Jobs in Golang using K8s client-go API
Tags: kubernetes, cloudnative, tutorial, github
Published: 2021-02-13 08:45:51

Few months back, I was building a system that involved launching and monitoring jobs on Kubernetes programmatically, basically I was writing a controller kind of a service that would take requests from an API and schedule jobs on the Kubernetes cluster. I was using Go and there is only one fully fledged client API in go called [client-go](https://github.com/kubernetes/client-go). In this post I am just documenting the process of creating k8s jobs using client-go, so that anyone working on the same problem would find this helpful. 

### Prerequisites:
1. A working Kubernetes setup, if you don't have a multi-node cluster, you can setup a local kubernetes environment using [minikube](https://kubernetes.io/docs/tutorials/hello-minikube/) or [microk8s](https://microk8s.io/).
2. Basic knowledge of Go is necessary.
3. Must have setup a working Go modules directory.

### Installing client-go package:
The client-go library is an official kubernetes client SDK by K8s community, you can use this library to programmatically manipulate your kubernetes cluster. Basically all Kubernetes controllers and components are built using this library. The kubernetes client tool `kubectl` is also built using client-go. client-go has following packages, which you must be aware of, before installing it:
   1. `kubernetes` : This package offers you APIs using which you can connect to your Kubernetes service. It provides different kinds of connection APIs. It also provides you the access to `clientset` interface, which can be used to specify and manipulate K8s objects.
   2. `discovery` : This package offers you APIs to discover K8s APIs. 
   3. `dynamic` : client-go provides you built-in types and constructs to specify any object, this strongly typed nature helps developers to identify type-errors at compile-type, but at the same time, the type system is rigid and offers less flexibility, on the other hand, dynamic API can be used to gain more flexibility because it allows developers to specify the objects as a nested map (can be de-serialized from JSON, YAML easily), this nested map can be changed easily as the need changes without having to re-compile the codebase, since the map is created dynamically and type-checking happens at runtime.
   4. `plugin/pkg/client/auth` : This module offers you authentication plugins.
   5. `transport` : This package is used to set up auth and start a connection.
   6. `tools/cache` : This package is useful for writing controllers. 

To install client-go as a module, you need to have the latest Go version which supports module system. See this [tutorial](https://golang.org/doc/tutorial/create-module) if you want to know more about Go module system. Let's create a simple go module called "github.com/k8sjobs"
```
mkdir k8sjobs
cd k8sjobs && go mod init github.com/k8sjobs
```
If everything is ok, you must see `go.mod` file created. Let's check.
```
cat k8sjobs/go.mod
``` 
Output:
```
module github.com/k8sjobs

go 1.15
```
So, we have the module ready, now let's install client-go. Client-go has many versions, we will install one of the latest stable versions of client-go. From the module root, i.e from `k8sjobs` directory, run : 
```
go get k8s.io/client-go@v0.20.2 
``` 
You many have to wait for installation to get complete. We can verify the installation by checking `go.mod` file. 
```
module github.com/k8sjobs

go 1.15

require k8s.io/client-go v0.20.2 // indirect
```
The `go.mod` has `client-go` as one of it's indirect dependencies which is fine as of now because we don't have any source file which imports `k8s.io/client-go` yet.

### What we will build?
We will build a simple command line tool that takes job name, container name and entrypoint as arguments and creates a job on the kubernetes cluster. We will be using golang's built-in package - `flag` to do this. We will create a file called `main.go` inside our module's root, which does everything that's required. So, let's start coding.
```go
import (
	"flag"
	"fmt"
)
func main() {
	jobName := flag.String("jobname", "test-job", "The name of the job")
	containerImage := flag.String("image", "ubuntu:latest", "Name of the container image")
	entryCommand := flag.String("command", "ls", "The command to run inside the container")

	flag.Parse()

	fmt.Printf("Args : %s %s %s\n", *jobName, *containerImage, *entryCommand)
}
```
So now we have a basic `main` function setup, this will simply accept arguments `jobName`, `containerName` and `imageName` and prints them. Now let's begin using `client-go` library.

#### 1. Connect to the cluster:
The first step is to connect to the cluster. Connection to the cluster requires K8s cluster `config` file, since we are connecting to an external remote cluster. Since it is assumed that you have a cluster set-up, you must be having the `config` file ready. It is assumed that the cluster config file is located at `$HOME/.kube/config`. We will write a function called `connectToK8s`.

Our imports till now:
```
import (
	"flag"
	"log"
	"os"
	"path/filepath"

	clientcmd "k8s.io/client-go/1.5/tools/clientcmd"
	"k8s.io/client-go/kubernetes"
)
```
The function `connectToK8s` connects to the cluster and then creates a `clientset` and returns it.
```go
func connectToK8s() *kubernetes.Clientset {
	home, exists := os.LookupEnv("HOME")
	if !exists {
		home = "/root"
	}

	configPath := filepath.Join(home, ".kube", "config")

	config, err := clientcmd.BuildConfigFromFlags("", configPath)
	if err != nil {
		log.Panicln("failed to create K8s config")
	}

	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		log.Panicln("Failed to create K8s clientset")
	}

	return clientset
}
```
Now we call the function inside `main`:
```
clientset := connectToK8s()
```
Now we have the clientset, we can create a job on the cluster. To do this, we will write a function called `launchK8sJob` which will be of the form:
```
func launchK8sJob(clientset *kubernetes.Clientset, jobName *string, image *string, cmd *string)
```
But before we begin, let's recall how the K8s job specification looks like:
```YAML
apiVersion: batch/v1
kind: Job
metadata:
  name: ls-job
spec:
  template:
    spec:
      containers:
      - name: ls-job
        image: ubuntu:latest
        command: ["ls", "-aRil"]
      restartPolicy: Never
  backoffLimit: 4
```
This is the sample specification taken from [here](https://kubernetes.io/docs/concepts/workloads/controllers/job/). The K8s specification uses `batch/v1` API to create Jobs, since everything in Kubernetes has a pre-defined template, we have a template for Jobs creation as well, which is shown above. The kubernetes `clientset` type provides `Batchv1` construct which we can use to manipulate jobs. Let's create a job using `Batchv1` construct.

Here are the imports till now:
```go
import (
	"context"
	"flag"
	"log"
	"os"
	"path/filepath"
	"strings"

	batchv1 "k8s.io/api/batch/v1"
	v1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	kubernetes "k8s.io/client-go/kubernetes"
	clientcmd "k8s.io/client-go/tools/clientcmd"
)
```
Now the function `launchK8sJob`:
```go
func launchK8sJob(clientset *kubernetes.Clientset, jobName *string, image *string, cmd *string) {
	jobs := clientset.BatchV1().Jobs("default")
	var backOffLimit int32 = 0

	jobSpec := &batchv1.Job{
		ObjectMeta: metav1.ObjectMeta{
			Name:      *jobName,
			Namespace: "default",
		},
		Spec: batchv1.JobSpec{
			Template: v1.PodTemplateSpec{
				Spec: v1.PodSpec{
					Containers: []v1.Container{
						{
							Name:    *jobName,
							Image:   *image,
							Command: strings.Split(*cmd, " "),
						},
					},
					RestartPolicy: v1.RestartPolicyNever,
				},
			},
			BackoffLimit: &backOffLimit,
		},
	}

	_, err := jobs.Create(context.TODO(), jobSpec, metav1.CreateOptions{})
	if err != nil {
		log.Fatalln("Failed to create K8s job.")
	}

	//print job details
	log.Println("Created K8s job successfully")
}
```
That's it! We just need to call this function from `main`. Here is the entire source file:
```go
package main

import (
	"context"
	"flag"
	"log"
	"os"
	"path/filepath"
	"strings"

	batchv1 "k8s.io/api/batch/v1"
	v1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	kubernetes "k8s.io/client-go/kubernetes"
	clientcmd "k8s.io/client-go/tools/clientcmd"
)

func connectToK8s() *kubernetes.Clientset {
	home, exists := os.LookupEnv("HOME")
	if !exists {
		home = "/root"
	}

	configPath := filepath.Join(home, ".kube", "config")

	config, err := clientcmd.BuildConfigFromFlags("", configPath)
	if err != nil {
		log.Fatalln("failed to create K8s config")
	}

	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		log.Fatalln("Failed to create K8s clientset")
	}

	return clientset
}

func launchK8sJob(clientset *kubernetes.Clientset, jobName *string, image *string, cmd *string) {
	jobs := clientset.BatchV1().Jobs("default")
	var backOffLimit int32 = 0

	jobSpec := &batchv1.Job{
		ObjectMeta: metav1.ObjectMeta{
			Name:      *jobName,
			Namespace: "default",
		},
		Spec: batchv1.JobSpec{
			Template: v1.PodTemplateSpec{
				Spec: v1.PodSpec{
					Containers: []v1.Container{
						{
							Name:    *jobName,
							Image:   *image,
							Command: strings.Split(*cmd, " "),
						},
					},
					RestartPolicy: v1.RestartPolicyNever,
				},
			},
			BackoffLimit: &backOffLimit,
		},
	}

	_, err := jobs.Create(context.TODO(), jobSpec, metav1.CreateOptions{})
	if err != nil {
		log.Fatalln("Failed to create K8s job.")
	}

	//print job details
	log.Println("Created K8s job successfully")
}

func main() {
	jobName := flag.String("jobname", "test-job", "The name of the job")
	containerImage := flag.String("image", "ubuntu:latest", "Name of the container image")
	entryCommand := flag.String("command", "ls", "The command to run inside the container")

	flag.Parse()

	clientset := connectToK8s()
	launchK8sJob(clientset, jobName, containerImage, entryCommand)
}
```
Note that, we are setting `RestartPolicy` as `Never` and `BackOffLimit` to zero, it means, if the job fails, it will never be restarted by the job controller because of `BackOffLimit` and the Pod created by the job will die gracefully because of `RestartPolicy` being set to `Never`.

  Let's build the module and run it. Before building, we set `GOBIN` to current working directory so that we will get the binary `k8sjobs` in the current working directory. 
```
export GOBIN=$PWD
go install
```
If compilation is successful, you should see `k8sjobs` binary generated in the current working directory. Let's run the binary by providing the arguments.
```
./k8sjobs --jobname=test --image=alpine:latest --command="ls -aRil"
```
If the cluster is configured properly, the command must exit successfully and we should see the output something like this.
```
2021/02/13 13:39:49 Created K8s job successfully
```
Now let's check the job using `kubectl`. (I am using `microk8s`, so I use `kubectl` binary that comes with `microk8s` toolchain. You can just use `kubectl` if you have installed it separately, or just alias `microk8s kubectl` to `kubectl`).

```
microk8s kubectl get jobs
```
Output:
```
NAME   COMPLETIONS   DURATION   AGE
test   1/1           73s        13m
```
The output shows that the job has been created and is completed successfully. Now let's describe the job to check if it's the same job we created.
```
microk8s kubectl describe job test
```
Output:
```
Name:           test
Namespace:      default
Selector:       controller-uid=48b7b61d-4ead-4033-91cd-035f539c27bf
Labels:         controller-uid=48b7b61d-4ead-4033-91cd-035f539c27bf
                job-name=test
Annotations:    <none>
Parallelism:    1
Completions:    1
Start Time:     Sat, 13 Feb 2021 13:39:49 +0530
Completed At:   Sat, 13 Feb 2021 13:41:02 +0530
Duration:       73s
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Pod Template:
  Labels:  controller-uid=48b7b61d-4ead-4033-91cd-035f539c27bf
           job-name=test
  Containers:
   test:
    Image:      alpine:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      ls
      -aRil
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  15m   job-controller  Created pod: test-l9zwk
  Normal  Completed         14m   job-controller  Job completed
```
This is the description of the job we created. You can verify that the container image name and the command is same as what we passed as arguments.

So that's it. This is just a basic example of how you can use `client-go` library to create K8s jobs. If you have free time, explore the library because there are whole lot of functionalities the library offers.