Write your Kubernetes Infrastructure as Go code - Getting started with cdk8s
Tags: kubernetes, programming, go, tutorial
Published: 2022-07-14 10:40:46

Infrastructure as Code (IaC) is a well established paradigm and refers to the standard practice of treating  infrastructure (network, disk, storage, databases, message queues etc.) in the same way as application code and applying general software engineering practices including source control versioning, testing and more. For exampple, [Terraform](https://www.terraform.io/) and [AWS CloudFormation](https://docs.aws.amazon.com/cloudformation/index.html) are widely-adopted technologies that use configuration files/templates to represent the infrastructure components.

### Infrastructure-IS-Code - A different way of thinking about this

Imagine you have an application that comprises of a Serverless function fronted by an API Gateway along with a NoSQL database as the backend. Instead of defining it in a static way (using JSON, YAML etc.), one can represent these components using standard programming language constructs such as classes, methods, etc. Here is pseudo-code example:

```
DBTable table = new DBTable("demo-table");
table.addPrimaryKey("email", Type.String);

Function function = new Function("demo-func");
function.addEnvVars("TABLE_NAME", table.Name());

APIGateway apigw = new APIGateway();
apigw.addFunctionIntegration(function);
```

Notice the (hypothetical) classes `DBTable`, `Function` and `APIGateway` and the way they are used. For e.g. a `function` can reference the `table` object and get it's name - all this comes to life during the program runtime and taken care of by the implementation details of the underlying framework/platform.

**But, you don't have to write pseudo-code for your production infrastructure**

... thanks to existing solutions such as [cdk8s](https://cdk8s.io/), [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html), [Pulumi](https://www.pulumi.com/), [CDK for Terraform](https://www.terraform.io/cdktf) (`cdktf`) etc. Almost all these solutions follow a similar approach - write code to define infrastructure, then convert that into configuration, for e.g. Kubernetes manifest (YAML), [AWS CloudFormation template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/gettingstarted.templatebasics.html#gettingstarted.templatebasics.what), [HCL config](https://www.terraform.io/language/syntax/configuration) etc., which can then be applied using standard tooling.

While we are on this topic, its hard not to mention [the Go programming language](https://go.dev/) and its ubiquitous presence in the cloud services and infrastructure domain. It combines the safety of a compiled language with the speed a interpreted language (like `Python`), has a [robust standard library](https://pkg.go.dev/std) and compiles to a single binary. These and many more qualities have led to lots of cloud-native software (IaC, monitoring, observability etc.) written in Go, such as [Prometheus](https://github.com/prometheus/prometheus), [Terraform](https://github.com/hashicorp/terraform), [Grafana](https://github.com/grafana/grafana), [Jaeger](https://github.com/jaegertracing/jaeger) etc. 

> ["In fact, over 75 percent of projects in the Cloud Native Computing Foundation are written in Go."](https://go.dev/solutions/cloud)

---

## Applying "Infra-Is-Code" mantra to Kubernetes

Over the course of multiple blog posts, I will cover how Go developers can use the [cdk8s](https://cdk8s.io/docs/latest/) (Cloud Development Kit for Kubernetes) project for defining Kubernetes resources. It's an open-source framework (also part of CNCF)  that provides high-level abstractions which can be composed into larger Kubernetes applications. Instead of adopting YAML or other configuration/template driven approach, `cdk8s` supports multiple programming languages, which means you can work with Kubernetes resources using familiar concepts such as classes, methods, etc. Ultimately, `cdk8s` generates Kubernetes manifests which you can `apply` using `kubectl` - business as usual!

> At the time of writing, `cdk8s` supports Go, Typescript, Python and Java

This blog post will start things off and provide a gentle yet hands-on intro to `cdk8s`. By the end of it, you will be familiar with the key concepts and understand how to use `cdk8s` Go APIs to define a Kubernetes application, deploy (using `kubectl`) and test it out.

### Before you begin...

Make sure you have `Go` ([v1.16](https://go.dev/dl/) or above) and [cdk8s CLI](https://github.com/cdk8s-team/cdk8s-cli) installed. Also, you need to have access to a Kubernetes cluster. For learning and experimentation I would recommend using a single-node cluster running locally - such as [minikube](https://minikube.sigs.k8s.io/docs/start/), [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation), etc.

> I generally use `minikube`, so setting up a cluster is as simple as `minikube start`

**To install cdk8s CLI**

You can choose from the below options:

```bash
#homebrew
brew install cdk8s

#npm
npm install -g cdk8s-cli

#yarn
yarn global add cdk8s-cli
```

---

## Alright, lets get started!

> Although this blog post will provide step-by-step instructions, you can always refer to the complete [code on Github](https://github.com/abhirockzz/cdk8s-for-go-developers/tree/master/part1-getting-started)


`cdk8s` makes it really easy for you get started and bootstrap your application. You don't need to guess and figure out how to structure your project, setup dependencies etc. since the `cdk8s init` command does it for you!

```bash
cdk8s init go-app

#output
....

 Your cdk8s Go project is ready!

   cat help      Prints this message  
   cdk8s synth   Synthesize k8s manifests to dist/
   cdk8s import  Imports k8s API objects to "imports/k8s"

  Deploy:
   kubectl apply -f dist/
```

Once completed, you will get a directory structure as such:

```bash
.
├── cdk8s.yaml
├── dist
│   └── test.k8s.yaml
├── go.mod
├── go.sum
├── help
├── imports
│   └── k8s
│       ├── internal
│       │   └── types.go
│       ├── jsii
│       │   ├── jsii.go
│       │   └── k8s-0.0.0.tgz
│       ├── k8s.go
│       ├── k8s.init.go
│       └── version
└── main.go
```

Update the generate `go.mod` file, and replace it with the following - this is to make things simpler for you. 

> Feel free to use the latest version of the modules if needed.

```go
module getting-started-with-cdk8s-go

go 1.16

require (
	github.com/aws/constructs-go/constructs/v10 v10.1.42
	github.com/aws/jsii-runtime-go v1.60.1
	github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2 v2.3.29
)
```

### You're all set to write some write some Go code!

The canonical Kubernetes "hello world" is to get a `nginx` server up and running. The  easiest option is to use simply use `kubectl run` e.g. `kubectl run nginx --image=nginx`. But, since this is imperative, we switch to a declarative way where we define our desired state (in a `yaml` file) and ask Kubernetes to figure things out. 

For e.g. we can write a `Deployment` manifest and submit it to Kubernetes using `kubectl apply -f <name of the yaml file`>.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-nginx
  template:
    metadata:
      labels:
        app: hello-nginx
    spec:
      containers:
        - image: nginx
          name: nginx-container
          ports:
            - containerPort: 8080
```

### But we are here to minimise yaml...

So, open the `main.go` file and copy the below Go code. Don't worry, I will walk you through it!

```go
package main

import (
	"getting-started-with-cdk8s-go/imports/k8s"
	"github.com/aws/constructs-go/constructs/v10"
	"github.com/aws/jsii-runtime-go"
	"github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2"
)

type NginxChartProps struct {
	cdk8s.ChartProps
}

func NewNginxChart(scope constructs.Construct, id string, props *NginxChartProps) cdk8s.Chart {
	var cprops cdk8s.ChartProps
	if props != nil {
		cprops = props.ChartProps
	}
	chart := cdk8s.NewChart(scope, jsii.String(id), &cprops)

	selector := &k8s.LabelSelector{MatchLabels: &map[string]*string{"app": jsii.String("hello-nginx")}}

	labels := &k8s.ObjectMeta{Labels: &map[string]*string{"app": jsii.String("hello-nginx")}}

	nginxContainer := &k8s.Container{Name: jsii.String("nginx-container"), Image: jsii.String("nginx"), Ports: &[]*k8s.ContainerPort{{ContainerPort: jsii.Number(80)}}}

	k8s.NewKubeDeployment(chart, jsii.String("deployment"),
		&k8s.KubeDeploymentProps{
			Spec: &k8s.DeploymentSpec{
				Replicas: jsii.Number(1),
				Selector: selector,
				Template: &k8s.PodTemplateSpec{
					Metadata: labels,
					Spec: &k8s.PodSpec{
						Containers: &[]*k8s.Container{nginxContainer}}}}})

	return chart
}

func main() {
	app := cdk8s.NewApp(nil)
	NewNginxChart(app, "nginx", nil)
	app.Synth()
}
```

When writing `cdk8s` based code in any language, you will come across a set of common concepts/terminologies - these include `Construct`, `App` and `Chart`. I will explain these as we walk through the code. 

## Slight detour (code walk-through and concepts)

Start with the `main` function first - we use [cdk8s.NewApp](https://pkg.go.dev/github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2#NewApp) to create an [App](https://pkg.go.dev/github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2#App).

**Well, what exactly in an `App`?** It's is a [construct](https://github.com/aws/constructs), and you can think of `construct`s as higher-level building blocks to represent **state**. The key thing to note is that these constructs are **composable**. What that means is that you can define levels of these constructs (each level provides/exposes a different abstraction layer) and combine them to create your desired end state - in this case it happens to be a Kubernetes manifest with objects such as `Deployment`, but it could be something else. 

> For e.g. an [AWS CloudFormation template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) (if you were to use [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html), not be confused with `cdk8s`)

Back to the `App` - so, an `App` is also a construct. In fact you can think of it as the **root** in a tree (hierarchy) of constructs. So what else is there in that tree? Look the second line in the main function - `NewNginxChart(app, "getting-started", nil)` - this invokes a function  `NewNginxChart` that returns a [cdk8s.Chart](https://pkg.go.dev/github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2#Chart) which is the next component in the hierarchy. AA cdk8s `App` can contain **multiple** charts and each chart can be converted (or in precise cdk8s terminology - **synthesized**) into a separate Kubernetes manifest file (you will see this action very soon). 

Finally, draw your attention to the `NewNginxChart` function. It has a bunch of things, but notice the call to `k8s.NewKubeDeployment` function. This is where we actually define a Kubernetes `Deployment` in code (in the next section, we will also add a `Service` to the chart.) 

> You can define multiple Kubernetes components in a chart, such a `Pod`, `Service`, `Ingress`, `Job` etc. - what ever you need for your application to work on Kubernetes.

To summarise, here is a visual representation of what I just explained - remember everything is a `Construct` (`App`, `Chart` etc.)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/axuf2vbi328tormugbbc.png)

**Wait, what about the Kubernetes API dependencies??**

If you've spent time working on accessing Kubernetes programmatically, this is an obvious (and great!) question. if you were to deal with k8s object using go, at the minimum you will need Kubernetes [client-go](https://github.com/kubernetes/client-go/), [API machinery](https://github.com/kubernetes/apimachinery) etc. Guess what, `cdk8s` has got you covered there too!

You actually don't need to pull in these dependencies because `cdk8s` allows you to treat these Kubernetes API Objects as `constructs` - remember, everything is s construct! They are *automatically imported* to your project when you run the `cdk8s init` command, but you can do it explicitly using [cdk8s import](https://cdk8s.io/docs/latest/cli/import/) as well. The resulting API is available as part of the `imports` folder (yes, go ahead and check that again!). On the top of `main.go`, check the package that is imported - its just refers to the `imports` folder.

> There is more to `cdk8s import` though. But you will have to wait for other blog posts to see that in action - we are just getting started!

## Alright, lets get back on track...

.. and continue with the practical bits. It's time to generate some yaml - you can't eliminate it, but at least you don't have to write it by hand! To do so, simply run:

```bash
cdk8s synth
```

Once that completes (should be quick!), check the `dist` directory to check what `cdk8s` has generated.
To make it easier to understand, here is a diagram which has a one-to-one mapping (*notice the labels 1, 2,3, etc.?*) between the the `cdk8s` code objects/properties to their respective counterparts in yaml e.g. `spec.replicas`, `spec.selector`, `template.spec` etc.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a6dt1ck7a4n9ba9qj8xu.png)

You can now use good old `kubectl` to deploy this to Kubernetes since `cdk8s` is not going to do that for you, at least not yet ;)


```
kubectl apply -f dist/
kubectl get pods -w
```

Once the `Deployment` is ready, the `Pod` should be in `Running` state. Simply use `port-forward` to access the `nginx` container port locally:

```
kubectl port-forward <enter nginx pod name> 8080:80
```

To access `nginx` home page, navigate to `http://localhost:8080` using your browser

> You also use a CLI tool e.g. `curl localhost:8080`.

## That's not all!

Instead of port forwarding, let's use the standard Kubernetes way of accessing applications by defining a `Service` resource, which is typically defined like this:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
    - port: 9090
      targetPort: 8080
  selector:
    app: hello-nginx
  type: LoadBalancer
```

But you know the rule - no yaml writing by hand! So, in the `NewNginxChart` function in the `main.go` file, add this piece of code after the part you defined the `Deployment`:

```go
	k8s.NewKubeService(chart, jsii.String("service"), &k8s.KubeServiceProps{
		Spec: &k8s.ServiceSpec{
			Type:     jsii.String("LoadBalancer"),
			Ports:    &[]*k8s.ServicePort{{Port: jsii.Number(9090), TargetPort: k8s.IntOrString_FromNumber(jsii.Number(80))}},
			Selector: &map[string]*string{"app": jsii.String("hello-nginx")}}})
```

First, delete the existing `Deployment` - `kubectl delete -f dist/`. Then, run `cdk8s synth` again to create the new manifest in the `dist` folder.

> Both the `Service` and `Deployment` are in the same file - this is because they are part of the *same* Chart.

How you access the service will depend on the Kubernetes cluster. If you are using a cloud provider, it will likely provision a Load Balancer service native to that cloud e.g. [Application Load Balancer in AWS](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). Please adjust this as per your setup.

> For `minikube`, you can simply follow these instructions https://minikube.sigs.k8s.io/docs/handbook/accessing/#loadbalancer-access - *"Services of type LoadBalancer can be exposed via the `minikube tunnel` command."*

In a terminal, run this command (it runs as a separate process):

```bash
minikube tunnel
```

In another terminal, *delete* the existing `Deployment` and then apply the new manifest:

```bash
kubectl apply -f dist/
kubectl get pods -w
```

Check the `Service`:

```bash
kubectl get svc
```

To access the `nginx` server, navigate to the external IP (as per the `Service`). In the case of minikube, you can simply use `localhost:9090` or `127.0.0.0:9090`

> Remember to use port `9090` since that's the external port we specified in the Service configuration in our code

## Before wrapping up...

.. I want to call out a couple of other useful things in `cdk8s`.

**Reference and reuse existing manifests and Helm charts**

Say you have a `Service` already defined in a `service.yaml` file. You can [include](https://pkg.go.dev/github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2#NewInclude) it in your `cdk8s` as part of a larger application/chart that you may have. Here is an example:

```
cdk8s.NewInclude(chart, jsii.String("existing service"), &cdk8s.IncludeProps{Url: jsii.String("service.yaml")})
```

Similarly, you can also include `Helm` charts. Say you wanted to add `bitnami/nginx`:

```go
cdk8s.NewHelm(chart, jsii.String("bitnami nginx helm chart"), &cdk8s.HelmProps{
		Chart:  jsii.String("bitnami/nginx"),
		Values: &map[string]interface{}{"service.type": "ClusterIP"}})
```

> Well, you do need to have `helm` installed locally and also add the repo first `helm repo add bitnami https://charts.bitnami.com/bitnami` 

**Another handy feature is...**

... the ability to declare dependencies between any two `cdk8s` constructs. For instance, in the previous example, we had a `Deployment` and a `Service`. You could create a dependency as such:

```
deployment := k8s.NewKubeDeployment(...)
service := k8s.NewKubeService(...)

deployment.AddDependency(service)
```

Thanks to `AddDependency`, the resulting manifest will be such that the `Service` is placed before the `Deployment` object. 

> Dependency is *not* limited to individual constructs in a chart. If you have multiple charts as part of your `cdk8s` app, you can establish dependencies *across* charts as well.

## Conclusion

Awesome. So you were able to "code" your way through trouble and ignore YAML. Hope you enjoyed it! To keep things simple, I demonstrated a `Deployment` and `Service`, but you can choose from other Kubernetes components such as `Ingress`, `Job` etc. They are all exposed using a similar pattern i.e. NewKube<Resource> for e.g. `NewKubeJob`, `NewKubeIngress` etc. 

But there is still a lot of boilerplate code involved in defining Kubernetes components. Writing Go code sounds way better than YAML engineering (at least to me), it seems as if we are translating existing YAML into Go structs (and fields). In a subsequent blog post, we will explore how to improve this further.

Happy coding!