Write your Kubernetes Infrastructure as Go code - Combine cdk8s with AWS CDK
Tags: kubernetes, go, tutorial, devops
Published: 2022-11-16 06:11:45

In an [earlier blog post](https://dev.to/abhirockzz/write-your-kubernetes-infrastructure-as-go-code-manage-aws-services-1no8) you saw how to use [cdk8s](https://cdk8s.io/docs/latest/) with [AWS Controllers for Kubernetes](https://aws-controllers-k8s.github.io/community/docs/community/overview/) (also known as **ACK**), thanks to the fact that you can import existing Kubernetes Custom Resource Definitions using `cdk8s`! This made is possible to deploy `DynamoDB` along with a client application, from  using `cdk8s` and Kubernetes.

But, what if you continue using [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html) for AWS infrastructure and harness the power `cdk8s` (and `cdk8s-plus`!) to define Kubernetes resources using regular code? Thanks to the [native integration between
the AWS EKS module and cdk8s](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks-readme.html#cdk8s-charts), you can have the best of both worlds! 

The goal of this blog post is to demonstrate that with a few examples. We will start off with a simple (nginx based) example before moving on to a full-fledged application stack (including `DynamoDB` etc.). Both will be using the Go programming language which is well supported in [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-go.html) as well as [cdk8s](https://cdk8s.io/docs/latest/getting-started/#prerequisites).

> All the code discussed in this blog is available in [this GitHub repo](https://github.com/abhirockzz/cdk8s-for-go-developers/tree/master/part6-cdk-eks-cdk8s)

## Prerequisites

To follow along step-by-step, in addition to an AWS account, you will need following CLIs - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [cdk8s CLI](https://cdk8s.io/docs/latest/getting-started/#install-the-cli) and [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). Also, dont' forget to install [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install), the [Go programming language](https://go.dev/dl/) (v1.16 or above) as well as [Docker](https://docs.docker.com/engine/install/), if you don't have them already.


## Keeping it simple with Nginx on EKS

As with most things in life, there are two ways - the easy way or the hard way ;) You will see both of them! Let's try things out first, see them working and then look at the code.

To start off, clone the repo and change to the right directory:

```bash
git clone https://github.com/abhirockzz/cdk8s-for-go-developers
cd cdk8s-for-go-developers/part6-cdk-eks-cdk8s/cdk-cdk8s-nginx-eks
```

To setup everything, all you need is a single command:

```bash
cdk deploy
```

> you can also use `cdk synth` to generate and inspect the Cloud Formation template first

You will be prompted to confirm. Once you do that, the process will kick off - it will take some time since lots of AWS resources will be created, including VPC, EKS cluster etc.

> Feel free to check the AWS Cloud Formation console to track the progress.

Once the process is complete, you need to connect to the EKS cluster using `kubectl`. The command required for this will be available as a result of the `cdk deploy` process (in the terminal) or you can refer to the **Outputs** section of the AWS Cloud Formation stack.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kc9nqjnoy6wtxboweglv.png)

Once you've configured `kubectl` to point to your EKS cluster, you can check the Nginx `Deployment` and `Service`.

```bash
kubectl get deployment

# output

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment-cdk8s        1/1     1            1           1m
nginx-deployment-cdk          1/1     1            1           1m
```

You will see that two `Deployment`s have been created - more on this soon. Similiarly, if you check the `Service` (`kubectl get svc`), you should see two of them - `nginx-service-cdk` and `nginx-service-cdk8s`.

To access Nginx, pick the `EXTERNAL-IP` of any of the two `Service`s. For example:

```bash
APP_URL=$(kubectl get service/nginx-service-cdk -o jsonpath="{.status.loadBalancer.ingress[0].hostname}")

echo $APP_URL

# to access nginx (notice we are using port 9090)
curl -i http://$APP_URL:9090
```

> If you get a `Could not resolve host` error while accessing the LB URL, wait for a minute or so and re-try

### Behind the scenes

Let's [look at the code](https://github.com/abhirockzz/cdk8s-for-go-developers/blob/master/part6-cdk-eks-cdk8s/cdk-cdk8s-nginx-eks/main.go) now - this will clarify why we have two Nginx `Deployment`s.

Thanks to AWS CDK, VPC creation is a one liner with [awsec2.NewVpc](https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awsec2#NewVpc) function and creating an [EKS cluster](https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awseks#NewCluster) isn't too hard either!

```go
func NewNginxOnEKSStack(scope constructs.Construct, id string, props *CdkStackProps) awscdk.Stack {
  //...
  vpc := awsec2.NewVpc(stack, jsii.String("demo-vpc"), nil)

	eksSecurityGroup := awsec2.NewSecurityGroup(stack, jsii.String("eks-demo-sg"),
		&awsec2.SecurityGroupProps{
			Vpc:               vpc,
			SecurityGroupName: jsii.String("eks-demo-sg"),
			AllowAllOutbound:  jsii.Bool(true)})

	eksCluster := awseks.NewCluster(stack, jsii.String("demo-eks"),
		&awseks.ClusterProps{
			ClusterName:   jsii.String("demo-eks-cluster"),
			Version:       awseks.KubernetesVersion_V1_21(),
			Vpc:           vpc,
			SecurityGroup: eksSecurityGroup,
			VpcSubnets: &[]*awsec2.SubnetSelection{
				{Subnets: vpc.PrivateSubnets()}},
			DefaultCapacity:         jsii.Number(2),
			DefaultCapacityInstance: awsec2.InstanceType_Of(awsec2.InstanceClass_BURSTABLE3, awsec2.InstanceSize_SMALL), DefaultCapacityType: awseks.DefaultCapacityType_NODEGROUP,
			OutputConfigCommand: jsii.Bool(true),
			EndpointAccess:      awseks.EndpointAccess_PUBLIC()})
//...
```

## Nginx on Kubernetes, the hard way!

Now we look at two different ways of creating Nginx, starting with the "hard" way. In this case, we use AWS CDK (*not* `cdk8s`) to define the `Deployment` and `Service` resources.

```go
func deployNginxUsingCDK(eksCluster awseks.Cluster) {

	appLabel := map[string]*string{
		"app": jsii.String("nginx-eks-cdk"),
	}

	deployment := map[string]interface{}{
		"apiVersion": jsii.String("apps/v1"),
		"kind":       jsii.String("Deployment"),
		"metadata": map[string]*string{
			"name": jsii.String("nginx-deployment-cdk"),
		},
		"spec": map[string]interface{}{
			"replicas": jsii.Number(1),
			"selector": map[string]map[string]*string{
				"matchLabels": appLabel,
			},
			"template": map[string]interface{}{
				"metadata": map[string]map[string]*string{
					"labels": appLabel,
				},
				"spec": map[string][]map[string]interface{}{
					"containers": {
						{
							"name":  jsii.String("nginx"),
							"image": jsii.String("nginx"),
							"ports": []map[string]*float64{
								{
									"containerPort": jsii.Number(80),
								},
							},
						},
					},
				},
			},
		},
	}

	service := map[string]interface{}{
		"apiVersion": jsii.String("v1"),
		"kind":       jsii.String("Service"),
		"metadata": map[string]*string{
			"name": jsii.String("nginx-service-cdk"),
		},
		"spec": map[string]interface{}{
			"type": jsii.String("LoadBalancer"),
			"ports": []map[string]*float64{
				{
					"port":       jsii.Number(9090),
					"targetPort": jsii.Number(80),
				},
			},
			"selector": appLabel,
		},
	}

	eksCluster.AddManifest(jsii.String("app-deployment"), &service, &deployment)
}
```

Finally, to create this in EKS we invoke [AddManifest](https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awseks#Cluster.AddManifest) (think of its like the programmatic equivalent of `kubectl apply`). This works, but there are a few gaps in this approach:

- We are not able to reap the benefits of Go which is a *strongly typed* language. That's because the API is *loosely* typed, thanks to `map[string]interface{}` everywhere. This makes it very error prone (I made a few mistakes too!)
- The verbosity is apparent as well. It seems as if we are writing `YAML` in `Go` - not too much of an improvement!

### Is there a better way..?

Let's look at the second function `deployNginxUsingCDK8s` - by the name its obvious that we used `cdk8s`, not just CDK)

```go
func deployNginxUsingCDK8s(eksCluster awseks.Cluster) {

	app := cdk8s.NewApp(nil)
	eksCluster.AddCdk8sChart(jsii.String("nginx-eks-chart"), NewNginxChart(app, "nginx-cdk8s", nil), nil)
}
```

This looks "too easy" to to be true! But it's made possible due to the inter-operability between CDK and `cdk8s`. What this implies is that, you can use *define* Kubernetes resources using `cdk8s` `Chart`s and apply them to an EKS cluster created with CDK (this makes it sort of a hybrind system). 

The hero of our story is [AddCdk8sChart](https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awseks#Cluster.AddCdk8sChart) function, which accepts a [constructs.Construct](https://pkg.go.dev/github.com/aws/constructs-go/constructs/v10#Construct) (remember, [everything is a construct](https://dev.to/abhirockzz/write-your-kubernetes-infrastructure-as-go-code-getting-started-with-cdk8s-37n)!). In this case, the `Construct` happens to be a [cdk8s.Chart](https://pkg.go.dev/github.com/cdk8s-team/cdk8s-core-go/cdk8s/v2#Chart) thats returned by `NewNginxChart` function - so [lets take a look at that](https://github.com/abhirockzz/cdk8s-for-go-developers/blob/master/part6-cdk-eks-cdk8s/cdk-cdk8s-nginx-eks/nginx.go).

```go
func NewNginxChart(scope constructs.Construct, id string, props *MyChartProps) cdk8s.Chart {
  //....

	dep := cdk8splus22.NewDeployment(chart, jsii.String("nginx-deployment"),
		&cdk8splus22.DeploymentProps{
			Metadata: &cdk8s.ApiObjectMetadata{
				Name: jsii.String("nginx-deployment-cdk8s")}})

	dep.AddContainer(&cdk8splus22.ContainerProps{
		Name:  jsii.String("nginx-container"),
		Image: jsii.String("nginx"),
		Port:  jsii.Number(80)})

	dep.ExposeViaService(&cdk8splus22.DeploymentExposeViaServiceOptions{
		Name:        jsii.String("nginx-service-cdk8s"),
		ServiceType: cdk8splus22.ServiceType_LOAD_BALANCER,
		Ports: &[]*cdk8splus22.ServicePort{{
			Port:       jsii.Number(9090),
			TargetPort: jsii.Number(80)}}})

	return chart
}
```

If you have worked with `cdk8s` (and Go) or read some of my previous blogs on this topic, this should look familiar - a strongly-typed, compact and expressive API! I don't even need to walk you through this since its so readable - we use `cdk8s-plus` to create a Nginx `Deployment`, add the container info and finally expose it via a `Service` so that we can access the Nginx from outside of EKS. 

This was a simple enough example to help bring out difference between the two approaches. The next scenario is different - in addition to the EKS cluster it has DynamoDB along with a URL shortener application that will be deployed to EKS.

## End to end example: DynamoDB along with an application on EKS

Instead of creating a new EKS cluster from scratch, we will re-use the existing cluster created as a result of the previous example - this is a good opportunity to take a look at how you can reference an existing EKS cluster in your CDK code. As expected, we need to create `DynamoDB` table as well. 

Just like in the previous example, let's try out the solution first. Change into the right directory first:

```bash
cd part6-cdk-eks-cdk8s/cdk-cdk8s-dynamodb-app-eks
```

Since the URL shortener application has to make API calls to `DynamoDB`, we need to configure [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (also known as **IRSA**).

> Refer to https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html

**Define IAM roles for the application**

Start by creating a [Kubernetes Service Account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/):

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-dynamodb-app-sa
EOF
```

> To confirm - `kubectl get serviceaccount/eks-dynamodb-app-sa -o yaml`

Set your AWS Account ID and [OIDC Identity provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) as environment variables:

```bash
ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

export EKS_CLUSTER_NAME=<enter cluster name>
export AWS_REGION=<enter region e.g. us-east-1>

OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
```

Create a JSON file with *Trusted Entities* for the role:

```bash
read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-dynamodb-app-sa"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json
```

> Check - `cat trust.json`

Now, create the IAM role:

```bash
export ROLE_NAME=dynamodb-app-irsa

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for DynamoDB app on EKS"
```

We will need to create and attach policy to role since we only want to allow `PutItem` and `GetItem` operations from our application. [Here is the policy JSON file](https://github.com/abhirockzz/cdk8s-for-go-developers/blob/master/part6-cdk-eks-cdk8s/policy.json):

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PutandGet",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem"
            ],
            "Resource": "arn:aws:dynamodb:*:*:table/urls"
        }
    ]
}
```

Create and attach the policy to the role we just created:

```bash
aws iam create-policy --policy-name dynamodb-irsa-policy --policy-document file://policy.json

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::<enter AWS account ID>:policy/dynamodb-irsa-policy
```

Finally, we need to associate the IAM role and Service Account:

```bash
kubectl annotate serviceaccount -n default eks-dynamodb-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::<enter AWS account ID>:role/dynamodb-app-irsa
```

**Get the EKS `kubectl` role ARN**

To [reference existing EKS cluster](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks-readme.html#using-existing-clusters) in AWS CDK, you need EKS cluster name and `kubectl` role ARN.

You can find the role ARN in the **Outputs** section of the AWS Cloud Formation stack.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qr9kgedjvmyor0t1dhxr.png)

We are ready to deploy the application using CDK. Set the required environment variables followed by `cdk deploy`:

> you can also use `cdk synth` to generate and inspect the Cloud Formation template first

```bash
export EKS_CLUSTER_NAME=<enter name of EKS cluster>
export KUBECTL_ROLE_ARN=<enter kubectl role ARN>
export SERVICE_ACCOUNT_NAME=eks-dynamodb-app-sa
export APP_PORT=8080
export AWS_REGION=<enter region e.g. us-east-1>

cdk deploy
```

CDK (and `cdk8s`) will do all the heavy lifting (we will look at the code very soon):

- New DynamoDB table will be created
- The docker image for our application will be built and pushed to ECR
- Kubernetes resources for the URL shortener application will be deployed to the existing EKS cluster

Once the stack creation is complete, check the Kubernetes `Deployment` and `Service`:

```bash
kubectl get deployment/dynamodb-app
kubectl get pods
kubectl get service/dynamodb-app-service
```

Testing the URL shortener service is easy. But I will not repeat it here since its already covered in [this blog](https://dev.to/abhirockzz/write-your-kubernetes-infrastructure-as-go-code-manage-aws-services-3pgi. All you need is the load balancer URL to access the service and use yout browser or `curl` to save and access URLs.

**Back to exploring Go code again**

Within the stack, we define the `DynamoDB` table (using [awsdynamodb.NewTable](https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awsdynamodb#NewTable)) along with the docker image for our application (with [awsecrassets.NewDockerImageAsset](https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awsecrassets#NewDockerImageAsset))


```go
func NewDynamoDBAppStack(scope constructs.Construct, id string, props *CdkStackProps) awscdk.Stack {
  //...
  	table := awsdynamodb.NewTable(stack, jsii.String("dynamodb-table"),
		&awsdynamodb.TableProps{
			TableName: jsii.String(tableName),
			PartitionKey: &awsdynamodb.Attribute{
				Name: jsii.String(dynamoDBPartitionKey),
				Type: awsdynamodb.AttributeType_STRING,
			},
			BillingMode:   awsdynamodb.BillingMode_PAY_PER_REQUEST,
			RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
		})

    appDockerImage := awsecrassets.NewDockerImageAsset(stack, jsii.String("app-image"),
		&awsecrassets.DockerImageAssetProps{
			Directory: jsii.String(appDirectory)})
  //...
```

Then comes the interesting part where we get the reference to our exsiting EKS cluster and use `AddCdk8sChart` (just like before) to deploy the application to EKS.

```go
//...
  eksCluster := awseks.Cluster_FromClusterAttributes(stack, jsii.String("existing cluster"),
		&awseks.ClusterAttributes{
			ClusterName:    jsii.String(eksClusterName),
			KubectlRoleArn: jsii.String(kubectlRoleARN)})
      
	app := cdk8s.NewApp(nil)
	appProps := NewAppChartProps(appDockerImage.ImageUri(), table.TableName())

	eksCluster.AddCdk8sChart(jsii.String("dynamodbapp-chart"), NewDynamoDBAppChart(app, "dynamodb-cdk8s", &appProps), nil)
```

The [NewDynamoDBAppChart function](https://github.com/abhirockzz/cdk8s-for-go-developers/blob/master/part6-cdk-eks-cdk8s/cdk-cdk8s-dynamodb-app-eks/dynamodb_app.go#L59) defines the `Deployment` and `Service`. Unlike the earlier Nginx example which had static values, this application takes in dynamic values - specifically the `DynamoDB` table name (which is used as the container environment variable `TABLE_NAME`). Also notice the fact that we explicitly add the the name of the Kubernetes service account (for *IRSA*) that we had created in the previous step.

```go
func NewDynamoDBAppChart(scope constructs.Construct, id string, props *AppChartProps) cdk8s.Chart {
  //...
	dep := cdk8splus22.NewDeployment(chart, jsii.String("dynamodb-app-deployment"), &cdk8splus22.DeploymentProps{
		Metadata: &cdk8s.ApiObjectMetadata{
			Name: jsii.String("dynamodb-app")},
		ServiceAccount: cdk8splus22.ServiceAccount_FromServiceAccountName(
			chart,
			jsii.String("aws-irsa"),
			jsii.String(props.serviceAccountName))})

	container := dep.AddContainer(//.. omitted for brevity)

	container.Env().AddVariable(jsii.String("TABLE_NAME"), cdk8splus22.EnvValue_FromValue(props.tableName))
	container.Env().AddVariable(jsii.String("AWS_REGION"), cdk8splus22.EnvValue_FromValue(&props.region))

	dep.ExposeViaService(//.. omitted for brevity)
	return chart
}
```

## Wrap up

That's all for this blog! We started off with a simple example to showcase the integration between AWS CDK and `cdk8s` and how easy it makes things (compared to just using CDK to deploy apps to EKS). Then, we moved on to explore a full fledged scenario where you deployed the infrastructure (`DynamoDB` etc.) along with the client application on EKS.

Happy Building!