Scaling an AWS EKS with Karpenter using Helm Provider with Terraform. Kubernetes Series - Episode 4
Tags: eks, terraform, karpenter, aws
Published: 2024-01-17 19:30:29

Cloud people!

In the last Episode of this series we covered the steps to configure velero using helm charts within Kubernetes to backup or restore all objects in the cluster, or filter objects.

{% embed https://dev.to/segoja7/deploying-an-aws-eks-and-velero-backup-using-helm-provider-with-terraform-kubernetes-series-episode-3-286m %}

In this episode, the focus is on deploying Karpenter within Kubernetes to scale the cluster to meet demand.

## Requirements
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
* [Kubectl](https://kubernetes.io/docs/tasks/tools/)
* [Terraform](https://www.terraform.io/downloads.html)
* [k9s](https://k9scli.io/topics/install/)

Let's see how we can do this using terraform and a new submodule of eks to setup karpenter with terraform.

> Karpenter is a tool opensource developed in [AWS Labs](https://github.com/awslabs) that allow solved many challenges that  a traditional auto scaling not resolved, AWS EKS supports two products for auto scaling. Cluster Autoscaler and Karpenter. check this [link](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html#cluster-autoscaler) for more information.

The focus is on adding karpenter to the EKS cluster, considering that the EKS cluster was deployed.

This is the link for all code terraform in branch episode4.{% embed https://github.com/segoja7/EKS.git %} 

## Step 1.


> It is necessary to add providers to connect and create resources within the eks cluster with helm and kubectl, the kubectl provider is used for deploying manifests and testing.

```
terraform {
  required_version = ">= 1.4.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.9"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.2"
    }
  }
}

provider "aws" {
  region  = local.region
  profile = local.profile
}


provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    # This requires the awscli to be installed locally where Terraform is executed
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name, "--profile", local.profile]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      # This requires the awscli to be installed locally where Terraform is executed
      args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name, "--profile", local.profile]
    }
  }
}

provider "kubectl" {
  apply_retry_count      = 5
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  load_config_file       = false

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    # This requires the awscli to be installed locally where Terraform is executed
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name, "--region", "us-east-1", "--profile", local.profile]
  }
}
```

## Step 2.
It is necessary, to add tags in the subnets and security groups for that karpenter can be work, these tags are used later.

```
  private_subnet_tags = {
    "karpenter.sh/discovery"              = local.name
  }
```

```
  tags = merge(local.tags, {
    # NOTE - if creating multiple security groups with this module, only tag the
    # security group that Karpenter should utilize with the following tag
    # (i.e. - at most, only one security group should have this tag in your account)
    "karpenter.sh/discovery" = "${local.name}"
  })
```

## Step 3.

This module configures all the necessary for using karpenter. 
This scenario is based in this example from [registry](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest/submodules/karpenter#:~:text=In%20this%20scenario%2C%20Karpenter%20would%20run%20atop%20the%20EKS%20Managed%20Node%20group%20and%20scale%20out%20nodes%20as%20needed%20from%20there%3A)

```
module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "19.20.0"

  cluster_name                    = module.eks.cluster_name
  irsa_oidc_provider_arn          = module.eks.oidc_provider_arn
  irsa_namespace_service_accounts = ["karpenter:karpenter"]

  create_iam_role      = false
  iam_role_arn         = module.eks.eks_managed_node_groups["cloud-people"].iam_role_arn
  irsa_use_name_prefix = false

  tags = local.tags
}
```

![karpenterIRSArole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vsn3why5r2g4e9eqyv3p.gif)

## Step 4.

In this step is to add karpenter using generic helm releases.

```
resource "helm_release" "karpenter" {
  namespace        = "karpenter"
  create_namespace = true

  name                = "karpenter"
  repository          = "oci://public.ecr.aws/karpenter"
  repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart               = "karpenter"
  version             = "v0.31.3"

  set {
    name  = "settings.aws.clusterName"
    value = module.eks.cluster_name
  }

  set {
    name  = "settings.aws.clusterEndpoint"
    value = module.eks.cluster_endpoint
  }

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.karpenter.irsa_arn
  }

  set {
    name  = "settings.aws.defaultInstanceProfile"
    value = module.karpenter.instance_profile_name
  }

  set {
    name  = "settings.aws.interruptionQueueName"
    value = module.karpenter.queue_name
  }
}
```
## Step 5.

With karpenter installed, only need to make test with Autoscaling Karpenter, but first is necessary to add a definition of personalized resources (CRD) check this [link](https://karpenter.sh/v0.31/concepts/provisioners/) for more information.

With this CRD it is possible to add various configurations based on taints, availability zones, instances type, processor architecture and others.

```
resource "kubectl_manifest" "karpenter_provisioner" {
  yaml_body = <<-YAML
    apiVersion: karpenter.sh/v1alpha5
    kind: Provisioner
    metadata:
      name: default
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"] #"spot"
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["c5.large","c5a.large", "c5ad.large", "c5d.large", "c6i.large", "t2.medium", "t3.medium", "t3a.medium"]
      limits:
        resources:
          cpu: 1000
      providerRef:
        name: default
      ttlSecondsAfterEmpty: 30
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}
```
> Note: When a CRD provisioner has not specified the processor architecture or any instance type for the new nodes, by default karpenter will use all instances and all architectures. This means that a node is created for your application and errors may occur due to incompatibilities with the processor architecture.


## Step 6.

Node templates allow you to configure AWS specific parameters(NodeTemplate) check this [link](https://karpenter.sh/v0.31/concepts/node-templates/) for more information.
> In this step karpenter use the tags created in step2.

```
resource "kubectl_manifest" "karpenter_node_template" {
  yaml_body = <<-YAML
    apiVersion: karpenter.k8s.aws/v1alpha1
    kind: AWSNodeTemplate
    metadata:
      name: default
    spec:
      subnetSelector:
        karpenter.sh/discovery: ${module.eks.cluster_name}
      securityGroupSelector:
        karpenter.sh/discovery: ${module.eks.cluster_name}
      tags:
        karpenter.sh/discovery: ${module.eks.cluster_name}
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}
```

## Step 7.

This step is created a sample pause deployment to demonstrate scaling.

```
# Example deployment using the [pause image](https://www.ianlewis.org/en/almighty-pause-container)
# and starts with zero replicas
resource "kubectl_manifest" "karpenter_example_deployment" {
  yaml_body = <<-YAML
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: inflate
    spec:
      replicas: 0
      selector:
        matchLabels:
          app: inflate
      template:
        metadata:
          labels:
            app: inflate
        spec:
          terminationGracePeriodSeconds: 0
          containers:
            - name: inflate
              image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
              resources:
                requests:
                  cpu: 1
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}
```

## Step 8.
Scale up the sample pause deployment to see Karpenter respond by provisioning nodes to support the workload.

```
kubectl scale deployment inflate --replicas 50
```
To view logs

```
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
```

![Scaling nodes karpenter](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xzz2vq71xj6x4n0iy55d.gif)

## Step 9.
Clean up nodes.

```
 kubectl scale deployment inflate --replicas 0
```

![Clean up nodes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j98wk18opozlyt3q3yv0.gif)

Conclusion, Karpenter is tool for the lifecyle of the node, additional is a great option to handle costs using the best options based in pricing.

If you have any questions, please leave them in the comments!

Successful!!