Running Stormforge with Karpenter and Kubecost
Tags: cloud, programming, devops, kubernetes
Published: 2024-03-11 11:03:51

As with all environments, including Dev, you want to ensure that whatever is running on said environment is operating at the utmost performance. Not only will it make the application stack more reliable, but it’s less calls at 2:00 AM for the engineers on-call and less support tickets from angry customers using your application.

In this blog post, you’ll learn about the three key ways to optimize performance across all fronts with Karpenter, Kubecost, and Stormforge.

## Prerequisites

To follow along from a hands-on perspective with this blog post, you’ll need to sign up for the 30-day Stormforge trial. You can find a link to it [here](https://app.stormforge.io/signup).

## The Benefits Of All Three

When you think about performance optimization, you may be thinking “Why do I need three tools?”, and you’re absolutely right. Technically, you don’t. You could use Stormforge for all three needs (Stormforge can work directly with the Cluster Autoscaler). However, when implementing tools and platforms, the key is to ensure that what you’re implementing is the best for the job. That’s why implementing open-source tools like Kubecost and Karpenter to add value makes sense.

Let’s break down what each is for.

First, there’s Karpenter. Karpenter will be used for scaling nodes up and down in the most performant way possible. If you’re wondering why you wouldn’t just use Cluster Autoscaler because it’s built in, you absolutely can, but it’s slower. From a performance benchmark perspective, Karpenter wins in terms of getting nodes up and bringing them down faster.

Next, there's Stormforge. Stormforge will take care of the resource optimization for resources running on the cluster. It’ll manage the memory and CPU needs of the workloads. It can take care of all Requests and limits. Karpenter takes care of the infrastructure and Stormforge takes care of the workloads.

Lastly, there’s Kubecost. Kubecost takes care of… you guessed it, the cost. It ensures that cost optimization is properly implemented within your Kubernetes environment for both the cluster itself and the the workloads running within the cluster. It’ll give you suggestions on what can be optimized to save money and still perform as well as before.

## Karpenter Configuration

The overall installation and configuration for Karpenter takes three key steps:

1. Ensuring the proper IAM permissions.
2. The installation of Karpenter itself.
3. The configuration of Karpenter.

In this section, you’re going to learn how to set up all three to ensure Karpenter is properly running within your environment.

### AWS Role

First things first - Karpenter needs permissions to scale up and scale down Worker Nodes. It doesn’t have the permission needed out of the box, so during the installation, you’ll point Karpenter to the proper IAM Role.

First, create a policy that uses the same config in the `policy.json` below.

```jsx
{

"Statement": [
    {
        "Action": [
            "ssm:GetParameter",
            "iam:PassRole",
            "ec2:DescribeImages",
            "ec2:RunInstances",
            "ec2:DescribeSubnets",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeLaunchTemplates",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceTypes",
            "ec2:DescribeInstanceTypeOfferings",
            "ec2:DescribeAvailabilityZones",
            "ec2:DeleteLaunchTemplate",
            "ec2:CreateTags",
            "ec2:CreateLaunchTemplate",
            "ec2:CreateFleet",
            "ec2:DescribeSpotPriceHistory",
            "pricing:GetProducts"
        ],
        "Effect": "Allow",
        "Resource": "*",
        "Sid": "Karpenter"
    },
    {
        "Action": "ec2:TerminateInstances",
        "Condition": {
            "StringLike": {
                "ec2:ResourceTag/Name": "*karpenter*"
            }
        },
        "Effect": "Allow",
        "Resource": "*",
        "Sid": "ConditionalEC2Termination"
    }
],
"Version": "2012-10-17"
}
```

This policy will ensure that Karpenter has the right permissions.

Next, create the policy.

```jsx
aws iam create-policy --policy-name KarpenterControllerPolicy --policy-document file://policy.json
```

After the policy is created, ensure that the OIDC provider is active and ready to use AWS IAM.

```jsx
eksctl utils associate-iam-oidc-provider --cluster k8squickstart-cluster --approve
```

Lastly, add the service account with the policy to your cluster.

```jsx
eksctl create iamserviceaccount \
  --cluster "k8squickstart-cluster" --name karpenter --namespace karpenter \
  --role-name "KarpenterInstanceNodeRole" \
  --attach-policy-arn "arn:aws:iam::912101370089:policy/KarpenterControllerPolicy" \
  --role-only \
  --approve
```

### Installation

Now that the policy is ready, you can start the installation process for Karpenter.

First, configure a few environment various which contain:

1. Your clusters name.
2. The IAM role.
3. The cluster endpoint

An example is below.

```jsx
export CLUSTER_NAME="k8squickstart-cluster"
export KARPENTER_IAM_ROLE_ARN=
export CLUSTER_ENDPOINT=
```

Next, install Karpenter with the associated environment variables.

```jsx
helm upgrade --install --namespace karpenter --create-namespace \
karpenter karpenter/karpenter \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
--set clusterName=${CLUSTER_NAME} \
--set clusterEndpoint=${CLUSTER_ENDPOINT} \
--set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME}
```

Lastly, install the Karpenter CRD’s.

```jsx
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/main/pkg/apis/crds/karpenter.sh_nodepools.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/main/pkg/apis/crds/karpenter.sh_nodeclaims.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/main/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml
```

You’re now ready for the Karpenter configuration.

### Configuration

Once Karpenter is installed, you’ll then need to configure it for the Worker Nodes that you want it to manage as Karpenter doesn’t do this out of the box.

As an example, the configuration below sets up Karpenter to manage Linux-based nodes that are of instance type A or M.

Ensure that you specify the IAM Role name that you created for the config below.

```jsx
kubectl apply -f - <<EOF
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["a", "m"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h # 30 * 24h = 720h
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-k8squickstart-eks"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "k8squickstart-eks"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "k8squickstart-eks"
EOF
```

You can then test out the configuration to ensure that proper scaling takes place.

Deploy a test workload.

```jsx
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
EOF
```

Scale the workload.

```jsx
kubectl scale deployment inflate --replicas 5
```

Check to ensure that Karpenter is working as expected.

```jsx
kubectl logs -f -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller
```

## Stormforge Configuration

Now that the Karpenter configuration is complete, you’ll implement the workload optimization piece, Stormforge. In the **prerequisite** section of this blog post, you went through the necessary prerequisites to sign up for Stormforge.

Once you’re signed in, you’ll see a screenshot similar to the one below.

Click the green **+ Add Cluster** button.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gql3jfbg88ki600wz3r5.png)



Next, go through the steps of putting in information about your cluster.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sfw6u10dzatbqqgvzao3.png)



You’ll reach a point where you need to download a `values.yaml` file which will contain the credentials Stormforge needs to have permissions to your Kubernetes cluster. Save the output in a `values.yaml` file.

```jsx
clusterName: karpentertest
stormforge:
  address: https://api.stormforge.io/
authorization:
  issuer: https://api.stormforge.io/
  clientID: ef0a6fd
  clientSecret: AwTG
```

The last step is to run the Stormforge Agent installation.

```markdown
helm install stormforge-agent oci://registry.stormforge.io/library/stormforge-agent \
  --namespace stormforge-system \
  --create-namespace \
  --values values.yaml
```

## Kubecost Configuration

When implementing Kubecost, at first, you’ll most likely google “How do I install Kubecost?”. You’ll see several articles that show the installation, but the thing is it shows the installation with a Kubecost product key.

To get that product key, you have to contact Kubecost.

If you don’t want to do that, you can instead just install the 100% open-source version.

To install the open-source version, run the following Helm Chart.

```jsx
helm upgrade --install kubecost \                            
  --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
  --namespace kubecost --create-namespace
```

You can ensure the resources are properly running with the command below.

```jsx
kubectl get all -n kubecost
```

## Wrapping Up

Consolidating the tools you use into one usable and manageable stack is key for any implementation, and when it comes to overall performance optimization, this three-prong stack makes the most sense to ensure that the application stack you’re running is as efficient as possible.