Kubernetes Cost Monitoring With Kubecost
Tags: kubernetes, vcluster, devspace, loft
Published: 2022-06-16 16:38:10

By Aniket Bhattacharyea

Kubernetes has taken the software development world by storm. It gives you an excellent framework to deploy your application with and abstracts away the low-level details of the underlying infrastructure. But just like everything great, it comes with a tradeoff.

Since it makes deployments easy and smooth, it's possible to overprovision resources that can end up driving cloud costs up. In my previous company, we used [Amazon EKS](https://aws.amazon.com/eks/) to host our infrastructure. Our cost used to be around $1000 per month—much higher than we had anticipated based on our actual production utilization. The culprit turned out to be unutilized clusters. We created some clusters for testing and experimentation, and unlike the production workload that ran all the time, these clusters were used sparingly. Since these were not deleted, they kept accumulating costs even though they were unused most of the time.

Due to the containerized nature of Kubernetes, the traditional methods of estimating cost and allocating resources [don't work](https://cast.ai/blog/kubernetes-cost-estimation-4-problems-and-how-to-solve-them). You need detailed insight into how your resources are being utilized throughout the various components of Kubernetes in order to make decisions.

[Kubecost](https://www.kubecost.com/) is a tool that helps you with cost-optimization and estimation for Kubernetes. It gives you real-time visibility into your resource utilization and provides you with a detailed breakdown of your costs. Kubecost also lets you assign out-of-cluster costs (eg, database or storage costs) to get complete insight across the entire spectrum of your cloud costs. With the help of Kubecost, you can also set up notifications to quickly catch cost spikes and take action accordingly.

In this article, you will learn how to install Kubecost in a Kubernetes cluster. You'll also get introduced to the sleep mode by [Loft Labs](https://loft.sh/), which can help you take action against high costs incurred by unused resources.

## Installing Kubecost

In order to install Kubecost, you need the following prerequisites:

1. A Kubernetes cluster. You can use any cloud-based cluster like [EKS](https://aws.amazon.com/eks/) or [GKE](https://cloud.google.com/kubernetes-engine/), or you can use a local cluster with [Minikube](https://minikube.sigs.k8s.io/docs/).
2. [Kubectl](https://kubernetes.io/docs/reference/kubectl/kubectl/) installed and configured for your cluster.
3. Helm 3.0+ [installed and configured](https://helm.sh/docs/intro/install/).

Use the following command to create a `kubecost` namespace and install Prometheus, Grafana, and kube-state-metrics in the namespace. You can further customize the installation with additional configuration as described [here](https://github.com/kubecost/cost-analyzer-helm-chart/blob/master/README.md#config-options).

```shell
kubectl create namespace kubecost
helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm install kubecost kubecost/cost-analyzer --namespace kubecost --set kubecostToken="YW5pa2V0QGFiaGF0dGFjaGFyeWVhLmRldg==xm343yadf98"
```

Next, run `kubectl get pods -n kubecost` and ensure all the pods are in the `Running` state.

![The pods must be in running state](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-1.png)

Once all the pods are running, run the following command to enable port-forwarding:

```shell
kubectl port-forward --namespace kubecost deployment/kubecost-cost-analyzer 9090
```

Note that if you’re running Kubecost on Minikube, you need to perform some additional steps:

1. Run `kubectl edit cm nginx-conf -n kubecost`, which will open the NGINX ConfigMap in your editor.
2. Search for **kubecost-cost-analyzer.kubecost:9001** and **kubecost-cost-analyzer.kubecost:9003** and change them to **localhost:9001** and **localhost:9003** respectively. Save the file and close the editor.
3. Restart the `kubecost-cost-analyzer` pod. There are many ways to do it, but the most straightforward is running the command `kubectl rollout restart deployment kubecost-cost-analyzer -n kubecost`.

Now you can visit `localhost:9090` in your browser, and you should see the following Kubecost screen.

![Screenshot of Kubecost welcome screen](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-2.png)

In case it shows *No available clusters* or throws some other error during installation, you can consult the [troubleshooting guide](https://docs.kubecost.com/troubleshoot-install.html).

You can click on the available cluster, which should take you to the dashboard where you can see various information about your cluster at a glance.

![Screenshot of Kubecost dashboard](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-3.png)

## Using Kubecost

Once Kubecost has been installed, you can start by looking at the cost allocations. The **Cost Allocation** page shows you a breakdown of your expenses. By default, the breakdown is grouped by namespaces. For each namespace, you see the cost incurred across various aspects like CPU, GPU, Memory, and PV.

![Screenshot of Cost Allocation page](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-4.png)

You can use the **Aggregate By** dropdown to see the breakdown by various Kubernetes concepts like clusters, containers, deployments,and pods. The breakdown is available by other non-Kubernetes organizational concepts like Team, Department, and Product. These aggregations are based on [Kubernetes labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) referenced at the pod or namespace level.

![Screenshot of Aggregate By dropdown](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-5.png)

Using the **Date Range** dropdown, you can change the date range of the report. You can use some common ranges like **Last 7 days**, **Today**, **Yesterday**, and **Last 30 days**, or you can enter a custom start and end date.

![Screenshot of Date Range dropdown](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-6.png)

Click on the options menu to modify some additional options. You can change the chart type to show cost over time or proportional cost. The cost metric can be changed to daily, hourly, or monthly rate, as well as cumulative cost over the date range.

You can also apply filters to fine-tune the report. Documentation for these options can be found [here](https://docs.kubecost.com/cost-allocation.html#additional-options).

![Screenshot of the additional options](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-7.png)

Finally, you can use the Save, Load, and Download buttons to save the report settings, load saved reports, or download the report in CSV format.

The **Savings** page lists some recommended actions that can be taken to save some cost. These actions include making reserved instance commitments, managing unclaimed volumes, managing pods with over-provisioned requests, and managing local disks with low utilization. Click on each recommendation to learn more about them.

![Screenshot of the Savings page](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-8.png)

On the **Health** page, you can see the health score of your cluster, which is an assessment of infrastructure reliability and performance. Kubecost performs a few health tests like monitoring for high CPU and memory utilization, checking for crash looping pods, looking for failing jobs, network issues, and CPU throttling. These health checks ensure your cluster is running smoothly.

![Screenshot of the health page](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-9.png)

Set up Slack or email alerts by visiting the **Notifications** page. You can set a daily threshold at the cluster level or for individual namespaces and get alerts when the cost exceeds the threshold. Optionally, set it up to send you weekly updates and cluster health updates.

![Screenshot of the Notifications page](https://loft.sh/blog/images/content/cost-monitoring-with-kubecost-10.png)

## Using Loft's Sleep Mode to Save Cost

While Kubecost is an excellent tool for gaining insight into how resources are being utilized in your Kubernetes cluster, there’s still a manual aspect to it. Kubecost only provides insights; the cluster admins must take manual action based on the reports. This is where [Loft's sleep mode](https://loft.sh/docs/self-service/sleep-mode) comes in.

As mentioned before, one of the biggest reasons for a high cost is idle resources. Loft's sleep mode takes care of this by putting [spaces](https://loft.sh/docs/self-service/namespaces) or [virtual clusters](https://loft.sh/docs/self-service/virtual-clusters) to sleep after a specified period of inactivity. They automatically wake up when a kubectl, Helm, or any other command involving the space or virtual cluster is executed.

Loft does this by storing the replica number for Deployments, StatefulSets, DaemonSets, and other ReplicaSet-based resources and scales down the replica number to 0, causing Kubernetes to delete all pods and containers. When the first activity is performed in the space or virtual cluster, Loft restores this replica number, essentially restarting the space/virtual cluster. 

This "first activity" that wakes up the space can be any interaction with the space, either via kubectl, Helm, or any other tool. It's also possible to [manually wake up](https://loft.sh/docs/self-service/sleep-mode#manual-wake-up) spaces.

Loft's sleep mode grants you the ability to [enforce automatic sleep for all spaces](https://loft.sh/docs/self-service/sleep-mode#automatic-for-entire-account) or [individual spaces](https://loft.sh/docs/self-service/sleep-mode#automatic-for-individual-spaces) after a specified period of inactivity. It's also possible to [manually trigger](https://loft.sh/docs/self-service/sleep-mode#manual-sleep) sleep mode for individual spaces. You can even [exclude](https://loft.sh/docs/self-service/sleep-mode#how-can-i-prevent-certain-resources-within-a-space-from-sleeping) resources from being put into sleep for more fine-grained control.

## Conclusion

While Kubernetes is an excellent tool for managing containers, it can cost a fortune if extra thought is not given to resource management. With Kubecost at your disposal, you can gain a real-time overview of your resource usage and catch cost overruns early. And using Loft's sleep mode, you can automatically shut down unused resources to save money. With the combination of Kubecost and Loft's sleep mode, you can have a highly performant cluster with minimal cost.

If you’re interested in creating a Kubernetes-based architecture that’s easy to use for your developers, DevOps, IT operations, and sales teams, take a look at [Loft](https://loft.sh/). Loft provides a platform for Kubernetes self-service and multi-tenancy that’s highly flexible and performant.

*Photo by [Pepi Stojanovski](https://unsplash.com/@timbatec?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/money?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)*