Karpenter: its monitoring, and Grafana dashboard for Kubernetes WorkerNodes
Tags: aws, devops, monitoring, grafana
Published: 2024-02-24 12:57:19

---
title: Karpenter: its monitoring, and Grafana dashboard for Kubernetes WorkerNodes
published: true
date: 2024-02-18 14:00:07 UTC
tags: aws,devops,monitoring,grafana
canonical_url: https://rtfm.co.ua/en/karpenter-its-monitoring-and-grafana-dashboard-for-kubernetes-workernodes/
---

Karpenter — monitoring: its metrics and creating a Grafana dashboard for Kubernetes WorkerNodes with examples of PromQL queries

![](https://cdn-images-1.medium.com/max/1024/1*OX1wkc5U5n27ZuXsQQ6m5g.png)

We have an AWS Elastic Kubernetes Service cluster with Karpenter which is responsible for EC2 auto-scaling, see [AWS: Getting started with Karpenter for autoscaling in EKS, and its installation with Helm](https://rtfm.co.ua/en/aws-getting-started-with-karpenter-for-autoscaling-in-eks-and-its-installation-with-helm/).

In general, there are no problems with it so far, but in any case we need to monitor it. For its monitoring, Karpenter provides metrics out of the box that we can use in Grafana and Prometheus/VictoriaMetrics alerts.

So, what we will do today:

- add Karpenter metrics collection to Prometheus/VictoriaMetrics
- will see which metrics can be useful to us
- will create a Grafana Dashboard for WorkerNodes + Karpenter

In general, this post is more about Grafana than Karpenter, but the in Grafana graphs we will mainly use metrics from Karpenter.

We will also need to create alerts, but that’s for another time. Having an idea of the available Karpenter metrics and Prometheus queries for graphs in Grafana, you shouldn’t have any problems with alerts.

Let’s go.

### Collecting Karpenter metrics with VictoriaMetrics VMAgent

Our VictoriaMetrics has its own chart, see [VictoriaMetrics: deploying a Kubernetes monitoring stack](https://rtfm.co.ua/en/victoriametrics-deploying-a-kubernetes-monitoring-stack/).

To add a new `target`, update the values of this chart, and add the endpoint `karpenter.karpenter.svc.cluster.local:8000`:

```
...
  vmagent:
    enabled: true
    spec: 
      replicaCount: 1
      inlineScrapeConfig: |
        - job_name: yace-exporter
          metrics_path: /metrics
          static_configs:
            - targets: ["yace-service:5000"]
        - job_name: github-exporter
          metrics_path: /
          static_configs:
            - targets: ["github-exporter-service:8000"]     
        - job_name: karpenter
          metrics_path: /metrics
          static_configs:
            - targets: ["karpenter.karpenter.svc.cluster.local:8000"]  
...
```

Deploy it and open a port to VMAgent to check the target:

```
$ kk port-forward svc/vmagent-vm-k8s-stack 8429
```

Check the targets:

![](https://cdn-images-1.medium.com/max/1024/0*YhmizLDi4v27MU2U.png)

To check the metrics, open a port to VMSingle:

```
$ kk port-forward svc/vmsingle-vm-k8s-stack 8429
```

And look for data by the `{job="karpenter"}` query:

![](https://cdn-images-1.medium.com/max/1024/0*viH93FgK_h-t5hqr.png)

### Useful Karpenter metrics

Now let’s see which metrics can be useful to us.

But before we get into the metrics, let’s talk about the basic concepts in Karpenter (besides the obvious ones like Node, NodeClaim, or Pod):

- **controller** : a Karpenter component that is responsible for a certain aspect of its operation, for example, the Pricing controller is responsible for checking the cost of instances, and the Disruption Controller is responsible for managing the process of changing the state of WorkerNodes
- **reconciliation** : the process when Karpenter reconciles the desired state and the current state, for example — when a Pod appears where there are no free resources on existing WorkerNodes, Karpenter will create a new Node on which the Pod can run, and its status will become Running — then the _reconciliation_ process of the status of this Pod will be completed
- **consistency** : internal control process and ensuring compliance with the required parameters (for example, checking that the created WorkerNode has a disk of exactly 30 GB)
- **disruption** : the process of changing WorkerNodes in the cluster, for example, recreating a WorkerNode (to replace it with an instance with more CPU or Memory), or deleting an existing node that has no running Pods
- **interruption** : cases when an EC2 instance is stopped due to hardware errors, instance termination (when you do a Stop or Terminate instance), or in the case of Spot, when AWS “takes back” that instance (actually, its resources like CPU and Mem); these events go to a corresponding SQS, from where Karpenter receives them to launch a new instance to replace it
- **provisioner** : a component that analyzes the current needs of the cluster, such as requests for creating new Pods, determines which resources need to be created (WorkerNodes), and initiates the creation of new ones _(in general,_ [_Provisioner has been replaced by NodePool_](https://rtfm.co.ua/en/karpenter-the-beta-version-an-overview-of-changes-and-upgrade-from-v0-30-0-to-v0-32-1/)_, but some metrics for it remain_)

Here I’ve collected only the metrics that I found most useful at the moment, but you should check out the [Inspect Karpenter Metrics](https://karpenter.sh/docs/reference/metrics/) documentation for yourself and there’s a bit more detail in the [Datadog documentation](https://docs.datadoghq.com/integrations/karpenter/):

**Controller** :

- `controller_runtime_reconcile_errors_total`: the number of errors when updating WorkerNodes (i.e. in the [Disruption Controller](https://karpenter.sh/v0.32/concepts/disruption/#disruption-controller) when performing Expiration, Drift, Interruption and Consolidation operations) - it is useful to have a graph in Grafana or an alert
- `controller_runtime_reconcile_total`: the total number of such operations - it is useful to have an idea of Karpenter activity and possibly have an alert if it happens too often

**Consistency** :

- `karpenter_consistency_errors`: looks like a useful metric, but I have it empty (at least for now)

**Disruption** :

- `karpenter_disruption_actions_performed_total`: the total number of disruption actions (deleting/recreating WorkerNodes), the metric labels indicate a _disruption method_ - it is useful to have an idea of Karpenter activity and possibly have an alert if it happens too often
- `karpenter_disruption_eligible_nodes`: the total number of WorkerNodes to perform disruption (deletion/recreation of WorkerNodes), the metric labels indicate _disruption method_
- `karpenter_disruption_replacement_nodeclaim_failures_total`: the total number of errors when creating new WorkerNodes to replace the old ones, the metric labels indicate _disruption method_

**Interruption** :

- `karpenter_interruption_actions_performed`: the number of actions on [EC2 Interruption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) notifications (from SQS) - maybe it makes sense, but I did not see this during the week of collecting metrics

**Nodeclaims** :

- `karpenter_nodeclaims_created`: the total number of created NodeClaims with labels by reason of creation and the corresponding NodePool
- `karpenter_nodeclaims_terminated`: similarly, but for deleted NodeClaims

**Provisioner** :

- `karpenter_provisioner_scheduling_duration_seconds`: it may make sense to monitor it, because if it grows but is too high, it may be a sign of problems; however, I have `karpenter_provisioner_scheduling_duration_seconds_bucket` histogram unchanged

**Nodepool** :

- `karpenter_nodepool_limit`: NodePool CPU/Memory limit set in its Provider ([`spec.limits`](https://karpenter.sh/docs/concepts/nodepools/#speclimits))
- `karpenter_nodepool_usage`: NodePool resource utilization - CPU, Memory, Volumes, Pods

**Nodes** :

- `karpenter_nodes_allocatable`: information on existing WorkerNodes - types, amount of CPU/Memory, Spot/On-Demand, Availability Zone, etc.
- you can have a graph by the number of Spot/On-Demand instances
- can be used to obtain data on available CPU/memory resources — `sum(karpenter_nodes_allocatable) by (resource_type)`
- `karpenter_nodes_created`: total number of Nodes created
- `karpenter_nodes_terminated`: total number of deleted Nodes
- `karpenter_nodes_total_pod_limits`: the total number of all Pod Limits (except DaemonSet) on each WorkerNode
- `karpenter_nodes_total_pod_requests`: total number Pod Requests (except DaemonSet) on every WorkerNode

**Pods** :

- `karpenter_pods_startup_time_seconds`: the time from the creation of a Pod until to its transition to the Running state (the sum of all tax returns)
- `karpenter_pods_state`: is a pretty useful metric because it has the status of a Pod in the labels, on which Node it was launched on, its namespace, etc.

**Cloudprovider** :

- `karpenter_cloudprovider_errors_total`: number of errors from AWS
- `karpenter_cloudprovider_instance_type_price_estimate`: the cost of instances by type - you can display the cost of cluster compute power on the dashboard

### Creating Grafana dashboard

There is a ready-made dashboard for Grafana — [Display all useful Karpenter metrics](https://grafana.com/grafana/dashboards/18862-karpenter/), but it is not informative at all. However, you can get some graphs and/or queries from it.

Now I have my own dashboard to check the status and resources for each individual WorkerNode:

![](https://cdn-images-1.medium.com/max/1024/0*6ONIXKbh4cgSii6D.png)

In the graphs of this board, I have [Data Links](https://grafana.com/docs/grafana/latest/panels-visualizations/configure-data-links/) to dashboard with details on a specific Pod:

![](https://cdn-images-1.medium.com/max/1024/0*imUMq2VtvLynGbyE.png)

The ALB graphs below are built from logs in Loki.

So what we’ll do is create a new board with all the WorkerNodes on it, and with the Data Links on the graphs of this board, we’ll link to the first dashboard.

Then there will be good navigation:

1. a general board for all WorkerNodes with the ability to go to a dashboard with more detailed information on a specific Node
2. on the board for a particular Node there will already be information about Pods on this Node, and data links to the board for a particular Pod

### The dashboard planning

Let’s think about what we would like to see on the new board.

Filters/variables:

- be able to see all WorkerNodes together, or select one or more separately
- to be able to see resources by specific Namespaces or Applications (in my case, each service has its own namespace, so we will use them)

Next, information on Nodes:

- general information on nodes:  
- number of Nodes  
- number of Pods  
- number of CPUs  
- number of Mem  
- spot vs on-demand ratio  
- cost of all Nodes per day
- percentage of allocatable used:  
- cpu — from Pods requested  
- mem — from Pods requested  
- Pods allocation
- real resources utilization — graphs by Nodes:  
- CPU and Memory by Nodes  
- number of Nodes — percentage of the maximum on the Node  
- created-deleted Nodes (by Karpenter)  
- Nodes cost  
- percentage of EBS used  
- network — in/our byes/sec

About Karpenter:

- `controller_runtime_reconcile_errors_total` - total number of errors
- `karpenter_provisioner_scheduling_duration_seconds` - time of creation of Pods
- `karpenter_cloudprovider_errors_total` - total number of errors

Looks like a plan?

Let’s go.

### Creating the dashboard

Create a new board and set the main parameters:

![](https://cdn-images-1.medium.com/max/998/0*b6pkPUkxJftm6P5_.png)

### Grafana variables

We need two variables — Nodes and Namespaces.

Nodes can be selected from the `karpenter_nodes_allocatable`, namespaces can be taken from the `karpenter_pods_state`.

Create the first variable  —  `node_name`, include the ability to select _All_ or _Multi-value_:

![](https://cdn-images-1.medium.com/max/903/0*3Lu7h--aGqt-7Wz-.png)

![](https://cdn-images-1.medium.com/max/1024/0*cZNgjVWHEEVkDoQz.png)

Create a second variable — $namespace.

To select namespaces only from the Nodes selected in the first filter, add the ability to filter by the `$node_name` that we created above and use a regular expression "`=~`" - if several Nodes are selected:

![](https://cdn-images-1.medium.com/max/1024/0*RnsW6thtdR44BMr0.png)

Let’s move on to the graphs.

### Number of WorkerNodes in the cluster

Query — use the `node_name=~"$node_name"` filter for the selected nodes:

```
count(sum(karpenter_nodes_allocatable{node_name=~"$node_name"}) by (node_name))
```

![](https://cdn-images-1.medium.com/max/1024/0*qFzudWtIx3mfkDO4.png)

### Number of Pods in the cluster

Query — here you can filter by both Nodes and Namespaces:

```
sum(karpenter_pods_state{node=~"$node_name", namespace=~"$namespace"})
```

![](https://cdn-images-1.medium.com/max/1024/0*HMmfVP9MKHujJOCG.png)

### Number of CPU cores on all nodes

Some of the resources are occupied by the system or by DemonSets — they will not be taken into account in `karpenter_nodes_allocatable`. You can check with the query `sum(karpenter_nodes_system_overhead{resource_type="cpu"})`.

Therefore, we can output either the total amount — `karpenter_nodes_allocatable{resourcetype="cpu"} + karpenter_nodes_system_overhead{resourcetype="cpu"}`, or only the amount actually available for our workloads - `karpenter_nodes_allocatable{resourcetype="cpu"}`.

Since we want to see the total number here, let’s use the sum:

```
sum(karpenter_nodes_allocatable{node_name=~"$node_name", resource_type="cpu"}) + sum(karpenter_nodes_system_overhead{node_name=~"$node_name", resource_type="cpu"})
```

![](https://cdn-images-1.medium.com/max/1024/0*54GNAVLGY267KJ2-.png)

### Total available Memory capacity

JFYI:

- SI standard: 1000 bytes in a **kilobyte**.
- IEC standard: 1024 bytes in a  **kibibyte**

But let’s just make `/ 1024` ourselves and use Kilobytes:

```
sum(sum(karpenter_nodes_allocatable{node_name=~"$node_name", resource_type="memory"}) + sum(karpenter_nodes_system_overhead{node_name=~"$node_name", resource_type="memory"})) / 1024
```

![](https://cdn-images-1.medium.com/max/1024/0*ilmgUKP5XFczM9nR.png)

### Spot instances — % of the total number of Nodes

In addition to the Nodes created by Karpenter itself, I have a dedicated “default” Node that is created when the cluster is created — for all controllers. It’s also a Spot (for now), so let’s count it as well.

The formula will be as follows:

```
spot nodes percent = all spot instances from karpenter + 1 default / total number of nodes
```

The query itself:

```
sum(count(sum(karpenter_nodes_allocatable{node_name=~"$node_name", nodepool!="", capacity_type="spot"}) by (node_name)) + 1) / count(sum(karpenter_nodes_allocatable{node_name=~"$node_name"}) by (node_name)) * 100
```

![](https://cdn-images-1.medium.com/max/1024/0*zj41ZAdiWmAL83dC.png)

### CPU requested — % of total allocatable

We take the query from the default Karpenter board and tweak it a bit to fit our filters:

```
sum(karpenter_nodes_total_pod_requests{node_name=~"$node_name", resource_type="cpu"}) / sum(karpenter_nodes_allocatable{node_name=~"$node_name", resource_type="cpu"})
```

![](https://cdn-images-1.medium.com/max/1024/0*9uJRIuP2O3DeI_qq.png)

### Memory requested — % of the total allocatable

Similarly:

```
sum(karpenter_nodes_total_pod_requests{node_name=~"$node_name", resource_type="memory"}) / sum(karpenter_nodes_allocatable{node_name=~"$node_name", resource_type="memory"})
```

![](https://cdn-images-1.medium.com/max/1024/0*HpdTEgCy3z_z8qfB.png)

### Pods allocation — % of the total allocatable

How many Pods do we have from the total capacity:

```
sum(karpenter_pods_state{node=~"$node_name", namespace=~"$namespace"} / sum(karpenter_nodes_allocatable{node_name=~"$node_name", resource_type="pods"})) * 100
```

![](https://cdn-images-1.medium.com/max/1024/0*sA8fBHCHlvZFWJts.png)

### Controller errors

The `controller_runtime_reconcile_errors_total` metric includes controllers from VictoriaMetrcis, so we exclude them with `{container!~".*victoria.*"}`:

```
sum(controller_runtime_reconcile_errors_total{container!~".*victoria.*"})
```

![](https://cdn-images-1.medium.com/max/1024/0*w38onxdqLEba3tXB.png)

### Cloudprovider errors

Count as a rate per second:

```
sum(rate(karpenter_cloudprovider_errors_total[15m]))
```

![](https://cdn-images-1.medium.com/max/1024/0*gJMBrInzS4f730UR.png)

### Nodes cost 24h — the cost of all Nodes per day

But here it turned out to be very interesting.

First of all, AWS has [default billing metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html) from CloudWatch, but our project uses AWS credits and these metrics are empty.

So let’s use Karpenter’s metrics — `karpenter_cloudprovider_instance_type_price_estimate`.

To display the cost of servers, we need to select each type of instance that is used and then calculate the total cost for each type and their number.

What do we have:

- the “default” Node: with Spot type, but not created by Karpenter — we can ignore it
- nodes created by Karpenter: can be either spot or on-demand, and can be of different types (`t3.medium`, `c5.large`, etc.)

First, we need to get the number of nodes for each type:

```
count(sum(karpenter_nodes_allocatable) by (node_name, instance_type,capacity_type)) by (instance_type, capacity_type)
```

We get 4 spots, and one instance without the `capacity_type` label, because it is from the default Node Group:

![](https://cdn-images-1.medium.com/max/1024/0*WUrYmfMn-uRnoGEw.png)

We can exclude it from `{capacity_type!=""}` - we have one, without autoscaling, we can ignore it, because it is only for the `CriticalAddons`.

For a better picture, let’s take a longer period of time, because there was also `t3.small`:

![](https://cdn-images-1.medium.com/max/1024/0*5H7IOsYbn5NNYJhU.png)

Next, we have the `karpenter_cloudprovider_instance_type_price_estimate` metric, using which we need to calculate the cost of all instances for each `instance_type` and `capacity_type`.

The request will look like this (thanks to ChatGPT):

```
sum by (instance_type, capacity_type) (
    count(sum(karpenter_nodes_allocatable) by (node_name, instance_type, capacity_type)) by (instance_type, capacity_type)
    * on(instance_type, capacity_type) group_left 
    avg(karpenter_cloudprovider_instance_type_price_estimate) by (instance_type, capacity_type)
)
```

Here:

1. the “inner” query `sum(karpenter_nodes_allocatable) by (node_name, instance_type, capacity_type)`: calculates the sum of all CPUs, memory, etc. for each combination of the `node_name`, `instance_type`, `capacity_type`
2. the “outer” `count(...) by (instance_type, capacity_type)`: the result of the previous query is counted with count to get the number of each combination - we get the number of WorkerNodes of each `instance_type` and `capacity_type`
3. second query — `avg(karpenter_cloudprovider_instance_type_price_estimate) by (instance_type, capacity_type)`: returns us the average price for each `instance_type` and `capacity_type`
4. using `* on(instance_type, capacity_type)`: multiply the number of nodes from query number **2** (`count(...)`) by the result from query number **3** (`avg(...)`) for the matching combinations of metrics `instance_type` and `capacity_type`
5. and the very first “outer” query `sum by (instance_type, capacity_type) (...)`: returns the sum for each combination

As a result, we have the following graph:

![](https://cdn-images-1.medium.com/max/1024/0*KF46xSvzHamET4Ra.png)

So, what do we have here?

- 4 `t3.medium` and 2 `t3.small` instances
- the total cost of all `t3.medium` per hour is 0.074, all `t3.small` - 0.017

To check, let’s calculate it manually.

First, by the `t3.small`:

```
{instance_type="t3.small", capacity_type="spot"}
```

It turns out to be 0.008:

![](https://cdn-images-1.medium.com/max/1024/0*-gfFEOPGcL7ueYB6.png)

And by the `t3.medium`:

```
{instance_type="t3.medium", capacity_type="spot"}
```

This turns out to be 0.018:

![](https://cdn-images-1.medium.com/max/1024/0*DsB6OcO-USaV5Kww.png)

So:

- 4 instances of `t3.medium` at 0.018 == 0.072 USD/hour
- 2 instances of `t3.small` at 0.008 == 0.016 USD/hour

![](https://cdn-images-1.medium.com/max/1024/0*HB2ahrqeufDAAMdW.png)

Everything looks correct.

All that’s left is to put it all together and display the total cost of all servers for 24 hours — use `avg()` and multiply the result by 24 hours:

```
avg(
  sum by (instance_type, capacity_type) (
    count(sum(karpenter_nodes_allocatable{capacity_type!=""}) by (node_name, instance_type, capacity_type)) by (instance_type, capacity_type)
    * on(instance_type, capacity_type) group_left 
    avg(karpenter_cloudprovider_instance_type_price_estimate) by (instance_type, capacity_type)
  )
) * 24
```

![](https://cdn-images-1.medium.com/max/1024/0*rPeJbBT9Gy7t25-j.png)

And as a result, everything looks like this:

![](https://cdn-images-1.medium.com/max/1024/0*NztZEcUZSD0NlFiC.png)

Let’s move on to the graphs.

### CPU % use by Node

Here, we will use the default metrics from Node Exporter — `node_cpu_seconds_total`, but they have instance labels in the form `instance="10.0.32.185:9100"`, and not `node_name` or node as in Karpenter metrics (`karpenter_pods_state{node="ip-10-0-46-221.ec2.internal"}`).

So to apply `node_cpu_seconds_total` with our variable `$node_name` we can add a new variable `node_ip`, which will be formed from the metric `kube_pod_info` with a filter by the label node, where we use our old variable `node_name` - to select Pods only from the Nodes selected in the filters.

Add a new variable, and for now, let to be the “_Show on dashboard_” enabled, for testing:

![](https://cdn-images-1.medium.com/max/693/0*rNcqCm_Alvim0MUU.png)

Now we can create a graph with a query:

```
100 * avg(1 - rate(node_cpu_seconds_total{instance=~"$node_ip:9100", mode="idle"}[5m])) by (instance)
```

![](https://cdn-images-1.medium.com/max/1024/0*w1BcCluTQHJ31_ph.png)

But in this case, the `instance` will return the results as "_10.0.38.127:9100_" - but we are using "_ip-10-0-38-127.ec2.internal_" everywhere. Also, we won't be able to add data links because the second panel uses the format the same format - _ip-10-0-38-127.ec2.internal_.

So we can use the [`label_replace()`](https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace) function and rewrite the request like this:

```
100 * avg by (instance) (
    label_replace(
        rate(node_cpu_seconds_total{instance=~"$node_ip:9100", mode="idle"}[5m]), 
        "instance", 
        "ip-${1}-${2}-${3}-${4}.ec2.internal", 
        "instance", 
        "(.*)\\.(.*)\\.(.*)\\.(.*):9100"
    )
)
```

Here, `label_replace` gets 4 aggregates:

1. first — the metric on which we will perform the transformation (result `rate(node_cpu_seconds_total)`)
2. the second is the label we will be transforming  —  `instance`
3. third — a new value format for labels —  `"ip-${1}-${2}-${3}-${4}.ec2.internal`"
4. the fourth is the name of the label from which we will keep the data using regex

And lastly, we describe the `regex “(.\*)\\.(.\*)\\.(.\*)\\.(.\*):9100"`, which should be used to get each octet from IPs like _10.0.38.127_, and then write each result in the `${1}-${2}-${3}-${4}` regex groups.

Now we have a graph like this:

![](https://cdn-images-1.medium.com/max/1024/0*lD3Zd7XnvHYT4p_3.png)

### Memory used by Node

Everything is the same here:

```
sum by (instance) (
  label_replace(
    (
      1 - (
        node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
      )
    ) * 100,
    "instance", "ip-${1}-${2}-${3}-${4}.ec2.internal", "instance", "(.*)\\.(.*)\\.(.*)\\.(.*):9100"
  )
)
```

![](https://cdn-images-1.medium.com/max/1024/0*dF_LxrkrjNPziZik.png)

### Pods use % by Node

Here we need to query between two metrics — `karpenter_pods_state` and `karpenter_nodes_allocatable`:

```
(
  sum by (node) (kube_pod_info{node=~"$node_name", created_by_kind!="Job"})
  / 
  sum by (node) (kube_node_status_allocatable{node=~"$node_name", resource="pods"})
) * 100
```

Or we can exclude our default Node “ip-10–0–41–2.ec2.internal” and display only the Nodes of Karpenter itself by adding a selection by `karpenter_nodes_allocatable{capacity_type!=""}` - because we are more interested in how busy the Nodes created by Karpenter itself for our applications.

But for this, we need the `karpenter_nodes_allocatable` metric, where we can check for the presence of the `capacity_type` label - `capacity_type!=""`.

However, `karpenter_nodes_allocatable` has a label `node_name` and not node as in the previous two, so we can add `label_replace` again and make this request:

```
(
  sum by (node) (kube_pod_info{node=~"$node_name", created_by_kind!="Job"})
  / 
  on(node) group_left
  sum by (node) (kube_node_status_allocatable{node=~"$node_name", resource="pods"})
) * 100
and on(node)
label_replace(karpenter_nodes_allocatable{capacity_type!=""}, "node", "$1", "node_name", "(.*)")
```

Here, in the `and on(node)`, we use the node label in the query results on the left (`sum by()`) and in the result on the right to select only those Nodes from the list of Nodes in `karpenter_nodes_allocatable{capacity_type!=""}` (that is, all Nodes except our "default" one) to select only those that are in the results of the first query:

![](https://cdn-images-1.medium.com/max/1024/0*M9h1562mEdps5BpZ.png)

### EBS use % by Node

It’s easier here:

```
sum(kubelet_volume_stats_used_bytes{instance=~"$node_name", namespace=~"$namespace"}) by (instance) 
/ 
sum(kubelet_volume_stats_capacity_bytes{instance=~"$node_name", namespace=~"$namespace"}) by (instance) 
* 100
```

### Nodes created/terminated by Karpenter

To display autoscaling activity, add a graph with two queries:

```
increase(karpenter_nodes_created[1h])
```

And:

```
- increase(karpenter_nodes_terminated[1h])
```

Here, in the `increase()` function, we check how much the value has changed in an hour:

![](https://cdn-images-1.medium.com/max/1024/0*MV6zd3Z5zwJAAKBJ.png)

And to get rid of these “steps”, we can additionally wrap the result in the `avg_over_time()` function:

![](https://cdn-images-1.medium.com/max/882/0*c5TWxoN4tE9HGryg.png)

### Grafana dashboard: the final result

And everything together now looks like this:

![](https://cdn-images-1.medium.com/max/1024/0*1GiqU64qxLRs8X93.png)

### Adding Data links

The last step is to add Data Links to the panels: we need to add a link to another dashboard, for a specific node.

This dashboard has the following URL: `https://monitoring.ops.example.co/d/kube-node-overview/kubernetes-node-overview?var-node_name=ip-10-0-41-2.ec2.internal`.

Where `var-node_name=ip-10-0-41-2.ec2.internal` specifies the name of a Node on which the data should be displayed:

![](https://cdn-images-1.medium.com/max/1024/0*zEr7i-YtFYl1-8yk.png)

So open the panel and find Data links:

![](https://cdn-images-1.medium.com/max/1024/0*Vff-PwmKi27Tqccb.png)

Set the name and URL — you can get a list of all fields by pressing Ctrl+Space:

![](https://cdn-images-1.medium.com/max/923/0*YIfOXFTwcmdiPzJM.png)

The `__field` will take values from the `labels.node` from the query result in the panel:

![](https://cdn-images-1.medium.com/max/703/0*1fEHvkFvx2xtOR1g.png)

And will generate a link in the form “_https://monitoring.ops.example.co/d/kube-node-overview/kubernetes-node-overview?var-node_name=ip-10-0-38-110.ec2.internal_".

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/karpenter-its-monitoring-and-grafana-dashboard-for-kubernetes-workernodes/)_._

* * *