The case of disappearing metrics in Kubernetes
Tags: kubernetes, devops, prometheus, aws
Published: 2020-05-24 15:46:30

---
title: The case of disappearing metrics in Kubernetes
published: true
description: How I debugged when metrics failed to appear in the grafana dashboard
cover_image: https://i.imgflip.com/42nrku.jpg
tags: kubernetes, devops, prometheus, aws
---
## 🧐 Context 🧐

My team hosts the kubernetes platform running on [EKS](https://aws.amazon.com/eks/) for several teams. When EKS announced the support for Kubernetes version 1.16 we upgraded our infrastructure. This is pretty routine process for us. We use Prometheus and grafana dashboard to observe and monitor our infrastructure. 

After we upgraded, our metrics dashboard stopped showing RAM and CPU utilization of pods. I set out to find why?

![detective](https://media.giphy.com/media/42wQXwITfQbDGKqUP7/giphy.gif)

## ⚠️ Disclaimer ⚠️

Before we move ahead, 
* I am a newbie to Kubernetes learning the tricks on the job
* This is my journey on how I debugged the issue, so what is obvious to a Kubernetes expert is not obvious to me.

### TL;DR 
If you already know what caused this issue, jump to [final section](#finally)

## Debugging Step 1:

So the dashboard is not showing metrics, normally Kubectl has commands that shows the CPU and RAM utilization of the Nodes and Pods. So I checked if those worked.

```bash
 > kubectl top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-xx-xx-xxx-xxx.ec2.internal   8xxm         5%     2xxxxMi         23%
ip-xx-xx-xxx-xxx.ec2.internal   2xxxxm       15%    3xxxxMi         29%
ip-....

> kubectl top pods
NAME         CPU(cores)   MEMORY(bytes)
xxxx-wnpgj   xxm          xxMi
xxxx-r4k85   xxm          xxxMi
...
```
###  🤔 What does this mean? 🤔
Kubernetes is able to fetch the metrics. These are the metrics that have to be used by the [HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#how-does-the-horizontal-pod-autoscaler-work) to scale the Pods based on the resource utilization. 

## Debugging Step 2:

Since the dashboard does not show the resource utilization metrics, but the `kubectl top` command works, my next thought was may be the `scaling does not work properly anymore`. 

According to [Resource usage monitoring](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#resource-metrics-pipeline)

> components such as the Horizontal Pod Autoscaler controller, as well as the kubectl top utility. These metrics are collected by the lightweight, short-term, in-memory metrics-server and are exposed via the metrics.k8s.io API

So now I wanted to see what is the status with one of our teams.
```bash
> kubectl get hpa -n xxx 
NAME         REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
xxx-worker   Deployment/xxx-worker   82/150, 5%/100%   1         6         1          4d19h
```
Since the Targets look fine in the above output. I thought let me see if at any point the scaling actually happened.

```bash
> kubectl describe hpa -n xxx 

Type     Reason                   Age                      From                       Message
----     ------                   ----                     ----                       -------
Warning  FailedGetResourceMetric  25m (x24 over 2d23h)     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
Normal   SuccessfulRescale        9m56s (x233 over 2d23h)  horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
Normal   SuccessfulRescale        4m9s (x233 over 2d22h)   horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
```
###  🤔 What does this mean? 🤔
As we can see at some point fetching the resource metric failed but we can also see that the system recovered and was able to scale successfully. 

Since there is a `FailedGetResourceMetric` at some point, I thought this could lead me to why metrics do not appear in the dashboard.

## Debugging Step 3:

As we have seen from several documentation mentioned above and also from the error `no metrics returned from resource metrics API`. There is some component in Kubernetes that is responsible for the metrics. So I cast a wide net

```bash
> kubectl get all -n metrics | grep metrics | grep pod
pod/metrics-server-xxxxxxx-xxxxx                        1/1     Running            0          4d21h
pod/prometheus-kube-state-metrics-xxxx-xxxxx            1/1     Running            0          4d21h
```

This sound promising. So `metrics-server` sounds like some server dealing with metrics. Naturally I am interested in logs

```bash
> k logs -n metrics pod/metrics-server-xxxxxxx-xxxxx
I0524 11:17:54.371856       1 manager.go:120] Querying source: kubelet_summary:ip-xx-xx-xxx-xxx.ec2.internal
I0524 11:17:54.388840       1 manager.go:120] Querying source: kubelet_summary:ip-xx-xx-xxx-xxx.ec2.internal
I0524 11:17:54.495829       1 manager.go:148] ScrapeMetrics: time: 233.232388ms, nodes: x, pods: xxx
E0524 11:18:18.542051       1 reststorage.go:160] unable to fetch pod metrics for pod my-namespace/team-worker-xxxxx-xxxx: no metrics known for pod
```

###  🤔 What does this mean? 🤔

As soon as I saw the words `ScrapeMetrics` (my brain translated it to Prometheus related) and the error `unable to fetch pod metrics for pod my-namespace/team-worker-xxxxx-xxxx: no metrics known for pod`
I thought I have found the issue.


## Debugging Step 4:

As any seasoned developer would do, I took the error message and googled it. Which resulted in some links and on a quick scan, I found couple of interesting Links 
![Links from google search](https://dev-to-uploads.s3.amazonaws.com/i/n0stnswd90ro145nvx0a.png)

After quickly scanning through the links, every one seem to suggest this following change in metrics-server
```yaml
command:
- /metrics-server
- --kubelet-insecure-tls #<-- 🤓
- --kubelet-preferred-address-types=InternalIP #<-- 🤓
```
`--kubelet-insecure-tls` seems like something you would do in a test setup. So I wanted to understand a bit more what this flags do.
From the [documentation](https://github.com/kubernetes-sigs/metrics-server#configuration), it is clear that the above flag is intended for testing purposes only and in the logs of metrics server I did not find any error pertaining to certificates. So I did not add the flag.

`--kubelet-preferred-address-types` this configuration is to determine which address type to use to connect to the nodes for getting the metrics. 
So before adding this config I wanted to verify if the metrics server api was reachable or not.

```bash
> kubectl get apiservices | grep metrics-server
v1beta1.metrics.k8s.io                 kube-system/metrics-server       True        4d23h
```
since the status of this service is available, those two flags would not make any sense. I wanted to see if I can get some raw metrics by querying this API.

```bash
> kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq '.items | .[] | select(.metadata.namespace == "xxx")'
{
  "metadata": {
    "name": "xxx-worker-6c5bd59857-xxxxx",
    "namespace": "xxx",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/xxx/pods/xxx-worker-6c5bd59857-xxxxx",
    "creationTimestamp": "2020-05-24T12:06:45Z"
  },
  "timestamp": "2020-05-24T12:05:48Z",
  "window": "30s",
  "containers": [
    {
      "name": "xxx-worker",
      "usage": {
        "cpu": "xxxxxn",
        "memory": "xxxxxxxKi"
      }
    }
  ]
} 
```
###  🤔 What does this mean? 🤔
![Frustration](https://media.giphy.com/media/ql4LidslabKpi/giphy.gif)

So everything seems to function well. I do not know why the metrics I am clearly able to query via `Kubectl` is not available for the dashboard.

## Debugging Step 5:
Now I want to see the source code of [metrics-server](https://github.com/kubernetes-sigs/metrics-server/blob/d1f4f6fc09cd3134e8ea5ba4e0bd2db4e8002ed8/pkg/storage/podmetrics/reststorage.go#L133) to figure out under what circumstance this error happens

```Go
	podMetrics, err := m.getPodMetrics(pod)
	if err == nil && len(podMetrics) == 0 {
		err = fmt.Errorf("no metrics known for pod \"%s/%s\"", pod.Namespace, pod.Name)
	}
	if err != nil {
		klog.Errorf("unable to fetch pod metrics for pod %s/%s: %v", pod.Namespace, pod.Name, err)
		return nil, errors.NewNotFound(m.groupResource, fmt.Sprintf("%v/%v", namespace, name))
	}
```
It is very clear from the code that the Pod exists but metrics is not available. So I tailed the logs and got the pod for which metrics is not found

```bash
 > kubectl get po -n xxx
NAME                          READY   STATUS    RESTARTS   AGE
xxx-worker-6c5bd59857-xxxxx   1/1     Running   0          1m5s
```
###  🤔 What does this mean? 🤔
It is very clear from above that all these pods where metrics are not available are pods that have been recently created 
and since metrics server scrapes for metrics periodically, it's just not available at the moment.

In my opinion it is really not an `ERROR` rather an expected behaviour. A `WARN` or `INFO` would be reasonable. It is good to see am not alone in this

{% github https://github.com/kubernetes-sigs/metrics-server/issues/349 %}

I have been chasing a wrong lead after all.

![back to square one](https://i.imgflip.com/42nax3.jpg)

## Debugging Step 5:

I have eliminated that Metrics server is not the reason for the bug. This got me to thinking, I should find how Prometheus is scraping the metrics.

After a lot of reading I figured out Prometheus gets the metrics exposed by the `kube-state-metrics`. It is very important to understand the difference between `metrics-server` and `kube-state-metrics`

{% github https://github.com/kubernetes-sigs/metrics-server/issues/55#issuecomment-387229420 %}

Looking at the logs of prometheus server and kube-state-metrics nothing jumped out of ordinary.

## <a name="finally">Finally</a>
I asked for help, I explained what I have done so far and my team mate suggested the obvious, did you check if the query works!!

so I went to prometheus query explorer and executed 
```promql
container_cpu_usage_seconds_total{namespace="xxx", pod_name="xxx-worker-6c5bd59857-xxxxx"}

Returned no results

container_cpu_usage_seconds_total{namespace="xxx"}

Returned results

container_cpu_usage_seconds_total{namespace="xxx", pod="xxx-worker-6c5bd59857-xxxxx"}

Returned results
```
###  🤔 What does this mean? 🤔
So some where during the EKS version upgrade the support for label `pod_name` is [dropped](https://github.com/kubernetes/kubernetes/pull/80376/commits/e0b66c792b8daabc8f2dcc5209356bca2cb2b197).

After fixing the queries in grafana, I could see the metrics finally.
![case closed](https://i.imgflip.com/42nepy.jpg)

## Lessons Learned

- Learn the components and its purpose before assuming
- Always read the release notes when [upgrading even minor versions](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#removed-metrics)

> Removed cadvisor metric labels pod_name and container_name to match instrumentation guidelines. Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead. (#80376, @ehashman)

- The solution is always the obvious solution
![do not overthink just because it is kubernetes](https://i.imgflip.com/42nqn2.jpg)

*I welcome all constructive feedback and comments*