Kubernetes: Statistics in Your Console
Tags: devops, kubernetes, productivity
Published: 2022-03-23 18:01:51

> As intended by the author, this article is the first in a series about the community-developed tools we work with at Otomato. And the second one about resource management in K8s (these talks had begun [in January](https://dev.to/otomato_io/moderating-pods-appetites-on-a-node-the-brief-1km7)). Fortunately, the world of open source does not let us get bored. 

Even in the world of containers, you’re still going to have to deal with CPU and memory resources. In the daily work of a DevOps engineer with a Kubernetes cluster, it is often in demand to quickly and with minimal effort obtain statistical information about the resources consumed. And, of course, no one of Ops would want to feel himself like [MV RENA](https://en.wikipedia.org/wiki/MV_Rena)'s captain or [Ever Given](https://en.wikipedia.org/wiki/Ever_Given)'s one!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i6yl6k0i2knzw9uo8uav.jpeg)

In general, resource consumption should be closely monitored.

Gurus will say that monitoring systems such as Prometheus/Grafana (and maybe even [relying on](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/) something like `metrics-server` subsystem) [should be used](https://getbetterdevops.io/setup-prometheus-and-grafana-on-kubernetes/) for these purposes, we'll talk about them some other time. Younger gurus will say, "don't show off and open the [Lens](https://k8slens.dev) application!". But, come on, guys, what if we _just want_ to display what we need on the screen with one console command? Fast, short and clear? But something more sophisticated than `kubectl top`'s output in the same time?

Well, thanks to developers community, there is a tool. Now I will tell you how to do it and with what tool.

##A few words about extensibility

As you may know, the Kubernetes command-line tool, `kubectl`, allows you to run commands against Kubernetes clusters. The Kubernetes CLI special interest group [added](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/) a built-in _plugin_ system to `kubectl` that allows anyone to add new sub-commands _to extend the basic command set it offers_. This does not require editing `kubectl`’s source code or recompiling it.

Any executable file in your [PATH](https://opensource.com/article/17/6/set-path-linux) that starts with `kubectl-` can be called with the kubectl command. To try this out, let’s write a very basic plugin called `kubectl-hello`.
```
#!/usr/bin/env bash
echo "Congrats ${USER}, you just used a kubectl plugin! 💪"
```
Make this script executable and add it to your PATH:
```
chmod +x kubectl-hello
ln -s $PWD/kubectl-hello /usr/local/bin/kubectl-hello
```
That’s it! You can now use this plugin with `kubectl`:
```
kubectl hello
```
Well, that's pretty much how it works. Let's go further.

##Krew: all plugins in one box

Installing plugins manually and keeping track of them can be tedious. Thankfully, `kubectl` has a plugin manager. The community-driven plugin manager for `kubectl` is called `krew`.

For Linux, follow [these instructions](https://krew.sigs.k8s.io/docs/user-guide/setup/install/) to set it up on your computer.

This will install the `kubectl-krew` binary. Notice that `krew` is itself a `kubectl` plugin.

The open-source community keeps a list of plugins you can install in the official `krew-index` repository. Use `krew` to download the index:

```
kubectl krew update
```
You can view the list of available plugins with the search sub-command:

```
kubectl krew search
```
Installing a plugin is easy with the `install` sub-command. To try it out, install the `whoami` plugin:

```
kubectl krew install whoami
```
If you installed a plugin a while ago, you may want to upgrade to a newer version, which could include new features or bug-fixes. Use the upgrade sub-command to upgrade one or all installed plugins. You can upgrade a single plugin with 

```
kubectl krew upgrade whoami
```
 Or upgrade all installed plugins with 

```
kubectl krew upgrade
```
 commands, respectively. 

## Ok, but what about our resources and statistics?

The reader is right. Let's go back to the question in the title. So, we _got_ a tool, let's _sharpen_ and _use_ it, now! 

This plugin we're interested in and can be installed with our  `krew` is `kube-capacity`. [It is](https://github.com/robscott/kube-capacity) a simple CLI written by Rob Scott from Google that provides an overview of the _resource requests, limits, and utilization_ in a Kubernetes cluster. It attempts to combine the best parts of the output from kubectl top and kubectl describe into an easy-to-use CLI focused on cluster resources. As a result, it can save you a lot of commands to figure out how a node is being used and which pods might be using more resources than they should be. 

The installation process is easy:

```
kubectl krew install resource-capacity
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z7coy5tv0dp411qxo7ey.jpg)

## Use cases

The simple `resource-capacity` command with `kubectl` would return the CPU [requests and limits](https://dev.to/otomato_io/kubernetes-magic-numbers-22jm) and memory requests and limits of each Node available in the cluster.

You can use the `--sort cpu.limit` flag to sort by the CPU limit. There are more sorts available, we will see next.

```
$ kubectl resource-capacity --sort cpu.limit
NODE                            CPU REQUESTS   CPU LIMITS    MEMORY REQUESTS   MEMORY LIMITS
*                               5120m (65%)    5950m (75%)   5815Mi (19%)      6690Mi (22%)
ip-192-168-21-48.ec2.internal   2760m (70%)    3275m (83%)   2995Mi (20%)      3645Mi (24%)
ip-192-168-18-83.ec2.internal   2360m (60%)    2675m (68%)   2820Mi (19%)      3045Mi (20%)
```
By default,  this would not return the Utilization data for each node.

If you need to see utilization report for each node, you need to use `--util` flag:

```
$ kubectl resource-capacity --sort cpu.limit --util
NODE                            CPU REQUESTS   CPU LIMITS    CPU UTIL    MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*                               5120m (65%)    5950m (75%)   581m (7%)   5815Mi (19%)      6690Mi (22%)    6511Mi (22%)
ip-192-168-21-48.ec2.internal   2760m (70%)    3275m (83%)   257m (6%)   2995Mi (20%)      3645Mi (24%)    2708Mi (18%)
ip-192-168-18-83.ec2.internal   2360m (60%)    2675m (68%)   324m (8%)   2820Mi (19%)      3045Mi (20%)    3803Mi (26%)
```
### What if we need a more customized report?

As we have seen above, `kubectl resource-capacity` plugin supports sort by fields. The `--sort` flag supports the following values: `cpu.util`, `cpu.request`, `cpu.limit`, `mem.util`, `mem.request`, `mem.limit`, `name`.

If no flag applied, by default it is sorted in ascending order by the `name`.

Moreover, adding `--pods` flag to your command will dump you all the similar stats, including running pods. The sorting order will be preserved. We can even further break it down to finer detail such as containers in each pods by adding `--containers` flag. The whole command will look like this: 

```
kubectl resource-capacity --sort cpu.util --util --pods --containers
```
### Is there any way to summarize values for namespaces?

It is a very good question. To get the CPU and memory usage of the namespace with `kube-capacity` plugin, use the following command:
```
kubectl resource-capacity -n kube-system -p -c
```
or
```
kubectl resource-capacity -n kube-system --pods --containers
```
### What if I would like to get YAML output for using it in automation frameworks?

The described `kube-capacity` plugin is configured to give output as a table by default, but the output can be customized into other formats too. The supported output formats are also YAML and JSON. You can define the output by using the flag `-o` or `--output`.

## I don't want to be a captain, I want to be a sea pilot!

Every Ops professional should be inquisitive, no less than a Dev one! Yes, there is something to strive for: with the help of the `kube-capacity` plugin, you can, figuratively speaking, not only find out the reason for the _overload_ of the ship, but also _what maps it used_ to plow the seas! 

So, what was in the pilot's briefcase? Let's see. _Labels_ will help us. We can use either labels to attach metadata to Kubernetes objects. Labels can be used to select objects and to find collections of objects that satisfy certain conditions. 

With `kube-capacity` plugin, you can filter the _nodes_ based on their _label_ and get the CPU and memory usage of these _specific_ nodes by using `--node-labels` flag. Also, based on the labels, we can "pull" useful information from our cluster, and use it to understand what went wrong, where we made a mistake, and what parameters should be modified or should be improved.

In our case, we want to filter the nodes with this Label `node.kubernetes.io/instance-type=m5.xlarge` and get the resource capacity and usage of those nodes which are hosted on such [AWS instances](https://aws.amazon.com/ec2/instance-types/). Let's see the sample output:

```
$ kubectl resource-capacity --node-labels node.kubernetes.io/instance-type=m5.xlarge
NODE                            CPU REQUESTS   CPU LIMITS    MEMORY REQUESTS   MEMORY LIMITS
*                               5120m (65%)    5950m (75%)   5815Mi (19%)      6690Mi (22%)
ip-192-168-21-48.ec2.internal   2660m (67%)    3125m (79%)   2895Mi (19%)      3495Mi (23%)
ip-192-168-18-83.ec2.internal   2460m (62%)    2825m (72%)   2920Mi (20%)      3195Mi (21%)
```
There are also some infrastructure specific or cloud topology specific Labels which may be useful to construct queries during your investigations. Among them are: `kubernetes.io/hostname`, `topology.kubernetes.io/zone`, `topology.kubernetes.io/region`, `eks.amazonaws.com/capacityType` (`ON_DEMAND | SPOT`, [AWS context](https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-provisioning-and-managing-ec2-spot-instances-in-managed-node-groups/)) and others.

To display the pod count of each node and the whole cluster, you can pass `--pod-count` argument:

```
$ kubectl resource-capacity --pod-count
NODE                            CPU REQUESTS   CPU LIMITS    MEMORY REQUESTS   MEMORY LIMITS   POD COUNT
*                               5120m (65%)    5950m (75%)   5815Mi (19%)      6690Mi (22%)    61/116
ip-192-168-21-48.ec2.internal   2660m (67%)    3125m (79%)   2895Mi (19%)      3495Mi (23%)    33/58
ip-192-168-18-83.ec2.internal   2460m (62%)    2825m (72%)   2920Mi (20%)      3195Mi (21%)    28/58
```

A similiar but less informative result may be dumped even without using any plugins. You may run:

```
kubectl get po -o json --all-namespaces | \
  jq '.items | group_by(.spec.nodeName) | map({"nodeName": .[0].spec.nodeName, "count": length}) | sort_by(.count)'
```

In general, as you may clearify, both sea pilots and captains will be able to defend the honor of the uniform. And if your toolkit is limited, don't worry! There are, though more complex, but also [quite working](https://github.com/eldada/kubernetes-scripts) approaches. 

However, this is a topic for a separate article. Like another topic: imagine, before you can query the Kubernetes Metrics API or run `kubectl top` commands to retrieve so called _metrics_, you’ll need to ensure that Metrics Server is deployed to your cluster. The author [has something to work on](https://dev.to/otomato_io/kubernetes-native-metrics-and-states-2p68) in the future.

Fast and robust clusters to you!

> 🇩🇪🇫🇷🇮🇱 The author is thankful to Sarav Ak, Tapan Hegde, Guillaume Vincent and Eldad Assis for their watchfulness in solving DevOps tasks. Pic source: Platform9.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7loc56nl5f4w70pznet4.jpeg)