From Valencia to Detroit: Observability is Stealing All the Thunder at KubeCon 2022
Tags: kubernetes, kubecon, observability, ebpf
Published: 2022-07-13 10:37:02

In the world of Kubernetes, observability is the talk of the town. Not only have sessions about novel takes on observability become recurring features at KubeCon events, but the blogo- and Twitterspheres are also rife with thoughts and posts on improving the way teams observe Kubernetes environments – and taking better advantage of tools like eBPF, which weren't around when Kubernetes debuted eight years ago.

This trend reflects what we are inclined to think of as an inflection point for K8s observability. Although the idea of observing K8s is obviously not new, practitioners and thought leaders are increasingly rethinking the tools, goals, and practices surrounding observability for Kubernetes.

**Kubernetes observability: The traditional approach**

When Kubernetes came along in 2014, most admins did the obvious thing by applying the same monitoring tools and practices to K8s clusters that had worked in other environments. Their approach boiled down to these basic steps:

- Collect as many metrics and logs as possible, typically with the help of monitoring agents that run inside the cluster.
- Dump those metrics and logs into some kind of time-series monitoring platform, like Prometheus.
- Spend a nice chunk of time studying dashboards and manually comparing data sets to identify and troubleshoot Kubernetes performance issues.

This approach to Kubernetes observability may have worked well enough in some cases, but it was never ideal. It required high effort and resulted in relatively low visibility. It also costs a lot of money to collect, analyze and store all of that monitoring data.

**How the K8s community is rethinking observability**

Today, however, evidence abounds that Kubernetes admin teams are not satisfied with the traditional approach to K8s observability.

For example, the Valencia KubeCon in May included a [dozen sessions](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/schedule/) dedicated to observability – not to mention a co-located "[eBPF Day](https://kccnceu2022.sched.com/event/z9cp/live-stream-cloud-native-ebpf-day-europe-hosted-by-cncf-additional-registration-499-fee-required-for-in-person-attendees?iframe=no&w=100%&sidebar=yes&bg=no)." The blogosphere is [abuzz](https://devops.com/open-source-observability-comes-of-age-in-2022/) with [articles ](https://securityboulevard.com/2022/06/using-ebpf-to-enhance-monitoring-and-observability/)on the potential of eBPF. And it's increasingly hard to find discussions of [Kubernetes observability](https://mobile.twitter.com/ogurechkin/status/1531073371382992896) on [Twitter ](https://twitter.com/kcdchennai/status/1532636124442939392)that don't mention eBPF. It's a pretty safe bet, too, that observability talks will be front and center at the next KubeCon in Detroit this fall (whose schedule has yet to be released).

If you look at these wide-ranging discussions on how to improve Kubernetes observability, you'll notice a few key themes and goals.

**Reducing cost**

First, we are seeing an increased focus on Kubernetes observability strategies that reduce overall cost.

One way to do this is to deploy lighter-weight observability stacks, which reduce the overhead that observability tools place on your cluster. For example, as Alex Jones [writes](https://twitter.com/AlexJonesax/status/1543678834100297730):


![Alex Jones Tweet](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5r4swyac1ocqneudwe5k.jpg)

Jones's thoughts are reinforced by Logz.io's latest [DevOps Pulse survey](https://logz.io/blog/devops-pulse-2022/), which found that 27 percent of respondents identified "total cost of ownership and the large volumes of data being ingested into the tools among their main challenges in maintaining effective observability into the cloud."

Perhaps observability spending wasn't as much of a concern in the heady days of 2021, when the tech industry remained flush with cash. But in today's increasingly volatile economic environment, controlling cost is more and more of a priority – and K8s management teams are realizing that they can do their part by deploying lighter-weight observability stacks.

**The turn to eBPF**

A second K8s observability trend – and one that ties into the cost-reduction concerns we just discussed – is a turn toward [eBPF as the foundation for Kubernetes observability](https://www.groundcover.com/blog/what-is-ebpf-anyway-and-why-should-kubernetes-admins-care).

It's not at all an exaggeration to say that eBPF, which lets lightweight programs run in the Linux kernel, provides revolutionary new opportunities when it comes to observability. eBPF makes it possible to collect deep, focused observability data from Kubernetes nodes (as long as they are Linux-based, at least) without burdening clusters with a full-scale observability stack, or having to worry about work like deploying sidecar containers.

**More traces**

One of the many tasks that eBPF makes faster and cheaper to perform is tracing – which is, indeed, an increasingly important component of Kubernetes observability. The [Logz.io survey](https://logz.io/blog/devops-pulse-2022/) found that 27 percent of practitioners are now including tracing in their observability strategies.

The concept of tracing is not exactly new, of course. For several years, we've been told that traces are one of the three "pillars" of observability (the others are metrics and logs). But until tools like eBPF came along, actually implementing distributed traces was a lot of work – not to mention a high degree of overhead.

But as teams look to gain more visibility than they can achieve via metrics and logs alone, and as eBPF makes it increasingly easy to look beyond metrics and logs, traces are evolving from a nice-to-have asset, to a core component of Kubernetes observability.

**Change is going to come**

Thanks to the ongoing conversations about rethinking Kubernetes observability – and taking greater advantage of newer tools, like eBPF – admins can look forward to a future where [observability is easier, less costly and more meaningful than ever before](https://www.groundcover.com/blog/no-strings-attached-observability-with-limitless-data-depth). The new approaches that are being highlighted in contexts like KubeCon – and which, again, we expect to see even more of at upcoming K8s events – are underlining how it has become easier than ever to trace what is happening to Kubernetes clusters and apps from deep within the kernel that powers each node. By extension, deploying lightweight, low-cost Kubernetes observability tools has also become much simpler.

For now, a lot of these discussions remain in the talking stages more than the action stages – teams are pioneering or validating new approaches, but they may not have implemented them fully. But the fact that there is so much buzz around K8s observability suggests that we're at an inflection point, and that within maybe a year's time, more and more organizations will be putting these new tools and approaches into practice.

We certainly hope they do, because to be the most it can be, Kubernetes needs simple, effective and low-cost observability – which only these new approaches can deliver.