Build On Live | Observability - Show notes
Tags: observability, aws, twitch, notes
Published: 2022-10-20 20:19:11

Welcome to our second **Build On Live Event** which is all about **Observability**. AWS experts and community members cover topics ranging from tracing, OpenTelemetry, SLOs, eBPFs and more!

This event was hosted by [Jacquie Grindrod](https://twitter.com/devopsjacquie) and [Ricardo Ferreira](https://twitter.com/riferrei) and it was a sight to be seen. With both in-person and remote guests, the industry experts covered a variety of specialty topics within the observability field

These are the show notes from that event, which was live streamed on the 29th of September 2022. The recording of this event is available on our [YouTube channel](https://www.youtube.com/c/BuildOnAWS), but in this article I will be linking each segment notes with its video, so make sure to hit that subscribe button! 👏

---

## Individual session notes:
  * [Intro and Welcome](#intro)
  * [Go Beyond Observability with AI](#gobeyond)
  * [How to Migrate Observability Platforms With OpenSLO](#openslo)
  * [Intro to eBPF: Explain to Me As if I Was Five](#ebpfs)
  * [OpenTelemetry—The Industry Telemetry Standard](#otel)
  * [That's a lot of data! how to manage ingestion and storage costs?](#data)
  * [Observability and Distributed Tracing at CNCF](#cncf)
  * [Creating trace data with OpenTelemetry](#oteltrace)
  * [Talking Observability with Liz Fong-Jones](#ollywliz)

## Intro and Welcome <a name="intro"></a>

Guest: [Curtis Evans](https://twitter.com/curtEYEvan), Developer Advocate at AWS

We kick off the day with your hosts Ricardo, Jacquie and a guest star Curtis. Discussing the basics of Observability and what can we expect throughout the day.

So, "explain to us like we are 5", Curtis takes an example of a e-Commerce website and what all goes on in an application and how we track each execution and process that happens in each step in your application, that can in the future give us more insights and helps us detect and solve a problem before it even occurs.

## Go Beyond Observability with AI <a name="gobeyond"></a>

Guest: [A.B. Vijay Kumar](https://twitter.com/abvijaykumar), IBM Fellow

{% embed https://www.youtube.com/watch?v=A2OSEDwY-Q4 %}

Did you know AI can be used to help you understand the stream of events from observability data and alert you when something is bit off? In this session, A.B Vijay Kumar from IBM gives you an overview of how to leverage AI in what the industry calls AIOps (also called Augmented SRE), which can be a powerful component of your observability strategy. He shows a nice demo where AI is used to detecting anomalies and trigger conversations on Slack that people on-call could discuss what the problems are, and what actions the AI recommends taking based on past incidents.

Also, check out the 1:14 minute mark in the [video](https://www.youtube.com/watch?v=A2OSEDwY-Q4) something funky happened to the camera 😅

![Ricardo and Jacquie talking to A.B. remotely](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bin6uolh5voshjqq9z70.png)

### Some links that were shared:
- [What is AIOps](https://www.gartner.com/en/information-technology/glossary/aiops-artificial-intelligence-operations)
- [AI Ops prescriptive guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-operations-integration/aiops.html)
- [Well Architected Analytics Lens](https://docs.aws.amazon.com/wellarchitected/latest/analytics-lens/operational-analytics.html)

---

## How to Migrate Observability Platforms With OpenSLO <a name="openslo"></a>

Guest: [Ian Bartholomew](https://www.linkedin.com/in/the-ian-bartholomew/), Site Reliability Engineering Manager at Nobl9

{% embed https://www.youtube.com/watch?v=Rh8zunybRYM %}

One of the best ways to measure the availability of services is having SLOs (Service Level Objective) set to them. They help in giving you a more pragmatic way to measure things and calculate how much does it costs having a service unavailable. What you may not know is that you can define them using an agnostic standard called OpenSLO using YAML. In this session, Ian Bartholomew from [Nobl9](https://www.nobl9.com/) get into the weeds of how to do that, showing step by step the definition, implementation, and migration of defined OpenSLO's from on-premise to the cloud. And he does all of this with a hands-on approach using only the terminal. How cool is that?

How do you explain/justify Error budgets to Engineering/Upper management? Well, you need to very clearly state that service uptimes will never be a 100%, it's gonna be very close but we also need to understand that all the extra nines is a lot of extra money - exponentially. So this error budget allows for an X amount to be unreliable.

Quote from [Charity Majors](https://twitter.com/mipsytipsy): _"Nine nines mean nothing if your customers are unhappy"_

![Ian, Ricardo and Jacquie in the Studio](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v9afyadqu12om267czh3.png)

### Some links that were shared:
- [OpenSLO](https://openslo.com)
- [SLO DLC](https://www.slodlc.com/)
- [What is an error budget](https://www.atlassian.com/incident-management/kpis/error-budget#:~:text=An%20error%20budget%20is%20the,can%20fail%20without%20contractual%20consequences.)
- [The code Ian has used in the session](https://github.com/nobl9/build-on-aws-o11y-demo)
- [Free trial for Nobl9](https://app.nobl9.com/signup/)
- [SLOConf Monthly meetup](https://www.meetup.com/sloconf-monthly/events/)

---

## Intro to eBPF: Explain to Me As if I Was Five <a name="ebpfs"></a>

Guest: [Antón Rodríguez](https://twitter.com/antonmry), Principal Software engineer at New Relic

{% embed https://www.youtube.com/watch?v=vg5BssWmvL0 %}

Implementing observability these days is a no turning back trip to the world of instrumentation. But instrumenting services can be a lot of work depending on the approach you use. To minimize a bit of this workload, you can use eBPF — a non-intrusive way to instrument sandbox programs in an operating system kernel. In this session, Antón Rodriguez from New Relic explains how he got involved with this technology, how he applies it to his daily job as a software engineer, and also shows a cool demo of eBPF using the CNCF project called Pixie.

![Anton showing us the Pixie dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1y04qpfvgvsk0h2fowbl.png)

### Some links that were shared:
- [Anton's Blog](https://www.galiglobal.com/)
- [What is eBPF](https://ebpf.io/)
- [Pixie - CNCF project](https://pixielabs.ai/)
- [Using Pixie to monitor a Strimzi kafka cluster](https://www.galiglobal.com/blog/2022/20220823-Using-Pixie-with-Strimzi.html)
- [Anton's talk at the Kafka Summit 2022](https://www.confluent.io/events/kafka-summit-london-2022/monitoring-kafka-without-instrumentation-using-ebpf/)

---

## OpenTelemetry—The Industry Telemetry Standard <a name="otel"></a>

Guest: [Michael Hausenblas](https://twitter.com/mhausenblas), Open Source - Observability at AWS

{% embed https://www.youtube.com/watch?v=I6Rtq9D5u7U %}

OpenTelemetry these days is arguably a synonym for observability. This CNCF project that provides an agnostic way for developers to instrument their services to produce telemetry data such as tracing and metrics is catching on. But a project of that magnitude can be often hard to understand. In this session, Michael Hausenblas from AWS gives us an overview of OpenTelemetry, his involvement in this project with CNCF, and how can you get started with this using the right resources. He also clarifies the many sub-projects that exist for OpenTelemetry, which are handy to know as you find your way into this technology.

![Michael, Jacquie and Ricardo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7kf4l2egs3kduzr3f29t.png)

### Some links that were shared:
- [OpenTelemetry](https://opentelemetry.io/)
- [OTLP Protocol](https://opentelemetry.io/docs/reference/specification/protocol/)

---

## That's a lot of data! how to manage ingestion and storage costs? <a name="data"></a>

Guest: [Richard "RichiH" Hartmann](https://twitter.com/TwitchiH), Director of Community at Grafana

{% embed https://www.youtube.com/watch?v=UIqB75PRbuc %}

All monitoring initiatives start the same: you collect as much data as you can, you store them infinitely so you can analyze them someday, but one day the bill comes and upper management asks you to decrease infra costs. Suddenly, you shrink your monitoring infra and end up with a shared cluster for all the 100+ services that need some visibility. In this session, Richard Hartmann from Grafana Labs explains why this is an anti-pattern that should not occur every time people panic about infra costs related to monitoring. He shares some best practices about how to optimize ingestion and the storage costs of telemetry data. As he walks us over the different telemetry data, he also explains what are golden signals and how you should implement them.

![Richard and the hosts on stream](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/03wi0zsrm67eqedau6j7.png)

### Some links that were shared:
- [Grafana Blog](https://grafana.com/blog/)
- [SRE eBook by Google](https://sre.google/sre-book/table-of-contents/)
- [Richard's Talks](https://github.com/richih/talks)

---

## Observability and Distributed Tracing at CNCF <a name="cncf"></a>

Guest: [Yuri Shkuro](https://twitter.com/YuriShkuro), Creator of Jaeger, Co-founder of OpenTelemetry

{% embed https://www.youtube.com/watch?v=vJskdmyPHOM %}

Despite of most people think, logs are not the only way to understand the execution path of code. Traces can tell that story too, and when combined with logs, provide even a more interesting story. This is the reason distributed tracing is so mainstream these days. In this session, Yuri Shkuro, from Meta, gives his perspective about the importance of distributed tracing, how he, by accident, bumped into this technology, and how they helped it to grow during his tenure at CNCF. He also explains how projects like OpenTracing, Jaeger, OpenCensus, and Zipkin helped to pave the way to what we know as OpenTelemetry.

![Yuri, Ricardo and Jacquie on stream](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9q7t7ujvcmoi8gpukb9r.png)

### Some links that were shared:
- [Jaeger](https://www.jaegertracing.io/)
- [Mastering Distributed tracing - Book](https://www.shkuro.com/books/2019-mastering-distributed-tracing/)
- [Yuri's talks](https://www.shkuro.com/talks/)
- [Yuri's Github](https://github.com/yurishkuro)
- [TEMPLE: Six Pillars of Observability](https://medium.com/@YuriShkuro/temple-six-pillars-of-observability-4ac3e3deb402)
- [Positional Paper: Schema-First Application Telemetry](https://research.facebook.com/publications/positional-paper-schema-first-application-telemetry/)

---

## Creating trace data with OpenTelemetry <a name="oteltrace"></a>

Guest: [Curtis Evans](https://twitter.com/curtEYEvan), Developer Advocate at AWS

{% embed https://www.youtube.com/watch?v=9jA3HDZYiaA %}

There is no better way for developers to learn how to use a specific technology than rolling up their sleeves and writing some code for it to build an example. Curtis Evans from AWS strongly believes that, and he steps up to help us understand OpenTelemetry in a more hands-on way. In this session, he takes his time to instrument a microservice written in Python to produce telemetry data for traces and metrics. He highlights the difference between a piece of code with and without instrumentation, so you can have a sense of how much work is needed.

![Curtis showing some telemetry data on screen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2gey4rq00ql2nhmp7nin.png)

### Some links that were shared:
- [OpenTelemetry Instrumentation](https://opentelemetry.io/docs/instrumentation/)
- [OpenTelemetry Python Guide](https://opentelemetry.io/docs/instrumentation/python/)
- [AWS Management and Governance page on Observability](https://aws.amazon.com/products/management-and-governance/use-cases/monitoring-and-observability/?trk=a4eae550-3d3e-42d1-916d-6737f0b4436c&sc_channel=ps&s_kwcid=AL!4422!3!600238537393!p!!g!!system%20observability&ef_id=CjwKCAjwg5uZBhATEiwAhhRLHqABpCUdBkJwpSVaBAMhbCUzNj6upH5ykomnOlchNYsoa3fQPweF8RoC9zwQAvD_BwE:G:s&s_kwcid=AL!4422!3!600238537393!p!!g!!system%20observability&whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc&blog-posts-cards.sort-by=item.additionalFields.createdDate&blog-posts-cards.sort-order=desc)
- [OpenTelemetry Registry](https://opentelemetry.io/registry/)

---

## Talking Observability with Liz Fong-Jones <a name="ollywliz"></a>

Guest: [Liz Fong-Jones](https://twitter.com/lizthegrey), Field CTO at Honeycomb.io

{% embed https://www.youtube.com/watch?v=2G6fyROWXgI %}

Working with observability sometimes looks like building a puzzle with curvy edge pieces. The task at hands is to understand each piece and never lose track of the big picture. The problem is, with observability, knowing how the big picture looks like is not always possible. In this session, Liz Fong-Jones from Honeycomb shares her extensive experience with SRE and observability and shows us how observability looks like and how to pull each technology together to build that bigger picture. With an outstanding end to end example of troubleshooting a problem, fixing it in production, and coming back to development to apply the lessons learned with instrumentation, Liz shares when and how to use observability technologies to improve the system's reliability and customer satisfaction. 

Also, to quote Liz from the day: "Do not be afraid of failure, failure is a learning opportunity. Break you systems as long as you learn from it, so you don't break them the same way twice!"

![Liz running a demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/llexy74y8acxlb8wjhh0.png)

### Some links that were shared:
- [Intro to O11y](https://glitch.com/~intro-to-o11y-nodejs)
- [Pyroscope](https://pyroscope.io/)
- [Liz's blog](https://www.lizthegrey.com/)