The four Golden Signals of Kubernetes monitoring
Tags: kubernetes, monitoring, prometheus
Published: 2022-10-28 09:20:10

---
title: The four Golden Signals of Kubernetes monitoring
published: true
description: Have you heard of the Four Golden Signals of Monitoring? They are a set of essential metrics to look in your system. 
tags: kubernetes, monitoring, prometheus
cover_image: https://sysdig.com/wp-content/uploads/BlogImages-GHAminer-featured-1.png
canonical_url: https://sysdig.com/blog/golden-signals-kubernetes/
# Use a ratio of 100:42 for best results.
---

<!-- wp:paragraph -->
<p><strong>Golden Signals</strong> are a reduced set of metrics that offer a wide view of a service from a user or consumer perspective: Latency, Traffic, Errors and Saturation. By focusing on these, you can be quicker at detecting potential problems that might be directly affecting the behavior of the application.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Google introduced the term "Golden Signals" to refer to the essential metrics that you need to measure in your applications. They are the following:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><a rel="noopener nofollow noreferrer" href="#errors">Errors</a> - rate of requests that fail.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a rel="noopener nofollow noreferrer" href="#saturation">Saturation</a> - consumption of your system resources.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a rel="noopener nofollow noreferrer" href="#traffic">Traffic</a> - amount of use of your service per time unit.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a rel="noopener nofollow noreferrer" href="#latency">Latency</a> - the time it takes to serve a request.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:image {"align":"center","id":60373,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image9-10.png"><img src="https://sysdig.com/wp-content/uploads/image9-10-1170x644.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60373"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><a href="https://sysdig.com/wp-content/uploads/BlogImages-GHAminer-featured-1.png"><br></a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is just a set of essential signals to start monitoring in your system. In other words, if you’re wondering which signals to monitor, you will need to look at these four first.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter: Goldilocks and the four Monitoring Signals</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><!-- wp:paragraph -->
<p>Once upon a time, there was a little girl called Goldilocks, who lived at the other side of the wood and had been sent on an errand by her mother, passed by the house, and looked in at the window…</p>
<!-- /wp:paragraph --></blockquote>
<!-- /wp:quote -->

<!-- wp:heading -->
<h2 id="errors">Errors</h2>
<!-- /wp:heading -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><!-- wp:paragraph -->
<p>Goldilocks then tried the little chair, which belonged to the Little Bear, and found it just right, but she sat in it so hard that she broke it.</p>
<!-- /wp:paragraph --></blockquote>
<!-- /wp:quote -->

<!-- wp:image {"align":"center","id":60368,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image4-19.png"><img src="https://sysdig.com/wp-content/uploads/image4-19-1170x644.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60368"/></a><figcaption class="wp-element-caption">The error rate for the chairs is ⅓</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The <strong>Errors</strong> golden signal measures the rate of requests that fail.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note that measuring the bulk amount of errors might not be the best course of action. If your application has a sudden peak of requests, then logically the amount of failed requests may increase.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>That’s why usually monitoring systems focus on the error rate, calculated as the percent of calls that are failing from the total.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you’re managing a web application, typically you will discriminate between those calls returning HTTP status in the 400-499 range (client errors) and 500-599 (server errors).</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Measuring errors in Kubernetes</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>One thermometer for the errors happening in Kubernetes is the Kubelet. You can use several Kubernetes State Metrics in Prometheus to measure the amount of errors.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The most important one is <code>kubelet_runtime_operations_errors_total</code>, which indicates low level issues in the node, like problems with container runtime.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you want to visualize errors per operation, you can use <code>kubelet_runtime_operations_total</code> to divide.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Errors example</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here's the Kubelet Prometheus metric for error rate in a Kubernetes cluster:</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>sum(rate(kubelet_runtime_operations_errors_total{cluster="",
job="kubelet", metrics_path="/metrics"}&#91;$__rate_interval])) 
by (instance, operation_type)
</code></pre>
<!-- /wp:code -->

<!-- wp:image {"align":"center","id":60380,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image16-2-1170x445.png"><img src="https://sysdig.com/wp-content/uploads/image16-2-1170x445.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60380"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="saturation">Saturation</h2>
<!-- /wp:heading -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><!-- wp:paragraph -->
<p>Goldilocks tasted the porridge in the dear little bowl, and it was just right, and it tasted so good that she tasted and tasted, and tasted and tasted until she was full.</p>
<!-- /wp:paragraph --></blockquote>
<!-- /wp:quote -->

<!-- wp:image {"align":"center","id":60381,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image17-2-1170x644.png"><img src="https://sysdig.com/wp-content/uploads/image17-2-1170x644.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60381"/></a><figcaption class="wp-element-caption">After eating one small bowl, Goldilocks is unable to eat more. That’s saturation.</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>Saturation</strong> measures the consumption of your system resources, usually as a percentage of the maximum capacity. Examples include:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>CPU usage</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Disk space</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Memory usage</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Network bandwidth</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>In the end, cloud applications run on machines, which have a limited amount of these resources.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In order to correctly measure, you should be aware of the following:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>What are the consequences if the resource is depleted? It could be that your entire system is unusable because this space has run out. Or maybe further requests are throttled until the system is less saturated.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Saturation is not always about resources about to be depleted. It’s also about over-resourcing, or allocating a higher quantity of resources than what is needed. This one is crucial for cost savings.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Measuring saturation in Kubernetes</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Since saturation depends on the resource being observed, you can use different metrics for Kubernetes entities:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><code>node_cpu_seconds_total</code> to measure machine CPU utilization.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><code>container_memory_usage_bytes</code> to measure the memory utilization at container level (paired with <code>container_memory_max_usage_bytes</code>).</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>The amount of Pods that a <a target="_blank" href="https://sysdig.com/learn-cloud-native/kubernetes-101/what-is-a-kubernetes-node/" rel="noopener">Node</a> can contain is also a Kubernetes resource.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Saturation example</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here’s a PromQL example of a Saturation signal, measuring CPU usage percent in a Kubernetes node.</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>100 - (avg by (instance) (rate(node_cpu_seconds_total{}&#91;5m])) * 100)
</code></pre>
<!-- /wp:code -->

<!-- wp:image {"align":"center","id":60377,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image13-8-1170x410.png"><img src="https://sysdig.com/wp-content/uploads/image13-8-1170x410.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60377"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="traffic">Traffic</h2>
<!-- /wp:heading -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><!-- wp:paragraph -->
<p>And the Middle-sized Bear said:<br>“Somebody has been tumbling my bed!”<br>And the Little bear piped:<br>“Somebody has been tumbling my bed, and here she is!”</p>
<!-- /wp:paragraph --></blockquote>
<!-- /wp:quote -->

<!-- wp:image {"align":"center","id":60376,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image12-7-1170x644.png"><img src="https://sysdig.com/wp-content/uploads/image12-7-1170x644.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60376"/></a><figcaption class="wp-element-caption">One of the beds is being used, but none should be being used instead. That’s an unusual traffic.</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>Traffic</strong> measures the amount of use of your service per time unit.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In essence, this will represent the usage of your current service. This is important not only for business reasons, but also to detect anomalies.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Is the amount of requests too high? This could be due to a peak of users or because of a misconfiguration causing retries.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Is the amount of requests too low? That may reflect that one of your systems is failing.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Still, traffic signals should always be measured with a time reference. As an example, this blog receives more visits from Tuesday to Thursday.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Depending on your application, you could be measuring traffic by:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Requests per minute for a web application</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Queries per minute for a database application</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Endpoint requests per minute for an API</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Traffic example</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here’s a Google Analytics chart displaying traffic distributed by hour:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":60367,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-full"><a href="https://sysdig.com/wp-content/uploads/image3-23.png"><img src="https://sysdig.com/wp-content/uploads/image3-23.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60367"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="latency">Latency</h2>
<!-- /wp:heading -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><!-- wp:paragraph -->
<p>At that, Goldilocks woke in a fright, and jumped out of the window and ran away as fast as her legs could carry her, and never went near the Three Bears’ snug little house again.</p>
<!-- /wp:paragraph --></blockquote>
<!-- /wp:quote -->

<!-- wp:image {"align":"center","id":60375,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image11-6-1170x644.png"><img src="https://sysdig.com/wp-content/uploads/image11-6-1170x644.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60375"/></a><figcaption class="wp-element-caption">Goldilocks ran down the stairs in just two seconds. That’s a very low latency.</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>Latency</strong> is defined as the time it takes to serve a request.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Average latency</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>When working with latencies, your first impulse may be to measure average latency, but depending on your system that might not be the best idea. There may be very fast or very slow requests distorting the results.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Instead, consider using a percentile, like p99, p95, and p50 (also known as median) to measure how the fastest 99%, 95%, or 50% of requests, respectively, took to complete.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Failed vs. successful</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>When measuring latency, it’s also important to discriminate between failed and successful requests, as failed ones might take sensibly less time than the correct ones.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Apdex Score</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As described above, latency information may not be informative enough:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Some users might perceive applications as slower, depending on the action they are performing.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Some users might perceive applications as slower, based on the default latencies of the industry.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>This is where the Apdex (Application Performance Index) comes in. It’s defined as:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":60365,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image size-full"><a href="https://sysdig.com/wp-content/uploads/image1-37.png"><img src="https://sysdig.com/wp-content/uploads/image1-37.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60365"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Where t is the target latency that we consider as reasonable.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Satisfied will represent the amount of users with requests under the target latency.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Tolerant will represent the amount of non-satisfied users with requests below four times the target latency.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Frustrated will represent the amount of users with requests above the tolerant latency.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>The output for the formula will be an index from 0 to 1, indicating how performant our system is in terms of latency.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Measuring latency in Kubernetes</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In order to measure the latency in your Kubernetes cluster, you can use metrics like <code>http_request_duration_seconds_sum</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can also measure the latency for the api-server by using Prometheus metrics like <code>apiserver_request_duration_seconds</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Latency example</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here’s an example of a Latency PromQL query for the 95% best performing HTTP requests in Prometheus:</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>histogram_quantile(0.95, sum(rate(prometheus_http_request_duration_seconds_bucket&#91;5m]))
by (le))
</code></pre>
<!-- /wp:code -->

<!-- wp:image {"align":"center","id":60374,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image10-10-1170x408.png"><img src="https://sysdig.com/wp-content/uploads/image10-10-1170x408.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60374"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="red-method">RED Method</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The <strong>RED Method</strong> was created by Tom Wilkie, from Weaveworks. It is heavily inspired by the Golden Signals and it’s focused on microservices architectures.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>RED stands for:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Rate</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Error</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Duration</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong>Rate</strong> measures the number of requests per second (equivalent to Traffic in the Golden Signals).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Error</strong> measures the number of failed requests (similar to the one in Golden Signals).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Duration</strong> measures the amount of time to process a request (similar to Latency in Golden Signals).</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="use-method">USE Method</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The <strong>USE Method</strong> was created by Brendan Gregg and it’s used to measure infrastructure.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>USE stands for:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Utilization</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Saturation</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Errors</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>That means for every resource in your system (CPU, disk, etc.), you need to check the three elements above.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Utilization</strong> is defined as the percentage of usage for that resource.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Saturation</strong> is defined as the queue for requests in the system.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Errors</strong> is defined as the number of errors happening in the system.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>While it may not be intuitive, Saturation in Golden Signals is not similar to the Saturation in USE, but rather Utilization.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":60372,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://sysdig.com/wp-content/uploads/image8-12-1170x439.png"><img src="https://sysdig.com/wp-content/uploads/image8-12-1170x439.png" alt="The four Golden Signals of Kubernetes monitoring" class="wp-image-60372"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="practical-example">A practical example of Golden signals in Kubernetes</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As an example to illustrate the use of Golden Signals, here’s a simple go application example with Prometheus instrumentation. This application will apply a random delay between 0 and 12 seconds in order to give usable information of latency. Traffic will be generated with curl, with several infinite loops.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>An <a rel="noopener nofollow noreferrer" target="_blank" href="https://prometheus.io/docs/practices/histograms/">histogram</a> was included to collect metrics related to latency and requests. These metrics will help us obtain the initial three Golden Signals: latency, request rate and error rate. To obtain saturation directly with Prometheus and node-exporter, use percentage of CPU in the nodes.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><script src="https://gist.github.com/b7678e205e27c34f446d86e54dedccc0.js"></script><br><noscript></noscript></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>
File: main.go
-------------
package main
import (
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "time"
    "github.com/gorilla/mux"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
    //Prometheus: Histogram to collect required metrics
    histogram := prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "greeting_seconds",
        Help:    "Time take to greet someone",
        Buckets: &#91;]float64{1, 2, 5, 6, 10}, //Defining small buckets as this app should not take more than 1 sec to respond
    }, &#91;]string{"code"}) //This will be partitioned by the HTTP code.
    router := mux.NewRouter()
    router.Handle("/sayhello/{name}", Sayhello(histogram))
    router.Handle("/metrics", promhttp.Handler()) //Metrics endpoint for scrapping
    router.Handle("/{anything}", Sayhello(histogram))
    router.Handle("/", Sayhello(histogram))
    //Registering the defined metric with Prometheus
    prometheus.Register(histogram)
    log.Fatal(http.ListenAndServe(":8080", router))
}
func Sayhello(histogram *prometheus.HistogramVec) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        //Monitoring how long it takes to respond
        start := time.Now()
        defer r.Body.Close()
        code := 500
        defer func() {
            httpDuration := time.Since(start)
            histogram.WithLabelValues(fmt.Sprintf("%d", code)).Observe(httpDuration.Seconds())
        }()
        if r.Method == "GET" {
            vars := mux.Vars(r)
            code = http.StatusOK
            if _, ok := vars&#91;"anything"]; ok {
                //Sleep random seconds
                rand.Seed(time.Now().UnixNano())
                n := rand.Intn(2) // n will be between 0 and 3
                time.Sleep(time.Duration(n) * time.Second)
                code = http.StatusNotFound
                w.WriteHeader(code)
            }
            //Sleep random seconds
            rand.Seed(time.Now().UnixNano())
            n := rand.Intn(12) //n will be between 0 and 12
            time.Sleep(time.Duration(n) * time.Second)
            name := vars&#91;"name"]
            greet := fmt.Sprintf("Hello %s \n", name)
            w.Write(&#91;]byte(greet))
        } else {
            code = http.StatusBadRequest
            w.WriteHeader(code)
        }
    }
}
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The application was deployed in a Kubernetes cluster with Prometheus and Grafana, and generated a dashboard with Golden Signals. In order to obtain the data for the dashboards, these are the PromQL queries:</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Latency:</h3>
<!-- /wp:heading -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>sum(greeting_seconds_sum)/sum(greeting_seconds_count)  //Average
histogram_quantile(0.95, sum(rate(greeting_seconds_bucket&#91;5m])) by (le)) //Percentile p95</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Request rate:</h3>
<!-- /wp:heading -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>sum(rate(greeting_seconds_count{}&#91;2m]))  //Including errors
rate(greeting_seconds_count{code="200"}&#91;2m])  //Only 200 OK requests</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Errors per second:</h3>
<!-- /wp:heading -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>sum(rate(greeting_seconds_count{code!="200"}&#91;2m]))</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Saturation:</h3>
<!-- /wp:heading -->

<!-- wp:code {"language":"javascript"} -->
<pre class="wp-block-code"><code>100 - (avg by (instance) (irate(node_cpu_seconds_total{}&#91;5m])) * 100)</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2 id="conclusion">Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Golden Signals, RED, and USE are just guidelines on what you should be focusing on when looking at your systems. But these are just the bare minimum on what to measure.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Understand the <strong>errors</strong> in your system. They will be a thermometer of all the other metrics, as they will point to any unusual behavior. And remember that you need to correctly mark requests as erroneous, but only the ones that should be exceptionally incorrect. Otherwise, your system will be prone to false positives or false negatives.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Measure <strong>latency</strong> of your requests. Try to understand your bottlenecks and what the negative experiences are when latency is higher than expected.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Visualize <strong>saturation</strong> and understand the resources involved in your solution. What are the consequences if a resource gets depleted?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Measure <strong>traffic</strong> to understand your usage curves. You will be able to find the best time to take down your system for an update, or you could be alerted when there’s an unexpected amount of users.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Once metrics are in place, it’s important to set up alerts, which will notify you in case any of these metrics reach a certain threshold.</p>
<!-- /wp:paragraph -->

<!-- wp:separator -->
<hr class="wp-block-separator has-alpha-channel-opacity"/>
<!-- /wp:separator -->

<!-- wp:html -->
<div class="sysdig-box" style="background-color: #f8f8f8; padding: 2em;">
<h2 style="margin-top: 0;">Track golden signals easily with Sysdig Monitor</h2>
<p>With Sysdig Monitor, you can quickly review the golden signals in your system, out of the box.</p>
<p>Review easily the Latency, Errors, Saturation and Traffic for the Pods in your cluster. And thanks to its Container Observability with eBPF, you can do this without adding any app or code instrumentation.</p>
<p>Sysdig Advisor accelerates mean time to resolution (MTTR) with live logs, performance data, and suggested remediation steps. It’s the easy button for Kubernetes troubleshooting!</p>
</div>
<!-- /wp:html -->

<!-- wp:image {"id":58085,"linkDestination":"custom"} -->
<figure class="wp-block-image"><a href="https://sysdig.com/wp-content/uploads/GoldenSignals-11.png"><img src="https://sysdig.com/wp-content/uploads/GoldenSignals-11.png" alt="Sysdig Monitor Golden Signals" class="wp-image-58085"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><a target="_blank" href="https://sysdig.com/company/free-trial-monitor/" rel="noopener">Try it free for 30 days!</a></p>
<!-- /wp:paragraph -->