Monitoring with Custom Metrics
Tags: prometheus, monitoring, devops
Published: 2023-03-02 10:19:53

---
title: Monitoring with Custom Metrics
published: true
description: Custom Metrics are an essential part of cloud monitoring, since they provide a way to start the observability journey into application level and business related data. Read more on this article:

tags: prometheus, monitoring, devops
cover_image: https://sysdig.com/wp-content/uploads/BlogImages-MonitoringWithCustomMetrics-featured.png
canonical_url: https://sysdig.com/blog/monitoring-custom-metrics/
# Use a ratio of 100:42 for best results.
---
<!-- wp:paragraph -->
<p><strong>Custom metrics</strong> are application-level or business-related tailored metrics, as opposed to the ones that come directly out-of-the-box from monitoring systems like Prometheus (e.g: kube-state-metrics or node exporter)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By kickstarting a monitoring project with Prometheus, you might realize that you get an initial set of out-of-the-box metrics with just Node Exporter and Kube State Metrics. But, this will only get you so far since you will just be performing black box monitoring. How can you go to the next level and observe what’s beyond?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>They are an essential part of the day-to-day monitoring of cloud-native systems, as they provide an additional dimension to the business and app level.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Metrics provided by an exporter</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Tailored metrics designed by the customer</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>An aggregate from previous existing metrics</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>In this article, you will see:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li> <a href="https://docs.google.com/document/d/19Z6q_lY60IqRz4DcnubmU4U1HwjxBPJJy3q_okC6-rE/edit#heading=h.hl0n2legtxy2">Why custom metrics are important</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="https://docs.google.com/document/d/19Z6q_lY60IqRz4DcnubmU4U1HwjxBPJJy3q_okC6-rE/edit#heading=h.1pkldw1lpoee">When to use custom metrics</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="https://docs.google.com/document/d/19Z6q_lY60IqRz4DcnubmU4U1HwjxBPJJy3q_okC6-rE/edit#heading=h.7o1c1smwtm7b">Considerations when creating custom metrics</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="https://docs.google.com/document/d/19Z6q_lY60IqRz4DcnubmU4U1HwjxBPJJy3q_okC6-rE/edit#heading=h.8hxg63te0986">Kubernetes Metric API</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="https://docs.google.com/document/d/19Z6q_lY60IqRz4DcnubmU4U1HwjxBPJJy3q_okC6-rE/edit#heading=h.q5qjsc4mpwar">Prometheus custom metrics</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="https://docs.google.com/document/d/19Z6q_lY60IqRz4DcnubmU4U1HwjxBPJJy3q_okC6-rE/edit#heading=h.lfrto1shfhfv">Challenges when using custom metrics</a></li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Why custom metrics are important</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Custom metrics allow companies to:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Monitor Key Performance Indicators (KPIs).</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Detect issues faster.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Track resource utilization.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Measure latency.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Track specific values from their services and systems.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Examples of custom metrics:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Latency of transactions in milliseconds.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Database open connections.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>% cache hits / cache misses.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>orders/sales in e-commerce site.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>% of slow responses.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>% of responses that are resource intensive.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>As you can see, any metrics retrieved from an exporter or created ad hoc will fit into the definition for custom metric.</p>
<!-- /wp:paragraph -->

<!-- wp:block {"ref":67741} /-->

<!-- wp:heading -->
<h2>When to use Custom Metrics</h2>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3>Autoscaling</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By providing specific visibility over your system, you can define rules on how the workload should scale.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Horizontal autoscaling: add or remove replicas of a Pod.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Vertical autoscaling: modify limits and requests of a container.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Cluster autoscaling: add or remove nodes in a cluster.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>If you want to dig deeper, check <a target="_blank" href="https://sysdig.com/blog/kubernetes-autoscaler/" rel="noopener">this article about autoscaling in Kubernetes</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Latency monitoring</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Latency measures the time it takes for a system to serve a request. This monitoring <a href="https://sysdig.com/blog/golden-signals-kubernetes/" target="_blank" rel="noopener">golden signal</a> is essential to understand what the end-user experience for your application is.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>These are considered custom metrics as they are not part of the out-of-the-box set of metrics coming from Kube State Metrics or Node Exporter. In order to measure latency, you might want to either track individual systems (database, API) or end-to-end.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Application level monitoring</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kube-state-metrics or node-exporter might be a good <a href="https://sysdig.com/blog/cloud-monitoring-journey/" target="_blank" rel="noopener">starting point for observability</a>, but they just scratch the surface as they perform black-box monitoring. By instrumenting your own application and services, you create a curated and personalized set of metrics for your own particular case.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Considerations when creating Custom Metrics</h2>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3>Naming</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Check for any <a href="https://prometheus.io/docs/practices/naming/" target="_blank" rel="noreferrer noopener">existing convention on naming</a>, as they might be either colliding with existing names or confusing. Custom metric name is the first description for its purpose.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Labels</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Thanks to labels, we can add parameters to our metrics, as we will be able to filter and refine through additional characteristics. Cardinality is the number of possible values for each label and since each combination of possible values will require a time series entry, that can increase resources drastically. Choosing the correct labels carefully is key to avoiding this cardinality explosion, which is one of the causes of resource spending spikes.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Costs</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Custom metrics may have some costs associated with them depending on the monitoring system you are using. Double-check what is the dimension used to scale costs:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Number of time series</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Number of labels</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Data storage</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Custom Metric lifecycle</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In case the Custom Metric is related to a job or a short-living script, consider using <a href="https://github.com/prometheus/pushgateway" target="_blank" rel="noopener nofollow noreferrer">Pushgateway</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Kubernetes Metric API</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>One of the most important features of Kubernetes is the ability to scale the workload based on the values of metrics automatically.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Metrics API are defined in the <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/kubernetes/metrics">official repository from Kubernetes</a>:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><code>metrics.k8s.io</code></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><code>custom.metrics.k8s.io</code></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><code>external.metrics.k8s.io</code></li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Creating new metrics</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>You can set new metrics by calling the K8s metrics API as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">curl -X POST \
  -H 'Content-Type: application/json' \
  http://localhost:8001/api/v1/namespaces/custom-metrics/services/custom-metrics-apiserver:http/proxy/write-metrics/namespaces/default/services/kubernetes/test-metric \
  --data-raw '"300m"'
</pre>
<!-- /wp:preformatted -->

<!-- wp:heading -->
<h2>Prometheus custom metrics</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As we mentioned, every exporter that we include in our Prometheus integration will account for several custom metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Check the following post for a <a target="_blank" href="https://sysdig.com/blog/prometheus-metrics/" rel="noopener">detailed guide on Prometheus metrics</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Challenges when using custom metrics</h2>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3>Cardinality explosion</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>While the resources consumed by some metrics might be negligible, the moment these are available to be used with labels in queries, things might get out of hand.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Cardinality refers to the cartesian products of metrics and labels. The result will be the amount of time series entries that need to be used for that single metric.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":67815,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://sysdig.com/wp-content/uploads/Custom-metrics-image-1-1-1170x644.png" alt="Custom metrics - cardinality example" class="wp-image-67815"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Also, every metric will be scraped and stored in a time series database based on your <code>scrape_interval.</code> The higher this value, the higher the amount of time series entries.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>All these factors will eventually lead to:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Higher resource consumption.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Higher storage demand.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Monitoring performance degradation.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Moreover, most common monitoring tools don’t give <a target="_blank" href="https://sysdig.com/use-cases/cloud-monitoring/" rel="noopener">visibility on current cardinality of metrics or costs associated</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Exporter over usage</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Exporters are a great way to include relevant metrics to your system. With them, you can easily instrument relevant metrics bound to your microservices and containers. But with great power comes great responsibility. Chances are that many of the metrics included in the package may not be relevant to your business at all.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By enabling custom metrics and exporters in your solution, you may end up having a burst in the amount of time series database entries.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Cost spikes</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Because of the elements explained above, monitoring costs could increase suddenly, as your current solution might be consuming more resources than expected, or your current monitoring solution has certain thresholds that were surpassed.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Alert fatigue</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>With metrics, most companies and individuals would love to start adding alerts and notifications when their values exceed certain thresholds. However, this could lead to higher notification sources and a reduced attention span.<br>Learn more about <a target="_blank" href="sysdig.com/blog//prometheus-alertmanager" rel="noopener">Alert Fatigue and how to mitigate it</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Custom metrics represent the next step for cloud-native monitoring as they represent the core of business observability. While using Prometheus along kube-state-metrics and node exporter is a nice starting step, eventually companies and organizations will need to take the next step and create tailored and on-point metrics to suit their needs.</p>
<!-- /wp:paragraph -->

<!-- wp:block {"ref":67433} /-->