Top metrics for Elasticsearch monitoring with Prometheus
Tags: elasticsearch, prometheus, monitoring, devops
Published: 2023-05-09 08:18:13

---
title: Top metrics for Elasticsearch monitoring with Prometheus
published: true
description: Elasticsearch is one of the most important search engines available. Read this article for insights on how monitor the right metrics.

tags: elasticsearch, prometheus, monitoring, devops
cover_image: https://sysdig.com/wp-content/uploads/BlogImages-TopMetricsInElasticsearch-featured.png
canonical_url: https://sysdig.com/blog/elasticsearch-monitoring/
# Use a ratio of 100:42 for best results.
---

<!-- wp:paragraph -->
<p>Starting the journey for Elasticsearch monitoring is crucial to get the right visibility and transparency over its behavior.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Elasticsearch is the most used <strong>search and analytics engine</strong>. It provides both scalability and redundancy to provide a high-availability search. As of 2023, more than sixty thousand companies of all sizes and backgrounds are using it as their search solution to track a diverse range of data, like analytics, logging, or business information.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By distributing data in JSON documents and indexing that data into several shards, Elastic search provides high availability, quick search, and redundancy capabilities.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In this article, we will evaluate the most important Prometheus metrics provided by the Elasticsearch exporter.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You will learn what are the main areas to focus on when monitoring an Elasticsearch system:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><a href="#start-monitoring">Start monitoring Elasticsearch with Prometheus.</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#monitor-golden-signals">How to monitor Golden Signals.</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#monitor-elasticsearch-infra-metrics">How to monitor infra metrics.</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#monitor-elasticsearch-index-performance">How to monitor index performance.</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#monitor-elasticsearch-search-performance">How to monitor search performance.</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#monitor-elasticsearch-cluster">How to monitor cluster performance.</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#advanced-monitoring">Advanced monitoring and next steps</a></li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="start-monitoring">How to start monitoring ElasticSearch with Prometheus</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As usual, the easiest way to start your Prometheus monitoring journey with Elasticsearch is to use <a href="https://promcat.io" target="_blank" rel="noopener nofollow noreferrer">PromCat.io</a> to find the best configs, dashboards, and alerts. The <a href="https://promcat.io/apps/elasticsearch#SetupGuide" target="_blank" rel="noopener nofollow noreferrer">Elasticsearch setup guide in Promcat</a> includes the Elasticsearch exporter with a series of out-of-box metrics that will be automatically scrapped to Prometheus. It also includes a collection of curated alerts and dashboards to start monitoring Elasticsearch right away.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":71755,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://sysdig.com/wp-content/uploads/Top-metrics-elasticsearch-image1-1170x383.png" alt="Top metrics for Elasticsearch - metric list" class="wp-image-71755" title="image_tooltip"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>You can combine these metrics with the <a href="https://sysdig.com/blog/exporters-and-target-labels/" target="_blank" rel="noopener">Node Exporter</a> to get more insights into your infrastructure. Also, if you're running Elasticsearch on Kubernetes, you can <a href="https://sysdig.es/blog/kubernetes-monitoring-prometheus/#kube-state-metrics" target="_blank" rel="noopener nofollow noreferrer">use KSM and CAdvisor</a> to combine Kubernetes metrics with Elasticsearch metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="monitor-golden-signals">How to monitor Golden Signals in Elasticsearch</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To review a bare minimum of important metrics, remember to check the so-called <a href="https://sysdig.com/blog/golden-signals-kubernetes/" target="_blank" rel="noopener">Golden Signals</a>:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Errors.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Traffic.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Saturation.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Latency.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>These represent a set of the essential metrics to look for in a system, in order to track black-box monitoring (focus only on what’s happening in the system, not why). In other words, Golden Signals will measure symptoms, not solutions to the current problem. This could be a good starting point for creating an Elasticsearch monitoring dashboard.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Errors</h3>
<!-- /wp:heading -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading" style="margin-top:0">elasticsearch_cluster_health_status</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Cluster health in Elasticsearch is measured by the colors green, yellow, and red, as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><strong>Green</strong>: Data integrity is correct, no shard is missing.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><strong>Yellow</strong>: There’s at least one shard missing, but data integrity can be preserved due to replicas.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><strong>Red</strong>: A primary shard is missing or unassigned, and there’s a data loss.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>With <code>elasticsearch_cluster_health_status</code>, you can quickly check the current situation for Elasticsearch data on a particular cluster. Remember that this won’t retrieve the actual causes of the data integrity loss, just that you need to act in order to prevent further problems.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Traffic</h3>
<!-- /wp:heading -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading" style="margin-top:0">elasticsearch_indices_search_query_total</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This metric is a counter with the total number of search queries executed, which by itself won’t give you much information as a number.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><span style="text-size: smaller; font-style: italic; border: 1px solid #FF9E1B; display: block; border-radius: 25px; padding: 25px; max-width: 96%; margin: 2em auto;"><em>Consider as well using <code>rate()</code> or <code>irate()</code>, to detect sudden changes or spikes in traffic. Dig deeper into Prometheus queries with our <a href="https://sysdig.com/blog/getting-started-with-promql-cheatsheet/" target="_blank" rel="noreferrer noopener">Getting started with PromQL guide</a></em><span style="font-size: revert; color: initial; font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Oxygen-Sans, Ubuntu, Cantarell, &quot;Helvetica Neue&quot;, sans-serif;"></span></span></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Saturation</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>For a detailed latency analysis, check the section on <a href="#monitor-elasticsearch-infra-metrics">How to monitor Elasticsearch infra metrics</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Latency</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>For a detailed latency analysis, check the section on <a href="#monitor-elasticsearch-index-performance">How to monitor Elasticsearch index performance</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="monitor-elasticsearch-infra-metrics">How to monitor Elasticsearch infra metrics</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Infrastructure monitoring focuses on tracking the overall performance of the servers and nodes of a system. As with similar cloud applications, most of the effort will be spent on monitoring CPU and Memory consumption. </p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Monitoring Elasticsearch CPU</h3>
<!-- /wp:heading -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading" style="margin-top:0">elasticsearch_process_cpu_percent</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This is a gauge metric used to measure the current CPU usage percent (0-100) of the Elasticsearch process. Since chances are that you’re running several Elasticsearch nodes, you will need to track each one separately.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":71757,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://sysdig.com/wp-content/uploads/Top-metrics-elasticsearch-image2-1170x553.png" alt="Top metrics for Elasticsearch - CPU usage" class="wp-image-71757" title="image_tooltip"/></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">elasticsearch_indices_store_throttle_time_seconds_total</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In case you’re using a file system as an index store, you can expect a certain level of delays in input and output operations. This metric represents how much your Elasticsearch index store is being throttled.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Since this is a counter metric that will only aggregate the total number of seconds, consider using <code>rate</code> or <code>irate</code> for an evaluation of how much it’s suddenly changing.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Monitoring Elasticsearch JVM Memory</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Elasticsearch is based on <a rel="noopener nofollow noreferrer" target="_blank" href="https://lucene.apache.org/">Lucene</a>, which is built in Java. This means that monitoring the Java Virtual Machine (JVM) memory is crucial to understand the current usage of the whole system.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">elasticsearch_jvm_memory_used_bytes</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This metric is a gauge that represents the memory usage in bytes for each area.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":71758,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://sysdig.com/wp-content/uploads/Top-metrics-elasticsearch-image3-1170x455.png" alt="Top metrics for Elasticsearch - Memory used" class="wp-image-71758" title="image_tooltip"/></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="monitor-elasticsearch-index-performance">How to monitor Elasticsearch index performance</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Indices in ElasticSearch partition the data as a logical namespace. Elasticsearch indexes documents in order to retrieve or search them as fast as possible.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Every time a new index is created, you can define the number of shards and replicas for it:</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"yaml"} -->
<pre class="wp-block-code"><code>{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">elasticsearch_indices_indexing_index_time_seconds_total</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This metric is a counter of the seconds accumulated spent on indexing. It can give you a very approximated idea of the Elasticsearch indexing performance.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note that you can divide this metric by elasticsearch_indices_indexing_index_total in order to get the average indexing time per operation.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">elasticsearch_indices_refresh_time_seconds_total</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>For an index to be searchable, Elasticsearch needs a refresh to be executed. This is set up with the config index.refresh_interval, which is set by default to one minute.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This metric elasticsearch_indices_refresh_time_seconds_total represents a counter with the total time dedicated to refreshing in Elasticsearch.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In case you want to measure the average time for refresh, you can divide this metric by <code>elasticsearch_indices_refresh_total</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="monitor-elasticsearch-search-performance">How to monitor Elasticsearch search performance</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>While Elasticsearch promises near-instant query speed, chances are that in the real world, you may feel that is not the case. The number of shards, the storage solution chosen, or the cache configuration might impact search performance, and it’s crucial to track what is the current behavior.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Additionally, the usage of wildcards, joins or the number of fields being searched will affect drastically the overall processing time of search queries.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading" style="margin-top:0">elasticsearch_indices_search_fetch_time_seconds</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>A counter metric aggregating the total amount of seconds dedicated to fetching results in search.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In case you want to retrieve the average fetch time per operation, just divide the result by <code>elasticsearch_indices_search_fetch_total</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="monitor-elasticsearch-cluster">How to monitor Elasticsearch cluster performance</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Apart from the usual cloud requirements, an Elasticsearch system would like to look at:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Number of shards.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Number of replicas.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>As a rule of thumb, the ratio between the number of shards and GB of heap space should be less than 20.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note as well that it’s suggested to have a separate cluster dedicated to monitoring.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">elasticsearch_cluster_health_active_shards</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This metric is a gauge that will indicate the number of active shards (both primary and replicas) from all the clusters.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">elasticsearch_cluster_health_relocating_shards</h5>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Elasticsearch will dynamically move shards between nodes based on balancing or current usage. With this metric, you can control when this movement is happening.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 class="wp-block-heading" id="advanced-monitoring">Advanced Monitoring</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Remember that the Prometheus exporter will give you a set of out-of-the-box metrics that are relevant enough to kickstart your monitoring journey. But the real challenge comes when you take the step to create your own custom metrics tailored to your application.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">REST API</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Additionally, mind that Elasticsearch <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html">provides a REST API</a> that you can query for more fine-grained monitoring.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">VisualVM</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The <a rel="noopener nofollow noreferrer" target="_blank" href="https://visualvm.github.io/">Java VisualVM</a> project is an advanced dashboard for Memory and CPU monitoring. It features advanced resource visualization, as well as process and thread utilization.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 class="wp-block-heading">Download the Dashboards</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>You can download the dashboards with the metrics seen in this article <a rel="noopener nofollow noreferrer" target="_blank" href="https://promcat.io/apps/elasticsearch">through the Promcat official page</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is a curated selection of the above metrics that can be easily integrated with your Grafana or Sysdig Monitor solution.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":71759,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://sysdig.com/wp-content/uploads/Top-metrics-elasticsearch-image4-1170x739.png" alt="Top metrics for Elasticsearch - Grafana dashboards" class="wp-image-71759" title="image_tooltip"/></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 class="wp-block-heading">Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Elasticsearch is one of the most important search engines available, featuring high availability, high scalability, and distributed capabilities through redundancy.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Using the Elasticsearch exporter for Prometheus you can kickstart the monitoring journey in an easy way, by automatically receiving the important metrics directly.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>As with many other applications, CPU, and Memory are crucial to understand system saturation. You should be aware of the current CPU throttling and the memory handling of the JVM.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Finally, it’s important to dig deeper into the particularities of Elasticsearch, like indices and search capabilities, to truly understand the challenges of monitoring and visualization.</p>
<!-- /wp:paragraph -->

<!-- wp:separator -->
<hr class="wp-block-separator has-alpha-channel-opacity"/>
<!-- /wp:separator -->

<!-- wp:acf/cta-form {"name":"acf/cta-form","data":{"title":"Want to dig deeper? Download the Prometheus monitoring guide","_title":"field_62fa7265767bf","sub_title":"Start your monitoring journey with the standard metrics solution ","_sub_title":"field_62fa7293767c0","is_form":"0","_is_form":"field_62fa72a6767c1","cta_one":{"title":"Download","url":"https://sysdig.com/content/c/pf-prometheus-monitoring-guide?x=u_WFRi\u0026utm_source=gated-organic\u0026utm_medium=website#page=1","target":"_blank"},"_cta_one":"field_62fa72ed767c4","cta_two":"","_cta_two":"field_62fa7305767c5","form":"","_form":"field_632b87c7b0724"},"mode":"auto"} /-->