JMeter, Prometheus, and Grafana Integration
Tags: prometheus, grafana, jmeter, performancetesting
Published: 2020-03-05 01:22:38

This post originally published at [QAInsights Blog](https://qainsights.com/jmeter-prometheus-and-grafana-integration/).

<!-- wp:paragraph -->
<p>You can check out my videos about LoadRunner integration with Grafana in my <a href="https://www.youtube.com/user/QAInsights/search?query=grafana">YouTube channel</a>. Recently I have started working extensively on JMeter, primarily in HTTP, MQTT, and JDBC Sampler and will be working on NeoLoad pretty soon. I got a challenge in my projects where developers would like to monitor the critical metrics when the performance test is on. I started sharing the Dynatrace dashboard link, but for non-technical members Dynatrace information is overwhelming. Our stack has Prometheus - time series database with Grafana in the front end. Then I started exploring about the integration and successfully implemented <strong>JMeter &gt; Prometheus &gt; Grafana</strong> integration and shared the link with my team. It is well-received. In this blog post, we will see about JMeter, Prometheus, and Grafana Integration.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Little Background</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In JMeter Plugins repository, there are backend listeners available for Kafka, ElasticSearch, and Azure. By default, JMeter supports Graphite and InfluxDB. But there is no backend listener available for Prometheus. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Fortunately, I found <a href="https://github.com/johrstrom/jmeter-prometheus-plugin">this excellent open source project</a> to export the JMeter metrics for Prometheus. Thanks to <strong>Jeff Ohrstrom</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now we shall see high-level diagram about the integration.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6958,"width":670,"height":401,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-5-1024x614.png" alt="JMeter, Prometheus, and Grafana Integration" class="wp-image-6958" width="670" height="401"/><figcaption>JMeter, Prometheus, and Grafana Integration</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As you observed in the above diagram, JMeter + Prometheus plugin exports the JMeter metrics to Prometheus database. If you want to see the visualization, you need to configure the data source in Grafana. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The main intention of the plugin is to export the results to database (Prometheus).</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>What is Prometheus?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><a href="https://github.com/prometheus">Prometheus</a> is an open-source systems monitoring and alerting toolkit originally built at <a href="https://soundcloud.com">SoundCloud</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>What is Grafana?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Grafana is the open source analytics and monitoring solution for every database.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>What is JMeter-Prometheus Plugin?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>It is an open source project which helps to export the JMeter metrics to Prometheus. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Prerequisites</h2>
<!-- /wp:heading -->

<!-- wp:list {"ordered":true} -->
<ol><li>Apache JMeter</li><li>JMeter-Prometheus Plugin</li><li>Prometheus</li><li>Grafana</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Recommended versions for above components is latest and stable release. I am in Windows 10 Home Edition.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Installing JMeter-Prometheus Plugin</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>There are multiple ways to place the jar file for JMeter. Very simple method is to download the jar file from the latest releases in GitHub and place it in the &lt;JMETER_HOME&gt;\lib\ext.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://github.com/johrstrom/jmeter-prometheus-plugin/releases">Download JMeter-Prometheus Plugin from here</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Validate the installation by creating a simple script in JMeter, and then navigate to Right Click on Thread Group &gt; Add &gt; Listener &gt; Prometheus Listener</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6961,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-6.png" alt="Prometheus Listener" class="wp-image-6961"/><figcaption>Prometheus Listener</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Or Right Click on Thread Group &gt; Add &gt; Config Element &gt; Prometheus Metrics</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6962,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-7.png" alt=" Prometheus Metrics" class="wp-image-6962"/><figcaption> Prometheus Metrics</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>Install Prometheus</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>It is very straight forward to install Prometheus. <a href="https://prometheus.io/download/">Download Prometheus for your OS </a>and extract the contents and keep it ready. In few moments, we will fire up Prometheus.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Install Grafana</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Like Prometheus, it is easy to install Grafana. Head to this <a href="https://grafana.com/get">download section</a> and install it. By default, it uses the port 3000.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Configuring JMeter-Prometheus Plugin in JMeter</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>After designing your test, you need to configure JMeter-Prometheus plugin in your test plan. Detailed instruction has already available in the GitHub page. Please read it twice for better understanding. I am not going to reiterate again in here :) </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To demonstrate, I have configured the listener as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6965,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-8-1024x199.png" alt="Prometheus Listener" class="wp-image-6965"/><figcaption>Prometheus Listener</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Configuring Prometheus</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now, it is time to configure Prometheus so that JMeter-Prometheus plugin will export the results to Prometheus database.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Go to Prometheus installation folder and open prometheus.yml in your favorite editor.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Add the below job details to your yml file. Below configuration helps to scrape the metrics from the port 9270.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code># A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=&lt;job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'jmeter'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9270']</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Launch Prometheus</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Once the configuration is completed, launch command prompt and change the directory to the prometheus installation folder and fire up the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>prometheus.exe --config.file=prometheus.yml</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now launch, Prometheus by launching http://localhost:9090/graph as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6967,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-9.png" alt="Prometheus UI" class="wp-image-6967"/><figcaption>Prometheus UI</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Launch JMeter Test</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We are good with Prometheus. Now it is time to validate the JMeter test plan. Configure your thread group with couple threads and loop infinite and hit run.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Launch, http://localhost:9270/metrics as shown below. If you are seeing the below metrics, everything is working fine. If not, please let me know the issue you are facing.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6968,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-10-1024x491.png" alt=" Prometheus Metrics" class="wp-image-6968"/><figcaption>Prometheus Metrics</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Viewing JMeter metrics in Prometheus</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As highlighted in the above metrics snapshot, we will see how to view the metrics in Prometheus. Just copy <code>jvm_classes_loaded</code> and go to Prometheus and paste it in the query text box and hit enter. You can see the value as shown below. </p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6970,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-12-1024x277.png" alt="Prometheus Query" class="wp-image-6970"/><figcaption>Prometheus Query</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click on <strong>Graph </strong>tab to view the visualization. You can customize how you want to visualize. Below is the graph which displays 95 percentile response time of each transaction.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6973,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-14-1024x503.png" alt=" Prometheus Graph" class="wp-image-6973"/><figcaption> Prometheus Graph</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Where is Grafana :) ?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you do not want to integrate Grafana, you can stop here. But Grafana is more powerful than the Prometheus visualization features.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now, let us configure Grafana to fetch the metrics from Prometheus and create beautiful visualization.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Launch Grafana by navigating to http://localhost:3000, you may need to change the password if you are logging in for first time.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Hover your mouse over the Settings icon and click on Data Sources as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6975,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-15.png" alt="Grafana Data Sources" class="wp-image-6975"/><figcaption>Grafana Data Sources</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click on <strong>Add data source </strong>as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6976,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-16.png" alt="Add Data Source" class="wp-image-6976"/><figcaption>Add Data Source</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Fill up the details as shown below and hit <strong>Save &amp; Test</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6977,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-17.png" alt="Prometheus Data Source" class="wp-image-6977"/><figcaption>Prometheus Data Source</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Let us validate the Prometheus data source in Grafana</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Hover your mouse over <strong>Explore </strong>icon and click on it.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6978,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-18.png" alt="Explore  Prometheus Data Source" class="wp-image-6978"/><figcaption>Explore  Prometheus Data Source</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Enter <code>jmeter_threads{}</code> and hit enter the query text box. If the configuration is correct, you will see the graph. Note that your JMeter test is still running. If stopped, you need to change the time picker value at top right corner.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6979,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-19-1024x403.png" alt="Prometheus Query in Grafana" class="wp-image-6979"/><figcaption>Prometheus Query in Grafana</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Creating Grafana Dashboard</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Hover your mouse over plus icon and click on <strong>Dashboard</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6981,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-20.png" alt="Creating Grafana Dashboard" class="wp-image-6981"/><figcaption>Creating Grafana Dashboard</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click on <strong>Add Query </strong>button.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6982,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-21.png" alt="Add Query" class="wp-image-6982"/><figcaption>Add Query</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Change the <strong>Query </strong>dropdown to <strong>Prometheus </strong>as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":6983,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-22.png" alt="Prometheus Query" class="wp-image-6983"/><figcaption>Prometheus Query</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Enter the query <code>jmeter_summary{quantile="0.95"}</code> and hit enter. You can customize the graph as desired.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6984,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-23-1024x433.png" alt="Prometheus Query and Graph in Grafana" class="wp-image-6984"/><figcaption>Prometheus Query and Graph in Grafana</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here is my simple dashboard.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":6986,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://qainsights.com/wp-content/uploads/2020/03/image-24-1024x513.png" alt="Sample Grafana Dashboard" class="wp-image-6986"/><figcaption>Sample Grafana Dashboard</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":4} -->
<h4>Sample Grafana Dashboard</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><a href="https://github.com/johrstrom/jmeter-prometheus-plugin/tree/master/docs/examples">https://github.com/johrstrom/jmeter-prometheus-plugin/tree/master/docs/examples</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Adding an excerpt from one of the author.</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>Another point of the plugin is that it does not need to send metrics to the JMeter controller to aggregate results as this is done by Prometheus itself.&nbsp; </p></blockquote>
<!-- /wp:quote -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now you have successfully integrated JMeter with Prometheus and Grafana. If you are satisfied with Prometheus visualization, no need to configure Grafana. But Grafana has more powerful visualization features than Prometheus. You need to setup JMeter-Prometheus plugin where ever your are running JMeter and streamline the data flow into centralized Prometheus database. This will help you to monitor the runtime metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You need to carefully craft the query in Grafana, otherwise you will end up in viewing false metrics. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Please let me know in the comments, if you face any issues. Happy to help you out :)</p>
<!-- /wp:paragraph -->