JMeter Integration with Elastic
Tags: testing, performance, tutorial, elasticsearch
Published: 2021-01-19 15:39:14

<!-- wp:paragraph -->
<p>I work in Kibana almost every day at work to debug and monitor the test execution. I always wanted to integrate my JMeter logs with Kibana to deep-dive into JMeter logs in real time. This blog post will talk about JMeter Integration with Elastic suite of products ElasticSearch, Kibana, and Filebeat. We are going to deploy ElasticSearch, Kibana, and Filebeat components locally and visualize the logs in Kibana.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There are many other integrations possible to deep-dive into JMeter metrics. But using the ELK (Elastic, Logstash, and Kibana) stack, it is easy to implement and maintain.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-about-elastic">About Elastic</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Elastic offers various products and solutions for search, analytics, security, logging and more. Their solutions are powered by <strong>ElasticSearch </strong>and <strong>Kibana</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>ElasticSearch </strong>is the search and analytics engine which is the heart of the Elastic Stack.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Kibana </strong>is the visualization layer of ELK.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Beats </strong>and <strong>Logstash </strong>are the Ingest engine.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8162,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-30.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-30.png" alt="JMeter Integration with Elastic - ELK Stack" class="wp-image-8162"/></a><figcaption>JMeter Integration with Elastic - ELK Stack</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-installations">Installations</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We are going to install the following components to integrate ELK with JMeter.</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>ElasticSearch</li><li>Kibana</li><li>Filebeat</li></ol>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3 id="h-let-us-begin-with-elasticsearch">Let us begin with ElasticSearch</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Go to https://www.elastic.co/start to begin downloading ElasticSearch. Click on the respective download links based on your operating system. This integration tested with Windows 10 Pro OS.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Download the zip file for Windows, extract it and place it in your favorite location. Once you extracted the zip file, you will get the below folders and files. By default, ElasticSearch uses localhost:9200. You can change the port and other configuration by editing the <strong>config\elasticsearch.yml</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8164,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-31.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-31.png" alt="ElasticSearch in Windows 10" class="wp-image-8164"/></a><figcaption>ElasticSearch in Windows 10</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To launch ElasticSearch, open Powershell and navigate to bin folder and enter <code>.\elasticsearch.bat</code></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-installing-kibana">Installing Kibana</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In https://www.elastic.co/start page, click on the appropriate download link for Kibana. In Windows, extract the downloaded zip file and place it in your favorite folder.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8165,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-32.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-32.png" alt="Kibana in Windows 10" class="wp-image-8165"/></a><figcaption>Kibana in Windows 10</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>By default, Kibana uses localhost:5601. To configure the properties, go to <code>config\kibana.yml</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To start Kibana, open Powershell, navigate to bin folder and enter <code>.\kibana.bat</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-installing-filebeat">Installing Filebeat</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Filebeat is the lightweight log analysis for Elastic. Go to https://www.elastic.co/downloads/beats/filebeat to download the appropriate package. For Windows, there is a MSI package available for Filebeat. Download the package and install it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It will get install in <code>C:\Program Files\Filebeat</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Install the Filebeat as a service which will help you to start the service for the unexpected restarts.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To install it as a service, open Powershell and navigate to <code>C:\Program Files\Filebeat</code> and run </p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">.\install-service-filebeat.ps1</pre>
<!-- /wp:preformatted -->

<!-- wp:heading {"level":3} -->
<h3 id="h-configuring-filebeat">Configuring Filebeat</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now we completed all the installations which are required to integrate. Next step is to configure the Filebeat and Kibana.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To configure Filebeat, open <code>Filebeat.yml</code> from <code>C:\Program Files\Filebeat</code> location and enter the below configurations.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below configuration is to mention the inputs and the JMeter log location. Multiline pattern is to identify the timestamp in the JMeter log file.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:\Tools\apache-jmeter-5.4\bin\jmeter.log
  multiline.pattern: ^&#91;0-9]{4}-&#91;0-9]{2}-&#91;0-9]{2}
  multiline.negate: true
  multiline.match: after</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Below configuration is for Kibana.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>setup.kibana:
  host: "localhost:5601"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Below configuration is for ElasticSearch.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>output.elasticsearch:
  hosts: &#91;"localhost:9200"]
  username: "&lt;username&gt;"
  password: "&lt;password&gt;"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Save the yml file and start the service using the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>Start-Service filebeat</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To stop the service, use <code>Stop-Service filebeat</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Launch Kibana by navigating to http://localhost:5601. Go to <strong>Management &gt; Stack Management</strong> page.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-configuring-kibana">Configuring Kibana</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Click on <strong>Index Patterns</strong> and make sure <code>filebeat-*</code> pattern is exists. Else, you can create one by clicking on <strong>Create Index pattern </strong>button. Index pattern helps to retrieve the data from ElasticSearch.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8166,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-33.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-33-1024x292.png" alt="Index Patterns" class="wp-image-8166"/></a><figcaption>Index Patterns</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-jmeter-test-execution">JMeter Test Execution</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now, it is time to start the test in JMeter. Launch JMeter and open the test plan which you want to execute. In the below example, I have one simple JSR223 Sampler to log info.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8167,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-34.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-34.png" alt="Sample Test Plan" class="wp-image-8167"/></a><figcaption>Sample Test Plan</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Start the execution in CLI mode. This test plan will simply log the string in the <strong>jmeter.log</strong> file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Launch Kibana and click on Discover as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8169,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-36.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-36.png" alt="Kibana Discover" class="wp-image-8169"/></a><figcaption>Kibana Discover</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Change the time filter and click on <strong>message </strong>and then click on <strong>Refresh </strong>button.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8170,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-37.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-37-1024x324.png" alt="Filter" class="wp-image-8170"/></a><figcaption>Filter</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>This will filter out only the timestamp and the message and display the hits stats.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8171,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-38.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-38-1024x661.png" alt="Output" class="wp-image-8171"/></a><figcaption>Output</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-closing-words">Closing Words</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By integrating JMeter with ElasticSearch, you can stream the performance stats real time to Kibana and visualize the data as per your requirement. This will help performance engineers to drill-down to the root cause when the test is running.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Kibana comes with powerful visualization, machine learning, canvas features which will expedite the process of performance testing and engineering.</p>
<!-- /wp:paragraph -->