Servers Performance Monitoring in JMeter
Tags: performance, testing, webperf, tutorial
Published: 2021-01-14 16:05:50

<!-- wp:paragraph -->
<p>As a performance engineer, it is our responsibility to gather the performance stats from all the layers. The statistics from the tool alone will not help us in identifying the performance bottlenecks. As the technology getting complex every day, it is critical to monitor and gather the metrics from across the tiers. In this blog article, we are going to focus on Servers Performance Monitoring in JMeter. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>When the test is on, you should not go out for a coffee or browse the internet. As a responsible performance engineer, you should monitor the execution in the tool or in APM solutions or the logs. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-servers-performance-monitoring">Servers Performance Monitoring</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>JMeter stats reveals the end-to-end performance stats in runtime. But it doesn't reveal how your backend is performing. When the load is being injected, your servers will undergo a lot of processing and sometimes it will even break. By monitoring the servers' performance, you can identify when the server is getting throttled and you can get started with your detective work.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-how-to-enable-servers-performance-monitoring-in-jmeter">How to enable Servers Performance Monitoring in JMeter?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>There are two setups you need to take care to monitor the servers performance in JMeter. </p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Server Agent setup</li><li>PerfMon Metrics Collector setup</li></ol>
<!-- /wp:list -->

<!-- wp:heading -->
<h2 id="h-server-agent-setup">Server Agent setup</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As I mentioned, by default JMeter doesn't collect the servers' health metrics (except Tomcat). But you can leverage open source library called <a href="https://github.com/hyperic/sigar" target="_blank" rel="noreferrer noopener">SIGAR</a> which should run on the each server where the application under test is hitting.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Download the <a href="https://github.com/undera/perfmon-agent/blob/master/README.md" target="_blank" rel="noreferrer noopener nofollow">PerfMon Server Agent from GitHub</a>. Extract it and keep it in each server which you would like to gather the metrics. No need to have a root or administrator access.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Go to the extractor folder and run <code>startAgent.bat</code> if you are in Windows or run <code>startAgent.sh</code> in macOS/Linux operating system.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will start the agent in the default port 4444 as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8108,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-1.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-1.png" alt="Server Agent" class="wp-image-8108"/></a><figcaption>Server Agent</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To change the port you can use this command <code>.\startAgent.bat --tcp-port 3450</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-perfmon-metrics-collector-setup">PerfMon Metrics Collector setup</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Open the JMeter test plan which you will be executing. Open the Plugins Manager to install the <strong>PerfMon Metrics Collector</strong> listener. Click on <strong>Available Plugins </strong>and search for <strong>PerfMon </strong>to install the listener.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8109,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-2.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-2.png" alt="PerfMon Plugin" class="wp-image-8109"/></a><figcaption>PerfMon Plugin</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Once the plugin is installed, restart the JMeter and add the <code>jp@gc - PerfMon Metrics Collector</code> by right clicking on the <strong>Thread Group &gt; Add &gt; Listener &gt; jp@gc - PerfMon Metrics Collector</strong>. This will add a listener to your test plan as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8111,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-4.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-4-1024x514.png" alt="PerfMon Plugin" class="wp-image-8111"/></a><figcaption>PerfMon Plugin</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3 id="h-configuring-jp-gc-perfmon-metrics-collector">Configuring <strong>jp@gc - PerfMon Metrics Collector</strong></h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Click on <strong>Add Row</strong> button which will add the <strong>Host, Port, and Metric to collect </strong>with the default values.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8113,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-6.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-6-1024x196.png" alt="Add Metric" class="wp-image-8113"/></a><figcaption>Add Metric</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>You can add multiple metrics to collect by clicking on <strong>Add Row</strong> button.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8114,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-7.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-7.png" alt="Add mulitple metrics" class="wp-image-8114"/></a><figcaption>Add multiple metrics</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3 id="h-testing-the-server-agent">Testing the Server Agent</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>After adding the listener, validate your test plan by right clicking on the <strong>Thread Group &gt; Validate. </strong>This will send one iteration of your requests in the test plan. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Head back to the command prompt (or terminal) where the Server Agent is up and running where you will see the <strong>Yep, we received the 'test' command</strong> output as shown below. This means the server agent is ready to measure the metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8116,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-9.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-9.png" alt="Testing the Server Agent" class="wp-image-8116"/></a><figcaption>Testing the Server Agent</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-start-the-test">Start the test</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Once the validation is successful, you can start your test in CLI mode. Just for demonstration purpose, I am executing the test in GUI mode. In this example, I have configured to monitor the CPU and Memory metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After few seconds of the test started, listener will plot the graphs which it got from the server agent.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8118,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-11.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-11-1024x456.png" alt="Servers Performance Monitoring in JMeter" class="wp-image-8118"/></a><figcaption>Servers Performance Monitoring in JMeter</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>This is my <strong>Memory </strong>screenshot from the <strong>Resource Monitor</strong> in Windows which matches the JMeter screenshot.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8120,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-13.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-13.png" alt="Resource Monitor" class="wp-image-8120"/></a><figcaption>Resource Monitor</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3 id="h-configuring-metrics">Configuring Metrics</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To configure the metrics in the listener follow the below steps as shown in the screenshot. If the quality of the image is not good, right click on the image and open it.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8121,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-14.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-14-1024x433.png" alt="Configure Metrics" class="wp-image-8121"/></a><figcaption>Configure Metrics</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Apart from typical metrics such as CPU, Memory, Swap etc, there are two metrics: <strong>EXEC </strong>and <strong>TAIL </strong>will be helpful based on the use case.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>EXEC </strong>helps to execute command and collect the data and display it in the chart. This command helps to get the process count in Powershell.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>C\:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe:(Get-Process).Count</code></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8125,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-18.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-18.png" alt="PerfMon Parameters Help" class="wp-image-8125"/></a><figcaption>PerfMon Parameters Help</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":8122,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-15.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-15-1024x179.png" alt="Exec" class="wp-image-8122"/></a><figcaption>Exec</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here is the runtime chart for the above command.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8123,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-16.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-16-1024x418.png" alt="EXEC Output" class="wp-image-8123"/></a><figcaption>EXEC Output</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>TAIL </strong>helps to parse the file and extract the metrics and display it in runtime. Suppose, if you want to parse through some logs file which has one column, you can do that using TAIL.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8124,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-17.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-17.png" alt="TAIL Help" class="wp-image-8124"/></a><figcaption>TAIL Help</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:image {"id":8127,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-19.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-19-1024x182.png" alt="TAIL Metric" class="wp-image-8127"/></a><figcaption>TAIL Metric</figcaption></figure>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":8129,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/01/image-21.png"><img src="https://qainsights.com/wp-content/uploads/2021/01/image-21-1024x455.png" alt="TAIL Output" class="wp-image-8129"/></a><figcaption>TAIL Output</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-troubleshooting">Troubleshooting</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you are trying to start the server agent, you might face issues in the command prompt or terminal with the below error.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>INFO    2021-01-12 19:39:53.797 &#91;kg.apc.p] (): *** Logging available processes ***
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000010014ed4, pid=10284, tid=10180
#
# JRE version: OpenJDK Runtime Environment (15.0+36) (build 15+36-1562)
# Java VM: OpenJDK 64-Bit Server VM (15+36-1562, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# C  &#91;sigar-amd64-winnt.dll+0x14ed4]
#
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Tools\ServerAgent-2.2.3\hs_err_pid10284.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To solve this issue, please check this <a href="https://github.com/hyperic/sigar/issues/141" target="_blank" rel="noreferrer noopener nofollow">GitHub Issue</a>. Download the <code>sigar-amd64-winnt.dll</code> and put it under <strong>lib </strong>folder. It is advisable to take a backup.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you are not able to start, the reason might be a firewall. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Following commands could be useful for the server agent.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>.\startAgent.bat --sysinfo

.\startAgent.bat --interval &lt;seconds&gt;

.\startAgent.bat --udp-port 0 --tcp-port 5555</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2 id="h-final-words">Final Words</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By leveraging the Server Agent and PerfMon Metrics Collector, you can monitor the performance of the hosts from the JMeter which will help you to find the bottlenecks and validate the health metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Please let me know if you face any issues while implementing this solution.</p>
<!-- /wp:paragraph -->