The Best Garbage Collection for JMeter
Tags: performance, testing, tutorial, java
Published: 2021-07-06 20:08:16

<!-- wp:paragraph -->
<p>The Best Garbage Collection for JMeter: Automated memory management is one of the core features of Java. By default, JMeter uses the G1GC algorithm to clean up the unreferenced objects in the heap. GC algorithms have been evolving for the past many years. In this blog article, I have done some experiments using the various GCs in JMeter to identify which GC algorithm yields us more threads to run in a bare metal medium powered laptop.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Before getting started, I am sure that some of you may not agree with the environment setup to validate the GC, base lining the maximum number of threads using the load model etc. If you are not satisfied with this setup, please share your comments so that I can learn along with you.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I am not going to bore you with the formula to come up with the maximum number of threads that we can inject. Instead, let us focus on the experiment.</p>
<!-- /wp:paragraph -->


<div class="wp-block-buttons is-content-justification-center"><!-- wp:button {"gradient":"vivid-cyan-blue-to-vivid-purple"} -->
<div class="wp-block-button"><a class="wp-block-button__link has-vivid-cyan-blue-to-vivid-purple-gradient-background has-background" href="https://github.com/QAInsights/Best-Garbage-Collection-for-JMeter" target="_blank" rel="noreferrer noopener">This experiment's GitHub Repo</a></div>
</div>


<!-- wp:heading {"level":1} -->
<h1 id="h-back-to-basics">Back to basics</h1>
<!-- /wp:heading -->

<!-- wp:heading -->
<h2 id="h-what-is-garbage-collection">What is Garbage Collection?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In Java, everything is an object. Object needs a place to live and it will be huge. When you start the Java application, your Java Virtual Machine creates new objects in the Young Generation which is expected to be short- lived. But sometimes, objects will live longer and takes up the heap space.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The longer the object lives, it is likely to become garbage collected in the Old Generation. Watch the phrase, <em>it is likely</em>. JVM may clean up the unreferenced object or it may not. It is up to JVM’s decision.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So, what is an unreferenced object? If the object has a reference in the stack memory, then it will not be garbage collected. Else, it might be garbage collected.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Above example is from the G1GC perspective, at least at a high level. If you want to learn more, please check my <a href="https://youtube.com/playlist?list=PLJ9A48W0kpRJ6Za-PG87scNCJp7kFgW2g" target="_blank" rel="noreferrer noopener">Performance Engineers</a> playlist on YouTube.</p>
<!-- /wp:paragraph -->

<!-- wp:embed {"url":"https://youtu.be/BWGxFcgQcYc","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://youtu.be/BWGxFcgQcYc
</div></figure>
<!-- /wp:embed -->

<!-- wp:heading -->
<h2 id="h-types-of-garbage-collectors">Types of Garbage Collectors</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Without deep diving into the GC algorithms, let us see the high-level overview about the following types of Garbage Collectors in Java.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Serial Garbage Collector</li><li>Parallel Garbage Collector</li><li>CMS Garbage Collector</li><li>G1 Garbage Collector</li><li>Epsilon Garbage Collector</li><li>Z Garbage Collector</li><li>Shenandoah Garbage Collector</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>In this experiment, I did multiples execution using commonly used GC algorithms namely: G1GC, Z GC, and Shenandoah GC.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Serial GC</strong> is one of the simplest GC algorithms.&nbsp;It uses single thread to carry out the GC activity. It is not ideal for multi-threaded application like JMeter.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Parallel GC</strong> uses multiple threads to carry out the GC activity, but it injects the pauses while doing the GC.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>CMS (Concurrent Mark Sweep) GC </strong>is deprecated algorithm in Java where it marks the referenced object and sweeps the remaining. It needs more CPU to achieve more throughput. Since it is deprecated, I have not included this GC for the experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Epsilon GC</strong> is a passive GC. Meaning, it will not perform any GC. Once the heap is full. It just terminates the application. It is useful to baseline the configuration and for testing your application. Hence, I am not using Epsilon GC in this experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8444,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/Epsilon.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/Epsilon.jpg" alt="Epsilon GC - Meme by NaveenKumar" class="wp-image-8444"/></a><figcaption>Epsilon GC - Meme by NaveenKumar</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>For more memes, please check <a href="https://qainsights.github.io/Testing-Jokes-Memes/" target="_blank" rel="noreferrer noopener">this repo</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>G1GC (Garbage First Garbage Collector)</strong> is one of the commonly used GC algorithms in Java and it is the default GC in Java 11. It separated heaps into multiple regions as I mentioned above and carry out the GC activity efficiently. G1GC is performant than CMS GC.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>ZGC</strong> yields less than 10ms GC time. It performs all the expensive work concurrently. ZGC yielded surprising results in this experiment. It works well with the large volume of heaps.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Shenandoah GC</strong> is from Red Hat team which the low pause time garbage collector that reduces GC pause times by performing more garbage collection work concurrently with the running Java application. Whether your heap size is 200&nbsp;GB or 2 GB, the semantics is identical.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":1} -->
<h1 id="h-experiment-setup">Experiment Setup</h1>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>I have not used Docker setup to run these tests. I want to measure the performance of each GC for JMeter in a bare metal machine. My laptop is at least six years old. But it is running, great apart from fan speed.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below is the system configuration:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>OS Name: Microsoft Windows 10 Pro</li><li>System Model: Inspiron 15-7579</li><li>System Type:  x64-based PC</li><li>Processor: Intel(R) Core (TM) i7-7500U CPU @ 2.70GHz, 2901 Mhz, 2 Core(s), 4 Logical Processor(s)</li><li>Installed Physical Memory (RAM): 16.0 GB</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Java and JMeter versions are as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Java version for G1GC and Shenandoah is Java 11.<ul><li>AdoptOpenJDK with Hotspot JVM</li></ul></li><li>Java version for ZGC is Java 16.<ul><li>AdoptOpenJDK with Hotspot JVM</li></ul></li><li>JMeter 5.4.1<ul><li>No JMeter plugins</li></ul><ul><li>Max heap size allocated is 4 GB for all the experiments.</li></ul></li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong>Demo application set up:</strong></p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Jetty 9.4.38.v20210224 hitting sample demo application at localhost:8080.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong>JMeter Test Plan</strong></p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Total number of HTTP Samplers: 8</li><li>Constant Timer of 300 milliseconds</li><li>Total Bytes for all the requests: 894.388 Kilobyte</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Before I started the experiments, I have closed all the resource consuming programs such as Vivaldi browser, MS Outlook, One Drive, Docker, MS Office programs, recording software, and more.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I have kept only the following programs open:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Two terminal sessions - to run Jetty application and JMeter in CLI mode.</li><li>Notepad – to take notes.</li><li>Snagit – to capture the screenshots.</li><li>Task Manager – to view the CPU and Memory trends.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Of course, there will be background processes running e.g., Anti-virus, daemon apps etc. I cannot kill those.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Following are the assumptions we can make:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>50% of RAM i.e., 8 GB is for Operating System</li><li>4 GB is allocated to JMeter which is consistent across the multiple runs.<ul><li>1 GB might be allocated for the daemon threads.</li></ul><ul><li>We are making use of 3 GB of RAM slice.</li></ul></li><li>Remaining might be consumed by Jetty app.</li><li>No network overhead as we are using locally hosted app.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>From JMeter perspective, there is no resource consuming elements in the test plan.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Also, after each run, I gave some breath time for my system, so that it can come back to normal utilization.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-how-to-change-the-heap-size-in-jmeter">How to change the heap size in JMeter?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Open jmeter.bat in your favorite editor and edit the line which has <code>SET HEAP</code> and configure the heap size to 4g e.g. <code>set HEAP=-Xms4g -Xmx4g -XX:MaxMetaspaceSize=256m</code></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-the-best-garbage-collection-for-jmeter-experiment">The Best Garbage Collection for JMeter Experiment</h2>
<!-- /wp:heading -->

<!-- wp:heading -->
<h2 id="h-g1gc">G1GC</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>I started my experiment with G1GC as it is the default GC in Java 11 and with 5000 threads with a ramp up of 300 seconds. I am aware that this number is high considering my system configuration. But surprisingly it performed well.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Throughout this experiment I am focusing only on the number of threads that JMeter can inject. &nbsp;Jetty will throw high response time eventually. The response time, number of failed transactions, and other parameters are secondary for this experiment. But in your actual project, you need to consider all the parameters.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below is the 5000 threads – G1GC screenshot. It is a clean run. For every run, I terminated the execution after few minutes of ramp up.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8445,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/G1GC-5000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/G1GC-5000-1024x624.jpg" alt="G1GC - 5000 Threads" class="wp-image-8445"/></a><figcaption>G1GC - 5000 Threads</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Next run I injected 7000 threads which is also performed well. But it 10000 threads run, I started seeing failures as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8446,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/G1GC-10000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/G1GC-10000-1024x656.jpg" alt="Garbage Collection for JMeter - G1GC - 7000 Threads" class="wp-image-8446"/></a><figcaption>G1GC - 7000 Threads</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In 12000 and 15000 threads run as well, the trend persisted. After these runs, I injected 25000 because of over confidence. But surprisingly, 20809<sup>th</sup> thread JMeter threw Out of Memory (heap space) error. To know more about this error, please check my YouTube tutorial.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8447,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/G1GC-25000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/G1GC-25000.jpg" alt="G1GC - 25000 Threads" class="wp-image-8447"/></a><figcaption>G1GC - 25000 Threads</figcaption></figure>
<!-- /wp:image -->

<!-- wp:embed {"url":"https://youtu.be/6gkCnhcohe4","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://youtu.be/6gkCnhcohe4
</div></figure>
<!-- /wp:embed -->

<!-- wp:embed {"url":"https://youtu.be/eZIArJEkwjo","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://youtu.be/eZIArJEkwjo
</div></figure>
<!-- /wp:embed -->

<!-- wp:paragraph -->
<p>I repeated the same run, the outcome is similar.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let us conclude G1GC experiment stating that JMeter could inject around 20000 threads in my system configuration.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-shenandoah-gc">Shenandoah GC</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To change the GC algo in JMeter, open jmeter.bat and add the below in ` set GC_ALGO` line:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>-XX:+UseShenandoahGC</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>As we know, Shenandoah GC should perform better than G1GC. Let us see the Shenandoah GC results. I started my first test with the over confidence by injecting 20000 threads. Shenandoah GC handled well. Here is the screenshot.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8450,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/SD-20000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/SD-20000-1024x487.jpg" alt="Garbage Collection for JMeter - Shenandoah - 20000 Threads" class="wp-image-8450"/></a><figcaption>Shenandoah - 20000 Threads</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Then, I increased the thread count to 25000. It broke at 21439<sup>th</sup> thread with heap space out of memory error. If you see the error count, it is surprisingly less.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8451,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/SD-25000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/SD-25000.jpg" alt="Shenandoah - 25000 Threads" class="wp-image-8451"/></a><figcaption>Shenandoah - 2000 Threads</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-zgc">ZGC</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To change the GC algo to ZGC, open jmeter.bat and add the below in `set GC_ALGO` line:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>-XX:+UseZGC</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here comes the next surprise, I started ZGC with 25000 threads. My bad. See it for yourself. It threw heap space out of memory after 8000 threads. I am not sure why. If you are expert in ZGC, please share your inputs in the comment section.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8452,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/ZGC-25000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/ZGC-25000-1024x431.jpg" alt="ZGC - 10000" class="wp-image-8452"/></a><figcaption>ZGC - 10000</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Even for 10000 threads, I got out of memory error again.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8453,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/07/ZGC-10000.jpg"><img src="https://qainsights.com/wp-content/uploads/2021/07/ZGC-10000-1024x307.jpg" alt="ZGC - 10000 OOM" class="wp-image-8453"/></a><figcaption>ZGC - 10000 OOM</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>But it ran well for 5000 threads. As the ZGC, threw OOM for 10000 threads, I did not purse my experiment further.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-observations">Observations</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>G1GC injected more than 20000 threads without any issue. But there were some stutters during the execution, I am not sure is it because of my OS or JMeter. I should have executed with Visual VM, but I do not want to have overhead again. G1GC had lot of errors in the samplers consistently across high volume runs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Shenandoah GC handled the load well. The number of errors were less, observe the trend in the above screenshots, you can easily correlate. Response times were high. From my perspective, I never bothered about that. My focus is only on active threads.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I expected much from ZGC, but it ran out of memory too soon. Maybe it needs more slices to work with. But clearly ZGC is not apt to run high volumes test.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you can carry out this experiment in a high-powered instance in cloud, you can begin with ZGC by slicing more memory into it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I can hear you saying that, why you have not tuned the GC parameters to yield performance. My intention is to try just vanilla flavors without touching the other parameters. The main reason is there are too many parameters to begin with. E.g. Shenandoa GC has three modes: normal/satb, iu, and passive. Default is normal/satb. Tuning these parameters might have yielded more number of threads. If you take G1GC, we could have achieved more performance. I am not a tuning expert. Let me conclude the experiment here.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-conclusion">Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>From multiple runs with varying number of threads, G1GC and Shenandoah performs almost similar in terms of performance. Further tuning of G1GC might have outperformed Shenandoah GC and vice-versa.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>ZGC algo has failed in my system configuration. But it could perform better in a better environment. To choose the best GC, you need to run your tests with each GC and baseline each number in your environment setup. You will get surprising results as me.</p>
<!-- /wp:paragraph -->