Chaos Testing in LoadRunner using Gremlin
Tags: performance, testing, tutorial, aws
Published: 2022-03-31 15:21:37

<!-- wp:paragraph -->
<p>In my last post, I listed out the <a href="https://qainsights.com/whats-new-in-micro-focus-loadrunner-professional-2022/" target="_blank" rel="noreferrer noopener">new features of LoadRunner Professional 2022</a>. One of the tech preview features in LoadRunner 2022 is Chaos Testing with Gremlin. This blog post will detail the steps to running chaos experiments in LoadRunner Professional using Gremlin integration. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>What is Chaos Engineering?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Simply put, you break things on purpose and learn how you can make your systems reliable. I urge you to check this excellent page from Gremlin, https://www.gremlin.com/chaos-engineering/?ref=qainsights </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Chaos testing is a subset of chaos engineering. Think of performance testing as a subset of performance engineering. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Gremlin Integration with LoadRunner Professional 2022</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To integrate Gremlin with LoadRunner Professional 2022, following are the prerequisites required:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>LoadRunner Professional 2022 </li><li>Gremlin account (which is free)</li><li>Sample application</li><li>Server with Gremlin installed </li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>To demonstrate the integration, let us use an EC2 instance on AWS with nginx installed on your favorite Linux distro.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>nginx on EC2</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Login to your AWS console and spin up an instance. In this example, let us use Ubuntu. Once the instance is spanned up, SSH into it to install nginx and Gremlin.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$ sudo apt update -y

$ sudo apt install nginx -y</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Launch http://&lt;ec2_ip_address&gt; to view the nginx home page. If you do not see the nginx home page as below, please validate the security groups.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9245,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-29.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-29.png" alt="nginx demo app" class="wp-image-9245"/></a><figcaption>nginx demo app</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":9246,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-30.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-30-1024x266.png" alt="Security Group" class="wp-image-9246"/></a><figcaption>Security Group</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>Gremlin Installation Steps</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The next step is to download and install Gremlin on the Linux box. Login to your Gremlin account. On the home page, you can find the steps to follow.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After successful installation, let us configure the Gremlin Control Plan using the Team ID and Secret Key. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter the below command and hit enter to copy the Team ID and Secret Key.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$ gremlin init</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To grab the Team ID, go to your <strong>Company &gt; Team &gt; Configuration</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To grab a Secret Key, check your registered email ID or you can reset it.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9247,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-31.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-31-1024x326.png" alt="Gremlin Configuration" class="wp-image-9247"/></a><figcaption>Gremlin Configuration</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Once the configuration is successful, you should get the below success message.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9248,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-32.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-32.png" alt="Gremlin Identifier" class="wp-image-9248"/></a><figcaption>Gremlin Identifier</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Creating a Scenario in Gremlin</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before we create a script in VuGen, let us create a scenario in Gremlin. Go to your <strong>Dashboard</strong>, click the <strong>Create Scenario</strong> button as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9250,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-33.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-33-1024x503.png" alt="Create Scenario" class="wp-image-9250"/></a><figcaption>Create Scenario</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Let us attack the CPU of a Linux box where nginx is running. Name your scenario, select the target hosts, select the <strong>Category </strong>as <strong>Resource</strong>, and <strong>Attack </strong>as <strong>CPU</strong>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The length of the attack is 300 seconds, CPU capacity as 100%. and target all the cores. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Add to Scenario</strong>, then <strong>Save Scenario</strong>. </p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9251,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-34.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-34-1024x654.png" alt="Attack Configuration" class="wp-image-9251"/></a><figcaption>Attack Configuration</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":9253,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-35.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-35-1024x367.png" alt="Scenario Details" class="wp-image-9253"/></a><figcaption>Scenario Details</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Creating LoadRunner Script</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Launch VuGen to create a Web HTTP/HTML protocol script and add the below code in <strong>Action</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>	lr_start_transaction("T00_HomePage");


        // Replace the EC2 IP address

	web_custom_request("web_custom_request",
		"URL=http://EC2_IP_ADDRESS",
		"Method=GET",
		"TargetFrame=",
		"Resource=0",
		"Referer=",
		"Body=",
		LAST);

	lr_end_transaction("T00_HomePage", LR_AUTO);</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Compile and run the script to validate the request. Let us configure the pacing as shown below in <strong>Runtime Settings</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9254,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-36.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-36.png" alt="Runtime Settings" class="wp-image-9254"/></a><figcaption>Runtime Settings</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p> Close <strong>VuGen </strong>and launch <strong>Controller</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Designing a scenario in Controller 2022</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Upon launching the Controller, add the above created script as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9256,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-37.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-37.png" alt="New Scenario in Controller" class="wp-image-9256"/></a><figcaption>New Scenario in Controller</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To add a disruption event, click on <strong>Disruption Events</strong> as shown below under <strong>Scenario Schedule</strong> in the <strong>Design </strong>tab.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9257,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-38.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-38.png" alt="Disruption Events" class="wp-image-9257"/></a><figcaption>Disruption Events</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click on the <strong>Add Event</strong> button to configure the API Key, Team ID, and other details. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To generate an API Key from Gremlin, go to your <strong>Account Settings &gt; API Keys &gt; New API Key</strong>. Enter a valid name and click <strong>Save</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Paste your API Key and Team ID as shown below and click <strong>Add</strong>. After adding the API key and team ID, the created scenario in Gremlin should be displayed as shown below. </p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9276,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-49.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-49.png" alt="Add Event" class="wp-image-9276"/></a><figcaption>Add Event</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Before you click <strong>Save</strong>, configure the start time as shown below. The end time will be automatically configured. The CPU attack will get started in the third minute and will last for a minute.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9275,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-48.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-48.png" alt="Disruption Events" class="wp-image-9275"/></a><figcaption>Disruption Events</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Below is the <strong>Global Schedule</strong> which we are going to run.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9263,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-42.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-42.png" alt="Global Schedule" class="wp-image-9263"/></a><figcaption>Global Schedule</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Save the <strong>Controller </strong>scenario before you hit <strong>Run</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Running Chaos Experiment</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before you start running the scenario in the Controller, go to your EC2 instance, enter <code>top</code> command to monitor the CPU utilization.</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>Creating chaos experiments is a blend of art and science.</p><cite>- myself :) </cite></blockquote>
<!-- /wp:quote -->

<!-- wp:heading {"level":4} -->
<h4>Experiment Objective</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The above experiment will validate the response time of nginx server for 50 vusers under the high CPU utilization. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click on the <strong>Run </strong>button to get started and monitor the graphs in the Controller.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Before the CPU Attack</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":9264,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-43.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-43.png" alt="" class="wp-image-9264"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>During the CPU Attack</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9266,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-45.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-45.png" alt="CPU Attack" class="wp-image-9266"/></a><figcaption>CPU Attack</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>After the CPU Attack</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9267,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-46.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-46.png" alt="After CPU Attack" class="wp-image-9267"/></a><figcaption>After CPU Attack</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here is the complete graph of our test where I highlighted the response time during the CPU attack. There is a slight elevation in the response time graph during high CPU utilization. After the attack, the response time trend reverted.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9268,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/03/image-47.png"><img src="https://qainsights.com/wp-content/uploads/2022/03/image-47-1024x358.png" alt="Response Time Vs Time" class="wp-image-9268"/></a><figcaption>Response Time Vs Time</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Congratulations! You have successfully executed a chaos experiment using Gremlin in LoadRunner Professional 2022. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After the experiment, make sure you terminate the instance on AWS, otherwise it will accrue charges.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Though it is a tech preview, the positive scenarios are working fine without any issues. Using the free tier in Gremlin, you can perform various experiments. Based on your requirement, you can define the blast radius and break things on purpose in a controlled manner on production. Integration with LoadRunner makes performance testers effort easy and effective. </p>
<!-- /wp:paragraph -->