Upload files to AWS S3 in JMeter using Groovy
Tags: performance, aws, java
Published: 2022-04-20 14:41:22

<!-- wp:paragraph -->
<p>I use my personal AWS S3 to store all my personal and confidential documents. There are three primary reasons for choosing AWS S3: affordable, speed and reliable. If you are working on the AWS cloud, the usage of S3 is inevitable. S3 plays a critical role in storing objects in hot and cold storage. Sometimes you need to upload a payload or file objects to S3 programmatically via your performance test script. This blog article will help you to upload files to AWS S3 in JMeter using Groovy. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>What is S3?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>S3 comes with various storage classes: S3 Standard, S3 Intelligent Tiering, S3 Glacier Instant Retrieval and more. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Prerequisites </h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The following are the prerequisites for upload files to AWS S3:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>AWS Account</li><li>AWS IAM User with S3 access policy,</li><li>basic knowledge of AWS services,</li><li>basic knowledge of JMeter</li><li>basic knowledge of AWS SDK (Java)</li><li>file(s) to upload</li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>AWS IAM User</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Login into your AWS account and open <strong>IAM </strong>service. Click <strong>Users </strong>under <strong>Access Management</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Add users</strong> button to set the user details and credential type as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter <code>jmeter-s3-access</code> in <strong>User name</strong> and check <strong><code>Access key - Programmatic access</code></strong>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This user will have access to the AWS service programmatically, not from the user console. Click <strong>Next: Permissions</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In the <strong>Set permissions</strong> section, click <strong>Attach existing policies directly</strong> and filter the policies of S3 by typing <code>s3</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For the demonstration purpose, let us go with <code><code>AmazonS3FullAccess</code></code>. Check <code>AmazonS3FullAccess</code> and then click <strong>Next: Tags</strong>. But for the production server, follow the zero trust framework. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Adding tags is optional, but it is recommended to have relevant key-pair values. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Review </strong>and then click on <strong>Create user</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Copy the <strong>Access key ID</strong> and <strong>Secret access key</strong> to a secured location. Alternatively, you can download the .csv file. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>JMeter Test Plan</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By default, JMeter doesn't have the feature to upload the artifacts to AWS S3. To extend the functionality, we must leverage the JSR223 Sampler. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is the complete playlist of JMeter series which will help you to become a hero within 1 week. </p>
<!-- /wp:paragraph -->

<!-- wp:embed {"url":"https://www.youtube.com/playlist?list=PLJ9A48W0kpRIjLkZ32Do9yDZXnnm7_uj_","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/playlist?list=PLJ9A48W0kpRIjLkZ32Do9yDZXnnm7_uj_
</div></figure>
<!-- /wp:embed -->

<!-- wp:heading {"level":3} -->
<h3>JSR223 Sampler</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>It is not possible to write upload to S3 code block natively. We must leverage the latest version of AWS SDK for Java. To add the AWS SDK as a dependency to the JSR223 Sampler, the easiest way is to leverage Grape. Personally, I have not tried the JMeter Maven plugin. I found Grape is simple to get start.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>What is Grape?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The&nbsp;<em><strong>Gr</strong>oovy <strong>A</strong>daptable <strong>P</strong>ackaging <strong>E</strong>ngine</em>&nbsp;or&nbsp;<em><strong>Gr</strong>oovy <strong>A</strong>dvanced <strong>P</strong>ackaging <strong>E</strong>ngine</em>, Grape, is a JAR dependency manager which is built-in with Groovy. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Using the <code>@Grab</code> annotations in JSR223, you can add maven repository dependencies to the classpath. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>@Grab(group='software.amazon.awssdk', module='s3', version='2.17.172', scope='test')</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The above annotation will download the AWS SDK S3 dependencies. To download multiple dependencies, use <code>@Grapes</code> annotations. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>@Grapes(
    @Grab(group='software.amazon.awssdk', module='s3', version='2.17.172', scope='test'),
    @Grab(group='software.amazon.awssdk', module='sts', version='2.17.172', scope='test')

)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p> To change the source, use <code>@GrabResolver</code> annotation.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>@GrabResolver(name='restlet', root='http://maven.restlet.org/')</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Hello Time using Grape in JSR223</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Let us understand how Grape works in JSR223 Sampler in JMeter by writing a simple snippet with @Grab annotations. To explain it with a simple <code>Hello world</code>, pardon <code>Hello time</code> example, let us use the below code snippet. Copy and paste the below code into your JSR223 Sampler in JMeter.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>@Grapes(
    @Grab(group='joda-time', module='joda-time', version='2.10.14')
)
import org.joda.time.LocalDateTime

LocalDateTime currentDateTime = new LocalDateTime()
log.info "Local Time is " + currentDateTime</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let us slice each line. @Grapes is the optional inception annotation which has @Grab annotation. One or more @Grab annotations can be placed anywhere in the JSR223. In this example, @Grab annotation will manage the <code>joda-time</code> dependency and prints the current date and time in the <strong>Log Viewer</strong>. By default, it will download the dependencies from mvnrepository.com.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click on the <strong>Run</strong> button in JMeter to see the log message in the Log Viewer. Once you click on <strong>Run</strong> button, the above script will download the dependencies from the source and keep the JARs in <code>/Users/&lt;user&gt;/.groovy/grapes</code> in Mac, <code>C:\Users\&lt;user&gt;\.groovy\grapes</code> in Windows OS.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The dependencies download will happen only for the first execution. Eventual execution in JMeter will be fast.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>AWS S3 SDK</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>There are multiple methods available to upload the artifacts to S3, e.g. via AWS CLI, AWS SDK, HTTP requests and more. The official AWS SDK opens the door to building applications for AWS via its API.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By leveraging the AWS SDK for Java, it is easy to interact with the AWS services. The latest version of the AWS SDK for Java is v2. v2 is a major rewrite of v1. The v2 version is packed with a nonblocking I/O architecture using Netty where it achieves high concurrency with fewer threads. Also, it supports HTTP/2 and automatic pagination. </p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Upload files to S3 in Groovy</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We are going to leverage Grape in Groovy in the JSR223 Sampler by adding AWS SDK for S3. Head to https://mvnrepository.com/artifact/software.amazon.awssdk/s3 and select the latest version. At this time of writing, the latest version is 2.17.172.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Grape </strong>and copy the annotation as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9321,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/04/image-12.png"><img src="https://qainsights.com/wp-content/uploads/2022/04/image-12.png" alt="Upload files to AWS S3 in JMeter using Groovy - Grape" class="wp-image-9321"/></a><figcaption>Upload files to AWS S3 in JMeter using Groovy - Grape</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Then, copy and paste the below snippet into the JSR223 Sampler.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>@Grapes(
    @Grab(group='software.amazon.awssdk', module='s3', version='2.17.172', scope='test')
)
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.*

import java.io.File
import java.nio.file.Paths

// Configurations
String accessKey = vars.get("AWS_ACCESS_KEY")
String secretKey = vars.get("AWS_SECRET_KEY")
String bucketName = vars.get("AWS_BUCKET_NAME")
String strFilename = "C:\\temp\\result.json"

try {
	// Set region
	Region region = Region.US_EAST_1
	
	// Create credentials
	AwsBasicCredentials awsCreds = AwsBasicCredentials.create(
	      accessKey,
	      secretKey)
	
	// Build S3 Client
	S3Client s3 = S3Client.builder()
	      .region(region)
	      .credentialsProvider(StaticCredentialsProvider.create(awsCreds))
	      .build()
	
	// Create file object
	File s3Obj = new File(strFilename)
	
	// Create PUT request
	PutObjectRequest request = PutObjectRequest.builder()
	      .bucket(bucketName)
	      .key(s3Obj.getName())
	      .build()
	
	// Upload file
	s3.putObject(request, Paths.get(strFilename))
}

// Catching exception and displaying it in the Sample result
catch (S3Exception e){
	SampleResult.setSuccessful(false)
	SampleResult.setResponseMessage(e.getMessage())
	SampleResult.setResponseCode(null)
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let us slice the above code.</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>After adding the <code>import</code> statements, the first block represents the configurations such as AWS Access, Secret Key, and Bucket name.</li><li>The next configuration is the AWS region.</li><li><code>AwsBasicCredentials</code> block creates the credentials to access the bucket</li><li><code>S3Client</code> block creates a S3 client using the <code>AwsBasicCredentials</code> credentials.</li><li><code>s3Obj</code> is the file to be uploaded.</li><li><code>PutObjectRequest</code> will build the request.</li><li><code>s3.putObject</code> will upload the file by leveraging the <code>request.</code></li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>If any exceptions occurs, JMeter will display the exceptions in the sampler with the exception details for troubleshooting.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The variables <code>AWS_ACCESS_KEY</code>, <code>AWS_BUCKET_NAME</code>, and <code>AWS_SECRET_KEY</code> is available in the <code>Test Plan</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is the <a href="https://github.com/QAInsights/S3-Upload-JMeter-Groovy" target="_blank" rel="noreferrer noopener">repository to download</a> the sample JMeter test plan for your reference.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Never ever store the AWS credentials in the test plan. Pass the credentials via command line, environment variables, or programmatically generate them.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Congratulations! Now you know how to upload artifacts to S3 programmatically in JMeter using JSR223 Sampler.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Uploading artifacts to S3 is just the beginning. By leveraging the AWS SDK for Java, it is possible to interact with AWS services from JMeter using Groovy. E.g, after JMeter test execution, you can upload the results in a zip file using the above snippet. If you have any other usecases for JMeter, please let me know in the comments. </p>
<!-- /wp:paragraph -->