Upload files to AWS S3 in k6
Tags: aws, tutorial, performance
Published: 2022-05-09 15:10:15

<!-- wp:paragraph -->
<p>In my last post, we discussed how to upload files to <a href="https://qainsights.com/upload-files-to-aws-s3-in-jmeter-using-groovy/" target="_blank" rel="noreferrer noopener">AWS S3 in JMeter using Groovy</a>. We have also seen Grape package manager on JMeter. Recently, k6 announced its next iteration with a lot of new features and fixes. In this blog post, we are going to see how to upload files to AWS S3 in k6.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>What's new in k6 0.38.1?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>k6 0.38.1 is a minor patch. To view all the new features, check out <a href="https://github.com/grafana/k6/releases/tag/v0.38.0" target="_blank" rel="noreferrer noopener">k6 0.38.0 tag</a>. Following are the noteworthy features in 0.38.0.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>AWS JSLib</li><li>Tagging metric values</li><li>Dumping SSL keys to an NSS formatted key log file</li><li>Accessing the consolidated and derived options from the default function</li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>AWS JSLib Features</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Amazon Web Services is one of the widely used public cloud platforms. AWS JSLib packed with three modules (as of today's writing) :</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>S3 Client</li><li>Secrets Manager Client</li><li>AWS Config</li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>S3 Client</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>S3Client modules help to list S3 buckets, create, upload, and delete S3 objects. To access the AWS services, each client (in this case S3Client) uses the AWS Credentials such as Access Key ID and Secret Access Key. The credentials must have required privileges to perform the tasks, otherwise the request will fail. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Prerequisites to upload objects to S3</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The following are the prerequisites to upload objects to S3 in k6.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The latest version of k6 0.38.0 or above</li><li>AWS Access Key ID and Secret Access Key which has relevant S3 permissions.</li><li>Basic knowledge in k6</li></ul>
<!-- /wp:list -->

<!-- wp:embed {"url":"https://www.youtube.com/playlist?list=PLJ9A48W0kpRJKmVeurt7ltKfrOdr8ZBdt","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/playlist?list=PLJ9A48W0kpRJKmVeurt7ltKfrOdr8ZBdt
</div><figcaption>Learn k6 Series</figcaption></figure>
<!-- /wp:embed -->

<!-- wp:heading -->
<h2>How to upload files to S3?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Create a new k6 script in your favorite editor and name it as <code>uploadFiletoS3.js</code>. To upload files to S3, the first step is to import the following directives to your k6 script. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>import exec from 'k6/execution'

import {
    AWSConfig,
    S3Client,
} from 'https://jslib.k6.io/aws/0.3.0/s3.js'</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The next step is to read the AWS credentials from the command line. It is not recommended to hard code the secrets into the code, which will raise a security concern. Using the <code>__ENV</code> variables, it is easy to pass the values into the k6 script. To pass the AWS config such as region, access key, and secret, use the object <code>AWSConfig</code> as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>const awsConfig = new AWSConfig(
    __ENV.AWS_REGION,
    __ENV.AWS_ACCESS_KEY_ID,
    __ENV.AWS_SECRET_ACCESS_KEY
)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The next step is to create a S3 client by wrapping <code>AWSConfig</code> into it using the below code.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>const s3 = new S3Client(awsConfig);</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>S3Client <code>s3</code> has the following methods:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>listBuckets()</li><li>listObjects(bucketName, [prefix])</li><li>getObject(bucketName, objectKey)</li><li>putObject(bucketName, objectKey, data)</li><li>deleteObject(bucketName, objectKey)</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Now, we know the <code>s3</code> methods. The next step is to create a dummy file to upload. Create a file with sample contents and save it in a current directory, e.g. <code>test.txt</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After creating a dummy file to upload, we need to load that file into the script using the <code>open()</code> method. Copy and paste the below code:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>const data = open('test.txt', 'r')
const testBucketName = 'k6test'
const testFileKey = 'test.txt'</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><code>open()</code> method reads the contents of a file and loads them into memory which will be used in the script. <code>open()</code> method takes two arguments: file path and the mode. By default, it will read it as text <code>r</code>, to read it as binary use <code>b</code>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The above <code>open</code> method works only in an <code>init</code> context. Please make a note. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The above variables <code>data</code>, <code>testBucketName</code> and <code>testFileKey</code> hold the data to upload, bucket name in S3, and file key respectively. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The next step is to define the <code>main()</code> context. Let us begin with the listing of the buckets. The below variable <code>buckets</code> will return the array which will contain each bucket object.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>const buckets = s3.listBuckets()</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Optionally, if you would like to loop through the bucket, use the below code snippet.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>for (let bucket in buckets) {
        console.log(buckets&#91;bucket].name);
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Or you can use <code>filter()</code> method as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>buckets.filter(bucket =&gt; bucket.name === testBucketName)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let us add a checkpoint whether the bucket is present or not. If the bucket is present, it will proceed to upload, else the execution will abort. Copy and paste the below snippet.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>if (buckets.filter((bucket) =&gt; bucket.name === testBucketName).length == 0) {
        exec.test.abort()
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The next step is to upload the object to S3 using <code>putObject()</code> method.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>s3.putObject(testBucketName, testFileKey,data)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Here is the final script.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>import exec from "k6/execution";

import { AWSConfig, S3Client } from "https://jslib.k6.io/aws/0.3.0/s3.js";

const awsConfig = new AWSConfig(
  __ENV.AWS_REGION,
  __ENV.AWS_ACCESS_KEY_ID,
  __ENV.AWS_SECRET_ACCESS_KEY
);

const s3 = new S3Client(awsConfig);

const data = open("test.txt", "r");
const testBucketName = "k6test";
const testFileKey = "test.txt";

// main function
export default function () {
  const buckets = s3.listBuckets();


  if (buckets.filter((bucket) =&gt; bucket.name === testBucketName).length == 0) {
    exec.test.abort();
  }

  s3.putObject(testBucketName, testFileKey, data);
  console.log("Uploaded " + testFileKey + " to S3");

}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Save the above script and execute the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>k6 run -e AWS_REGION=ZZ-ZZZZ-Z -e AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXX -e AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYYYYYYY <code>uploadFiletoS3<span style="background-color: inherit; font-size: inherit; color: inherit;">.js</span></code></code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To store the variables in PowerShell, you can use the below command e.g.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>Set-Variable -Name "AWS_REGION" -Value "us-east-2"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To execute, you can use the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>k6 run -e AWS_REGION=$AWS_REGION -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY <code>uploadFiletoS3<span style="background-color: inherit; font-size: inherit; color: inherit;">.js</span></code></code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Navigate to the S3 console and go to the bucket to check the file object. You can download the file to verify the contents.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9349,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-1024x347.png" alt="S3 Validation" class="wp-image-9349"/></a><figcaption>S3 Validation</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Congratulations! You successfully uploaded the file to S3. If you would like to delete the file object, use the below code snippet.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>s3.deleteObject(testBucketName, testFileKey)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To read the content from S3 bucket, you can use the below snippet.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>const fileContent = s3.getObject(testBucketName, testFileKey);
console.log(fileContent.data);</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Final Thoughts</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The k6 AWS library is neatly designed with frequently used AWS services and methods. Right now, it supports S3 Client, Secret Manager Client, and AWS Config. Hopefully, the k6 team will add more services which will help developers and performance engineers.</p>
<!-- /wp:paragraph -->