Using EC2 to enhance k6 load tests
Tags: testing, aws, clojure, k6
Published: 2022-04-11 08:17:58

k6 is the best load test tool available, but it's pricey. I wanted to start unlimited VUs (virtual users) without paying a lot of money and found a neat way to do it by spawning many short-lived EC2 instances!

## Too many templates

I used a simple `user-data` script with _heredocs_. Make sure you have an AMI with Docker installed. You will also need S3 access.

Start by setting up AWS credentials.

```bash
# load_test.user-data
echo "setting up aws-cli"
mkdir -p /home/ec2-user/.aws
cat <<EOF > /home/ec2-user/.aws/config
[default]
output=json
region=us-west-1
EOF

cat <<EOF > /home/ec2-user/.aws/credentials
[default]
aws_access_key_id={{ aws-creds.access-key-id }}
aws_secret_access_key={{ aws-creds.secret-access-key }}
EOF
```

Next, you can write out the load-test and make it executable:

```bash
# load_test.user-data
echo "writing load test: {{load-test}}"
mkdir -p /home/ec2-user/load_test
chmod a+rw /home/ec2-user/load_test
cat <<EOF > /home/ec2-user/load_test/{{load-test}}
{{load-test-content|safe}}
EOF
```

We pull the k6 docker image and execute it with a single command.

```bash
# load_test.user-data
echo "running load test"
sudo -u ec2-user docker run --rm \
  --ulimit nofile=65536:65536 \
  --network host \
  -v /home/ec2-user/load_test/:/mnt/load_test:rw \
  -i loadimpact/k6 \
  run --summary-export=/mnt/load_test/out.json \
  /mnt/load_test/{{load-test}} > /dev/null
```

Finally, upload the test results to s3.

```bash
# load_test.user-data
echo "uploading results"
sudo -u ec2-user aws s3api put-object \
  --bucket load-test-results \
  --key {{uuid}}/{{instance-name}} \
  --body /home/ec2-user/load_test/out.json > /dev/null
```

IMPORTANT: You should set a `trap` function to call `systemctl halt` to shutdown the EC2 instances. Leaving EC2 instances running would be counter-productive to our goal of saving money.

You may have noticed that I have used templates in the `user-data`. We can use Selmer to supply all of the important stuff.

```clojure
(defn build-user-data
  [aws-creds uuid instance-name load-test load-test-template aws-creds]
  (selmer/render-file
   "load_test.user-data"
   {:aws-creds aws-creds
    :uuid uuid
    :instance-name instance-name
    :load-test (.getName (io/file (io/resource load-test)))
    :load-test-content (selmer/render-file load-test load-test-template)}))
```

`aws-creds` should be a map with keys `:access-key-id` and `:secret-access-key`. These are your S3 credentials.

You may have noticed that `load-test-content` is itself a template. That is so we can point the load test to an arbitrary address and host our web app from anywhere. Here is an example load-test.

```javascript
import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
    stages: [
        { duration: '30s', target: 5000 }, // ramp up
        { duration: '30s', target: 5000 }, // sustained load
        { duration: '30s', target: 0 }, // cool down
    ],
};

export default function () {
    let res = http.get('{{url-prefix}}/ping')

    check(res, {'status is 200': (r) => r && r.status === 200});

    sleep(1);
}
```

`load-test-template` should be a map with a single key `:url-prefix`. You may find more exciting uses for using templates in your load tests.  :wink:

## Orchestration

We can use the excellent Cognitect AWS library to kick off our tests.

```clojure
(defn launch-load-test-instance
  [ec2 aws-creds uuid instance-name load-test load-test-template]
  (aws/invoke ec2
              {:op :RunInstances
               :request {:ImageId "my-ami"
                         :InstanceType "my-size"
                         :MinCount 1 :MaxCount 1
                         :InstanceInitiatedShutdownBehavior "terminate"
                         :UserData (-> (build-user-data aws-creds uuid instance-name load-test load-test-template)
                                       byte-streams/to-byte-array
                                       base64/encode
                                       byte-streams/to-string)
                         :TagSpecifications [{:ResourceType "instance"
                                              :Tags [{:Key "Name" :Value instance-name}
                                                     {:Key "load-test-id" :Value uuid}]}]
                         :SecurityGroupIds [...]
                         :KeyName "my-company"}}))

(defn launch-load-test-instances
  [ec2 aws-creds uuid instance-prefix load-test load-test-template num-instances]
  (doall
   (map
    #(launch-load-test-instance ec2 aws-creds uuid
                      (str instance-prefix %)
                      load-test load-test-template)
    (range num-instances))))
```

There are some gaps in the above code that you will need to fill.

I feel like a mad scientist after writing that code. Wreak havoc, my minions!


![image of gene wilder as young Frankenstein doing his best mad scientist impression](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d8rlh8iakjwgvfzevoif.jpg)


## Gathering test results

In the previous code, you may have noticed that we supplied a `uuid` to each ec2 instance and used it in the `user-data` script as a shared bucket for all of our k6 results. I hope you kept track of it.  :smiling_imp:

```clojure
(def s3-bucket "load-test-results")

(defn get-k6-summary [s3 key]
  (->> {:op :GetObject
        :request {:Bucket s3-bucket
                  :Key key}}
       (aws/invoke s3)
       :Body
       io/reader
       json/decode-stream))

(defn get-k6-summaries [s3 uuid]
  (map
   (comp (partial get-k6-summary s3) :Key)
   (->> {:op :ListObjects
         :request {:Bucket s3-bucket
                   :Prefix uuid}}
        (aws/invoke s3)
        :Contents)))
```

Calling `get-k6-summaries` will return a sequence of all the generated results. Those individual results may be helpful for some analysis, but you should aggregate as many metrics as can be combined.

```clojure
(defn metric-type [m]
  (condp = (set (keys m))
    #{"count" "rate"} :counter
    #{"min" "max" "p(90)" "p(95)" "med" "avg"} :trend
    #{"fails" "value" "passes"} :rate
    #{"min" "max" "value"} :gauge
    :unknown))

(defn merge-summaries
  "summaries is a vector of k6 output summaries converted from JSON to EDN.
   get-k6-summaries returns this structure precisely. This function works with the various types
   of k6 metrics (https://k6.io/docs/using-k6/metrics) and does the following:
    - min values are the minimum in all the maps
    - max values are the maximum in all the maps
    - avg values are the average in all the maps
    - count is the sum in all the maps
    - rate is the average in all the maps
    - fails is the sum in all the maps
    - passes is the sum in all the maps
   p, med, & 'value' values are not included in the merged summary, but they can still be viewed
   for individual machines by inspecting the output from get-k6-summary"
  [summaries]
  (letfn [(merge-metrics [v1 v2]
            (condp = (metric-type v2)
              :counter {"count" (+ (get v1 "count") (get v2 "count"))
                        "rate" (double (/ (+ (get v1 "rate") (get v2 "rate")) 2))}
              :trend {"min" (min (get v1 "min") (get v2 "min"))
                      "max" (max (get v1 "max") (get v2 "max"))
                      "avg" (double (/ (+ (get v1 "avg") (get v2 "avg")) 2))}
              :rate {"fails" (+ (get v1 "fails") (get v2 "fails"))
                     "passes" (+ (get v1 "passes") (get v2 "passes"))}
              :gauge {"min" (min (get v1 "min") (get v2 "min"))
                      "max" (max (get v1 "max") (get v2 "max"))}
              {}))
          (merge-checks [v1 v2]
            {"passes" (+ (get v1 "passes") (get v2 "passes"))
             "fails" (+ (get v1 "fails") (get v2 "fails"))})
          (max-vus [m]
            (get-in m ["metrics" "vus_max" "max"]))]
    {:metrics (apply
               merge-with
               merge-metrics
               (map #(get % "metrics")
                    summaries))
     :checks (apply
              merge-with
              merge-checks
              (map
               #(get-in % ["root_group" "checks"])
               summaries))
     :max-vus (->> summaries
                   (map max-vus)
                   (apply +))}))
```

Call `merge-summaries` to get aggregate results.


## Tradeoffs

A paid k6 plan will provide you with a friendlier interface and better result-sets. But is $25k/year worth that amount of money? I don't know. That's up to each user.

If you're willing to manage some ec2 instances yourself and work with less information in aggregate, then perhaps this approach is better.

One problem I didn't go into any detail about is managing failed EC2 instances. Certain failure states require an external `TerminateInstances` operation: management of 'dangling' ec2 instances is now your problem.

k6 cloud will also distribute traffic across regions. I think it will also simulate clients with slower bandwidth. Could you provide all of this if you need it?

I believe you could get close to k6's paid feature-set, but only with a lot of effort. Hopefully, I have given you a starting point.

Don't forget to have fun!