A Guide to Load Testing Node.js APIs with Artillery
Tags: node
Published: 2021-11-24 12:43:18

[Artillery](https://artillery.io/) is an open-source command-line tool purpose-built for load testing and smoke testing web applications. It is written in JavaScript and it supports testing HTTP, Socket.io, and WebSockets APIs.

This article will get you started with load testing your Node.js APIs using Artillery. You'll be able to detect and fix critical performance issues before you deploy code to production.

Before we dive in and set up Artillery for a Node.js app, though, let's first answer the question: what is load testing and why is it important?

## Why Should You Do Load Tests in Node.js?

Load testing is essential to quantify system performance and identify breaking points at which an application starts to fail. A load test generally involves simulating user queries to a remote server.

Load tests reproduce real-world workloads to measure how a system responds to a specified load volume over time. You can determine if a system behaves correctly under loads it is designed to handle and how adaptable it is to spikes in traffic. It is closely related to [stress testing](https://www.blazemeter.com/blog/performance-testing-vs-load-testing-vs-stress-testing), which assesses how a system behaves under extreme loads and if it can recover once traffic returns to normal levels.

Load testing can help validate if an application can withstand realistic load scenarios without a degradation in performance. It can also help uncover issues like:

- Increased response times
- Memory leaks
- Poor performance of various system components under load

As well as other design issues that contribute to a suboptimal user experience.

In this article, we'll focus on the free and open-source version of Artillery to explore load testing. However, bear in mind that a [pro version of Artillery](https://artillery.io/pro/) is also available for those whose needs exceed what can be achieved through the free version. It provides added features for testing at scale and is designed to be usable even if you don't have prior DevOps experience.

## Installing Artillery for Node.js

Artillery is an [npm package](https://www.npmjs.com/package/artillery) so you can install it through `npm` or `yarn`:

```bash
$ yarn global add artillery
```

If this is successful, the `artillery` program should be accessible from the command line:

```bash
$ artillery -V
        ___         __  _ ____                  _
  _____/   |  _____/ /_(_) / /__  _______  __  (_)___  _____
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ / / / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__, (_)_/\____/
                                    /____/

------------ Version Info ------------
Artillery: 1.7.7
Artillery Pro: not installed (https://artillery.io/pro)
Node.js: v16.7.0
OS: linux/x64
--------------------------------------
```

## Basic Artillery Usage

Once you've installed the Artillery CLI, you can start using it to send traffic to a web server. It provides a `quick` subcommand that lets you run a test without writing a test script first.

You'll need to specify:

- an endpoint
- the rate of virtual users per second or a fixed amount of virtual users
- how many requests should be made per user

```bash
$ artillery quick --count 20 --num 10 http://localhost:4000/example
```

The `--count` parameter above specifies the total number of virtual users, while `--num` indicates the number of requests that should be made per user. Therefore, 200 (20*10) GET requests are sent to the specified endpoint. On successful completion of the test, a report is printed out to the console.

```bash
All virtual users finished
Summary report @ 14:46:26(+0100) 2021-08-29
  Scenarios launched:  20
  Scenarios completed: 20
  Requests completed:  200
  Mean response/sec: 136.99
  Response time (msec):
    min: 0
    max: 2
    median: 1
    p95: 1
    p99: 2
  Scenario counts:
    0: 20 (100%)
  Codes:
    200: 200
```

This shows several details about the test run, such as the requests completed, response times, time taken for the test, and more. It also displays the response codes received on each request so that you can determine if your API handles failures gracefully in cases of overload.

While the `quick` subcommand is handy for performing one-off tests from the command line, it's quite limited in what it can achieve. That's why Artillery provides a way to configure different load testing scenarios through test definition files in YAML or JSON formats. This allows great flexibility to simulate the expected flows at one or more of your application's endpoints.

## Writing Your First Artillery Test Script

In this section, I'll demonstrate a basic test configuration that you can apply to any application. If you want to follow along, you can set up a test environment for your project, or run the tests locally so that your production environment is not affected. Ensure you install Artillery as a development dependency so that the version you use is consistent across all deployments.

```bash
$ yarn add -D artillery
```

An Artillery test script consists of two main sections: `config` and `scenarios`. `config` includes the general configuration settings for the test such as the target, response timeouts, default HTTP headers, etc. `scenarios` consist of the various requests that virtual users should make during a test. Here's a script that tests an endpoint by sending 10 virtual users every second for 30 seconds:

```yaml
config:
  target: "http://localhost:4000"
  phases:
    - duration: 30
      arrivalRate: 10

scenarios:
  - name: "Retrieve data"
    flow:
      - get:
          url: "/example"
```

In the above script, the `config` section defines the base URL for the application that's being tested in the `target` property. All the endpoints defined later in the script will run against this base URL.

The `phases` property is then used to set up the number of virtual users generated in a period of time and how frequently these users are sent to specified endpoints.

In this test, `duration` determines that virtual users will be generated for 30 seconds and `arrivalRate` determines the number of virtual users sent to the endpoints per second (10 users).

On the other hand, the `scenarios` section defines the various operations that a virtual user should perform. This is controlled through the `flow` property, which specifies the exact steps that should be executed in order. In this case, we have a single step: a GET request to the `/example` endpoint on the base URL. Every virtual user that Artillery generates will make this request.

Now that we've written our first script, let's dive into how to run a load test.

## Running a Load Test in Artillery

Save your test script to a file (such as `load-test.yml`) and execute it through the command below:

```bash
$ artillery run path/to/script.yml
```

This command will start sending virtual users to the specified endpoints at a rate of 10 requests per second. A report will be printed to the console every 10 seconds, informing you of the number of test scenarios launched and completed within the time period, and other statistics such as mean response time, HTTP response codes, and errors (if any).

Once the test concludes, a summary report (identical to the one we examined earlier) is printed out before the command exits.

```bash
All virtual users finished
Summary report @ 15:38:48(+0100) 2021-09-02
  Scenarios launched:  300
  Scenarios completed: 300
  Requests completed:  300
  Mean response/sec: 9.87
  Response time (msec):
    min: 0
    max: 1459
    median: 1
    p95: 549.5
    p99: 1370
  Scenario counts:
    Retrieve data: 300 (100%)
  Codes:
    200: 300
```

## How to Create Realistic User Flows

The test script we executed in the previous section is not very different from the `quick` example in that it makes requests to only a single endpoint. However, you can use Artillery to test more complex user flows in an application.

In a SaaS product, for example, a user flow could be: someone lands on your homepage, checks out the pricing page, and then signs up for a free trial. You'll definitely want to find out how this flow will perform under stress if hundreds or thousands of users are trying to perform these actions at the same time.

Here's how you can define such a user flow in an Artillery test script:

```yaml
config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
      name: "Warming up"
    - duration: 240
      arrivalRate: 20
      rampTo: 100
      name: "Ramping up"
    - duration: 500
      arrivalRate: 100
      name: "Sustained load"
  processor: "./processor.js"

scenarios:
  - name: "Sign up flow"
    flow:
      - get:
          url: "/"
      - think: 1
      - get:
          url: "/pricing"
      - think: 2
      - get:
          url: "/signup"
      - think: 3
      - post:
          url: "/signup"
          beforeRequest: generateSignupData
          json:
            email: "{{ email }}"
            password: "{{ password }}"
```

In the above script, we define three test phases in `config.phases`:

- The first phase sends 20 virtual users per second to the application for 60 seconds.
- In the second phase, the load will start at 20 users per second and gradually increase to 100 users per second over 240 seconds.
- The third and final phase simulates a sustained load of 100 users per second for 500 seconds.

By providing several phases, you can accurately simulate real-world traffic patterns and test how adaptable your system is to a sudden barrage of requests.

The steps that each virtual user takes in the application are under `scenarios.flow`. The first request is `GET /` which leads to the homepage. Afterward, there is a pause for 1 second (configured with `think`) to simulate user scrolling or reading before making the next GET request to `/pricing`. After a further delay of 2 seconds, the virtual user makes a GET request to `/signup`. The last request is `POST /signup`, which sends a JSON payload in the request body.

The `{{ email }}` and `{{ password }}` placeholders are populated through the `generateSignupData` function, which executes before the request is made. This function is defined in the `processor.js` file referenced in `config.processor`. In this way, Artillery lets you specify custom hooks to execute at specific points during a test run. Here are the contents of `processor.js`:

```javascript
const Faker = require('faker');

function generateSignupData(requestParams, ctx, ee, next) {
  ctx.vars['email'] = Faker.internet.exampleEmail();
  ctx.vars['password'] = Faker.internet.password(10);

  return next();
}

module.exports = {
  generateSignupData,
};
```

The `generateSignupData` function uses methods provided by [Faker.js](https://github.com/marak/Faker.js/) to generate a random email address and password each time it is called. The results are then set on the virtual user's context, and `next()` is called so that the scenario can continue to execute. You can use this approach to inject dynamic random content into your tests so they're as close as possible to real-world requests.

Note that other [hooks](https://artillery.io/docs/guides/guides/http-reference.html#Function-signatures) are available aside from `beforeRequest`, including the following:

- `afterResponse` - Executes one or more functions after a response has been received from the
endpoint:

```yaml
- post:
    url: "/login"
    afterResponse:
      - "logHeaders"
      - "logBody"
```

- `beforeScenario` and `afterScenario` - Used to execute one or more functions before or after each request in a scenario:

```yaml
scenarios:
  - beforeScenario: "setData"
    afterScenario: "logResults"
    flow:
      - get:
          url: "/auth"
```

- `function` - Can execute functions at any point in a scenario:

```yaml
- post:
    url: "/login"
    function: "doSomething"
```

## Injecting Data from a Payload File

Artillery also lets you inject custom data through a payload file in CSV format. For example, instead of generating fake email addresses and passwords on the fly as we did in the previous section, you can have a predefined list of such data in a CSV file:

```plaintext
Dovie32@example.net,rwkWspKUKy
Allen.Fay@example.org,7BaFHbaWga
Jany30@example.org,CWvc6Bznnh
Dorris47@example.com,1vlT_02i6h
Imani.Spencer21@example.net,1N0PRraQU7
```

To access the data in this file, you need to reference it in the test script through the `config.payload.path` property. Secondly, you need to specify the names of the fields you'd like to access through `config.payload.fields`. The [`config.payload` property provides several other options](https://artillery.io/docs/guides/guides/test-script-reference.html#Payload-file-options) to configure its behavior, and it's also possible to specify multiple payload files in a single script.

```yaml
config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./auth.csv"
    fields:
      - "email"
      - "password"

scenarios:
  - name: "Authenticating users"
    flow:
      - post:
          url: "/login"
          json:
            email: "{{ email }}"
            password: "{{ password }}"
```

## Capturing Response Data From an Endpoint

Artillery makes it easy to capture the response of a request and reuse certain fields in a subsequent request. This is helpful if you're simulating flows with requests that depend on an earlier action's execution.

Let's assume you're providing a geocoding API that accepts the name of a place and returns its longitude and latitude in the following format:

```json
{
  "longitude": -73.935242,
  "latitude": 40.730610
}
```

You can populate a CSV file with a list of cities:

```plaintext
Seattle
London
Paris
Monaco
Milan
```
Here's how you can configure Artillery to use each city's longitude and latitude values in another request. For example, you can use the values to retrieve the current weather through another endpoint:

```yaml
config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./cities.csv"
    fields:
      - "city"

scenarios:
  - flow:
      - get:
          url: "/geocode?city={{ city }}"
          capture:
            - json: "$.longitude"
              as: "lon"
            - json: "$.latitude"
              as: "lat"
      - get:
          url: "/weather?lon={{ lon }}&lat={{ lat }}"
```

The `capture` property above is where all the magic happens. It's where you can access the JSON response of a request and store it in a variable to reuse in subsequent requests. The `longitude` and `latitude` properties from the `/geocode` response body (with the aliases `lon` and `lat`, respectively) are then passed on as query parameters to the `/weather` endpoint.

## Using Artillery in a CI/CD Environment

An obvious place to run your load testing scripts is in a CI/CD pipeline so that your application is put through its paces before being deployed to production.

When using Artillery in such environments, it's necessary to set failure conditions that cause the program to exit with a non-zero code. Your deployment should abort if performance objectives are not met. Artillery provides support for this use case through its `config.ensure` property.

Here's an example that uses the `ensure` setting to assert that 99% of all requests have an aggregate response time of 150 milliseconds or less and that 1% or less of all requests are allowed to fail:

```yaml
config:
  target: "https://example.com"
  phases:
    - duration: 60
      arrivalRate: 20
  ensure:
    p99: 150
    maxErrorRate: 1
```

Once you run the test, it will continue as before, except that assertions are verified at the end of the test and cause the program to exit with a non-zero exit code if requirements are not met. The reason for a test failure is printed at the bottom of the summary report.

```plaintext
All virtual users finished
Summary report @ 07:45:48(+0100) 2021-09-03
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 4
  Response time (msec):
    min: 1
    max: 487
    median: 2
    p95: 443.5
    p99: 487
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20

ensure condition failed: ensure.p99 < 200
```

Aside from checking the aggregate latency, you can also run assertions on `min`, `max`, and `median` — the minimum, maximum, and median response times, respectively. Here's how to assert that requests never take more than 500 milliseconds to complete during a test run:

```yaml
config:
  ensure:
    max: 500
```

The report for a failed test will indicate the reason for failure:

```yaml
All virtual users finished
Summary report @ 08:29:59(+0100) 2021-09-03
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 3.64
  Response time (msec):
    min: 1
    max: 603
    median: 305.5
    p95: 602.5
    p99: 603
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20

ensure condition failed: ensure.max < 500
```

## Generating Status Reports in Artillery

Artillery prints a summary report for each test run to the standard output, but it's also possible to output detailed statistics for a test run into a JSON file by utilizing the `--output` flag:

```bash
$ artillery run config.yml --output test.json
```

Once the test completes, its report is placed in a `test.json` file in the current working directory. This JSON file can be visualized through [Artillery's online report viewer](https://reportviewer.artillery.io/) or converted into an HTML report through the `report` subcommand:

```bash
$ artillery report --output report.html test.json
Report generated: report.html
```

You can open the `report.html` file in your browser to view a full report of the test run. It includes tables and several charts that should give you a good idea of how your application performed under load:

![Artillery HTML report](https://blog.appsignal.com/images/blog/2021-11/artillery-report.png)

## Extending Artillery With Plugins

Artillery's built-in tools for testing HTTP, Socket.io, and Websocket APIs can take you quite far in your load testing process. However, if you have additional requirements, you can [search for plugins on NPM](https://www.npmjs.com/search?q=artillery-plugin-) to extend Artillery's functionality.

Here are some official Artillery plugins that you might want to check out:

- [artillery-plugin-expect](https://artillery.io/docs/guides/plugins/plugin-expectations-assertions.html): Helps with adding expectations to HTTP requests for functional or acceptance testing.
- [artillery-plugin-publish-metrics](https://artillery.io/docs/guides/plugins/plugin-expectations-assertions.html#Metrics): Used to send statistics from test runs to some external monitoring and observability systems.
- [artillery-plugin-fuzzer](https://artillery.io/docs/guides/plugins/plugin-fuzzer.html): Helps you [fuzz test](https://www.contrastsecurity.com/knowledge-hub/glossary/fuzz-testing) your APIs with random and unexpected payloads to your API endpoints so you can catch errors. It is based on the [Big List Of Naughty Strings](https://github.com/minimaxir/big-list-of-naughty-strings/).
- [artillery-plugin-metrics-by-endpoint](https://artillery.io/docs/guides/plugins/plugin-metrics-by-endpoint.html): Breaks down response time metrics by endpoint rather than displaying aggregate values across all endpoints.

You can also [extend Artillery by creating your own plugins](https://artillery.io/blog/extend-artillery-by-creating-your-own-plugins/).

## Use Artillery for Node.js Apps to Avoid Downtime

In this article, we've described how you can set up a load testing workflow for your Node.js applications with Artillery. This setup will ensure that your application performance stays predictable under various traffic conditions. You'll be able to account well for traffic-heavy periods and avoid downtime, even when faced with a sudden influx of users.

We've covered a sizeable chunk of what Artillery can do for you, but there's still lots more to discover. 

Ensure you read the [Artillery official documentation](https://artillery.io/docs/guides/overview/welcome.html) to learn about the other features on offer.

Thanks for reading, and happy coding!

**P.S. If you liked this post, [subscribe to our JavaScript Sorcery list](https://blog.appsignal.com/javascript-sorcery) for a monthly deep dive into more magical JavaScript tips and tricks.**

**P.P.S. If you need an APM for your Node.js app, go and [check out the AppSignal APM for Node.js](https://www.appsignal.com/nodejs).**

*Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him [through his blog](https://freshman.tech/).*