Performance Testing via Artillery.io - user guide
Tags: artilleryio, testing, automation, loadtesting
Published: 2022-03-09 12:33:50


Valor's Head of QA Dmitry has shared his experience using Artillery.io including tips and tricks for beginners. The following story is on his behalf, and the original material was published on [the company's blog](https://valor-software.com/articles/performance-testing-via-artillery-io.html).

## Intro
Today, I’m going to tell you all about setting up and using [Artillery.io](https://artillery.io/) on your projects. We have a lot of popular testing tools like JMeter, Gatling, LoadNinja, etc. So you might be wondering why I’ve chosen Artillery. Well, I’m going to show its advantages and share handy usage tips later in the article.
‍
‌Artillery.io is a modern performance testing software that is powerful yet quite simple to navigate. It suits different types of performance testing, on local machines and as part of [CI](https://en.wikipedia.org/wiki/Continuous_integration). What’s even better? – Artillery provides [a free ‘Dev’ option](https://www.artillery.io/pricing) if you want to test the waters before diving in. So, let's make some exploration!

## Why Artillery.io?
Artillery offers many services that make performance and load testing faster and more reliable. Now let me outline a few reasons why I choose to use it over the better-known software.

1. Node.js tool
Java-based tools can be complicated and time-consuming to work with. Artillery.io is a Node.js tool with a basic СLI, which is a far simpler way to run performance tests.

2. Broad support of protocols, frameworks, and libraries
Out of the box, Artillery.io will support:

- [HTTP](https://www.artillery.io/docs/guides/guides/http-reference)
- [Socket.IO](https://www.artillery.io/docs/guides/guides/socketio-reference)
- [WebSocket](https://www.artillery.io/docs/guides/guides/ws-reference)
However, you can use Artillery.io to test any stack, thanks to the plugin interface. Here are a few examples:

- Apache Kafka
- Amazon Kinesis
- [HLS](https://www.artillery.io/docs/guides/plugins/plugin-hls) (HTTP Live Streaming)‌
3. YAML format
Unlike popular competitors’ testing tools, Artillery.io writes tests in YAML format which makes them readable and easy to understand. Artillery also supports a JSON format for testing.

4. Accessible scenarios
With Artillery's scenarios and phases, you can build more flexible tests and cover multiple use cases in parallel. Also, it takes you just a few steps and conditions to set up tests for complex user behaviours. Finally, it's easy to reuse scenarios for comparison and future testing.

5. Easy integration
Artillery.io seamlessly integrates with CI (for example, check [the integration guide for GitHub Actions](https://artillery.io/docs/guides/integration-guides/github-actions.html)), making the process very organized. It also [works with Docker](https://artillery.io/docs/guides/guides/docker.html), so you can run your Artillery tests inside a Docker container.

6. Detailed reports
After running tests, you can create a JSON report through the --output flag. Also, you can generate an HTML report out of JSON to save it as a file or export it to your browser. Find details on reporting in the Scenario section below.

7. Artillery Docs
Artillery provides [numerous documents](https://artillery.io/docs/guides/overview/welcome.html) to browse for instructions and other useful info. This helps you get the performance testing software up and running much smoother than with other similar tools. Also, you can look up support from other developers on [community forums](https://github.com/artilleryio/artillery/discussions).
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s88f6p7zky1ap20j7xn6.png)

## How does Artillery.io work?
I’m going to walk you through some basic commands to get started with Artillery.io using [ngx-bootstrap](https://valor-software.com/ngx-bootstrap/#/) in my examples. Ngx-bootstrap, together with other community libraries, were created by [Valor Software](https://valor-software.com/), the company where I’m working. And as we have hundreds of thousands of engineers using our products monthly, we need scalable performance testing to make sure everything runs smoothly. So, let’s get started!

First, you need to install the latest version of the software.

**Install**
To install the latest version, enter:
`npm install -g artillery@latest`

**Fast test**
To run a fast test, enter:
`artillery quick --count 15 -n 30 https://valor-software.com/ngx-bootstrap/`

With this command, the test will run 15 virtual users with each of them making 30 HTTP GET requests to the URL address that you specified.‌

This command also supports other flags to configure your fast tests:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/geb82bf05wunp49f870w.png)
 
Below is a list of possible configurations of the run command:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l4ttiuprlhrqhv3cehs5.png)
 
Config
For standard test scenarios, you should create a ‘my.yml’ file with config.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k6gigbngiodg0n69k125.png)
 
The my.yml file should contain ‘config:’ paired with the target URL. To set up environments for your tests, you can write environments into the command after the target URL has been entered.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kff05qltnvm7vfuuqmjd.png)
 
**Phases in the config**
Artillery.io allows you to enter multiple sequential load options for the application. The testing phase consists of several aspects:

- duration: the time of one phase;
- arrivalRate: the number of users added each second;
- rampTo: up to how many users per second the load will grow by;
- name: a name of the phases.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/diagwqenw78jk2wu7qex.png)
 
If you only have one target URL, the different phases of the performance testing are placed right after it. If there are multiple environments, then you should add phases to each environment variable.

**Plugins**
In the Artillery npm utility, you can find [lots of plugins](https://www.npmjs.com/search?q=artillery-plugin-&page=0&perPage=20) that can help you in your performance testing. Install [artillery-plugin-expect](https://www.npmjs.com/package/artillery-plugin-expect) to compare the expected result with the actually received result.

Then, after the phases inside the config, enter:
`plugins:
   expect: {}`
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f5bue1uwi6g93oalkwn6.png)

Now, with the config completed, we can finally write some tests!

**Scenario**
All tests should be written in the `scenarios` section and should contain:

- GET, POST, PUT, DELETE, and some other commands;
- a URL for every endpoint;
- the body text in JSON format;
- all checks you want to run.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x1mvdne038vksxu2z0g2.png)
 
To run tests, you should enter:
`artillery run my.yml`

To run a test and then generate a report in a .txt file, run:
`artillery run -o first.txt my.yml`

After generating a .txt report, to generate a second report in HTML, run the command:
`artillery report first.txt`

Here's the report example:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pmupyebb5o01au5auoub.png)
 
**Authentication**
With Artillery.io, you can use basic authentication or get authorized by tokens uploading CSV with your credentials. Just pick an option that suits your project best.

Check their [official documentation](https://artillery.io/docs/guides/guides/http-reference.html#Basic-Authentication) to find information on authentication and many other aspects of Artillery.io.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/km1rl4kya5lf2wgzml0r.png)
 
Let's first revise how you can get authorized by tokens. To upload a CSV file with credentials, you should add the following lines into the config file:

- payload – to use the payload functionality;
- path – to write a path for the CSV file that Artillery needs to use;
- fields – names of fields that you need to use.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zh3j4vj4rsephj5q2erz.png)
 
See an example with a CSV file:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4b03vkcvyjuo4yfwc66n.png)
 
The second way to get authorized is through environment variables. And we have several options for this. The first one is to execute an export command in your console as below:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/idpscavzgskdfqhl61kk.png)
 
And after it's done, you can use your variable inside your yml file:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e12488bygprnifi65hki.png)
 
Another option is to write a new script with an environment variable in your package.json file to execute your tests without typing the export variable in the terminal each time. It will look like this:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/onpjnvgu8gzanf7j4fdt.png)
 
Then all you need to do is run this script in the terminal:
`npm run artillery:run`

Finally, you can upload CSV to pass parameters for your test requests: GET, POST, PUT, etc.
Check an example with POST:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cb2wa8og00905mjcef7w.png)
  
**Final thoughts**
Artillery.io is an accessible and comprehensive tool that I use and can surely recommend for performance testing.

With a free [Artillery Core](https://artillery.io/docs/guides/getting-started/installing-artillery.html) version, you can run tests from a local or virtual machine. And by switching to [Artillery Pro](https://artillery.io/docs/guides/getting-started/installing-artillery-pro.html), you're getting a self-hosted AWS performance testing platform.

Other features of Artillery Pro include:

- runs millions of tests per second across 13 geographical regions;
- works with existing security systems;
- no repeat charges or paid maintenance;
- VPC internal test services.

I hope this article has given you some clarity on Artillery’s software, and helped you make a choice regarding the performance testing tool you want to work with. ‌‍

In case you’re looking for help in software testing, or your project needs an advanced quality assurance pipeline – [drop us a line](https://valor-software.com/contact.html)!

**Useful Links**
- [Installation through npm utility guide](https://www.artillery.io/docs/guides/getting-started/installing-artillery)
- [Artillery docs](https://artillery.io/docs/guides/overview/welcome.html)
- [Artillery plugins on npm](https://www.npmjs.com/search?q=artillery-plugin-&page=0&perPage=20)