Introducing TestRail in your K6 tests
Tags: k6, testrail, azurepipeline, performance
Published: 2021-08-16 08:05:00

## Introducing TestRail in your K6 tests
Working smarter than harder or being lazy is the reason why we automate things. Let me try to show you my attempt to optimise TestRail integration with K6...

### Use case
In my day-to-day work I am using Azure Pipeline. I will describe the process of Continuous Integration. Some parts might be project-specific, but the general idea and approach won't change much if you are using different technologies.  
 
### TestRail
There is a couple of prerequisites that you have to do before we get started.
1. Go to you project and add a Test Suit:
![Screenshot 2021-08-13 at 13.32.46](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1xuo5flgcfcyw6b2r40m.png)
2. After that we want to feed the Suit with the tests. Add some tests to that Suit. Remember that everyone will be using that, so even though you know what TestX is, it is better to give it a more descriptive name:
![Screenshot 2021-08-13 at 13.38.06](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ik05xdkkqglnaa0i99b.png)
3. We will need a user that is able to create Test Runs and update the results of the tests. You can use your credentials, but I advise to use (create) the user that whole company, project or team is using during automation tests.
4. If you have a user we should create an API token (that is not mandatory, we can use our password instead, but the password might change...). In upper right corner click on user and go to "My Settings". From there create a token:
![Screenshot 2021-08-13 at 14.01.50](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/af6rt7bbewgda55dy2x2.png)

### TestRail API
Whole documentation can be found [here](https://www.gurock.com/testrail/docs/api), but we will focus only on [Runs](https://www.gurock.com/testrail/docs/api/reference/runs#addrun) and [Results](https://www.gurock.com/testrail/docs/api/reference/results#addresultsforcases).

#### Creating a Test Run:
1. You will need **Project ID** (69, in our case), which you can find in the URL when you view the project: *company.testrail.io/index.php?/projects/overview/69*
2. And **Test Suit ID** (2137 in our case) we just created: *company.testrail.io/index.php?/suites/view/2137*

#### Playing around:
I prefer to have my user:token (or user:password) encoded. So it looks "nice" in the header. Let us encode the user:token [here](https://mixedanalytics.com/knowledge-base/api-connector-encode-credentials-to-base-64/). Now we can send the cUrl:
```bash
curl -X POST \
  'https://company.testrail.io/index.php?/api/v2/add_run/69' \
  -H 'Authorization: Basic myTokenHere' \
  -H 'Content-Type: application/json' \
  -d '{
	"suite_id": 2137,
	"name": "Performance Test Run",
	"include_all": true
}'
```
You just created a new test run! If the above is working for you, we can proceed to the next step. Keep in mind what was the response, we will need that later.

#### Creating Test Run in our framework
We want to create a Test Run each time we run the scenarios from our Test Suit. Meaning we only want to create it once and then change the results of the tests in that Test Run when our K6 tests are done.
I am using Azure Pipeline so I have a step in my K6 job that looks like this:
```YAML
- task: PowerShell@2
  name: CreateTestRun
  inputs:
    targetType: 'filePath'
    filePath: $(System.DefaultWorkingDirectory)/scripts/CreateTestRun.ps1
    arguments: -token $(trtoken)
```
and the content of the file CreateTestRun.ps1 is:
```powershell
param ($token)

$Uri = "https://company.testrail.io/index.php?/api/v2/add_run/69"
$Body = @{
        suite_id = 2137
        name = "Performance Test Run: $(Get-Date -Format "dd/MM/yyyy, HH:mm")"
        include_all = true
} | ConvertTo-Json
$Headers = @{
        "Authorization" = "Basic $token"
        "Content-Type" = "application/json"
}
$Result = Invoke-RestMethod -Uri $Uri -Method 'Post' -Headers $Headers -Body $Body
$runId = $Result.id
    
Write-Host "##vso[task.setvariable variable=runId;isOutput=true]$runId"
```
The script is grabbing and saving the ID of the created Test Run so I can pass it later to K6:
```YAML
-e RUNID=$(CreateTestRun.runId)
-e TRTOKEN=$(trtoken)
```

#### Updating test results in K6
After we created a Test Run and we have its ID we can start our K6 tests. Each time we finish one of our scenarios, we should update the results in that run.
K6 scenario is passed if it passes all of our [thresholds](https://k6.io/docs/using-k6/thresholds/). So we have to work around that. After the test a report is created. In each of my scenarios I have a [handleSummary](https://k6.io/docs/results-visualization/end-of-test-summary/#handlesummary-callback) function:
```javascript
import tools from '../tools/tools.js'
import { textSummary } from 'https://jslib.k6.io/k6-summary/0.0.1/index.js';

export function handleSummary(data) {
    const success = tools.CheckThresholds(data, 'iteration_duration')
    tools.UpdateTestrail(success, 142927)
    return {
        'logs/DataGridTest2.json': JSON.stringify(data),
        'stdout': textSummary(data, { indent: ' ', enableColors: true}),
    }
}
```
Now, there is a couple of things I have to explain. We won't go deep into my framework so I will just explain what we are seeing here.
```javascript
tools.CheckThresholds()
```
Is the function that checks the status of my thresholds. It is a custom function that I created and stored in tools.js file:
```javascript
function CheckThresholds(data, metric) {
    const res = data['metrics'][metric]['thresholds'] ? data['metrics'][metric]['thresholds'] : true
    return Object.values(res).every(obj => obj['ok'])
}
```
It checks the chosen metric, either built-in or custom. 

OK, so we now know if the test is passed or failed. Time to update it! Let us take a look at
```javascript
tools.UpdateTestrail()
```
and unfold the mistery what *142927* number means...
```javascript
function UpdateTestrail(success, caseId) {
    const host = 'https://company.testrail.io'
    let status = 3 //untested

    if (success) {
        status = 1
    } else {
        status = 5
    }

    let body = {
        "results": [
            {
                "case_id": caseId,
                "status_id": status,
            }
        ]
    };
    let params = { 
        headers: {
            Authorization: `Basic ${__ENV.TRTOKEN}`,
            'content-type': 'application/json'
        }
    };

    http.post(`${host}/index.php?/api/v2/add_results_for_cases/${__ENV.RUNID}`,JSON.stringify(body),params);
}
```
Let us go back to our original Test Suit, which is a base for our Test Run:
![Screenshot 2021-08-13 at 13.38.06](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ik05xdkkqglnaa0i99b.png)Number *142927* is simply one of our Tests!

#### What happen?
We created Test Run and grabbed its ID.
Then we passed that ID to K6 and run a scenario.
We have thresholds set up so we know when our scenario fails.
After the test is done we updated the results in TestRail.
Now we can proceed to the next scenario, where the only thing that is different is the TestId (142927). And our updated Test Run will look something like that:![Screenshot 2021-08-17 at 09.13.18](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zvgy1qx6v7t4dwc66920.png)