On-Call for QA Engineers
Tags: slack, results, reporting, automation
Published: 2022-09-10 05:28:43

QA Engineers spend a considerable amount of **time** and **effort** in developing automation frameworks and even more on adding tests and maintaining them. 

Given the flaky nature of these end-to-end automation tests, most of the teams design **alerting** mechanisms for test failures.


Communication platforms like slack or microsoft teams are becoming the new **graveyard** for these alerts as most engineers end-up ignoring them over a period of time. The primary reasons for ignoring these alerts are

- Excess Alerts
- Lack of Visibility
- Poor Accountability

In the following sections, sample config files of [Test Results Reporter](https://test-results-reporter.github.io/) cli tool are shared to follow the best practices while sending your test results to slack, microsoft teams or any other communication platforms.

{% github test-results-reporter/reporter %}

## Excess Alerts

During the initial phases, looking at these alerts in your slack channels or any other place is going to be a fancy experience. After some days, we get used to ignoring them if there are more alerts than we can handle. Due to this, the **real problems** are lost in the noise.

One way to solve this is by sending the failure alerts to a **different channel** by keeping your primary channel clean and out of noise.

#### Primary Channel

![Slack Test Summary Slim](https://github.com/test-results-reporter/test-results-reporter.github.io/blob/main/docs/assets/images/slack/slack-test-summary-slim.png?raw=true)

#### Failure Channel

![Slack Failure Details](https://github.com/test-results-reporter/test-results-reporter.github.io/blob/main/docs/assets/images/slack/slack-failure-details.png?raw=true)

#### Config

```json
[
  {
    "name": "slack",
    "inputs": {
      "url": "<primary-channel-webhook-url>",
      "publish": "test-summary-slim"
    }
  },
  {
    "name": "slack",
    "condition": "fail",
    "inputs": {
      "url": "<failure-channel-webhook-url>",
      "publish": "failure-details"
    }
  }
]
```

## Lack of Visibility

Test Failure Analysis is where value is realised and quality is improved. It is highly critical for the velocity of your releases. Failures happen for a wide variety of reasons and keeping a track of these failure reasons is a tedious task. 

[ReportPortal](https://reportportal.io/) is an AI powered test results portal. It uses the collected history of executions and fail patterns to achieve hands-free analysis of your latest results. With auto analysing power, it helps to distinguish between real failures and noise (*known failures or flaky tests*). 

It provides the following failure categories

- Product Bug (**PB**)
- Automation Bug (**AB**)
- System Issue (**SI**)
- No Defect (**ND**)
- To Investigate (**TI**)

#### [Report Portal Analysis](https://test-results-reporter.github.io/extensions/report-portal-analysis.html)

![Slack Report Portal Analysis](https://github.com/test-results-reporter/test-results-reporter.github.io/blob/main/docs/assets/images/slack/slack-report-portal-analysis.png?raw=true)

#### Config

```json
{
  "name": "report-portal-analysis",
  "inputs": {
    "url": "<report-portal-base-url>",
    "api_key": "<api-key>",
    "project": "<project-id>",
    "launch_id": "<launch-id>"
  }
}
```

## Poor Accountability

Investigating and fixing the test failures is the most important part of maintaining automation test suites. It makes the automation **healthy** and **trustworthy**.

One way we often see to infuse accountability is by asking the test owners (*who has written or adopted the tests*) to look at the failures. In this approach, the knowledge is restricted and creates a dependency on others.

Another way is to make the entire team responsible for all test failures. It would be a disaster, if the entire team tries to investigate the same failure at the same time or no one takes a look at them assuming others are investigating. It would be worth creating a roster for the person to look at the failures.

**Auto tagging** the person who is on-call to investigate the test failures would definitely help in maintaining the automation suites healthy. 

#### [Mentions](https://test-results-reporter.github.io/extensions/mentions.html)

![Slack Mentions](https://github.com/test-results-reporter/test-results-reporter.github.io/blob/main/docs/assets/images/slack/slack-mentions.png?raw=true)

#### Config

```json
{
  "name": "mentions",
  "inputs": {
    "schedule": {
      "layers": [
        {
          "rotation": {
            "every": "week",
            "users": [
              {
                "name": "Jon",
                "slack_uid": "ULA15K66N"
              },
              {
                "name": "Anudeep",
                "slack_uid": "ULA15K66M"
              }
            ]
          }
        }
      ]
    }
  }
}
```