Testing and QA
Tags: a11y, testing, qa, ci
Published: 2018-10-22 08:26:57

---
title: Testing and QA
published: true
description: The fifth post in a series aimed at giving you a solid basic understanding of accessibility
tags: Accessibility, Testing, QA, CI
series: Accessibility 101
---

## Introduction

In this article we will go over how to test our applications and websites for Accessibility criteria. 

> "Automated tests can’t cover everything however; only 20 to 50% of accessibility issues can be detected automatically. For example, we can’t yet automate the comparison of an alt attribute with an image’s content, and there are some screen reader tests that need to be carried out by hand too." - [Seren Davies](https://24ways.org/2017/automating-your-accessibility-tests/)

As you can see, only 20% - 50% of accessibility criterion can be tested in an automated manner. Thusly, automated testing alone should never wholly replace using best practices in development or testing with real users in user testing and looking at real data and user interactions via usability testing in the design and development stages.

For this article I will show you how to use [the aXe accessibility engine](https://www.deque.com/axe/) to test for accessibility in key areas of the testing pipeline.

Let's make a very advanced single page website:

*index.html*
```html
<!DOCTYPE html>
<html lang="en">
<head>
    <title>Greatest website in the world</title>
    <link rel="stylesheet" href="/styles/main.css">
</head>
<body>
    <h1>My awesome product</h1>
    <small>Why didn't anyone make this sooner?</small>
    <img src="https://fillmurray.com/500/500" />
</body>
</html>
```

*styles/main.css*
```css
h1 {
    font-size: 3rem;
}

small {
    color: lightgrey;
}
```

And when we open out index.html in the browser, we see the best website ever, right? Well, at the least, it will fit our purposes for now.

## Testing

Before we begin writing a test, make sure you have a valid install of [node.js](https://nodejs.org/en/).

Assuming you do, lets run `npm init -y` in a terminal that is currently cd'd into our project.

Now run the following commands: 

```bash
npm install selenium-webdriver axe-webdriverjs chromedriver -D
```

```bash
npm install express -S
```

Then we need to install the binary for the browser we wish to test on, these can be found in the [Selenium webdriver documentation](https://www.npmjs.com/package/selenium-webdriver).

After your chosen binaries are successfully installed, open the `package.json` file in the project root.

It should looks something like this:

```json
{
  "name": "testing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "axe-webdriverjs": "^2.0.1",
    "selenium-webdriver": "^4.0.0-alpha.1",
    "chromedriver": "^2.41.0"
  },
  "dependencies": {
    "express": "^4.16.3"
  }
}
```

Now, in the `scripts` section, lets add the following lines:

```json
"scripts": {
  "test": "npm run test:accessibility",
  "test:accessibility": "TEST_URL=localhost:3000 node ./tests/accessibility",
  "start:dev": "PORT=3000 node .",
  "start:prod": "node ."
}
```

And now our `package.json` should look like so:

```json
{
  "name": "testing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "npm run test:accessibility",
    "test:accessibility": "TEST_URL=localhost:3000 node ./tests/accessibility",
    "start:dev": "PORT=3000 node .",
    "start:prod": "node ."
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "axe-webdriverjs": "^2.0.1",
    "selenium-webdriver": "^4.0.0-alpha.1",
    "chromedriver": "^2.41.0"
  },
  "dependencies": {
    "express": "^4.16.3"
  }
}
```

Now we will create our server:

*index.js*
```js
const express = require("express");
const app = express();
const port = process.env.PORT || 8080;

app.use("/styles", express.static("styles"));

app.get("/", (request, response, nextRoute) => {
    response.sendFile(`${__dirname}/index.html`);
 });


app.listen(port, () => {
    console.log(`app listening on port: ${port}`)
});
```

Finally we can lastly write `npm run start:dev` into our terminal and if we navigate to `localhost:3000`, we should see our lovely website up and running!

Ok, now we have our setup complete, lets get to the fun part and write a simple test to show us all the violations our simple page contains.

*tests/accessibility/index.js*
```js
const AxeBuilder = require('axe-webdriverjs');
const { Builder } = require('selenium-webdriver');

async function TestPageByUrl(url) {
  const driver = await new Builder()
  .forBrowser('chrome')
  .build();

  try {
      await driver.get(url);
        
      const { violations } = await AxeBuilder(driver).analyze();

      if(violations.length > 0) {
        console.log(
          JSON.stringify(violations, null, 4)
        );
        process.exitCode = 1;
      }
    } catch(error) {
      throw new Error(error);
    } finally {
      await driver.close();
      await driver.quit();
    }
}

TestPageByUrl(process.env.TEST_URL)
  .catch(console.error);
```

Using this method, we ensure that if the test runs fine, nothing goes wrong but if there are violations, we can cleanly exit the process after the script has closed the `driver` instance properly. This means that in our pipelines, the tests would clearly fail, should there be any violations in the given url.

With this method, you can also do something along the lines of a `tests/accessibility/routes.js` file which contains an array of routes and tests each one in turn. Alternatively you can go further and add integration and acceptance tests by using the selenium and axe driver APIs to test whole core user flows but this is a little out of context for this article.

As you can see though, getting results is really easy and you can do alot of things based on the data returned such as generate documentation or raise tickets for each issue using the JIRA API for example.

That aside, if we stop our server instance and re-run `npm run start:dev` in our terminal and then the test with `npm run test:accessibility` in a new window, we will see a few issues are raised: Let's fix these!

## Fixing the results

### Images must have alternate text

Lets alter the image and add an alt tag
```html
  <img 
    src="https://fillmurray.com/500/500"
    alt="Bill Murray portrait" />
```
### Ensures the contrast between foreground and background colors meets WCAG 2 AA contrast ratio thresholds

Lets alter the colour of the small tag
```css
small {
  color: #666;
}
```

### Ensures all content is contained within a landmark region

Lets wrap our content in a `main` tag:

```html
<main>
  <h1>My awesome product</h1>
  <small>Why didn't anyone make this sooner?</small>
  <img 
    src="https://fillmurray.com/500/500"
    alt="Bill Murray portrait" />
</main>
```

And now when we run `npm run test:accessibility` we will see that no errors are displayed, good work!

## Gitlab CI pipeline (Continuous integration pipelines)

As I use Gitlab or Travis more often than not for pipelines when required, I figured Gitlab was the easiest to setup from my experience and the following yaml file will basically run all of the tests we detailed above once pushed into your Gitlab repo.

```yaml
image: node:9.4.0

cache:
  paths:
    - node_modules/

stages:
  - testAccessibility

TestAccessibility:
  stage: testAccessibility
  tags:
    - node
  before_script: 
    - npm install
  script:
    - npm run test:accessibility
```

Once pushed, gitlab will run all of our tests and should our pipeline pass, we have just implemented a whole load of accessibility tests which will help us consistenly push content that is as accessible as is possible via automation, pretty nice, right?

## Conclusion

As you can see, after the initial setup process, automated testing for accessibility is pretty simple and well worth implementing. Working with aXe is really intuitive and personally I feel it provides the best in market solution at the moment, so give it a go on your website or application and see how you perform overall, iteratively improve and honestly, you'll reap the rewards in the end as users will be happier, more of them will be included and overall things will be better because of your work.

## Resources

- [the aXe accessibility engine](https://www.deque.com/axe/)
- [aXe Core](https://axe-core.org/)
- [Axe webdriver](https://www.npmjs.com/package/axe-webdriverjs)
- [Selenium webdriver](https://www.npmjs.com/package/selenium-webdriver)
- [GitLab Continuous Integration & Deployment](https://about.gitlab.com/features/gitlab-ci-cd/)