Automated Accessibility Testing
Tags: testing, showdev, a11y, webdev
Published: 2021-09-21 13:24:48

As described in our <a href="https://developers.livechat.com/updates/all?category=accessibility" target='_blank'>recent series of blog posts</a>, our team has been working on improving the accessibility of the Chat Widget. While making the improvements, we wanted to maintain top-notch quality and ensure that while introducing new changes, there was no drop in standards. We use <a href='https://www.cypress.io/' target='_blank'>Cypress</a> for all our automated testing, and so we decided to extend the existing testing suite by adding automated accessibility tests. In this article, I’d like to share with you what we’ve learned and how you can do this yourself!

![LiveChat Automated Accessibility Testing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u97qkhdsi091ux501l1z.png)

## Example app overview

In order to make it easier for you to follow, I’ve created a simple app that allows the user to submit a dummy form. You can find the repository <a href='https://github.com/klarzynskik/cypress-a11y-testing' target='_blank'>here</a>. Don’t worry, the data you submit in the app is not sent anywhere, so you can play around with it as much as you want.

![Simple form with two three inputs and two buttons](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h3om04alhld7x1kze0rn.png)

It’s created using the <a href='https://reactjs.org/docs/create-a-new-react-app.html#create-react-app' target='_blank'>Create React App</a> scripts and using `yarn` as a dependency manager (<a href='https://classic.yarnpkg.com/en/docs/install/' target='_blank'>installation instructions</a>). After cloning the repository, all you need to do is install the dependencies by simply typing `yarn` (or `npm install` if you'll choose to do it without `yarn`) in the console. Then, with `yarn start` (or `npm start`), you can run the app, and it will be available at http://localhost:3000. Each step described in the article is reflected on the corresponding git branch.

The repository also contains a simple integration test created with Cypress, located in the `functional.spec.js` file.

## `cypress-axe`

In the previous article describing our <a href='https://developers.livechat.com/updates/livechat-testing-chat-widget-accessibility' target='_blank'>journey with accessibility testing</a>, I mentioned the <a href='https://www.deque.com/axe/browser-extensions/' target='_blank'>axe DevTools</a> browser extension. It allows the user to perform audits on the website, so that any accessibility issues can be discovered. This extension is based on the <a href='https://github.com/dequelabs/axe-core' target='_blank'>axe-core</a> engine — a testing tool, validating against different types of rules and best practices.

The same engine can be used with Cypress, thanks to the <a href='https://github.com/component-driven/cypress-axe' target='_blank'>cypress-axe</a> package. It adds several custom commands to the Cypress API, which utilizes “axe-core” under the hood. It makes it possible to perform a11y audits directly while testing.

Let’s, then, add the required dependencies to the project — we need the `cypress-axe` and `axe-core` (with the `--dev` flag, so that these are added as developer dependencies):

```bash
yarn add cypress-axe axe-core --dev
# or
npm install cypress-axe axe-core --save-dev
```

Now, we need to let Cypress know that we have extended the list of commands. Therefore, we should add the following import statement in `cypress/support/index.js`:

```js
​​import 'cypress-axe'
```

Great, with that in place, we can start using the library.

## Running audits

The first command we’re going to use is `injectAxe`. Basically, it inserts the `axe-core` code into the application under test. Therefore, it needs to be run after the `visit` call (and after each reload) to make sure the runtime is available on the website.

Let’s create a separate `accessibility.spec.js` test suite and utilize it in there:

```js
it('should not report any errors during the accessibility audit', () => {
    cy.injectAxe()
})
```

We can run the tests now. To get more feedback while developing, we’ll run it in Cypress Test Runner with a full GUI. This can be done via the `open` command (keep in mind to start the app earlier with `yarn start`):

```bash
yarn run cypress open
# or
npx cypress open
```

And in the runner, we select `accessibility.spec.js` to run it:

![Cypress GUI with two specs: accessibility and functional](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uiq7g6wbvv0ujglfkr4c.png)

So far, so good — the test runs without any errors. After opening the developer tools and switching the console context to our app, we should have access to the `axe` object, which indicates that it was indeed correctly injected:

![Cypress Test Runner with developers console opened and AXE object logged there](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8wc78l9ya1bv2t78uvqt.png)

It’s worth noting that the library also adds a `configureAxe` command. This corresponds to `axe.configure` from `axe-core`, described <a href='https://www.deque.com/axe/core-documentation/api-documentation/#api-name-axeconfigure' target='_blank'>here</a>. Using it is not needed in most cases but might be useful.

We can now run the audit. This is done with the `checkA11y` command. It can be run with the following parameters, all optional:

- `context` — a selector to a DOM element that should be analyzed.
- `options` — overrides for the global axe configuration whenever you want to set custom rules or checks for just that particular audit.
- `violationCallback` — custom handler for reported errors.
- `skipFailures` — allows you to prevent the test from failing on error and instead only log the error to the console.

Let’s add it to the test and run it (on branch: `step-1`):

```js
it('should not report any errors during the accessibility audit', () => {
    cy.injectAxe()
    cy.checkA11y()
})
```

The app fails a test, informing the user about the violations. Upon clicking on each of the violations, the console logs more details, and the element causing the violation in each case is highlighted:

![Cypress Test Runner with failing accessibility audit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hs3pq78ph1du2i25tbgv.png)

The log for each violation contains the following set of information:

- `command` — informing us whether a violation is an error, warning, etc.
- `id` — identifier of the violated rule, for example, `button-name` or `color-contrast.`
- `impact` — defining the severity of the violation.
- `tags` — making it easier to identify which rulesets were violated (e.g., `wcag2a`).
- `description` — brief details about the rule.
- `help` — information on how to fix the issue.
- `helpurl` — link to more details about the issue.
- `nodes` — affected elements.

## Fixing the bugs

In the example, we had three bugs reported. The first one was about the “clear the form” button missing “<a href='https://dequeuniversity.com/rules/axe/4.3/button-name?application=axeAPI' target='_blank'>discernible text</a>.” As you can see, it’s an “icon button.” This means that screen reader users won’t be able to identify what it does! One way of fixing that is by adding an `aria-label` attribute to the button.

The second issue reported is informing us that the color contrast on the submit button is incorrect. Using “inspect” from the browser’s dev tools, indeed we can see that the contrast is 1.41, while <a href='https://dequeuniversity.com/rules/axe/4.3/color-contrast' target='_blank'>the rule</a> requires at least 4.5 for smaller texts:

![Browser inspect on "Submit" button from the example app, showing the contrast being 1.41](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/27p4562f5fpco7u7my0i.png)

It can be easily fixed, for example by changing the text colors on the buttons to “black” instead of “white”:

![Browser inspect on "Submit" button from the example app, showing the contrast being 14.81](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w1uvpi3a4nm9ag08yo6a.png)

The last error is caused by the form input missing a label, and <a href='https://dequeuniversity.com/rules/axe/4.3/label' target='_blank'>each one of them should have it</a> so that screen reader users can easily navigate around. This can be fixed by either setting a `for` attribute to the `id` of the given `input` for the `label`, or if a `label` is not present, by adding a corresponding `aria-` attribute.

Let’s apply all of the fixes (branch: `step-2`) and re-run the tests:

![Cypress Test Runner with passing accessibility audit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nu89cy935l2ywar07l40.png)

Nice! The app has passed the test, meaning that there were no accessibility violations found during the audit. With that test included in the pipeline, we can make sure we have not introduced any regressions in that area.

## Audits are not all there is

Audits are a powerful way of verifying accessibility violations; however, these are not always enough. The application might pass all of the automated audits, yet still not be accessible.

One of the additional layers you can add to make sure you are building a good, accessible app, is to use  “accessible queries” for elements you interact with in the tests. In other words, you find elements by text, labels, roles, aria-attributes, etc., rather than using the IDs or data-test attributes. This applies to all layers of testing. There is a very useful project called <a href='https://testing-library.com/' target='_blank'>Testing Library</a>, which provides you with UI-centric queries out of the box. It has a separate package for many different testing methods and libraries, including the Cypress one: <a href='https://github.com/testing-library/cypress-testing-library' target='_blank'>Cypress Testing Library</a>.

Let’s take a look at the test we initially had in the repository (file: `functional.spec.js`):

```js
it('should allow submitting data', () => {
    cy.get('[id=name]').type('Dummy')
    cy.get('[id=last-name]').type('Tester')
    cy.get('[id=books]').click()
    cy.get('[type=submit]').click()
    cy.contains('Thanks for submitting the form!').should('be.visible')
})
```

It heavily relies on the IDs — these don’t matter much for the end-user since they rarely browse the web with the developer tools opened. They care much more about the labels we mentioned before, so let’s try to rewrite that test with that in mind.

First, we need to install the dependency:

```bash
yarn add @testing-library/cypress --dev
# or
npm install @testing-library/cypress --save-dev
```

Now, we add an appropriate import statement in `cypress/support/index.js`:

```js
import “@testing-library/cypress/add-commands”
```

That’s it — we can make use of all the <a href='https://testing-library.com/docs/queries/about/' target='_blank'>queries</a>! Let’s apply that to the test (branch: `step-3`):

```js
it('should allow submitting data', () => {
    cy.findByLabelText('Name:').type('Dummy')
    cy.findByLabelText('Last Name:').type('Tester')
    cy.findByLabelText('Books').click()
    cy.findByText('Submit').click()
    cy.findByText('Thanks for submitting the form!').should('be.visible')
})
```

After running the test, it still passes:

![Cypress Test Runner with passing test after introducing queries from Testing Library](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vu0w1k74hu3ozpes0wjd.png)

Using the `findByLabelText` command, we would have found the bug reported by the audit much sooner. Even if you were to decide not to use that package, referring to the <a href='https://testing-library.com/docs/queries/about/#priority' target='_blank'>priority</a> of the queries description in the Testing Library docs would be beneficial for your users, and the quality of your tests.

## Keyboard navigation

Additionally, you can test the flow of your app for just keyboard navigation. A lot of users use just the keyboard to browse the web, and it’s important that the whole application flow can be completed without using the mouse. You can read more about its importance in the <a href='https://developers.livechat.com/updates/livechat-accessibility-keyboard-navigation' target='_blank'>“Keyboard navigation”</a> article.

The way Cypress is built does not allow simulating “real” user actions out of the box. All of the actions there are triggered via JavaScript. Usually, this is not a problem; however, in terms of keyboard navigation, it is — <a href='https://docs.cypress.io/api/commands/type#Tabbing' target='_blank'>we won’t be able to use a “Tab” key</a>, which is necessary to simulate how the user would navigate around. Luckily, the <a href='https://github.com/dmtrKovalenko/cypress-real-events' target='_blank'>Cypress Real Events</a> package comes in handy (or its alternatives, e.g. <a href='https://github.com/Bkucera/cypress-plugin-tab' target='_blank'>Cypress Plugin Tab</a>). It uses the <a href='https://chromedevtools.github.io/devtools-protocol/' target='_blank'>Chrome DevTools Protocol</a> (so it only works in Chrome) in order to fire actual system events. Still, it has its downsides, though — you won’t be able to interact with certain system elements like default selects. If you were to determine that this is an important part of your app, other testing frameworks might be necessary.

## Creating a simple test for keyboard navigation

Let’s create a simple test for the keyboard flow. Let’s add the package to our project:

```bash
yarn add cypress-real-events --dev
# or
npm install cypress-real-events --save-dev
```

And, as previously, import it in the `cypress/support/index.js` to let Cypress know about the new goodies:

```js
import "cypress-real-events/support"
```

Now, we should have access to all of the additional <a href='https://github.com/dmtrKovalenko/cypress-real-events#api' target='_blank'>API commands</a>. What interests us most are `realPress` (for single key presses) and `realType` (for typing).

Let’s add another simple test case, then, in the `accesibility.spec.js` suite. We must use only the keyboard to navigate around and fill in all of the inputs in the form.

Based on the application overview, after visiting the page, the first `Tab` stroke should focus us on the "Name" field within the form. Let’s check that, then, using both the `realPress` command from `cypress-real-events` and the `focused` one (<a href='https://docs.cypress.io/api/commands/focused' target='_blank'>from Cypress</a>). The latter is from the Cypress API and gets the currently focused DOM element. However, since Cypress wraps the Application Under Test (AUT) in its own, separate iframe, and the events are fired using the CDP, we would need an extra click somewhere in the AUT. This would make sure the events are correctly fired for the AUT and not the Cypress runner.

```js
it('should allow submitting with just keyboard', () => {
    cy.findByText('Dummy form').should('be.visible').realClick()
    cy.realPress('Tab')
})
```

So far so good. We would need an assertion to make sure that the focused element indeed has the “Name:” label. Let’s add a custom assertion for that:

```js
const assertFocusedInputLabel = (expectedLabel) => {
  return cy.focused().then(($el) => {
    cy.get(`label[for="${$el.attr("id")}"]`).should(
      "have.text",
      expectedLabel
    );
  });
};
```

Now, let’s add that to the test, along with the typing action:

```js
it('should allow submitting with just keyboard', () => {
    cy.findByText('Dummy form').should('be.visible').realClick()
    cy.realPress('Tab')
    assertFocusedInputLabel("Name:").realType("Dummy");
})
```

And now, let’s run it!

![Cypress Test Runner with passing test after introducing simple keyboard navigation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gvmey3uvxjsjzwmdeg5s.png)

The test passes, hooray! We can now do the same for other form elements, and finally, submit a form using the `Enter` key (branch: `step-4`):

```js
it("should allow submitting with just keyboard", () => {
    cy.findByText("Dummy form").should("be.visible").realClick();
    cy.realPress("Tab");
    assertFocusedInputLabel("Name:").realType("Dummy");
    cy.realPress("Tab");
    assertFocusedInputLabel("Last Name:").realType("User");
    cy.realPress("Tab");
    assertFocusedInputLabel("Books").realPress("Space");
    cy.realPress("Enter");
    cy.findByText("Thanks for submitting the form!").should("be.visible");
});
```

Let’s run the test and see how it does:

![Cypress Test Runner with failing test after introducing keyboard navigation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wc3u2vnjigwcharbrjdh.png)

Oh no, a failure! Upon investigation, it seems that someone has added a `tabindex=-1` attribute to the “Last name” field. That attribute will make the element “not tabbable” and as a result hide it from keyboard users. Let’s remove it (branch: `step-5`) and re-run the tests:

![Cypress Test Runner with passing test after introducing keyboard navigation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qpg92yao3its4fn34uvp.png)

Perfect, we can see that it would find a bug in such a case. As you can see, the audit has passed, as this attribute is not disallowed. This is why having an additional layer for that would be highly beneficial in the long run.

## Static analysis

The other things — apart from automated tests — you can add to prevent accessibility mistakes, are accessible plugins for the linters. In the same way they let you know about missing semicolons, they can warn you whenever necessary attributes are missing, or any other violations are made.

A good example of such a tool is the <a href='https://marketplace.visualstudio.com/items?itemName=deque-systems.vscode-axe-linter' target='_blank'>axe Accessibility Linter</a> VS code plugin. With the same `axe-core` rules engine, the plugin will inform you whenever an accessibility bug appears.

The other example is an accessibility eslint plugin for JSX elements: <a href='https://www.npmjs.com/package/eslint-plugin-jsx-a11y' target='_blank'>eslint-plugin-jsx-a11y</a>. Although it would not inform you about all of the errors, it would for sure prevent you from making the most basic mistakes. The eslint plugin has a big advantage over any editor plugins — it can be run as a <a href='https://gist.github.com/estorgio/e8bcaa8e87d0fcdcf85fdf598956e34c' target='_blank'>pre-commit hook</a> or on the CI before the build process. Thanks to that, the verification can be forced within the project configuration.

## Wrap-up

Making our Chat Widget accessible has been really enlightening for us, in many areas. It was also hard work, requiring a lot of learning. We believe in the value automated tests bring, and since accessibility is one of our biggest priorities, we want to make sure that it’s being thoroughly tested. We hope that with this description of our test setup, you’ll try it yourself.

Be sure to let us know if you have any questions!