Automating a11y testing: Part 2 - Beyond Axe
Published: 2023-07-20 09:51:40

Earlier this year I gave a talk on the topic of automating web accessibility (a11y) testing at the John Lewis Partnership Tech Profession Conference 2023 which I’m delighted to be sharing here in an article series format.

In [Part 1](https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-1-axe-ed3d215de126) of this series I covered a number of tools from the [Axe suite](https://www.deque.com/axe/) for static analysis of sites to find a11y violations, from framework specific packages to VSCode integrations, and sharing a few learnings with caveats and gotchas from using Axe tools.

However, this is just a small subset of available test automation tools. In this article we will start to explore some of the non-Axe tooling (though Axe will certainly pop up again!) in the wider a11y test automation space. Let’s get going!

## Linting for a11y

Although some aspects of web accessibility require manual validation as they are subjective regarding the user experience, many requirements of the [Web Content Accessibility Guidelines (WCAG)](https://www.w3.org/WAI/standards-guidelines/wcag/) are for ensuring you use the correct semantic markup and associated attributes to correctly present your content to users. Such requirements are well suited to static analysis checks as they are deterministic and rule based — indeed there is such a set of official rules set out in the [Accessibility Conformance Tests (ACT) specification](https://www.w3.org/WAI/standards-guidelines/act/) which are used by static analysis tools such as Axe that we covered in [Part 1](https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-1-axe-ed3d215de126).

> "Throughout this series I will be mostly focusing on the automation side of a11y testing, but it is also key to remember the importance of manual validation and exploratory testing" — Key considerations from [Part 1](https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-1-axe-ed3d215de126) still apply!

Although we can run static analysis checks in CI, as a form of push left and also early feedback, a really natural and effective place to run these checks is in code linting. What is linting if not static analysis of code / markup! We’ve already seen the [VSCode Axe Linter](https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-1-axe-ed3d215de126#2279) in the previous article, let’s look at some others.

### **Eslint**

If you are already plugged into the [Eslint](https://eslint.org/) ecosystem, there are a number of plugins that you can reach for depending on your framework:

* React and other JSX based frameworks — [eslint-plugin-jsx-a11y](https://www.npmjs.com/package/eslint-plugin-jsx-a11y)
* React Native — [eslint-plugin-react-native-a11y](https://github.com/FormidableLabs/eslint-plugin-react-native-a11y)
* Vue — [eslint-plugin-vue-a11y](https://github.com/maranran/eslint-plugin-vue-a11y) and / or [eslint-plugin-vuejs-accessibility](https://github.com/vue-a11y/eslint-plugin-vuejs-accessibility)
* Angular — [@angular-eslint/eslint-plugin](https://www.npmjs.com/package/@angular-eslint/eslint-plugin)
* Lit Web Components — [eslint-plugin-lit-a11y](https://www.npmjs.com/package/eslint-plugin-lit-a11y)

Mileage may vary with coverage from plugin to plugin, but each comes with the same great developer experience of real time feedback *as* you write your code as well as all the other awesome stuff you get with using Eslint such as IDE integrations with intellisense, auto-fix, etc.

![VSCode tab open on a "ProductPod Slidedown" React component. In the component a div element is being rendered with a number of event handler props for mouse enter, click, and key down. The entire div JSX is underlined in yellow and there is a hover tooltip with a warning reading: "Avoid non-native interactive elements. If using native HTML is not possible, add an appropriate role and support for tabbing, mouse, keyboard, and touch inputs to an interactive content element."](https://cdn-images-1.medium.com/max/2786/1*Y7Z3XCk4i3Kbym-tsFRy_Q.png) _A warning from [eslint-plugin-jsx-a11y](https://www.npmjs.com/package/eslint-plugin-jsx-a11y) in VSCode_

Same as your other linting rules, you can also lean into the power of running the linting both locally, whether manually or through git hooks, as well as in CI to ensure you catch issues before you even consider raising that pull request.

### Svelte

Those reading closely may have noticed that the Eslint plugin list above (which was by no means exhaustive) was missing the popular framework Svelte.

> "SvelteKit strives to provide an accessible platform for your app by default. Svelte’s [compile-time accessibility checks](https://svelte.dev/docs#accessibility-warnings) will also apply to any SvelteKit application you build." — [https://kit.svelte.dev/docs/accessibility](https://kit.svelte.dev/docs/accessibility)

Svelte (and SvelteKit) have a number of [accessibility warnings](https://svelte.dev/docs/accessibility-warnings) built into the compiler itself which cover a subset of a11y rules. It’s super cool to have these built into the framework itself, but it is worth flagging that these aren’t a catch-all. Even within the limited scope of static analysis checkers, [as flagged by one of the Svelte maintainers Geoff Rich](https://twitter.com/geoffrich_/status/1383170662651482115?s=20), there are a number of gotchas around these compiler checks.

For example, the Svelte compiler is only able to check hard-coded values in markup. If you provide a dynamic value via a variable the Svelte compiler isn’t able to determine the possible values for that attribute and thus doesn’t yeild the desired warning.

```html
<script>
  let href = '#';
</script>
    
<a href={href}>More Information</a>
```

In this example the `href="#"` should normally triggers an a11y warning from the compiler, but using the variable results in the warning being surpressed.

### Other

There are a limited number of other linting options out there for a11y, some of which are paid for options:

* [ember-template-lint](https://www.npmjs.com/package/ember-template-lint) — A linter for Ember projects that includes a11y rules.
* [AccessLint ](https://github.com/marketplace/accesslint)— A paid for (with free trial) GitHub App Integration: "AccessLint brings automated web accessibility testing into your development workflow."
* [Axe DevTools Linter](https://www.deque.com/axe/devtools/linter/) — In addition to the VSCode plugin covered in Part 1, Axe also have paid for (with free trial) integrations with GitHub Actions, SonarQube, a REST API, and a CLI for their a11y linter.

## Visual testing

Let’s pivot over to some ideas around a11y visual testing!

### Emulated vision deficiencies

Back in Chrome 83 the Chromium team released a DevTools change that allowed you to [emulate different vision deficiencies](https://developer.chrome.com/blog/new-in-devtools-83/#vision-deficiencies) in the browser. This is accessible from the [Chrome DevTools Command Palette](https://developer.chrome.com/docs/devtools/command-menu/) (open DevTools and key CMD+SHIFT+P on Mac or CTRL+SHIFT+P for Windows) by searching for "Rendering", where you can then pick from six different emulated vision deficiencies.

![Chrome open on Waitrose No.1 Dine In Meal Deal web page. Chrome DevTools is open on the "Rendering" drawer. Under "Emulate vision deficiencies" the "Blurred vision" option has been selected in the dropdown.](https://cdn-images-1.medium.com/max/7360/1*-Jxv80wsLzSQ8l1zr1lxbA.png)

Something that I’ve learned recently is that Chrome also boasts a [Chrome DevTools Protocol (CDP)](https://chromedevtools.github.io/devtools-protocol/) that allows you to set almost every feature you get in DevTools via an API. For example, for those using Playwright this example demonstrates how we can set up a CDP session and request to set an emulated blurred vision:

```ts
type EmulatedVisionDeficiency =
  | "none"
  | "blurredVision"
  | "reducedContrast"
  | "achromatopsia"
  | "deuteranopia"
  | "protanopia"
  | "tritanopia";

test("visually acceptable when have blurred vision", async ({ page }) => {
  const client = await page.context().newCDPSession(page);

  await client.send("Emulation.setEmulatedVisionDeficiency", {
    type: "blurredVision",
  });

  // ... visual test with Playwright / other third party lib
});
```

Provided your test framework supports a Chromium based browser then CDP should be available and you can make use of this capability in your based visual tests, e.g. using [Playwright screenshots](https://playwright.dev/docs/test-snapshots) or [Cypress image snapshots](https://docs.cypress.io/guides/tooling/visual-testing).

For example, here is emulated achromatopsia (where you can’t perceive colour) for a snapshot test on John Lewis Women’s Dresses product listing page (PLP):

![VSCode window with two tabs open. One containing Playwright tests for setting up different vision deficiencies via CDP and then using the Playwright screenshot API. The other tab shows the resulting screenshot for the achromatopsia test — a greyscale image of the John Lewis Women’s Dresses product listing page.](https://cdn-images-1.medium.com/max/8128/1*-CJlRbC_mCG8M07RMi2spA.png)

Here’s the same PLP from the blurred vision test:

![The John Lewis Women’s Dresses product listing page as viewed by a user with blurred vision.](https://cdn-images-1.medium.com/max/4668/1*rlJbjYw6fvWyx3DpukfTLA.png)

And finally one from the protanopia (where you can’t perceive Red light) test:

![The John Lewis Women’s Dresses product listing page as viewed by a user with protanopia.](https://cdn-images-1.medium.com/max/4668/1*K0NwIRn7RvpCdKJ0KbGbsw.png)

I wouldn’t recommend going overboard with running all of these variations for every test — visual tests are typically heavy and slow by nature, and fragile to browser upgrades, anti-aliasing, and other false positive issues (although this is somewhat mitigated through using third party vendors who typically have intelligent diffing algorithms).

>  Mileage may vary with third part integrations — if your provider performs the snapshot on the browser instance you’re running (locally or in CI) then this may well be an option, but if they do HTML + CSS snapshotting to recreate the page in different browsers on their servers then this is likely not a goer.

It could be worth considering adding to a couple to your suites for golden path smoke tests. Alternatively, it can be nice to have such tests running in production on a nightly, or weekly, where the snapshots form the basis of design review discussion opposed to a blocking gate in release.

> "In the UK, more than 2 million people are living with sight loss. Of these, around 340,000 are registered as blind or partially sighted." — [https://www.nhs.uk/conditions/vision-loss/](https://www.nhs.uk/conditions/vision-loss/)

> "Globally, at least 2.2 billion people have a near or distance vision impairment." — [https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment](https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment)

> "Colour blindness (colour vision deficiency, or CVD) affects approximately 1 in 12 men (8%) and 1 in 200 women in the world. In Britain this means that there are approximately 3 million colour blind people (about 4.5% of the entire population), most of whom are male. Worldwide, there are approximately 300 million people." — [https://www.colourblindawareness.org/](https://www.colourblindawareness.org/)

The key is to be aware and empathetic in your design and implementation towards folks who have visual impairments, and understanding of their experience of your site. A few extra, simple tests can help bring colour (pun very much intended) to otherwise dry requirements around font-size and colour contrast which are hard to appreciate in isolation — a quick change to a base font-size in your design system could be a game change for some folks experience on your site.

### Automating zoom

Another important aspect of visual accessibility is supporting higher zoom levels, typically of up to 200%. This is laid out in several success criteria in WCAG for text resize and content reflow.

> "This Success Criterion helps people with low vision by letting them increase text size in content so that they can read it." — [https://www.w3.org/WAI/WCAG21/Understanding/resize-text.html](https://www.w3.org/WAI/WCAG21/Understanding/resize-text.html)

Unfortunately most major frameworks don’t support APIs to instrument browser zoom at the moment. Searching around you might see references to the --force-device-scale-factor=2.0 device scale factor flag for Chrome, but this relates to pixel density and is not for zoom (can be passed to launchOptions.args array within your framework’s browser setup options if you want to try and see).

This limitation has been flagged for some testing frameworks so we can hope it might be supported in future. For now you can upvote the likes of [this Playwright issue for adding an option for browser zoom](https://github.com/microsoft/playwright/issues/2497).

While we wait for frameworks to catch up, the best solution that I’ve found for the time being is to simulate zoom by adding a before hook in your tests which sets the zoom style of the body. For example, this is a snippet from a Playwright visual test in which the a 200% zoom is applied through CSS:

```js
await page.evaluate('document.body.style.zoom=2.0');
```

![Waitrose The King’s Coronation Jewel The Jack Russell Cake product detail page with 200% zoom on desktop.](https://cdn-images-1.medium.com/max/3860/1*1ueqotHdWdvwnlqSNrhUWQ.png)

In many cases this isn’t necessarily representative of using the native browser zoom feature, nor the OS level zoom features for enlarging text, which is what the WCAG guidance is really for. Nevertheless, it somewhat provides an idea as to how your page behaves when magnified, and can be a useful addition to your golden path test suite to understand if critical functionality still works.

If anything this goes to show how important it is to keep up manual and exploratory testing when it comes to a11y, there is no way to get the coverage otherwise! When it comes to vision deficiencies we are really still quite limited on how much we can shift to automation — some smoke tests on Chrome are great, but what about other browsers? What about tablet and mobile?!

If anyone has any ideas on automation in this space I would love to hear them!

## Time to get funky

In this last section I wanted to share one last tool called [Funkify](https://www.funkify.org/).

>  "Funkify is an extension for Chrome that helps you experience the web and interfaces through the eyes of extreme users with different abilities and disabilities." — [https://www.funkify.org/](https://www.funkify.org/)

It is questionably accessibility test automation, as it is a browser extension... but I’ve decided it qualifies as something that automates customer personas to supplement your exploratory testing. It’s also just very cool so worth the share on that premise alone.

Because it is rather good unfortunately (but quite understandably) the maintainers have introduced a premium level around a lot of the functionality. But even if you just have a dabble with the free features I think it’s well worth it for expanding horizons.

![Waitrose groceries homepage open in a Chrome browser. In the toolbar the Funkify extension has been activated, displaying a popup with a number of personna simulators that can be selected: "Blurry Bianca, Color Carl, Dyslexia Dani, Trembling Trevor, Tunnel Toby, Peripheral Pierre".](https://cdn-images-1.medium.com/max/2532/1*COcasHsknmhnKGAUS-QOHQ.png)

The extension works well on most sites, and as you can see, provides a series of simple personas or modes that you can adopt while browsing the site. Everything from vision defects to simulated dyslexia with scrambling letters to trembling hands.

For example, here is the Waitrose groceries landing page with the Dyslexia Dani persona example where we can get a flavour of what the user experience might be like (be wary this is an example simulation, it is not necessarily representative of the experience for all folks with Dyslexia):

![Waitrose groceries homepage open in a Chrome browser. In the toolbar the Funkify extension has been activated, displaying a popup where the "Dyslexia Dani" persona has been activated. The letters in the text on the Waitrose page is slightly jumbled.](https://cdn-images-1.medium.com/max/2570/1*RSUwzgqRLdXBi2O-o5LKLg.png)

One option that really struck home for me is the Trembling Trevor persona which simulates how it might be like to use the site if you suffer from motor degenerative disorders such as Parkinson’s. Unfortunately Medium doesn’t support embedding videos directly to show this off, so I really encourage giving it a go with the free trial at least once! Once you’ve had a play with the extension on desktop, have a think about touch devices — ask yourself how confident are you with your site’s buttons, links, and dropdowns being usable on a small touch screen?

## Closing notes

In this article I’ve covered off a few additional tools and techniques for expanding a11y coverage, exploring extensions to the likes of visual testing and how this can supplement manual and exploratory testing.

This article hasn’t been extensive by far, but hopefully a flavour of what tooling is out there. If you’re keen to explore further, these are some awesome sites where you can learn more and find other tools in the a11y automation space:

* [https://www.a11yproject.com/resources/](https://www.a11yproject.com/resources/) — absolute wealth of informationa and resources, from blogs and books to tools and organisations.
* [https://a11y-automation.dev/automated-tools](https://a11y-automation.dev/automated-tools) —a comprehensive list of automated tools for a11y testing.
* [https://www.w3.org/WAI/ER/tools/](https://www.w3.org/WAI/ER/tools/) — a list of evaluation tools for web accessibility as curated by W3.

Stay tuned for part 3 of this series where we will take a look at the popular "accessibility first" [Testing Library](https://testing-library.com/) framework and start to explore the emerging field of screen reader automation tooling. See you soon! 👋

---

*Hi, my name is [Craig Morten](https://twitter.com/CraigMorten). I am a senior product engineer at the John Lewis Partnership. When I’m not hunched over my laptop I can be found drinking excessive amounts of tea or running around in circles at my local athletics track.*

---

The John Lewis Partnership are hiring across a range of roles. If you love web development and are excited by accessibility, performance, SEO, and all the other awesome tech in web, we would love to hear from you! See our open positions [here](https://www.jlpjobs.com/engineering-jobs/).