Testing for Accessibility with Playwright
Tags: playwright, testing, javascript
Published: 2023-11-27 04:18:11

**Merging my favorite tool with my passion:** 🎭 **+ A11y** 

---

For my first accessibility tests, I decided to start with a couple of ready-made scripts provided in the documentation. These tests use the `@axe-core/playwright` package to evaluate a wide range of accessibility rules.

Some of these rules align with specific criteria from the Web Content Accessibility Guidelines (WCAG), while others are considered best practices but not explicitly required by WCAG. 
<br>

## Testing for WCAG A Compliance
If you are interested in scanning an entire page for accessibility violations, the following test is a good start:

```js
test.describe('homepage', () => { 
  test('should not have any automatically detectable accessibility issues', async ({ page }) => {
    await page.goto('https://corinamurg.netlify.app/'); 

    const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); 

    expect(accessibilityScanResults.violations).toEqual([]); 
  });
});
```

If you prefer to focus on rules that correspond to particular WCAG criteria, you could use `AxeBuilder.withTags()`. 

I also started with `AxeBuilder.withTags()`. My targets were the **WCAG A** guidelines for which I used the `wcag2a` and `wcag21a` tags.

```js
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('should not have any automatically detectable WCAG A violations', async ({ page }, testInfo) => {
    await page.goto('https://corinamurg.netlify.app/');
  
    const accessibilityScanResults = await new AxeBuilder({ page })
        .withTags(['wcag2a', 'wcag21a'])
        .analyze();
  
    expect(accessibilityScanResults.violations).toEqual([]);
});

```

I did not expect to see any violations, and indeed there were none. Moving on!
<br>

## Unexpected Results with WCAG AA
A different story with **WCAG AA**! Again, I did not expect any violations, but there were quite a few!

```js
test('should not have any automatically detectable WCAG AA violations', async ({ page }, testInfo) => {
    await page.goto('https://corinamurg.netlify.app/');
  
    const accessibilityScanResults = await new AxeBuilder({ page })
        .withTags(['wcag2aa', 'wcag21aa'])
        .analyze();
  
    expect(accessibilityScanResults.violations).toEqual([]);
});
```

I had previously run the **axe Dev Tools** scan and the page received a clean bill of health:

![This image shows the results of an accessibility test run on the website 'https://corinamurg.netlify.app/' using axe DevTools. The results display a total of zero issues, indicating no detected accessibility problems. There are additional categories such as 'Critical', 'Serious', 'Moderate', and 'Minor', all of which also show zero issues. The 'Best Practices' switch is toggled on, and the WCAG 2.1 AA standards have been applied to the test. The interface includes a 'Re-run scan' button, suggesting that the test can be performed again. There are options to sign up or sign in at the top right corner of the tool's interface.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tidj8er3o5erktvqtc1h.png)


But the Playwright scan revealed SIX(!) elements with color contrast issues:
- three with the `sample-project--title` class, and 
- three with the `sample-project--description` class.

All failure summaries were similar to this one:

```js
+ "failureSummary": "Fix any of the following:
+    Element has insufficient color contrast of 2.61 
       (foreground color: #a0a0a0, 
       background color: #ffffff, 
       font size: 14.4pt (19.2px), 
       font weight: normal). 
       Expected contrast ratio of 4.5:1",
 +   "html": "<p class=\"sample-project--description\">
                    React, Typescript
              </p>",
 +    "impact": "serious",
 +    "none": Array [],
 +    "target": Array [
 +    ".my-portfolio-site > .sample-project > .sample-project--description",
 +         ],
```
<br>

## Troubleshooting the Color Contrast Issue
It was quite the head-scratching moment because the CSS declaration blocks told a different story.

```css
.sample-project--title {
    align-self: start;
    display: inline-block;
    font-size: 1.7rem;
    font-weight: 500;
    margin-top: 32px;
    padding-bottom: 10px;
    position: relative; 
}
```

```css
.sample-project--description {
    font-size: 1.2rem;
    font-weight: 400;
    margin-top: 0;
}
```

To determine the color contrast of an element we need to know the foreground and the background colors. In this case, neither element has specific colors, so they inherit from the body element: white background, and black text. Doesn't this combination give the best color contrast one can ask for?! 🤔

And yet, the test revealed a color-contrast violation … Not only that, but it was referring to a text color that my site does not even use: `foreground color #a0a0a0`. 

This actually gave me the idea to be more specific about the colors for each element. So, I added `color:inherit`. And … surprise! It worked ... for the `sample-project--title` elements, that is! Running the test again revealed only 3 errors, one for each of the three `sample-project--description` elements.

After this semi-victory, I assumed the test needed more specificity so I replaced `color:inherit` with the actual hex values. No luck! (and in hindsight, no surprise. I was at a point where I was just guessing …) 😕
<br>

## Looking for Help from the WebAIM’s Contrast Checker
Was the size of the text too small? I wouldn’t have thought that to be the cause of the violation, again because of the black text on white background combination. Still, I checked WebAIM’s color contrast tool just to make sure (and because I did not know what else to do!!!):


![The image is of a Contrast Checker tool displaying the results for a color contrast test between a black foreground color (#000000) and a white background color (#FFFFFF). The contrast ratio is indicated as 21:1, which is the highest possible score, and it is enclosed in a green-outlined box. Below the ratio, there are results for 'Normal Text' and 'Large Text' showing a green "Pass" for both WCAG AA and WCAG AAA levels of compliance. Sample text "The five boxing wizards jump quickly." is shown below each compliance level to illustrate the legibility of the text with the given contrast ratio.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qiu17p1c0dg88m5hdk30.png)

Here’s also WebAIM’s specification about the relationship between font size and expected color contrast:

> WCAG 2.0 level **AA** requires a contrast ratio of at least **4.5:1 for normal text** and **3:1 for large text**. 

> WCAG 2.1 requires a contrast ratio of at least 3:1 for graphics and user interface components (such as form input borders). 

> WCAG Level **AAA** requires a contrast ratio of at least **7:1 for normal text** and **4.5:1 for large text**.

> Large text is defined as 14 point (typically 18.66px) and bold or larger, or 18 point (typically 24px) or larger.

Back to my trouble-making elements: with a font size of 19.2px and a contrast ratio of 21:1, they should pass not just the AA guidelines, but the AAA ones as well!
<br>

## Will AAA Fail Too?!
Sigh of relief … running the **WCAG AAA** scan found no errors with those particular elements. But wait ... why not? They failed the lower standard. How come they passed the higher standard?! 😲  

There were a few errors regarding another elements, and for good reasons. This is an example of one of the failure summaries:

```js
+ "failureSummary": "Fix any of the following:
+    Element has insufficient color contrast of 4.5 
       (foreground color: #1a73c6, 
        background color: #f6f6f6,
        font size: 12.0pt (16px), 
        font weight: normal). 
        Expected contrast ratio of 7:1",
+    "html": "<span class=\"footer-heading\">Profiles</span>",
+    "impact": "serious",
+    "none": Array [],
+    "target": Array [
+           ".footer-right--social > .footer-heading",
+         ],
```

This is the CSS declaration block for one of the elements with the AAA violations:

```css
.footer-heading {
    background-color : #f6f6f6
    color: #1a73c6;
    font-size: 1rem;
    font-weight: 500;
    text-transform: uppercase;
}
```

At 16px, the font-size for this element is considered “normal” so it needs a contrast ratio of 7:1 to meet the AAA guideline. The failure summary confirmed (correctly this time!) the contrast checker’s result: the element’s contrast ratio is only 4.5:1, good enough for AA but falling short of the AAA guidelines.

![The image shows a web interface of a Contrast Checker tool. It indicates the contrast ratio between a blue foreground color (#1A73C6) and a light gray background color (#F6F6F6) as 4.5:1, which is outlined in a green box and labeled as the minimum ratio for WCAG AA compliance. The tool shows 'Pass' for large text compliance with both WCAG AA and AAA standards, but 'Fail' for normal text under WCAG AAA. Sample text "The five boxing wizards jump quickly." is used to illustrate the contrast for both normal and large text sizes.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q04ddbk5fbinly23ti8q.png)


<br>

## Back to the WCAG AA failed test … 
How much time is too much time investigating and troubleshooting a problem when you have no idea why it exists in the first place? After the failed attempts mentioned above, I scoured the documentation, reached out to Playwright’s Discord community, and opened a bug issue on Playwright’s GitHub repo. It had to be a bug, there was no other explanation I could find.

The response was swift: 

> This sounds like a bug which should be reported at @axe-core/playwright or axe-core since Playwright does not do anything related to axe.

Ahhh ... if it’s indeed a bug, as I had assumed, it must be from the package that actually checks for violations. Why did I not think about that?!

Another round of electronically leafing through another documentation. So far, no answers. But will soon find a way to submit a bug report. Stay tune! 😎
<br>
## Update
I did reach out to the axe-core team and they were super helpful. They explained that `AxeBuilder analyze()` runs as soon as the DOM is ready. Since my code has CSS fade-in transitions, they get in the way of the Playwright test. Adding a 2-second timeout solved the problem.

---

## Coming Up
In the mean time, I have another a11y bug to deal with. Found by **Google’s Lighthouse** extension:


![The image shows a Lighthouse web accessibility evaluation report for 'https://corinamurg.netlify.app/'. The report gives the following results: Performance 99, Accessibility 95, Best Practices 100, SEO 90.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a7siyztd35t4gi64y97q.png)


**Can you guess my next challenge?**

Write a test in Playwright that will check for the following WCAG guideline:

> For each user interface component that includes a visible text label, the accessible name MUST match (or include) the visible text in the label.


<br>

## Resources

[WebAIM Contrast Checker](https://webaim.org/resources/contrastchecker/)

[Debbie O’Brien’s Playwright Series on dev.to](https://dev.to/debs_obrien/series/25202)

[Playwright Documentation](https://playwright.dev/docs/intro)
<br>

**Image credit**: Photo by [Alex Kondratiev](https://unsplash.com/@alexkondratiev)

Description: The image shows a hand pouring a blue liquid from a glass test tube into a flask with red liquid, and another hand pouring a green liquid into the flask. The background is plain white.