What is Automated Accessibility Testing?

What is automated accessibility testing?

Automated accessibility testing uses software, such as a rules engine, a browser audit or a linter, to scan a page or component for accessibility failures that a program can detect: missing alt text, form fields without labels, low colour contrast, invalid ARIA. It is fast and repeatable, which makes it ideal for catching regressions, but it covers only part of WCAG. A human still has to judge the rest.

How it works

A tool loads the rendered DOM, runs a set of rules against it and reports each element that fails, usually with the rule, the WCAG criterion and a suggested fix. Common options are:

• Rules engines such as axe-core, run inside unit, component or end-to-end tests so a failing rule fails the build.
• Browser audits such as Lighthouse, which turns a set of automated audits into a 0 to 100 accessibility score. Lighthouse weights those audits using axe user impact assessments, and each audit is pass or fail, so one unlabelled button fails the whole button-name audit.
• Linters in the editor or CI that flag problems in markup before it ships.

What it cannot do

The W3C's evaluation guidance says that no tool alone can determine whether a site meets accessibility standards, and that knowledgeable human evaluation is required. axe-core's own README says it finds on average 57% of WCAG issues automatically, and reports uncertain elements as "incomplete" for manual review. That is the vendor's figure for its own rules, so treat it as a rough indication, not a guarantee. A tool can see that an image has alt text, but not whether the text is meaningful. It cannot tell whether the tab order makes sense, whether an error message is understandable, or what a screen reader user actually experiences. Lighthouse itself lists manual checks, such as logical tab order and focus handling, that do not enter the score.

An example in a pipeline

A team adds axe-core to its end-to-end tests for the sign-up and checkout pages. On each pull request the build fails if a new rule violation appears, such as an input without a label. Before each release, a tester tabs through those pages and uses a screen reader. The automated run catches regressions on every commit, and the manual pass finds what rules cannot, see manual tests for accessibility.

Common mistakes

• Treating a 100 Lighthouse accessibility score as proof of accessibility or WCAG conformance.
• Scanning only the home page, not logged-in states, dialogs, error states and checkout.
• Never reviewing "incomplete" or "needs review" results.
• Fixing the tool's warning without checking the real experience, for example adding a meaningless alt text to satisfy a rule.
• Running it once before launch instead of on every change.

Where it fits with testing and monitoring

Automated testing is one layer. It sits beside keyboard checks, screen reader testing and the formal methods described in accessibility conformance testing. Even a thorough audit is not a conformance report: see accessibility conformance reports for what that document is. LoadFocus page speed monitoring runs scheduled Lighthouse audits and records the accessibility category with Performance, SEO and Best Practices, so you can see the score as a trend. It is a monitor for regressions, not an accessibility audit and not a conformance report. For a longer discussion, see performance testing and accessibility metrics and a roundup of accessibility testing tools.

FAQ: Automated accessibility testing

What percentage of accessibility issues can automated tools find?

It varies by tool and by site. axe-core's README says on average 57% of WCAG issues, which is Deque's own estimate for its rules. Either way, a share needs manual testing.

Is Lighthouse an accessibility audit?

No. It runs a set of automated audits and gives a weighted score. A score of 100 means those audits passed, not that the page is accessible.

Can automated testing make my site WCAG compliant?

No. It helps you find and prevent many failures, but conformance needs manual evaluation against the success criteria.

When should I run automated accessibility tests?

On every pull request or build for key pages and components, and on a schedule for live pages, so regressions are caught early.

What should I test by hand?

Keyboard operation and focus order, screen reader behaviour, zoom and reflow, meaningful alt text, error messages and any custom widget.