Accessibility testing methods: the layered pass that finds real barriers

Why a green scanner report lies#

You ran axe. It reported zero violations. Then a real user, or an auditor, found barriers anyway. That gap is the pain most searchers arrive with, and it is not a bug in the tool. A scanner is doing exactly what it can do, which is less than a clean report implies.

Because the failure is structural, no better scanner fixes it. So the honest move is to see the gap first, before you trust any report. The demo below makes it observable.

Watch a scanner pass a page that is actually broken#

Here is a page built to break in five ways. Every element passes an automated scan. Every element is a real barrier for someone. Step through the elements, then flip the lens between what the scanner sees and what a keyboard or screen-reader user hits.

<img src="summer-sale.png" alt="image">
(the words "SAVE 20%, code SPRING" live in the pixels)

Notice what happened on the pay-now control. It has a valid role and an accessible name, so the scanner passes it. Still, no keyboard user can reach or press a plain div, so checkout dead-ends. The tool checked that a name and role exist. It could never check that the control actually works.

What every ranking page gets wrong: quoting the number as folklore#

Search this topic and every result repeats the same figure. Automated tools catch 30 to 40 percent of issues. Almost none say where the number comes from, and none map it to what a machine can actually decide. So the figure floats as folklore, and the reader is left to trust it.

The coverage number is not a statistic, it is a boundary you can compute#

This guide makes one core claim. The coverage figure is a property you can derive, not a statistic to memorize. Moreover, it falls out of one question asked of every success criterion. Can a machine settle this by checking presence or computing a value, or does it need a human to judge meaning or operability?

The decidability model: three classes of WCAG success criterion#

Sort every WCAG 2.2 A and AA success criterion into three classes. Fully decidable means a machine returns a real verdict, such as a contrast ratio or a present lang attribute. Partially decidable means a machine can flag a candidate but a person confirms it, such as whether an alt attribute that exists is actually meaningful. Human-only means the machine is blind, such as whether focus order is logical. Step through the three class tabs below and read what shifts.

Two of the sampled criteria are worth a closer look. Focus Not Obscured (2.4.11) and Focus Appearance (2.4.13) are partially decidable, because a tool can measure a focus indicator yet cannot confirm it is perceivable in context. For a worked micro-example of getting one focus indicator right, see how focus versus focus-visible actually behaves in CSS. That single selector is one row; this guide is the whole method.

Why the boundary sits exactly at presence-check versus meaning and operability#

The boundary is not arbitrary. A machine decides two things well. It decides existence, such as whether an alt attribute or a label is present. It decides computation, such as a contrast ratio or a target size in pixels. Both are formal properties of the markup and the styles.

Meaning and operability are different in kind. For example, whether alt text describes the image is a semantic judgment. Meanwhile, whether a custom widget is operable is a behavioral one. In contrast, whether the reading order carries sense is a judgment about intent. No static check settles any of these, because the answer lives in a human reading or a human interaction, not in the source. Therefore the boundary sits precisely at presence-check versus judgment, and the coverage figure is just the size of the decidable side.

Working the real numbers: WebAIM Million 2026, GDS, and Deque#

Now put real data on the boundary. The WebAIM Million 2026 report scanned the top one million home pages and found 56.1 detectable errors per page. Six machine-decidable classes account for 96 percent of all detected errors, so a scanner genuinely earns its place as the cheap first pass.

Here is the twist the scan cannot show you. Apply the independent UK Government Digital Service tool audit. On a pass or fail basis, the best automated tool detected only about 30 percent of accessibility barrier types. So a page that scans to zero errors has cleared roughly a third of the criteria surface. The rest is unexamined.

The close: full conformance certainly lower than 4.1 percent#

WebAIM closes the loop with a hard number. Because only auto-detectable failures were counted, the true full-conformance rate was, in their words, certainly lower than 4.1 percent of pages. Read that carefully. Even most of the small share of pages that scanned clean still fail manual criteria. Consequently, a green report tells you almost nothing about whether the page actually works for a disabled user.

The layered accessibility testing methods: what to run, when, and in what order#

These layered accessibility testing methods run in a fixed order, cheapest first. Each layer catches what the one before it cannot. The automated gate runs on every commit, the manual passes run before release, and real users close the loop. The funnel below is the spine of the method.

Layer 1: automated scanning in CI (the cheap first pass)#

In practice, Layer 1 is the automated pass, wired into CI so it runs on every pull request. Tools like axe-core, Lighthouse, WAVE, and pa11y reliably catch the machine-decidable classes: contrast, alt presence, label association, ARIA validity, and page language. Gate the build on serious and critical findings, and warn on the rest so a moderate issue never buries the signal. Here are three real configurations you can lift.

name: accessibility
on: [pull_request]
jobs:
  a11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build
      - run: npx serve -l 4321 dist &
      - run: npx wait-on http://localhost:4321
      # pa11y-ci reads .pa11yci and fails the job on any error-level issue.
      - run: npx pa11y-ci --config .pa11yci

{
  "defaults": {
    "runners": ["axe"],
    "standard": "WCAG2AA",
    "timeout": 30000,
    "threshold": 0
  },
  "urls": [
    "http://localhost:4321/",
    "http://localhost:4321/pricing/",
    "http://localhost:4321/contact/"
  ]
}

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

// Block the build only on serious or critical impact. Warn on the rest, so a
// moderate finding never drowns the signal (an anti alert-fatigue baseline).
test('home page has no serious or critical violations', async ({ page }) => {
  await page.goto('/');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
    .analyze();

  for (const v of results.violations) {
    if (v.impact !== 'serious' && v.impact !== 'critical') {
      console.warn(`warn ${v.impact}: ${v.id} (${v.nodes.length} nodes)`);
    }
  }
  const blocking = results.violations.filter(
    (v) => v.impact === 'serious' || v.impact === 'critical',
  );
  expect(blocking, JSON.stringify(blocking, null, 2)).toHaveLength(0);
});

The severity gate matters more than the tool choice. Because a raw scan surfaces every impact level, an unfiltered gate trains the team to ignore red builds. Instead, block on serious and critical, warn on moderate, and hold a baseline so known debt does not re-fail the build every run.

Layer 2: manual keyboard-only testing checklist#

Layer 2 needs no tool, only your keyboard. Unplug the mouse and drive the whole page. This one pass catches a huge share of operability barriers that no scanner reports.

Layer 3: screen reader testing (NVDA, VoiceOver, JAWS)#

Layer 3 is the screen-reader pass, and the choice of reader matters. Ground it in the WebAIM Screen Reader User Survey #10, which found JAWS is the primary desktop reader for 40.5 percent of users, with NVDA and VoiceOver making up most of the rest. Pair each reader with a browser its users actually run.

For each widget, know what you are listening for. When a modal opens, focus should move into it and the reader should announce its name. Next, when a field is invalid, the error should be tied to the input and read on focus. Finally, when a custom toggle flips, its new state should be announced, not silent.

Layer 4: zoom, reflow, reduced-motion, and cognitive load#

Layer 4 is the pass most guides skip in a line or two. Here it gets concrete pass or fail procedures. First, zoom the browser to 400 percent at a 1280 pixel width. The page passes only if content reflows to one column with no two-axis scrolling. Second, apply a 200 percent text-spacing bookmarklet. The page passes if no text is clipped or overlapped. Third, set the operating system to reduce motion. Any non-essential animation should stop or shorten. Finally, read the flow for cognitive load. A form that re-asks for data you already gave, or a login that forces a memory puzzle, is a real barrier under WCAG 2.2.

Layer 5: the WCAG 2.2 checklist, including the seven criteria competitors omit#

Layer 5 walks the full WCAG 2.2 checklist. This is the differentiator, because most guides still cite WCAG 2.1 and miss the seven criteria added in 2.2. Several of them are human judgments, so they belong on your manual checklist, not your scanner.

Layer 6: where real assistive-tech users are irreplaceable#

Finally, Layer 6 closes the funnel with real assistive-technology users. Still, automated checks and even your own manual passes are proxies. A daily screen-reader user hits barriers you will not predict, because they know the shortcuts, the habits, and the failure patterns of their tools. For the human-only criteria, such as consistent help and redundant entry, lived operability is the only true test. So budget for it on any flow that matters.

Triaging what you find: severity before you fix#

The accessibility testing methods above will surface more findings than you can fix at once. So triage by barrier severity, not by scanner impact score. A barrier that blocks task completion outranks one that merely annoys, and a barrier on a critical journey outranks one on a dead-end page.

How teams triage the bugs automated scans surface#

Still, a severity matrix only sets the order. A team must argue the edge cases. For instance, a low-severity barrier on a revenue path can outrank a high-severity one almost nobody hits. For the mechanics of that decision, and why the room matters, read how teams triage the bugs automated scans surface. The same forum logic applies to accessibility findings.

When NOT to lean on this, and the honest thesis#

These accessibility testing methods are not a checklist to game. So here is where the approach breaks, stated plainly.

Why manual QA still catches what automation misses#

This is the same argument shape you see across testing, not just accessibility. Automation has structural blind spots, and layered human review closes them. For the general version of the case, applied to test suites, read why manual QA still catches what automation misses. Accessibility is one sharp instance of that broader truth.

A broader look at layered testing techniques#

Layering is a general testing discipline, not an accessibility trick. Safety-critical fields formalize it into named stages for exactly the same reason. For the wider taxonomy of how testing techniques stack up, read a broader look at layered testing techniques. The pattern carries across domains: cheap automated checks first, expensive human judgment where it counts.

Here is the thesis to carry away. The coverage number is not a statistic to memorize, it is a boundary you can compute from what a machine can decide. Accessibility is therefore a design and development property you build in, not a test you bolt on at the end. Run the automated gate for what it settles cheaply, then spend your real effort on the judgment layers, because that is where the barriers live.