Pa11y finds accessibility issues, but its exit codes make them operational

An accessibility report that nobody can gate, filter or reproduce is a to-do list wearing a lab coat. pa11y makes the report part of a command-line or Node.js workflow: give it a page, select a test runner, and receive concrete findings plus an exit status that a build system can understand.

Pa11y can test a URL or a local HTML file. Its default runner is HTML_CodeSniffer, while axe is also supported. Both runners can be enabled in one invocation:

pa11y https://example.com --runner axe --runner htmlcs

The important boundary is in the word "automated." Pa11y reports what the configured runners found within the configured page scope. The axe-specific --level-cap-when-needs-review option exists precisely because some findings still require manual review. An empty report is therefore not, by itself, a declaration of accessibility conformance.

The useful part is the testing contract

The smallest command is deliberately unceremonious:

pa11y https://example.com

From there, Pa11y applies a runner and reports issues. HTML_CodeSniffer is the documented default. The --standard option selects WCAG2A, WCAG2AA or WCAG2AAA, with WCAG2AA as the default, but that option applies only to the htmlcs runner. It does not configure axe.

The JavaScript interface exposes the same basic operation as a Promise:

const pa11y = require('pa11y');

pa11y('https://example.com').then((results) => {
    // Use the results
});

The resolved result contains pageUrl, documentTitle and an issues array. Each documented issue includes enough information to support both triage and later processing:

That combination matters. A rule identifier without context sends developers hunting through markup like archaeologists with a CSS selector. A selector without the message merely identifies where the confusion lives.

Configure detection separately from build policy

Pa11y has several controls that look related but answer different questions. Keeping them separate produces tests that are easier to explain when they fail at an inconvenient hour.

--include-warnings and --include-notices control which lower-level findings appear in the report. --level controls which issue levels produce exit code 2. Its documented settings are:

• error: fail on errors only
• warning: fail on errors or warnings
• notice: fail on errors, warnings or notices
• none: always return exit code 0 for accessibility findings

The command also supports --ignore for named issue types or codes and --threshold for allowing a specified quantity before the accessibility gate fails. These are policy controls, not corrections. An ignored issue remains in the page; it has merely persuaded the build not to discuss it.

Pa11y's three exit codes make that distinction explicit:

• 0: the run completed without findings that violate the configured gate
• 1: the run failed because of a technical fault
• 2: the run completed, but the page contained findings that violate the configured gate

This is the operational heart of Pa11y. Exit code 1 says the test did not successfully answer the question. Exit code 2 says it answered, and the answer was no.

Scope is part of the result

--root-element limits testing to one CSS-selected part of the page. --hide-elements removes selected elements from testing and accepts comma-separated selectors. These are useful for deliberately bounded checks, but their presence changes what the result means.

A report scoped to a component says something about that component, not the whole page. Likewise, hiding an element is not remediation. A navigation bar can achieve a flawless automated result by being excluded from the exam, an achievement normally reserved for paperwork.

Timing is configurable through --wait, which delays the start of testing, and --timeout, which limits the run. --screen-capture can save an image of the tested page. Together, these options provide useful evidence when the page state at test time matters more than a later attempt to reproduce it.

Pa11y also supports --add-rule for adding WCAG 2.1 rules to an HTML_CodeSniffer run. Like --standard, this is documented as an htmlcs-only control. Runner selection should therefore be recorded alongside the report rather than treated as an incidental command-line detail.

Configuration that survives more than one command

The CLI automatically looks for pa11y.json in the current directory. A different JSON or JavaScript configuration file can be supplied with --config:

pa11y https://example.com --config ./path/to/config.json

When the same setting appears in both the configuration file and the command line, the command-line value wins. This is sensible for deliberate overrides and wonderfully confusing when a copied CI argument quietly defeats the configuration everyone is reading.

For project use, Pa11y can be installed as a development dependency:

npm install pa11y --save-dev

For direct command-line use, the documented global installation is:

npm install -g pa11y

Pa11y 9 requires Node.js 20, 22 or 24. The README directs users of older Node.js versions to Pa11y 8 or below. The npm badge provides the current published version without freezing a soon-stale release number into the documentation. The --environment option prints details about the environment Pa11y will use, which is the useful first check when two machines behave differently.

Reports for people, pipelines and peculiar local requirements

The documented built-in reporters are cli, csv, html, json and tsv. The CLI help excerpt names only cli, csv and json, while the fuller reporter section additionally documents html and tsv. That documentation mismatch is worth noticing before somebody concludes that the shorter help line is the complete inventory.

CSV output can be redirected to a file:

pa11y https://example.com > report.csv --reporter csv

Custom reporters must be CommonJS modules. Given --reporter rainbows, Pa11y first attempts to resolve an installed package named pa11y-reporter-rainbows, then a module at <cwd>/rainbows. Resolution follows Node.js require behavior.

A custom reporter must export a supports string containing the compatible Pa11y semver range. It should also export begin, error, debug, info and results methods, each returning a string or a Promise resolving to one. That compatibility declaration is small but important: formatting a report should not become an accidental wager on an incompatible result shape.

Where pa11y goes sideways

The build fails even though Pa11y completed

Symptom: CI labels the command as failed, but Pa11y printed a normal accessibility report rather than a technical error.

Cause: Exit code 2 means the test ran successfully and found issues at or above the configured --level. It is different from exit code 1, which indicates a technical fault.

What to do: Preserve the numeric exit code in CI logs. Treat 1 as an execution problem and 2 as an accessibility-policy failure.

The report is clean because it omitted the inconvenient levels

Symptom: A run reports no blocking problems even though warnings or notices are expected.

Cause: Reporting and gating are separate. Warnings and notices have explicit inclusion flags, while the default exit policy reacts only to errors.

What to do: Add --include-warnings and --include-notices when those findings belong in the artifact, then set --level independently. A green light certifies the configured policy, not every possible concern. Build lights are literal-minded little bureaucrats.

The threshold boundary does not mean what its label suggests

Symptom: A build configured with --threshold 10 returns exit code 2 when the count reaches exactly ten.

Cause: The current CLI description says the threshold permits the stated number, but the README's worked example says nine errors return 0 and ten or more return 2. Those two descriptions disagree at the boundary.

What to do: Test the exact boundary with the Pa11y version used in CI and record the expected exit code. Do not build release policy around an inferred interpretation of the word "permit."

Changing --standard does nothing to an axe-only run

Symptom: The command accepts a different standard, but an axe-only test does not behave as though that setting changed its runner.

Cause: Pa11y documents --standard as an HTML_CodeSniffer-only option. The same limitation applies to --add-rule.

What to do: Apply those flags only to htmlcs runs. Record the selected runner with the result so that WCAG2AA is not mistakenly presented as an axe configuration.

An important part of the page vanishes from the findings

Symptom: A region known to be present produces no issues or selectors in the report.

Cause: --root-element can confine the entire test to one selector, while --hide-elements removes matching elements from testing. Either may also come from the configuration file rather than the visible command.

What to do: Inspect both command-line arguments and configuration. Remove or narrow the selectors, then rerun the same page before treating the earlier report as page-wide evidence.

The configuration file appears to be ignored

Symptom: Editing pa11y.json changes nothing, or CI behaves differently from an apparently identical local run.

Cause: Command-line values override values from the configuration file. Pa11y also searches for pa11y.json in the current directory, so invocation location affects automatic discovery.

What to do: Examine the complete command and working directory. Remove stale overrides or pass the intended file explicitly with --config.

A local HTML file cannot be tested as supplied

Symptom: A URL works, but a local file path does not.

Cause: Pa11y's documentation requires absolute paths for local HTML files.

What to do: Resolve the file to an absolute path before invoking Pa11y. Do not assume a path relative to the shell's current directory will be accepted merely because the file exists there.

A custom reporter cannot be loaded

Symptom: --reporter works with a built-in format but errors with a custom name.

Cause: Pa11y searches first for pa11y-reporter-<name> and then for <cwd>/<name>, using CommonJS require resolution. The reporter must also export a supports semver range compatible with the Pa11y version.

What to do: Verify the resolved package or local module, its working-directory location, its CommonJS exports and its supports value. Reporter discovery is Node module resolution wearing a name badge.

A Pa11y 9 environment starts with the wrong Node.js line

Symptom: A Pa11y 9 installation is being debugged on a Node.js version outside the documented set.

Cause: Pa11y 9 requires Node.js 20, 22 or 24. Older Node.js versions are directed to Pa11y 8 or below.

What to do: Run pa11y --environment, compare the reported runtime with the supported versions, and align the runtime before investigating page-level findings.

References

• Pa11y project and current README
• Pa11y source and LGPL-3.0 licence record
• Node.js CommonJS module documentation

---