-
Notifications
You must be signed in to change notification settings - Fork 455
Add quarto call axe, a hidden experimental site accessibility scanner
#14815
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
d63e5db
M0: hidden `quarto dev-call axe` skeleton command
cwickham f320513
M1: scan an already-rendered site end-to-end
cwickham 84f1b6e
Scan the whole document: drop the axe context excludes
cwickham 1dc920e
M2: aggregate, report and the two contracts
cwickham 3bc8d9b
Version the signature scheme, and pin the normalization with tests
cwickham 10e7628
Let Zod's .strict() do the baseline error reporting
cwickham 0f79345
M3: fixture site, unit tests and a browser smoke test
cwickham 64d2612
M4: survive the bundle, and stop importing a browser module to do it
cwickham 44494ec
M4a: select the author's themes, not just the user's OS preference
cwickham 8162c39
Fix: light-only colour-scheme links are not a dark theme
cwickham e7f8563
Split the axe fixture site into one site per concern
cwickham 9f3545c
M4b: per-page mode discovery replaces the runtime theme probe
cwickham 40f59d7
Never discover the scanner's own output or vendored libs as pages
cwickham 84127d4
B1: a rejected CDP send fails its cell closed, not the whole scan
cwickham bee408a
Fail closed when the page redirects away from the served site
cwickham b7719c7
Wait for readiness instead of a fixed 500ms settle per cell
cwickham d7f0df7
Add --exclude: site-relative globs to skip, after --pages
cwickham 5739f5f
Skip redirect stubs at discovery, and harden the scan per review
cwickham ce9e11b
Ship report.md and a generated README; retire the HTML report
cwickham cc31326
report.md: link table rows to occurrences; state detail once
cwickham c63c274
report.md: drop the redundant rule column
cwickham 73847e1
Remove the systemic/localized label; drop the occurrence detail column
cwickham 9edb955
findings.json: emit anchor-relative paths
cwickham 20437ee
Self-ignore the _axe-checks artifact dir
cwickham 8c04019
scan: collision-proof cell names, percent-encoded page URLs
cwickham d00b227
discover: two-mode marker matches the toggle definition, not its name
cwickham 32ce785
axe action: exit through exitWithCleanup
cwickham 01224ed
Add --fail-on: exit 1 on new findings by impact
cwickham e144a57
Await the site server's teardown before exiting
cwickham e1a2475
Close the v1 review's test gaps: flag parsing and exit codes
cwickham d7f36ec
Write llm-docs/axe-scan-architecture.md; repoint dead references
cwickham ef13b94
dev-docs/axe-scan.md: experimental how-to
cwickham 86b9312
changelog: quarto dev-call axe (PR number pending)
cwickham f0717fd
report.md: backtick code spans instead of <code> raw HTML
cwickham 191a63d
Move axe from dev-call to quarto call
cwickham f8e82fb
Test the browser-launch failure path
cwickham c1a2167
Don't dangle an empty colon when Chrome discovery finds nothing
cwickham f0261c7
Docs: cross-page grouping is a heuristic, not a law
cwickham 257376c
Docs: post-render usage and a sticky PR comment
cwickham d07b536
Help text: say EXPERIMENTAL, not Prototype
cwickham c2b8cbf
changelog: fill in the axe PR number (#14815)
cwickham 20e870c
Docs: how to read and render report.md
cwickham 379c6ad
Default narrow viewport: 390x844 -> 320x568 (WCAG's reflow width)
cwickham d4a2de0
llm-doc: separate CDP transport from scan commands
cwickham 5319ff6
Docs: say plainly that posting the report is workflow config
cwickham 7d8768e
--max-pages counts scannable pages, not raw HTML files
cwickham ee7f633
Usage errors exit 3, not 1 or 2
cwickham 06f0db9
Reject duplicate baseline signatures instead of merging them
cwickham 5141d88
Skip the scan on an incremental project render
cwickham 82da67d
Bound the timeout recovery, and stop the cell it abandoned
cwickham 08742c1
Make the changelog's docs pointer a link; drop it from --help
cwickham b9c6852
axe: don't tell the reader an unseen baseline entry is resolved
cderv 1741f3b
axe: reject a --report destination the scan owns
cderv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| --- | ||
| paths: | ||
| - "src/command/call/axe/**" | ||
| - "tests/unit/axe-*.test.ts" | ||
| - "tests/smoke/axe/**" | ||
| - "tests/docs/axe-scan/**" | ||
| --- | ||
|
|
||
| # Axe Scan Command (`quarto call axe`) | ||
|
|
||
| For the scanner's architecture — pipeline, mode discovery, signatures, | ||
| baseline semantics, exit codes, and the rationale behind each — see | ||
| `llm-docs/axe-scan-architecture.md`. | ||
|
|
||
| For how to use the command (flags, baseline workflow, CI recipe), see | ||
| `dev-docs/axe-scan.md`. | ||
|
|
||
| The signature normalization scheme is pinned by | ||
| `tests/unit/axe-signature.test.ts`: changing `normalizeSelector` or | ||
| `signatureOf` in a way that re-keys signatures requires bumping | ||
| `kSignatureScheme` in `schemas.ts` on purpose. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| --- | ||
| paths: | ||
| - "src/format/html/format-html-info*" | ||
| - "src/format/html/format-html-bootstrap*" | ||
| - "src/command/render/pandoc-html*" | ||
| - "src/resources/formats/html/templates/quarto-html-*body.ejs" | ||
| - "src/core/brand/**" | ||
| - "src/command/dev-call/axe/**" | ||
| --- | ||
|
|
||
| # HTML Dark Mode | ||
|
|
||
| `formatDarkMode(format)` (`src/format/html/format-html-info.ts`) is the single | ||
| predicate for "does this page have a dark mode" — `undefined` means no. | ||
| The `data-mode` link attribute measures compiled CSS darkness; it does not | ||
| declare the author's slot. The dark slot's stable identity is the | ||
| `quarto-color-alternate` class. | ||
|
|
||
| For the full picture — configuration surface (`theme`, `brand`/`_brand.yml`), | ||
| rendered DOM markers, programmatic mode switching, and the known traps | ||
| (light-only brand, light-colored dark slots, `theme: darkly`) — see | ||
| `llm-docs/html-dark-mode-architecture.md`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,201 @@ | ||
| # Scanning a site with `quarto call axe` | ||
|
|
||
| `quarto call axe` scans a rendered Quarto site for accessibility | ||
| violations with axe-core. It drives headless Chrome over every page, at | ||
| desktop and mobile widths, in each colour mode the page ships. It groups | ||
| violations by root cause, compares them against a committed baseline, and | ||
| writes a report you can read, commit review comments from, or gate CI on. | ||
|
|
||
| **Status: experimental.** The command is hidden (it does not show in | ||
| `quarto call` help) and makes no stability promise — flags, artifact shapes, | ||
| and semantics can change between prereleases. This page is contributor-facing | ||
| documentation; it graduates to quarto.org when the command is unhidden (the | ||
| same path the `axe:` render option's docs took). | ||
|
|
||
| Prerequisite: a Chromium the scanner can find. `quarto install | ||
| chrome-headless-shell` is the reliable route; an installed system Chrome or | ||
| Edge also works. | ||
|
|
||
| ## When to reach for it | ||
|
|
||
| The render-time `axe:` option checks the page you are looking at, in your | ||
| browser, while you author. This command checks the *site*: every page, both | ||
| viewports, light and dark, at audit time or in CI. Reach for it when you | ||
| want a site-wide inventory, a regression gate, or output an agent can work | ||
| through. | ||
|
|
||
| ## First scan | ||
|
|
||
| ```sh | ||
| quarto render | ||
| quarto call axe _site | ||
| ``` | ||
|
|
||
| The scan prints its matrix up front (`43 pages (40 light+dark, 3 default) × | ||
| 2 viewports — 166 cells`), then one line per cell, then a findings summary. | ||
| Artifacts land at the project root (the nearest `_quarto.yml` at or above | ||
| the site dir): | ||
|
|
||
| - `_axe-checks/findings.json` — every finding, machine-readable. | ||
| - `_axe-checks/report.md` — the human summary, GitHub-flavored markdown. | ||
| - `_axe-checks/README.md` — generated docs for the artifacts, including the | ||
| baseline how-to. Read this after a scan; it is written for whoever (or | ||
| whatever) has only the artifact directory in front of them. | ||
| - `_axe-checks/cells/` — raw axe output per page × viewport × mode cell. | ||
|
|
||
| `_axe-checks/` ignores itself (`.gitignore` with `*`): it is a disposable | ||
| snapshot, not a thing to commit. | ||
|
|
||
| ### Reading report.md | ||
|
|
||
| The report is GitHub-flavored markdown, made to be read where markdown | ||
| already renders: your editor's preview, GitHub, or the sticky PR comment | ||
| from the CI recipe below. Rendering it is optional. For a standalone HTML | ||
| view: | ||
|
|
||
| ```sh | ||
| quarto render _axe-checks/report.md | ||
| ``` | ||
|
|
||
| then open `_axe-checks/report.html` in your browser. The output lands | ||
| beside the report, inside the self-ignoring artifact directory. | ||
|
|
||
| `quarto preview _axe-checks/report.md` does **not** work from inside a | ||
| project: `_`-prefixed directories are not project inputs, so preview stops | ||
| with `No output created by quarto render report.md`. Use render-then-open, | ||
| or pass `--report` a path inside your site source (see the flags table) to | ||
| render and preview the report as part of the site. | ||
|
cderv marked this conversation as resolved.
|
||
|
|
||
| A finding is one *root cause*, not one element: an alt-less image in a | ||
| shared include shows up as one finding with an instance count, not once per | ||
| page. Grouping keys on axe's element selector (normalized), so this holds | ||
| when the pages describe the element the same way — reliably true for | ||
| Quarto's own chrome and for repeated template output, and occasionally | ||
| wrong for an anonymous element whose surrounding DOM differs page to page | ||
| (axe then picks different selectors, and one cause splits into two | ||
| findings). Findings on many pages usually come from a shared source — a | ||
| template, the theme, Quarto's own chrome — and one fix clears them all. | ||
|
|
||
| ## Flags | ||
|
|
||
| | flag | default | | | ||
| |---|---|---| | ||
| | `--pages <globs>` | all `*.html` | comma-separated site-relative globs | | ||
| | `--exclude <globs>` | — | skip globs, applied after `--pages` | | ||
| | `--max-pages <n>` | ∞ | deterministic cap (sorted, first n scannable pages — redirect stubs don't use up the cap) | | ||
| | `--viewports <WxH,...>` | `1440x900,320x568` | | | ||
| | `--themes <light,dark>` | `light,dark` | filters two-mode pages; one-mode pages always scan once | | ||
| | `--timeout <ms>` | `30000` | per-cell budget | | ||
| | `--settle <ms>` | `50` | extra delay after the page reports ready | | ||
| | `--fail-on <impact>` | off | exit 1 on new findings at/above `minor`/`moderate`/`serious`/`critical` | | ||
| | `--report <path>` | `_axe-checks/report.md` | put the report elsewhere, e.g. inside your site source — can't name the baseline, `findings.json`, `README.md`, `.gitignore`, or anything inside `cells/` | | ||
|
|
||
| The narrow default viewport is 320 CSS px — the width WCAG's reflow | ||
| criterion (SC 1.4.10) names, equivalent to 400% zoom on a 1280 px window — | ||
| so every rule runs against the reflowed mobile layout. | ||
|
|
||
| A subset scan (`--pages`, `--exclude`, `--max-pages`) says so loudly in | ||
| every artifact: its counts describe the subset, not the site. | ||
|
|
||
| ## Running it as a post-render step | ||
|
|
||
| The command works as a project `post-render` script, so a full render scans | ||
| itself: | ||
|
|
||
| ```yaml | ||
| project: | ||
| type: website | ||
| post-render: | ||
| - quarto call axe _site/ | ||
|
cderv marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| The script runs from the project directory after the outputs are written, | ||
| so the relative site dir and the artifact anchor resolve exactly as they do | ||
| on the command line. Exit codes propagate: without `--fail-on`, findings | ||
| never fail the render; with `--fail-on`, a new finding at the threshold — | ||
| or an incomplete scan — fails `quarto render` itself, with the scan's error | ||
| line in the render output. This adds the scan's runtime to each full render. | ||
|
|
||
| Only *full* renders scan. Quarto runs post-render scripts on incremental | ||
| renders and on preview reloads too, and the scan skips both with a note and | ||
| exit 0 — a partly-rebuilt site would mix this render's output with the last | ||
| one's. Run `quarto render` with no file argument to scan. | ||
|
|
||
| ## The baseline workflow | ||
|
|
||
| The first scan of a real site reports findings you will not fix today: | ||
| upstream defects, deferred best-practice items, known false positives. The | ||
| baseline is the committed ledger of those decisions — `_axe-baseline.json` | ||
| at the project root, beside (not inside) `_axe-checks/`. | ||
|
|
||
| 1. Scan, then fix what you can from `report.md`. | ||
| 2. For each finding you are accepting instead: copy it out of | ||
| `findings.json` into the baseline's `findings` array, trim it, and write | ||
| a `note` saying why. The generated `_axe-checks/README.md` documents the | ||
| entry shape and the scoping rules (`pages: []` accepts site-wide; | ||
| a listed `pages` re-alerts anywhere else). | ||
| 3. Commit `_axe-baseline.json`. | ||
|
|
||
| From then on, reports separate **new** findings from **baselined** ones, | ||
| and only new findings can fail CI. A baselined finding that escalates in | ||
| impact, or shows up on a page outside its scope, re-alerts as new. Entries | ||
| a scan no longer sees are reported as stale — prune them by hand, but only | ||
| after a scan of the full matrix (every page, viewport and mode) in which | ||
| every cell completed. Narrowing any axis with `--pages`, `--viewports` or | ||
| `--themes` means the scan can't tell "fixed" from "not scanned", and neither | ||
| can a full scan that lost a cell; the report says which applies to it. | ||
|
|
||
| There is deliberately no `--update-baseline`: every entry exists because | ||
| someone wrote it and said why. | ||
|
|
||
| ## Exit codes and the CI recipe | ||
|
|
||
| | exit | meaning | | ||
| |---|---| | ||
| | `0` | scan complete; no new findings at/above `--fail-on` (when given) | | ||
| | `1` | complete scan, new findings at/above the `--fail-on` threshold | | ||
| | `2` | scan incomplete — a cell timed out or errored, no browser, nothing to scan. Takes precedence over 1: an incomplete scan never reads as a pass | | ||
| | `3` | usage error — a flag value the command can't accept (`--fail-on serius`), or a filter that matches nothing (`--themes dark` on a site with no dark mode). A typo is not a result, so it gets its own code | | ||
|
|
||
| A minimal GitHub Actions gate: | ||
|
|
||
| ```yaml | ||
| - uses: quarto-dev/quarto-actions/setup@v2 | ||
| - run: quarto install chrome-headless-shell --no-prompt | ||
| - run: quarto render | ||
| - run: quarto call axe _site --fail-on serious | ||
|
cderv marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| `report.md` is GitHub-flavored markdown, so a workflow can post it straight | ||
| into the PR — no Quarto-side tooling needed. Use a sticky comment, which | ||
| updates in place on each push rather than adding one comment per run: | ||
|
|
||
| ```yaml | ||
| - if: always() | ||
| uses: marocchino/sticky-pull-request-comment@v2 | ||
| with: | ||
| header: axe | ||
| path: _axe-checks/report.md | ||
| ``` | ||
|
|
||
| (`if: always()` keeps the comment current when `--fail-on` fails the job; | ||
| `header` keys the comment so other sticky comments are untouched. GitHub | ||
| caps comment bodies at 65,536 characters; a whole-site report on a large | ||
| site can exceed it — trim or attach as an artifact instead.) | ||
|
|
||
| ## Reading the output as an agent | ||
|
|
||
| Point the agent at `_axe-checks/` and let it read the generated `README.md` | ||
| first. Finding ids are stable across runs (`image-alt-6e3b76`), so "fix | ||
| `image-alt-6e3b76`" is a well-defined instruction, and each finding's | ||
| `occurrences[]` carries real selectors and HTML excerpts. Fixes belong in | ||
| Quarto *source* (`.qmd`, `_quarto.yml`, `_brand.yml`, theme `.scss`) — never | ||
| in the rendered site directory. | ||
|
cderv marked this conversation as resolved.
|
||
|
|
||
| ## Where the pieces are documented | ||
|
|
||
| - How it works and why: `llm-docs/axe-scan-architecture.md`. | ||
| - What the artifacts mean, baseline entry shape: the generated | ||
| `_axe-checks/README.md` (regenerated every scan, always matches the build | ||
| that wrote it). | ||
| - Render-time single-page checking: the `axe:` HTML format option. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.