[Testing 20] test(e2e): accessibility scans + nightly run (stacked on #220)
From the PR description
Stacked on #220 - merge that first; this diff includes its commits until then. Once #220 lands, this PR shrinks to just the one commit described below (300d91a).
Three additions that make the e2e suite from #220 catch more problems with no extra work on your part:
1. Accessibility scans (e2e/accessibility.spec.ts)
Every run now scans the login page, the assistant, the projects list, and the tabular reviews list with axe-core, the standard tool for automated accessibility checks. The policy is deliberately gentle to start: only critical problems (things that outright break the page for screen-reader users) fail the check; serious problems are printed in the test output so the backlog stays visible without blocking merges. Once that backlog is cleared, one line in the spec (BLOCKING_IMPACTS) promotes "serious" to blocking too. These scans need no API key and run on every trigger.
2. Nightly run
The workflow now also runs every night at 03:47 UTC against main. That catches breakage that lands between PRs - a dependency update, a Supabase CLI change - within a day instead of surfacing on someone's unrelated PR.
3. When something fails, you get the evidence
This was mostly in place from #220 and is now documented in docs/e2e-ci.md: failed specs retry twice on CI, record a step-by-step trace of what the browser did, and the report + traces upload as a downloadable artifact (playwright-report) even when the job times out. From a red run's page in the Actions tab: download the artifact, then npx playwright show-report playwright-report locally.
One new dev dependency at the repo root: @axe-core/playwright (installed via npm; lockfile updated by npm itself).
The first CI run of this PR proved the spec earns its keep: it flagged 3 real critical violations - the icon-only send button on the assistant page had no accessible name, and the select-all checkboxes on the projects and tabular-reviews lists had no labels. The second commit on this branch fixes all three (5 lines of aria-label, no markup or styling changes), so screen-reader users get a working send button and checkboxes out of this PR, not just a scanner.
Maintainer action required
- Set the
ANTHROPIC_API_KEYsecret (Settings → Secrets and variables → Actions → New repository secret). Without it everything still passes, but the 4 specs that send a real chat message and check the streamed answer skip themselves. With it, they run on every PR and every nightly. (#236 documents the exact setup.) - Make the checks required (Settings → Branches → rule for
main→ "Require status checks to pass before merging"): adde2e / playwrightplus the existing CI jobs. Until a check is marked required, a red run does not actually block the Merge button. Step-by-step indocs/e2e-ci.md.
🤖 Generated with Claude Code
Our analysis
Harden e2e checks with accessibility scans — read the full analysis →
Think the analysis missed something the PR description covers?
Commits in this PR (4)
| SHA | Subject | Author | Date | |
|---|---|---|---|---|
cf4821ad | test(e2e): axe accessibility spec, failure traces, nightly schedule | Amalanand Muthukumaran | 2026-07-25 | ↗ GitHub |
commit body- e2e/accessibility.spec.ts: axe-core scans (@axe-core/playwright) over /login (pre-auth), /assistant, /projects, and /tabular-reviews. Two-tier policy: critical-impact violations fail the build; serious-impact violations are logged without failing, to be ratcheted into the blocking tier once the backlog is cleared. - .github/workflows/e2e.yml: nightly schedule (03:47 UTC) so drift landing between PRs is caught within a day; LLM-gated specs run on schedule only when the ANTHROPIC_API_KEY secret is configured, else they self-skip. - docs/e2e-ci.md: document the accessibility scans, the nightly run, and how to use the uploaded playwright-report artifact (retries + traces were already configured; no config change needed there). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
b66c0cbc | fix(a11y): accessible names for icon button and unlabeled inputs | Amalanand Muthukumaran | 2026-07-25 | ↗ GitHub |
commit bodyFixes the 3 critical axe violations flagged by e2e/accessibility.spec.ts
on its first CI run (upstream PR #243):
- /assistant, `button-name`: the icon-only send/stop button in ChatInput
(ArrowRight/Square icon, no text) had no accessible name. Add a
state-aware aria-label ("Send message" / "Stop response"), matching the
aria-label convention of its siblings (AddDocButton, workflows button).
- /projects, `label`: the select-all checkbox in the table header row had
no label/aria-label. Add aria-label="Select all projects".
- /tabular-reviews, `label`: same select-all header checkbox pattern.
Add aria-label="Select all reviews".
No markup or styling changes - aria-label attributes only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
4669cc01 | fix(e2e): keep the HTML report when the CI reporter is set to "github" | Amal | 2026-08-05 | ↗ GitHub |
commit bodyWHY THIS MATTERS
docs/e2e-ci.md ("Failure artifacts") tells a maintainer debugging a red
nightly run to download the `playwright-report` artifact and open it with
`npx playwright show-report playwright-report`. But the config set
`reporter: "github"` on CI, and naming ANY reporter REPLACES Playwright's
default `html` reporter rather than adding to it - so `playwright-report/`
was never written, the artifact upload had nothing to ship (it only
survived because `if-no-files-found: ignore` hid the gap), and the
documented debugging flow failed at exactly the moment it was needed:
when a nightly run is red and someone needs the per-spec traces.
WHAT IS A PLAYWRIGHT REPORTER (AND WHY "REPLACES", NOT "ADDS")
A reporter is a plugin that consumes test events and renders them
somewhere: `list` prints one line per test to the terminal, `github`
emits workflow annotations (`::error file=...`) that GitHub renders
inline on the PR diff, and `html` writes a browsable report with
screenshots and traces to `playwright-report/`. When no `reporter` is
configured, Playwright uses `html` on CI by default - but the `reporter`
option is a full override, not a merge. Setting it to a single reporter
silently drops the default. To get several outputs you must list them
all explicitly as an array of tuples:
reporter: process.env.CI
? [["github"], ["html", { open: "never" }]]
: "list",
Each entry is `[name]` or `[name, options]`.
HOW THE FIX WORKS
CI now runs BOTH reporters: `github` keeps the inline PR annotations,
and `html` regenerates `playwright-report/` so the workflow's existing
`actions/upload-artifact` step (which already lists `playwright-report/`
in its `path`) has real content to upload. The `open: "never"` option
matters on CI: the html reporter's default (`on-failure`) tries to
launch a local browser to display the report after a red run, which is
useless on a headless runner. The workflow runs `npx playwright test`
from the repo root, so the reporter's default output folder
(`playwright-report/` relative to the config) lands exactly where the
artifact step looks. Local runs keep the terse `list` reporter.
Verified: `npx tsc --noEmit` passes and `CI=1 npx playwright test
--list` loads the config and enumerates all 31 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
3294e914 | docs(e2e): correct stale spec counts in the workflow's comments | Amal | 2026-08-05 | ↗ GitHub |
commit bodyWHY THIS MATTERS
Two comments in .github/workflows/e2e.yml still said a keyless run
"passes the other 23 specs". That was true of an earlier revision of
this branch, but the suite has since grown (4 accessibility specs were
added): it now has 31 specs, of which 4 are LLM-gated, so a keyless run
passes 27. Stale numbers in CI comments are worse than no numbers: a
maintainer eyeballing a green keyless run against the comment ("23
passed? but I see 27...") is left wondering whether extra specs
accidentally ran with a leaked key, or whether the skip mechanism is
broken - the comment turns a healthy run into a false alarm. The
companion doc (docs/e2e-ci.md) already states 31/27; the workflow now
agrees with it.
WHAT IS COMMENT DRIFT
Comments are not checked by any compiler or test, so they rot silently
when the code they describe changes - here, the spec count changed in a
later commit on the same branch and nothing forced the comment to keep
up. The practical defenses are (a) keeping numbers in as few places as
possible and pointing everywhere else at that one place, and (b)
sweeping comments whenever the quantity they cite changes. This commit
applies (b); both comments also point at docs/e2e-ci.md, which remains
the authoritative source for expected pass/skip totals.
HOW THE FIX WORKS
Pure comment edits, no behavioral change:
- the workflow header now reads "of the 31 specs, the 4 LLM-dependent
ones ... skip themselves ... so a keyless run passes the other 27
specs";
- the ANTHROPIC_API_KEY env comment now reads "still green on the other
27 of the 31 specs".
Verified the file still YAML-parses (python3 -c "import yaml;
yaml.safe_load(...)") and that `CI=1 npx playwright test --list`
reports "Total: 31 tests", matching the corrected figures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-243.md from
inside the repo you want the changes in.