[Testing 21] test depth: mutation testing + SSE load harness (on-demand)
From the PR description
Two tools that go a level deeper than the regular test suite. Neither one gates merges - both are run-on-demand, so they cost you nothing day to day.
1. Mutation testing for the security-critical code. Line coverage says a test ran a line; it doesn't say a test would notice if that line's behavior changed. Mutation testing checks exactly that: Stryker plants hundreds of small deliberate bugs (flip a comparison, delete an early return, weaken a regex) and re-runs the suite for each one. Any bug the suite doesn't catch is listed as "survived" - a concrete missing assertion. It's scoped to the four places where a hollow test would be dangerous: sharing/access checks (access.ts), signed download tokens (downloadTokens.ts), secret redaction (safeError.ts), and citation extraction (chat/citations.ts). Current score: ~74-76% of planted bugs caught; the run fails only if that drops below 69, so it flags real regressions, not noise. Run it with cd backend && npm run test:mutation (~3 min) or from the Actions tab; a monthly scheduled run catches slow drift. The HTML report shows each surviving mutant inline in the code - the surviving safeError redaction-regex mutants are ready-made test cases for whoever wants a small follow-up PR.
2. A load test for chat streaming. The SSE stream behind POST /chat is the hot path and has caused incidents before (streams timing out during long tool calls). loadtest/sse-stream.js (k6) ramps up to N simultaneous chat streams and verifies each one starts quickly and actually runs to completion ([DONE] arrives) - with deliberately lenient, documented thresholds: a red run means "streams are hanging," not "we missed an SLO we never set." Run it locally per docs/test-depth.md, or via the manual SSE load test workflow, which boots nothing itself - you point it at an already-running staging stack (the one from #210 is the intended target) and store a test user's token in the LOADTEST_AUTH_TOKEN secret. Never point it at production: it creates real chats and burns real tokens.
Why not merge gates? Mutation testing multiplies test runtime; the load test needs a live stack with real keys. Both are too slow/stateful to put in front of every PR, and a flaky required check is worse than none. If the project gains contributors and a permanent staging stack, the natural upgrade path is in docs/test-depth.md.
Verified: two full local Stryker runs completed green (2m51s / 2m25s); k6 script validated via k6 inspect + a dry run in docker; both workflows YAML-parse. Lockfile changes are npm-generated (npm install, never hand-edited).
🤖 Generated with Claude Code
Our analysis
Add on-demand security mutation and chat stream load tests — read the full analysis →
Think the analysis missed something the PR description covers?
Commits in this PR (2)
| SHA | Subject | Author | Date | |
|---|---|---|---|---|
f80722a7 | test: mutation testing (security libs) + SSE load harness, on-demand | Amalanand Muthukumaran | 2026-07-25 | ↗ GitHub |
commit bodyTwo on-demand depth tools, neither a merge gate: - Stryker mutation testing scoped to the security-critical backend libs (access.ts, downloadTokens.ts, safeError.ts, chat/citations.ts). Measured 74.0-76.4% mutation score across runs; thresholds.break=69 fails only on real regressions. `npm run test:mutation` locally (~3 min), .github/workflows/mutation.yml on demand + monthly cron, HTML report uploaded as artifact. - k6 load harness for the SSE chat stream (loadtest/sse-stream.js): ramps to N concurrent POST /chat streams, checks TTFB and that every stream delivers events through to the [DONE] sentinel - the past incident class (streams timing out on long tool calls). Lenient, documented thresholds. .github/workflows/loadtest.yml is workflow_dispatch-only and boots nothing: point it at a staging stack (PR #210). docs/test-depth.md explains how to read the mutation report, how to run the k6 harness against the local stack, and why neither gates merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
64ddcf4c | docs: replace dead PR #210 staging-stack references with self-contained target guidance | Amal | 2026-08-05 | ↗ GitHub |
commit bodyWHY THIS MATTERS
The load-test harness (workflow, doc, and k6 script) all told operators
to point it at "the staging stack from PR #210". PR #210 was CLOSED
unmerged - it was rejected for leaking admin credentials - so the one
piece of infrastructure the harness claims to require does not exist.
A newcomer following the docs would hit a dead end: the reference reads
as "there is a blessed staging stack somewhere", when in reality there
is no such stack and never will be from that PR. Worse, linking to a
credential-leaking PR as recommended reading is itself a small security
smell. Docs that point at dead artifacts erode trust in all the other
docs around them.
WHAT IS A SELF-CONTAINED REFERENCE
Documentation can either point at an artifact ("use the stack from PR
#210") or describe a contract ("use any stack that serves X behind Y").
Pointers are fragile: PRs get closed, branches get deleted, staging
environments get torn down, and the doc silently rots. A contract-style
reference survives all of that because it tells the operator what the
target must PROVIDE rather than where one specific instance lived. For
an on-demand load harness - which by design boots nothing itself - the
contract is the only stable thing to document.
HOW THE FIX WORKS
Every "#210" reference in the three harness files is replaced with the
actual contract the target stack must satisfy:
- a deployed, NON-production backend the operator owns,
- serving the backend API's streaming endpoint:
POST {BASE_URL}/chat
Authorization: Bearer <supabase access token>
- with real LLM provider keys configured (each iteration performs a
real chat completion).
Concretely:
- .github/workflows/loadtest.yml: the header comment and the
`target_url` input description now describe that contract, e.g.
description: "Backend base URL of a non-production stack you
deployed (serving the backend API with auth), ..."
- docs/test-depth.md: the "Running from GitHub Actions" section
describes the same contract instead of naming PR #210.
- loadtest/sse-stream.js: the header comment does likewise.
No behavior changes - the workflow inputs, k6 scenario, thresholds, and
safety warnings ("never point at production") are untouched. Verified by
grepping the three files for "#210" (no matches), YAML-parsing the
workflow, and `node --check` on the k6 script.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-254.md from
inside the repo you want the changes in.