ci: make every quality gate actually execute (or remove it)
From the PR description
Summary
Three of the repo's quality gates were mechanisms that never actually ran, discovered while auditing the CI story end to end:
- The backend coverage ratchet was never enforced.
ci.ymlrannpm test --if-present- thetest:coveragescript and its documented floors (backend/vitest.config.mts) have existed since the harness landed, but no CI job ever invoked them. The frontend job has run its ratchet all along. - The
evalsjob green-checked a directory that does not exist. It guardedevals/run.mjs; the eval-harness PRs were rejected upstream, so every PR since has carried a green "Eval harness" check that verified nothing. mutation.ymlandloadtest.ymlhad zero runs ever (both created 2026-08-12; verified via the Actions API). Worse, the mutation scope excluded the two most safety-critical modules added since:privateIp.ts(the SSRF guard) andverifyCitations.ts(the "verified" badge).
This PR makes each gate real or removes it:
ci.ymlbackend job now runsnpm run test:coverage; floors re-measured and raised to the current tree (57/50/58/58, from 52/46/53/54 - coverage rose unenforced, and a ratchet resumes from today, not from stale floors). Theevalsjob is deleted.stryker.config.jsonnow mutates all five security libs. The first measured run with the expanded scope scored 68.52 - below the 69 break floor: the exempt files were under the bar the moment they were included (privateIp.ts: 57.9, 66 surviving mutants). New adversarial tests written from the surviving-mutant report (NAT64 prefix-confusion with public embedded IPv4s, the0xffffhextet boundary, junk passed directly toisPrivateIpv6) bring the score to 70.01, back above the floor.ignoreStaticis enabled for the module-load-timeBlockListtable mutants that mutation switching cannot toggle.mutation.ymlnow also runs onpull_requestwhen the mutated modules, their tests, or the harness change (path filter) - the stepdocs/test-depth.mditself named as the natural next stage. A measured run takes ~2 minutes, not the ~10 the old header guessed. This PR triggers it, so the run on this PR is the gate proving itself. I also dispatched the workflow's first-ever run onmainfrom the Actions tab.loadtest.ymlis removed: it required an externally deployed stack and aLOADTEST_AUTH_TOKENsecret, neither of which has ever existed - it could not run, only appear runnable. The k6 scenario and its documented local invocation are unchanged;docs/test-depth.mdrecords why the workflow left and what would justify its return.
Reproduce the gaps on main
# 1. Backend coverage ratchet not wired (frontend runs test:coverage; backend doesn't):
grep -n "test --if-present\|test:coverage" .github/workflows/ci.yml
# 2. The evals job guards a file that doesn't exist, and passes:
ls evals/ 2>&1; grep -n "evals/run.mjs" .github/workflows/ci.yml
# 3. Zero runs ever for mutation + loadtest:
gh api repos/Open-Legal-Products/mike/actions/workflows/mutation.yml/runs -q .total_count
gh api repos/Open-Legal-Products/mike/actions/workflows/loadtest.yml/runs -q .total_count
# 4. The security modules exempt from their own mutation harness:
grep -n "privateIp\|verifyCitations" backend/stryker.config.json
Verify on this branch
npm run test:coverage --prefix backend # suite + floors: 57.35/50.40/58.27/58.83 vs 57/50/58/58 - passes
npm run test:mutation --prefix backend # 5 files, score 70.01 vs break 69 - passes (~1 min locally)
And on the PR itself: the "Mutation testing" check appears and runs because this PR touches the harness - the first PR-gated mutation run in the repo.
Demo
Live recording: main's inert gate lines, then the coverage ratchet passing at the raised floors, then the five-file mutation run finishing at 70.01 against the 69 floor:

Testing performed
npm run test:coverage --prefix backend: passes at the raised floors.npm run test:mutation --prefix backend: 70.01 total (privateIp 58.1 → 65.9 with the new tests; measured before/after logs in the demo GIF).- Full
npm test --prefix backend: 851 passed, unchanged. git diff --check: clean.- Dispatched
mutation.ymlonmain(its first run ever) to verify the harness executes in CI, not just locally.
Tradeoffs & design decisions
- Raising ratchet floors in the same PR that wires the gate: an alternative was enforcing the old floors (52/46/53/54), but a ratchet's contract is "no regression from now" - enforcing stale floors would allow a ~5-point silent drop on day one. In-flight PRs that reduce backend coverage below 57/50/58/58 will now fail CI; that is the ratchet doing its job, but it is a behavior change for open branches.
- Functions floor margin is thin (58.27 measured vs 58 floor). Consistent with the established round-down-to-whole-percent rule, but the first PR that deletes a small tested function may need to add a test or justify a floor adjustment.
evalsjob deleted, not fixed: resurrecting a harness the maintainer declined twice (#231, #257) is not this PR's call. If evals return, the job should return with them.- Mutation as a path-filtered PR gate: unrelated PRs never pay the ~2 minutes; PRs touching guard code do. The path list must be kept in sync with
stryker.config.jsonmanually - noted in both files. Equivalent-mutant noise remains (most survivingprivateIpmutants are masked by the fail-closed backstop, which is itself evidence the design fails safe). ignoreStatic: trueremoves module-load-time mutants from the score rather than pretending tests can toggle them; the BlockList tables' runtime behavior is asserted directly by tests instead. This is the documented Stryker remedy, not a score fudge (the score moved only 68.52 → 68.72 from it).loadtest.ymlremoved rather than made self-booting: a CI loadtest would need the full stack plus a stubbed LLM, and this repo has explicitly rejected stub-heavy harnesses ("does not call a model or exercise Mike's real chat" - #257 review). A load test against a stub measures the stub. The honest state is: local tool, documented, no workflow until a staging stack exists.
🤖 Generated with Claude Code
Our analysis
Make CI quality gates enforceable — read the full analysis →
Think the analysis missed something the PR description covers?
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-394.md from
inside the repo you want the changes in.