ci: make every quality gate actually execute (or remove it)

🟢 open · #394 · open-legal-products/mike ← amal66/mike · opened 12d ago by amal66 · +148-173 across 7 files · ↗ on GitHub

From the PR description

Summary

Three of the repo's quality gates were mechanisms that never actually ran, discovered while auditing the CI story end to end:

  1. The backend coverage ratchet was never enforced. ci.yml ran npm test --if-present - the test:coverage script and its documented floors (backend/vitest.config.mts) have existed since the harness landed, but no CI job ever invoked them. The frontend job has run its ratchet all along.
  2. The evals job green-checked a directory that does not exist. It guarded evals/run.mjs; the eval-harness PRs were rejected upstream, so every PR since has carried a green "Eval harness" check that verified nothing.
  3. mutation.yml and loadtest.yml had zero runs ever (both created 2026-08-12; verified via the Actions API). Worse, the mutation scope excluded the two most safety-critical modules added since: privateIp.ts (the SSRF guard) and verifyCitations.ts (the "verified" badge).

This PR makes each gate real or removes it:

  • ci.yml backend job now runs npm run test:coverage; floors re-measured and raised to the current tree (57/50/58/58, from 52/46/53/54 - coverage rose unenforced, and a ratchet resumes from today, not from stale floors). The evals job is deleted.
  • stryker.config.json now mutates all five security libs. The first measured run with the expanded scope scored 68.52 - below the 69 break floor: the exempt files were under the bar the moment they were included (privateIp.ts: 57.9, 66 surviving mutants). New adversarial tests written from the surviving-mutant report (NAT64 prefix-confusion with public embedded IPv4s, the 0xffff hextet boundary, junk passed directly to isPrivateIpv6) bring the score to 70.01, back above the floor. ignoreStatic is enabled for the module-load-time BlockList table mutants that mutation switching cannot toggle.
  • mutation.yml now also runs on pull_request when the mutated modules, their tests, or the harness change (path filter) - the step docs/test-depth.md itself named as the natural next stage. A measured run takes ~2 minutes, not the ~10 the old header guessed. This PR triggers it, so the run on this PR is the gate proving itself. I also dispatched the workflow's first-ever run on main from the Actions tab.
  • loadtest.yml is removed: it required an externally deployed stack and a LOADTEST_AUTH_TOKEN secret, neither of which has ever existed - it could not run, only appear runnable. The k6 scenario and its documented local invocation are unchanged; docs/test-depth.md records why the workflow left and what would justify its return.

Reproduce the gaps on main

# 1. Backend coverage ratchet not wired (frontend runs test:coverage; backend doesn't):
grep -n "test --if-present\|test:coverage" .github/workflows/ci.yml

# 2. The evals job guards a file that doesn't exist, and passes:
ls evals/ 2>&1; grep -n "evals/run.mjs" .github/workflows/ci.yml

# 3. Zero runs ever for mutation + loadtest:
gh api repos/Open-Legal-Products/mike/actions/workflows/mutation.yml/runs -q .total_count
gh api repos/Open-Legal-Products/mike/actions/workflows/loadtest.yml/runs -q .total_count

# 4. The security modules exempt from their own mutation harness:
grep -n "privateIp\|verifyCitations" backend/stryker.config.json

Verify on this branch

npm run test:coverage --prefix backend   # suite + floors: 57.35/50.40/58.27/58.83 vs 57/50/58/58 - passes
npm run test:mutation --prefix backend   # 5 files, score 70.01 vs break 69 - passes (~1 min locally)

And on the PR itself: the "Mutation testing" check appears and runs because this PR touches the harness - the first PR-gated mutation run in the repo.

Demo

Live recording: main's inert gate lines, then the coverage ratchet passing at the raised floors, then the five-file mutation run finishing at 70.01 against the 69 floor:

PR #394 demo: gates executing

Testing performed

  • npm run test:coverage --prefix backend: passes at the raised floors.
  • npm run test:mutation --prefix backend: 70.01 total (privateIp 58.1 → 65.9 with the new tests; measured before/after logs in the demo GIF).
  • Full npm test --prefix backend: 851 passed, unchanged.
  • git diff --check: clean.
  • Dispatched mutation.yml on main (its first run ever) to verify the harness executes in CI, not just locally.

Tradeoffs & design decisions

  • Raising ratchet floors in the same PR that wires the gate: an alternative was enforcing the old floors (52/46/53/54), but a ratchet's contract is "no regression from now" - enforcing stale floors would allow a ~5-point silent drop on day one. In-flight PRs that reduce backend coverage below 57/50/58/58 will now fail CI; that is the ratchet doing its job, but it is a behavior change for open branches.
  • Functions floor margin is thin (58.27 measured vs 58 floor). Consistent with the established round-down-to-whole-percent rule, but the first PR that deletes a small tested function may need to add a test or justify a floor adjustment.
  • evals job deleted, not fixed: resurrecting a harness the maintainer declined twice (#231, #257) is not this PR's call. If evals return, the job should return with them.
  • Mutation as a path-filtered PR gate: unrelated PRs never pay the ~2 minutes; PRs touching guard code do. The path list must be kept in sync with stryker.config.json manually - noted in both files. Equivalent-mutant noise remains (most surviving privateIp mutants are masked by the fail-closed backstop, which is itself evidence the design fails safe).
  • ignoreStatic: true removes module-load-time mutants from the score rather than pretending tests can toggle them; the BlockList tables' runtime behavior is asserted directly by tests instead. This is the documented Stryker remedy, not a score fudge (the score moved only 68.52 → 68.72 from it).
  • loadtest.yml removed rather than made self-booting: a CI loadtest would need the full stack plus a stubbed LLM, and this repo has explicitly rejected stub-heavy harnesses ("does not call a model or exercise Mike's real chat" - #257 review). A load test against a stub measures the stub. The honest state is: local tool, documented, no workflow until a staging stack exists.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S4VRDD1DZat4SLYtt2q2eU

Our analysis

Make CI quality gates enforceable — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-394.md from inside the repo you want the changes in.

⬇ Download capture-pull-394.md