[Testing 10] test: offline eval harness - citation accuracy, injection resistance, privilege leakage

⛔ closed · #231 · open-legal-products/mike ← amal66/mike · opened 3mo ago by amal66 · closed 3mo ago · +790 across 9 files · ↗ on GitHub

From the PR description

Summary

The offline eval harness from the reference index (#205, row amal66#45): deterministic prompt-safety and behavior gates that run against committed fixtures - no network, no LLM calls, no secrets. Fully standalone (no dependency on any other PR in the testing series; doesn't even need the vitest harness).

Changes

  • evals/ - runner (run.mjs), engine, scorers, fixture runner, golden dataset (8 cases + JSON schema), fixture model outputs, and a README covering how to add cases and how to plug in a live-model runner later.
  • Checks cover: citation accuracy against source text, prompt-injection resistance, and privilege/PII leakage.

Why

An LLM legal product can regress in ways unit tests can't see - a prompt tweak that starts leaking privileged content or following injected instructions. This gives a cheap, deterministic scorecard for exactly those behaviors, gated at --threshold 1.0, cheap enough to run on every PR (wired up by the CI workflow PR in this series, but runnable on its own today).

Testing

  • node evals/run.mjs --threshold 1.0 - 8/8 cases passing locally on this branch.
  • The failure gate works: the fork row proved a failing fixture fails the run (non-zero exit).

Based directly on main; no shared commits with the other testing PRs.

🤖 Generated with Claude Code

Our analysis

Add offline prompt-safety evaluation harness — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-231.md from inside the repo you want the changes in.

⬇ Download capture-pull-231.md