[Testing 10] test: offline eval harness - citation accuracy, injection resistance, privilege leakage
From the PR description
Summary
The offline eval harness from the reference index (#205, row amal66#45): deterministic prompt-safety and behavior gates that run against committed fixtures - no network, no LLM calls, no secrets. Fully standalone (no dependency on any other PR in the testing series; doesn't even need the vitest harness).
Changes
evals/- runner (run.mjs), engine, scorers, fixture runner, golden dataset (8 cases + JSON schema), fixture model outputs, and a README covering how to add cases and how to plug in a live-model runner later.- Checks cover: citation accuracy against source text, prompt-injection resistance, and privilege/PII leakage.
Why
An LLM legal product can regress in ways unit tests can't see - a prompt tweak that starts leaking privileged content or following injected instructions. This gives a cheap, deterministic scorecard for exactly those behaviors, gated at --threshold 1.0, cheap enough to run on every PR (wired up by the CI workflow PR in this series, but runnable on its own today).
Testing
node evals/run.mjs --threshold 1.0- 8/8 cases passing locally on this branch.- The failure gate works: the fork row proved a failing fixture fails the run (non-zero exit).
Based directly on main; no shared commits with the other testing PRs.
🤖 Generated with Claude Code
Our analysis
Add offline prompt-safety evaluation harness — read the full analysis →
Think the analysis missed something the PR description covers?
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-231.md from
inside the repo you want the changes in.