Sloth-ninja gives JessicaOSS a quality-loss tripwire

A new evaluation threshold is designed to catch meaningful drops in answer quality without overreacting to normal variation from an AI judge.

workflowinfrastructure

Sloth-ninja has turned a documented quality target into an active regression check, using the fork's first fully live evaluation run as its starting point. Six judged scenarios averaged 4.83 out of 5, but the gate is set slightly lower at 4.5.

That margin is deliberate. A single scenario being reassessed from a 5 to a 4 should not block progress, while a genuinely weak result or failed scenario should trigger attention. The threshold is meant to become stricter once repeated runs show how consistently the AI judge scores the work.

So what Legal-AI teams that need to improve a product without quietly eroding its practical usefulness should watch this as a modest but important quality-control step.

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

1 commit from Sloth-ninja/JessicaOSS, oldest first. Source extracted verbatim from the harvested git log.

SHA	Subject	Author	Date
`286dee21`	Evals: commit judged-score baseline for the regression gate	sloth-ninja	2026-07-12	↗ GitHub
commit body Measured mean 4.83 on main (425b91b); baseline set at 4.5 to absorb single-point judge variance while still catching real regressions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6

SHA

Subject

Author

Date

286dee21

Evals: commit judged-score baseline for the regression gate

sloth-ninja

2026-07-12

↗ GitHub

commit body

Measured mean 4.83 on main (425b91b); baseline set at 4.5 to absorb
single-point judge variance while still catching real regressions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1199.md from inside the repo you want the changes in.