Sloth-ninja completes the legal-AI report card

The fork now has a fuller set of checked legal scenarios for testing whether its workflows stay accurate under pressure.

complianceworkflow

Sloth-ninja has completed four previously unfinished benchmark scenarios, taking the set to 35 reviewed examples. Each pairs an invented legal source document with the resulting workflow output, so legal reviewers can check the reasoning against known facts and regenerate the test later.

The new scenarios cover:

Spanish-language document review.
Commercial lease issues under the Landlord and Tenant Act 1954.
Minimum employment-rights questions.
Disclosure review.

The reported first-pass assessments gave each new example top marks for issue spotting, legal accuracy, UK terminology and factual discipline. The statutory references were also checked against the live legislation source, while the outputs intentionally avoid case-law citations. These are evaluation materials rather than a new product feature, but they make future changes easier to challenge against a concrete legal baseline.

So what Legal teams assessing AI reliability should care because this gives the fork a more credible, reviewable way to detect when legal outputs drift.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

1 commit from Sloth-ninja/JessicaOSS, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date

8802849d Evals: populate the four remaining judged golden-set fixtures sloth-ninja 2026-07-12 ↗ GitHub

SHA	Subject	Author	Date
`8802849d`	Evals: populate the four remaining judged golden-set fixtures	sloth-ninja	2026-07-12	↗ GitHub
commit body Fill input.text for jud-spa-review, jud-lease-lta1954, jud-employment-minima and jud-disclosure-review with PROVISIONAL workflow outputs generated over synthetic source documents (committed under evals/fixtures/ for provenance and AGL associate review), and remove their pending markers so they run once ANTHROPIC_API_KEY is configured. Each output scored 5/5 on all four rubric dimensions by an interim Opus judge pass; all 37 statutory citations resolve against live legislation.gov.uk (HTTP 200, 12/07/2026). Synthetic parties throughout per docs/safe-local-testing.md; no case-law citations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6

commit body

Fill input.text for jud-spa-review, jud-lease-lta1954,
jud-employment-minima and jud-disclosure-review with PROVISIONAL
workflow outputs generated over synthetic source documents (committed
under evals/fixtures/ for provenance and AGL associate review), and
remove their pending markers so they run once ANTHROPIC_API_KEY is
configured. Each output scored 5/5 on all four rubric dimensions by an
interim Opus judge pass; all 37 statutory citations resolve against
live legislation.gov.uk (HTTP 200, 12/07/2026). Synthetic parties
throughout per docs/safe-local-testing.md; no case-law citations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1202.md from inside the repo you want the changes in.

⬇ Download capture-thread-1202.md