⚖MikeWatch Forks of the Mike legal-AI codebase, distilled for legal-tech readers
Home · Sloth-ninja/JessicaOSS · this thread

Sloth-ninja makes AI changes earn their way into JessicaOSS

This fork is putting a measurable quality gate between AI changes and a production merge.

workflowinfrastructure

Sloth-ninja has added a disciplined evaluation setup: known examples are scored, a baseline is recorded, and the automated checks can block a change that makes those results worse.

That matters because legal AI quality often erodes quietly. A polished interface can conceal weaker answers, missed details, or less reliable workflows until users spot the problem. Here, the fork treats tested AI behaviour as something that must be preserved, not merely demonstrated once.

The team has also adjusted its automated checks to tolerate a known test-runner glitch only when every test has otherwise passed. It is a practical choice, but worth inspecting on GitHub: the safeguard is only credible if it cannot wave through a genuine failure.

So what Legal teams and product leaders evaluating JessicaOSS should care if they want evidence that future AI changes are being held to a repeatable standard.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

11 commits from Sloth-ninja/JessicaOSS, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
257a8c7e Add CI merge-gate workflow (BUILD_PLAN §7) sloth-ninja 2026-07-12 ↗ GitHub
commit body 
Three jobs on PRs to main and pushes to main, with per-branch
concurrency cancellation for superseded PR runs:

- backend: npm ci, tsc --noEmit, vitest run (Node 22, npm cache)
- frontend: npm ci, tsc --noEmit, eslint, next build - public
  NEXT_PUBLIC_* values come from repository variables with safe
  placeholder fallbacks (Supabase client construction at module
  scope makes next build throw on empty URL/key)
- evals: npm ci in evals/, full golden set including the citation
  hard gate; COMPANIES_HOUSE_API_KEY and ANTHROPIC_API_KEY from
  repository secrets - the runner skips those cases loudly when
  the keys are absent, so the job passes on forks without secrets

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6
8802849d Evals: populate the four remaining judged golden-set fixtures sloth-ninja 2026-07-12 ↗ GitHub
commit body 
Fill input.text for jud-spa-review, jud-lease-lta1954,
jud-employment-minima and jud-disclosure-review with PROVISIONAL
workflow outputs generated over synthetic source documents (committed
under evals/fixtures/ for provenance and AGL associate review), and
remove their pending markers so they run once ANTHROPIC_API_KEY is
configured. Each output scored 5/5 on all four rubric dimensions by an
interim Opus judge pass; all 37 statutory citations resolve against
live legislation.gov.uk (HTTP 200, 12/07/2026). Synthetic parties
throughout per docs/safe-local-testing.md; no case-law citations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6
286dee21 Evals: commit judged-score baseline for the regression gate sloth-ninja 2026-07-12 ↗ GitHub
commit body 
Measured mean 4.83 on main (425b91b); baseline set at 4.5 to absorb
single-point judge variance while still catching real regressions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsUrp6sw58RCSeqJtx5RH6
20ed2698 Merge pull request #10 from Sloth-ninja/eval-judged-fixtures Ezana.H 2026-07-12 ↗ GitHub 
Evals: populate the four remaining judged golden-set fixtures (provisional, 5/5 interim-judged)
425b91bf Merge pull request #11 from Sloth-ninja/ci-merge-gate Ezana.H 2026-07-12 ↗ GitHub 
CI merge-gate workflow: backend, frontend, evals (BUILD_PLAN §7)
8112fe1a Merge pull request #12 from Sloth-ninja/eval-baseline Ezana.H 2026-07-14 ↗ GitHub 
Evals: commit judged-score baseline (regression gate)
6eaac458 chore(backend): typecheck project + ignore rule for operator scripts sloth-ninja 2026-08-14 ↗ GitHub
commit body 
`scripts/` holds operator tools that are RUN with tsx and never compiled
into dist/, so the build tsconfig keeps its `src`-only rootDir/include (the
same separation evals/ gets from its own project). tsconfig.scripts.json
typechecks them anyway (`npm run typecheck:scripts`), and the probe runner's
JSON output is git-ignored.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
75f465da harden(stop-check): tolerate the vitest worker IPC flake when every test passed sloth-ninja 2026-08-14 ↗ GitHub
commit body 
Owner-approved 14/08/2026 (in-session, exact patch presented and accepted).
Both vitest summary lines must show only passes AND the known
'Timeout calling "onTaskUpdate"' signature must be present; the tolerance
logs a one-line note each time. Any failed test, missing summary, or other
error blocks exactly as before. Verified against the real 13/08 flake
capture and synthetic failure/other-error/no-signature outputs.
Docs (CLAUDE.md, HANDOVER §0, BUILD_LOG) updated from "awaiting sign-off"
to applied.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
295f5e35 docs: every stop-check status mention now agrees (review fixes 1-5) sloth-ninja 2026-08-14 ↗ GitHub
commit body 
The hook commit updated three mentions but missed the DURABLE_LESSONS
Status paragraph (dated addendum appended - the file is append-only) and
four lines inside this PR's own BUILD_LOG entry (Scope, two Key-changes
bullets, the Deferred list, and a stale supersede clause).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aa10010c docs: the Deferred list no longer contradicts the Decisions bullet (re-review finding 2) sloth-ninja 2026-08-14 ↗ GitHub 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
39dae312 Merge pull request #87 from Sloth-ninja/close-out-pm-feedback Ezana.H 2026-08-14 ↗ GitHub 
Close-out: PM surface + 13/08 pilot-feedback wave shipped; stop-check hardening (owner-approved)

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-852.md from inside the repo you want the changes in.

⬇ Download capture-thread-852.md