Sloth-ninja puts safety rails around Clio integration testing

This fork turns unanswered practice-management questions into tightly controlled live checks, without letting routine testing touch client data.

integrationworkflow

Sloth-ninja has rebuilt a set of owner-run probes for Clio, the legal practice-management platform. They are designed to settle practical questions before the integration is relied on, while keeping normal application behaviour separate from live testing.

  • Checks how large result sets are split across pages and filtered.
  • Tests how activities, financial records and contact lookups behave in real accounts.
  • Restricts ordinary probes to read-only requests and blocks access to unrelated database services.
  • Gates the one edit test behind explicit owner approval and a designated test matter, with safeguards against deleting billed time.

The work also records when a result cannot yet be conclusive, rather than pretending an incomplete test answered the question.

So what Legal teams considering a Clio-connected workflow should care because the fork is doing the unglamorous validation needed to make integrations safer to trust.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

6 commits from Sloth-ninja/JessicaOSS, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
6eaac458 chore(backend): typecheck project + ignore rule for operator scripts sloth-ninja 2026-08-14 ↗ GitHub
commit body
`scripts/` holds operator tools that are RUN with tsx and never compiled
into dist/, so the build tsconfig keeps its `src`-only rootDir/include (the
same separation evals/ gets from its own project). tsconfig.scripts.json
typechecks them anyway (`npm run typecheck:scripts`), and the probe runner's
JSON output is git-ignored.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aa97f630 feat(clio): committed live-probe runner for the Practice Management spec sloth-ninja 2026-08-14 ↗ GitHub
commit body
Regenerates the lost scratchpad probe script, this time in the repo. Answers
docs/PRACTICE_MANAGEMENT_SPEC.md's four open questions against the live EU
tenant with a stored connection: offset continuation on /matters.json (Q1),
the corrected billable_matters selector (Q2), billed-entry write restrictions
(Q3, write probe), and the corrected contacts selector (Q4) - plus the five
remaining reads the Matters seam depends on, so sign-off is not granted to
calls that were never exercised live.

Safety rails are structural, not promised:
- the Supabase handle is a proxy that throws on insert/update/upsert/delete/
  rpc, so the client's self-healing paths (refresh persistence, dead-grant
  prune) can never rewrite or delete a production connection row - they are
  surfaced instead;
- probes 1-9 can only call probeGet, which cannot express a method or a body;
- probe 10 needs BOTH --write-probe and CLIO_PROBE_TEST_MATTER_ID, creates one
  non-billable entry, answers the billed question with a NO-OP patch (never a
  destructive delete of billed time), and cleans up in a finally with honest
  reporting of anything left behind;
- sequential requests, small pages, and an abort after two consecutive 429s
  keep the run inside the 50 req/min budget shared with production.

A decryption failure is detected and explained (USER_API_KEYS_ENCRYPTION_SECRET
must match Fly) rather than surfacing as "not connected", and results record
shapes - field presence, counts, statuses - never client data.

The Matters seam's selector constants are exported so the probes test the exact
shipped strings rather than copies that could drift.

Verified: npx tsc --noEmit clean; tsc -p tsconfig.scripts.json clean; prettier
clean; 180 lib/clio tests pass; --dry-run walks all ten probes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fd29a7b5 fix(clio-probes): review wave - write-probe boundary, rail verdicts, coverage sloth-ninja 2026-08-14 ↗ GitHub
commit body
Independent review of the probe runner: one Critical, three Important, six
minors. All applied.

CRITICAL - the write probe escaped its own boundary. The billed sub-probe took
its candidate from probe 9, which runs against an ARBITRARY live matter (probe
2's first row or --matter-id), not CLIO_PROBE_TEST_MATTER_ID - so the "no-op"
PATCH would have landed on a real client's invoiced time entry, contradicting
the file's own promise. Worse, a null/withheld note degraded to "", so the
no-op could have BLANKED a note on an invoiced record. The sub-probe now runs
its own search on the test matter, restricted to the caller's own entries, and
skips with an UNANSWERED verdict unless the note is readable (also skips when
the caller's Clio user id is unknown, rather than widening to colleagues').

IMPORTANT - inner catches swallowed the safety rails. Probe 3 scored ANY
failure as "pass"; probes 6/8 absorbed a blocked DB write or a rate-limit abort
into `controlRejected` and carried on issuing requests. Every inner catch now
calls rethrowRailErrors first, and probe 3's non-400 branch is "inconclusive" -
only an outright rejection answers open question 1.

IMPORTANT - the "never initiates OAuth" claim was imprecise: a 401 can still
make the shipped client attempt one refresh-token exchange before the (blocked)
persist. Reworded to say what is actually true - no authorisation flow ever, and
the stored row is never rewritten. client.ts deliberately untouched.

IMPORTANT - probe 2 now also exercises the list `query=` search and `status=`
filter, the exact surface #71 rebuilt and the one part of the Matters header
that had never been called live.

Minors: cleanup DELETE sleeps once and retries a single time past a drained
rate-limit window (what it is deleting is live data); a billed-edit refusal
counts only 401/403/409/422, so a 500 no longer reads as "Clio refuses";
`npm run typecheck:scripts` added to the backend CI job; the readOnlyDb proxy
inverted to an ALLOW-list (only a wrapped `from`) after the reviewer showed
`schema().from().update()` walks past a deny-list; the contacts selector hoisted
into an exported CONTACT_SEARCH_FIELDS in manageTools.ts so the probe tests the
shipped string, not a copy; matter-id bound tightened to 15 digits. Also fixed
en route: a preflight failure printed "Unknown error" because Supabase rejects
with a plain object, not an Error.

Verified: tsc --noEmit clean; tsc -p tsconfig.scripts.json --noEmit clean;
prettier clean; full backend suite 888 passed / 50 files; --dry-run walks all
ten probes; all four refusal paths plus the missing-env path refuse correctly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
711ea9b3 fix(clio-probes): make the read-only DB rail a genuine allow-list, and test it sloth-ninja 2026-08-14 ↗ GitHub
commit body
The previous proxy CLAIMED to allow only from() while implementing a deny-list
of property names, and the reviewer walked past it twice: `db.rest` is a public
property holding the PostgrestClient, so `db.rest.from(t).update(...)` and
`db.rest.rpc(...)` never met the guard, and `db.schema("public").from(t)`
returns a fresh, unwrapped builder. Enumerating the ways out of a client is a
game you lose on the next dependency bump; naming the one way in is not.

Reading ANY property other than `from` now throws. Symbols and `then` are the
two deliberate exceptions and return undefined rather than throwing - a proxy
that throws on Symbol.toStringTag explodes the moment anything logs it, and one
returning a function for `then` looks thenable to `await`. Neither can reach the
database.

Moved into scripts/readOnlyDb.ts so the guarantee can be unit-tested without
importing a script that runs on import, with scripts/readOnlyDb.test.ts (15
tests) covering the happy read path, all four builder writes, every refused
entry point, both proven bypasses, and the inert symbol/then handling. The
reviewer's point that a manual one-off smoke adds no regression protection is
correct - this survives refactors. vitest.config.ts include extended to
scripts/**/*.test.ts so `npm test` runs it.

Verified: tsc --noEmit + tsc -p tsconfig.scripts.json clean; prettier clean;
15/15 new tests; --dry-run clean; and the real Supabase client still reads
through the stricter proxy (offline smoke with dummy env reaches the network
rather than a blocked-access throw).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b22a0df2 fix(clio-probes): rethrow rails in the stale-etag catch; assert billed ownership sloth-ninja 2026-08-14 ↗ GitHub
commit body
Two one-line review fixes in the write phase, where the stakes are highest.

- The stale-If-Match catch was the one remaining rail-swallowing site: a blocked
  DB access or a rate-limit abort would have been filed as "Clio did not enforce
  concurrency" AND left the run issuing further writes. It now rethrows first,
  like every other inner catch.
- The billed candidate's ownership no longer rests solely on Clio honouring
  `user_id=` server-side; the row's own `user.id` must match the caller's stored
  Clio user id. If that param were ever ignored or loosened, the "no-op" patch
  would otherwise have landed on a colleague's billed time.

Verified: tsc both projects clean; prettier clean; --dry-run clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b129ca04 docs: BUILD_LOG entry for the committed Clio live-probe runner sloth-ninja 2026-08-14 ↗ GitHub
commit body
Scope, the derivation of probes 1-9 (the docs name only the four open
questions), the structural safety rails, verification, and both independent
review rounds with the findings they caught - the write probe escaping its test
matter, and the read-only rail's deny-list being bypassable via `db.rest` and
`db.schema()`.

Decisions recorded: billed DELETE is deliberately never probed; and the owner
prerequisite that the designated test matter must carry the owner's own billed
time with a readable note, or open question 3 returns UNANSWERED by design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1011.md from inside the repo you want the changes in.

⬇ Download capture-thread-1011.md