eliziff puts a hard ceiling on legal chat's memory

This Beaver fork treats a long-running matter as a managed workspace, not an ever-growing prompt.

chat-uiworkflow

The headline change is prompt discipline: the assistant tracks the cost of its instructions and document context, keeps large matter rosters to a fixed size, and warns when the working budget is halfway spent before anything gets clipped. One measured cleanup cut 2,214 tokens from the fixed prompt overhead.

Alongside that, eliziff is adding practical matter handling:

  • Email intake, reflecting the reality that many legal matters live in inboxes.
  • Document attachments that announce their presence without loading their full text into every chat.
  • A recycle bin for chats, with restore and permanent deletion.
  • Spreadsheet-style review that can retrieve individual cells and preserve the review route.

The assistant also asks for missing blockers rather than vague permission to proceed.

So what Teams building document-heavy legal assistants should look closely: this is a concrete attempt to keep chat useful and predictable as a matter grows.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

31 commits from eliziff/Beaver, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
dc1c3f46 refactor: emit helper and shared citationUrls in cloud chat transport Eli Ziff 2026-07-27 ↗ GitHub
commit body
runLLMStream destructures its params in the signature, a local emit()
replaces nine hand-rolled SSE data-frame writes, the single-caller
flushVisibleTail folds into flushText, and citationUrls() moves to
citations.ts so both transports share the URL-extraction flatMap.
Public surface unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o
2b62ebf7 feat: chat turn store rename, stop route, attached-doc context; local transport compaction Eli Ziff 2026-07-27 ↗ GitHub
commit body
anonymousChatTurns becomes chatTurns with abortChatTurn powering the
new POST /chat/:chatId/stop route; PATCH accepts project_id to move a
chat between projects; user-attached documents get an explicit context
preamble in both transports. routes/chat.ts also lands its compaction:
persistTurnEvents() replaces the four copies of the three-way turn
persistence fan-out, beginSseTurn()/sseFinishTurn()/fail()/conflict()
collapse the repeated SSE and error plumbing, and writes after the
socket is gone are uniformly skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o
bb677f32 feat: matter-state schema, store, and bounded prompt assembler (Issue 9 stages 1-3) Eli Ziff 2026-07-27 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
e0973749 feat: chat recycling bin backend (soft delete, restore, purge) Eli Ziff 2026-07-27 ↗ GitHub
commit body
Deleted chats keep their transcript under deleted_at instead of vanishing;
restore and purge endpoints for both cloud and account-free planes, with
deleted_at filtering across chat listings and project routes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
797c9a74 feat: tabular review cell reads and route persistence Eli Ziff 2026-07-27 ↗ GitHub
commit body
readTabularCells lets chat tools read selected review columns/rows; tabular
routes gain persistence covered by restart-surviving integration tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
7a2c31a5 feat: document automation events on the local plane Eli Ziff 2026-07-27 ↗ GitHub
commit body
localAutomationEvent records automation runs against library documents;
deterministic DOCX cleanup extended to cover automation output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
15846735 feat: recycling bin UI, app settings modal, document automation UI, and interface compaction Eli Ziff 2026-07-27 ↗ GitHub
commit body
Recycling bin and delete-warning flows in the sidebar; settings move from
account pages into an in-app modal; document automation surfaces in the
library; UI copy cut to the content rule (no restated labels, bounded lists,
stable first-frame geometry); focused component tests added alongside,
redundant snapshot-style suites removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
45ceba7b refactor: remove duplicate project chat and dead context code Eli Ziff 2026-07-28 ↗ GitHub
48df7224 Simplify project chat actions Eli Ziff 2026-07-28 ↗ GitHub
d17bd74a Remove write-only active chat state Eli Ziff 2026-07-28 ↗ GitHub
772fa3e4 Record document and chat state cuts Eli Ziff 2026-07-28 ↗ GitHub
f9e0f84b Make pending chat handoff one-shot Eli Ziff 2026-07-28 ↗ GitHub
6e7f5bf8 Record pending chat handoff result Eli Ziff 2026-07-28 ↗ GitHub
f1adf5a4 refactor local runtime and collapse duplicate chat paths Eli Ziff 2026-07-28 ↗ GitHub
2d59ad07 Announce attachments, do not preload their full text Eli Ziff 2026-07-28 ↗ GitHub
commit body
Drop the provider-side inlining of every attached document's extracted
text. formatChatMessageContent already prepends a deterministic manifest
(filename + document_id) to the user message, so the model always knows
what is attached and pulls content via the Library tools only when it
needs it. Aligns chat.ts with the test committed in f1adf5a, which
asserts the manifest is present and full text is not.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
dbbe47c6 chore: remove dead chat helpers Eli Ziff 2026-07-28 ↗ GitHub
7b5f9129 perf: keep local app free of chat tool dispatcher Eli Ziff 2026-07-28 ↗ GitHub
0d3f3a04 shrink: remove project chat dropdowns Eli Ziff 2026-07-28 ↗ GitHub
b3917960 shrink: keep chat file drops local Eli Ziff 2026-07-28 ↗ GitHub
49a2d15f shrink: keep compact chat renderer native Eli Ziff 2026-07-28 ↗ GitHub
6c7cc077 shrink: apply chat auth once per router Eli Ziff 2026-07-28 ↗ GitHub
9b74c8b3 Benchmark runs: remove ask_inputs tool, raise child cap to 3h Eli Ziff 2026-07-28 ↗ GitHub
commit body
Sonnet through the Beaver arm paused the banking run on ask_inputs
- there is no user in a benchmark run to answer, and the reference
harness has no ask-user affordance either (parity). Gate mirrors
MIKE_DISABLE_RESEARCH_TOOLS. The 30-min child cap would also have
killed any claude-p whole-deliverable run mid-flight (LAB's own
sonnet row took 48 min).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
a2dd9938 Ingest .eml: legal matters are mostly email Eli Ziff 2026-07-28 ↗ GitHub
commit body
Beaver had no email ingester at all -- .eml was outside
ALLOWED_DOCUMENT_TYPES, and the LAB arm works around that by wrapping the
message as a .docx with its content unchanged. Replaying the smoke run's
audit showed what "unchanged" costs: the fixture is a real
quoted-printable message, so soft line breaks land mid-digit and the
reader sees "$85,0=\n00". Three amounts in one client email were being
read wrong by both the model and the deterministic auditors.

Measured on the LAB fixture (deterministic, no model):
  money anchors     33 raw -> 33 decoded
  malformed anchors  2 raw ->  0 decoded   ($47,00 and $85,0)
  recovered                   $47,000, $85,000, $320

lib/emailText.ts parses RFC 5322 headers (with folding and RFC 2047
encoded-words) and MIME bodies: quoted-printable and base64 transfer
decoding, charset decoding via TextDecoder, multipart traversal that
prefers the plain-text rendition of an alternative and strips HTML only as
a fallback, and embedded message/rfc822. Part bodies are handled as latin1
so string offsets slice the Buffer byte-for-byte; charset decoding happens
strictly after transfer decoding, since both encodings yield bytes.

What it will not decode becomes a typed abstention rather than a guess:
not_an_email, unsupported_encoding, unsupported_charset, part_too_large,
nesting_too_deep, encrypted. Attachments are named with their type and
size instead of inlined -- an attachment is evidence that something exists
which this file does not contain, and the reader needs to know that.

Wired through textParserFor, which is the single seam both the production
read path and the local library path already share.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
fda59230 Put a floor under what one tool result costs the transcript Eli Ziff 2026-07-29 ↗ GitHub
commit body
Surveying six open-source harnesses turned up one thing every single one of
them does that Beaver did not: cap the size of a tool result at the
transport, and pair the cap with a message telling the model how to fetch
the rest. Codex caps at 10,000 tokens middle-out; pi and opencode at 2,000
lines or 50KB with "Use offset={next} to continue"; goose spills over
200,000 chars to a tempfile; Cline truncates at 400KB and points at search.

Beaver had excellent per-organ caps and no backstop. result() was a bare
JSON.stringify, and an untargeted library_read defaulted to 300,000
characters -- and that is not paid once, because the adapters re-send the
whole transcript every round, so one unbounded read is re-billed for the
rest of the turn.

  - result() now enforces 64,000 chars. That sits above every deliberate
    read (section reads cap at 60,000, as does PDF lookup), so it bites
    untargeted whole-document reads and nothing else. It trims the one
    oversized string field rather than the envelope, so what the model
    parses stays well-formed JSON, and it keeps head AND tail because a
    clause's proviso lives at its end.
  - The untargeted library_read default drops from 300,000 to 24,000. The
    300,000 ceiling stays for a caller that deliberately asks.
  - A truncated result names its own recovery, and Beaver can be more
    specific than a byte offset: library_outline for the section map, then
    library_read(section=...), or a library_find hit's `at` as `offset`.

scripts/lean-smoke.ts is the modelful smoke this came out of: one real turn
over the real /chat route, reporting the tool sequence the model actually
chose and what the turn cost. Two runs on claude -p sonnet, ~70s each.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
4199ff59 Tell the model what documents exist; price the prompt Eli Ziff 2026-07-29 ↗ GitHub
commit body
A plainly worded question ("the lease", "my client's email") got zero tool
calls and "I don't have access to your Beaver Library in this session".
The same question naming the files got five tool calls and correct answers.
The prompt already said "Use library_list before claiming a Library
document is unavailable" -- it is one sentence inside a 20-sentence block
with no line breaks, and it was ignored. Knowing the tools exist is not the
same as knowing the matter exists.

libraryInventory.ts names the in-scope documents and their ids in the
system prompt. ~20 tokens per document, and it removes the library_list
discovery round trip since the ids the tools need are already there.
Structure stays on demand: this is an inventory, not an outline.

scripts/prompt-budget.ts prices the fixed overhead, because the first
instinct (mine) was to blame the prose and that was wrong:

  tool schemas   6,964 tok  (params 4,147 / descriptions 2,248)
  system prose   1,059 tok  (largest block 928 tok, 20 sentences, 0 breaks)

and prices leaner encodings of the same information, for the routes where
we own the wire format:

  V0 JSON Schema (today)              6,964
  V1 JSON, param descriptions dropped 4,574   -34%
  V3 signature + full descriptions    2,913   -58%
  V5 index only                       1,070   -85%

V3 keeps every word of every tool description; the saving is JSON Schema
structure plus per-parameter prose. Not shipped -- dropping parameter
descriptions is exactly where argument correctness could break, so it needs
an A/B first.

MIKE_TOOL_RESULT_CAP, MIKE_READ_DEFAULT_CHARS and MIKE_LIBRARY_INVENTORY
make the three thrift changes togglable so one binary runs both arms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
46450c62 ask_inputs: ask for blockers, not permission Eli Ziff 2026-07-29 ↗ GitHub
commit body
Real-world report: asked to "read the first line of this, then pause", the
assistant called ask_inputs to ask "Would you like me to read only the
first line and then stop?" - a round trip spent restating the instruction
it had just been given.

All three places that instructed on this tool were purely permissive. Each
said when to ask ("decisions, clarifications, or document uploads",
"clarify a missing premise", "would materially change the work"); none said
when not to, and "clarification" is an open invitation.

Whether to ask is judgment, so it stays prose rather than becoming a guard.
The instruction now draws the line lawyers already work to - blocker versus
deferrable: an instruction only the user can give, or a document never
provided, versus ambiguity you resolve on the most reasonable reading and
flag as an assumption. Plus the one hard prohibition the failure needed:
never seek confirmation of an instruction already given.

Same wording across the tool description (read at the point of use) and
both system prompts, so the three cannot drift apart again.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
949656c4 Every rule gets one home: 2,214 tokens off the fixed overhead Eli Ziff 2026-07-29 ↗ GitHub
commit body
Measured first: 12,833 fixed tokens per turn (2,131 prompt + 10,702
tool array). Then cut by single-homing - a tool's own contract lives in
its description, cross-tool routing in the prompt, corrective feedback
in result strings only where not taught elsewhere. Seventeen rules
deduplicated across prompt, descriptions, next_required_action, and
error strings; result-string re-teaching alone drops 599 tokens per
full fire set. Smoke first-input: full 8,369 to 6,937, lean 7,817 to
6,790, four-for-four behaviours and correct answers on both. Stale
assertions on the removed version_id requirement and the [^id] note
contract updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
3410fdcf Matter roster: constant-size above eight documents Eli Ziff 2026-07-29 ↗ GitHub
commit body
Per-line rosters scale per turn with matter size; past the cap one line
now names the matter's existence and count and sends discovery back to
the tools, preserving the measured anti-failure (vague wording without
any roster produced zero tool calls). The 12-document antitrust dev
task is the deciding A/B cell.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c66a410a Meter the tool budget at halfway instead of enforcing it silently Eli Ziff 2026-07-29 ↗ GitHub
commit body
Both adapters append one line to the last tool result of the middle
round naming rounds used and remaining, so the model plans the back
half rather than discovering the wall.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
d6d03a4f retrieval: put the passage lane on the product chat path Eli Ziff 2026-07-30 ↗ GitHub
commit body
Wires today's passage stack into a2aj_search behind MIKE_PASSAGE_SEARCH=1.
All additive and flag-gated; the document-level lane is untouched and is
the silent fall-back whenever the lane cannot serve the request.

- build-passage-index.ts: --mode chars|clause, part of sidecar identity;
  the MissingPassageIndexError hint now names --mode when one is asked
  for.
- a2ajPassageSearch: mode passthrough, and rerank {model} (or
  MIKE_RETRIEVAL_RERANK_MODEL) widening the pool to max(48, size*8) at
  perDocCap 24 before rerankPassages cuts back to size. Rerank is async,
  so searchLocalA2AJPassages now returns a Promise - a signature change,
  its only callers were tests.
- citation short-circuit expands each detected key through the citator's
  resolution evidence (new citationAliasKeys wrapper over the existing
  private keysForQuery), so a French twin or parallel reporter cite
  reaches the same decision. No citator graph, or an ambiguous key,
  degrades to exactly today's literal-key behaviour.
- chat/tools/a2ajTools: a2aj_search prefers the passage lane when the
  env is set AND the sidecar exists (never built inline), returning the
  same result shape with the passage in `snippet` plus passage_start/
  passage_end. Name search and dataset/date/sort filters stay on the
  document lane rather than being silently dropped.

Tests are model-call-free (a completeText tripwire asserts zero calls):
clause-mode round trip against clauseChunkText spans, --mode in the
refusal, rerank short-circuit and env-absent no-op, alias pin with and
without a fixture citator graph, and chat-lane serve/fall-back.
18 passed across a2ajPassageWiring, a2ajPassageSearch, passageRetrieval;
tsc --noEmit clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
34fca3ea reflow narrow chat controls Eli Ziff 2026-08-10 ↗ GitHub

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-839.md from inside the repo you want the changes in.

⬇ Download capture-thread-839.md