eliziff brings Canadian and US legal sources closer to home

logRawOpinionPayload had zero call sites and was the only consumer of
the fs/path imports; buildTnaPinpointUrl had no production caller
(superseded by provider-native anchors); opinionText's html/plain_text
branches were unreachable because compactOpinion already sets text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o

Live probes across 14 Decisia/Norma hosts (FCA, FC, TCC, ONCA, NSCA,
CIRB, SST, CART, CMAC, CT, FPSLREB, CITT, CHRT, SCC) show the identical
behaviour Beaver only handled for decisions.scc-csc.ca: the default URL
is an iframe shell with no document text, so anchors and text fragments
silently no-op; ?iframe=true serves the inline document with name=parN
anchors. One host+path predicate replaces the hostname special case,
the 22-entry LEXUM_PARAGRAPH_DATASETS table (wrong in both directions),
isLexumParagraphSource, and the scrollableLexum option. The flag now
applies for every locator kind and every caller, not just A2AJ
paragraph lookups. site_preference=mobile is dropped: unnecessary and
it pinned a persistent layout cookie on the court's whole site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o

Removing it broke the feature: in the desktop rendering a text-fragment
jump locks the viewport on the matched text and the page cannot scroll.
mobile is load-bearing for scrollable pinpoints, and the preference
cookie is irrelevant because users only reach these URLs through
Beaver's deep links. Comment now records this so no future cleanup
repeats the mistake. The 14-host Decisia generalization stands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o

One immutable content-hashed artifact per fetched source: text, lazy
token spans, ordered blocks with native/heuristic origin, and a
locator index. Providers are compilers, consumers are queries.

Every fixture shape was captured live from api.a2aj.ca and two of the
brief's claims died on contact: the 'inline heading marker' SCC shape
is really a table-of-contents block (which halves the paragraph spine
- recorded in ranges.missing), and Oakes 1986 is dot-numbered, not
unnumbered (R. v. Duarte 1990 is the true abstain case). Newly
indexed: federal bold-emphasis sections (Criminal Code 0 -> 11,165
blocks live), alphanumeric regulation sections (CRC 870 0 -> 7,911),
Ontario out-of-order cross-reference traps. Two section-map nesting
bugs fixed and pinned: Number('01')===Number('1') dropped s.231(6.1),
and (C)/(D) parsed as Roman numerals. Byte-identical parity with the
legacy engine everywhere legacy produced anything.

Compile: 2.26 MB Criminal Code in 29-39 ms; tokenization deferred to
a memoized getter (144 ms, first quote query only).

Browser gate (scripts/deeplink-gate.mjs) run on the generated URLs:
Decisia anchors and text fragments land (including anchor-less 2001
SCC 1, where the fragment alone jumps - anchor availability is
per-document); A2AJ's raw-XML and JSON-API source URLs are proven
non-pinpointable (fragment never jumps) and recorded as negative
controls for the stage-3 host/anchor table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o

firstString() collapses the asString ?? asString chains (order
preserved per chain - API and bulk paths intentionally differ),
attachOpinionStructure() dedupes the WeakMap + structure tail shared
by API and R2 opinion mapping, failRow() replaces four identical
error-row pushes, and the sanitizer allowlists collapse to split
strings with identical contents. -165 lines; wire shapes untouched.

Verified: tsc clean; courtlistenerLocalBulk, docxCitationLinking,
legalSourceStructure suites green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsMXzBsEJ27DjJVPSfW5o

Imports huggingface.co/datasets/a2aj/hansard (Ontario, 1.35M interventions)
into the local SQLite legal-data plane via scripts/import_a2aj_hansard.py;
a2ajHansard.ts exposes FTS search and fetch over interventions. Tests run
the real import script over captured fixture rows, no network.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga

Add fetch_courtlistener_bulk.py: downloads CourtListener's official,
no-auth bulk CSVs (citations 127 MB, opinion-clusters 2.46 GB in full)
and a deterministic HTTP-Range prefix of the 54.6 GB opinions file,
cut at the last complete CSV record with a quote/escape-aware scanner.
The existing import_courtlistener_bulk.py loads the slice unchanged.
Measured: 2.68 GB downloaded, 30m26s import, 5.87 GB SQLite with
18.1M citations, 10.07M clusters, 17,343 full-text opinions; citation
lookup, case-name search, and opinion retrieval all serve locally with
no COURTLISTENER_API_TOKEN.

Test drives the slicer over a truncated multi-stream bz2 fixture via
file:// and the real importer, proving slicer, importer, and reader
agree on schema.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga

hansard_search / hansard_fetch expose lib/a2ajHansard.ts (store,
importer, and tests existed; the read path was imported by nothing).
Registered in the research-gated group so sealed benchmark runs
never see it; typed hansard_not_installed reply when no database
has been imported. Tool-layer tests drive the real import script
over the captured fixture rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns

Generalizes the parseCache pattern (evaluation-context plan §12) to
a keyed, TTL-bounded, atomically written cache and threads it
through the two authority fetch seams that had only per-session
in-memory caching: CourtListener API GETs (24h ceiling, token never
in the key) and public legal source text/JSON fetches (7d/24h).
A2AJ already had its own persistent response cache. Stands aside
under NODE_ENV=test so fetch-stubbing suites keep their call
assertions; failures fall through to real fetches.

Chunks/embeddings/retrieval-result/preview caches from the plan's
menu stay unbuilt: no live consumer exists for them yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns

searchLocalA2AJPassages puts passageRetrieval on the product search
path additively - searchLocalA2AJ and every existing export are
unchanged. Two lanes, deterministic first: citation_lookup exact
matches on citation-shaped substrings of the query prepend those
documents' best passages ahead of the bm25 ranking; identity beats
ranking. Results carry document metadata joined from the source db and
verbatim char offsets.

The 5.5 GB-scale sidecar is never built inline (a build would hang a
user request): a missing index is a typed MissingPassageIndexError
naming scripts/build-passage-index.ts, the one place it is built.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD

The product passage lane pulled `SELECT * FROM document WHERE id = ?`
once per pooled hit. That row carries unofficial_text_en and _fr, both
unofficial_sections maps and both cases_cited lists - hundreds of KB -
and the loop keeps a citation, a name, a date, a URL and a dataset, all
short strings. The passage text itself comes from the hit, which
searchPassages already sliced, so the bodies on that row were never
read at all. Worse, the dedupe key is docId:start, so a document that
supplies many hits (perDocCap 24 under rerank) refetched its whole
payload once per hit.

Same defect as 17884220, one module over: it was fixed for the text
lookup in passageRetrieval and left standing here.

Narrow column list + a per-call memo. Same rows, same values, same
ordering; the memo dies with the call.

Measured on 300 LegalBench mini queries (size 8, rerankHits 48) against
a 69-document / 106 KB-mean bulk db: 157.93 -> 54.92 ms/query, 2.9x.
Unchanged-output proof: sha256 of every returned A2AJPassageResult[] is
byte-identical over all 300 queries (digest of digests
e4f136fc8a327666447bafea8f873ecdda3aa72c9bf1a6e26d5410d56dadff31). The
production A2AJ corpus populates both languages plus the section maps,
which this bed leaves empty, so the real saving is larger.

28 tests pass across a2ajPassageSearch, a2ajPassageWiring,
passageRetrieval.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H

Inert-by-measurement is not scoped. Two independent differentials say the
recoveries do not move legislation - 23,531 statutes / 2,924,267 nodes
byte-identical, and a 6,129-document sweep whose 114 skeleton changes are
ALL LegalBench-RAG (cuad 79, maud 35), zero in statutes, cases, amending
acts or DOCX - but a measurement is evidence about today's corpus, not a
guarantee about tomorrow's.

`sourceDocA2AJ` never calls `compileAgreementSkeleton`, so the bulk
legislation compiler was always out of reach. The live path that is not is
`passageRetrieval.ts` in clause mode, where `ensurePassageIndex` feeds A2AJ
`unofficial_text_en/fr` for doc_type laws straight in. That call now passes
`recoverExtraction: false`.

A2AJ ships the publisher's line breaks, so there is no extraction damage to
recover there and nothing to compete: legislation and case law keep the
structure the hardened statute and paragraph work gives them, because the
new code does not run at all. The option defaults to true for the Library
lane, which compiles PDF- and DOCX-derived agreements and is what the
recoveries exist for. Contract numbers are unchanged: mini 16 refused /
4,414 resolved / 9.1% missing.

Also recorded from the same sweep, the soundness invariant of the space-run
recovery: zero of 6,129 documents changed skeleton output without containing
an internal run of 2+ spaces, and 5,516 of them contain none.

Rejected on the way, with its cost, so it is not retried: gating instead on
"a statute spine wins" is blunt and charges contracts whose page-number runs
fake a spine (mini 16 -> 21 refused, holdout 9 -> 13); conditioning that on
the spine reading also addressing 50% of the document's own references gets
the cost down to 5 and 85 resolved but is a heuristic, not a scope, and
Eli asked for a scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H

- seeded capture/verify harness (harness.mjs) with layered text/claims caches
- worker surfaces opinion delimiters at semantic (par N-M), offset, and
  combined page resolutions from SourceDoc geometry
- legalOpinionBoundaries partition now emits spans and judges
- broaden A2AJ heading-join grammar (roman numerals, abbreviations)
- docx conformance, legal-grounding, and LAB audit worktree updates