cortex: exact staleness, flags before fetches, and stop OCR-ing what Cortex already read

↗ view on GitHub · emileriksenkeev · 2026-08-07 · 8784f1e3

Three changes to the materialize path, all of which stop guessing at something
the corpus can now answer directly.

STALENESS ASKS THE FLAGS FIRST (§W3.1(4)). checkCortexStaleness used to fetch
every row it was given. Now: a stamped cortex_stale is the answer; a copy the
feed has been watching continuously since it was taken (bootstrap older than the
copy, cursor not behind, last run ok) is answered "not stale" from the absence
of an event; and only what neither covers costs an /api/fetch - whose verdict is
then stamped, so the same document is free next time. The 25-cap now bounds
fetches rather than coverage, and the route's own ceiling goes 50 → 200 with
project access memoized, because a matter's documents share one project and
asking 200 times is how a raised ceiling becomes a slow page.

EXACT COMPARISON WHEN THERE IS ONE (§W3.2). materialize now records
sha256 of the bytes it stored in metadata.cortex_blob_sha256 - computed locally,
not copied from the response, so it describes what Syd HAS. When /api/fetch
starts returning blob_sha256 (v8 §W1.2) the comparison is two shas and nothing
else; until then the whole existing heuristic chain still decides, unchanged.

The chain gained one guard on the way past. blob_sha256 was already in its hash
bucket, which meant the day Cortex started returning it, every copy anchored on
a blob PATH would have compared a sha against a path and reported the entire
pre-W1 library stale in one page load. It now carries its own fingerprint_kind
and comparison requires matching kinds: mismatched kinds are "we could not tell",
which is the answer this code has always distinguished from "it is current".

NO SECOND OCR (§W3.3). A materialized scan was transcribed twice - once by
Cortex on ingest, once again here by Claude vision the moment embedDocument
reported no extractable text. /api/fetch already returned the transcription in
the same response we downloaded the bytes with. It is now persisted to ocr_text
in the statement that marks the document ready, before the embed runs, so the
embedder finds text and the vision tail never fires. Only when it is worth
trusting: not for docx/xlsx (tracked changes and grid fidelity need the local
parse), not when text_truncated (a cut transcription embedded as the whole
document is a search index that omits the end of every long contract), and not
for the ~12% of the corpus still queued for OCR upstream, where Syd's own pass
is the only thing that will produce text. Bytes still land in R2 unchanged.

Refresh keeps all three honest: it re-imports or clears the imported text for
the new bytes, re-pins the sha, clears the stale flags it just answered - and
keeps cortex_superseded, because pulling the same doc_id's newer bytes does not
make a document Cortex ruled superseded operative again. It also now merges into
existing metadata instead of replacing the column, which had been quietly
dropping cortex_filed_id, cortex_sync and cortex_backfill on every refresh.

The UI reads the flags straight off the row it already has: a distinct
"superseded in Cortex" badge, and "newer in Cortex" the moment the page loads
rather than after a round trip - with the reason in the tooltip, including
reasons this build has never heard of. Staleness badging stays gated on rows
with a real corpus anchor: a Syd upload that was pushed to Cortex is not
"behind" anything.

Build Plan v8 §W3.1(4), §W3.2, §W3.3.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repository emileriksenkeev/sydOS
Author emileriksenkeev <emil@keev.co>
Authored
Parents 92fa89a7
Stats 5 files changed , +144 , -7
Part of Cortex - company-wide document corpus integration

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-8784f1e3.md from inside the repo you want the change in.

⬇ Download capture-commit-8784f1e3.md