cortex: exact staleness, flags before fetches, and stop OCR-ing what Cortex already read
Three changes to the materialize path, all of which stop guessing at something the corpus can now answer directly. STALENESS ASKS THE FLAGS FIRST (§W3.1(4)). checkCortexStaleness used to fetch every row it was given. Now: a stamped cortex_stale is the answer; a copy the feed has been watching continuously since it was taken (bootstrap older than the copy, cursor not behind, last run ok) is answered "not stale" from the absence of an event; and only what neither covers costs an /api/fetch - whose verdict is then stamped, so the same document is free next time. The 25-cap now bounds fetches rather than coverage, and the route's own ceiling goes 50 → 200 with project access memoized, because a matter's documents share one project and asking 200 times is how a raised ceiling becomes a slow page. EXACT COMPARISON WHEN THERE IS ONE (§W3.2). materialize now records sha256 of the bytes it stored in metadata.cortex_blob_sha256 - computed locally, not copied from the response, so it describes what Syd HAS. When /api/fetch starts returning blob_sha256 (v8 §W1.2) the comparison is two shas and nothing else; until then the whole existing heuristic chain still decides, unchanged. The chain gained one guard on the way past. blob_sha256 was already in its hash bucket, which meant the day Cortex started returning it, every copy anchored on a blob PATH would have compared a sha against a path and reported the entire pre-W1 library stale in one page load. It now carries its own fingerprint_kind and comparison requires matching kinds: mismatched kinds are "we could not tell", which is the answer this code has always distinguished from "it is current". NO SECOND OCR (§W3.3). A materialized scan was transcribed twice - once by Cortex on ingest, once again here by Claude vision the moment embedDocument reported no extractable text. /api/fetch already returned the transcription in the same response we downloaded the bytes with. It is now persisted to ocr_text in the statement that marks the document ready, before the embed runs, so the embedder finds text and the vision tail never fires. Only when it is worth trusting: not for docx/xlsx (tracked changes and grid fidelity need the local parse), not when text_truncated (a cut transcription embedded as the whole document is a search index that omits the end of every long contract), and not for the ~12% of the corpus still queued for OCR upstream, where Syd's own pass is the only thing that will produce text. Bytes still land in R2 unchanged. Refresh keeps all three honest: it re-imports or clears the imported text for the new bytes, re-pins the sha, clears the stale flags it just answered - and keeps cortex_superseded, because pulling the same doc_id's newer bytes does not make a document Cortex ruled superseded operative again. It also now merges into existing metadata instead of replacing the column, which had been quietly dropping cortex_filed_id, cortex_sync and cortex_backfill on every refresh. The UI reads the flags straight off the row it already has: a distinct "superseded in Cortex" badge, and "newer in Cortex" the moment the page loads rather than after a round trip - with the reason in the tooltip, including reasons this build has never heard of. Staleness badging stays gated on rows with a real corpus anchor: a Syd upload that was pushed to Cortex is not "behind" anything. Build Plan v8 §W3.1(4), §W3.2, §W3.3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| Repository | emileriksenkeev/sydOS |
|---|---|
| Author | emileriksenkeev <emil@keev.co> |
| Authored | |
| Parents | 92fa89a7 |
| Stats | 5 files changed , +144 , -7 |
| Part of | Cortex - company-wide document corpus integration |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-8784f1e3.md
from inside the repo you want the change in.