cortex: reconcile the back catalogue, sha first

↗ view on GitHub · emileriksenkeev · 2026-08-06 · 807ab3eb

The integration only ever looked forward. 1,481 documents and 2.0 GB were
already sitting in Syd when it shipped, and a document that exists in exactly
one system is a document the company can lose.

backfillCortexSync hashes each document's bytes on the droplet, asks
/api/ingest the cheap question first (JSON sha precheck: a hit answers with the
doc_id and no byte moves, and costs no rate-limit token), and posts the bytes
only for the misses. It stamps the Syd row with the Cortex id, the sha and the
corpus fingerprint, so the retry sweep will not re-push it and the phase-2
staleness check covers the old library.

Work products are excluded by default and stamped as such rather than skipped
in silence: a generated document with an upload-sourced version in its history
sends the source material instead, and one without is left for Emil to file
deliberately (§8.4).

Supporting changes: cortexCall now surfaces Cortex's Retry-After on 429 (and
stops burning retries on a wait it cannot fit in its budget) so a bulk caller
can honour it; ingestToCortex takes a `tool` override; checkCortexStaleness
admits backfilled rows, whose source_system stays syd_os because Syd made them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repository emileriksenkeev/sydOS
Author emileriksenkeev <emil@keev.co>
Authored
Parents f67a71f8
Stats 4 files changed , +943 , -5
Part of Cortex - company-wide document corpus integration

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-807ab3eb.md from inside the repo you want the change in.

⬇ Download capture-commit-807ab3eb.md