cortex: Syd pulls too - the changes feed, and a cursor that survives a restart
Syd has pushed to the corpus since §5.S3 and never pulled. The only way a
working copy learned that Cortex had moved on was somebody opening the project
page: /cortex/staleness, one /api/fetch per row, capped at 25. A matter nobody
opened was never checked, and a reclassification or a supersede edge is not
something a fingerprint diff can see at all.
/api/changes is a cursor over everything Cortex touched. This adds the resident
consumer (backend runs for days on the droplet, so setInterval, same shape as
the 15-minute pass-through sweep it sits beside), bootstrapping at cursor='now'
because the feed is not a backfill tool and the back catalogue is already
reconciled.
What it does with an event is deliberately almost nothing: it stamps
metadata.cortex_stale / cortex_stale_reason / cortex_stale_at on local documents
matching metadata.cortex_id, plus cortex_superseded when Cortex says so. No
auto-refresh - that mints a version row and re-embeds, and doing it underneath a
lawyer mid-review is worse than being a day behind. The flag is the signal; the
existing "Refresh from Cortex" button is the remedy.
Three rules carry the correctness:
- the cursor advances only past changes that actually finished. A failed page,
a failed write, or the per-run page cap leaves it on the last completed item
and the next tick resumes there. Re-reading is free (stamping is idempotent
and a repeat event is a no-op write); skipping is unrecoverable.
- an unrecognized reason is still news. The pinned v8 consumer rule: never
switch on an exhaustive list, so anything Cortex invents later is flagged
with its reason recorded verbatim and logged once as new vocabulary.
- a doc_id with no local match is ignored. Syd tracks working copies, not the
corpus.
The cursor lives in a new sync_state(key, value jsonb) table - generic on
purpose, so the next resident consumer costs a row rather than a migration -
with expression indexes on the documents.metadata keys the feed and the health
counts now query every ten minutes.
/cortex/status grows the other half of the picture: cursor age, last run, the
sweep's backlog and the stale-flag counts, with WARN lines in the journal past
3h of cursor age or 25 queued documents. A corpus link that has silently stopped
looks exactly like a corpus where nothing changed, which is the whole reason
this endpoint exists.
Also documents CORTEX_API_URL/KEY, the CF Access pair and both interval vars in
.env.example, which have been load-bearing and undocumented since the bridge
landed.
Build Plan v8 §W3.1, §W3.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| Repository | emileriksenkeev/sydOS |
|---|---|
| Author | emileriksenkeev <emil@keev.co> |
| Authored | |
| Parents | 6a7bbacb |
| Stats | 6 files changed , +737 , -5 |
| Part of | Cortex - company-wide document corpus integration |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-92fa89a7.md
from inside the repo you want the change in.