[Architecture] feat: default-on durable DB job queue (Postgres) - audit trails, account erasure, storage cleanup, async exports

From the PR description

Stacked on #294 (durable BullMQ queues) - review the last 3 commits (0b8d34c0, da07f581, 6bf3135f); the earlier ones are #294's and will disappear from the diff when it merges.

Design (ADR)

Context. #294 made the two heaviest workloads (conversion, extraction) durable - but opt-in, because they need Redis. A whole-tree survey found a second class of workloads that must be durable in every deployment, not just ones that opted in: audit trails (fire-and-forget inserts racing process teardown after SSE [DONE]), account erasure (an unbounded inline cascade that a restart leaves half-done), storage cleanup (every deleteFile(...).catch(() => {}) in the tree silently leaks bytes on a hiccup), and export builds (whole account serialized inside a GET request).

Decision. A second, default-on queue backed by the one dependency every deployment already has - Postgres:

• db_jobs table + claim_db_jobs() RPC built on FOR UPDATE SKIP LOCKED, the standard Postgres work-queue idiom: any number of backend replicas partition the work race-free, no coordinator, no new infrastructure. The Word add-in stack and the Mac app's Docker stack run the same backend + Postgres, so the queue is functional there with zero configuration - which is exactly why it can default on where the Redis queues cannot.
• State machine: pending → running → done, failure → pending with exponential backoff (30s/90s/270s... capped 30min), attempts exhausted → failed (kept 30 days for inspection). Attempts increment at claim time so a crash-looping job cannot retry forever. Crash recovery is folded into the claim itself: a running job whose claim is older than 10 minutes was orphaned by a dead worker and gets re-claimed.
• Optional dedupe_key under a partial unique index over live jobs only: double submits collapse race-free; a finished job frees the key.
• An in-process runner polls every 5s (one cheap partial-index query), runs handlers with at-least-once semantics (throw = retry), and sweeps retention hourly. DB_JOBS_ENABLED=false is an operational escape hatch, not a feature flag.

Why not reuse the BullMQ queues: default-on + Redis-required contradict; these workloads are low-throughput and latency-tolerant, exactly what a polled DB queue is for. Why not transactions (for deletion): the cascade spans Postgres and object storage, which cannot commit atomically - the job supplies what a transaction can't: cross-system at-least-once completion.

What rides the queue

Plus one boot-time fix in the same spirit: the workflow add-on catalog syncs at server boot instead of inside the first unlucky GET /workflow-addons after a deploy (lazy latch kept as fallback).

Replication

Setup: local stack (backend + Supabase). Apply the migration (backend/migrations/20260821_01_db_jobs.sql) or bootstrap fresh from schema.sql.

Base case (on main / #294):

1. Audit loss: send a chat message and restart the backend the instant [DONE] renders (or point the DB away for a moment) - the turn (and its generated-document rows) never appear in History, with nothing logged but a swallowed console.error.
2. Deletion: watch the backend logs while deleting a project whose documents live in storage; kill storage (or the process) mid-delete - DB rows gone, objects still in the bucket, forever. Delete an account and kill the backend mid-cascade - the account still logs in with partial data.
3. Exports: on a large account, Settings → Privacy & Data → Export waits the whole build inside one request; close the tab and the work is discarded; click twice and it builds twice.

On this branch:

1. Send a chat message → one audit.chat_turn row appears in db_jobs and flips to done within ~5s; the audit rows appear even if you restart the backend in between (the pending job is picked up at next boot - the smoke's stale-reclaim path).
2. Delete a project → rows vanish immediately, a storage.cleanup job drains the objects; stop storage first and watch the job retry with backoff until storage returns. Delete an account → login is dead instantly, and the cascade completes across restarts (account.delete job).
3. Export buttons now schedule a job (202), poll every 2s, then download the artifact; double-click → one build (deduped); the artifact 404s after 24h (retention sweep).

Tradeoffs & design decisions (flagged)

• Default-on is the point of this PR - but it degrades safely: if the migration hasn't been applied yet, the runner logs the failed claim each tick and does nothing else; enqueue failures fall back to the old inline behavior at every call site (chat audit → direct inserts; storage cleanup → best-effort inline deletes; account delete → inline cascade). No path is worse than main under any failure.
• At-least-once, not exactly-once: a retry after a partial audit fan-out can duplicate an audit row. Chosen deliberately - for an audit trail a rare duplicate beats a silent gap.
• Polling (5s), not LISTEN/NOTIFY: supabase-js has no NOTIFY channel; a 5s indexed poll is negligible and works over PostgREST everywhere. Latency-sensitive work does not belong on this queue.
• Account deletion reverses ordering (auth first, data second): the user-visible contract improves (account gone instantly, retriable if auth delete fails), at the cost that a data-cascade failure now has no user to retry it - which is exactly what the 20-attempt durable job is for.
• Export downloads get their own endpoint rather than the signed /download/:token route: that route only serves paths backed by a live document_versions row (and would 404 on artifacts); the new endpoint re-checks ownership + MFA on every request.
• Audit CSV and document-ZIP stay synchronous: bounded (filter limit / user selection) with immediate-download UX; queueing them would add polling friction to the common small case. Follow-up candidates.
• Single audit events stay direct inserts: enqueuing them would double every audit write for no durability gain (the enqueue is an insert into the same database). Only the multi-row post-[DONE] fan-out gains from the queue, so only it rides it.
• Legacy sync export routes kept for curl users and older clients; the frontend alone switches to the async flow.

Testing

• Backend: tsc clean, 786 tests (+26 on this branch: state machine, backoff, batch isolation, retention ordering, dedupe-as-success, storage-cleanup fallback, all four handlers). Frontend: tsc clean, 482 tests (+2: schedule→poll→download wiring, failed-build surfacing), production build green.
• Live against real Postgres/PostgREST (local Supabase, migration applied): enqueue→claim→done with result readback; dedupe unique-violation; retry rescheduled with future run_at and not re-claimed early; stale-running re-claim; and two concurrent claims over 10 jobs partitioning 5/5 with zero overlap - FOR UPDATE SKIP LOCKED proven, not assumed.
• Schema drift: migration and schema.sql DDL verified structurally identical (comments aside); the drift workflow builds both install paths in CI.

Maintenance

Same commitment as the other rows: triage within 48h on anything this breaks; the runner can be disabled operationally with DB_JOBS_ENABLED=false and the PR reverts cleanly (the migration is additive).

🤖 Generated with Claude Code