[Architecture] feat: durable job queues - BullMQ-primary with Postgres fallback, worker threads, async-on for new installs

From the PR description

[!TIP] Reviewer & testing companion: Durable Queues Field Guide - the mechanism in one diagram, the 27 commits in a five-act reading order, a risk-ordered review checklist, the judgment calls that need a maintainer, and copy-paste runbooks for both transports (~10 min each). This description remains the authoritative record; the guide is the map of it.

Row amal66#40 of the reference index (#205), grown into the app's full background-job architecture. Supersedes and absorbs #369.

Stack mechanics (read this first)

This is the bottom of a three-PR architecture stack. Each PR is based on the one below it, so each diff shows only its own commits:

• The two intermediate base branches (olp-pr/durable-queues, olp-pr/service-layers) exist in this repo purely as mirrors of the identical branches on the amal66/mike fork, so that GitHub can compute a clean stacked diff. They are force-pushed together on every rebase - origin and fork tips are kept byte-identical (both currently f59aee93 / b82bbbed). Nothing else should branch off them.
• When this PR merges into main, #295 should be retargeted to main (its diff is already only its own 2 commits, so nothing changes about what it contains), and so on down the stack. GitHub will do the retarget automatically for #295 if olp-pr/durable-queues is deleted on merge - please don't delete it until #295 has been retargeted, or #356's base disappears with it.
• Rebasing the stack means: rebase olp-pr/durable-queues onto main, then rebase olp-pr/service-layers onto it, then olp-pr/code-quality, then force-push all three to the fork and the two mirror branches to origin. Head tip today: f59aee93, rebased onto main @ 54681b55.

Design (ADR)

Context. Two kinds of workloads need to leave the request thread: user-attended heavy work (DOCX→PDF conversion, tabular extraction) and work that must be durable in every deployment (audit trails, account erasure, storage cleanup, export builds, token refresh). The first kind benefits from instant pickup and live progress; the second kind must not depend on infrastructure a default deployment doesn't have.

Decision: one queue contract, two interchangeable transports.

• Redis configured → BullMQ. Conversion/extraction run on their native BullMQ queues (instant pickup, live cell-progress over pub/sub), and the registry jobs get instant delivery via a transactional outbox: the db_jobs Postgres row stays the single durable record; enqueue also hands the row id to an app-jobs BullMQ queue; the worker claims the row through a conditional claim_db_job() RPC, so a duplicated delivery matches zero rows and a lost delivery is recovered by the poller (which drops to a 60s backstop). BullMQ is never a second source of truth.
• No Redis → the Postgres queue carries everything. Same job bodies, same dedupe identities (the BullMQ jobId doubles as the DB dedupe key), same retry budgets; claim_db_jobs() uses FOR UPDATE SKIP LOCKED so claims partition work safely across replicas; crash recovery is folded into the claim (rows stuck running past p_stale_seconds, default 600s, get re-claimed - there is no separate reaper). cancel_db_jobs(text[]) gives the Postgres transport the cross-process cancellation that BullMQ gets natively. Live progress degrades to the SSE views' existing DB-poll backstops (5s poll vs the 60s Redis-mode backstop).
• Driver resolution (backend/src/lib/dbq/driver.ts): explicit QUEUE_DRIVER=redis|postgres wins; else REDIS_URL set → redis; else a legacy ASYNC_DOCUMENT_CONVERSION/ASYNC_TABULAR_EXTRACTION flag on → redis (those flags always meant BullMQ); else postgres. (QUEUE_DRIVER=auto is documentation shorthand for "unset" - any unrecognized value falls through the same ladder.)

Where workers run (WORKERS_MODE, backend/src/index.ts): thread (default) - a worker_thread inside the API process, off the HTTP event loop, respawned on crash; inline - the historical single-thread mode; none - a standalone worker process (node dist/worker.js, source backend/src/worker.ts) on the same Postgres/Redis: a separate container or machine, N instances safe by construction. Graceful shutdown is coordinated in every mode (SIGTERM → drain → 15s force-exit).

New installs vs existing installs. Code defaults never flipped - flipping them would break every deployment that upgrades in place (only forks pinned to old commits would be safe). Instead the bootstrap artifacts changed: docker-compose.yml ships a Redis service (redis:7-alpine, --appendonly yes AOF, loopback-only) with ASYNC_* on at the compose level, so fresh docker compose up installs run BullMQ for everything, while an existing bare-metal deployment that pulls this branch sees byte-identical behavior until it opts in.

Alternatives considered. Always-Redis (rejected: breaks the £20-VPS deployment and every in-place upgrade); Postgres-only (rejected: loses instant pickup and live progress where Redis exists); LISTEN/NOTIFY instead of polling for the fallback (not reachable through supabase-js/PostgREST); separate worker deployment as the default (rejected: single-box stays single-box; the seam exists for those who want it).

What runs on the queue now

Eight durable job kinds (backend/src/lib/dbq/handlers.ts): conversion.convert, extraction.extract, audit.chat_turn, account.delete, storage.cleanup, export.build, mcp.refresh_token, document.precompute_text - plus the app-jobs BullMQ delivery queue and the two native conversion/extraction queues.

Also in this diff, and worth a reviewer's attention beyond the queue itself:

• routes/tabular.ts is decomposed into backend/src/lib/tabular/{extract,extractRow,generate,generateStream,prompt,rows,shared}.ts (~1,540 new lines, net -701 in the route) so the SSE route and the async worker provably share one extraction core. #295 later moves this whole directory into modules/tabular/.
• Stale-work reaper (backend/src/lib/maintenance/staleWork.ts): documents stuck processing and cells stuck generating flip to error (30 min stale / 10 min sweep, first sweep 30s after boot, bounded limit 500).
• New shared infra: lib/queue/runProgress.ts (Redis pub/sub cell progress), workers/registry.ts (driver-gated worker registry), workerThread.ts, workerRuntime.ts, lib/sseHeartbeat.ts, lib/auditExport.ts, lib/pdfjs.ts.
• Frontend: frontend/src/app/lib/asyncExport.ts (schedule → poll → download, 2s cadence, 150-poll ceiling) wired into the history page, DocTable, TabularReviewView, and privacy/data settings.
• Two new backend dependencies: bullmq ^5.34.0, ioredis ^5.11.1. One new migration: backend/migrations/20260824_01_db_jobs.sql, mirrored into backend/schema.sql.

Base-case replication - see the problems on main

All of these are on pristine origin/main; none needs this branch.

1. Synchronous conversion blocks the upload request (root cause of #8). git grep -n "await docxToPdf" origin/main -- backend/src/routes/documents.ts → the upload handler awaits LibreOffice inline (lines 650, 864). Upload a large DOCX from the web app with the Network tab open: the POST /documents request stays pending for the whole LibreOffice run. Behind Cloudflare, a bulk upload of these hits the 100s edge timeout and surfaces as a phantom CORS error - that is issue #8.
2. Audit rows are fire-and-forget. git grep -n "void recordChatTurn" origin/main → routes/chat.ts:686,702, routes/projectChat.ts:357. Run a chat turn and docker kill the backend the instant the stream emits [DONE]: the turn happened, no audit_events row exists, and nothing ever retries. On this branch the same kill leaves a db_jobs row that completes after restart.
3. Storage deletes are swallowed. git show origin/main:backend/src/lib/userDataCleanup.ts | sed -n 115p → deleteFile(path).catch(() => {}). Stop MinIO/S3, delete a document from the UI: the row disappears, the object leaks permanently, and nothing is recorded.
4. Account erasure is an inline cascade. Delete an account and kill the API mid-cascade → a half-erased user with no resumption path.
5. Tabular extraction dies with the tab. Start a review generation on main and close the tab → the run stops; there is no server-side job to resume.
6. Word add-in chat turns produce no audit trail at all on main (recordChatTurn has no wordChat.ts caller).
7. Exports are built inside the request - a large account/documents export on main holds one HTTP connection for its entire build.

PR replication - see it working

A. Fresh install, Redis transport. docker compose up --build with fresh volumes.

• Boot logs must show workers: thread, three BullMQ workers started in the worker thread, and [dbq] runner (poll 60000ms, driver redis).
• Upload a DOCX → POST returns 201 instantly; [conversion-worker] converted follows; PDF rendition lands in document_versions.
• Run a tabular review → live SSE cell updates; job ids are extract_<reviewId>_<rowId>[_<col>] (underscores, no colons - see the tradeoff note below).
• Kill the browser tab mid-run, reopen → the grid is complete; the workers never noticed.
• Run a chat turn → select kind, status from db_jobs order by created_at desc limit 1 shows audit.chat_turn flipping to done sub-second (outbox delivery, not the 60s poll).
• docker stop MinIO, delete a document → the delete still succeeds (rows first), the storage.cleanup job fails and re-queues with backoff; restart MinIO → the job reaches done on a later attempt.

B. Postgres transport, no Redis at all. QUEUE_DRIVER=postgres (or simply unset REDIS_URL and the ASYNC_* flags).

• Boot logs: driver postgres, poll 5000ms, no BullMQ workers, no Redis dial.
• Start a review generation, docker kill the backend mid-run → cells stranded generating, jobs stranded running. Restart: pending jobs are claimed within ~5s; the stranded running rows are re-claimed once claimed_at is older than 600s (to test without waiting, age it: update db_jobs set claimed_at = claimed_at - interval '11 minutes' where status = 'running';). All cells reach done.
• Same jobs, same dedupe keys, same retry budgets as (A) - only progress granularity changes.

C. Upgrade-in-place invariance (the property that matters most). On a deployment with no REDIS_URL and no ASYNC_* flags, boot the backend: no Redis dial (verify with an unroutable REDIS_URL - every module still imports cleanly), conversion/extraction stay inline exactly as on main, and the Postgres queue quietly carries audit/deletes/exports. The only behavioral deltas vs main are the durability fixes.

D. Worker placement. Default boot logs workers: thread + [worker-thread] background workers started. WORKERS_MODE=none on the API plus node dist/worker.js in a second container (compose has a commented worker service) moves the same worker set out of process. SIGTERM in every mode ends with Shutdown complete.

Tradeoffs & design decisions (flagged)

• At-least-once everywhere: a retry after a partial audit fan-out can duplicate a row - for an audit trail, a rare duplicate beats a silent gap. Alternative rejected: exactly-once via a dedupe table per artifact - more schema and more failure modes than the problem justifies.
• Delivery jobs never retry (attempts: 1) - the durable row + poller are the retry mechanism; two retry systems fighting is how jobs run twice.
• Fallback-mode progress is poll-granular (5s) - accepted; the alternative was LISTEN/NOTIFY, unreachable through PostgREST/supabase-js.
• Compose-level defaults, not code defaults - deliberate; see ADR. Alternative rejected: flipping the code defaults, which silently changes behavior for every in-place upgrade.
• Job ids use _, never : - BullMQ rejects most colon-containing custom jobIds (all but a legacy 3-segment form). This is a real constraint the schema now encodes, not a style choice; see the live-verification note below.
• MFA asymmetry on bulk downloads: DocTable's >10-doc ZIP path now runs under requireMfaIfEnrolled (the export routes' gate) while the small sync ZIP only requires auth. Flagged, not hidden; unifying either way is a small follow-up.
• Local-storage-mode Word turns now write one audit row - metadata only (chat.message, surface: "word"); the title is the chat/document title and never the prompt, holding the no-server-side-conversation contract. Easy to gate on persistChat if the maintainers prefer zero rows - say the word and I'll add the gate here rather than in a follow-up. This is the one deliberate privacy judgement call in the PR, so it deserves a maintainer's opinion.
• Token-refresh failure classification is deliberately pessimistic: any 4xx from the token endpoint is treated as a dead grant (RFC 6749 puts invalid_grant at 400). Wrong toward "permanent" costs one skipped background refresh; wrong the other way replays a dead grant forever. A 24h expired-age floor stops abandoned connectors from enqueueing doomed jobs indefinitely.
• Zip export builds sequentially - this path exists for selections big enough that concurrency is the memory problem. Capped at 500 documents.
• Precompute cache keyed on versionId - sound today (legacy .doc never gains assistant-edit versions, which are DOCX); flagged as the one soft spot in cache identity.
• The frontend Docker fix rides along (commit b0e52648) - it is not queue work. docker compose up is broken on current main (a #368 regression): the frontend type-imports from backend/src, which the frontend-only Docker context doesn't contain, so next build's type check fails inside the image while host builds and CI pass. Fixed here by building the frontend image from the repo root with a Dockerfile.dockerignore. Called out explicitly because it is an unrelated fix in this diff - happy to split it out if you'd rather take it separately.
• Prior rounds' tradeoffs still apply: permanent-failure cleanup only errors claimed cells; clear-cells cancellation is best-effort atop guarded terminal writes; chat-generated documents never flip error on rendition failure; edit_document/generate_docx deliberately get no renditions; flag-on PPTX has a first-open gap until its rendition lands.

Testing evidence

Automated. Backend tsc clean, 80 test files / ~820 cases. Frontend tsc clean, 97 test files / ~520 cases, production build green. (Case counts include it.each expansion; file counts are exact - main has 64 backend / 95 frontend test files, so this PR adds 16 backend and 2 frontend test files.) All CI checks are green on f59aee93 - CodeQL, gitleaks, backend, frontend, playwright, Supabase stack tests, eval harness, and the "Fresh install vs upgraded deployment" schema-drift job that builds both install paths.

Live cross-transport test round - posted publicly as comment 5400174994. That comment is the primary evidence for this PR and is worth reading in full: a fresh docker compose up --build (fresh volumes), Chrome driven over the DevTools protocol, a real Anthropic key, with Postgres and container logs cross-checked at every step. It covers 11 Redis-transport scenarios (A1-A11: fresh-boot contract, queued conversion, tabular run with live SSE, kill-tab durability, clear-cells + queued regenerate-cell, sub-second outbox audit delivery, async CSV/ZIP/JSON exports, MinIO-stopped fault injection with recovery on restart, full account erasure) and 2 Postgres-fallback scenarios (B1 crash-recovery with stale-claim reclaim, B2 upgrade-in-place invariance), plus main-regression spot checks for the features that landed during the rebase window (#365 onboarding, #374 tabular UX, #376/#380 workflow catalog, #335 dark mode). It also lists what was not covered live and why (Word-host audit, MCP grant refresh, mid-flight stream reattach).

Earlier rounds, still valid. Live claim-RPC semantics on real Supabase Postgres + Redis (dedupe unique-violation, backoff-not-reclaimed-early, stale-running reclaim, two concurrent claimers partitioning 10 jobs 5/5 with zero overlap); outbox end-to-end incl. duplicate-delivery no-op; BullMQ cancellation semantics (the smoke that caught Job#discard() being an in-memory no-op cross-process); boot smokes for all three WORKERS_MODE homes in dev-tsx and compiled prod, both drivers, clean SIGTERM. Gated Supabase stack tests 15/15 against real GoTrue + RLS. The Word add-in Playwright suite (22 spec files, chromium + webkit) was run as a regression check and is green - this PR changes zero files under word-addin/; its Word work is entirely server-side in backend/src/routes/wordChat.ts.

Two real defects were found by hand-testing and fixed on the branch - both invisible to every unit test:

1. BullMQ rejects most colon-containing custom jobIds (commit 52e5a212). Our convert:<id> and 4-segment regenerate ids threw at the first real upload, while extract:<a>:<b> had sailed through the smokes inside BullMQ's legacy 3-segment carve-out. All custom ids now use underscores. (The live comment cites this as b71a56c1 - that is the pre-rebase hash of the same commit; on the current branch it is 52e5a212.)
2. docker compose up broken on main (commit b0e52648) - see the tradeoff note above.

CodeQL. The three highs CodeQL raised on this branch are fixed in 5264b007 with a pinning regression test (backend/src/lib/tabular/__tests__/tabular.extract.sanitize.test.ts): tag-stripping now loops to a fixed point (<scr<script>ipt> reassembly), entity decoding does & last (double-unescaping), and a user-controlled filename moved out of console.error's format-string position. f59aee93 then raised the frontend coverage floor that the new tests made stale (statements 99 → 100; branches stays 97 at a measured 97.66%).

Provenance

Re-derived from amal66#40 against current row-based main (see earlier revisions of this description for the port details); the fork's embedding queue still rides with the RAG row. The Postgres queue, outbox, worker-thread placement, and full-app coverage are new in this PR. The Mac desktop shell's compose (separate branch) inherits the Redis service when it rebases.

Related issues

Addresses the root cause of #8 (bulk uploads → Cloudflare 524 timeouts surfacing as phantom CORS errors): synchronous LibreOffice conversion inside the upload request leaves this PR's queue, so upload responses return immediately. Note the rollout caveat for existing deployments - async conversion is on by default only for fresh compose installs; an in-place upgrade must set ASYNC_DOCUMENT_CONVERSION=true (with Redis, or QUEUE_DRIVER=postgres). The client-side half of #8 (unbounded parallel uploads, per-file outcomes) is fixed separately in #381.

Maintenance

Same commitment as the other index rows: triage within 48h on anything this breaks. Escape hatches at every layer: QUEUE_DRIVER=postgres, DB_JOBS_ENABLED=false, WORKERS_MODE=inline, and the whole feature set degrades to main's behavior when the relevant enqueue can't reach its backend.

🤖 Generated with Claude Code

Review fix round - 2026-08-27

Nine findings from a review of 8f134f9a were re-verified against the code before anything was changed. Eight reproduced and are fixed; one did not reproduce and is deliberately left alone (see #8 below). Every fix ships with a regression test that was proven failing against the pre-fix code (git stash, run red, restore, run green).

Verification

npx tsc --noEmit clean; npx vitest run in backend/ - 1007 passed, 31 skipped, 0 failed (was 996 before this round). npm run build --prefix backend clean. git diff --check clean. No frontend files touched. SQL verified against a throwaway local Supabase stack on its own ports (never the compose database), including the fresh-schema.sql vs baseline + migrations fingerprint comparison the drift workflow performs - no drift.

Left for the maintainer (deliberately not changed)

• compose ASYNC_* defaults are still true and the Redis port is still published - flagged as policy decisions in review, so they are untouched here.
• Merge-order constraint with the organisations stack (finding 8): whichever of the two lands second, accessibleProjectIds must end up on the orgs stack's grant lookup, not on shared_with. The comment in lib/auditExport.ts is the only thing standing in a resolver's way.
• DB_JOBS_ENABLED=false now changes producer behaviour, not just consumer behaviour (inline delete / 503 export). If a split topology is ever meant to express "workers live elsewhere", that is WORKERS_MODE=none, not this flag - worth a line in docs/deployment.md if you disagree with the reading.

Deploy-order note - 2026-08-27

The maintainer decided the landing order: this PR (and #295 stacked on it) merges and deploys first; the organizations stack (#267/#268/#363) lands after. The org stack's migrations were renumbered to 20260831_01..05 accordingly, so this PR's 20260829_01_db_jobs.sql watermark no longer skips them. The combined sequence - main schema → 20260829_01 → 20260831_01..05 - was verified end to end on a real local Postgres (applied twice; all objects disjoint; org-stack integration suites 22/22 against the combined database). A follow-up commit (e18c30ec) also updates the DB_JOBS_ENABLED documentation to match the fix round's producer-side semantics.