feat: direct upload sessions rebased onto durable job queues (#397 + #400 on #294)

✅ merged · #409 · open-legal-products/mike ← open-legal-products/mike · opened 7d ago by amal66 · merged 7d ago by willchen96 · self · +12,774-2,605 across 101 files · ↗ on GitHub

From the PR description

Summary

This branch is #397 (feat/direct-upload-sessions) plus its review-fix stack #400, rebased commit-by-commit onto #294 (olp-pr/durable-queues). Neither #397 nor #400 was modified - all 19 of their commits are replayed here on top of the queues work, with conflicts resolved so the two features integrate rather than collide. Targets olp-pr/durable-queues, so the diff below is exactly the upload-sessions work as it lands in the durable-queues world.

What changed relative to the original PRs

Everything in #397 and #400 is carried over verbatim where the two branches didn't touch, plus one integration commit. The deliberate integration decisions:

  • Upload workers run inside the durable-queues worker runtime. #397 started startUploadProcessingWorkers() directly in index.ts at listen time. Here the pool is registered in workerRuntime.ts (startAllWorkers/stopAllWorkers) instead, so upload processing follows WORKERS_MODE like every other background worker: it runs in the worker thread by default, inline with WORKERS_MODE=inline, and in the standalone worker process (node dist/worker.js) with WORKERS_MODE=none - including graceful shutdown. index.ts is untouched relative to #294 (the only file-set difference vs #397+#400: index.ts out, workerRuntime.ts in).
  • Legacy multipart endpoints are retired as in #397, including the versions of them that #294 had extended with deferred conversion (POST /single-documents, POST/PUT .../versions*, workflow reference-file POST/PUT, both handleDocumentUpload copies). #294's single-document poll route (GET /single-documents/:documentId) is kept - the deferred-conversion UI needs it.
  • DocTable download keeps both features: #397's mixed file/folder selection download, with #294's >10-document async ZIP export preserved for doc-only selections (folder selections stay on the direct streaming path so the archive keeps its folder structure).
  • schema.sql / grants: union of both branches' function grants; create_upload_session carries #397's final 7-arg signature.
  • Lockfiles: package-lock.json regenerated from the merged package.json (bullmq/ioredis kept, multer dropped - verified the diff is exactly the multer removal); bun.lock resolved the same way.
  • New integration commit (fix(compose)): mounts the consolidated 20260828_02_upload_sessions.sql migration into the compose db-init service - #294's composeMigrations test requires every migration to replay there, a rule that postdates #400's consolidation. It applies in filename order, before 20260829_01_db_jobs.sql.

Known interaction (no code change)

#294's stale-work sweep flips documents stuck in processing for >30 min (STALE_DOC_PROCESSING_MS) unless a live conversion-queue job exists; it does not know about upload-processing jobs. A slow upload job past that age can be flipped to error transiently - the job's own finalizing write still lands afterwards, and the sweep remains the owner of last resort for genuinely dead jobs. Flagging for review rather than papering over it here.

Why

#294 and #397 both rework the upload/processing pipeline and conflict heavily (21 shared files). This branch resolves that collision once, in #294's architecture, so the upload-sessions work can land on top of the durable-queues work without touching either original PR.

Testing

  • npm test --prefix backend - 98 files passed, 5 skipped; 1093 tests passed, 33 skipped (includes the compose-migration replay test that caught the missing mount).
  • npm test --prefix frontend - 107 files passed; 712 tests passed.
  • npm run build --prefix backend, npm run build --prefix frontend, npm run typecheck --prefix word-addin - all clean.
  • git diff --check clean.
  • Changed-file set verified identical to #397+#400's (± the index.tsworkerRuntime.ts seam described above).

Browser E2E / live-stack runs were not repeated here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HPZSeEeCkaKj9zGLj5RhHF

Our analysis

Integrate upload sessions with durable queues — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-409.md from inside the repo you want the changes in.

⬇ Download capture-pull-409.md