feat: add resilient direct upload sessions

⛔ closed · #397 · open-legal-products/mike ← open-legal-products/mike · opened 12d ago by willchen96 · closed 7d ago · self · +10,848-2,368 across 93 files · ↗ on GitHub

From the PR description

Summary

  • Add direct-to-object-storage upload sessions for the web app and Word add-in.
  • Process each file independently as soon as its upload completes, while allowing multiple concurrent sessions.
  • Improve document-table selection, actions, optimistic state, upload placeholders, and context-aware default sorting.
  • Bound memory and fan-out for document processing and folder ZIP downloads.

What changed

Direct upload architecture

  • Replaced multipart API uploads with signed R2/S3 PUT URLs and upload-session manifests.
  • Added per-file completion, independent processing jobs, lease-based worker claims, retries, and partial-success reporting.
  • Removed the redundant bulk session-completion path so clients use one per-file protocol.
  • Added resilient status polling with backoff and transient-error retries.
  • Made mutation, polling, and hourly session limits configurable with safe defaults.
  • Persisted uploader email on sessions so worker-generated audit events retain attribution.
  • Added browser upload support to both the web app and Word add-in.

Document tables and actions

  • Unified file and folder selection, including Shift-range selection, Cmd/Ctrl additive selection, and select-all.
  • Applied context-menu actions to the full current selection and added relevant View/Open actions.
  • Added mixed file/folder download and delete behavior with optimistic row removal.
  • Persisted upload placeholder rows across table remounts and animated the Uploading. / Uploading.. / Uploading... state.
  • Kept library files and templates sorted by last updated by default, while project-root contents default to name sorting.
  • Updated workflow actions so reference files can be viewed and Edit routes to the workflow editor.

Memory and reliability hardening

  • Streamed ZIP exports from object storage through the backend instead of buffering all source files and the complete archive.
  • Added resolved-document and total-byte caps, bounded metadata/download work, and collision-safe archive names.
  • Streamed sealed upload objects to temporary files for document processing to avoid holding whole files and duplicate copies in memory.
  • Improved per-file client error collection so failures surface immediately instead of becoming delayed upload_incomplete errors.
  • Added deterministic cleanup ownership and storage error handling.

Database, CI, and deployment

  • Consolidated the branch's upload-session SQL into a single pre-merge migration and synchronized backend/schema.sql.
  • Added MinIO CORS configuration and verification to browser-upload CI and local-stack setup.
  • Updated deployment documentation and environment examples for upload-session limits and browser-reachable storage endpoints.
  • Updated the legacy upload regression test to cover both the new session validation path and the retired endpoint's 410 response.

Why

Large uploads previously passed through the API and delayed document creation until an entire batch finished. This change sends bytes directly to object storage, lets completed files enter processing immediately, supports overlapping upload batches, and avoids request-sized memory spikes in processing and ZIP export paths. The table changes make mixed file/folder operations consistent with familiar desktop selection behavior.

Testing

  • npm test --prefix backend - 76 files passed, 4 skipped; 896 tests passed, 26 skipped.
  • npm test --prefix frontend - 104 files passed; 667 tests passed.
  • npm run build --prefix backend
  • npm run build --prefix frontend
  • npm run typecheck --prefix word-addin
  • Targeted upload-session, storage, processing, selection, and ZIP-export tests.
  • git diff --check

Full Playwright browser E2E was not run locally; the CI storage CORS setup is included for that suite.

Our analysis

Add direct object-storage upload sessions — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-397.md from inside the repo you want the changes in.

⬇ Download capture-pull-397.md