[feat-009] Vision perf: parallel render + tiered cache + progress UI

↗ view on GitHub · Nick Whitehouse · 2026-05-06 · 6bf6d52d

Four wins for the vision-mode wait time, ordered by user impact:

1) PROGRESS UI. Backend emits vision_render_start/done SSE events
   around the pdftoppm call. Frontend renders a "Reading <filename>..."
   block (matching the existing DocReadBlock pattern) instead of a
   dead spinner. SSE stream now opens BEFORE the render so the
   placeholder reaches the browser immediately. ~10s+ wait now feels
   intentional rather than broken.

2) PARALLEL RENDER. pdftoppm is CPU-bound; one process handles only
   one page at a time. Split into 4 workers via -f/-l page ranges,
   each writing to its own subdir to avoid filename collisions. 75
   pages went from 28s → 11.7s on bench. Page count discovered via
   pdfinfo before splitting (also from poppler-utils).

3) IN-MEMORY LRU CACHE (visionCache.ts). 5-entry cap (composites are
   1-2MB each, ~30MB per 75-page doc - keeps worst-case ≤150MB
   resident on the 512MB Railway box). Subsequent turns against the
   same doc skip render entirely; sub-millisecond hit. No SSE
   placeholder events on a memory hit so the UI doesn't flicker.

4) R2 PERSISTENT CACHE (visionR2Cache.ts). Sits behind memory cache.
   Single JSON manifest at vision-cache/<base64url(storagePath|p|d)>.json
   contains the array of base64 composites. Survives backend restarts
   and Railway redeploys. Render → memory write → fire-and-forget R2
   write; subsequent processes hit R2 once, then promote to memory.
   Errors swallowed - cache is best-effort.

Combined effect on a 75-page PDF:
- First chat ever:      ~12s render, ~5MB R2 write
- Same chat session:    sub-ms (memory)
- After backend restart: ~1-2s (R2 read + parse)
- New process or doc:    back to first-chat numbers

Files:
- backend/src/lib/pdfRender.ts: parallelise pdftoppm; pdfinfo page count
- backend/src/lib/visionCache.ts: new - in-memory LRU
- backend/src/lib/visionR2Cache.ts: new - R2-backed manifest
- backend/src/lib/visionContext.ts: tiered lookup + SSE events + write hookup
- backend/src/routes/chat.ts: open SSE before render so placeholder ships
- frontend/src/app/hooks/useAssistantChat.ts: handle vision_render_start/done
- frontend/src/app/components/assistant/AssistantMessage.tsx: VisionRenderBlock
- frontend/src/app/components/shared/types.ts: vision_render variant on
  AssistantEvent

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repository nwhitehouse/mike
Author Nick Whitehouse <nick.whitehouse@mccarthyfinch.com>
Authored
Parents 26ef15f4
Stats 8 files changed , +472 , -45
Part of Vision mode: PDF page images → Olava (feat-007a / 008 / 009 / 010 / bug-005)

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-6bf6d52d.md from inside the repo you want the change in.

⬇ Download capture-commit-6bf6d52d.md