feat: structured JSON logger with per-request id and access log

🟢 open · #156 · Open-Legal-Products/mike ← Shawnaldinho/mike · opened 2mo ago by Shawnaldinho · +149-5 across 5 files · ↗ on GitHub

From the PR description

Summary

Backend logs were all free-form `console.log`/`console.error`, so an error you see on the client can't be correlated to the line that produced it on the server, and there's no way to filter by route/status/user_id from a log aggregator. This PR puts the floor in: a tiny structured logger, a per-request id, and an access log line per response.

Changes

  • `backend/src/lib/logger.ts` - zero-dep JSON-line logger. `logger.{debug,info,warn,error}({ fields }, msg)` writes one JSON object per call (stdout for info/debug, stderr for warn/error). Level is controlled by `LOG_LEVEL` (defaults: `info` in production, `debug` elsewhere). `logger.child({ ... })` returns a logger that prefixes every event with the given fields. `errFields(err)` pulls `err_name`/`err_msg`/`err_stack` out of an `Error`. Hand-written instead of pulling pino - only ~80 lines today; the call sites won't change if it's later swapped.
  • `backend/src/middleware/requestContext.ts` - assigns a per-request id (inbound `X-Request-Id` if present, else a UUID), echoes it back on the response, parks a child logger on `res.locals.log`, and on response `close` emits one access-log line with method/route/status/elapsed_ms/user_id. Wired into `index.ts` before the rate limiters.
  • Demonstrate adoption in the most user-visible places: the two streaming error paths (`POST /chat`, `POST /projects/:projectId/chat`) and the title-generation handler now use the request-scoped logger so their error lines correlate with the access log via `request_id`. Debug-style `console.log` calls in `chatTools.ts` / `documents.ts` were left alone - that's a separate, broader cleanup.

Why

Concrete things this enables that weren't possible before:

  • `request_id= level=error` filter in CloudWatch / Loki / Datadog pulls every line tied to one failing request.
  • Frontend can surface the `X-Request-Id` so a user-reported "chat broke" comes with the exact id you grep for.
  • Distinguishing client-side cancels (`status=499`) from server-side errors (`status=500`) without touching the console.

Addresses the observability concern from https://insights.flank.ai/where-mikeoss-falls-short.html (gap 14). Doesn't try to also tackle tracing / metrics / Sentry - those are reasonable follow-ups built on this foundation.

Testing

  • `npm run build --prefix backend` passes.
  • Hand-traced a request through `requestContext` → rate-limiter → handler → `close` and confirmed (a) inbound `X-Request-Id` is preserved, (b) generated UUID echoed back when absent, (c) access-log line emits with the same id, (d) the streaming error logs share that id.

Our analysis

Add structured logging with per-request IDs and access logs — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-156.md from inside the repo you want the changes in.

⬇ Download capture-pull-156.md