Phase 6: audit logging for LLM + tool invocations
From the PR description
Summary
Phase 6 of the Mike → GordonOSS finance fork build plan: append-only audit logging for every LLM call and every tool invocation. Required groundwork before paid data connectors (Phase 8) so we can prove who saw what data and which model handled it.
What changed
- New
audit_logtable with immutability triggers (UPDATE/DELETE raiseaudit_log entries are immutable). Migration inbackend/migrations/audit_log.sqland appended tobackend/schema.sql. Indexes on(user_id, created_at),(project_id, created_at), andevent_type. backend/src/lib/audit.ts-AuditEntryshape,recordAudit()(fire-and-forget; never throws),hashContent()SHA-256 helper,AUDIT_LOG_ENABLEDfeature flag.- Tool dispatcher (
lib/tools/registry.ts) records onetool_callrow per invocation withduration_ms, input/output hashes, and resolveddocument_ids(extracted from both args and side effects). Errors are recorded then re-thrown. - LLM adapter (
lib/llm/index.ts) records onellm_callrow perstreamChatWithToolscall with provider, model, hashes, and duration. Audit context flows throughrunLLMStream(used by/chat,/projects/:id/chat,/tabular-review/:id/chat) and the tabulargeneratepath. GET /audit-log- user-scoped reader withproject_id,event_type,from,to,limit(default 100, max 1000), andoffsetfilters.- Unit tests cover
hashContentdeterminism, insert shape, feature-flag no-op, and error swallowing.
What's NOT in this PR
The schema's event_type CHECK reserves connector_fetch, document_upload, and document_download but no code path emits them yet - they land in Phase 8 (connectors) and a follow-up that instruments the documents/downloads routes. No migration needed when those wire up.
Reviewer notes
- Audit failures are silent by design:
recordAuditcatches all errors andconsole.errors them. The build plan calls this out - audit logging must never break a user request. - Input/output hashes only: we store SHA-256 of the inputs and outputs, never raw content. Forensic traceability without leaking message content into a second table.
user_emailis denormalized:user_idisON DELETE SET NULLso historical entries survive a user deletion; the email column preserves attribution for investigators.- Per-iteration vs per-call:
streamChatWithToolsmay internally drive multiple provider iterations (tool-use loops). We record one row per outer call rather than one per iteration - keeps the table compact and matches "one user message = one llm_call row."
Test plan
-
npx vitest runinbackend/- 65/65 passing (7 new inaudit.test.ts) -
npx tsc --noEmitclean -
npx eslint src- 0 errors (pre-existing warnings unrelated) - Apply
backend/migrations/audit_log.sqlto Supabase, send a chat message, confirmaudit_logcontains bothllm_callandtool_callrows -
UPDATE audit_log SET status='error' WHERE id = ...should raise the immutability exception -
GET /audit-logreturns only the caller's own rows
🤖 Generated with Claude Code
Our analysis
Append-only audit logging for LLM and tool calls — read the full analysis →
Think the analysis missed something the PR description covers?
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-6.md from
inside the repo you want the changes in.