feat: legalbench external benchmark adapter and first live baselines

↗ view on GitHub · Eli Ziff · 2026-07-28 · c3928839

Nine Stanford LegalBench tasks (arXiv:2308.11462, HF nguha/legalbench)
covering claimed capabilities: rule application/QA (abercrombie, hearsay,
personal_jurisdiction, ucc_v_common_law), issue spotting
(corporate_lobbying), contract/merger clause understanding (contract_qa,
cuad_anti-assignment, maud_specific_performance), and citation support
(citation_prediction_classification). All CC BY 4.0; Learned Hands
excluded (CC BY-NC-SA), rule_qa excluded (manual eval only).

Official test splits and upstream base_prompt.txt files are downloaded to
git-ignored benchmarks/legalbench/data/ and pinned byte-for-byte in the
committed manifest, which also records per-task license and the paper's
GPT-4 balanced accuracy with table citation. Scoring ports evaluation.py
normalize() + sklearn balanced accuracy; chat-style label extraction is
deterministic and scores ambiguity as wrong. Limited runs use a
class-stratified prefix of the official split (several splits group rows
by gold label).

First live gate, gpt-5-mini, 25 stratified examples/task:
abercrombie balanced acc 88.0 (GPT-4 paper: 85.3),
cuad_anti-assignment 96.2 (GPT-4 paper: 91.4), 0 unparsed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 7fb565c7
Stats 7 files changed , +1698
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-c3928839.md from inside the repo you want the change in.

⬇ Download capture-commit-c3928839.md