feat: legalbench external benchmark adapter and first live baselines
Nine Stanford LegalBench tasks (arXiv:2308.11462, HF nguha/legalbench) covering claimed capabilities: rule application/QA (abercrombie, hearsay, personal_jurisdiction, ucc_v_common_law), issue spotting (corporate_lobbying), contract/merger clause understanding (contract_qa, cuad_anti-assignment, maud_specific_performance), and citation support (citation_prediction_classification). All CC BY 4.0; Learned Hands excluded (CC BY-NC-SA), rule_qa excluded (manual eval only). Official test splits and upstream base_prompt.txt files are downloaded to git-ignored benchmarks/legalbench/data/ and pinned byte-for-byte in the committed manifest, which also records per-task license and the paper's GPT-4 balanced accuracy with table citation. Scoring ports evaluation.py normalize() + sklearn balanced accuracy; chat-style label extraction is deterministic and scores ambiguity as wrong. Limited runs use a class-stratified prefix of the official split (several splits group rows by gold label). First live gate, gpt-5-mini, 25 stratified examples/task: abercrombie balanced acc 88.0 (GPT-4 paper: 85.3), cuad_anti-assignment 96.2 (GPT-4 paper: 91.4), 0 unparsed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yXxuitxX5h4wm1v88hnga
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 7fb565c7 |
| Stats | 7 files changed , +1698 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-c3928839.md
from inside the repo you want the change in.