Portable DB framework with italaw and ICSID scrapers
Gadoes builds a self-hosted alternative to API-based MCP sources - a scraper framework for jurisdictions with no published MCP server - and ships two concrete implementations: italaw (international investment arbitration) and ICSID (World Bank arbitration awards). Three chunks, six commits.
Chunk 14 - Framework. Three files under backend/src/lib/mcp/portable/: baseScraper.ts (69 lines) defines the BaseScraper contract with an async generator scrape() method; portableMcpServer.ts (57 lines) wraps a scraped corpus as an MCP-compatible server; freshnessManager.ts (120 lines) tracks when a corpus was last refreshed and whether it's within its configured staleness window. New runtime dependencies: better-sqlite3@^12.9.0 (for the local corpus DB), cheerio@^1.2.0 (for HTML parsing), and p-throttle@^8.1.0 (to rate-limit scrape requests).
Chunk 15 - italaw. portable/italaw/scraper.ts (104 lines) fetches and parses case pages from italaw.com into better-sqlite3 rows. server.ts (40 lines) wraps it as an MCP server. Five HTML fixtures cover cases like Metalclad v. Mexico, Vattenfall v. Germany (Energy Charter Treaty), and Siemens v. Argentina. A scripts/scrape-italaw.ts CLI entry point runs the scrape manually: npm run scrape:italaw [path/to/italaw.db]. 216-line test suite.
Chunk 16 - ICSID. Same pattern: portable/icsid/scraper.ts (102 lines) and server.ts (40 lines), five HTML fixtures covering awards, jurisdiction decisions, and an annulment (Siemens ARB/03/8; Occidental v. Ecuador ARB/06/11, the $1.7B award; Salini v. Morocco for the jurisdiction test). npm run scrape:icsid. This chunk also drops supabase/config.toml (406 lines) - local Supabase dev environment setup, unrelated to the portable DB feature.
The italaw and ICSID citation verification_status is auto-set to verified on save (see Chunk 20) because the corpus is a local scrape rather than a live API call.
Spotted something wrong? Or know the PR text has fresher detail than the writeup above?