RAG sidecar squeezed into 320MB on Render's free tier
Davemaina1 pulled embedding and search out of Node entirely, built a FastAPI sidecar, then spent four commits cutting its memory footprint from 1.5GB down to 320MB peak RSS -- small enough to run on Render's free tier with 192MB to spare.
The starting point (c0684a4) is an architectural split: kenyaLawSearch.ts goes from ~200 lines of ONNX-in-Node to a thin HTTP client, and all the ML work moves into rag-service/app.py. The motivation was real -- the in-process ONNX session was causing silent segfaults and tsx-watch failures under concurrent inference because @xenova/transformers v2 holds a single ONNX session that isn't safe to call in parallel. Moving it to Python sidesteps that entirely.
The memory work unfolds across the next three commits. First, torch (1.5GB) and 90MB). The sentence-transformers get replaced with onnxruntime + tokenizers + huggingface-hub, running the same all-MiniLM-L6-v2 via its ONNX export (CrossEncoder reranker gets dropped in favor of RRF fusion. That lands around 340MB.
The second cut eliminates ChromaDB. The 86K-chunk corpus is pre-exported to .npz files on a Supabase Storage public bucket. At startup the service downloads ~50MB, dequantizes int8 embeddings back to float32, builds a BM25 index in memory, and answers queries via brute-force numpy dot product -- around 180ms per query at that scale.
The final pass (3ec3223) writes corpus files to /tmp and memory-maps the embedding array instead of loading it into RAM. BM25 gets dropped entirely (saved 150MB), leaving semantic-only search. Peak RSS measured at 320MB. chromadb and rank-bm25 disappear from requirements.
Spotted something wrong? Or know the PR text has fresher detail than the writeup above?