--- title: ProbabilityRAG emoji: 🎲 colorFrom: indigo colorTo: purple sdk: docker app_port: 7860 pinned: false short_description: Local-first RAG Probability tutor --- # 🎲 ProbabilityRAG **A local-first RAG tutor that answers probability questions straight from the textbook β€” with citations you can open to the exact page.** Ask a probability question and it retrieves the relevant passages from *Grinstead & Snell* (hybrid dense + sparse search, then a cross-encoder rerank), grounds an LLM on them, and returns a cited answer you can trace back to the source PDF page. It refuses instead of guessing when the material doesn't cover the question β€” and it runs entirely offline on a single 8 GB GPU. > ### πŸ”΄ [**Try the live demo β†’**](https://piero7-probabilityrag.hf.space) > Free-tier hosting, so be patient: ~40s wake-up if it's been sleeping, ~30–90s per answer > (the same question answers in ~8 s on local GPU). Limited to 5 questions/day per visitor. ![ProbabilityRAG demo β€” ask, sources render, answer streams with LaTeX, citation opens the PDF at the exact page](docs/demo.gif) ``` question β†’ decompose (gated: split comparison/multi-part questions into sub-queries) β†’ retrieve (BGE-M3 dense + sparse, fused in Qdrant via RRF) β†’ rerank (bge-reranker-v2-m3 cross-encoder) β†’ generate (qwen2.5:3b via Ollama locally Β· GLM-4.7-Flash in the hosted demo) β†’ answer + clickable PDF-page citations ``` --- ## ✨ Features - 🟒 **Grounded, cited answers** β€” every claim cites its source chunk; click a citation to open the source PDF at that exact page. - 🟒 **Hybrid retrieval** β€” BGE-M3 dense *and* sparse vectors fused (RRF), then reranked by a cross-encoder. - 🟒 **Query decomposition** β€” comparison / multi-part questions are split into sub-queries, each reranked on its own, then round-robin merged. - 🟒 **Three answer registers** β€” the default terse answer, "Why? ↳ go deeper," and a plain-language "ELI5." - 🟒 **Refuses when unsupported** β€” replies "I don't know based on the provided material" instead of hallucinating. - 🟒 **Streaming** β€” sources render first, then tokens stream in (SSE), with a stop button. - 🟒 **Fully local & offline** β€” retrieval models and the LLM co-reside on one 8 GB GPU; no cloud API needed. - 🟒 **Model switcher** β€” a server-side allowlist maps names to local Ollama models or any OpenAI-compatible API (the demo runs GLM-4.7-Flash); the UI selector appears only when there's a real choice. - 🟒 **Deployed public demo** β€” one Docker image on HF Spaces: index and model weights baked at build, per-IP + global daily rate limits (hashed IPs, SQLite), CORS pinned, LLM failures surfaced as SSE error frames. - 🟒 **Retrieval eval harness** β€” a 45-question golden set scored by Hit@k / MRR drives every change, plus 57 unit tests (~1 s, GPU-free) run by CI on every push. --- ## 🧰 Tech stack | Area | Tooling | | --- | --- | | PDF β†’ math parsing | [Marker](https://github.com/VikParuchuri/marker) (Markdown + LaTeX) | | Embeddings | [BGE-M3](https://huggingface.co/BAAI/bge-m3) (dense + sparse) via [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) | | Vector store | [Qdrant](https://qdrant.tech/) (native hybrid + RRF fusion) | | Reranker | [bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | | Generation | [Ollama](https://ollama.com/) β€” `qwen2.5:3b-instruct` locally Β· [GLM-4.7-Flash](https://z.ai) (free API) in the demo | | Backend | [FastAPI](https://fastapi.tiangolo.com/) + [Uvicorn](https://www.uvicorn.org/) | | Deployment | Multi-stage [Docker](Dockerfile) β†’ [HF Spaces](https://huggingface.co/spaces/Piero7/ProbabilityRAG) free tier (index baked at build) | | CI | GitHub Actions β€” `pytest` on every push (torch stubbed, no GPU wheels) | | Frontend | [React](https://react.dev/) + [Vite](https://vite.dev/) + [Tailwind](https://tailwindcss.com/) | | Math rendering | [KaTeX](https://katex.org/) + [remark-math](https://github.com/remarkjs/remark-math) / [rehype-katex](https://github.com/remarkjs/remark-math) | | Env | [uv](https://github.com/astral-sh/uv) + Python 3.12 | --- ## πŸ› οΈ How I built it (the process) I'm taking a hard probability course this fall, and I wanted a tutor that answered from the actual textbook instead of confidently making things up. I also just wanted to learn RAG properly so I wrote every stage in raw Python instead of reaching for LlamaIndex. Slower, but I actually understand what each piece is doing now. The constraint that made it fun was keeping everything **local on an 8 GB GPU** β€” the retrieval models and the LLM have to share the card, no cloud API to lean on. I built the riskiest stuff first (PDFβ†’math parsing, then retrieval) and checked every change against a golden set before moving on. The part that took longest was comparison questions like *"compare the binomial and geometric variance."* A single embedding can't sit near two distributions at once, so it'd only ever retrieve one side. Splitting those into sub-queries and merging the results took multi-hop retrieval from 42/45 to 45/45 β€” and taught me the annoying lesson that my reranker was fine the whole time; I was just handing it too small a candidate pool to ever see the right chunk. --- ## πŸ“š What I learned - **Most of my "the model got it wrong" moments were actually retrieval** β€” the passage just wasn't in the top-k. Fix retrieval before you touch the prompt. - **A reranker can only reorder what you give it** β€” I spent a while "fixing ranking" when the real bug was the pool size upstream, starving it of the right chunk. - **Without the eval set I was just guessing** β€” Hit@k / MRR turned "this feels better" into a number I could actually point at. - **Pin your transitive dependencies before deploying** β€” the demo's first build crashed in production because unpinned `transformers` resolved to a new major version that removed an API my reranker library called. Worked on my machine; died in the container. - **Silent failures are the most expensive kind** β€” the hosted LLM call was failing and the response stream just… stopped, with nothing to debug from. Making the server emit the provider's actual error as an SSE frame turned a mystery into a 5-minute fix (wrong model id), and made every future failure diagnosable. --- ## πŸš€ How it could be improved - **Comparison queries are ~12s vs ~8s single-hop** β€” the decomposition lanes run sequentially; parallelizing the independent embedβ†’retrieveβ†’rerank lanes would cut that materially. - **Eval covers retrieval, not generation quality** β€” add RAGAS Faithfulness / Answer-Relevancy with a local judge for an end-to-end number. - **The free-tier demo is slow** (~30–90 s/answer) β€” retrieval runs fp32 on 2 shared vCPUs behind one worker. Paid CPU/GPU hardware or quantized ONNX embeddings would cut it dramatically. - **Covariance has no corpus coverage** β€” Grinstead & Snell never defines it, so that query (correctly) refuses; ingesting a supplementary source would close the gap. --- ## ▢️ How to run the project > Just want to try it? Use the [live demo](https://piero7-probabilityrag.hf.space) β€” no setup. To deploy your own copy, see [DEPLOY.md](DEPLOY.md). **Prerequisites:** [Ollama](https://ollama.com/) running with `qwen2.5:3b-instruct` pulled, and the *Grinstead & Snell* PDF at `data/raw/grinstead_snell.pdf` (the corpus is git-ignored β€” licensing). ### 1. Install deps ```bash uv venv && uv pip install -r requirements.txt # Python 3.12 env (torch, FlagEmbedding, qdrant-client, fastapi…) cd web && npm install # frontend ``` ### 2. Build the index (one time) ```bash .venv/Scripts/python -u scripts/embed_store.py --build ``` ### 3. Run it ```bash # backend (loads BGE-M3 + reranker once, ~40s): .venv/Scripts/python -m uvicorn app.server:app --port 8000 # frontend: cd web && npm run dev # http://localhost:5173 ```
CLI (no web UI) ```bash .venv/Scripts/python -u scripts/ask.py "What is the mgf of a Poisson?" .venv/Scripts/python -u scripts/ask.py --think "Derive Var of a binomial via its mgf" ```
Operational gotchas (important) - **Exactly ONE Ollama server, on the GPU** β€” the desktop app OR `ollama serve`, never both (the loser serves on CPU and everything looks mysteriously slow). Verify: `ollama ps` β†’ `PROCESSOR = GPU`. - **Qdrant single-writer lock** β€” stop the FastAPI server before running any script that opens `qdrant_storage/` (`ask.py`, `eval_retrieval.py`, `embed_store.py`). The eval also needs Ollama up (query decomposition calls it). - **Higher-quality model:** `PROBRAG_MODEL=deepseek-r1:7b` (better answers, but spills to CPU on 8 GB β†’ minutes/answer). Or enable several and pick in the UI: `PROBRAG_MODELS=qwen,glm-local` (full env reference in [DEPLOY.md](DEPLOY.md)). - Full dev notes in [`eval/IMPROVEMENTS.md`](eval/IMPROVEMENTS.md).
Tests / eval ```bash # unit tests β€” pure pipeline logic (chunking, retrieval merge, refusal, stream protocol); ~1s, no GPU needed .venv/Scripts/python -m pytest tests/ -q # retrieval eval β€” Hit@k / MRR on the 45-question golden set (stop the backend first; Ollama up) .venv/Scripts/python scripts/eval_retrieval.py ```
**Machine:** Dell XPS 16 Β· Win 11 Β· Intel Core Ultra 9 Β· 32 GB RAM Β· RTX 4060 (8 GB VRAM).