README.md

---
title: ProbabilityRAG
emoji: 🎲
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
short_description: Local-first RAG Probability tutor
---

# 🎲 ProbabilityRAG

**A local-first RAG tutor that answers probability questions straight from the textbook — with citations you can open to the exact page.**

Ask a probability question and it retrieves the relevant passages from *Grinstead & Snell* (hybrid dense + sparse search, then a cross-encoder rerank), grounds an LLM on them, and returns a cited answer you can trace back to the source PDF page. It refuses instead of guessing when the material doesn't cover the question — and it runs entirely offline on a single 8 GB GPU.

> ### 🔴 [**Try the live demo →**](https://piero7-probabilityrag.hf.space)
> Free-tier hosting, so be patient: ~40s wake-up if it's been sleeping, ~30–90s per answer
> (the same question answers in ~8 s on local GPU). Limited to 5 questions/day per visitor.

![ProbabilityRAG demo — ask, sources render, answer streams with LaTeX, citation opens the PDF at the exact page](docs/demo.gif)

```
question
  → decompose   (gated: split comparison/multi-part questions into sub-queries)
  → retrieve    (BGE-M3 dense + sparse, fused in Qdrant via RRF)
  → rerank      (bge-reranker-v2-m3 cross-encoder)
  → generate    (qwen2.5:3b via Ollama locally · GLM-4.7-Flash in the hosted demo)
  → answer + clickable PDF-page citations
```

---

## ✨ Features

- 🟢 **Grounded, cited answers** — every claim cites its source chunk; click a citation to open the source PDF at that exact page.
- 🟢 **Hybrid retrieval** — BGE-M3 dense *and* sparse vectors fused (RRF), then reranked by a cross-encoder.
- 🟢 **Query decomposition** — comparison / multi-part questions are split into sub-queries, each reranked on its own, then round-robin merged.
- 🟢 **Three answer registers** — the default terse answer, "Why? ↳ go deeper," and a plain-language "ELI5."
- 🟢 **Refuses when unsupported** — replies "I don't know based on the provided material" instead of hallucinating.
- 🟢 **Streaming** — sources render first, then tokens stream in (SSE), with a stop button.
- 🟢 **Fully local & offline** — retrieval models and the LLM co-reside on one 8 GB GPU; no cloud API needed.
- 🟢 **Model switcher** — a server-side allowlist maps names to local Ollama models or any OpenAI-compatible API (the demo runs GLM-4.7-Flash); the UI selector appears only when there's a real choice.
- 🟢 **Deployed public demo** — one Docker image on HF Spaces: index and model weights baked at build, per-IP + global daily rate limits (hashed IPs, SQLite), CORS pinned, LLM failures surfaced as SSE error frames.
- 🟢 **Retrieval eval harness** — a 45-question golden set scored by Hit@k / MRR drives every change, plus 57 unit tests (~1 s, GPU-free) run by CI on every push.

---

## 🧰 Tech stack

| Area | Tooling |
| --- | --- |
| PDF → math parsing | [Marker](https://github.com/VikParuchuri/marker) (Markdown + LaTeX) |
| Embeddings | [BGE-M3](https://huggingface.co/BAAI/bge-m3) (dense + sparse) via [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) |
| Vector store | [Qdrant](https://qdrant.tech/) (native hybrid + RRF fusion) |
| Reranker | [bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) |
| Generation | [Ollama](https://ollama.com/) — `qwen2.5:3b-instruct` locally · [GLM-4.7-Flash](https://z.ai) (free API) in the demo |
| Backend | [FastAPI](https://fastapi.tiangolo.com/) + [Uvicorn](https://www.uvicorn.org/) |
| Deployment | Multi-stage [Docker](Dockerfile) → [HF Spaces](https://huggingface.co/spaces/Piero7/ProbabilityRAG) free tier (index baked at build) |
| CI | GitHub Actions — `pytest` on every push (torch stubbed, no GPU wheels) |
| Frontend | [React](https://react.dev/) + [Vite](https://vite.dev/) + [Tailwind](https://tailwindcss.com/) |
| Math rendering | [KaTeX](https://katex.org/) + [remark-math](https://github.com/remarkjs/remark-math) / [rehype-katex](https://github.com/remarkjs/remark-math) |
| Env | [uv](https://github.com/astral-sh/uv) + Python 3.12 |

---

## 🛠️ How I built it (the process)

I'm taking a hard probability course this fall, and I wanted a tutor that answered from the actual textbook instead of confidently making things up. I also just wanted to learn RAG properly so I wrote every stage in raw Python instead of reaching for LlamaIndex. Slower, but I actually understand what each piece is doing now.

The constraint that made it fun was keeping everything **local on an 8 GB GPU** — the retrieval models and the LLM have to share the card, no cloud API to lean on. I built the riskiest stuff first (PDF→math parsing, then retrieval) and checked every change against a golden set before moving on. The part that took longest was comparison questions like *"compare the binomial and geometric variance."* A single embedding can't sit near two distributions at once, so it'd only ever retrieve one side. Splitting those into sub-queries and merging the results took multi-hop retrieval from 42/45 to 45/45 — and taught me the annoying lesson that my reranker was fine the whole time; I was just handing it too small a candidate pool to ever see the right chunk.

---

## 📚 What I learned

- **Most of my "the model got it wrong" moments were actually retrieval** — the passage just wasn't in the top-k. Fix retrieval before you touch the prompt.
- **A reranker can only reorder what you give it** — I spent a while "fixing ranking" when the real bug was the pool size upstream, starving it of the right chunk.
- **Without the eval set I was just guessing** — Hit@k / MRR turned "this feels better" into a number I could actually point at.
- **Pin your transitive dependencies before deploying** — the demo's first build crashed in production because unpinned `transformers` resolved to a new major version that removed an API my reranker library called. Worked on my machine; died in the container.
- **Silent failures are the most expensive kind** — the hosted LLM call was failing and the response stream just… stopped, with nothing to debug from. Making the server emit the provider's actual error as an SSE frame turned a mystery into a 5-minute fix (wrong model id), and made every future failure diagnosable.

---

## 🚀 How it could be improved

- **Comparison queries are ~12s vs ~8s single-hop** — the decomposition lanes run sequentially; parallelizing the independent embed→retrieve→rerank lanes would cut that materially.
- **Eval covers retrieval, not generation quality** — add RAGAS Faithfulness / Answer-Relevancy with a local judge for an end-to-end number.
- **The free-tier demo is slow** (~30–90 s/answer) — retrieval runs fp32 on 2 shared vCPUs behind one worker. Paid CPU/GPU hardware or quantized ONNX embeddings would cut it dramatically.
- **Covariance has no corpus coverage** — Grinstead & Snell never defines it, so that query (correctly) refuses; ingesting a supplementary source would close the gap.

---

## ▶️ How to run the project

> Just want to try it? Use the [live demo](https://piero7-probabilityrag.hf.space) — no setup. To deploy your own copy, see [DEPLOY.md](DEPLOY.md).

**Prerequisites:** [Ollama](https://ollama.com/) running with `qwen2.5:3b-instruct` pulled, and the *Grinstead & Snell* PDF at `data/raw/grinstead_snell.pdf` (the corpus is git-ignored — licensing).

### 1. Install deps
```bash
uv venv && uv pip install -r requirements.txt   # Python 3.12 env (torch, FlagEmbedding, qdrant-client, fastapi…)
cd web && npm install                            # frontend
```

### 2. Build the index (one time)
```bash
.venv/Scripts/python -u scripts/embed_store.py --build
```

### 3. Run it
```bash
# backend (loads BGE-M3 + reranker once, ~40s):
.venv/Scripts/python -m uvicorn app.server:app --port 8000
# frontend:
cd web && npm run dev         # http://localhost:5173
```

<details>
<summary><b>CLI (no web UI)</b></summary>

```bash
.venv/Scripts/python -u scripts/ask.py "What is the mgf of a Poisson?"
.venv/Scripts/python -u scripts/ask.py --think "Derive Var of a binomial via its mgf"
```
</details>

<details>
<summary><b>Operational gotchas (important)</b></summary>

- **Exactly ONE Ollama server, on the GPU** — the desktop app OR `ollama serve`, never both (the loser serves on CPU and everything looks mysteriously slow). Verify: `ollama ps` → `PROCESSOR = GPU`.
- **Qdrant single-writer lock** — stop the FastAPI server before running any script that opens `qdrant_storage/` (`ask.py`, `eval_retrieval.py`, `embed_store.py`). The eval also needs Ollama up (query decomposition calls it).
- **Higher-quality model:** `PROBRAG_MODEL=deepseek-r1:7b` (better answers, but spills to CPU on 8 GB → minutes/answer). Or enable several and pick in the UI: `PROBRAG_MODELS=qwen,glm-local` (full env reference in [DEPLOY.md](DEPLOY.md)).
- Full dev notes in [`eval/IMPROVEMENTS.md`](eval/IMPROVEMENTS.md).
</details>

<details>
<summary><b>Tests / eval</b></summary>

```bash
# unit tests — pure pipeline logic (chunking, retrieval merge, refusal, stream protocol); ~1s, no GPU needed
.venv/Scripts/python -m pytest tests/ -q

# retrieval eval — Hit@k / MRR on the 45-question golden set (stop the backend first; Ollama up)
.venv/Scripts/python scripts/eval_retrieval.py
```
</details>

**Machine:** Dell XPS 16 · Win 11 · Intel Core Ultra 9 · 32 GB RAM · RTX 4060 (8 GB VRAM).