dev_docs_rag_implementation_plan.md

# Production Grade RAG — Developer Documentation Assistant
## Implementation Plan

---

## Project Overview

A domain-specific "Ask my Docs" system built for developer documentation.
Users can query any SDK/API/framework documentation and receive accurate answers with citations.

**Initial Target Domain:** FastAPI Documentation

## Locked MVP Choices

The MVP is intentionally constrained to one implementation path. Any alternatives listed in earlier notes are out of scope until the baseline system is working and evaluated.

| Area | MVP Choice |
|---|---|
| Documentation source | Official FastAPI documentation pages only |
| Ingestion method | `BeautifulSoup` crawler + local normalized markdown/json output |
| Extra sources | No GitHub `/docs` sync, no PDF upload, no arbitrary `.md` ingestion in MVP |
| Chunking | Structure-aware chunking with LangChain `RecursiveCharacterTextSplitter` |
| Embeddings | OpenAI `text-embedding-3-small` |
| Vector store | `ChromaDB` only |
| Keyword retrieval | `rank_bm25` |
| Fusion | Reciprocal Rank Fusion (RRF) |
| Re-ranker | Local `cross-encoder/ms-marco-MiniLM-L-6-v2` |
| Answer model | OpenAI `gpt-4o-mini` |
| API layer | `FastAPI` |
| Evaluation | `Ragas` + a hand-written FastAPI eval set |
| CI gate | Lightweight retrieval/regression checks on PRs; full eval run on demand or nightly |

## MVP Scope Boundaries

- Index only the official FastAPI docs corpus.
- Treat the MVP as a single-tenant local/dev system.
- Optimize for answer quality and citation correctness before UI polish.
- Defer multi-doc search, version-aware retrieval, PDF ingestion, and production vector DB migration until after MVP validation.

---

## Phase 1: Fundamentals (Data Ingestion Pipeline)

### Step 1 — Document Collection
- Crawl official FastAPI documentation using `BeautifulSoup`
- Normalize each page into local markdown/json before chunking
- Store raw HTML plus cleaned content locally for reproducibility
- Exclude GitHub docs sync, PDF upload, and arbitrary file ingestion from MVP
- Acceptance criteria:
  - All target FastAPI pages are fetched successfully
  - Each page is stored with `source_url`, `page_title`, and crawl timestamp
  - Re-running ingestion updates changed pages without duplicating records

### Step 2 — Chunking Strategy
- Use LangChain `RecursiveCharacterTextSplitter`
- Chunk size target: **700 tokens**
- Overlap: **100 tokens**
- Preserve code blocks and heading boundaries whenever possible
- Tag each chunk with metadata:
  - `source_url`
  - `page_title`
  - `section_title`
  - `chunk_id`
  - `doc_version`
- Acceptance criteria:
  - No chunk splits in the middle of fenced code blocks
  - Every chunk can be traced back to an exact source page and section
  - Chunk output is deterministic for unchanged source documents

### Step 3 — Embeddings + Vector Store
- Embedding model: OpenAI `text-embedding-3-small`
- Store embeddings in **ChromaDB**
- Each vector carries full metadata from Step 2
- Acceptance criteria:
  - Full FastAPI corpus is embedded successfully
  - Re-indexing does not create duplicate vectors for unchanged chunks
  - Querying Chroma returns chunk metadata required for citation display

---

## Phase 2: Hybrid Retrieval

### Step 4 — BM25 + Semantic Search
- BM25 via `rank_bm25` library for keyword matching
- Vector similarity search via ChromaDB
- Merge both result sets using **Reciprocal Rank Fusion (RRF)**
- Retrieve top 10 BM25 hits and top 10 vector hits before fusion
- Acceptance criteria:
  - Known keyword-heavy queries surface exact-term matches
  - Known semantic queries surface relevant conceptual matches
  - Fused retrieval performs better than vector-only on the seed eval set

### Step 5 — Cross Encoder Re-ranker
- Use local `cross-encoder/ms-marco-MiniLM-L-6-v2`
- Re-rank top 20 results → pass top 5 to LLM
- Biggest quality boost in the pipeline
- Acceptance criteria:
  - Re-ranking is applied on every answer path
  - Top 5 contexts are logged for debugging and eval review
  - Reranked results improve answer grounding on the seed eval set

### Step 6 — Citation Enforcement
- Each retrieved chunk carries `source_url` + `section_title`
- Prompt the LLM to answer only using provided context
- Force structured output: answer + list of cited chunks
- If no relevant chunk found → return "I don't know" (no hallucination)
- Validate that every cited chunk ID exists in the retrieved context set
- Acceptance criteria:
  - API response returns answer text plus machine-readable citations
  - Unsupported answers are rejected or converted to "I don't know"
  - Manual spot checks confirm citations map to relevant evidence

---

## Phase 3: Evaluation Pipeline

### Step 7 — Build Eval Dataset
- Manually write **30-50 Q&A pairs** from FastAPI docs for the initial eval set
- Cover: factual questions, code questions, comparison questions
- Store as JSON/CSV in the repo
- Expand to 100+ only after the first reliable baseline is in place
- Acceptance criteria:
  - Eval set covers multiple sections of the docs
  - Each question has an expected answer and supporting source reference
  - The dataset is stable enough to compare runs over time

### Step 8 — Ragas Evaluation Metrics
| Metric | What it measures |
|---|---|
| `faithfulness` | Does the answer match retrieved context? |
| `answer_relevancy` | Is the answer on-topic? |
| `context_precision` | Are the right chunks being retrieved? |
| `context_recall` | Are all relevant chunks being found? |

### Step 9 — CI Integration
- Run lightweight regression checks on every PR via **GitHub Actions**
- Reserve full Ragas evaluation for scheduled or manually triggered runs
- Use an initial target such as `faithfulness >= 0.85` as a baseline, not a permanent hard-coded gate
- Store eval results over time to track regression
- Acceptance criteria:
  - PR workflow catches obvious retrieval or API regressions quickly
  - Full eval workflow produces repeatable metrics artifacts
  - Baseline metrics are documented before strict CI thresholds are enforced

---

## Tech Stack

| Layer | Tool |
|---|---|
| Orchestration | LangChain |
| Vector Store (dev) | ChromaDB |
| Vector Store (prod) | Deferred until post-MVP |
| Reranker | `cross-encoder/ms-marco-MiniLM-L-6-v2` |
| Evaluation | Ragas |
| LLM | `gpt-4o-mini` |
| Scraping | BeautifulSoup |
| CI/CD | GitHub Actions |
| API Layer | FastAPI |
| Frontend (optional) | Deferred until post-MVP |

---

## Suggested Build Timeline

| Week | Focus |
|---|---|
| Week 1 | Phase 1 — Ingest, chunk, embed, store |
| Week 2 | Phase 2 — BM25 + rerank + citation enforcement |
| Week 3 | Phase 3 — Eval dataset + Ragas + CI pipeline |
| Week 4 | API hardening + README + baseline evaluation review |

---

## Bonus Features (Post-MVP)

- **Version-aware retrieval** — "In v2 vs v3, how does X work?"
- **Code snippet extraction** — return relevant code blocks alongside answers
- **Multi-doc search** — compare two frameworks side by side
- **Query rewriting** — auto-expand vague queries before retrieval

---

*Based on project notes from January 10–11, 2026*