sumaiyaamir/agentic_rag_poisoning

Agentic RAG Poisoning Framework

Autonomous Decision-Making LLM Agent with Data Poisoning Evaluation

---

Project Structure

agentic_rag_poisoning/
├── data/
│   ├── raw_pdfs/                  ← source PDFs (generated by generate_pdfs.py)
│   └── faiss_index/               ← saved vector store (auto-created)
├── src/
│   ├── __init__.py
│   ├── ingestion.py               ← Phase 1 & 2: PDF → chunks → embeddings → FAISS
│   ├── rag_pipeline.py            ← Phase 3: retrieval + Ollama LLM generation
│   ├── agent_monitor.py           ← Phase 4: autonomous validation & refinement
│   ├── poisoning.py               ← Phase 5: 4 attack strategies
│   └── evaluation.py              ← Phase 6: RAGAS metrics comparison
├── results/
│   └── evaluation_report.json     ← auto-generated after evaluation
├── generate_pdfs.py               ← creates the 3 sample PDFs
├── main.py                        ← unified pipeline runner
├── requirements.txt
├── .env
└── README.md

---

FULL SETUP PROCEDURE (VS Code + Windows)

Follow every step in order. Commands marked [CMD] run in Command Prompt / PowerShell. Commands marked [VS CODE TERMINAL] run in the VS Code integrated terminal.

---

STEP 1 — Install Prerequisites

1a. Install Python 3.11

• Download from: https://www.python.org/downloads/
• During installation: tick "Add Python to PATH"
• Verify in CMD:

  python --version
  

  Expected: Python 3.11.x

1b. Install Ollama (local LLM runtime)

• Download from: https://ollama.com/download
• Run the installer — it installs as a background service
• Verify in CMD:

  ollama --version
  

1c. Install VS Code

• Download from: https://code.visualstudio.com/
• Install the Python extension (search "Python" in Extensions tab)

---

STEP 2 — Create the Project Folder

In [CMD]:

mkdir C:\Projects\agentic_rag_poisoning
cd C:\Projects\agentic_rag_poisoning

Copy all project files into this folder so the structure matches above.

---

STEP 3 — Open in VS Code

In [CMD]:

code C:\Projects\agentic_rag_poisoning

OR: Open VS Code → File → Open Folder → select agentic_rag_poisoning

---

STEP 4 — Create Virtual Environment

In [VS CODE TERMINAL] (Terminal → New Terminal):

python -m venv venv

Activate it:

• Windows:

  venv\Scripts\activate
  

• Mac/Linux:

  source venv/bin/activate
  

You should see (venv) at the start of your terminal prompt.

Select the venv interpreter in VS Code:

• Press Ctrl+Shift+P → type "Python: Select Interpreter"
• Choose the one that says .\venv\Scripts\python.exe

---

STEP 5 — Install Dependencies

In [VS CODE TERMINAL] (with venv active):

pip install --upgrade pip
pip install -r requirements.txt

This installs ~1.5 GB of packages. Takes 5–15 minutes depending on internet speed.

If torch fails (common on Windows CPU-only), install it separately:

pip install torch --index-url https://download.pytorch.org/whl/cpu

---

STEP 6 — Pull the Ollama LLM Model

Open a new [CMD] window (NOT the VS Code terminal — Ollama runs separately):

ollama pull mistral

Downloads ~4 GB. This is the Mistral 7B model. Takes 5–20 minutes.

Verify Ollama is running:

ollama serve

Leave this CMD window open while using the project.

In another CMD, test it:

ollama run mistral "Hello, are you working?"

---

STEP 7 — Generate Sample PDFs

Back in [VS CODE TERMINAL] (venv active):

python generate_pdfs.py

Expected output:

Created: data/raw_pdfs/machine_learning_intro.pdf
Created: data/raw_pdfs/rag_systems_guide.pdf
Created: data/raw_pdfs/data_poisoning_attacks.pdf

All 3 PDFs generated successfully in data/raw_pdfs/

You can open these PDFs in VS Code or any PDF viewer to inspect them.

---

STEP 8 — Run Phase 1 & 2: Ingestion

python src/ingestion.py

Expected output:

Loading PDFs from 'data/raw_pdfs'...
  ✓ machine_learning_intro.pdf (5 pages)
  ✓ rag_systems_guide.pdf (4 pages)
  ✓ data_poisoning_attacks.pdf (4 pages)

Chunking complete: 13 pages → ~80 chunks

Loading embedding model: sentence-transformers/all-MiniLM-L6-v2
  (First run downloads ~90 MB — subsequent runs use cache)

Building FAISS index from 80 chunks...
✓ Vector store saved to 'data/faiss_index'

---

STEP 9 — Run Phase 3: RAG Pipeline

Make sure Ollama is running (Step 6).

python src/rag_pipeline.py

You will see question → answer → source citations for 4 test questions.

---

STEP 10 — Run Phase 4: Agentic Monitor

python src/agent_monitor.py

Watch the agent validate each response and trigger refinement when needed.

---

STEP 11 — Run Phase 5: Poisoning Attacks

python src/poisoning.py

This runs all 4 attack strategies and shows you sample poisoned chunks.

---

STEP 12 — Run Phase 6: Evaluation

python src/evaluation.py

This builds clean + poisoned vector stores, runs all questions through both, and prints a comparison table like:

══════════════════════════════════════════════════════════════
  Evaluation: Clean RAG  vs  Poisoned RAG [Label Flipping]
══════════════════════════════════════════════════════════════
  Metric                 Clean    Poisoned      Drop
  ──────────────────────────────────────────────────────
  faithfulness           0.842       0.531    ▼ 0.311
  answer_relevancy       0.791       0.624    ▼ 0.167
  context_recall         0.768       0.592    ▼ 0.176
  context_precision      0.803       0.641    ▼ 0.162
══════════════════════════════════════════════════════════════

Results are saved to results/evaluation_report.json.

---

STEP 13 — Run Full Pipeline at Once

python main.py

Or run specific phases:

python main.py --phase ingestion
python main.py --phase rag
python main.py --phase agent
python main.py --phase poison
python main.py --phase eval
python main.py --rebuild    # force rebuild vector store

---

Adding Your Own PDFs

1. Drop any PDF into data/raw_pdfs/
2. Run: python src/ingestion.py (force rebuild)
3. Or in code: run_ingestion(force_rebuild=True)

---

Common Errors & Fixes

Error	Fix
ollama: connection refused	Run ollama serve in a separate CMD window
ModuleNotFoundError	Activate venv: venv\Scripts\activate
torch not found	pip install torch --index-url https://download.pytorch.org/whl/cpu
No PDF files found	Run python generate_pdfs.py first
FAISS index not found	Run python src/ingestion.py first
out of memory	Reduce CHUNK_SIZE in ingestion.py to 256

---

Tech Stack (All Free / Open Source)

Component	Tool	Version
LLM	Mistral 7B via Ollama	0.3.x
Embeddings	all-MiniLM-L6-v2	sentence-transformers 3.x
Vector DB	FAISS (CPU)	faiss-cpu 1.9
RAG Framework	LangChain	0.3.7
PDF Loading	PyMuPDF	1.24
Evaluation	RAGAS	0.2.5
PDF Generation	ReportLab	4.2