Initial commit: SPARKNET framework

.dockerignore

@@ -0,0 +1,90 @@
+# Git
+.git
+.gitignore
+.gitattributes
+
+# Python
+__pycache__
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+.mypy_cache/
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Virtual environments
+venv/
+ENV/
+env/
+.venv/
+sparknet/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs
+*.log
+logs/
+
+# Local data (will be mounted as volumes)
+data/vectorstore/
+data/embedding_cache/
+uploads/
+outputs/
+
+# Tests
+tests/
+.pytest_cache/
+
+# Documentation
+docs/
+*.md
+!README.md
+
+# Notebooks
+*.ipynb
+.ipynb_checkpoints/
+
+# Backup files
+.backup/
+*.bak
+
+# Screenshots
+screenshots/
+
+# Development files
+*.env.local
+*.env.development
+*.env.test
+
+# Large files
+*.pdf
+*.pptx
+*.docx
+Dataset/
+presentation/

.streamlit/config.toml

@@ -0,0 +1,14 @@
+[server]
+headless = true
+port = 8501
+enableCORS = false
+maxUploadSize = 50
+
+[theme]
+primaryColor = "#4ECDC4"
+backgroundColor = "#0e1117"
+secondaryBackgroundColor = "#1a1a2e"
+textColor = "#ffffff"
+
+[browser]
+gatherUsageStats = false

CHANGELOG.md

@@ -0,0 +1,232 @@
+# SPARKNET Changelog
+
+All notable changes to the SPARKNET project are documented in this file.
+
+## [1.2.0] - 2026-01-20
+
+### Added (Phase 1B Continuation)
+
+#### Table Extraction Preservation (FG-002) - HIGH PRIORITY
+- **Enhanced SemanticChunker** (`src/document/chunking/chunker.py`)
+  - Table structure reconstruction from OCR regions
+  - Markdown table generation with proper formatting
+  - Header row detection using heuristics
+  - Structured data storage in `extra.table_structure`
+  - Cell positions preserved for evidence highlighting
+  - Searchable text includes header context for better embedding
+  - Configurable row/column thresholds
+
+- **ChunkerConfig enhancements**
+  - `preserve_table_structure` - Enable markdown conversion
+  - `table_row_threshold` - Y-coordinate grouping threshold
+  - `table_col_threshold` - X-coordinate clustering threshold
+  - `detect_table_headers` - Automatic header detection
+
+#### Nginx Configuration (TG-005)
+- **Nginx Reverse Proxy** (`nginx/nginx.conf`)
+  - Production-ready reverse proxy configuration
+  - Rate limiting (30 req/s API, 5 req/s uploads)
+  - WebSocket support for Streamlit
+  - SSE support for RAG streaming
+  - Gzip compression
+  - Security headers (XSS, CSRF protection)
+  - SSL/TLS configuration (commented, ready for production)
+  - Connection limits and timeout tuning
+
+#### Integration Tests (TG-006)
+- **API Integration Tests** (`tests/integration/test_api_v2.py`)
+  - TestClient-based testing without server
+  - Health/status endpoint tests
+  - Authentication flow tests
+  - Document upload/process/index workflow
+  - RAG query and search tests
+  - Error handling verification
+  - Concurrency tests
+  - Performance benchmarks (marked slow)
+
+- **Table Chunker Unit Tests** (`tests/unit/test_table_chunker.py`)
+  - Table structure reconstruction tests
+  - Markdown generation tests
+  - Header detection tests
+  - Column detection tests
+  - Edge case handling
+
+#### Cross-Module State Synchronization (Phase 1B)
+- **Enhanced State Manager** (`demo/state_manager.py`)
+  - Event system with pub/sub pattern
+  - `EventType` enum for type-safe events
+  - Evidence highlighting synchronization
+  - Page/chunk selection sync across modules
+  - RAG query/response sharing
+  - Module-specific state storage
+  - Sync version tracking for change detection
+  - Helper components: `render_evidence_panel()`, `render_document_selector()`
+
+---
+
+## [1.1.0] - 2026-01-20
+
+### Added
+
+#### REST API (Phase 1B - TG-003)
+- **Document API** (`api/routes/documents.py`)
+  - `POST /api/documents/upload` - Upload and process documents
+  - `GET /api/documents` - List all documents with filtering
+  - `GET /api/documents/{doc_id}` - Get document by ID
+  - `GET /api/documents/{doc_id}/detail` - Get detailed document info
+  - `GET /api/documents/{doc_id}/chunks` - Get document chunks
+  - `POST /api/documents/{doc_id}/process` - Trigger processing
+  - `POST /api/documents/{doc_id}/index` - Index to RAG
+  - `POST /api/documents/batch-index` - Batch index multiple documents
+  - `DELETE /api/documents/{doc_id}` - Delete a document
+
+- **RAG API** (`api/routes/rag.py`)
+  - `POST /api/rag/query` - Execute RAG query with 5-agent pipeline
+  - `POST /api/rag/query/stream` - Stream RAG response (SSE)
+  - `POST /api/rag/search` - Semantic search without synthesis
+  - `GET /api/rag/store/status` - Get vector store status
+  - `DELETE /api/rag/store/collection/{name}` - Clear collection
+  - `GET /api/rag/cache/stats` - Get cache statistics
+  - `DELETE /api/rag/cache` - Clear query cache
+
+- **API Schemas** (`api/schemas.py`)
+  - Request/response models for all endpoints
+  - Document, Query, Search, Citation schemas
+  - Pydantic validation with comprehensive field definitions
+
+#### Authentication (Phase 1C - TG-002)
+- **JWT Authentication** (`api/auth.py`)
+  - OAuth2 password bearer scheme
+  - `POST /api/auth/token` - Get access token
+  - `POST /api/auth/register` - Register new user
+  - `GET /api/auth/me` - Get current user info
+  - `GET /api/auth/users` - List users (admin only)
+  - `DELETE /api/auth/users/{username}` - Delete user (admin only)
+  - Password hashing with bcrypt
+  - Default admin user creation on startup
+
+#### Extended Document Support (Phase 1B - FG-001)
+- Added support for new document formats in document processing:
+  - **Word (.docx)** - Full text and table extraction
+  - **Excel (.xlsx, .xls)** - Multi-sheet extraction
+  - **PowerPoint (.pptx)** - Slide-by-slide text extraction
+  - **Text (.txt)** - Plain text processing
+  - **Markdown (.md)** - Markdown file support
+
+#### Caching (Phase 1B - TG-004)
+- **Cache Manager** (`src/utils/cache_manager.py`)
+  - Redis-based caching with in-memory fallback
+  - `QueryCache` - Cache RAG query results (1 hour TTL)
+  - `EmbeddingCache` - Cache embeddings (24 hour TTL)
+  - `@cached` decorator for function-level caching
+  - Automatic cache cleanup and size limits
+
+#### Docker Containerization (Phase 1C - TG-007)
+- **Dockerfile** - Multi-stage build
+  - Production stage with optimized image
+  - Development stage with hot reload
+  - Health checks and proper dependencies
+
+- **docker-compose.yml** - Full stack deployment
+  - SPARKNET API service
+  - Streamlit Demo service
+  - Ollama LLM service with GPU support
+  - ChromaDB vector store
+  - Redis cache
+  - Optional Nginx reverse proxy
+
+- **docker-compose.dev.yml** - Development configuration
+  - Volume mounts for code changes
+  - Hot reload enabled
+  - Connects to host Ollama
+
+- **.dockerignore** - Optimized build context
+
+### Changed
+
+#### API Main (`api/main.py`)
+- Enhanced lifespan initialization with graceful degradation
+- Added RAG component initialization
+- Improved health check with component status
+- New `/api/status` endpoint for comprehensive system status
+- Better error handling allowing partial functionality
+
+### Technical Details
+
+#### New Files Created
+```
+api/
+├── auth.py              # Authentication module
+├── schemas.py           # Pydantic models
+└── routes/
+    ├── documents.py     # Document endpoints
+    └── rag.py           # RAG endpoints
+
+src/utils/
+└── cache_manager.py     # Redis/memory caching
+
+docker/
+├── Dockerfile           # Multi-stage build
+├── docker-compose.yml   # Production stack
+├── docker-compose.dev.yml # Development stack
+└── .dockerignore        # Build optimization
+```
+
+#### Dependencies Added
+- `python-jose[cryptography]` - JWT tokens
+- `passlib[bcrypt]` - Password hashing
+- `python-multipart` - Form data handling
+- `redis` - Redis client (optional)
+- `python-docx` - Word document support
+- `openpyxl` - Excel support
+- `python-pptx` - PowerPoint support
+
+#### Configuration
+- `SPARKNET_SECRET_KEY` - JWT secret (environment variable)
+- `REDIS_URL` - Redis connection string
+- `OLLAMA_HOST` - Ollama server URL
+- `CHROMA_HOST` / `CHROMA_PORT` - ChromaDB connection
+
+### API Quick Reference
+
+```bash
+# Health check
+curl http://localhost:8000/api/health
+
+# Upload document
+curl -X POST -F "file=@document.pdf" http://localhost:8000/api/documents/upload
+
+# Query RAG
+curl -X POST http://localhost:8000/api/rag/query \
+  -H "Content-Type: application/json" \
+  -d '{"query": "What are the main findings?"}'
+
+# Get token
+curl -X POST http://localhost:8000/api/auth/token \
+  -d "username=admin&password=admin123"
+```
+
+### Docker Quick Start
+
+```bash
+# Production deployment
+docker-compose up -d
+
+# Development with hot reload
+docker-compose -f docker-compose.dev.yml up
+
+# Pull Ollama models
+docker exec sparknet-ollama ollama pull llama3.2:latest
+docker exec sparknet-ollama ollama pull mxbai-embed-large:latest
+```
+
+---
+
+## [1.0.0] - 2026-01-19
+
+### Initial Release
+- Multi-Agent RAG Pipeline (5 agents)
+- Document Processing Pipeline (OCR, Layout, Chunking)
+- Streamlit Demo Application (5 modules)
+- ChromaDB Vector Store
+- Ollama LLM Integration

Dockerfile

@@ -0,0 +1,109 @@
+# SPARKNET Dockerfile
+# Multi-stage build for optimized production image
+
+# ============== Build Stage ==============
+FROM python:3.11-slim as builder
+
+WORKDIR /app
+
+# Install build dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    gcc \
+    g++ \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements first for caching
+COPY requirements.txt .
+COPY api/requirements.txt ./api_requirements.txt
+
+# Create virtual environment and install dependencies
+RUN python -m venv /opt/venv
+ENV PATH="/opt/venv/bin:$PATH"
+
+RUN pip install --no-cache-dir --upgrade pip && \
+    pip install --no-cache-dir -r requirements.txt && \
+    pip install --no-cache-dir -r api_requirements.txt
+
+# ============== Production Stage ==============
+FROM python:3.11-slim as production
+
+LABEL maintainer="SPARKNET Team"
+LABEL description="SPARKNET: Multi-Agentic Document Intelligence Platform"
+LABEL version="1.0.0"
+
+WORKDIR /app
+
+# Install runtime dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    # PDF processing
+    poppler-utils \
+    libpoppler-cpp-dev \
+    # Image processing
+    libgl1-mesa-glx \
+    libglib2.0-0 \
+    libsm6 \
+    libxext6 \
+    libxrender-dev \
+    # OCR support
+    tesseract-ocr \
+    tesseract-ocr-eng \
+    # Utilities
+    curl \
+    wget \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy virtual environment from builder
+COPY --from=builder /opt/venv /opt/venv
+ENV PATH="/opt/venv/bin:$PATH"
+
+# Set Python environment
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PYTHONPATH=/app
+
+# Copy application code
+COPY src/ ./src/
+COPY api/ ./api/
+COPY config/ ./config/
+COPY demo/ ./demo/
+
+# Create necessary directories
+RUN mkdir -p /app/data/vectorstore \
+    /app/data/embedding_cache \
+    /app/uploads/documents \
+    /app/uploads/patents \
+    /app/outputs \
+    /app/logs
+
+# Set permissions
+RUN chmod -R 755 /app
+
+# Expose ports
+# 8000 - FastAPI
+# 4000 - Streamlit
+EXPOSE 8000 4000
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
+    CMD curl -f http://localhost:8000/api/health || exit 1
+
+# Default command - run FastAPI
+CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"]
+
+# ============== Development Stage ==============
+FROM production as development
+
+# Install development dependencies
+RUN pip install --no-cache-dir \
+    pytest \
+    pytest-asyncio \
+    pytest-cov \
+    black \
+    flake8 \
+    mypy \
+    ipython \
+    jupyter
+
+# Development command with hot reload
+CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]

IMPLEMENTATION_REPORT.md

@@ -0,0 +1,474 @@
+# SPARKNET Implementation Report
+## Agentic Document Intelligence Platform
+
+**Report Date:** January 2025
+**Version:** 0.1.0
+
+---
+
+## Executive Summary
+
+SPARKNET is an enterprise-grade **Agentic Document Intelligence Platform** that follows FAANG best practices for:
+- **Modular Architecture**: Clean separation of concerns with well-defined interfaces
+- **Local-First Privacy**: All processing happens locally via Ollama
+- **Evidence Grounding**: Every extraction includes verifiable source references
+- **Production-Ready**: Type-safe, tested, configurable, and scalable
+
+---
+
+## 1. What Has Been Implemented
+
+### 1.1 Core Subsystems
+
+| Subsystem | Location | Status | Description |
+|-----------|----------|--------|-------------|
+| **Document Intelligence** | `src/document_intelligence/` | Complete | Vision-first document understanding |
+| **Legacy Document Pipeline** | `src/document/` | Complete | OCR, layout, chunking pipeline |
+| **RAG Subsystem** | `src/rag/` | Complete | Vector search with grounded retrieval |
+| **Multi-Agent System** | `src/agents/` | Complete | ReAct-style agents with tools |
+| **LLM Integration** | `src/llm/` | Complete | Ollama client with routing |
+| **CLI** | `src/cli/` | Complete | Full command-line interface |
+| **API** | `api/` | Complete | FastAPI REST endpoints |
+| **Demo UI** | `demo/` | Complete | Streamlit dashboard |
+
+### 1.2 Document Intelligence Module (`src/document_intelligence/`)
+
+**Architecture (FAANG-inspired: Google DocAI pattern):**
+
+```
+src/document_intelligence/
+├── chunks/           # Core data models (BoundingBox, DocumentChunk, TableChunk)
+│   ├── models.py     # Pydantic models with full type safety
+│   └── __init__.py
+├── io/               # Document loading with caching
+│   ├── base.py       # Abstract interfaces
+│   ├── pdf.py        # PyMuPDF-based PDF loading
+│   ├── image.py      # PIL image loading
+│   └── cache.py      # LRU page caching
+├── models/           # ML model interfaces
+│   ├── base.py       # BaseModel, BatchableModel
+│   ├── ocr.py        # OCRModel interface
+│   ├── layout.py     # LayoutModel interface
+│   ├── table.py      # TableModel interface
+│   └── vlm.py        # VisionLanguageModel interface
+├── parsing/          # Document parsing pipeline
+│   ├── parser.py     # DocumentParser orchestrator
+│   └── chunking.py   # SemanticChunker
+├── grounding/        # Visual evidence
+│   ├── evidence.py   # EvidenceBuilder, EvidenceTracker
+│   └── crops.py      # Image cropping utilities
+├── extraction/       # Field extraction
+│   ├── schema.py     # ExtractionSchema, FieldSpec
+│   ├── extractor.py  # FieldExtractor
+│   └── validator.py  # ExtractionValidator
+├── tools/            # Agent tools
+│   ├── document_tools.py  # ParseDocumentTool, ExtractFieldsTool, etc.
+│   └── rag_tools.py       # IndexDocumentTool, RetrieveChunksTool, RAGAnswerTool
+└── agent_adapter.py  # EnhancedDocumentAgent integration
+```
+
+**Key Features:**
+- **Zero-Shot Capability**: Works across document formats without training
+- **Schema-Driven Extraction**: Define fields using JSON Schema or Pydantic
+- **Abstention Policy**: Never guesses - abstains when confidence is low
+- **Visual Grounding**: Every extraction includes page, bbox, snippet, confidence
+
+### 1.3 RAG Subsystem (`src/rag/`)
+
+**Architecture (FAANG-inspired: Meta FAISS + Google Vertex AI pattern):**
+
+```
+src/rag/
+├── store.py          # VectorStore interface + ChromaVectorStore
+├── embeddings.py     # OllamaEmbedding + OpenAIEmbedding (feature-flagged)
+├── indexer.py        # DocumentIndexer for chunked documents
+├── retriever.py      # DocumentRetriever with evidence support
+├── generator.py      # GroundedGenerator with citations
+├── docint_bridge.py  # Bridge to document_intelligence subsystem
+└── __init__.py       # Clean exports
+```
+
+**Key Features:**
+- **Local-First Embeddings**: Ollama `nomic-embed-text` by default
+- **Cloud Opt-In**: OpenAI embeddings disabled by default, feature-flagged
+- **Metadata Filtering**: Filter by document_id, chunk_type, page_range
+- **Citation Generation**: Answers include `[1]`, `[2]` references
+- **Confidence-Based Abstention**: Returns "I don't know" when uncertain
+
+### 1.4 Multi-Agent System (`src/agents/`)
+
+**Agents Implemented:**
+| Agent | Purpose | Model |
+|-------|---------|-------|
+| `ExecutorAgent` | Task execution with tools | llama3.1:8b |
+| `DocumentAgent` | ReAct-style document analysis | llama3.1:8b |
+| `PlannerAgent` | Task decomposition | mistral |
+| `CriticAgent` | Output validation | phi3 |
+| `MemoryAgent` | Context management | llama3.2 |
+| `VisionOCRAgent` | Vision-based OCR | llava (optional) |
+
+### 1.5 CLI Commands
+
+```bash
+# Document Intelligence
+sparknet docint parse document.pdf -o result.json
+sparknet docint extract invoice.pdf --preset invoice
+sparknet docint ask document.pdf "What is the total?"
+sparknet docint classify document.pdf
+
+# RAG Operations
+sparknet docint index document.pdf              # Index into vector store
+sparknet docint index-stats                     # Show index statistics
+sparknet docint retrieve "payment terms" -k 10  # Semantic search
+sparknet docint ask doc.pdf "question" --use-rag  # RAG-powered Q&A
+
+# Legacy Document Commands
+sparknet document parse invoice.pdf
+sparknet document extract contract.pdf -f "party_name"
+sparknet rag index *.pdf --collection my_docs
+sparknet rag search "query" --top 10
+```
+
+---
+
+## 2. How to Execute SPARKNET
+
+### 2.1 Prerequisites
+
+```bash
+# 1. System Requirements
+# - Python 3.10+
+# - NVIDIA GPU with CUDA 12.0+ (optional but recommended)
+# - 16GB+ RAM
+# - 50GB+ disk space
+
+# 2. Install Ollama (if not installed)
+curl -fsSL https://ollama.com/install.sh | sh
+
+# 3. Start Ollama server
+ollama serve
+```
+
+### 2.2 Installation
+
+```bash
+cd /home/mhamdan/SPARKNET
+
+# Option A: Use existing virtual environment
+source sparknet/bin/activate
+
+# Option B: Create new environment
+python3 -m venv sparknet
+source sparknet/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+pip install -r demo/requirements.txt
+
+# Install SPARKNET in development mode
+pip install -e .
+```
+
+### 2.3 Download Required Models
+
+```bash
+# Embedding model (required for RAG)
+ollama pull nomic-embed-text:latest
+
+# LLM models (at least one required)
+ollama pull llama3.2:latest     # Fast, 2GB
+ollama pull llama3.1:8b         # General purpose, 5GB
+ollama pull mistral:latest      # Good reasoning, 4GB
+
+# Optional: Larger models for complex tasks
+ollama pull qwen2.5:14b         # Complex reasoning, 9GB
+```
+
+### 2.4 Running the Demo UI
+
+**Method 1: Using the launcher script**
+```bash
+cd /home/mhamdan/SPARKNET
+./run_demo.sh 8501
+```
+
+**Method 2: Direct Streamlit command**
+```bash
+cd /home/mhamdan/SPARKNET
+source sparknet/bin/activate
+streamlit run demo/app.py --server.port 8501
+```
+
+**Method 3: Bind to specific IP (for remote access)**
+```bash
+streamlit run demo/app.py \
+  --server.address 172.24.50.21 \
+  --server.port 8501 \
+  --server.headless true
+```
+
+**Access at:** http://172.24.50.21:8501 or http://localhost:8501
+
+### 2.5 Running the API Server
+
+```bash
+cd /home/mhamdan/SPARKNET
+source sparknet/bin/activate
+uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload
+```
+
+**API Endpoints:**
+- `GET /health` - Health check
+- `POST /api/documents/parse` - Parse document
+- `POST /api/documents/extract` - Extract fields
+- `POST /api/rag/index` - Index document
+- `POST /api/rag/query` - Query RAG
+
+### 2.6 Running Examples
+
+```bash
+cd /home/mhamdan/SPARKNET
+source sparknet/bin/activate
+
+# Document Intelligence Demo
+python examples/document_intelligence_demo.py
+
+# RAG End-to-End Pipeline
+python examples/document_rag_end_to_end.py
+
+# Simple Agent Task
+python examples/simple_task.py
+
+# Document Agent
+python examples/document_agent.py
+```
+
+### 2.7 Running Tests
+
+```bash
+cd /home/mhamdan/SPARKNET
+source sparknet/bin/activate
+
+# Run all tests
+pytest tests/ -v
+
+# Run specific test suites
+pytest tests/unit/test_document_intelligence.py -v
+pytest tests/unit/test_rag_integration.py -v
+
+# Run with coverage
+pytest tests/ --cov=src --cov-report=html
+```
+
+---
+
+## 3. Configuration
+
+### 3.1 RAG Configuration (`configs/rag.yaml`)
+
+```yaml
+vector_store:
+  type: chroma
+  chroma:
+    persist_directory: "./.sparknet/chroma_db"
+    collection_name: "sparknet_documents"
+    distance_metric: cosine
+
+embeddings:
+  provider: ollama  # Local-first
+  ollama:
+    model: nomic-embed-text
+    base_url: "http://localhost:11434"
+  openai:
+    enabled: false  # Disabled by default
+
+generator:
+  provider: ollama
+  ollama:
+    model: llama3.2
+  abstain_on_low_confidence: true
+  abstain_threshold: 0.3
+```
+
+### 3.2 Document Configuration (`config/document.yaml`)
+
+```yaml
+ocr:
+  engine: paddleocr  # or tesseract
+  languages: ["en"]
+  confidence_threshold: 0.5
+
+layout:
+  enabled: true
+  reading_order: true
+
+chunking:
+  min_chunk_chars: 10
+  max_chunk_chars: 4000
+  target_chunk_chars: 500
+```
+
+---
+
+## 4. FAANG Best Practices Applied
+
+### 4.1 Google-Inspired Patterns
+- **DocAI Architecture**: Modular vision-first document understanding
+- **Structured Output**: Schema-driven extraction with validation
+- **Abstention Policy**: Never hallucinate, return "I don't know"
+
+### 4.2 Meta-Inspired Patterns
+- **FAISS Integration**: Fast similarity search (optional alongside ChromaDB)
+- **RAG Pipeline**: Retrieve-then-generate with citations
+
+### 4.3 Amazon-Inspired Patterns
+- **Textract-like API**: Structured field extraction with confidence scores
+- **Evidence Grounding**: Every output traceable to source
+
+### 4.4 Microsoft-Inspired Patterns
+- **Form Recognizer Pattern**: Pre-built schemas for invoices, contracts
+- **Confidence Thresholds**: Configurable abstention levels
+
+### 4.5 Apple-Inspired Patterns
+- **Privacy-First**: All processing local by default
+- **Opt-In Cloud**: OpenAI and cloud services disabled by default
+
+---
+
+## 5. Quick Start Commands
+
+```bash
+# === SETUP ===
+cd /home/mhamdan/SPARKNET
+source sparknet/bin/activate
+ollama serve &  # Start in background
+
+# === DEMO UI ===
+streamlit run demo/app.py --server.port 8501
+
+# === CLI USAGE ===
+# Parse a document
+python -m src.cli.main docint parse Dataset/IBM*.pdf -o result.json
+
+# Index for RAG
+python -m src.cli.main docint index Dataset/*.pdf
+
+# Ask questions with RAG
+python -m src.cli.main docint ask Dataset/IBM*.pdf "What is this document about?" --use-rag
+
+# === PYTHON API ===
+python -c "
+from src.document_intelligence import DocumentParser
+parser = DocumentParser()
+result = parser.parse('Dataset/IBM N_A.pdf')
+print(f'Parsed {len(result.chunks)} chunks')
+"
+
+# === RUN TESTS ===
+pytest tests/unit/ -v
+```
+
+---
+
+## 6. Troubleshooting
+
+### Issue: Ollama not running
+```bash
+# Check status
+curl http://localhost:11434/api/tags
+
+# Start Ollama
+ollama serve
+
+# If port in use
+pkill ollama && ollama serve
+```
+
+### Issue: Missing models
+```bash
+ollama list  # See installed models
+ollama pull nomic-embed-text  # Install embedding model
+ollama pull llama3.2  # Install LLM
+```
+
+### Issue: ChromaDB errors
+```bash
+# Reset vector store
+rm -rf .sparknet/chroma_db
+```
+
+### Issue: Import errors
+```bash
+# Ensure in correct directory
+cd /home/mhamdan/SPARKNET
+
+# Ensure venv activated
+source sparknet/bin/activate
+
+# Reinstall
+pip install -e .
+```
+
+---
+
+## 7. Architecture Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        SPARKNET Platform                        │
+├─────────────────────────────────────────────────────────────────┤
+│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐             │
+│  │  Streamlit  │  │  FastAPI    │  │    CLI      │  Interfaces │
+│  │    Demo     │  │    API      │  │  Commands   │             │
+│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘             │
+├─────────┴────────────────┴────────────────┴─────────────────────┤
+│                                                                 │
+│  ┌──────────────────────────────────────────────────────────┐  │
+│  │                   Agent Layer                            │  │
+│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐    │  │
+│  │  │ Document │ │ Executor │ │ Planner  │ │  Critic  │    │  │
+│  │  │  Agent   │ │  Agent   │ │  Agent   │ │  Agent   │    │  │
+│  │  └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘    │  │
+│  └───────┴────────────┴────────────┴────────────┴───────────┘  │
+│                                                                 │
+│  ┌────────────────────┐  ┌─────────────────────────────────┐   │
+│  │ Document Intel     │  │         RAG Subsystem           │   │
+│  │ ┌───────┐ ┌──────┐ │  │ ┌─────────┐ ┌─────────────────┐ │   │
+│  │ │Parser │ │Extract│ │  │ │Indexer  │ │   Retriever     │ │   │
+│  │ └───────┘ └──────┘ │  │ └─────────┘ └─────────────────┘ │   │
+│  │ ┌───────┐ ┌──────┐ │  │ ┌─────────┐ ┌─────────────────┐ │   │
+│  │ │Ground │ │Valid │ │  │ │Embedder │ │   Generator     │ │   │
+│  │ └───────┘ └──────┘ │  │ └─────────┘ └──────��──────────┘ │   │
+│  └────────────────────┘  └─────────────────────────────────┘   │
+│                                                                 │
+│  ┌─────────────────────────────────────────────────────────┐   │
+│  │                   Infrastructure                         │   │
+│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐   │   │
+│  │  │  Ollama  │ │ ChromaDB │ │   GPU    │ │  Cache   │   │   │
+│  │  │  Client  │ │  Store   │ │ Manager  │ │  Layer   │   │   │
+│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘   │   │
+│  └─────────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 8. Files Modified/Created in Recent Session
+
+| File | Action | Description |
+|------|--------|-------------|
+| `src/rag/docint_bridge.py` | Created | Bridge between document_intelligence and RAG |
+| `src/document_intelligence/tools/rag_tools.py` | Created | RAG tools for agents |
+| `src/document_intelligence/tools/__init__.py` | Modified | Added RAG tool exports |
+| `src/document_intelligence/tools/document_tools.py` | Modified | Enhanced AnswerQuestionTool with RAG |
+| `src/cli/docint.py` | Modified | Added index, retrieve, delete-index commands |
+| `src/rag/__init__.py` | Modified | Added bridge exports |
+| `configs/rag.yaml` | Created | RAG configuration file |
+| `tests/unit/test_rag_integration.py` | Created | RAG integration tests |
+| `examples/document_rag_end_to_end.py` | Created | End-to-end RAG example |
+
+---
+
+**Report Complete**
+
+For questions or issues, refer to the troubleshooting section above or check the test files for usage examples.

api/auth.py

@@ -0,0 +1,320 @@
+"""
+SPARKNET Authentication Module
+JWT-based authentication with OAuth2 support.
+"""
+
+from fastapi import Depends, HTTPException, status
+from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
+from jose import JWTError, jwt
+from passlib.context import CryptContext
+from pydantic import BaseModel
+from datetime import datetime, timedelta
+from typing import Optional, List
+from pathlib import Path
+import os
+import json
+import uuid
+
+# Configuration (use environment variables in production)
+SECRET_KEY = os.getenv("SPARKNET_SECRET_KEY", "sparknet-super-secret-key-change-in-production")
+ALGORITHM = "HS256"
+ACCESS_TOKEN_EXPIRE_MINUTES = 30
+
+# Password hashing
+pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
+
+# OAuth2 scheme
+oauth2_scheme = OAuth2PasswordBearer(tokenUrl="api/auth/token", auto_error=False)
+
+# Simple file-based user store (replace with database in production)
+USERS_FILE = Path(__file__).parent.parent / "data" / "users.json"
+USERS_FILE.parent.mkdir(parents=True, exist_ok=True)
+
+
+class User(BaseModel):
+    """User model."""
+    user_id: str
+    username: str
+    email: str
+    hashed_password: str
+    is_active: bool = True
+    is_admin: bool = False
+    scopes: List[str] = []
+    created_at: datetime = None
+
+    class Config:
+        json_encoders = {
+            datetime: lambda v: v.isoformat() if v else None
+        }
+
+
+class UserInDB(User):
+    """User model with password hash."""
+    pass
+
+
+class TokenData(BaseModel):
+    """JWT token payload."""
+    username: Optional[str] = None
+    user_id: Optional[str] = None
+    scopes: List[str] = []
+
+
+def _load_users() -> dict:
+    """Load users from file."""
+    if USERS_FILE.exists():
+        try:
+            with open(USERS_FILE) as f:
+                data = json.load(f)
+                return {u["username"]: User(**u) for u in data}
+        except Exception:
+            pass
+    return {}
+
+
+def _save_users(users: dict):
+    """Save users to file."""
+    with open(USERS_FILE, "w") as f:
+        json.dump([u.dict() for u in users.values()], f, default=str, indent=2)
+
+
+def verify_password(plain_password: str, hashed_password: str) -> bool:
+    """Verify a password against its hash."""
+    return pwd_context.verify(plain_password, hashed_password)
+
+
+def get_password_hash(password: str) -> str:
+    """Hash a password."""
+    return pwd_context.hash(password)
+
+
+def get_user(username: str) -> Optional[UserInDB]:
+    """Get a user by username."""
+    users = _load_users()
+    if username in users:
+        return UserInDB(**users[username].dict())
+    return None
+
+
+def authenticate_user(username: str, password: str) -> Optional[UserInDB]:
+    """Authenticate a user."""
+    user = get_user(username)
+    if not user:
+        return None
+    if not verify_password(password, user.hashed_password):
+        return None
+    return user
+
+
+def create_access_token(data: dict, expires_delta: Optional[timedelta] = None) -> str:
+    """Create a JWT access token."""
+    to_encode = data.copy()
+    if expires_delta:
+        expire = datetime.utcnow() + expires_delta
+    else:
+        expire = datetime.utcnow() + timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)
+    to_encode.update({"exp": expire})
+    encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)
+    return encoded_jwt
+
+
+async def get_current_user(token: str = Depends(oauth2_scheme)) -> Optional[UserInDB]:
+    """Get the current user from JWT token."""
+    if not token:
+        return None
+
+    try:
+        payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
+        username: str = payload.get("sub")
+        if username is None:
+            return None
+        token_data = TokenData(
+            username=username,
+            user_id=payload.get("user_id"),
+            scopes=payload.get("scopes", [])
+        )
+    except JWTError:
+        return None
+
+    user = get_user(token_data.username)
+    return user
+
+
+async def get_current_active_user(
+    current_user: Optional[UserInDB] = Depends(get_current_user)
+) -> Optional[UserInDB]:
+    """Get current active user (authentication optional)."""
+    if current_user and not current_user.is_active:
+        return None
+    return current_user
+
+
+async def require_auth(
+    current_user: Optional[UserInDB] = Depends(get_current_user)
+) -> UserInDB:
+    """Require authentication (raises exception if not authenticated)."""
+    credentials_exception = HTTPException(
+        status_code=status.HTTP_401_UNAUTHORIZED,
+        detail="Could not validate credentials",
+        headers={"WWW-Authenticate": "Bearer"},
+    )
+    if not current_user:
+        raise credentials_exception
+    if not current_user.is_active:
+        raise HTTPException(status_code=400, detail="Inactive user")
+    return current_user
+
+
+async def require_admin(
+    current_user: UserInDB = Depends(require_auth)
+) -> UserInDB:
+    """Require admin privileges."""
+    if not current_user.is_admin:
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Admin privileges required"
+        )
+    return current_user
+
+
+def create_user(username: str, email: str, password: str, is_admin: bool = False) -> User:
+    """Create a new user."""
+    users = _load_users()
+
+    if username in users:
+        raise ValueError(f"User {username} already exists")
+
+    user = User(
+        user_id=str(uuid.uuid4()),
+        username=username,
+        email=email,
+        hashed_password=get_password_hash(password),
+        is_active=True,
+        is_admin=is_admin,
+        scopes=["read", "write"] if not is_admin else ["read", "write", "admin"],
+        created_at=datetime.now()
+    )
+
+    users[username] = user
+    _save_users(users)
+    return user
+
+
+def delete_user(username: str) -> bool:
+    """Delete a user."""
+    users = _load_users()
+    if username in users:
+        del users[username]
+        _save_users(users)
+        return True
+    return False
+
+
+# Initialize default admin user if none exists
+def init_default_admin():
+    """Create default admin user if no users exist."""
+    users = _load_users()
+    if not users:
+        try:
+            create_user(
+                username="admin",
+                email="admin@sparknet.local",
+                password="admin123",  # Change in production!
+                is_admin=True
+            )
+            print("Default admin user created: admin / admin123")
+        except Exception as e:
+            print(f"Could not create default admin: {e}")
+
+
+# Auth routes
+from fastapi import APIRouter
+
+auth_router = APIRouter()
+
+
+@auth_router.post("/token")
+async def login_for_access_token(form_data: OAuth2PasswordRequestForm = Depends()):
+    """OAuth2 compatible token login."""
+    user = authenticate_user(form_data.username, form_data.password)
+    if not user:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Incorrect username or password",
+            headers={"WWW-Authenticate": "Bearer"},
+        )
+    access_token_expires = timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)
+    access_token = create_access_token(
+        data={
+            "sub": user.username,
+            "user_id": user.user_id,
+            "scopes": user.scopes
+        },
+        expires_delta=access_token_expires
+    )
+    return {
+        "access_token": access_token,
+        "token_type": "bearer",
+        "expires_in": ACCESS_TOKEN_EXPIRE_MINUTES * 60
+    }
+
+
+@auth_router.post("/register")
+async def register_user(
+    username: str,
+    email: str,
+    password: str,
+):
+    """Register a new user."""
+    try:
+        user = create_user(username, email, password)
+        return {
+            "user_id": user.user_id,
+            "username": user.username,
+            "email": user.email,
+            "message": "User created successfully"
+        }
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+
+@auth_router.get("/me")
+async def read_users_me(current_user: UserInDB = Depends(require_auth)):
+    """Get current user information."""
+    return {
+        "user_id": current_user.user_id,
+        "username": current_user.username,
+        "email": current_user.email,
+        "is_active": current_user.is_active,
+        "is_admin": current_user.is_admin,
+        "scopes": current_user.scopes
+    }
+
+
+@auth_router.get("/users")
+async def list_users(current_user: UserInDB = Depends(require_admin)):
+    """List all users (admin only)."""
+    users = _load_users()
+    return [
+        {
+            "user_id": u.user_id,
+            "username": u.username,
+            "email": u.email,
+            "is_active": u.is_active,
+            "is_admin": u.is_admin
+        }
+        for u in users.values()
+    ]
+
+
+@auth_router.delete("/users/{username}")
+async def delete_user_endpoint(
+    username: str,
+    current_user: UserInDB = Depends(require_admin)
+):
+    """Delete a user (admin only)."""
+    if username == current_user.username:
+        raise HTTPException(status_code=400, detail="Cannot delete yourself")
+    if delete_user(username):
+        return {"status": "deleted", "username": username}
+    raise HTTPException(status_code=404, detail=f"User not found: {username}")

api/routes/documents.py

@@ -0,0 +1,553 @@
+"""
+SPARKNET Document API Routes
+Endpoints for document upload, processing, and management.
+"""
+
+from fastapi import APIRouter, UploadFile, File, HTTPException, Query, Depends, BackgroundTasks
+from fastapi.responses import StreamingResponse
+from typing import List, Optional
+from pathlib import Path
+from datetime import datetime
+import hashlib
+import shutil
+import uuid
+import io
+import sys
+
+# Add project root to path
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from api.schemas import (
+    DocumentUploadResponse, DocumentResponse, DocumentMetadata,
+    DocumentDetailResponse, ChunksResponse, ChunkInfo,
+    OCRRegionInfo, LayoutRegionInfo, DocumentStatus,
+    IndexRequest, IndexResponse, BatchIndexRequest, BatchIndexResponse
+)
+from loguru import logger
+
+router = APIRouter()
+
+# In-memory document store (replace with database in production)
+_documents = {}
+_processing_tasks = {}
+
+# Supported file types
+SUPPORTED_EXTENSIONS = {
+    '.pdf': 'application/pdf',
+    '.png': 'image/png',
+    '.jpg': 'image/jpeg',
+    '.jpeg': 'image/jpeg',
+    '.tiff': 'image/tiff',
+    '.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
+    '.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
+    '.pptx': 'application/vnd.openxmlformats-officedocument.presentationml.presentation',
+    '.txt': 'text/plain',
+    '.md': 'text/markdown',
+}
+
+UPLOAD_DIR = PROJECT_ROOT / "uploads" / "documents"
+UPLOAD_DIR.mkdir(parents=True, exist_ok=True)
+
+
+def generate_doc_id(filename: str, content: bytes) -> str:
+    """Generate unique document ID from filename and content hash."""
+    content_hash = hashlib.md5(content[:4096]).hexdigest()[:8]
+    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+    return f"doc_{timestamp}_{content_hash}"
+
+
+async def process_document_task(doc_id: str, file_path: Path, file_type: str):
+    """Background task to process a document."""
+    try:
+        logger.info(f"Processing document: {doc_id}")
+        _documents[doc_id]["status"] = DocumentStatus.PROCESSING
+
+        # Try to use actual document processor
+        try:
+            from src.document.pipeline.processor import DocumentProcessor, PipelineConfig
+
+            config = PipelineConfig(
+                ocr_enabled=True,
+                layout_enabled=True,
+                chunking_enabled=True,
+            )
+            processor = DocumentProcessor(config)
+            result = processor.process(str(file_path))
+
+            # Extract data from result
+            chunks = []
+            for i, chunk in enumerate(getattr(result, 'chunks', [])):
+                chunks.append({
+                    "chunk_id": f"{doc_id}_chunk_{i}",
+                    "doc_id": doc_id,
+                    "text": getattr(chunk, 'text', str(chunk)),
+                    "chunk_type": getattr(chunk, 'chunk_type', 'text'),
+                    "page_num": getattr(chunk, 'page', 0),
+                    "confidence": getattr(chunk, 'confidence', 1.0),
+                    "bbox": getattr(chunk, 'bbox', None),
+                })
+
+            _documents[doc_id].update({
+                "status": DocumentStatus.COMPLETED,
+                "raw_text": getattr(result, 'raw_text', ''),
+                "chunks": chunks,
+                "page_count": getattr(result, 'page_count', 1),
+                "ocr_regions": getattr(result, 'ocr_regions', []),
+                "layout_regions": getattr(result, 'layout_regions', []),
+                "processing_time": getattr(result, 'processing_time', 0.0),
+                "updated_at": datetime.now(),
+            })
+
+            logger.success(f"Document {doc_id} processed successfully: {len(chunks)} chunks")
+
+        except Exception as proc_error:
+            logger.warning(f"Full processor unavailable: {proc_error}, using fallback")
+            # Fallback: simple text extraction
+            raw_text = ""
+
+            if file_type in ['.pdf']:
+                try:
+                    import fitz
+                    doc = fitz.open(str(file_path))
+                    for page in doc:
+                        raw_text += page.get_text() + "\n"
+                    page_count = len(doc)
+                    doc.close()
+                except Exception as e:
+                    logger.error(f"PDF extraction failed: {e}")
+                    page_count = 1
+
+            elif file_type in ['.txt', '.md']:
+                raw_text = file_path.read_text(errors='ignore')
+                page_count = 1
+
+            elif file_type == '.docx':
+                try:
+                    from docx import Document
+                    doc = Document(str(file_path))
+                    raw_text = "\n".join([p.text for p in doc.paragraphs])
+                    page_count = max(1, len(raw_text) // 3000)
+                except Exception as e:
+                    logger.error(f"DOCX extraction failed: {e}")
+                    page_count = 1
+
+            elif file_type == '.xlsx':
+                try:
+                    import pandas as pd
+                    df_dict = pd.read_excel(str(file_path), sheet_name=None)
+                    for sheet_name, df in df_dict.items():
+                        raw_text += f"\n=== Sheet: {sheet_name} ===\n"
+                        raw_text += df.to_string() + "\n"
+                    page_count = len(df_dict)
+                except Exception as e:
+                    logger.error(f"XLSX extraction failed: {e}")
+                    page_count = 1
+
+            elif file_type == '.pptx':
+                try:
+                    from pptx import Presentation
+                    prs = Presentation(str(file_path))
+                    for i, slide in enumerate(prs.slides):
+                        raw_text += f"\n=== Slide {i+1} ===\n"
+                        for shape in slide.shapes:
+                            if hasattr(shape, "text"):
+                                raw_text += shape.text + "\n"
+                    page_count = len(prs.slides)
+                except Exception as e:
+                    logger.error(f"PPTX extraction failed: {e}")
+                    page_count = 1
+
+            # Create simple chunks
+            chunks = []
+            chunk_size = 1000
+            text_chunks = [raw_text[i:i+chunk_size] for i in range(0, len(raw_text), chunk_size - 100)]
+            for i, text in enumerate(text_chunks):
+                if text.strip():
+                    chunks.append({
+                        "chunk_id": f"{doc_id}_chunk_{i}",
+                        "doc_id": doc_id,
+                        "text": text.strip(),
+                        "chunk_type": "text",
+                        "page_num": min(i * chunk_size // 3000 + 1, page_count),
+                        "confidence": 1.0,
+                        "bbox": None,
+                    })
+
+            _documents[doc_id].update({
+                "status": DocumentStatus.COMPLETED,
+                "raw_text": raw_text,
+                "chunks": chunks,
+                "page_count": page_count,
+                "ocr_regions": [],
+                "layout_regions": [],
+                "processing_time": 0.0,
+                "updated_at": datetime.now(),
+            })
+
+            logger.info(f"Document {doc_id} processed with fallback: {len(chunks)} chunks")
+
+    except Exception as e:
+        logger.error(f"Document processing failed for {doc_id}: {e}")
+        _documents[doc_id]["status"] = DocumentStatus.ERROR
+        _documents[doc_id]["error"] = str(e)
+
+
+@router.post("/upload", response_model=DocumentUploadResponse)
+async def upload_document(
+    background_tasks: BackgroundTasks,
+    file: UploadFile = File(...),
+    auto_process: bool = Query(True, description="Automatically process after upload"),
+    auto_index: bool = Query(False, description="Automatically index to RAG after processing"),
+):
+    """
+    Upload a document for processing.
+
+    Supported formats: PDF, PNG, JPG, DOCX, XLSX, PPTX, TXT, MD
+    """
+    # Validate file extension
+    file_ext = Path(file.filename).suffix.lower()
+    if file_ext not in SUPPORTED_EXTENSIONS:
+        raise HTTPException(
+            status_code=400,
+            detail=f"Unsupported file type: {file_ext}. Supported: {list(SUPPORTED_EXTENSIONS.keys())}"
+        )
+
+    # Read file content
+    content = await file.read()
+    if len(content) == 0:
+        raise HTTPException(status_code=400, detail="Empty file uploaded")
+
+    # Generate document ID
+    doc_id = generate_doc_id(file.filename, content)
+
+    # Save file
+    file_path = UPLOAD_DIR / f"{doc_id}{file_ext}"
+    with open(file_path, "wb") as f:
+        f.write(content)
+
+    # Create document record
+    _documents[doc_id] = {
+        "doc_id": doc_id,
+        "filename": file.filename,
+        "file_type": file_ext,
+        "file_path": str(file_path),
+        "status": DocumentStatus.PENDING,
+        "raw_text": "",
+        "chunks": [],
+        "page_count": 0,
+        "ocr_regions": [],
+        "layout_regions": [],
+        "indexed": False,
+        "indexed_chunks": 0,
+        "processing_time": None,
+        "created_at": datetime.now(),
+        "updated_at": None,
+        "auto_index": auto_index,
+    }
+
+    # Start processing in background
+    if auto_process:
+        background_tasks.add_task(process_document_task, doc_id, file_path, file_ext)
+        status = DocumentStatus.PROCESSING
+        message = "Document uploaded and processing started"
+    else:
+        status = DocumentStatus.PENDING
+        message = "Document uploaded successfully. Call /process to begin processing."
+
+    _documents[doc_id]["status"] = status
+
+    return DocumentUploadResponse(
+        doc_id=doc_id,
+        filename=file.filename,
+        status=status,
+        message=message,
+        created_at=_documents[doc_id]["created_at"]
+    )
+
+
+@router.get("", response_model=List[DocumentMetadata])
+async def list_documents(
+    status: Optional[DocumentStatus] = Query(None, description="Filter by status"),
+    indexed: Optional[bool] = Query(None, description="Filter by indexed status"),
+    limit: int = Query(50, ge=1, le=200),
+    offset: int = Query(0, ge=0),
+):
+    """List all documents with optional filtering."""
+    docs = list(_documents.values())
+
+    # Apply filters
+    if status:
+        docs = [d for d in docs if d["status"] == status]
+    if indexed is not None:
+        docs = [d for d in docs if d.get("indexed", False) == indexed]
+
+    # Apply pagination
+    docs = docs[offset:offset + limit]
+
+    return [
+        DocumentMetadata(
+            doc_id=d["doc_id"],
+            filename=d["filename"],
+            file_type=d["file_type"],
+            page_count=d.get("page_count", 0),
+            chunk_count=len(d.get("chunks", [])),
+            text_length=len(d.get("raw_text", "")),
+            status=d["status"],
+            indexed=d.get("indexed", False),
+            indexed_chunks=d.get("indexed_chunks", 0),
+            processing_time=d.get("processing_time"),
+            created_at=d["created_at"],
+            updated_at=d.get("updated_at"),
+        )
+        for d in docs
+    ]
+
+
+@router.get("/{doc_id}", response_model=DocumentResponse)
+async def get_document(
+    doc_id: str,
+    include_text: bool = Query(False, description="Include full raw text"),
+):
+    """Get document by ID."""
+    if doc_id not in _documents:
+        raise HTTPException(status_code=404, detail=f"Document not found: {doc_id}")
+
+    d = _documents[doc_id]
+
+    return DocumentResponse(
+        doc_id=d["doc_id"],
+        filename=d["filename"],
+        file_type=d["file_type"],
+        status=d["status"],
+        metadata=DocumentMetadata(
+            doc_id=d["doc_id"],
+            filename=d["filename"],
+            file_type=d["file_type"],
+            page_count=d.get("page_count", 0),
+            chunk_count=len(d.get("chunks", [])),
+            text_length=len(d.get("raw_text", "")),
+            status=d["status"],
+            indexed=d.get("indexed", False),
+            indexed_chunks=d.get("indexed_chunks", 0),
+            processing_time=d.get("processing_time"),
+            created_at=d["created_at"],
+            updated_at=d.get("updated_at"),
+        ),
+        raw_text=d.get("raw_text") if include_text else None,
+        preview=d.get("raw_text", "")[:500] if d.get("raw_text") else None,
+    )
+
+
+@router.get("/{doc_id}/detail", response_model=DocumentDetailResponse)
+async def get_document_detail(doc_id: str):
+    """Get detailed document information including chunks and regions."""
+    if doc_id not in _documents:
+        raise HTTPException(status_code=404, detail=f"Document not found: {doc_id}")
+
+    d = _documents[doc_id]
+
+    return DocumentDetailResponse(
+        doc_id=d["doc_id"],
+        filename=d["filename"],
+        status=d["status"],
+        metadata=DocumentMetadata(
+            doc_id=d["doc_id"],
+            filename=d["filename"],
+            file_type=d["file_type"],
+            page_count=d.get("page_count", 0),
+            chunk_count=len(d.get("chunks", [])),
+            text_length=len(d.get("raw_text", "")),
+            status=d["status"],
+            indexed=d.get("indexed", False),
+            indexed_chunks=d.get("indexed_chunks", 0),
+            processing_time=d.get("processing_time"),
+            created_at=d["created_at"],
+            updated_at=d.get("updated_at"),
+        ),
+        chunks=[ChunkInfo(**c) for c in d.get("chunks", [])],
+        ocr_regions=[OCRRegionInfo(**r) for r in d.get("ocr_regions", []) if isinstance(r, dict)],
+        layout_regions=[LayoutRegionInfo(**r) for r in d.get("layout_regions", []) if isinstance(r, dict)],
+    )
+
+
+@router.get("/{doc_id}/chunks", response_model=ChunksResponse)
+async def get_document_chunks(
+    doc_id: str,
+    page: Optional[int] = Query(None, description="Filter by page number"),
+    chunk_type: Optional[str] = Query(None, description="Filter by chunk type"),
+):
+    """Get all chunks for a document."""
+    if doc_id not in _documents:
+        raise HTTPException(status_code=404, detail=f"Document not found: {doc_id}")
+
+    d = _documents[doc_id]
+    chunks = d.get("chunks", [])
+
+    # Apply filters
+    if page is not None:
+        chunks = [c for c in chunks if c.get("page_num") == page]
+    if chunk_type:
+        chunks = [c for c in chunks if c.get("chunk_type") == chunk_type]
+
+    return ChunksResponse(
+        doc_id=doc_id,
+        total_chunks=len(chunks),
+        chunks=[ChunkInfo(**c) for c in chunks],
+    )
+
+
+@router.post("/{doc_id}/process")
+async def process_document(
+    doc_id: str,
+    background_tasks: BackgroundTasks,
+    force: bool = Query(False, description="Force reprocessing"),
+):
+    """Trigger document processing."""
+    if doc_id not in _documents:
+        raise HTTPException(status_code=404, detail=f"Document not found: {doc_id}")
+
+    d = _documents[doc_id]
+
+    if d["status"] == DocumentStatus.PROCESSING:
+        raise HTTPException(status_code=400, detail="Document is already being processed")
+
+    if d["status"] == DocumentStatus.COMPLETED and not force:
+        raise HTTPException(
+            status_code=400,
+            detail="Document already processed. Use force=true to reprocess."
+        )
+
+    file_path = Path(d["file_path"])
+    if not file_path.exists():
+        raise HTTPException(status_code=404, detail="Document file not found")
+
+    background_tasks.add_task(process_document_task, doc_id, file_path, d["file_type"])
+    _documents[doc_id]["status"] = DocumentStatus.PROCESSING
+
+    return {"doc_id": doc_id, "status": "processing", "message": "Processing started"}
+
+
+@router.delete("/{doc_id}")
+async def delete_document(doc_id: str):
+    """Delete a document."""
+    if doc_id not in _documents:
+        raise HTTPException(status_code=404, detail=f"Document not found: {doc_id}")
+
+    d = _documents[doc_id]
+
+    # Delete file
+    file_path = Path(d["file_path"])
+    if file_path.exists():
+        file_path.unlink()
+
+    # Remove from store
+    del _documents[doc_id]
+
+    return {"doc_id": doc_id, "status": "deleted", "message": "Document deleted successfully"}
+
+
+@router.post("/{doc_id}/index", response_model=IndexResponse)
+async def index_document(doc_id: str, force_reindex: bool = Query(False)):
+    """Index a document to the RAG vector store."""
+    if doc_id not in _documents:
+        raise HTTPException(status_code=404, detail=f"Document not found: {doc_id}")
+
+    d = _documents[doc_id]
+
+    if d["status"] != DocumentStatus.COMPLETED:
+        raise HTTPException(
+            status_code=400,
+            detail=f"Document not ready for indexing. Current status: {d['status']}"
+        )
+
+    if d.get("indexed") and not force_reindex:
+        return IndexResponse(
+            doc_id=doc_id,
+            status="already_indexed",
+            chunks_indexed=d.get("indexed_chunks", 0),
+            message="Document already indexed. Use force_reindex=true to reindex."
+        )
+
+    try:
+        # Try to use actual indexer
+        from src.rag.indexer import DocumentIndexer
+        from src.rag.embeddings import get_embedding_model
+        from src.rag.store import get_vector_store
+
+        embeddings = get_embedding_model()
+        store = get_vector_store()
+        indexer = DocumentIndexer(embeddings, store)
+
+        # Index chunks
+        chunks_to_index = d.get("chunks", [])
+        indexed_count = 0
+
+        for chunk in chunks_to_index:
+            try:
+                indexer.index_chunk(
+                    text=chunk["text"],
+                    document_id=doc_id,
+                    chunk_id=chunk["chunk_id"],
+                    metadata={
+                        "filename": d["filename"],
+                        "page_num": chunk.get("page_num"),
+                        "chunk_type": chunk.get("chunk_type", "text"),
+                    }
+                )
+                indexed_count += 1
+            except Exception as e:
+                logger.warning(f"Failed to index chunk {chunk['chunk_id']}: {e}")
+
+        _documents[doc_id]["indexed"] = True
+        _documents[doc_id]["indexed_chunks"] = indexed_count
+        _documents[doc_id]["status"] = DocumentStatus.INDEXED
+
+        return IndexResponse(
+            doc_id=doc_id,
+            status="indexed",
+            chunks_indexed=indexed_count,
+            message=f"Successfully indexed {indexed_count} chunks"
+        )
+
+    except Exception as e:
+        logger.error(f"Indexing failed for {doc_id}: {e}")
+        raise HTTPException(status_code=500, detail=f"Indexing failed: {str(e)}")
+
+
+@router.post("/batch-index", response_model=BatchIndexResponse)
+async def batch_index_documents(request: BatchIndexRequest):
+    """Batch index multiple documents."""
+    results = []
+    successful = 0
+    failed = 0
+
+    for doc_id in request.doc_ids:
+        try:
+            result = await index_document(doc_id, request.force_reindex)
+            results.append(result)
+            if result.status in ["indexed", "already_indexed"]:
+                successful += 1
+            else:
+                failed += 1
+        except HTTPException as e:
+            results.append(IndexResponse(
+                doc_id=doc_id,
+                status="error",
+                chunks_indexed=0,
+                message=e.detail
+            ))
+            failed += 1
+
+    return BatchIndexResponse(
+        total_requested=len(request.doc_ids),
+        successful=successful,
+        failed=failed,
+        results=results
+    )
+
+
+# Export document store for other modules
+def get_document_store():
+    """Get the in-memory document store."""
+    return _documents

api/routes/rag.py

@@ -0,0 +1,415 @@
+"""
+SPARKNET RAG API Routes
+Endpoints for RAG queries, search, and indexing management.
+"""
+
+from fastapi import APIRouter, HTTPException, Query, Depends
+from fastapi.responses import StreamingResponse
+from typing import List, Optional
+from pathlib import Path
+from datetime import datetime
+import time
+import json
+import sys
+import asyncio
+
+# Add project root to path
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from api.schemas import (
+    QueryRequest, RAGResponse, Citation, QueryPlan, QueryIntentType,
+    SearchRequest, SearchResponse, SearchResult,
+    StoreStatus, CollectionInfo
+)
+from loguru import logger
+
+router = APIRouter()
+
+# Simple in-memory cache for query results
+_query_cache = {}
+CACHE_TTL_SECONDS = 3600  # 1 hour
+
+
+def get_cache_key(query: str, doc_ids: Optional[List[str]]) -> str:
+    """Generate cache key for query."""
+    import hashlib
+    doc_str = ",".join(sorted(doc_ids)) if doc_ids else "all"
+    content = f"{query}:{doc_str}"
+    return hashlib.md5(content.encode()).hexdigest()
+
+
+def get_cached_response(cache_key: str) -> Optional[RAGResponse]:
+    """Get cached response if valid."""
+    if cache_key in _query_cache:
+        cached = _query_cache[cache_key]
+        if time.time() - cached["timestamp"] < CACHE_TTL_SECONDS:
+            response = cached["response"]
+            response.from_cache = True
+            return response
+        else:
+            del _query_cache[cache_key]
+    return None
+
+
+def cache_response(cache_key: str, response: RAGResponse):
+    """Cache a query response."""
+    _query_cache[cache_key] = {
+        "response": response,
+        "timestamp": time.time()
+    }
+    # Limit cache size
+    if len(_query_cache) > 1000:
+        oldest_key = min(_query_cache, key=lambda k: _query_cache[k]["timestamp"])
+        del _query_cache[oldest_key]
+
+
+def _get_rag_system():
+    """Get or initialize the RAG system."""
+    try:
+        from src.rag.agentic.orchestrator import AgenticRAG, RAGConfig
+
+        config = RAGConfig(
+            model_name="llama3.2:latest",
+            max_revision_attempts=2,
+            retrieval_top_k=10,
+            final_top_k=5,
+            min_confidence=0.5,
+        )
+        return AgenticRAG(config)
+    except Exception as e:
+        logger.error(f"Failed to initialize RAG system: {e}")
+        return None
+
+
+@router.post("/query", response_model=RAGResponse)
+async def query_documents(request: QueryRequest):
+    """
+    Execute a RAG query across indexed documents.
+
+    The query goes through the 5-agent pipeline:
+    1. QueryPlanner - Intent classification and query decomposition
+    2. Retriever - Hybrid dense+sparse search
+    3. Reranker - Cross-encoder reranking with MMR
+    4. Synthesizer - Answer generation with citations
+    5. Critic - Hallucination detection and validation
+    """
+    start_time = time.time()
+
+    # Check cache if enabled
+    if request.use_cache:
+        cache_key = get_cache_key(request.query, request.doc_ids)
+        cached = get_cached_response(cache_key)
+        if cached:
+            cached.latency_ms = (time.time() - start_time) * 1000
+            return cached
+
+    try:
+        # Initialize RAG system
+        rag = _get_rag_system()
+        if not rag:
+            raise HTTPException(status_code=503, detail="RAG system not available")
+
+        # Build filters
+        filters = {}
+        if request.doc_ids:
+            filters["document_id"] = {"$in": request.doc_ids}
+
+        # Execute query
+        logger.info(f"Executing RAG query: {request.query[:50]}...")
+
+        result = rag.query(
+            query=request.query,
+            filters=filters if filters else None,
+            top_k=request.top_k,
+        )
+
+        # Build response
+        citations = []
+        for i, source in enumerate(result.get("sources", [])):
+            citations.append(Citation(
+                citation_id=i + 1,
+                doc_id=source.get("document_id", "unknown"),
+                document_name=source.get("filename", source.get("document_id", "unknown")),
+                chunk_id=source.get("chunk_id", f"chunk_{i}"),
+                chunk_text=source.get("text", "")[:300],
+                page_num=source.get("page_num"),
+                relevance_score=source.get("relevance_score", source.get("score", 0.0)),
+                bbox=source.get("bbox"),
+            ))
+
+        # Query plan info
+        query_plan = None
+        if "plan" in result:
+            plan = result["plan"]
+            query_plan = QueryPlan(
+                intent=QueryIntentType(plan.get("intent", "factoid").lower()),
+                sub_queries=plan.get("sub_queries", []),
+                keywords=plan.get("keywords", []),
+                strategy=plan.get("strategy", "hybrid"),
+            )
+
+        response = RAGResponse(
+            query=request.query,
+            answer=result.get("answer", "I could not find an answer to your question."),
+            confidence=result.get("confidence", 0.0),
+            citations=citations,
+            source_count=len(citations),
+            query_plan=query_plan,
+            from_cache=False,
+            validation=result.get("validation"),
+            latency_ms=(time.time() - start_time) * 1000,
+            revision_count=result.get("revision_count", 0),
+        )
+
+        # Cache successful responses
+        if request.use_cache and response.confidence >= request.min_confidence:
+            cache_key = get_cache_key(request.query, request.doc_ids)
+            cache_response(cache_key, response)
+
+        return response
+
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"RAG query failed: {e}")
+        raise HTTPException(status_code=500, detail=f"Query failed: {str(e)}")
+
+
+@router.post("/query/stream")
+async def query_documents_stream(request: QueryRequest):
+    """
+    Stream RAG response for real-time updates.
+
+    Returns Server-Sent Events (SSE) with partial responses.
+    """
+    async def generate():
+        try:
+            # Initialize RAG system
+            rag = _get_rag_system()
+            if not rag:
+                yield f"data: {json.dumps({'error': 'RAG system not available'})}\n\n"
+                return
+
+            # Send planning stage
+            yield f"data: {json.dumps({'stage': 'planning', 'message': 'Analyzing query...'})}\n\n"
+            await asyncio.sleep(0.1)
+
+            # Build filters
+            filters = {}
+            if request.doc_ids:
+                filters["document_id"] = {"$in": request.doc_ids}
+
+            # Send retrieval stage
+            yield f"data: {json.dumps({'stage': 'retrieving', 'message': 'Searching documents...'})}\n\n"
+
+            # Execute query (in chunks if streaming supported)
+            result = rag.query(
+                query=request.query,
+                filters=filters if filters else None,
+                top_k=request.top_k,
+            )
+
+            # Send sources
+            yield f"data: {json.dumps({'stage': 'sources', 'count': len(result.get('sources', []))})}\n\n"
+
+            # Send synthesis stage
+            yield f"data: {json.dumps({'stage': 'synthesizing', 'message': 'Generating answer...'})}\n\n"
+
+            # Stream answer in chunks
+            answer = result.get("answer", "")
+            chunk_size = 50
+            for i in range(0, len(answer), chunk_size):
+                chunk = answer[i:i+chunk_size]
+                yield f"data: {json.dumps({'stage': 'answer', 'chunk': chunk})}\n\n"
+                await asyncio.sleep(0.02)
+
+            # Send final result
+            citations = []
+            for i, source in enumerate(result.get("sources", [])):
+                citations.append({
+                    "citation_id": i + 1,
+                    "doc_id": source.get("document_id", "unknown"),
+                    "chunk_text": source.get("text", "")[:200],
+                    "relevance_score": source.get("score", 0.0),
+                })
+
+            final = {
+                "stage": "complete",
+                "confidence": result.get("confidence", 0.0),
+                "citations": citations,
+                "validation": result.get("validation"),
+            }
+            yield f"data: {json.dumps(final)}\n\n"
+
+        except Exception as e:
+            logger.error(f"Streaming query failed: {e}")
+            yield f"data: {json.dumps({'error': str(e)})}\n\n"
+
+    return StreamingResponse(
+        generate(),
+        media_type="text/event-stream",
+        headers={
+            "Cache-Control": "no-cache",
+            "Connection": "keep-alive",
+        }
+    )
+
+
+@router.post("/search", response_model=SearchResponse)
+async def search_documents(request: SearchRequest):
+    """
+    Semantic search across indexed documents.
+
+    Returns matching chunks without answer synthesis.
+    """
+    start_time = time.time()
+
+    try:
+        from src.rag.store import get_vector_store
+        from src.rag.embeddings import get_embedding_model
+
+        store = get_vector_store()
+        embeddings = get_embedding_model()
+
+        # Generate query embedding
+        query_embedding = embeddings.embed_query(request.query)
+
+        # Build filter
+        where_filter = None
+        if request.doc_ids:
+            where_filter = {"document_id": {"$in": request.doc_ids}}
+
+        # Search
+        results = store.similarity_search_with_score(
+            query_embedding=query_embedding,
+            k=request.top_k,
+            where=where_filter,
+        )
+
+        # Filter by minimum score
+        search_results = []
+        for doc, score in results:
+            if score >= request.min_score:
+                search_results.append(SearchResult(
+                    chunk_id=doc.metadata.get("chunk_id", "unknown"),
+                    doc_id=doc.metadata.get("document_id", "unknown"),
+                    document_name=doc.metadata.get("filename", "unknown"),
+                    text=doc.page_content,
+                    score=score,
+                    page_num=doc.metadata.get("page_num"),
+                    chunk_type=doc.metadata.get("chunk_type", "text"),
+                ))
+
+        return SearchResponse(
+            query=request.query,
+            total_results=len(search_results),
+            results=search_results,
+            latency_ms=(time.time() - start_time) * 1000,
+        )
+
+    except Exception as e:
+        logger.error(f"Search failed: {e}")
+        # Fallback: return empty results
+        return SearchResponse(
+            query=request.query,
+            total_results=0,
+            results=[],
+            latency_ms=(time.time() - start_time) * 1000,
+        )
+
+
+@router.get("/store/status", response_model=StoreStatus)
+async def get_store_status():
+    """Get vector store status and statistics."""
+    try:
+        from src.rag.store import get_vector_store
+
+        store = get_vector_store()
+
+        # Get collection info
+        collection = store._collection
+        count = collection.count()
+
+        # Get unique documents
+        all_metadata = collection.get(include=["metadatas"])
+        doc_ids = set()
+        for meta in all_metadata.get("metadatas", []):
+            if meta and "document_id" in meta:
+                doc_ids.add(meta["document_id"])
+
+        collections = [CollectionInfo(
+            name=store.collection_name,
+            document_count=len(doc_ids),
+            chunk_count=count,
+            embedding_dimension=store.embedding_dimension if hasattr(store, 'embedding_dimension') else 1024,
+        )]
+
+        return StoreStatus(
+            status="healthy",
+            collections=collections,
+            total_documents=len(doc_ids),
+            total_chunks=count,
+        )
+
+    except Exception as e:
+        logger.error(f"Store status check failed: {e}")
+        return StoreStatus(
+            status="error",
+            collections=[],
+            total_documents=0,
+            total_chunks=0,
+        )
+
+
+@router.delete("/store/collection/{collection_name}")
+async def clear_collection(collection_name: str, confirm: bool = Query(False)):
+    """Clear a vector store collection (dangerous operation)."""
+    if not confirm:
+        raise HTTPException(
+            status_code=400,
+            detail="This operation will delete all data. Set confirm=true to proceed."
+        )
+
+    try:
+        from src.rag.store import get_vector_store
+
+        store = get_vector_store()
+        if store.collection_name != collection_name:
+            raise HTTPException(status_code=404, detail=f"Collection not found: {collection_name}")
+
+        # Clear collection
+        store._collection.delete(where={})
+
+        return {"status": "cleared", "collection": collection_name, "message": "Collection cleared successfully"}
+
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Collection clear failed: {e}")
+        raise HTTPException(status_code=500, detail=f"Clear failed: {str(e)}")
+
+
+@router.get("/cache/stats")
+async def get_cache_stats():
+    """Get query cache statistics."""
+    current_time = time.time()
+    valid_entries = sum(
+        1 for v in _query_cache.values()
+        if current_time - v["timestamp"] < CACHE_TTL_SECONDS
+    )
+
+    return {
+        "total_entries": len(_query_cache),
+        "valid_entries": valid_entries,
+        "expired_entries": len(_query_cache) - valid_entries,
+        "ttl_seconds": CACHE_TTL_SECONDS,
+    }
+
+
+@router.delete("/cache")
+async def clear_cache():
+    """Clear the query cache."""
+    count = len(_query_cache)
+    _query_cache.clear()
+    return {"status": "cleared", "entries_removed": count}

api/schemas.py

@@ -0,0 +1,302 @@
+"""
+SPARKNET API Schemas
+Pydantic models for request/response validation.
+"""
+
+from pydantic import BaseModel, Field, ConfigDict
+from typing import List, Dict, Any, Optional
+from datetime import datetime
+from enum import Enum
+
+
+# ==================== Enums ====================
+
+class DocumentStatus(str, Enum):
+    PENDING = "pending"
+    PROCESSING = "processing"
+    COMPLETED = "completed"
+    INDEXED = "indexed"
+    ERROR = "error"
+
+
+class QueryIntentType(str, Enum):
+    FACTOID = "factoid"
+    COMPARISON = "comparison"
+    AGGREGATION = "aggregation"
+    CAUSAL = "causal"
+    PROCEDURAL = "procedural"
+    DEFINITION = "definition"
+    LIST = "list"
+    MULTI_HOP = "multi_hop"
+
+
+class AnswerFormat(str, Enum):
+    PROSE = "prose"
+    BULLET_POINTS = "bullet_points"
+    TABLE = "table"
+    STEP_BY_STEP = "step_by_step"
+
+
+# ==================== Document Schemas ====================
+
+class DocumentUploadResponse(BaseModel):
+    """Response after uploading a document."""
+    model_config = ConfigDict(from_attributes=True)
+
+    doc_id: str = Field(..., description="Unique document identifier")
+    filename: str = Field(..., description="Original filename")
+    status: DocumentStatus = Field(..., description="Document status")
+    message: str = Field(..., description="Status message")
+    created_at: datetime = Field(default_factory=datetime.now)
+
+
+class DocumentMetadata(BaseModel):
+    """Document metadata information."""
+    model_config = ConfigDict(from_attributes=True)
+
+    doc_id: str
+    filename: str
+    file_type: str
+    page_count: int = 0
+    chunk_count: int = 0
+    text_length: int = 0
+    status: DocumentStatus
+    indexed: bool = False
+    indexed_chunks: int = 0
+    processing_time: Optional[float] = None
+    created_at: datetime
+    updated_at: Optional[datetime] = None
+
+
+class DocumentResponse(BaseModel):
+    """Full document response with metadata."""
+    model_config = ConfigDict(from_attributes=True)
+
+    doc_id: str
+    filename: str
+    file_type: str
+    status: DocumentStatus
+    metadata: DocumentMetadata
+    raw_text: Optional[str] = Field(None, description="Full extracted text (if requested)")
+    preview: Optional[str] = Field(None, description="Text preview (first 500 chars)")
+
+
+class ChunkInfo(BaseModel):
+    """Information about a document chunk."""
+    model_config = ConfigDict(from_attributes=True)
+
+    chunk_id: str
+    doc_id: str
+    text: str
+    chunk_type: str = "text"
+    page_num: Optional[int] = None
+    confidence: float = 1.0
+    bbox: Optional[Dict[str, float]] = None
+    metadata: Dict[str, Any] = Field(default_factory=dict)
+
+
+class ChunksResponse(BaseModel):
+    """Response containing document chunks."""
+    doc_id: str
+    total_chunks: int
+    chunks: List[ChunkInfo]
+
+
+class OCRRegionInfo(BaseModel):
+    """OCR region information."""
+    region_id: str
+    text: str
+    confidence: float
+    page_num: int
+    bbox: Dict[str, float]
+
+
+class LayoutRegionInfo(BaseModel):
+    """Layout region information."""
+    region_id: str
+    region_type: str
+    confidence: float
+    page_num: int
+    bbox: Dict[str, float]
+
+
+class DocumentDetailResponse(BaseModel):
+    """Detailed document response with all extracted data."""
+    doc_id: str
+    filename: str
+    status: DocumentStatus
+    metadata: DocumentMetadata
+    chunks: List[ChunkInfo]
+    ocr_regions: List[OCRRegionInfo] = Field(default_factory=list)
+    layout_regions: List[LayoutRegionInfo] = Field(default_factory=list)
+
+
+# ==================== RAG Query Schemas ====================
+
+class QueryRequest(BaseModel):
+    """RAG query request."""
+    query: str = Field(..., min_length=1, max_length=2000, description="Query text")
+    doc_ids: Optional[List[str]] = Field(None, description="Filter by document IDs")
+    top_k: int = Field(5, ge=1, le=20, description="Number of chunks to retrieve")
+    answer_format: AnswerFormat = Field(AnswerFormat.PROSE, description="Desired answer format")
+    include_sources: bool = Field(True, description="Include source citations")
+    min_confidence: float = Field(0.5, ge=0.0, le=1.0, description="Minimum confidence threshold")
+    use_cache: bool = Field(True, description="Use cached results if available")
+
+
+class Citation(BaseModel):
+    """Citation/source reference."""
+    citation_id: int = Field(..., description="Citation number [1], [2], etc.")
+    doc_id: str
+    document_name: str
+    chunk_id: str
+    chunk_text: str
+    page_num: Optional[int] = None
+    relevance_score: float
+    bbox: Optional[Dict[str, float]] = None
+
+
+class QueryPlan(BaseModel):
+    """Query planning information."""
+    intent: QueryIntentType
+    sub_queries: List[str] = Field(default_factory=list)
+    keywords: List[str] = Field(default_factory=list)
+    strategy: str = "hybrid"
+
+
+class RAGResponse(BaseModel):
+    """Complete RAG response."""
+    query: str
+    answer: str
+    confidence: float = Field(..., ge=0.0, le=1.0)
+    citations: List[Citation] = Field(default_factory=list)
+    source_count: int = 0
+    query_plan: Optional[QueryPlan] = None
+    from_cache: bool = False
+    validation: Optional[Dict[str, Any]] = None
+    latency_ms: Optional[float] = None
+    revision_count: int = 0
+
+
+class SearchRequest(BaseModel):
+    """Semantic search request."""
+    query: str = Field(..., min_length=1, max_length=1000)
+    doc_ids: Optional[List[str]] = None
+    top_k: int = Field(10, ge=1, le=50)
+    min_score: float = Field(0.0, ge=0.0, le=1.0)
+
+
+class SearchResult(BaseModel):
+    """Single search result."""
+    chunk_id: str
+    doc_id: str
+    document_name: str
+    text: str
+    score: float
+    page_num: Optional[int] = None
+    chunk_type: str = "text"
+
+
+class SearchResponse(BaseModel):
+    """Search response with results."""
+    query: str
+    total_results: int
+    results: List[SearchResult]
+    latency_ms: float
+
+
+# ==================== Indexing Schemas ====================
+
+class IndexRequest(BaseModel):
+    """Request to index a document."""
+    doc_id: str = Field(..., description="Document ID to index")
+    force_reindex: bool = Field(False, description="Force reindexing if already indexed")
+
+
+class IndexResponse(BaseModel):
+    """Indexing response."""
+    doc_id: str
+    status: str
+    chunks_indexed: int
+    message: str
+
+
+class BatchIndexRequest(BaseModel):
+    """Batch indexing request."""
+    doc_ids: List[str]
+    force_reindex: bool = False
+
+
+class BatchIndexResponse(BaseModel):
+    """Batch indexing response."""
+    total_requested: int
+    successful: int
+    failed: int
+    results: List[IndexResponse]
+
+
+# ==================== System Schemas ====================
+
+class HealthResponse(BaseModel):
+    """Health check response."""
+    status: str = Field(..., description="healthy, degraded, or unhealthy")
+    version: str
+    components: Dict[str, bool]
+
+
+class SystemStatus(BaseModel):
+    """Detailed system status."""
+    status: str
+    version: str
+    uptime_seconds: float
+    components: Dict[str, bool]
+    statistics: Dict[str, Any]
+    models: Dict[str, str]
+
+
+class CollectionInfo(BaseModel):
+    """Vector store collection information."""
+    name: str
+    document_count: int
+    chunk_count: int
+    embedding_dimension: int
+
+
+class StoreStatus(BaseModel):
+    """Vector store status."""
+    status: str
+    collections: List[CollectionInfo]
+    total_documents: int
+    total_chunks: int
+
+
+# ==================== Authentication Schemas ====================
+
+class UserCreate(BaseModel):
+    """User creation request."""
+    username: str = Field(..., min_length=3, max_length=50)
+    email: str
+    password: str = Field(..., min_length=8)
+
+
+class UserResponse(BaseModel):
+    """User response (no password)."""
+    user_id: str
+    username: str
+    email: str
+    is_active: bool = True
+    created_at: datetime
+
+
+class Token(BaseModel):
+    """JWT token response."""
+    access_token: str
+    token_type: str = "bearer"
+    expires_in: int
+
+
+class TokenData(BaseModel):
+    """Token payload data."""
+    username: Optional[str] = None
+    user_id: Optional[str] = None
+    scopes: List[str] = Field(default_factory=list)

config/document.yaml

@@ -0,0 +1,147 @@
+# SPARKNET Document Processing Configuration
+# ===========================================
+
+# OCR Configuration
+ocr:
+  # Engine selection: "paddleocr" (default) or "tesseract"
+  engine: paddleocr
+
+  # PaddleOCR settings
+  paddleocr:
+    lang: en
+    use_gpu: false
+    det_db_thresh: 0.3
+    det_db_box_thresh: 0.5
+    rec_algorithm: CRNN
+    show_log: false
+
+  # Tesseract settings
+  tesseract:
+    lang: eng
+    config: "--psm 3"  # Page segmentation mode
+    oem: 3  # OCR Engine mode (LSTM)
+
+  # Preprocessing
+  preprocessing:
+    deskew: true
+    denoise: false
+    contrast_enhance: false
+
+# Layout Detection Configuration
+layout:
+  # Detection method: "rule_based" (default) or "model_based"
+  method: rule_based
+
+  # Rule-based settings
+  rule_based:
+    merge_threshold: 20  # Pixels to merge nearby regions
+    column_detection: true
+    min_region_area: 100
+
+  # Confidence thresholds
+  thresholds:
+    text: 0.5
+    title: 0.7
+    table: 0.6
+    figure: 0.6
+    list: 0.5
+
+# Reading Order Configuration
+reading_order:
+  # Reconstruction method: "rule_based" (default)
+  method: rule_based
+
+  # Column detection
+  column_gap_threshold: 50  # Minimum gap between columns
+  reading_direction: ltr  # Left-to-right
+
+  # Line grouping
+  line_height_tolerance: 0.5
+
+# Chunking Configuration
+chunking:
+  # Chunk size limits
+  target_size: 512  # Target tokens per chunk
+  max_size: 1024  # Maximum tokens per chunk
+  min_size: 50  # Minimum tokens per chunk
+
+  # Overlap for context
+  overlap_size: 50  # Tokens to overlap between chunks
+
+  # Semantic chunking
+  semantic_boundaries: true
+  respect_paragraphs: true
+  respect_sections: true
+
+# Grounding/Evidence Configuration
+grounding:
+  # Image cropping for evidence
+  include_images: true
+  crop_padding: 10  # Pixels around regions
+  max_image_size: 512
+  image_format: PNG  # PNG or JPEG
+  image_quality: 85  # JPEG quality
+
+  # Snippet settings
+  max_snippet_length: 200
+  include_context: true
+
+# Pipeline Configuration
+pipeline:
+  # PDF rendering
+  render_dpi: 300
+
+  # Caching
+  enable_caching: true
+  cache_directory: ./data/cache
+
+  # Processing options
+  parallel_pages: false
+  max_pages: null  # Limit pages (null for all)
+
+  # Output options
+  include_ocr_regions: true
+  include_layout_regions: true
+  generate_full_text: true
+
+# Validation Configuration
+validation:
+  # Critic settings
+  critic:
+    confidence_threshold: 0.7
+    evidence_required: true
+    strict_mode: false
+    max_fields_per_request: 10
+
+  # Verifier settings
+  verifier:
+    fuzzy_match: true
+    case_sensitive: false
+    min_match_ratio: 0.6
+    strong_threshold: 0.9
+    moderate_threshold: 0.7
+    weak_threshold: 0.5
+
+# LLM Configuration for DocumentAgent
+agent:
+  # Ollama settings
+  ollama_base_url: http://localhost:11434
+  default_model: llama3.2:3b
+
+  # Model routing by complexity
+  model_routing:
+    simple: llama3.2:1b
+    standard: llama3.2:3b
+    complex: llama3.1:8b
+    analysis: llama3.1:70b  # For heavy analysis (optional)
+
+  # Agent behavior
+  max_iterations: 10
+  temperature: 0.1
+  timeout: 120  # Seconds
+
+# Logging Configuration
+logging:
+  level: INFO  # DEBUG, INFO, WARNING, ERROR
+  format: "{time} | {level} | {message}"
+  file: null  # Log file path (null for stderr only)

config/rag.yaml

@@ -0,0 +1,141 @@
+# SPARKNET RAG Configuration
+# ===========================
+
+# Vector Store Configuration
+vector_store:
+  # Store type: "chromadb" (default)
+  type: chromadb
+
+  # ChromaDB settings
+  chromadb:
+    persist_directory: ./data/vectorstore
+    collection_name: sparknet_documents
+    anonymized_telemetry: false
+
+  # Search settings
+  default_top_k: 5
+  similarity_threshold: 0.7
+
+# Embedding Configuration
+embeddings:
+  # Adapter type: "ollama" (default) or "openai"
+  adapter_type: ollama
+
+  # Ollama settings (local, default)
+  ollama:
+    base_url: http://localhost:11434
+    model: nomic-embed-text  # Options: nomic-embed-text, mxbai-embed-large, all-minilm
+
+  # OpenAI settings (optional, feature-flagged)
+  openai:
+    enabled: false
+    model: text-embedding-3-small  # Options: text-embedding-3-small, text-embedding-3-large
+    # api_key: ${OPENAI_API_KEY}  # Use env var
+
+  # Common settings
+  batch_size: 32
+  timeout: 60
+
+  # Caching
+  enable_cache: true
+  cache_directory: ./data/embedding_cache
+
+# Indexer Configuration
+indexer:
+  # Batch processing
+  batch_size: 32
+
+  # Metadata to index
+  include_bbox: true
+  include_page: true
+  include_chunk_type: true
+
+  # Filtering
+  skip_empty_chunks: true
+  min_chunk_length: 10
+
+# Retriever Configuration
+retriever:
+  # Search parameters
+  default_top_k: 5
+  similarity_threshold: 0.7
+  max_results: 20
+
+  # Reranking (future)
+  enable_reranking: false
+  rerank_top_k: 10
+
+  # Evidence settings
+  include_evidence: true
+  evidence_snippet_length: 200
+
+# Generator Configuration
+generator:
+  # LLM provider: "ollama" (default) or "openai"
+  llm_provider: ollama
+
+  # Ollama settings
+  ollama:
+    base_url: http://localhost:11434
+    model: llama3.2:3b  # Options: llama3.2:3b, llama3.1:8b, mistral
+
+  # OpenAI settings (optional)
+  openai:
+    model: gpt-4o-mini  # Options: gpt-4o-mini, gpt-4o
+    # api_key: ${OPENAI_API_KEY}  # Use env var
+
+  # Generation settings
+  temperature: 0.1
+  max_tokens: 1024
+  timeout: 120
+
+  # Citation settings
+  require_citations: true
+  citation_format: "[{index}]"
+
+  # Abstention settings
+  abstain_on_low_confidence: true
+  confidence_threshold: 0.6
+
+# Query Processing
+query:
+  # Query expansion
+  expand_queries: false
+  max_expansions: 3
+
+  # Hybrid search (future)
+  enable_hybrid: false
+  keyword_weight: 0.3
+  semantic_weight: 0.7
+
+# Metadata Filtering
+filters:
+  # Supported filter types
+  supported:
+    - document_id
+    - chunk_type
+    - page
+    - confidence_min
+
+  # Default filters (applied to all queries)
+  defaults: {}
+
+# Performance Settings
+performance:
+  # Connection pooling
+  max_connections: 10
+
+  # Timeouts
+  embedding_timeout: 60
+  search_timeout: 30
+  generation_timeout: 120
+
+  # Caching
+  query_cache_enabled: true
+  query_cache_ttl: 3600  # Seconds
+
+# Logging
+logging:
+  level: INFO
+  include_queries: false  # Log user queries (privacy consideration)
+  include_latency: true

configs/rag.yaml

@@ -0,0 +1,201 @@
+# RAG (Retrieval-Augmented Generation) Configuration
+# SPARKNET Document Intelligence Integration
+
+# =============================================================================
+# Vector Store Settings
+# =============================================================================
+vector_store:
+  # Store type: "chroma" (default) or "memory" (for testing)
+  type: chroma
+
+  # ChromaDB settings
+  chroma:
+    # Persistence directory for vector store
+    persist_directory: "./.sparknet/chroma_db"
+
+    # Collection name for document chunks
+    collection_name: "sparknet_documents"
+
+    # Distance metric: "cosine" (default), "l2", or "ip"
+    distance_metric: cosine
+
+    # Anonymized telemetry (set to false to disable)
+    anonymized_telemetry: false
+
+# =============================================================================
+# Embedding Settings
+# =============================================================================
+embeddings:
+  # Provider: "ollama" (default, local) or "openai" (cloud, requires API key)
+  provider: ollama
+
+  # Ollama settings (local, privacy-preserving)
+  ollama:
+    # Model name for embeddings
+    # Recommended: nomic-embed-text (768 dims) or mxbai-embed-large (1024 dims)
+    model: nomic-embed-text
+
+    # Ollama server URL
+    base_url: "http://localhost:11434"
+
+    # Request timeout in seconds
+    timeout: 30
+
+  # OpenAI settings (cloud, disabled by default)
+  openai:
+    # IMPORTANT: OpenAI is disabled by default for privacy
+    # Set to true only if you explicitly need cloud embeddings
+    enabled: false
+
+    # Model name (if enabled)
+    model: text-embedding-3-small
+
+    # API key (from environment variable OPENAI_API_KEY)
+    # Never store API keys in config files
+    api_key_env: OPENAI_API_KEY
+
+  # Caching settings
+  cache:
+    # Enable embedding cache for faster re-processing
+    enabled: true
+
+    # Maximum cache entries
+    max_entries: 10000
+
+# =============================================================================
+# Indexer Settings
+# =============================================================================
+indexer:
+  # Batch size for embedding generation
+  batch_size: 32
+
+  # Include bounding box metadata
+  include_bbox: true
+
+  # Include page numbers
+  include_page: true
+
+  # Include chunk type labels
+  include_chunk_type: true
+
+  # Skip empty chunks
+  skip_empty_chunks: true
+
+  # Minimum chunk text length (characters)
+  min_chunk_length: 10
+
+# =============================================================================
+# Retriever Settings
+# =============================================================================
+retriever:
+  # Default number of results to return
+  default_top_k: 5
+
+  # Maximum results to return
+  max_results: 20
+
+  # Minimum similarity score (0.0 - 1.0)
+  # Chunks below this threshold are filtered out
+  similarity_threshold: 0.5
+
+  # Enable result reranking (experimental)
+  enable_reranking: false
+
+  # Number of results to rerank
+  rerank_top_k: 10
+
+  # Include evidence references in results
+  include_evidence: true
+
+  # Maximum snippet length in evidence
+  evidence_snippet_length: 200
+
+# =============================================================================
+# Generator Settings (Answer Generation)
+# =============================================================================
+generator:
+  # LLM provider for answer generation: "ollama" (default) or "openai"
+  provider: ollama
+
+  # Ollama settings (local)
+  ollama:
+    # Model for answer generation
+    # Recommended: llama3.2, mistral, or phi3
+    model: llama3.2
+
+    # Ollama server URL
+    base_url: "http://localhost:11434"
+
+    # Request timeout in seconds
+    timeout: 60
+
+    # Generation parameters
+    temperature: 0.1
+    max_tokens: 1024
+
+  # OpenAI settings (cloud, disabled by default)
+  openai:
+    enabled: false
+    model: gpt-4o-mini
+    api_key_env: OPENAI_API_KEY
+    temperature: 0.1
+    max_tokens: 1024
+
+  # Confidence settings
+  min_confidence: 0.5
+
+  # Abstention policy
+  # When true, the system will refuse to answer if confidence is too low
+  abstain_on_low_confidence: true
+  abstain_threshold: 0.3
+
+  # Maximum context length for LLM
+  max_context_length: 8000
+
+  # Require citations in answers
+  require_citations: true
+
+# =============================================================================
+# Document Intelligence Integration
+# =============================================================================
+document_intelligence:
+  # Parser settings
+  parser:
+    render_dpi: 200
+    max_pages: null  # null = no limit
+
+  # Extraction settings
+  extraction:
+    min_field_confidence: 0.5
+    abstain_on_low_confidence: true
+
+  # Grounding settings
+  grounding:
+    enable_crops: true
+    crop_output_dir: "./.sparknet/crops"
+
+# =============================================================================
+# Performance Settings
+# =============================================================================
+performance:
+  # Number of parallel workers for batch processing
+  num_workers: 4
+
+  # Chunk processing batch size
+  chunk_batch_size: 100
+
+  # Enable async processing where supported
+  async_enabled: true
+
+# =============================================================================
+# Logging Settings
+# =============================================================================
+logging:
+  # Log level: DEBUG, INFO, WARNING, ERROR
+  level: INFO
+
+  # Log RAG queries and results
+  log_queries: false
+
+  # Log embedding operations
+  log_embeddings: false

demo/README.md

@@ -0,0 +1,185 @@
+# SPARKNET Demo Application
+
+An interactive Streamlit demo showcasing SPARKNET's document intelligence capabilities.
+
+## Features
+
+- **📄 Document Processing**: Upload and process documents with OCR
+- **🔍 Field Extraction**: Extract structured data with evidence grounding
+- **💬 RAG Q&A**: Interactive question answering with citations
+- **🏷️ Classification**: Automatic document type detection
+- **📊 Analytics**: Processing statistics and insights
+- **🔬 Live Processing**: Real-time pipeline visualization
+- **📊 Document Comparison**: Compare multiple documents
+
+## Quick Start
+
+### 1. Install Dependencies
+
+```bash
+# From project root
+pip install -r demo/requirements.txt
+
+# Or install all SPARKNET dependencies
+pip install -r requirements.txt
+```
+
+### 2. Start Ollama (Optional, for live processing)
+
+```bash
+ollama serve
+
+# Pull required models
+ollama pull llama3.2:3b
+ollama pull nomic-embed-text
+```
+
+### 3. Run the Demo
+
+```bash
+# From project root
+streamlit run demo/app.py
+
+# Or with custom port
+streamlit run demo/app.py --server.port 8501
+```
+
+### 4. Open in Browser
+
+Navigate to http://localhost:8501
+
+## Demo Pages
+
+| Page | Description |
+|------|-------------|
+| **Home** | Overview and feature cards |
+| **Document Processing** | Upload/select documents for OCR processing |
+| **Field Extraction** | Extract structured fields with evidence |
+| **RAG Q&A** | Ask questions about indexed documents |
+| **Classification** | Classify document types |
+| **Analytics** | View processing statistics |
+| **Live Processing** | Watch pipeline in real-time |
+| **Interactive RAG** | Chat-style document Q&A |
+| **Document Comparison** | Compare documents side by side |
+
+## Sample Documents
+
+The demo uses patent pledge documents from the `Dataset/` folder:
+
+- Apple 11.11.2011.pdf
+- IBM 11.01.2005.pdf
+- Google 08.02.2012.pdf
+- And more...
+
+## Screenshots
+
+### Home Page
+```
+┌─────────────────────────────────────────┐
+│  🔥 SPARKNET                            │
+│  Agentic Document Intelligence Platform │
+├─────────────────────────────────────────┤
+│  [Doc Processing] [Extraction] [RAG]    │
+│                                         │
+│  Feature cards with gradients...        │
+└─────────────────────────────────────────┘
+```
+
+### RAG Q&A
+```
+┌─────────────────────────────────────────┐
+│  💬 Ask a question...                   │
+├─────────────────────────────────────────┤
+│  User: What patents are covered?        │
+│                                         │
+│  Assistant: Based on the documents...   │
+│  [📚 View Sources]                      │
+│    [1] Apple - Page 1: "..."            │
+│    [2] IBM - Page 2: "..."              │
+└─────────────────────────────────────────┘
+```
+
+## Configuration
+
+### Environment Variables
+
+```bash
+# Ollama URL (default: http://localhost:11434)
+export OLLAMA_BASE_URL=http://localhost:11434
+
+# ChromaDB path (default: ./data/vectorstore)
+export CHROMA_PERSIST_DIR=./data/vectorstore
+```
+
+### Streamlit Config
+
+Create `.streamlit/config.toml`:
+
+```toml
+[theme]
+primaryColor = "#FF6B6B"
+backgroundColor = "#FFFFFF"
+secondaryBackgroundColor = "#F0F2F6"
+textColor = "#262730"
+
+[server]
+maxUploadSize = 50
+```
+
+## Development
+
+### Adding New Pages
+
+1. Create a new file in `demo/pages/`:
+   ```
+   demo/pages/4_🆕_New_Feature.py
+   ```
+
+2. Follow the naming convention: `{order}_{emoji}_{name}.py`
+
+3. Import project modules:
+   ```python
+   import sys
+   from pathlib import Path
+   PROJECT_ROOT = Path(__file__).parent.parent.parent
+   sys.path.insert(0, str(PROJECT_ROOT))
+   ```
+
+### Customizing Styles
+
+Edit the CSS in `app.py`:
+
+```python
+st.markdown("""
+<style>
+    .main-header { ... }
+    .evidence-box { ... }
+</style>
+""", unsafe_allow_html=True)
+```
+
+## Troubleshooting
+
+### "ModuleNotFoundError: No module named 'src'"
+
+Make sure you're running from the project root:
+```bash
+cd /path/to/SPARKNET
+streamlit run demo/app.py
+```
+
+### Ollama Not Connected
+
+1. Check if Ollama is running: `curl http://localhost:11434/api/tags`
+2. Start Ollama: `ollama serve`
+
+### ChromaDB Errors
+
+Install ChromaDB:
+```bash
+pip install chromadb
+```
+
+## License
+
+Part of the SPARKNET project. See main LICENSE file.

demo/app.py

@@ -0,0 +1,944 @@
+"""
+SPARKNET Demo Application
+
+A Streamlit-based demo showcasing:
+- Document Processing Pipeline
+- Field Extraction with Evidence
+- RAG Search and Q&A
+- Document Classification
+- Evidence Visualization
+"""
+
+import streamlit as st
+import sys
+import os
+from pathlib import Path
+import json
+import time
+from datetime import datetime
+
+# Add project root to path
+PROJECT_ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+# Page configuration
+st.set_page_config(
+    page_title="SPARKNET Document Intelligence",
+    page_icon="🔥",
+    layout="wide",
+    initial_sidebar_state="expanded",
+)
+
+# Custom CSS
+st.markdown("""
+<style>
+    .main-header {
+        font-size: 2.5rem;
+        font-weight: bold;
+        background: linear-gradient(90deg, #FF6B6B, #4ECDC4);
+        -webkit-background-clip: text;
+        -webkit-text-fill-color: transparent;
+        margin-bottom: 0.5rem;
+    }
+    .sub-header {
+        color: #666;
+        font-size: 1.1rem;
+        margin-bottom: 2rem;
+    }
+    .metric-card {
+        background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+        border-radius: 10px;
+        padding: 1rem;
+        color: white;
+    }
+    .evidence-box {
+        background-color: #f0f7ff;
+        border-left: 4px solid #4ECDC4;
+        padding: 1rem;
+        margin: 0.5rem 0;
+        border-radius: 0 8px 8px 0;
+    }
+    .chunk-card {
+        background-color: #fafafa;
+        border: 1px solid #e0e0e0;
+        border-radius: 8px;
+        padding: 1rem;
+        margin: 0.5rem 0;
+    }
+    .confidence-high { color: #22c55e; font-weight: bold; }
+    .confidence-medium { color: #eab308; font-weight: bold; }
+    .confidence-low { color: #ef4444; font-weight: bold; }
+    .stTabs [data-baseweb="tab-list"] {
+        gap: 8px;
+    }
+    .stTabs [data-baseweb="tab"] {
+        padding: 10px 20px;
+        background-color: #f0f2f6;
+        border-radius: 8px;
+    }
+</style>
+""", unsafe_allow_html=True)
+
+
+def get_sample_documents():
+    """Get list of sample documents from Dataset folder."""
+    dataset_path = PROJECT_ROOT / "Dataset"
+    if dataset_path.exists():
+        return sorted([f.name for f in dataset_path.glob("*.pdf")])
+    return []
+
+
+def format_confidence(confidence: float) -> str:
+    """Format confidence with color coding."""
+    if confidence >= 0.8:
+        return f'<span class="confidence-high">{confidence:.1%}</span>'
+    elif confidence >= 0.6:
+        return f'<span class="confidence-medium">{confidence:.1%}</span>'
+    else:
+        return f'<span class="confidence-low">{confidence:.1%}</span>'
+
+
+def render_header():
+    """Render the main header."""
+    col1, col2 = st.columns([3, 1])
+    with col1:
+        st.markdown('<div class="main-header">🔥 SPARKNET</div>', unsafe_allow_html=True)
+        st.markdown('<div class="sub-header">Agentic Document Intelligence Platform</div>', unsafe_allow_html=True)
+    with col2:
+        st.image("https://img.shields.io/badge/version-0.1.0-blue", width=100)
+
+
+def render_sidebar():
+    """Render the sidebar with navigation."""
+    with st.sidebar:
+        st.markdown("## Navigation")
+
+        page = st.radio(
+            "Select Feature",
+            [
+                "🏠 Home",
+                "📄 Document Processing",
+                "🔍 Field Extraction",
+                "💬 RAG Q&A",
+                "🏷️ Classification",
+                "📊 Analytics",
+            ],
+            label_visibility="collapsed",
+        )
+
+        st.markdown("---")
+        st.markdown("### System Status")
+
+        # Check component status
+        ollama_status = check_ollama_status()
+        st.markdown(f"**Ollama:** {'🟢 Online' if ollama_status else '🔴 Offline'}")
+
+        chromadb_status = check_chromadb_status()
+        st.markdown(f"**ChromaDB:** {'🟢 Ready' if chromadb_status else '🔴 Not initialized'}")
+
+        st.markdown("---")
+        st.markdown("### Sample Documents")
+        docs = get_sample_documents()
+        st.markdown(f"**Available:** {len(docs)} PDFs")
+
+        return page
+
+
+def check_ollama_status():
+    """Check if Ollama is running."""
+    try:
+        import httpx
+        with httpx.Client(timeout=2.0) as client:
+            resp = client.get("http://localhost:11434/api/tags")
+            return resp.status_code == 200
+    except:
+        return False
+
+
+def check_chromadb_status():
+    """Check if ChromaDB is available."""
+    try:
+        import chromadb
+        return True
+    except:
+        return False
+
+
+def render_home_page():
+    """Render the home page."""
+    st.markdown("## Welcome to SPARKNET")
+
+    st.markdown("""
+    SPARKNET is an enterprise-grade **Agentic Document Intelligence Platform** that combines:
+
+    - **📄 Document Processing**: OCR with PaddleOCR/Tesseract, layout detection, semantic chunking
+    - **🔍 RAG Subsystem**: Vector search with ChromaDB, grounded retrieval with citations
+    - **🤖 Multi-Agent System**: ReAct-style agents with tool use and validation
+    - **🏠 Local-First**: Privacy-preserving inference via Ollama
+    - **📎 Evidence Grounding**: Every extraction includes bbox, page, chunk_id references
+    """)
+
+    st.markdown("---")
+
+    # Feature cards
+    col1, col2, col3, col4 = st.columns(4)
+
+    with col1:
+        st.markdown("""
+        <div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+                    border-radius: 12px; padding: 1.5rem; color: white; text-align: center;">
+            <h3>📄</h3>
+            <h4>Document Processing</h4>
+            <p style="font-size: 0.9rem;">OCR, Layout Detection, Chunking</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+    with col2:
+        st.markdown("""
+        <div style="background: linear-gradient(135deg, #f093fb 0%, #f5576c 100%);
+                    border-radius: 12px; padding: 1.5rem; color: white; text-align: center;">
+            <h3>🔍</h3>
+            <h4>Field Extraction</h4>
+            <p style="font-size: 0.9rem;">Structured Data with Evidence</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+    with col3:
+        st.markdown("""
+        <div style="background: linear-gradient(135deg, #4facfe 0%, #00f2fe 100%);
+                    border-radius: 12px; padding: 1.5rem; color: white; text-align: center;">
+            <h3>💬</h3>
+            <h4>RAG Q&A</h4>
+            <p style="font-size: 0.9rem;">Grounded Answers with Citations</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+    with col4:
+        st.markdown("""
+        <div style="background: linear-gradient(135deg, #43e97b 0%, #38f9d7 100%);
+                    border-radius: 12px; padding: 1.5rem; color: white; text-align: center;">
+            <h3>🏷️</h3>
+            <h4>Classification</h4>
+            <p style="font-size: 0.9rem;">Document Type Detection</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+    st.markdown("---")
+
+    # Quick start
+    st.markdown("### Quick Start")
+
+    with st.expander("📚 How to Use This Demo", expanded=True):
+        st.markdown("""
+        1. **Document Processing**: Upload or select a PDF to process with OCR
+        2. **Field Extraction**: Define fields to extract with evidence grounding
+        3. **RAG Q&A**: Ask questions about indexed documents
+        4. **Classification**: Automatically classify document types
+
+        **Sample Documents**: The demo includes real patent documents from major tech companies.
+        """)
+
+    # Sample documents preview
+    st.markdown("### Available Sample Documents")
+    docs = get_sample_documents()
+
+    if docs:
+        cols = st.columns(4)
+        for i, doc in enumerate(docs[:8]):
+            with cols[i % 4]:
+                company = doc.split()[0] if doc else "Unknown"
+                st.markdown(f"""
+                <div style="background: #f8f9fa; border-radius: 8px; padding: 0.8rem;
+                            margin: 0.3rem 0; border: 1px solid #e0e0e0;">
+                    <strong>📄 {company}</strong>
+                    <br><small style="color: #666;">{doc[:30]}...</small>
+                </div>
+                """, unsafe_allow_html=True)
+
+
+def render_document_processing_page():
+    """Render the document processing page."""
+    st.markdown("## 📄 Document Processing Pipeline")
+
+    st.markdown("""
+    Process documents through our intelligent pipeline:
+    **OCR → Layout Detection → Reading Order → Semantic Chunking → Grounding**
+    """)
+
+    # Document selection
+    col1, col2 = st.columns([2, 1])
+
+    with col1:
+        upload_option = st.radio(
+            "Document Source",
+            ["Select from samples", "Upload new document"],
+            horizontal=True,
+        )
+
+        if upload_option == "Select from samples":
+            docs = get_sample_documents()
+            if docs:
+                selected_doc = st.selectbox("Select a document", docs)
+                doc_path = PROJECT_ROOT / "Dataset" / selected_doc
+            else:
+                st.warning("No sample documents found")
+                doc_path = None
+        else:
+            uploaded_file = st.file_uploader("Upload PDF", type=["pdf"])
+            if uploaded_file:
+                # Save temporarily
+                temp_path = PROJECT_ROOT / "data" / "temp" / uploaded_file.name
+                temp_path.parent.mkdir(parents=True, exist_ok=True)
+                with open(temp_path, "wb") as f:
+                    f.write(uploaded_file.read())
+                doc_path = temp_path
+            else:
+                doc_path = None
+
+    with col2:
+        st.markdown("### Processing Options")
+        ocr_engine = st.selectbox("OCR Engine", ["paddleocr", "tesseract"])
+        max_pages = st.slider("Max Pages", 1, 20, 5)
+        render_dpi = st.selectbox("Render DPI", [150, 200, 300], index=2)
+
+    st.markdown("---")
+
+    if doc_path and st.button("🚀 Process Document", type="primary"):
+        process_document_demo(doc_path, ocr_engine, max_pages, render_dpi)
+
+
+def process_document_demo(doc_path, ocr_engine, max_pages, render_dpi):
+    """Demo document processing."""
+
+    progress_bar = st.progress(0)
+    status_text = st.empty()
+
+    # Simulate processing stages
+    stages = [
+        ("Loading document...", 0.1),
+        ("Running OCR extraction...", 0.3),
+        ("Detecting layout regions...", 0.5),
+        ("Reconstructing reading order...", 0.7),
+        ("Creating semantic chunks...", 0.9),
+        ("Finalizing...", 1.0),
+    ]
+
+    for stage_text, progress in stages:
+        status_text.text(stage_text)
+        progress_bar.progress(progress)
+        time.sleep(0.5)
+
+    status_text.text("✅ Processing complete!")
+
+    # Try actual processing
+    try:
+        from src.document.pipeline import process_document, PipelineConfig
+        from src.document.ocr import OCRConfig
+
+        config = PipelineConfig(
+            ocr=OCRConfig(engine=ocr_engine),
+            render_dpi=render_dpi,
+            max_pages=max_pages,
+        )
+
+        with st.spinner("Running actual document processing..."):
+            result = process_document(str(doc_path), config=config)
+
+        # Display results
+        render_processing_results(result)
+
+    except Exception as e:
+        st.warning(f"Live processing unavailable: {e}")
+        st.info("Showing demo results instead...")
+        render_demo_processing_results(str(doc_path))
+
+
+def render_processing_results(result):
+    """Render actual processing results."""
+
+    # Metrics
+    col1, col2, col3, col4 = st.columns(4)
+
+    with col1:
+        st.metric("Pages", result.metadata.num_pages)
+    with col2:
+        st.metric("Chunks", result.metadata.total_chunks)
+    with col3:
+        st.metric("Characters", f"{result.metadata.total_characters:,}")
+    with col4:
+        conf = result.metadata.ocr_confidence_avg or 0
+        st.metric("OCR Confidence", f"{conf:.1%}")
+
+    st.markdown("---")
+
+    # Tabs for different views
+    tab1, tab2, tab3 = st.tabs(["📝 Extracted Text", "📦 Chunks", "🗺️ Layout"])
+
+    with tab1:
+        st.markdown("### Full Extracted Text")
+        st.text_area(
+            "Document Text",
+            result.full_text[:5000] + "..." if len(result.full_text) > 5000 else result.full_text,
+            height=400,
+        )
+
+    with tab2:
+        st.markdown("### Document Chunks")
+        for i, chunk in enumerate(result.chunks[:10]):
+            with st.expander(f"Chunk {i+1}: {chunk.chunk_type.value} (Page {chunk.page + 1})"):
+                st.markdown(f"**ID:** `{chunk.chunk_id}`")
+                st.markdown(f"**Confidence:** {format_confidence(chunk.confidence)}", unsafe_allow_html=True)
+                st.markdown(f"**BBox:** ({chunk.bbox.x_min:.0f}, {chunk.bbox.y_min:.0f}) → ({chunk.bbox.x_max:.0f}, {chunk.bbox.y_max:.0f})")
+                st.markdown("**Text:**")
+                st.text(chunk.text[:500])
+
+    with tab3:
+        st.markdown("### Layout Regions")
+        if result.layout_regions:
+            layout_data = []
+            for r in result.layout_regions:
+                layout_data.append({
+                    "Type": r.layout_type.value,
+                    "Page": r.page + 1,
+                    "Confidence": f"{r.confidence:.1%}",
+                    "Position": f"({r.bbox.x_min:.0f}, {r.bbox.y_min:.0f})",
+                })
+            st.dataframe(layout_data, width='stretch')
+        else:
+            st.info("No layout regions detected")
+
+
+def render_demo_processing_results(doc_path):
+    """Render demo processing results when actual processing unavailable."""
+
+    doc_name = Path(doc_path).name
+
+    # Simulated metrics
+    col1, col2, col3, col4 = st.columns(4)
+
+    with col1:
+        st.metric("Pages", 12)
+    with col2:
+        st.metric("Chunks", 47)
+    with col3:
+        st.metric("Characters", "15,234")
+    with col4:
+        st.metric("OCR Confidence", "94.2%")
+
+    st.markdown("---")
+
+    # Demo chunks
+    demo_chunks = [
+        {
+            "type": "title",
+            "page": 1,
+            "confidence": 0.98,
+            "text": f"PATENT PLEDGE - {doc_name.split()[0]}",
+            "bbox": "(100, 50) → (700, 100)",
+        },
+        {
+            "type": "text",
+            "page": 1,
+            "confidence": 0.95,
+            "text": "This Patent Pledge is made by the undersigned company to promote innovation and reduce patent-related barriers...",
+            "bbox": "(100, 150) → (700, 300)",
+        },
+        {
+            "type": "text",
+            "page": 1,
+            "confidence": 0.92,
+            "text": "The company hereby pledges not to assert any patent claims against any party making, using, or selling products...",
+            "bbox": "(100, 320) → (700, 500)",
+        },
+    ]
+
+    tab1, tab2 = st.tabs(["📝 Extracted Text", "📦 Chunks"])
+
+    with tab1:
+        st.markdown("### Full Extracted Text")
+        demo_text = f"""
+PATENT PLEDGE - {doc_name.split()[0]}
+
+This Patent Pledge is made by the undersigned company to promote innovation
+and reduce patent-related barriers in the technology industry.
+
+DEFINITIONS:
+1. "Covered Patents" means all patents and patent applications owned by
+   the Pledgor that cover fundamental technologies.
+2. "Open Source Software" means software distributed under licenses
+   approved by the Open Source Initiative.
+
+PLEDGE:
+The company hereby pledges not to assert any Covered Patents against
+any party making, using, selling, or distributing Open Source Software.
+
+This pledge is irrevocable and shall remain in effect for the life
+of all Covered Patents.
+
+[Document continues with legal terms and conditions...]
+        """
+        st.text_area("Document Text", demo_text, height=400)
+
+    with tab2:
+        st.markdown("### Document Chunks")
+        for i, chunk in enumerate(demo_chunks):
+            with st.expander(f"Chunk {i+1}: {chunk['type']} (Page {chunk['page']})"):
+                st.markdown(f"**Confidence:** {format_confidence(chunk['confidence'])}", unsafe_allow_html=True)
+                st.markdown(f"**BBox:** {chunk['bbox']}")
+                st.markdown("**Text:**")
+                st.text(chunk["text"])
+
+
+def render_extraction_page():
+    """Render the field extraction page."""
+    st.markdown("## 🔍 Field Extraction with Evidence")
+
+    st.markdown("""
+    Extract structured fields from documents with **evidence grounding**.
+    Every extracted value includes its source location (page, bbox, chunk_id).
+    """)
+
+    col1, col2 = st.columns([2, 1])
+
+    with col1:
+        # Document selection
+        docs = get_sample_documents()
+        if docs:
+            selected_doc = st.selectbox("Select Document", docs, key="extract_doc")
+
+        st.markdown("### Fields to Extract")
+
+        # Predefined schemas
+        schema_type = st.selectbox(
+            "Extraction Schema",
+            ["Patent/Legal Document", "Invoice", "Contract", "Custom"],
+        )
+
+        if schema_type == "Patent/Legal Document":
+            default_fields = ["document_title", "company_name", "effective_date", "key_terms", "parties_involved"]
+        elif schema_type == "Invoice":
+            default_fields = ["invoice_number", "date", "total_amount", "vendor_name", "line_items"]
+        elif schema_type == "Contract":
+            default_fields = ["contract_title", "parties", "effective_date", "term_length", "key_obligations"]
+        else:
+            default_fields = ["field_1", "field_2"]
+
+        fields = st.multiselect(
+            "Select fields to extract",
+            default_fields,
+            default=default_fields[:3],
+        )
+
+    with col2:
+        st.markdown("### Extraction Options")
+        validate = st.checkbox("Validate with Critic", value=True)
+        include_evidence = st.checkbox("Include Evidence", value=True)
+        confidence_threshold = st.slider("Min Confidence", 0.0, 1.0, 0.7)
+
+    st.markdown("---")
+
+    if fields and st.button("🔍 Extract Fields", type="primary"):
+        extract_fields_demo(selected_doc, fields, validate, include_evidence)
+
+
+def extract_fields_demo(doc_name, fields, validate, include_evidence):
+    """Demo field extraction."""
+
+    with st.spinner("Extracting fields..."):
+        time.sleep(1.5)
+
+    st.success("✅ Extraction complete!")
+
+    # Demo results
+    company = doc_name.split()[0] if doc_name else "Company"
+
+    demo_extractions = {
+        "document_title": {
+            "value": f"{company} Patent Non-Assertion Pledge",
+            "confidence": 0.96,
+            "page": 1,
+            "evidence": f"Found in header: '{company} Patent Non-Assertion Pledge' at position (100, 50)",
+        },
+        "company_name": {
+            "value": company,
+            "confidence": 0.98,
+            "page": 1,
+            "evidence": f"Identified as pledgor: '{company}' mentioned 15 times throughout document",
+        },
+        "effective_date": {
+            "value": doc_name.split()[-1].replace(".pdf", "") if len(doc_name.split()) > 1 else "N/A",
+            "confidence": 0.85,
+            "page": 1,
+            "evidence": "Date found in document header",
+        },
+        "key_terms": {
+            "value": "Patent pledge, Open source, Non-assertion, Royalty-free",
+            "confidence": 0.89,
+            "page": 2,
+            "evidence": "Key terms identified from definitions section",
+        },
+        "parties_involved": {
+            "value": f"{company}, Open Source Community",
+            "confidence": 0.82,
+            "page": 1,
+            "evidence": "Parties identified from pledge declaration",
+        },
+    }
+
+    # Display results
+    st.markdown("### Extracted Fields")
+
+    for field in fields:
+        if field in demo_extractions:
+            data = demo_extractions[field]
+
+            col1, col2 = st.columns([3, 1])
+
+            with col1:
+                st.markdown(f"""
+                <div class="chunk-card">
+                    <strong>{field.replace('_', ' ').title()}</strong>
+                    <p style="font-size: 1.2rem; margin: 0.5rem 0;">{data['value']}</p>
+                </div>
+                """, unsafe_allow_html=True)
+
+            with col2:
+                st.markdown(f"**Confidence:** {format_confidence(data['confidence'])}", unsafe_allow_html=True)
+                st.markdown(f"**Page:** {data['page']}")
+
+            if include_evidence:
+                st.markdown(f"""
+                <div class="evidence-box">
+                    📎 <strong>Evidence:</strong> {data['evidence']}
+                </div>
+                """, unsafe_allow_html=True)
+
+            st.markdown("")
+
+    # Validation results
+    if validate:
+        st.markdown("---")
+        st.markdown("### Validation Results")
+
+        col1, col2, col3 = st.columns(3)
+        with col1:
+            st.metric("Fields Validated", len(fields))
+        with col2:
+            st.metric("Valid", len(fields) - 1)
+        with col3:
+            st.metric("Uncertain", 1)
+
+        st.info("💡 Critic validation: All fields have supporting evidence in the document.")
+
+
+def render_rag_page():
+    """Render the RAG Q&A page."""
+    st.markdown("## 💬 RAG Question Answering")
+
+    st.markdown("""
+    Ask questions about indexed documents. Answers include **citations** pointing to
+    the exact source chunks with page numbers and text snippets.
+    """)
+
+    # Index status
+    col1, col2 = st.columns([2, 1])
+
+    with col1:
+        st.markdown("### Ask a Question")
+
+        # Preset questions
+        preset_questions = [
+            "What is the main purpose of this document?",
+            "What patents are covered by this pledge?",
+            "What are the key terms and definitions?",
+            "Who are the parties involved?",
+            "What are the conditions for the pledge?",
+        ]
+
+        question_mode = st.radio(
+            "Question Mode",
+            ["Select preset", "Custom question"],
+            horizontal=True,
+        )
+
+        if question_mode == "Select preset":
+            question = st.selectbox("Select a question", preset_questions)
+        else:
+            question = st.text_input("Enter your question")
+
+        col_a, col_b = st.columns(2)
+        with col_a:
+            top_k = st.slider("Number of sources", 1, 10, 5)
+        with col_b:
+            show_confidence = st.checkbox("Show confidence scores", value=True)
+
+    with col2:
+        st.markdown("### Index Status")
+        st.markdown("""
+        - **Documents indexed:** 3
+        - **Total chunks:** 147
+        - **Embedding model:** nomic-embed-text
+        - **Vector dimension:** 768
+        """)
+
+    st.markdown("---")
+
+    if question and st.button("🔍 Get Answer", type="primary"):
+        rag_query_demo(question, top_k, show_confidence)
+
+
+def rag_query_demo(question, top_k, show_confidence):
+    """Demo RAG query."""
+
+    with st.spinner("Searching documents and generating answer..."):
+        time.sleep(1.5)
+
+    # Demo answer based on question
+    demo_answers = {
+        "purpose": {
+            "answer": "The main purpose of this document is to establish a **Patent Non-Assertion Pledge** where the company commits not to assert certain patent claims against parties using, making, or distributing Open Source Software. This pledge aims to promote innovation and reduce patent-related barriers in the technology industry.",
+            "confidence": 0.92,
+            "citations": [
+                {"index": 1, "page": 1, "snippet": "This Patent Pledge is made to promote innovation and reduce patent-related barriers...", "confidence": 0.95},
+                {"index": 2, "page": 1, "snippet": "The company hereby pledges not to assert any patent claims against any party...", "confidence": 0.91},
+            ],
+        },
+        "patents": {
+            "answer": "The pledge covers **all patents and patent applications** owned by the Pledgor that relate to fundamental technologies used in Open Source Software. Specifically, these are referred to as 'Covered Patents' in the document, defined as patents that cover essential features or functionalities.",
+            "confidence": 0.88,
+            "citations": [
+                {"index": 1, "page": 2, "snippet": "'Covered Patents' means all patents and patent applications owned by the Pledgor...", "confidence": 0.93},
+                {"index": 2, "page": 2, "snippet": "Patents covering fundamental technologies essential to Open Source implementations...", "confidence": 0.85},
+            ],
+        },
+        "default": {
+            "answer": "Based on the available documents, this appears to be a **Patent Pledge** document from a major technology company. The document establishes terms for patent non-assertion related to Open Source Software, with specific definitions and conditions outlined in the legal text.",
+            "confidence": 0.75,
+            "citations": [
+                {"index": 1, "page": 1, "snippet": "Patent Pledge document establishing non-assertion terms...", "confidence": 0.80},
+            ],
+        },
+    }
+
+    # Select answer based on question keywords
+    if "purpose" in question.lower() or "main" in question.lower():
+        result = demo_answers["purpose"]
+    elif "patent" in question.lower() and "cover" in question.lower():
+        result = demo_answers["patents"]
+    else:
+        result = demo_answers["default"]
+
+    # Display answer
+    st.markdown("### Answer")
+
+    st.markdown(f"""
+    <div style="background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
+                border-radius: 12px; padding: 1.5rem; margin: 1rem 0;">
+        {result['answer']}
+    </div>
+    """, unsafe_allow_html=True)
+
+    if show_confidence:
+        st.markdown(f"**Overall Confidence:** {format_confidence(result['confidence'])}", unsafe_allow_html=True)
+
+    # Citations
+    st.markdown("### 📚 Citations")
+
+    for citation in result["citations"][:top_k]:
+        st.markdown(f"""
+        <div class="evidence-box">
+            <strong>[{citation['index']}] Page {citation['page']}</strong>
+            {f' - Confidence: {citation["confidence"]:.0%}' if show_confidence else ''}
+            <br>
+            <em>"{citation['snippet']}"</em>
+        </div>
+        """, unsafe_allow_html=True)
+
+
+def render_classification_page():
+    """Render the classification page."""
+    st.markdown("## 🏷️ Document Classification")
+
+    st.markdown("""
+    Automatically classify documents into predefined categories with confidence scores
+    and reasoning explanations.
+    """)
+
+    docs = get_sample_documents()
+
+    col1, col2 = st.columns([2, 1])
+
+    with col1:
+        if docs:
+            selected_doc = st.selectbox("Select Document to Classify", docs, key="classify_doc")
+
+        st.markdown("### Document Categories")
+        categories = [
+            "📜 Legal/Patent Document",
+            "📑 Contract/Agreement",
+            "📊 Financial Report",
+            "📋 Technical Specification",
+            "📄 General Business Document",
+        ]
+        st.markdown("\n".join([f"- {cat}" for cat in categories]))
+
+    with col2:
+        st.markdown("### Classification Options")
+        detailed_reasoning = st.checkbox("Show detailed reasoning", value=True)
+        multi_label = st.checkbox("Allow multiple categories", value=False)
+
+    st.markdown("---")
+
+    if st.button("🏷️ Classify Document", type="primary"):
+        classify_document_demo(selected_doc, detailed_reasoning)
+
+
+def classify_document_demo(doc_name, detailed_reasoning):
+    """Demo document classification."""
+
+    with st.spinner("Analyzing document..."):
+        time.sleep(1.0)
+
+    st.success("✅ Classification complete!")
+
+    # Demo classification results
+    col1, col2 = st.columns([2, 1])
+
+    with col1:
+        st.markdown("### Primary Classification")
+        st.markdown("""
+        <div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+                    border-radius: 12px; padding: 1.5rem; color: white; text-align: center;">
+            <h2 style="margin: 0;">📜 Legal/Patent Document</h2>
+            <p style="font-size: 1.2rem; margin: 0.5rem 0;">Patent Non-Assertion Pledge</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+    with col2:
+        st.markdown("### Confidence Scores")
+        st.markdown(f"**Legal/Patent:** {format_confidence(0.94)}", unsafe_allow_html=True)
+        st.markdown(f"**Contract:** {format_confidence(0.72)}", unsafe_allow_html=True)
+        st.markdown(f"**Technical:** {format_confidence(0.15)}", unsafe_allow_html=True)
+        st.markdown(f"**Financial:** {format_confidence(0.08)}", unsafe_allow_html=True)
+
+    if detailed_reasoning:
+        st.markdown("---")
+        st.markdown("### Classification Reasoning")
+
+        st.markdown("""
+        <div class="evidence-box">
+            <strong>Why Legal/Patent Document?</strong>
+            <ul>
+                <li>Contains legal terminology: "pledge", "assert", "patent claims", "royalty-free"</li>
+                <li>Structured as a formal legal declaration</li>
+                <li>References specific patent-related definitions</li>
+                <li>Contains commitment/obligation language</li>
+            </ul>
+        </div>
+        """, unsafe_allow_html=True)
+
+        st.markdown("""
+        <div class="chunk-card">
+            <strong>Key Indicators Found:</strong>
+            <br>
+            • "Patent Pledge" - Document title indicator (weight: 0.35)<br>
+            • "hereby pledges" - Legal commitment language (weight: 0.25)<br>
+            • "Covered Patents" - Patent-specific terminology (weight: 0.20)<br>
+            • "Open Source Software" - Tech/IP context (weight: 0.15)
+        </div>
+        """, unsafe_allow_html=True)
+
+
+def render_analytics_page():
+    """Render the analytics page."""
+    st.markdown("## 📊 Processing Analytics")
+
+    st.markdown("View statistics and insights about document processing.")
+
+    # Summary metrics
+    col1, col2, col3, col4 = st.columns(4)
+
+    with col1:
+        st.metric("Documents Processed", 24, delta="+3 today")
+    with col2:
+        st.metric("Total Chunks", 1247, delta="+156")
+    with col3:
+        st.metric("Avg. Confidence", "91.3%", delta="+2.1%")
+    with col4:
+        st.metric("Questions Answered", 89, delta="+12")
+
+    st.markdown("---")
+
+    # Charts
+    col1, col2 = st.columns(2)
+
+    with col1:
+        st.markdown("### Document Types Processed")
+        import pandas as pd
+
+        chart_data = pd.DataFrame({
+            "Type": ["Patent/Legal", "Contract", "Technical", "Financial", "Other"],
+            "Count": [12, 5, 4, 2, 1],
+        })
+        st.bar_chart(chart_data.set_index("Type"))
+
+    with col2:
+        st.markdown("### Processing Performance")
+        perf_data = pd.DataFrame({
+            "Stage": ["OCR", "Layout", "Chunking", "Indexing", "Retrieval"],
+            "Avg Time (s)": [2.3, 0.8, 0.5, 1.2, 0.3],
+        })
+        st.bar_chart(perf_data.set_index("Stage"))
+
+    st.markdown("---")
+
+    # Recent activity
+    st.markdown("### Recent Activity")
+
+    activities = [
+        {"time": "2 min ago", "action": "Processed", "document": "IBM N_A.pdf", "chunks": 42},
+        {"time": "15 min ago", "action": "Indexed", "document": "Apple 11.11.2011.pdf", "chunks": 67},
+        {"time": "1 hour ago", "action": "Queried", "document": "RAG Collection", "chunks": 5},
+        {"time": "2 hours ago", "action": "Classified", "document": "Google 08.02.2012.pdf", "chunks": 0},
+    ]
+
+    for activity in activities:
+        st.markdown(f"""
+        <div class="chunk-card">
+            <strong>{activity['time']}</strong> - {activity['action']} <em>{activity['document']}</em>
+            {f" ({activity['chunks']} chunks)" if activity['chunks'] > 0 else ""}
+        </div>
+        """, unsafe_allow_html=True)
+
+
+def main():
+    """Main application."""
+    render_header()
+    page = render_sidebar()
+
+    # Route to appropriate page
+    if page == "🏠 Home":
+        render_home_page()
+    elif page == "📄 Document Processing":
+        render_document_processing_page()
+    elif page == "🔍 Field Extraction":
+        render_extraction_page()
+    elif page == "💬 RAG Q&A":
+        render_rag_page()
+    elif page == "🏷️ Classification":
+        render_classification_page()
+    elif page == "📊 Analytics":
+        render_analytics_page()
+
+    # Footer
+    st.markdown("---")
+    st.markdown(
+        "<div style='text-align: center; color: #666;'>"
+        "🔥 SPARKNET Document Intelligence Platform | Built with Streamlit"
+        "</div>",
+        unsafe_allow_html=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

demo/llm_providers.py

@@ -0,0 +1,339 @@
+"""
+Free LLM Providers for SPARKNET
+
+Supports multiple free-tier LLM providers:
+1. HuggingFace Inference API (free, no payment required)
+2. Groq (free tier - very fast)
+3. Google Gemini (free tier)
+4. Local/Offline mode (simulated responses)
+"""
+
+import os
+import requests
+from typing import Optional, Tuple, List
+from dataclasses import dataclass
+from loguru import logger
+
+@dataclass
+class LLMResponse:
+    text: str
+    model: str
+    provider: str
+    success: bool
+    error: Optional[str] = None
+
+
+class HuggingFaceProvider:
+    """
+    HuggingFace Inference API - FREE tier available.
+
+    Models that work well on free tier:
+    - microsoft/DialoGPT-medium
+    - google/flan-t5-base
+    - mistralai/Mistral-7B-Instruct-v0.2 (may need Pro for heavy use)
+    - HuggingFaceH4/zephyr-7b-beta
+    """
+
+    API_URL = "https://api-inference.huggingface.co/models/"
+
+    # Free-tier friendly models
+    MODELS = {
+        "chat": "HuggingFaceH4/zephyr-7b-beta",
+        "chat_small": "microsoft/DialoGPT-medium",
+        "instruct": "google/flan-t5-large",
+        "embed": "sentence-transformers/all-MiniLM-L6-v2",
+    }
+
+    def __init__(self, api_token: Optional[str] = None):
+        """
+        Initialize HuggingFace provider.
+
+        Args:
+            api_token: HF token (optional but recommended for higher rate limits)
+                      Get free token at: https://huggingface.co/settings/tokens
+        """
+        self.api_token = api_token or os.environ.get("HUGGINGFACE_TOKEN") or os.environ.get("HF_TOKEN")
+        self.headers = {}
+        if self.api_token:
+            self.headers["Authorization"] = f"Bearer {self.api_token}"
+
+    def generate(self, prompt: str, model: Optional[str] = None, max_tokens: int = 500) -> LLMResponse:
+        """Generate text using HuggingFace Inference API."""
+        model = model or self.MODELS["chat"]
+        url = f"{self.API_URL}{model}"
+
+        payload = {
+            "inputs": prompt,
+            "parameters": {
+                "max_new_tokens": max_tokens,
+                "temperature": 0.7,
+                "do_sample": True,
+                "return_full_text": False,
+            }
+        }
+
+        try:
+            response = requests.post(url, headers=self.headers, json=payload, timeout=60)
+
+            if response.status_code == 503:
+                # Model is loading
+                return LLMResponse(
+                    text="Model is loading, please try again in a moment...",
+                    model=model,
+                    provider="huggingface",
+                    success=False,
+                    error="Model loading"
+                )
+
+            response.raise_for_status()
+            result = response.json()
+
+            if isinstance(result, list) and len(result) > 0:
+                text = result[0].get("generated_text", "")
+            else:
+                text = str(result)
+
+            return LLMResponse(
+                text=text,
+                model=model,
+                provider="huggingface",
+                success=True
+            )
+
+        except Exception as e:
+            logger.error(f"HuggingFace API error: {e}")
+            return LLMResponse(
+                text="",
+                model=model,
+                provider="huggingface",
+                success=False,
+                error=str(e)
+            )
+
+    def embed(self, texts: List[str], model: Optional[str] = None) -> Tuple[List[List[float]], Optional[str]]:
+        """Generate embeddings using HuggingFace."""
+        model = model or self.MODELS["embed"]
+        url = f"{self.API_URL}{model}"
+
+        payload = {
+            "inputs": texts,
+            "options": {"wait_for_model": True}
+        }
+
+        try:
+            response = requests.post(url, headers=self.headers, json=payload, timeout=60)
+            response.raise_for_status()
+            embeddings = response.json()
+            return embeddings, None
+        except Exception as e:
+            logger.error(f"HuggingFace embed error: {e}")
+            return [], str(e)
+
+
+class GroqProvider:
+    """
+    Groq - FREE tier with very fast inference.
+
+    Free tier includes:
+    - 14,400 requests/day for smaller models
+    - Very fast inference (fastest available)
+
+    Get free API key at: https://console.groq.com/keys
+    """
+
+    API_URL = "https://api.groq.com/openai/v1/chat/completions"
+
+    MODELS = {
+        "fast": "llama-3.1-8b-instant",  # Fastest
+        "smart": "llama-3.3-70b-versatile",  # Best quality
+        "small": "gemma2-9b-it",  # Good balance
+    }
+
+    def __init__(self, api_key: Optional[str] = None):
+        self.api_key = api_key or os.environ.get("GROQ_API_KEY")
+        if not self.api_key:
+            logger.warning("No Groq API key found. Get free key at: https://console.groq.com/keys")
+
+    def generate(self, prompt: str, model: Optional[str] = None, max_tokens: int = 500) -> LLMResponse:
+        """Generate text using Groq API."""
+        if not self.api_key:
+            return LLMResponse(
+                text="",
+                model="",
+                provider="groq",
+                success=False,
+                error="No Groq API key configured"
+            )
+
+        model = model or self.MODELS["fast"]
+
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json"
+        }
+
+        payload = {
+            "model": model,
+            "messages": [{"role": "user", "content": prompt}],
+            "max_tokens": max_tokens,
+            "temperature": 0.7,
+        }
+
+        try:
+            response = requests.post(self.API_URL, headers=headers, json=payload, timeout=30)
+            response.raise_for_status()
+            result = response.json()
+
+            text = result["choices"][0]["message"]["content"]
+
+            return LLMResponse(
+                text=text,
+                model=model,
+                provider="groq",
+                success=True
+            )
+
+        except Exception as e:
+            logger.error(f"Groq API error: {e}")
+            return LLMResponse(
+                text="",
+                model=model,
+                provider="groq",
+                success=False,
+                error=str(e)
+            )
+
+
+class OfflineProvider:
+    """
+    Offline/Demo mode - no API required.
+
+    Provides simulated responses for demonstration purposes.
+    """
+
+    def __init__(self):
+        pass
+
+    def generate(self, prompt: str, context: str = "", **kwargs) -> LLMResponse:
+        """Generate a simulated response based on context."""
+
+        # Extract key information from context if provided
+        if context:
+            # Simple extractive response
+            sentences = context.split('.')
+            relevant = [s.strip() for s in sentences if len(s.strip()) > 20][:3]
+
+            if relevant:
+                response = f"Based on the documents, {relevant[0].lower()}."
+                if len(relevant) > 1:
+                    response += f" Additionally, {relevant[1].lower()}."
+            else:
+                response = "Based on the available documents, I found relevant information but cannot generate a detailed response in offline mode."
+        else:
+            response = "I'm running in offline demo mode. To get AI-powered responses, please configure a free LLM provider (HuggingFace or Groq)."
+
+        return LLMResponse(
+            text=response,
+            model="offline",
+            provider="offline",
+            success=True
+        )
+
+    def embed(self, texts: List[str]) -> Tuple[List[List[float]], Optional[str]]:
+        """Generate simple bag-of-words style embeddings for demo."""
+        import hashlib
+
+        embeddings = []
+        for text in texts:
+            # Create deterministic pseudo-embeddings based on text hash
+            hash_bytes = hashlib.sha256(text.encode()).digest()
+            # Convert to 384-dim vector (same as MiniLM)
+            embedding = [((b % 200) - 100) / 100.0 for b in hash_bytes * 12][:384]
+            embeddings.append(embedding)
+
+        return embeddings, None
+
+
+class UnifiedLLMProvider:
+    """
+    Unified interface for all LLM providers.
+
+    Automatically selects the best available provider.
+    """
+
+    def __init__(self):
+        self.providers = {}
+        self.active_provider = None
+        self.active_embed_provider = None
+
+        # Try to initialize providers in order of preference
+        self._init_providers()
+
+    def _init_providers(self):
+        """Initialize available providers."""
+
+        # Check for Groq (fastest, generous free tier)
+        groq_key = os.environ.get("GROQ_API_KEY")
+        if groq_key:
+            self.providers["groq"] = GroqProvider(groq_key)
+            self.active_provider = "groq"
+            logger.info("Using Groq provider (free tier)")
+
+        # Check for HuggingFace (always available, even without token)
+        hf_token = os.environ.get("HUGGINGFACE_TOKEN") or os.environ.get("HF_TOKEN")
+        self.providers["huggingface"] = HuggingFaceProvider(hf_token)
+        if not self.active_provider:
+            self.active_provider = "huggingface"
+            logger.info("Using HuggingFace provider")
+
+        # HuggingFace for embeddings (always free)
+        self.active_embed_provider = "huggingface"
+
+        # Offline fallback
+        self.providers["offline"] = OfflineProvider()
+
+        logger.info(f"LLM Provider: {self.active_provider}, Embed Provider: {self.active_embed_provider}")
+
+    def generate(self, prompt: str, **kwargs) -> LLMResponse:
+        """Generate text using the best available provider."""
+        provider = self.providers.get(self.active_provider)
+
+        if provider:
+            response = provider.generate(prompt, **kwargs)
+            if response.success:
+                return response
+
+        # Fallback to offline
+        return self.providers["offline"].generate(prompt, **kwargs)
+
+    def embed(self, texts: List[str]) -> Tuple[List[List[float]], Optional[str]]:
+        """Generate embeddings using the best available provider."""
+        if self.active_embed_provider == "huggingface":
+            embeddings, error = self.providers["huggingface"].embed(texts)
+            if not error:
+                return embeddings, None
+
+        # Fallback to offline embeddings
+        return self.providers["offline"].embed(texts)
+
+    def get_status(self) -> dict:
+        """Get status of all providers."""
+        return {
+            "active_llm": self.active_provider,
+            "active_embed": self.active_embed_provider,
+            "available_providers": list(self.providers.keys()),
+            "groq_configured": "groq" in self.providers and self.providers["groq"].api_key is not None,
+            "huggingface_configured": self.providers["huggingface"].api_token is not None,
+        }
+
+
+# Global instance
+_llm_provider: Optional[UnifiedLLMProvider] = None
+
+
+def get_llm_provider() -> UnifiedLLMProvider:
+    """Get or create the unified LLM provider."""
+    global _llm_provider
+    if _llm_provider is None:
+        _llm_provider = UnifiedLLMProvider()
+    return _llm_provider

demo/pages/1_🔬_Live_Processing.py

@@ -0,0 +1,714 @@
+"""
+Live Document Processing Demo - SPARKNET
+
+Real-time document processing with integrated state management and auto-indexing.
+"""
+
+import streamlit as st
+import sys
+from pathlib import Path
+import time
+import io
+import base64
+from datetime import datetime
+import hashlib
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+sys.path.insert(0, str(PROJECT_ROOT / "demo"))
+
+# Import state manager and RAG config
+from state_manager import (
+    get_state_manager,
+    ProcessedDocument as StateDocument,
+    generate_doc_id,
+    render_global_status_bar,
+)
+from rag_config import (
+    get_unified_rag_system,
+    auto_index_processed_document,
+    check_ollama,
+)
+
+st.set_page_config(page_title="Live Processing - SPARKNET", page_icon="🔬", layout="wide")
+
+# Custom CSS
+st.markdown("""
+<style>
+.stage-card {
+    background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);
+    padding: 15px;
+    border-radius: 10px;
+    margin: 10px 0;
+    border-left: 4px solid #4ECDC4;
+}
+.stage-active {
+    border-left-color: #ffc107;
+    animation: pulse 1s infinite;
+}
+.stage-done {
+    border-left-color: #28a745;
+}
+.stage-error {
+    border-left-color: #dc3545;
+}
+@keyframes pulse {
+    0% { opacity: 1; }
+    50% { opacity: 0.7; }
+    100% { opacity: 1; }
+}
+.metric-card {
+    background: #161b22;
+    border-radius: 8px;
+    padding: 12px;
+    text-align: center;
+    border: 1px solid #30363d;
+}
+.metric-value {
+    font-size: 24px;
+    font-weight: bold;
+    color: #4ECDC4;
+}
+.metric-label {
+    font-size: 11px;
+    color: #8b949e;
+    text-transform: uppercase;
+}
+.action-btn {
+    margin: 5px;
+}
+.nav-card {
+    background: #0d1117;
+    border-radius: 10px;
+    padding: 15px;
+    margin: 10px 0;
+    border: 1px solid #30363d;
+    cursor: pointer;
+}
+.nav-card:hover {
+    border-color: #4ECDC4;
+}
+</style>
+""", unsafe_allow_html=True)
+
+# Initialize state manager
+state_manager = get_state_manager()
+
+
+def process_document_actual(file_bytes: bytes, filename: str, options: dict) -> dict:
+    """
+    Process document using the actual document processing pipeline.
+    Returns processing results with all extracted data.
+    """
+    import tempfile
+    import os
+
+    # Create temp file
+    suffix = Path(filename).suffix
+    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
+        tmp.write(file_bytes)
+        tmp_path = tmp.name
+
+    try:
+        # Try to use actual document processor
+        try:
+            from src.document.pipeline.processor import (
+                DocumentProcessor,
+                PipelineConfig,
+            )
+            from src.document.ocr import OCRConfig
+            from src.document.layout import LayoutConfig
+            from src.document.chunking.chunker import ChunkerConfig
+
+            # Configure chunking with table preservation options
+            chunker_config = ChunkerConfig(
+                preserve_table_structure=options.get("preserve_tables", True),
+                detect_table_headers=options.get("detect_headers", True),
+                chunk_tables=True,
+                chunk_figures=True,
+                include_captions=True,
+            )
+
+            # Configure layout detection
+            layout_config = LayoutConfig(
+                method="rule_based",
+                detect_tables=True,
+                detect_figures=True,
+                detect_headers=True,
+                detect_titles=True,
+                detect_lists=True,
+                min_confidence=0.3,  # Lower threshold to detect more regions
+                heading_font_ratio=1.1,  # More sensitive heading detection
+            )
+
+            # Configure pipeline with all options
+            config = PipelineConfig(
+                ocr=OCRConfig(engine=options.get("ocr_engine", "paddleocr")),
+                layout=layout_config,
+                chunking=chunker_config,
+                max_pages=options.get("max_pages", 10),
+                include_ocr_regions=True,
+                include_layout_regions=options.get("enable_layout", True),
+                generate_full_text=True,
+            )
+
+            processor = DocumentProcessor(config)
+            processor.initialize()
+
+            # Process document
+            result = processor.process(tmp_path)
+
+            # Convert to dict format for state
+            chunks_list = []
+            for chunk in result.chunks:
+                chunks_list.append({
+                    "chunk_id": chunk.chunk_id,
+                    "text": chunk.text,
+                    "page": chunk.page,
+                    "chunk_type": chunk.chunk_type.value,
+                    "confidence": chunk.confidence,
+                    "bbox": chunk.bbox.to_xyxy() if chunk.bbox else None,
+                })
+
+            ocr_regions = []
+            for region in result.ocr_regions:
+                ocr_regions.append({
+                    "text": region.text,
+                    "confidence": region.confidence,
+                    "page": region.page,
+                    "bbox": region.bbox.to_xyxy() if region.bbox else None,
+                })
+
+            layout_regions = []
+            for region in result.layout_regions:
+                layout_regions.append({
+                    "id": region.id,
+                    "type": region.type.value,
+                    "confidence": region.confidence,
+                    "page": region.page,
+                    "bbox": region.bbox.to_xyxy() if region.bbox else None,
+                })
+
+            return {
+                "success": True,
+                "raw_text": result.full_text,
+                "chunks": chunks_list,
+                "ocr_regions": ocr_regions,
+                "layout_regions": layout_regions,
+                "page_count": result.metadata.num_pages,
+                "ocr_confidence": result.metadata.ocr_confidence_avg or 0.0,
+                "layout_confidence": result.metadata.layout_confidence_avg or 0.0,
+            }
+
+        except Exception as e:
+            # Fallback: Use simple text extraction
+            return process_document_fallback(file_bytes, filename, options, str(e))
+
+    finally:
+        # Cleanup
+        if os.path.exists(tmp_path):
+            os.unlink(tmp_path)
+
+
+def process_document_fallback(file_bytes: bytes, filename: str, options: dict, reason: str) -> dict:
+    """
+    Fallback document processing using simple text extraction.
+    """
+    text = ""
+    page_count = 1
+
+    suffix = Path(filename).suffix.lower()
+
+    # Try PyMuPDF for PDFs
+    if suffix == ".pdf":
+        try:
+            import fitz
+            pdf_stream = io.BytesIO(file_bytes)
+            doc = fitz.open(stream=pdf_stream, filetype="pdf")
+            page_count = len(doc)
+            max_pages = min(options.get("max_pages", 5), page_count)
+
+            text_parts = []
+            for page_num in range(max_pages):
+                page = doc[page_num]
+                text_parts.append(f"--- Page {page_num + 1} ---\n{page.get_text()}")
+            text = "\n\n".join(text_parts)
+            doc.close()
+        except Exception as pdf_e:
+            text = f"PDF extraction failed: {pdf_e}"
+
+    elif suffix in [".txt", ".md"]:
+        try:
+            text = file_bytes.decode("utf-8")
+        except:
+            text = file_bytes.decode("latin-1", errors="ignore")
+
+    else:
+        text = f"Unsupported file type: {suffix}"
+
+    # Simple chunking
+    chunk_size = 500
+    overlap = 50
+    chunks = []
+
+    for i in range(0, len(text), chunk_size - overlap):
+        chunk_text = text[i:i + chunk_size]
+        if len(chunk_text.strip()) > 20:
+            chunks.append({
+                "chunk_id": f"chunk_{len(chunks)}",
+                "text": chunk_text,
+                "page": 0,
+                "chunk_type": "text",
+                "confidence": 0.9,
+                "bbox": None,
+            })
+
+    return {
+        "success": True,
+        "raw_text": text,
+        "chunks": chunks,
+        "ocr_regions": [],
+        "layout_regions": [],
+        "page_count": page_count,
+        "ocr_confidence": 0.9,
+        "layout_confidence": 0.0,
+        "fallback_reason": reason,
+    }
+
+
+def get_page_images(file_bytes: bytes, filename: str, max_pages: int = 5) -> list:
+    """Extract page images from PDF for visualization."""
+    images = []
+    suffix = Path(filename).suffix.lower()
+
+    if suffix == ".pdf":
+        try:
+            import fitz
+            pdf_stream = io.BytesIO(file_bytes)
+            doc = fitz.open(stream=pdf_stream, filetype="pdf")
+            page_count = min(len(doc), max_pages)
+
+            for page_num in range(page_count):
+                page = doc[page_num]
+                pix = page.get_pixmap(dpi=100)
+                img_bytes = pix.tobytes("png")
+                images.append({
+                    "page": page_num,
+                    "data": base64.b64encode(img_bytes).decode(),
+                    "width": pix.width,
+                    "height": pix.height,
+                })
+            doc.close()
+        except:
+            pass
+
+    return images
+
+
+# Header
+st.markdown("# 🔬 Live Document Processing")
+st.markdown("Process documents in real-time with auto-indexing to RAG")
+
+# Global status bar
+render_global_status_bar()
+
+st.markdown("---")
+
+# Main content
+col_upload, col_status = st.columns([2, 1])
+
+with col_upload:
+    st.markdown("### 📤 Upload Document")
+
+    uploaded_file = st.file_uploader(
+        "Choose a document",
+        type=["pdf", "txt", "md"],
+        help="Upload PDF, TXT, or MD files for processing"
+    )
+
+    # Or select from existing files
+    docs_path = PROJECT_ROOT / "Dataset"
+    existing_docs = sorted([f.name for f in docs_path.glob("*.pdf")]) if docs_path.exists() else []
+
+    if existing_docs:
+        st.markdown("**Or select from samples:**")
+        selected_sample = st.selectbox("Sample documents", ["-- Select --"] + existing_docs)
+
+with col_status:
+    st.markdown("### 📊 System Status")
+
+    ollama_ok, models = check_ollama()
+    rag_system = get_unified_rag_system()
+
+    status_cols = st.columns(2)
+    with status_cols[0]:
+        if ollama_ok:
+            st.success(f"Ollama ({len(models)})")
+        else:
+            st.error("Ollama Offline")
+    with status_cols[1]:
+        if rag_system["status"] == "ready":
+            st.success("RAG Ready")
+        else:
+            st.error("RAG Error")
+
+    # State summary
+    summary = state_manager.get_summary()
+    st.metric("Processed Docs", summary["total_documents"])
+    st.metric("Indexed Chunks", summary["total_indexed_chunks"])
+
+st.markdown("---")
+
+# Processing Options
+st.markdown("### ⚙️ Processing Options")
+
+opt_cols = st.columns(4)
+with opt_cols[0]:
+    ocr_engine = st.radio("OCR Engine", ["paddleocr", "tesseract"], horizontal=True,
+                          help="PaddleOCR is faster and more accurate for most documents")
+with opt_cols[1]:
+    max_pages = st.slider("Max pages", 1, 50, 10, help="Maximum number of pages to process")
+with opt_cols[2]:
+    enable_layout = st.checkbox("Layout detection", value=True,
+                                help="Detect tables, figures, headings and other layout elements")
+with opt_cols[3]:
+    auto_index = st.checkbox("Auto-index to RAG", value=True,
+                            help="Automatically index processed documents for RAG queries")
+
+# Advanced options (expanded by default for visibility)
+with st.expander("🔧 Advanced Options", expanded=False):
+    adv_cols = st.columns(3)
+    with adv_cols[0]:
+        preserve_tables = st.checkbox("Preserve table structure", value=True,
+                                     help="Convert tables to markdown format with structure")
+    with adv_cols[1]:
+        detect_headers = st.checkbox("Detect table headers", value=True,
+                                    help="Automatically identify header rows in tables")
+    with adv_cols[2]:
+        generate_embeddings = st.checkbox("Generate embeddings", value=True,
+                                         help="Create embeddings for semantic search")
+
+# Determine what to process
+file_to_process = None
+file_bytes = None
+filename = None
+
+if uploaded_file is not None:
+    file_bytes = uploaded_file.read()
+    filename = uploaded_file.name
+    file_to_process = "upload"
+elif existing_docs and selected_sample != "-- Select --":
+    file_path = docs_path / selected_sample
+    file_bytes = file_path.read_bytes()
+    filename = selected_sample
+    file_to_process = "sample"
+
+# Process button
+if file_to_process and st.button("🚀 Start Processing", type="primary", use_container_width=True):
+
+    # Generate document ID
+    content_hash = hashlib.md5(file_bytes[:1000]).hexdigest()[:8]
+    doc_id = generate_doc_id(filename, content_hash)
+
+    # Start processing in state manager
+    state_manager.start_processing(doc_id, filename)
+
+    # Pipeline stages
+    stages = [
+        ("loading", "📄 Loading Document", "Reading and preparing document..."),
+        ("ocr", f"🔍 {ocr_engine.upper()} Extraction", "Extracting text from document..."),
+        ("layout", "📐 Layout Detection", "Identifying document structure..."),
+        ("chunking", "✂️ Semantic Chunking", "Creating meaningful text chunks..."),
+        ("indexing", "📚 RAG Indexing", "Adding to vector store..."),
+    ]
+
+    # Progress container
+    progress_container = st.container()
+    results_container = st.container()
+
+    with progress_container:
+        progress_bar = st.progress(0)
+        status_text = st.empty()
+
+        # Metrics row
+        metric_cols = st.columns(5)
+        metric_placeholders = {
+            "pages": metric_cols[0].empty(),
+            "ocr_regions": metric_cols[1].empty(),
+            "layout_regions": metric_cols[2].empty(),
+            "chunks": metric_cols[3].empty(),
+            "confidence": metric_cols[4].empty(),
+        }
+
+        processing_start = time.time()
+        processing_result = None
+        error_msg = None
+
+        try:
+            # Stage 1: Loading
+            status_text.markdown("**📄 Loading document...**")
+            state_manager.update_processing(doc_id, "loading", 0.1, "Loading document...")
+            progress_bar.progress(10)
+            time.sleep(0.3)
+
+            # Get page images for visualization
+            page_images = get_page_images(file_bytes, filename, max_pages)
+            metric_placeholders["pages"].metric("Pages", len(page_images) if page_images else "N/A")
+
+            # Stage 2-3: OCR + Layout
+            status_text.markdown(f"**🔍 Running {ocr_engine.upper()}...**")
+            state_manager.update_processing(doc_id, "ocr", 0.3, f"Running {ocr_engine}...")
+            progress_bar.progress(30)
+
+            # Actual processing with all options
+            options = {
+                "ocr_engine": ocr_engine,
+                "max_pages": max_pages,
+                "enable_layout": enable_layout,
+                "preserve_tables": preserve_tables,
+                "detect_headers": detect_headers,
+                "generate_embeddings": generate_embeddings,
+            }
+            processing_result = process_document_actual(file_bytes, filename, options)
+
+            # Update metrics
+            metric_placeholders["pages"].metric("Pages", processing_result.get("page_count", 0))
+            metric_placeholders["ocr_regions"].metric("OCR Regions", len(processing_result.get("ocr_regions", [])))
+
+            status_text.markdown("**📐 Layout detection...**")
+            state_manager.update_processing(doc_id, "layout", 0.5, "Detecting layout...")
+            progress_bar.progress(50)
+            time.sleep(0.2)
+
+            metric_placeholders["layout_regions"].metric("Layout Regions", len(processing_result.get("layout_regions", [])))
+
+            # Stage 4: Chunking
+            status_text.markdown("**✂️ Creating chunks...**")
+            state_manager.update_processing(doc_id, "chunking", 0.7, "Creating chunks...")
+            progress_bar.progress(70)
+            time.sleep(0.2)
+
+            chunks = processing_result.get("chunks", [])
+            metric_placeholders["chunks"].metric("Chunks", len(chunks))
+            metric_placeholders["confidence"].metric(
+                "Confidence",
+                f"{processing_result.get('ocr_confidence', 0) * 100:.0f}%"
+            )
+
+            # Stage 5: RAG Indexing
+            indexed_count = 0
+            if auto_index and rag_system["status"] == "ready" and chunks:
+                status_text.markdown("**📚 Indexing to RAG...**")
+                state_manager.update_processing(doc_id, "indexing", 0.9, "Indexing to RAG...")
+                progress_bar.progress(90)
+
+                # Auto-index
+                index_result = auto_index_processed_document(
+                    doc_id=doc_id,
+                    text=processing_result.get("raw_text", ""),
+                    chunks=chunks,
+                    metadata={"filename": filename, "source": file_to_process}
+                )
+
+                if index_result["success"]:
+                    indexed_count = index_result["num_chunks"]
+                    state_manager.mark_indexed(doc_id, indexed_count)
+
+            # Complete
+            progress_bar.progress(100)
+            processing_time = time.time() - processing_start
+
+            # Add to state manager
+            state_doc = StateDocument(
+                doc_id=doc_id,
+                filename=filename,
+                file_type=Path(filename).suffix[1:].upper(),
+                raw_text=processing_result.get("raw_text", ""),
+                chunks=chunks,
+                page_count=processing_result.get("page_count", 1),
+                page_images=[img["data"] for img in page_images],
+                ocr_regions=processing_result.get("ocr_regions", []),
+                layout_data={"regions": processing_result.get("layout_regions", [])},
+                indexed=indexed_count > 0,
+                indexed_chunks=indexed_count,
+                processing_time=processing_time,
+            )
+            state_manager.add_document(state_doc)
+            state_manager.complete_processing(doc_id, success=True)
+            state_manager.set_active_document(doc_id)
+
+            status_text.success(f"✅ Processing complete in {processing_time:.2f}s!")
+
+        except Exception as e:
+            error_msg = str(e)
+            state_manager.complete_processing(doc_id, success=False, error=error_msg)
+            status_text.error(f"❌ Processing failed: {error_msg}")
+
+    # Results
+    if processing_result and processing_result.get("success"):
+        with results_container:
+            st.markdown("---")
+            st.markdown("### 📋 Processing Results")
+
+            # Summary cards
+            sum_cols = st.columns(5)
+            sum_cols[0].markdown(f"""
+            <div class="metric-card">
+                <div class="metric-value">{processing_result.get('page_count', 0)}</div>
+                <div class="metric-label">Pages</div>
+            </div>
+            """, unsafe_allow_html=True)
+            sum_cols[1].markdown(f"""
+            <div class="metric-card">
+                <div class="metric-value">{len(processing_result.get('ocr_regions', []))}</div>
+                <div class="metric-label">OCR Regions</div>
+            </div>
+            """, unsafe_allow_html=True)
+            sum_cols[2].markdown(f"""
+            <div class="metric-card">
+                <div class="metric-value">{len(processing_result.get('layout_regions', []))}</div>
+                <div class="metric-label">Layout Regions</div>
+            </div>
+            """, unsafe_allow_html=True)
+            sum_cols[3].markdown(f"""
+            <div class="metric-card">
+                <div class="metric-value">{len(chunks)}</div>
+                <div class="metric-label">Chunks</div>
+            </div>
+            """, unsafe_allow_html=True)
+            sum_cols[4].markdown(f"""
+            <div class="metric-card">
+                <div class="metric-value">{indexed_count}</div>
+                <div class="metric-label">Indexed</div>
+            </div>
+            """, unsafe_allow_html=True)
+
+            # Show fallback warning prominently if fallback was used
+            if processing_result.get("fallback_reason"):
+                st.error(f"⚠️ **Fallback Mode**: Document processor failed, using simple text extraction. Layout detection unavailable. Reason: {processing_result['fallback_reason']}")
+
+            # Tabs for detailed results
+            tab_text, tab_chunks, tab_layout, tab_pages = st.tabs([
+                "📝 Extracted Text",
+                "📦 Chunks",
+                "🗺️ Layout",
+                "📄 Pages"
+            ])
+
+            with tab_text:
+                text_preview = processing_result.get("raw_text", "")[:5000]
+                if len(processing_result.get("raw_text", "")) > 5000:
+                    text_preview += "\n\n... [truncated] ..."
+                st.text_area("Full Text", text_preview, height=400)
+
+                if processing_result.get("fallback_reason"):
+                    st.warning(f"Using fallback extraction: {processing_result['fallback_reason']}")
+
+            with tab_chunks:
+                for i, chunk in enumerate(chunks[:20]):
+                    chunk_type = chunk.get("chunk_type", "text")
+                    conf = chunk.get("confidence", 0)
+                    color = "#4ECDC4" if conf > 0.8 else "#ffc107" if conf > 0.6 else "#dc3545"
+
+                    with st.expander(f"[{i+1}] {chunk_type.upper()} - {chunk.get('text', '')[:50]}..."):
+                        col1, col2, col3 = st.columns([2, 1, 1])
+                        col1.markdown(f"**Chunk ID:** `{chunk.get('chunk_id', 'N/A')}`")
+                        col2.markdown(f"**Page:** {chunk.get('page', 0) + 1}")
+                        col3.markdown(f"**Confidence:** <span style='color:{color}'>{conf:.0%}</span>", unsafe_allow_html=True)
+                        st.code(chunk.get("text", ""), language=None)
+
+                if len(chunks) > 20:
+                    st.info(f"Showing 20 of {len(chunks)} chunks")
+
+            with tab_layout:
+                layout_regions = processing_result.get("layout_regions", [])
+                if layout_regions:
+                    # Group by type
+                    by_type = {}
+                    for r in layout_regions:
+                        t = r.get("type", "unknown")
+                        by_type[t] = by_type.get(t, 0) + 1
+
+                    st.markdown("**Detected Region Types:**")
+                    type_cols = st.columns(min(len(by_type), 6))
+                    for i, (rtype, count) in enumerate(by_type.items()):
+                        type_cols[i % 6].metric(rtype.title(), count)
+
+                    st.markdown("**Regions:**")
+                    for r in layout_regions[:15]:
+                        conf = r.get("confidence", 0)
+                        color = "#4ECDC4" if conf > 0.8 else "#ffc107" if conf > 0.6 else "#dc3545"
+                        st.markdown(f"- **{r.get('type', 'unknown').upper()}** (page {r.get('page', 0) + 1}) - Confidence: <span style='color:{color}'>{conf:.0%}</span>", unsafe_allow_html=True)
+                else:
+                    # Provide helpful message based on cause
+                    if processing_result.get("fallback_reason"):
+                        st.warning("Layout detection unavailable - document processor is using fallback mode. Check the error message above.")
+                    elif not enable_layout:
+                        st.info("Layout detection is disabled. Enable it in the options above.")
+                    else:
+                        st.info("No layout regions detected. The document may have minimal structure or the OCR results didn't contain enough text patterns for layout analysis.")
+
+            with tab_pages:
+                if page_images:
+                    for img_data in page_images:
+                        st.markdown(f"**Page {img_data['page'] + 1}** ({img_data['width']}x{img_data['height']})")
+                        st.image(
+                            f"data:image/png;base64,{img_data['data']}",
+                            use_container_width=True
+                        )
+                else:
+                    st.info("Page images not available")
+
+            # Navigation to other modules
+            st.markdown("---")
+            st.markdown("### 🔗 Continue With This Document")
+
+            nav_cols = st.columns(3)
+
+            with nav_cols[0]:
+                st.markdown("""
+                <div class="nav-card">
+                    <h4>💬 Interactive RAG</h4>
+                    <p style="color: #8b949e;">Ask questions about this document using the RAG system.</p>
+                </div>
+                """, unsafe_allow_html=True)
+                if st.button("Go to Interactive RAG", key="nav_rag", use_container_width=True):
+                    st.switch_page("pages/2_💬_Interactive_RAG.py")
+
+            with nav_cols[1]:
+                st.markdown("""
+                <div class="nav-card">
+                    <h4>📄 Document Viewer</h4>
+                    <p style="color: #8b949e;">View chunks, layout, and visual annotations.</p>
+                </div>
+                """, unsafe_allow_html=True)
+                if st.button("Go to Document Viewer", key="nav_viewer", use_container_width=True):
+                    st.switch_page("pages/5_📄_Document_Viewer.py")
+
+            with nav_cols[2]:
+                st.markdown("""
+                <div class="nav-card">
+                    <h4>🎯 Evidence Viewer</h4>
+                    <p style="color: #8b949e;">Inspect OCR regions and evidence grounding.</p>
+                </div>
+                """, unsafe_allow_html=True)
+                if st.button("Go to Evidence Viewer", key="nav_evidence", use_container_width=True):
+                    st.switch_page("pages/4_🎯_Evidence_Viewer.py")
+
+# Show recent processed documents
+st.markdown("---")
+st.markdown("### 📚 Recently Processed")
+
+all_docs = state_manager.get_all_documents()
+if all_docs:
+    for doc in reversed(all_docs[-5:]):
+        col1, col2, col3, col4 = st.columns([3, 1, 1, 1])
+        col1.markdown(f"**{doc.filename}** (`{doc.doc_id[:8]}...`)")
+        col2.markdown(f"📄 {doc.page_count} pages")
+        col3.markdown(f"📦 {len(doc.chunks)} chunks")
+        if doc.indexed:
+            col4.success(f"✓ Indexed ({doc.indexed_chunks})")
+        else:
+            col4.warning("Not indexed")
+else:
+    st.info("No documents processed yet. Upload or select a document above.")

demo/pages/2_💬_Interactive_RAG.py

@@ -0,0 +1,844 @@
+"""
+Interactive Multi-Agentic RAG - SPARKNET
+
+Query your documents using the unified RAG system with document filtering
+and real-time chunk inspection.
+"""
+
+import streamlit as st
+import sys
+from pathlib import Path
+import time
+import hashlib
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+sys.path.insert(0, str(PROJECT_ROOT / "demo"))
+
+# Import unified RAG configuration and state manager
+from rag_config import (
+    get_unified_rag_system,
+    get_store_stats,
+    index_document,
+    query_rag,
+    check_ollama,
+    get_indexed_documents,
+    search_similar_chunks,
+)
+from state_manager import (
+    get_state_manager,
+    render_global_status_bar,
+)
+import re
+from collections import Counter
+
+
+def clean_filename_for_question(filename: str) -> str:
+    """
+    Clean a filename to make it suitable for use in a question.
+    Handles cases like 'Red_Hat_NA.pdf' -> 'Red Hat' (removing short tokens).
+    """
+    # Remove extension
+    name = Path(filename).stem
+
+    # Replace separators with spaces
+    name = re.sub(r'[_\-\.]+', ' ', name)
+
+    # Split into words and filter
+    words = name.split()
+
+    # Remove very short tokens (like 'NA', 'V1', etc.) and numbers
+    cleaned_words = []
+    for word in words:
+        # Skip if too short (1-2 chars) unless it's a known acronym
+        if len(word) <= 2 and not word.upper() in ['AI', 'ML', 'NLP', 'API', 'UI', 'UX']:
+            continue
+        # Skip pure numbers or version-like strings
+        if re.match(r'^[vV]?\d+$', word):
+            continue
+        # Skip common file suffixes
+        if word.lower() in ['final', 'draft', 'copy', 'new', 'old', 'v1', 'v2']:
+            continue
+        cleaned_words.append(word)
+
+    # Join and clean up extra spaces
+    result = ' '.join(cleaned_words).strip()
+
+    # If result is too short, return None
+    if len(result) < 3:
+        return None
+
+    return result
+
+
+def generate_dynamic_questions(state_manager, indexed_docs, max_questions=4):
+    """
+    Generate dynamic suggested questions based on indexed document content.
+
+    Analyzes:
+    - Document titles and filenames
+    - Chunk content for key topics
+    - Table presence
+    - Document types
+    - Detected entities and keywords
+    """
+    questions = []
+
+    # Get all indexed documents from state manager
+    all_docs = state_manager.get_all_documents()
+    indexed_doc_list = [d for d in all_docs if d.indexed]
+
+    if not indexed_doc_list and not indexed_docs:
+        # No documents indexed - return generic questions
+        return [
+            "What is the main topic of this document?",
+            "Summarize the key points",
+            "What are the main findings?",
+            "List the important details",
+        ]
+
+    # Collect document info
+    doc_names = []
+    all_text_samples = []
+    has_tables = False
+    has_figures = False
+    doc_types = set()
+
+    for doc in indexed_doc_list:
+        doc_names.append(doc.filename)
+        doc_types.add(doc.file_type.lower())
+
+        # Sample text from chunks
+        for chunk in doc.chunks[:10]:  # First 10 chunks
+            chunk_text = chunk.get('text', '') if isinstance(chunk, dict) else str(chunk)
+            all_text_samples.append(chunk_text[:500])
+
+            # Check for tables
+            chunk_type = chunk.get('chunk_type', '') if isinstance(chunk, dict) else ''
+            if 'table' in chunk_type.lower():
+                has_tables = True
+            if 'figure' in chunk_type.lower() or 'chart' in chunk_type.lower():
+                has_figures = True
+
+    # Also check indexed_docs from RAG system
+    for doc_info in indexed_docs[:5]:
+        if isinstance(doc_info, dict):
+            doc_names.append(doc_info.get('filename', doc_info.get('doc_id', '')))
+
+    # Extract key topics from text samples
+    combined_text = ' '.join(all_text_samples).lower()
+
+    # Extract potential topics (simple keyword extraction)
+    stop_words = {
+        'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for',
+        'of', 'with', 'by', 'from', 'is', 'are', 'was', 'were', 'be', 'been',
+        'being', 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would',
+        'could', 'should', 'may', 'might', 'must', 'shall', 'can', 'need',
+        'this', 'that', 'these', 'those', 'it', 'its', 'as', 'if', 'when',
+        'than', 'so', 'no', 'not', 'only', 'own', 'same', 'too', 'very',
+        'just', 'also', 'now', 'here', 'there', 'where', 'why', 'how', 'all',
+        'each', 'every', 'both', 'few', 'more', 'most', 'other', 'some', 'such',
+        'into', 'through', 'during', 'before', 'after', 'above', 'below', 'between',
+        'under', 'again', 'further', 'then', 'once', 'any', 'about', 'which', 'who',
+        'page', 'document', 'file', 'section', 'chapter', 'figure', 'table',
+    }
+
+    # Extract words (3+ chars, not numbers)
+    words = re.findall(r'\b[a-z]{3,}\b', combined_text)
+    meaningful_words = [w for w in words if w not in stop_words and len(w) > 3]
+    word_freq = Counter(meaningful_words)
+    top_topics = [word for word, count in word_freq.most_common(15) if count > 2]
+
+    # Generate questions based on top topics (prioritize content-based questions)
+    if top_topics:
+        topic = top_topics[0]
+        questions.append(f"What does the document say about {topic}?")
+
+        if len(top_topics) > 1:
+            questions.append(f"Explain the {top_topics[1]} mentioned in the document")
+
+        if len(top_topics) > 2:
+            questions.append(f"How are {top_topics[0]} and {top_topics[2]} related?")
+
+    # Generate questions based on clean document names (only if name is meaningful)
+    for name in doc_names[:2]:
+        clean_name = clean_filename_for_question(name)
+        if clean_name and len(clean_name) > 5:
+            questions.append(f"Summarize the {clean_name} document")
+            break  # Only use one document name question
+
+    # Add table-specific question if tables detected
+    if has_tables:
+        questions.append("What data is presented in the tables?")
+
+    # Add figure-specific question if figures detected
+    if has_figures:
+        questions.append("What do the figures and charts show?")
+
+    # Add document-type specific questions
+    if 'pdf' in doc_types:
+        questions.append("What are the main conclusions?")
+    if 'docx' in doc_types or 'doc' in doc_types:
+        questions.append("What recommendations are made?")
+    if 'xlsx' in doc_types or 'xls' in doc_types:
+        questions.append("What trends are visible in the data?")
+
+    # Add content-aware generic questions
+    generic_questions = [
+        "Summarize the key points in this document",
+        "What are the main findings discussed?",
+        "What methodology or approach is described?",
+        "What are the important takeaways?",
+        "List the main topics covered",
+        "What problems or challenges are mentioned?",
+    ]
+
+    # Fill remaining slots with generic questions
+    for q in generic_questions:
+        if len(questions) >= max_questions:
+            break
+        if q not in questions:
+            questions.append(q)
+
+    # Ensure we have unique questions and limit to max
+    seen = set()
+    unique_questions = []
+    for q in questions:
+        q_lower = q.lower()
+        if q_lower not in seen:
+            seen.add(q_lower)
+            unique_questions.append(q)
+            if len(unique_questions) >= max_questions:
+                break
+
+    # Fallback if we don't have enough
+    while len(unique_questions) < max_questions:
+        fallback = [
+            "What is this document about?",
+            "Summarize the key points",
+            "What are the main findings?",
+            "What conclusions are drawn?",
+        ]
+        for q in fallback:
+            if q not in unique_questions:
+                unique_questions.append(q)
+                break
+        if len(unique_questions) >= max_questions:
+            break
+
+    return unique_questions[:max_questions]
+
+st.set_page_config(
+    page_title="Interactive RAG - SPARKNET",
+    page_icon="💬",
+    layout="wide"
+)
+
+# Custom CSS
+st.markdown("""
+<style>
+.chat-message {
+    padding: 15px;
+    border-radius: 12px;
+    margin: 10px 0;
+}
+.user-message {
+    background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+    color: white;
+}
+.assistant-message {
+    background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);
+    color: #eee;
+}
+.source-card {
+    background: #0d1117;
+    border-radius: 8px;
+    padding: 12px;
+    margin: 8px 0;
+    border-left: 4px solid #4ECDC4;
+}
+.source-header {
+    font-size: 11px;
+    color: #888;
+    margin-bottom: 6px;
+}
+.source-text {
+    font-size: 13px;
+    color: #c9d1d9;
+    font-family: monospace;
+}
+.metric-box {
+    background: #161b22;
+    border-radius: 8px;
+    padding: 10px;
+    text-align: center;
+}
+.metric-value {
+    font-size: 20px;
+    font-weight: bold;
+    color: #4ECDC4;
+}
+.metric-label {
+    font-size: 10px;
+    color: #888;
+    text-transform: uppercase;
+}
+.pipeline-bar {
+    display: flex;
+    justify-content: center;
+    gap: 5px;
+    padding: 10px;
+    background: #0d1117;
+    border-radius: 8px;
+    margin: 10px 0;
+}
+.pipeline-step {
+    padding: 5px 12px;
+    border-radius: 15px;
+    font-size: 11px;
+    background: #21262d;
+    color: #8b949e;
+}
+.pipeline-step.active {
+    background: linear-gradient(90deg, #4ECDC4, #44a08d);
+    color: white;
+}
+.pipeline-step.done {
+    background: #238636;
+    color: white;
+}
+.doc-filter-card {
+    background: #161b22;
+    border-radius: 8px;
+    padding: 10px;
+    margin: 5px 0;
+    border: 1px solid #30363d;
+}
+.doc-filter-card.selected {
+    border-color: #4ECDC4;
+}
+.chunk-preview {
+    background: #0d1117;
+    border-radius: 6px;
+    padding: 8px;
+    margin: 4px 0;
+    font-size: 12px;
+    font-family: monospace;
+    max-height: 100px;
+    overflow-y: auto;
+}
+</style>
+""", unsafe_allow_html=True)
+
+
+def get_chunk_color(index: int) -> str:
+    """Get distinct color for citations."""
+    colors = [
+        "#FF6B6B", "#4ECDC4", "#45B7D1", "#96CEB4",
+        "#FFEAA7", "#DDA0DD", "#98D8C8", "#F7DC6F",
+    ]
+    return colors[index % len(colors)]
+
+
+# Initialize state manager
+state_manager = get_state_manager()
+
+# Get system status
+rag_system = get_unified_rag_system()
+ollama_ok, models = check_ollama()
+stats = get_store_stats()
+indexed_docs = get_indexed_documents()
+
+# Session state
+if "messages" not in st.session_state:
+    st.session_state.messages = []
+if "quick_indexed" not in st.session_state:
+    st.session_state.quick_indexed = []
+if "doc_filter" not in st.session_state:
+    st.session_state.doc_filter = None  # None = all documents
+
+
+# Header
+st.markdown("# 💬 Interactive RAG Chat")
+st.markdown("Ask questions about your indexed documents with multi-agent pipeline")
+
+# Global status bar
+render_global_status_bar()
+
+# Pipeline indicator
+st.markdown("""
+<div class="pipeline-bar">
+    <span class="pipeline-step">📝 Query</span>
+    <span>→</span>
+    <span class="pipeline-step">🎯 Plan</span>
+    <span>→</span>
+    <span class="pipeline-step">🔍 Retrieve</span>
+    <span>→</span>
+    <span class="pipeline-step">📊 Rerank</span>
+    <span>→</span>
+    <span class="pipeline-step">💬 Generate</span>
+    <span>→</span>
+    <span class="pipeline-step">✅ Validate</span>
+</div>
+""", unsafe_allow_html=True)
+
+# Status bar
+cols = st.columns(5)
+with cols[0]:
+    if ollama_ok:
+        st.success(f"Ollama ({len(models)})")
+    else:
+        st.error("Ollama Offline")
+with cols[1]:
+    if rag_system["status"] == "ready":
+        st.success("RAG Ready")
+    else:
+        st.error("RAG Error")
+with cols[2]:
+    st.info(f"{rag_system.get('llm_model', 'N/A').split(':')[0]}")
+with cols[3]:
+    chunk_count = stats.get('total_chunks', 0)
+    if chunk_count > 0:
+        st.success(f"{chunk_count} Chunks")
+    else:
+        st.warning("0 Chunks")
+with cols[4]:
+    st.info(f"{rag_system.get('embed_model', 'N/A').split(':')[0]}")
+
+if rag_system["status"] == "error":
+    with st.expander("RAG Error Details"):
+        st.code(rag_system["error"])
+
+st.markdown("---")
+
+# Sidebar
+with st.sidebar:
+    st.markdown("## 📚 Document Filter")
+
+    if indexed_docs:
+        st.markdown(f"**{len(indexed_docs)} documents indexed**")
+
+        # All documents option
+        if st.button(
+            "All Documents",
+            key="filter_all",
+            type="primary" if st.session_state.doc_filter is None else "secondary",
+            use_container_width=True
+        ):
+            st.session_state.doc_filter = None
+            st.rerun()
+
+        st.markdown("---")
+        st.markdown("**Filter by document:**")
+
+        # Document list
+        for doc in indexed_docs[:10]:
+            doc_id = doc.get("document_id", "unknown")
+            chunk_count = doc.get("chunk_count", 0)
+            is_selected = st.session_state.doc_filter == doc_id
+
+            if st.button(
+                f"📄 {doc_id[:20]}... ({chunk_count})",
+                key=f"filter_{doc_id}",
+                type="primary" if is_selected else "secondary",
+                use_container_width=True
+            ):
+                st.session_state.doc_filter = doc_id
+                st.rerun()
+
+        if len(indexed_docs) > 10:
+            st.caption(f"... and {len(indexed_docs) - 10} more")
+
+        # Show selected filter
+        if st.session_state.doc_filter:
+            st.markdown("---")
+            st.info(f"Filtering: {st.session_state.doc_filter[:25]}...")
+            if st.button("Clear Filter"):
+                st.session_state.doc_filter = None
+                st.rerun()
+    else:
+        st.info("No documents indexed yet")
+
+    st.markdown("---")
+    st.markdown("## 📤 Quick Index")
+    st.caption("Index text directly without leaving this page")
+
+    quick_text = st.text_area("Paste text:", height=120, key="quick_text",
+                              placeholder="Paste document text here...")
+    quick_name = st.text_input("Name:", value="quick_doc", key="quick_name")
+
+    if st.button("📥 Index Now", type="primary", use_container_width=True,
+                disabled=(rag_system["status"] != "ready")):
+        if quick_text.strip():
+            with st.spinner("Indexing..."):
+                doc_id = f"{quick_name}_{hashlib.md5(quick_text[:50].encode()).hexdigest()[:8]}"
+                result = index_document(
+                    text=quick_text,
+                    document_id=doc_id,
+                    metadata={"filename": quick_name, "source": "quick_index"}
+                )
+                if result["success"]:
+                    st.session_state.quick_indexed.append(quick_name)
+                    st.success(f"{result['num_chunks']} chunks indexed!")
+                    st.rerun()
+                else:
+                    st.error(f"Error: {result['error']}")
+        else:
+            st.warning("Enter some text first")
+
+    # Recently indexed
+    if st.session_state.quick_indexed:
+        st.markdown("---")
+        st.markdown("### Recently Indexed")
+        for doc in st.session_state.quick_indexed[-5:]:
+            st.caption(f"• {doc}")
+
+    st.markdown("---")
+    st.markdown("### Options")
+
+    show_sources = st.checkbox("Show sources", value=True)
+    show_metrics = st.checkbox("Show metrics", value=True)
+    show_chunk_preview = st.checkbox("Show chunk preview", value=False)
+
+    if st.button("Clear Chat"):
+        st.session_state.messages = []
+        st.rerun()
+
+# Main chat area
+if stats.get('total_chunks', 0) == 0:
+    st.warning("No documents indexed yet!")
+    st.markdown("""
+    **To get started:**
+    1. Use the **Quick Index** in the sidebar to paste and index text
+    2. Or go to **🔬 Live Processing** page to upload and process documents
+
+    Once you've indexed some content, come back here to ask questions!
+    """)
+
+    # Sample text for quick start
+    with st.expander("Try with sample text"):
+        sample = """SPARKNET is a multi-agentic document intelligence framework.
+It uses RAG (Retrieval-Augmented Generation) for document Q&A.
+
+Key features:
+- PDF, TXT, MD document processing
+- Visual chunk segmentation
+- Hybrid retrieval (dense + sparse)
+- Cross-encoder reranking
+- Grounded answer generation with citations
+- Hallucination detection and validation
+
+The system uses multiple specialized agents:
+1. Query Planner - analyzes and decomposes queries
+2. Retriever - performs hybrid search
+3. Reranker - scores relevance with cross-encoder
+4. Synthesizer - generates grounded answers
+5. Critic - validates for hallucination"""
+
+        st.code(sample, language=None)
+        if st.button("Index This Sample"):
+            result = index_document(
+                text=sample,
+                document_id="sparknet_sample",
+                metadata={"filename": "sparknet_sample", "source": "sample"}
+            )
+            if result["success"]:
+                st.success(f"Indexed {result['num_chunks']} chunks!")
+                st.rerun()
+
+    # Navigation
+    col1, col2 = st.columns(2)
+    with col1:
+        if st.button("🔬 Go to Live Processing", type="primary", use_container_width=True):
+            st.switch_page("pages/1_🔬_Live_Processing.py")
+    with col2:
+        if st.button("📄 Go to Document Viewer", use_container_width=True):
+            st.switch_page("pages/5_📄_Document_Viewer.py")
+
+else:
+    # Check if we need to process a pending user message (from sample question click)
+    pending_query = None
+    if st.session_state.messages and st.session_state.messages[-1]["role"] == "user":
+        # Check if there's no assistant response after the last user message
+        pending_query = st.session_state.messages[-1]["content"]
+
+    # Display chat history (except pending query which we'll process below)
+    messages_to_display = st.session_state.messages[:-1] if pending_query else st.session_state.messages
+    for msg in messages_to_display:
+        with st.chat_message(msg["role"]):
+            st.markdown(msg["content"])
+
+            if msg["role"] == "assistant" and "metadata" in msg:
+                meta = msg["metadata"]
+
+                # Metrics
+                if show_metrics and meta:
+                    m_cols = st.columns(4)
+                    with m_cols[0]:
+                        st.markdown(f'<div class="metric-box"><div class="metric-value">{meta.get("latency_ms", 0):.0f}ms</div><div class="metric-label">Latency</div></div>', unsafe_allow_html=True)
+                    with m_cols[1]:
+                        st.markdown(f'<div class="metric-box"><div class="metric-value">{meta.get("num_sources", 0)}</div><div class="metric-label">Sources</div></div>', unsafe_allow_html=True)
+                    with m_cols[2]:
+                        conf = meta.get("confidence", 0)
+                        color = "#4ECDC4" if conf > 0.6 else "#ffc107" if conf > 0.3 else "#dc3545"
+                        st.markdown(f'<div class="metric-box"><div class="metric-value" style="color:{color}">{conf:.0%}</div><div class="metric-label">Confidence</div></div>', unsafe_allow_html=True)
+                    with m_cols[3]:
+                        val = "✓" if meta.get("validated") else "?"
+                        st.markdown(f'<div class="metric-box"><div class="metric-value">{val}</div><div class="metric-label">Validated</div></div>', unsafe_allow_html=True)
+
+                # Sources
+                if show_sources and "citations" in msg and msg["citations"]:
+                    with st.expander(f"Sources ({len(msg['citations'])})"):
+                        for i, cite in enumerate(msg["citations"]):
+                            color = get_chunk_color(i)
+                            st.markdown(f"""
+                            <div class="source-card" style="border-left-color: {color};">
+                                <div class="source-header">
+                                    <strong>[{cite.get('index', i+1)}]</strong> • Relevance: {cite.get('relevance_score', 0):.0%}
+                                </div>
+                                <div class="source-text">{cite.get('text_snippet', '')[:300]}...</div>
+                            </div>
+                            """, unsafe_allow_html=True)
+
+    # Show current filter
+    if st.session_state.doc_filter:
+        st.info(f"Searching in: **{st.session_state.doc_filter}** — [Clear filter in sidebar]")
+
+    # Process pending query from sample question click
+    if pending_query:
+        with st.chat_message("user"):
+            st.markdown(pending_query)
+
+        with st.chat_message("assistant"):
+            if rag_system["status"] != "ready":
+                st.error("RAG system not ready")
+                st.session_state.messages.append({"role": "assistant", "content": "RAG system not ready"})
+            else:
+                # Show progress
+                progress = st.progress(0)
+                status = st.empty()
+
+                stages = ["Planning", "Retrieving", "Reranking", "Generating", "Validating"]
+                for i, stage in enumerate(stages):
+                    status.markdown(f"**{stage}...**")
+                    progress.progress((i + 1) * 20)
+                    time.sleep(0.15)
+
+                # Build filters for document
+                filters = None
+                if st.session_state.doc_filter:
+                    filters = {"document_id": st.session_state.doc_filter}
+
+                # Query RAG
+                response, error = query_rag(pending_query, filters=filters)
+
+                progress.empty()
+                status.empty()
+
+                if error:
+                    st.error(f"Error: {error}")
+                    st.session_state.messages.append({"role": "assistant", "content": f"Error: {error}"})
+                elif response:
+                    # Display answer
+                    st.markdown(response.answer)
+
+                    # Build metadata
+                    metadata = {
+                        "latency_ms": response.latency_ms,
+                        "num_sources": response.num_sources,
+                        "confidence": response.confidence,
+                        "validated": response.validated,
+                    }
+
+                    # Display metrics
+                    if show_metrics:
+                        m_cols = st.columns(4)
+                        with m_cols[0]:
+                            st.markdown(f'<div class="metric-box"><div class="metric-value">{metadata.get("latency_ms", 0):.0f}ms</div><div class="metric-label">Latency</div></div>', unsafe_allow_html=True)
+                        with m_cols[1]:
+                            st.markdown(f'<div class="metric-box"><div class="metric-value">{metadata.get("num_sources", 0)}</div><div class="metric-label">Sources</div></div>', unsafe_allow_html=True)
+                        with m_cols[2]:
+                            conf = metadata.get("confidence", 0)
+                            color = "#4ECDC4" if conf > 0.6 else "#ffc107" if conf > 0.3 else "#dc3545"
+                            st.markdown(f'<div class="metric-box"><div class="metric-value" style="color:{color}">{conf:.0%}</div><div class="metric-label">Confidence</div></div>', unsafe_allow_html=True)
+                        with m_cols[3]:
+                            val = "✓" if metadata.get("validated") else "?"
+                            st.markdown(f'<div class="metric-box"><div class="metric-value">{val}</div><div class="metric-label">Validated</div></div>', unsafe_allow_html=True)
+
+                    # Build citations list
+                    citations = []
+                    if hasattr(response, 'citations') and response.citations:
+                        for i, cite in enumerate(response.citations):
+                            citations.append({
+                                "index": i + 1,
+                                "text_snippet": cite.text_snippet if hasattr(cite, 'text_snippet') else str(cite),
+                                "relevance_score": cite.relevance_score if hasattr(cite, 'relevance_score') else 0.0,
+                            })
+
+                    # Store message with metadata
+                    st.session_state.messages.append({
+                        "role": "assistant",
+                        "content": response.answer,
+                        "metadata": metadata,
+                        "citations": citations,
+                    })
+                else:
+                    st.warning("No response from RAG system")
+                    st.session_state.messages.append({"role": "assistant", "content": "No response from RAG system"})
+
+    # Chat input
+    if prompt := st.chat_input("Ask about your documents..."):
+        # Add user message
+        st.session_state.messages.append({"role": "user", "content": prompt})
+
+        with st.chat_message("user"):
+            st.markdown(prompt)
+
+        with st.chat_message("assistant"):
+            if rag_system["status"] != "ready":
+                st.error("RAG system not ready")
+                st.session_state.messages.append({"role": "assistant", "content": "RAG system not ready"})
+            else:
+                # Show progress
+                progress = st.progress(0)
+                status = st.empty()
+
+                stages = ["Planning", "Retrieving", "Reranking", "Generating", "Validating"]
+                for i, stage in enumerate(stages):
+                    status.markdown(f"**{stage}...**")
+                    progress.progress((i + 1) * 20)
+                    time.sleep(0.15)
+
+                # Build filters for document
+                filters = None
+                if st.session_state.doc_filter:
+                    filters = {"document_id": st.session_state.doc_filter}
+
+                # Query RAG
+                response, error = query_rag(prompt, filters=filters)
+
+                progress.empty()
+                status.empty()
+
+                if error:
+                    st.error(f"Error: {error}")
+                    st.session_state.messages.append({"role": "assistant", "content": f"Error: {error}"})
+                elif response:
+                    # Display answer
+                    st.markdown(response.answer)
+
+                    # Build metadata
+                    metadata = {
+                        "latency_ms": response.latency_ms,
+                        "num_sources": response.num_sources,
+                        "confidence": response.confidence,
+                        "validated": response.validated,
+                    }
+
+                    # Display metrics
+                    if show_metrics:
+                        m_cols = st.columns(4)
+                        with m_cols[0]:
+                            st.markdown(f'<div class="metric-box"><div class="metric-value">{response.latency_ms:.0f}ms</div><div class="metric-label">Latency</div></div>', unsafe_allow_html=True)
+                        with m_cols[1]:
+                            st.markdown(f'<div class="metric-box"><div class="metric-value">{response.num_sources}</div><div class="metric-label">Sources</div></div>', unsafe_allow_html=True)
+                        with m_cols[2]:
+                            conf_color = "#4ECDC4" if response.confidence > 0.6 else "#ffc107" if response.confidence > 0.3 else "#dc3545"
+                            st.markdown(f'<div class="metric-box"><div class="metric-value" style="color:{conf_color}">{response.confidence:.0%}</div><div class="metric-label">Confidence</div></div>', unsafe_allow_html=True)
+                        with m_cols[3]:
+                            val_icon = "✓" if response.validated else "?"
+                            st.markdown(f'<div class="metric-box"><div class="metric-value">{val_icon}</div><div class="metric-label">Validated</div></div>', unsafe_allow_html=True)
+
+                    # Display sources
+                    citations = []
+                    if show_sources and response.citations:
+                        with st.expander(f"Sources ({len(response.citations)})"):
+                            for i, cite in enumerate(response.citations):
+                                color = get_chunk_color(i)
+                                citations.append({
+                                    "index": cite.index,
+                                    "relevance_score": cite.relevance_score,
+                                    "text_snippet": cite.text_snippet,
+                                })
+                                st.markdown(f"""
+                                <div class="source-card" style="border-left-color: {color};">
+                                    <div class="source-header">
+                                        <strong>[{cite.index}]</strong> • Relevance: {cite.relevance_score:.0%}
+                                    </div>
+                                    <div class="source-text">{cite.text_snippet[:300]}...</div>
+                                </div>
+                                """, unsafe_allow_html=True)
+
+                    # Chunk preview (semantic search)
+                    if show_chunk_preview:
+                        with st.expander("Chunk Preview (Top Matches)"):
+                            chunks = search_similar_chunks(
+                                prompt,
+                                top_k=5,
+                                doc_filter=st.session_state.doc_filter
+                            )
+                            for i, chunk in enumerate(chunks):
+                                sim = chunk.get("similarity", 0)
+                                color = "#4ECDC4" if sim > 0.7 else "#ffc107" if sim > 0.5 else "#8b949e"
+                                st.markdown(f"""
+                                <div class="chunk-preview" style="border-left: 3px solid {color};">
+                                    <div style="font-size: 10px; color: #8b949e;">
+                                        Similarity: <span style="color: {color};">{sim:.0%}</span> |
+                                        Doc: {chunk.get('document_id', 'N/A')[:15]}...
+                                    </div>
+                                    <div style="margin-top: 4px;">{chunk.get('text', '')[:200]}...</div>
+                                </div>
+                                """, unsafe_allow_html=True)
+
+                    # Save to history
+                    st.session_state.messages.append({
+                        "role": "assistant",
+                        "content": response.answer,
+                        "citations": citations,
+                        "metadata": metadata,
+                    })
+
+# Dynamic suggested questions based on document content
+st.markdown("---")
+st.markdown("### 💡 Try asking")
+
+# Get indexed documents for question generation
+indexed_docs = get_indexed_documents()
+state_manager = get_state_manager()
+
+# Generate dynamic questions based on document content
+dynamic_questions = generate_dynamic_questions(state_manager, indexed_docs, max_questions=4)
+
+# Display as clickable buttons
+sample_cols = st.columns(len(dynamic_questions))
+for i, q in enumerate(dynamic_questions):
+    with sample_cols[i]:
+        # Truncate long questions for button display
+        display_q = q if len(q) <= 35 else q[:32] + "..."
+        if st.button(display_q, key=f"sample_{i}", use_container_width=True,
+                    disabled=(stats.get('total_chunks', 0) == 0),
+                    help=q if len(q) > 35 else None):
+            st.session_state.messages.append({"role": "user", "content": q})
+            st.rerun()
+
+# Show hint about dynamic questions
+if stats.get('total_chunks', 0) > 0:
+    st.caption("📌 Questions are generated based on your indexed documents")
+
+# Architecture info
+with st.expander("Multi-Agent RAG Architecture"):
+    st.markdown("""
+    ```
+    Query → [Query Planner] → [Retriever] → [Reranker] → [Synthesizer] → [Critic] → Answer
+                   ↓              ↓              ↓             ↓            ↓
+              Decompose      Dense+Sparse    Cross-Encoder  Grounded   Hallucination
+              & Expand       + RRF Fusion    Scoring        Citations  Detection
+    ```
+
+    **Agents:**
+    - **Query Planner**: Analyzes intent, decomposes complex queries, expands terms
+    - **Retriever**: Hybrid search combining dense (embedding) and sparse (BM25) retrieval
+    - **Reranker**: Cross-encoder scoring for precision, diversity via MMR
+    - **Synthesizer**: Generates grounded answers with proper citations
+    - **Critic**: Validates for hallucination, checks citation accuracy
+    """)

demo/pages/3_📊_Document_Comparison.py

@@ -0,0 +1,528 @@
+"""
+Document Comparison - SPARKNET
+
+Compare documents using semantic similarity, structure analysis,
+and content comparison with real embedding-based similarity.
+"""
+
+import streamlit as st
+import sys
+from pathlib import Path
+import pandas as pd
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+sys.path.insert(0, str(PROJECT_ROOT / "demo"))
+
+from state_manager import (
+    get_state_manager,
+    render_global_status_bar,
+)
+from rag_config import (
+    get_indexed_documents,
+    compute_document_similarity,
+    search_similar_chunks,
+    check_ollama,
+    get_unified_rag_system,
+)
+
+st.set_page_config(page_title="Document Comparison - SPARKNET", page_icon="📊", layout="wide")
+
+# Custom CSS
+st.markdown("""
+<style>
+.comparison-card {
+    background: #161b22;
+    border-radius: 10px;
+    padding: 15px;
+    margin: 10px 0;
+    border: 1px solid #30363d;
+}
+.doc-header {
+    font-size: 16px;
+    font-weight: bold;
+    color: #4ECDC4;
+    margin-bottom: 10px;
+}
+.similarity-badge {
+    display: inline-block;
+    padding: 8px 16px;
+    border-radius: 20px;
+    font-weight: bold;
+    font-size: 18px;
+}
+.sim-high {
+    background: linear-gradient(90deg, #4ECDC4 0%, #44a08d 100%);
+    color: white;
+}
+.sim-med {
+    background: linear-gradient(90deg, #ffc107 0%, #ff8800 100%);
+    color: black;
+}
+.sim-low {
+    background: linear-gradient(90deg, #dc3545 0%, #c82333 100%);
+    color: white;
+}
+.chunk-match {
+    background: #0d1117;
+    border-radius: 8px;
+    padding: 10px;
+    margin: 8px 0;
+    border-left: 4px solid;
+}
+.diff-added {
+    background: rgba(78, 205, 196, 0.1);
+    border-left-color: #4ECDC4;
+}
+.diff-removed {
+    background: rgba(220, 53, 69, 0.1);
+    border-left-color: #dc3545;
+}
+.diff-common {
+    background: rgba(139, 148, 158, 0.1);
+    border-left-color: #8b949e;
+}
+.metric-card {
+    background: #161b22;
+    border-radius: 8px;
+    padding: 15px;
+    text-align: center;
+}
+.metric-value {
+    font-size: 32px;
+    font-weight: bold;
+}
+.metric-label {
+    font-size: 11px;
+    color: #8b949e;
+    text-transform: uppercase;
+}
+</style>
+""", unsafe_allow_html=True)
+
+
+def get_similarity_class(sim: float) -> str:
+    """Get CSS class based on similarity."""
+    if sim >= 0.7:
+        return "sim-high"
+    elif sim >= 0.4:
+        return "sim-med"
+    return "sim-low"
+
+
+def get_similarity_color(sim: float) -> str:
+    """Get color based on similarity."""
+    if sim >= 0.7:
+        return "#4ECDC4"
+    elif sim >= 0.4:
+        return "#ffc107"
+    return "#dc3545"
+
+
+# Initialize state manager
+state_manager = get_state_manager()
+rag_system = get_unified_rag_system()
+
+# Header
+st.markdown("# 📊 Document Comparison")
+st.markdown("Compare documents using semantic similarity, structure analysis, and content comparison")
+
+# Global status bar
+render_global_status_bar()
+
+st.markdown("---")
+
+# Get documents
+all_docs = state_manager.get_all_documents()
+indexed_docs = get_indexed_documents()
+
+if not all_docs and not indexed_docs:
+    st.warning("No documents available for comparison")
+    st.markdown("""
+    ### Getting Started
+
+    To compare documents:
+    1. Go to **Live Processing** to upload and process documents
+    2. Process at least 2 documents
+    3. Come back here to compare them
+
+    Features:
+    - **Semantic Similarity**: Compare documents using embedding-based similarity
+    - **Structure Analysis**: Compare document structure (pages, chunks, regions)
+    - **Content Comparison**: Find similar passages between documents
+    """)
+
+    if st.button("🔬 Go to Live Processing", type="primary", use_container_width=True):
+        st.switch_page("pages/1_🔬_Live_Processing.py")
+
+else:
+    # Build document options
+    doc_options = {}
+    for doc in all_docs:
+        doc_options[f"{doc.filename} (State)"] = {"id": doc.doc_id, "source": "state", "doc": doc}
+    for doc in indexed_docs:
+        doc_id = doc.get("document_id", "unknown")
+        if doc_id not in [d["id"] for d in doc_options.values()]:
+            doc_options[f"{doc_id} (RAG)"] = {"id": doc_id, "source": "rag", "doc": doc}
+
+    if len(doc_options) < 2:
+        st.warning("Need at least 2 documents for comparison. Process more documents first.")
+    else:
+        # Document selection
+        st.markdown("### Select Documents to Compare")
+
+        col1, col2 = st.columns(2)
+        with col1:
+            doc1_name = st.selectbox("Document 1", list(doc_options.keys()), index=0)
+        with col2:
+            remaining = [k for k in doc_options.keys() if k != doc1_name]
+            doc2_name = st.selectbox("Document 2", remaining, index=0 if remaining else None)
+
+        doc1_info = doc_options.get(doc1_name)
+        doc2_info = doc_options.get(doc2_name)
+
+        # Comparison type
+        comparison_type = st.radio(
+            "Comparison Type",
+            ["Semantic Similarity", "Structure Analysis", "Content Comparison"],
+            horizontal=True,
+        )
+
+        if st.button("🔍 Compare Documents", type="primary", use_container_width=True):
+            st.markdown("---")
+
+            if comparison_type == "Semantic Similarity":
+                st.markdown("### Semantic Similarity Analysis")
+
+                with st.spinner("Computing document embeddings and similarity..."):
+                    # Use the compute_document_similarity function from rag_config
+                    if rag_system["status"] == "ready":
+                        result = compute_document_similarity(doc1_info["id"], doc2_info["id"])
+
+                        if result.get("error"):
+                            st.warning(f"Could not compute similarity: {result['error']}")
+                            # Use fallback based on text overlap
+                            if doc1_info["source"] == "state" and doc2_info["source"] == "state":
+                                doc1 = doc1_info["doc"]
+                                doc2 = doc2_info["doc"]
+                                # Simple word overlap
+                                words1 = set(doc1.raw_text.lower().split())
+                                words2 = set(doc2.raw_text.lower().split())
+                                overlap = len(words1 & words2) / max(len(words1 | words2), 1)
+                                similarity = overlap
+                            else:
+                                similarity = 0.5  # Default fallback
+                        else:
+                            similarity = result.get("similarity", 0)
+                    else:
+                        st.error("RAG system not ready for similarity computation")
+                        similarity = 0.5
+
+                # Display similarity score
+                sim_class = get_similarity_class(similarity)
+                sim_color = get_similarity_color(similarity)
+
+                st.markdown(f"""
+                <div style="text-align: center; padding: 30px;">
+                    <div class="similarity-badge {sim_class}">
+                        {similarity:.0%} Similarity
+                    </div>
+                    <p style="color: #8b949e; margin-top: 15px;">
+                        Based on embedding-based semantic similarity
+                    </p>
+                </div>
+                """, unsafe_allow_html=True)
+
+                # Similarity interpretation
+                if similarity >= 0.7:
+                    st.success("These documents are highly similar in content and meaning.")
+                elif similarity >= 0.4:
+                    st.warning("These documents have moderate similarity - some shared topics.")
+                else:
+                    st.info("These documents are quite different in content.")
+
+                # Document details
+                col1, col2 = st.columns(2)
+
+                with col1:
+                    st.markdown(f"#### 📄 {doc1_name.split(' (')[0]}")
+                    if doc1_info["source"] == "state":
+                        doc = doc1_info["doc"]
+                        st.metric("Pages", doc.page_count)
+                        st.metric("Chunks", len(doc.chunks))
+                        st.metric("Characters", f"{len(doc.raw_text):,}")
+                    else:
+                        doc = doc1_info["doc"]
+                        st.metric("Chunks", doc.get("chunk_count", "N/A"))
+
+                with col2:
+                    st.markdown(f"#### 📄 {doc2_name.split(' (')[0]}")
+                    if doc2_info["source"] == "state":
+                        doc = doc2_info["doc"]
+                        st.metric("Pages", doc.page_count)
+                        st.metric("Chunks", len(doc.chunks))
+                        st.metric("Characters", f"{len(doc.raw_text):,}")
+                    else:
+                        doc = doc2_info["doc"]
+                        st.metric("Chunks", doc.get("chunk_count", "N/A"))
+
+            elif comparison_type == "Structure Analysis":
+                st.markdown("### Document Structure Comparison")
+
+                col1, col2 = st.columns(2)
+
+                # Get structure data
+                def get_structure(info):
+                    if info["source"] == "state":
+                        doc = info["doc"]
+                        return {
+                            "Pages": doc.page_count,
+                            "Chunks": len(doc.chunks),
+                            "OCR Regions": len(doc.ocr_regions),
+                            "Layout Regions": len(doc.layout_data.get("regions", [])),
+                            "Characters": len(doc.raw_text),
+                            "Words": len(doc.raw_text.split()),
+                        }
+                    else:
+                        doc = info["doc"]
+                        return {
+                            "Chunks": doc.get("chunk_count", 0),
+                            "Source": doc.get("source_path", "N/A"),
+                        }
+
+                struct1 = get_structure(doc1_info)
+                struct2 = get_structure(doc2_info)
+
+                with col1:
+                    st.markdown(f"#### 📄 {doc1_name.split(' (')[0]}")
+                    for key, value in struct1.items():
+                        if isinstance(value, int) and value > 1000:
+                            st.metric(key, f"{value:,}")
+                        else:
+                            st.metric(key, value)
+
+                with col2:
+                    st.markdown(f"#### 📄 {doc2_name.split(' (')[0]}")
+                    for key, value in struct2.items():
+                        if isinstance(value, int) and value > 1000:
+                            st.metric(key, f"{value:,}")
+                        else:
+                            st.metric(key, value)
+
+                # Structure comparison chart
+                st.markdown("---")
+                st.markdown("### Comparison Chart")
+
+                common_keys = [k for k in struct1.keys() if k in struct2 and isinstance(struct1[k], (int, float))]
+                if common_keys:
+                    comparison_df = pd.DataFrame({
+                        "Metric": common_keys,
+                        doc1_name.split(' (')[0]: [struct1[k] for k in common_keys],
+                        doc2_name.split(' (')[0]: [struct2[k] for k in common_keys],
+                    })
+                    st.bar_chart(comparison_df.set_index("Metric"))
+
+                # Chunk type comparison (if available)
+                if doc1_info["source"] == "state" and doc2_info["source"] == "state":
+                    st.markdown("---")
+                    st.markdown("### Chunk Type Distribution")
+
+                    def get_chunk_types(doc):
+                        types = {}
+                        for chunk in doc.chunks:
+                            t = chunk.get("chunk_type", "unknown")
+                            types[t] = types.get(t, 0) + 1
+                        return types
+
+                    types1 = get_chunk_types(doc1_info["doc"])
+                    types2 = get_chunk_types(doc2_info["doc"])
+
+                    all_types = set(types1.keys()) | set(types2.keys())
+
+                    type_df = pd.DataFrame({
+                        "Type": list(all_types),
+                        doc1_name.split(' (')[0]: [types1.get(t, 0) for t in all_types],
+                        doc2_name.split(' (')[0]: [types2.get(t, 0) for t in all_types],
+                    })
+                    st.dataframe(type_df, width='stretch', hide_index=True)
+
+            else:  # Content Comparison
+                st.markdown("### Content Comparison")
+
+                if doc1_info["source"] == "state" and doc2_info["source"] == "state":
+                    doc1 = doc1_info["doc"]
+                    doc2 = doc2_info["doc"]
+
+                    # Word overlap analysis
+                    words1 = set(doc1.raw_text.lower().split())
+                    words2 = set(doc2.raw_text.lower().split())
+
+                    common_words = words1 & words2
+                    only_doc1 = words1 - words2
+                    only_doc2 = words2 - words1
+
+                    # Metrics
+                    metric_cols = st.columns(4)
+                    metric_cols[0].markdown(f"""
+                    <div class="metric-card">
+                        <div class="metric-value" style="color: #4ECDC4;">{len(common_words):,}</div>
+                        <div class="metric-label">Common Words</div>
+                    </div>
+                    """, unsafe_allow_html=True)
+                    metric_cols[1].markdown(f"""
+                    <div class="metric-card">
+                        <div class="metric-value" style="color: #FF6B6B;">{len(only_doc1):,}</div>
+                        <div class="metric-label">Only in Doc 1</div>
+                    </div>
+                    """, unsafe_allow_html=True)
+                    metric_cols[2].markdown(f"""
+                    <div class="metric-card">
+                        <div class="metric-value" style="color: #45B7D1;">{len(only_doc2):,}</div>
+                        <div class="metric-label">Only in Doc 2</div>
+                    </div>
+                    """, unsafe_allow_html=True)
+
+                    overlap_pct = len(common_words) / max(len(words1 | words2), 1)
+                    metric_cols[3].markdown(f"""
+                    <div class="metric-card">
+                        <div class="metric-value" style="color: #ffc107;">{overlap_pct:.0%}</div>
+                        <div class="metric-label">Word Overlap</div>
+                    </div>
+                    """, unsafe_allow_html=True)
+
+                    # Similar passages
+                    st.markdown("---")
+                    st.markdown("### Similar Passages")
+
+                    # Find similar chunks between documents
+                    with st.spinner("Finding similar passages..."):
+                        similar_passages = []
+
+                        # Compare first 10 chunks from doc1 against doc2
+                        for i, chunk1 in enumerate(doc1.chunks[:10]):
+                            text1 = chunk1.get("text", "")
+                            words_c1 = set(text1.lower().split())
+
+                            best_match = None
+                            best_score = 0
+
+                            for j, chunk2 in enumerate(doc2.chunks):
+                                text2 = chunk2.get("text", "")
+                                words_c2 = set(text2.lower().split())
+
+                                # Jaccard similarity
+                                if words_c1 and words_c2:
+                                    score = len(words_c1 & words_c2) / len(words_c1 | words_c2)
+                                    if score > best_score and score > 0.3:
+                                        best_score = score
+                                        best_match = {
+                                            "doc1_chunk": i,
+                                            "doc2_chunk": j,
+                                            "doc1_text": text1[:200],
+                                            "doc2_text": text2[:200],
+                                            "similarity": score,
+                                        }
+
+                            if best_match:
+                                similar_passages.append(best_match)
+
+                    if similar_passages:
+                        # Sort by similarity
+                        similar_passages.sort(key=lambda x: x["similarity"], reverse=True)
+
+                        for i, match in enumerate(similar_passages[:5]):
+                            sim_color = get_similarity_color(match["similarity"])
+                            with st.expander(f"Match {i+1} - Similarity: {match['similarity']:.0%}"):
+                                col1, col2 = st.columns(2)
+                                with col1:
+                                    st.markdown(f"**{doc1_name.split(' (')[0]}** (Chunk {match['doc1_chunk']+1})")
+                                    st.markdown(f"""
+                                    <div class="chunk-match diff-common">
+                                        {match['doc1_text']}...
+                                    </div>
+                                    """, unsafe_allow_html=True)
+                                with col2:
+                                    st.markdown(f"**{doc2_name.split(' (')[0]}** (Chunk {match['doc2_chunk']+1})")
+                                    st.markdown(f"""
+                                    <div class="chunk-match diff-common">
+                                        {match['doc2_text']}...
+                                    </div>
+                                    """, unsafe_allow_html=True)
+                    else:
+                        st.info("No significantly similar passages found between documents")
+
+                    # Key terms comparison
+                    st.markdown("---")
+                    st.markdown("### Key Terms Comparison")
+
+                    # Get most frequent words (simple approach)
+                    from collections import Counter
+
+                    def get_top_words(text, n=20):
+                        words = text.lower().split()
+                        # Filter out common words
+                        stopwords = {"the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
+                                    "have", "has", "had", "do", "does", "did", "will", "would", "could",
+                                    "should", "may", "might", "must", "and", "or", "but", "if", "then",
+                                    "so", "to", "of", "in", "for", "on", "with", "at", "by", "from",
+                                    "this", "that", "these", "those", "it", "its"}
+                        words = [w for w in words if len(w) > 3 and w not in stopwords]
+                        return Counter(words).most_common(n)
+
+                    top1 = get_top_words(doc1.raw_text)
+                    top2 = get_top_words(doc2.raw_text)
+
+                    col1, col2 = st.columns(2)
+                    with col1:
+                        st.markdown(f"**Top terms in {doc1_name.split(' (')[0]}:**")
+                        for word, count in top1[:10]:
+                            in_doc2 = word in [w for w, c in top2]
+                            color = "#4ECDC4" if in_doc2 else "#8b949e"
+                            st.markdown(f"<span style='color: {color};'>• {word}</span> ({count})", unsafe_allow_html=True)
+
+                    with col2:
+                        st.markdown(f"**Top terms in {doc2_name.split(' (')[0]}:**")
+                        for word, count in top2[:10]:
+                            in_doc1 = word in [w for w, c in top1]
+                            color = "#4ECDC4" if in_doc1 else "#8b949e"
+                            st.markdown(f"<span style='color: {color};'>• {word}</span> ({count})", unsafe_allow_html=True)
+
+                else:
+                    st.info("Content comparison requires both documents to be in processed state")
+
+        # Export options
+        st.markdown("---")
+        st.markdown("### Export Comparison")
+
+        export_cols = st.columns(3)
+        with export_cols[0]:
+            if st.button("📄 Export as JSON", use_container_width=True):
+                import json
+                export_data = {
+                    "document1": doc1_name,
+                    "document2": doc2_name,
+                    "comparison_type": comparison_type,
+                }
+                st.json(export_data)
+        with export_cols[1]:
+            st.button("📊 Export as CSV", disabled=True, use_container_width=True)
+        with export_cols[2]:
+            st.button("📋 Export as PDF", disabled=True, use_container_width=True)
+
+# Navigation
+st.markdown("---")
+st.markdown("### Navigation")
+nav_cols = st.columns(4)
+
+with nav_cols[0]:
+    if st.button("🔬 Live Processing", use_container_width=True):
+        st.switch_page("pages/1_🔬_Live_Processing.py")
+with nav_cols[1]:
+    if st.button("💬 Interactive RAG", use_container_width=True):
+        st.switch_page("pages/2_💬_Interactive_RAG.py")
+with nav_cols[2]:
+    if st.button("🎯 Evidence Viewer", use_container_width=True):
+        st.switch_page("pages/4_🎯_Evidence_Viewer.py")
+with nav_cols[3]:
+    if st.button("📄 Document Viewer", use_container_width=True):
+        st.switch_page("pages/5_📄_Document_Viewer.py")

demo/pages/4_🎯_Evidence_Viewer.py

@@ -0,0 +1,529 @@
+"""
+Evidence Viewer - SPARKNET
+
+Visualize extracted OCR regions, layout, and evidence grounding with
+confidence-based coloring and interactivity.
+"""
+
+import streamlit as st
+import sys
+from pathlib import Path
+import base64
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+sys.path.insert(0, str(PROJECT_ROOT / "demo"))
+
+from state_manager import (
+    get_state_manager,
+    render_global_status_bar,
+)
+from rag_config import (
+    get_indexed_documents,
+    get_chunks_for_document,
+    check_ollama,
+)
+
+st.set_page_config(page_title="Evidence Viewer - SPARKNET", page_icon="🎯", layout="wide")
+
+# Custom CSS with confidence-based colors
+st.markdown("""
+<style>
+.region-card {
+    background: #161b22;
+    border-radius: 10px;
+    padding: 12px;
+    margin: 8px 0;
+    border-left: 4px solid;
+    transition: transform 0.2s;
+}
+.region-card:hover {
+    transform: translateX(4px);
+}
+.confidence-high {
+    border-left-color: #4ECDC4 !important;
+    background: linear-gradient(90deg, rgba(78,205,196,0.1) 0%, transparent 100%);
+}
+.confidence-med {
+    border-left-color: #ffc107 !important;
+    background: linear-gradient(90deg, rgba(255,193,7,0.1) 0%, transparent 100%);
+}
+.confidence-low {
+    border-left-color: #dc3545 !important;
+    background: linear-gradient(90deg, rgba(220,53,69,0.1) 0%, transparent 100%);
+}
+.region-header {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    margin-bottom: 8px;
+}
+.region-type {
+    font-weight: bold;
+    text-transform: uppercase;
+    font-size: 12px;
+}
+.region-conf {
+    font-size: 14px;
+    font-weight: bold;
+}
+.region-text {
+    font-family: 'Monaco', 'Menlo', monospace;
+    font-size: 13px;
+    color: #c9d1d9;
+    line-height: 1.5;
+}
+.region-meta {
+    font-size: 10px;
+    color: #8b949e;
+    margin-top: 8px;
+}
+.bbox-display {
+    background: #0d1117;
+    padding: 4px 8px;
+    border-radius: 4px;
+    font-family: monospace;
+    font-size: 11px;
+}
+.page-thumbnail {
+    border: 2px solid #30363d;
+    border-radius: 8px;
+    padding: 10px;
+    background: #0d1117;
+}
+.page-thumbnail.active {
+    border-color: #4ECDC4;
+}
+.stats-card {
+    background: #161b22;
+    border-radius: 8px;
+    padding: 15px;
+    text-align: center;
+}
+.stats-value {
+    font-size: 28px;
+    font-weight: bold;
+    color: #4ECDC4;
+}
+.stats-label {
+    font-size: 11px;
+    color: #8b949e;
+    text-transform: uppercase;
+}
+.copy-btn {
+    background: #21262d;
+    border: none;
+    padding: 4px 8px;
+    border-radius: 4px;
+    font-size: 11px;
+    cursor: pointer;
+}
+</style>
+""", unsafe_allow_html=True)
+
+
+def get_confidence_class(conf: float) -> str:
+    """Get CSS class based on confidence."""
+    if conf >= 0.8:
+        return "confidence-high"
+    elif conf >= 0.6:
+        return "confidence-med"
+    return "confidence-low"
+
+
+def get_confidence_color(conf: float) -> str:
+    """Get color based on confidence."""
+    if conf >= 0.8:
+        return "#4ECDC4"
+    elif conf >= 0.6:
+        return "#ffc107"
+    return "#dc3545"
+
+
+def get_type_color(region_type: str) -> str:
+    """Get color for region type."""
+    colors = {
+        "title": "#FF6B6B",
+        "heading": "#FF8E6B",
+        "paragraph": "#4ECDC4",
+        "text": "#45B7D1",
+        "list": "#96CEB4",
+        "table": "#FFEAA7",
+        "figure": "#DDA0DD",
+        "header": "#98D8C8",
+        "footer": "#8b949e",
+    }
+    return colors.get(region_type.lower(), "#666")
+
+
+# Initialize state manager
+state_manager = get_state_manager()
+
+# Header
+st.markdown("# 🎯 Evidence Viewer")
+st.markdown("Visualize OCR regions, layout structure, and evidence grounding with confidence scoring")
+
+# Global status bar
+render_global_status_bar()
+
+st.markdown("---")
+
+# Get documents from state
+all_docs = state_manager.get_all_documents()
+indexed_docs = get_indexed_documents()
+
+# Sidebar for document selection
+with st.sidebar:
+    st.markdown("## 📚 Select Document")
+
+    if all_docs:
+        doc_options = {f"{d.filename} ({len(d.ocr_regions)} regions)": d.doc_id for d in all_docs}
+        selected_doc_name = st.selectbox("Processed Documents", list(doc_options.keys()))
+        selected_doc_id = doc_options.get(selected_doc_name)
+
+        if selected_doc_id:
+            state_manager.set_active_document(selected_doc_id)
+    else:
+        st.info("No documents processed yet")
+        selected_doc_id = None
+
+    st.markdown("---")
+    st.markdown("## 🎨 Display Options")
+
+    show_ocr = st.checkbox("Show OCR Regions", value=True)
+    show_layout = st.checkbox("Show Layout Regions", value=True)
+    show_bbox = st.checkbox("Show Bounding Boxes", value=True)
+
+    st.markdown("---")
+    st.markdown("## 🎚️ Filters")
+
+    min_confidence = st.slider("Min Confidence", 0.0, 1.0, 0.0, 0.1)
+
+    region_types = ["All", "title", "heading", "paragraph", "text", "list", "table", "figure"]
+    selected_type = st.selectbox("Region Type", region_types)
+
+# Main content
+active_doc = state_manager.get_active_document()
+
+if active_doc:
+    # Document header
+    col1, col2 = st.columns([3, 1])
+    with col1:
+        st.markdown(f"## 📄 {active_doc.filename}")
+        st.caption(f"ID: `{active_doc.doc_id}` | {active_doc.page_count} pages")
+    with col2:
+        if active_doc.indexed:
+            st.success("Indexed")
+        else:
+            st.warning("Not indexed")
+
+    # Statistics cards
+    stat_cols = st.columns(5)
+
+    # Calculate stats
+    ocr_regions = active_doc.ocr_regions
+    layout_regions = active_doc.layout_data.get("regions", [])
+
+    avg_ocr_conf = sum(r.get("confidence", 0) for r in ocr_regions) / len(ocr_regions) if ocr_regions else 0
+    high_conf_count = len([r for r in ocr_regions if r.get("confidence", 0) >= 0.8])
+    med_conf_count = len([r for r in ocr_regions if 0.6 <= r.get("confidence", 0) < 0.8])
+    low_conf_count = len([r for r in ocr_regions if r.get("confidence", 0) < 0.6])
+
+    stat_cols[0].markdown(f"""
+    <div class="stats-card">
+        <div class="stats-value">{len(ocr_regions)}</div>
+        <div class="stats-label">OCR Regions</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    stat_cols[1].markdown(f"""
+    <div class="stats-card">
+        <div class="stats-value">{len(layout_regions)}</div>
+        <div class="stats-label">Layout Regions</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    stat_cols[2].markdown(f"""
+    <div class="stats-card">
+        <div class="stats-value" style="color: #4ECDC4;">{avg_ocr_conf:.0%}</div>
+        <div class="stats-label">Avg Confidence</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    stat_cols[3].markdown(f"""
+    <div class="stats-card">
+        <div class="stats-value" style="color: #4ECDC4;">{high_conf_count}</div>
+        <div class="stats-label">High Conf (&gt;80%)</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    stat_cols[4].markdown(f"""
+    <div class="stats-card">
+        <div class="stats-value" style="color: #dc3545;">{low_conf_count}</div>
+        <div class="stats-label">Low Conf (&lt;60%)</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    st.markdown("---")
+
+    # Main view - Page images and regions
+    tab_regions, tab_pages, tab_export = st.tabs(["📋 Regions", "📄 Page View", "📥 Export"])
+
+    with tab_regions:
+        # Filter regions
+        filtered_ocr = ocr_regions
+        if min_confidence > 0:
+            filtered_ocr = [r for r in filtered_ocr if r.get("confidence", 0) >= min_confidence]
+
+        # Page selector
+        pages = sorted(set(r.get("page", 0) for r in filtered_ocr))
+        if pages:
+            selected_page = st.selectbox(
+                "Select Page",
+                pages,
+                format_func=lambda x: f"Page {x + 1} ({len([r for r in filtered_ocr if r.get('page') == x])} regions)"
+            )
+
+            page_regions = [r for r in filtered_ocr if r.get("page") == selected_page]
+
+            st.markdown(f"### OCR Regions on Page {selected_page + 1}")
+            st.caption(f"Showing {len(page_regions)} regions (filtered by confidence >= {min_confidence:.0%})")
+
+            # Display regions with confidence coloring
+            for i, region in enumerate(page_regions):
+                conf = region.get("confidence", 0)
+                conf_class = get_confidence_class(conf)
+                conf_color = get_confidence_color(conf)
+                text = region.get("text", "")
+                bbox = region.get("bbox")
+
+                st.markdown(f"""
+                <div class="region-card {conf_class}">
+                    <div class="region-header">
+                        <span class="region-type" style="color: {conf_color};">Region {i + 1}</span>
+                        <span class="region-conf" style="color: {conf_color};">{conf:.0%}</span>
+                    </div>
+                    <div class="region-text">{text}</div>
+                    <div class="region-meta">
+                        {f'<span class="bbox-display">Bbox: ({bbox[0]:.0f}, {bbox[1]:.0f}) - ({bbox[2]:.0f}, {bbox[3]:.0f})</span>' if bbox and show_bbox else ''}
+                    </div>
+                </div>
+                """, unsafe_allow_html=True)
+
+                # Copy button
+                col1, col2 = st.columns([4, 1])
+                with col2:
+                    if st.button("📋 Copy", key=f"copy_{i}"):
+                        st.toast(f"Copied region {i+1} text!")
+
+        else:
+            st.info("No OCR regions available for this document")
+
+        # Layout regions
+        if show_layout and layout_regions:
+            st.markdown("---")
+            st.markdown("### Layout Regions")
+
+            # Group by type
+            by_type = {}
+            for r in layout_regions:
+                rtype = r.get("type", "unknown")
+                if rtype not in by_type:
+                    by_type[rtype] = []
+                by_type[rtype].append(r)
+
+            # Type pills
+            st.markdown("**Detected types:**")
+            type_html = ""
+            for rtype, regions in by_type.items():
+                color = get_type_color(rtype)
+                type_html += f'<span style="background: {color}33; color: {color}; padding: 4px 10px; border-radius: 12px; margin: 4px; display: inline-block; font-size: 12px;">{rtype.title()} ({len(regions)})</span>'
+            st.markdown(type_html, unsafe_allow_html=True)
+
+            # Layout details
+            for rtype, regions in by_type.items():
+                with st.expander(f"{rtype.title()} ({len(regions)} regions)"):
+                    for r in regions[:10]:
+                        conf = r.get("confidence", 0)
+                        conf_color = get_confidence_color(conf)
+                        st.markdown(f"""
+                        <div style="background: #0d1117; padding: 8px; border-radius: 6px; margin: 4px 0; border-left: 3px solid {get_type_color(rtype)};">
+                            <span style="color: {conf_color};">{conf:.0%}</span> | Page {r.get('page', 0) + 1}
+                        </div>
+                        """, unsafe_allow_html=True)
+
+    with tab_pages:
+        st.markdown("### Page Images with Regions")
+
+        if active_doc.page_images:
+            page_select = st.selectbox(
+                "Page",
+                range(len(active_doc.page_images)),
+                format_func=lambda x: f"Page {x + 1}",
+                key="page_view_select"
+            )
+
+            if page_select is not None:
+                # Display page image
+                img_data = active_doc.page_images[page_select]
+                st.image(
+                    f"data:image/png;base64,{img_data}",
+                    caption=f"Page {page_select + 1}",
+                    use_container_width=True
+                )
+
+                # Regions on this page
+                page_ocr = [r for r in ocr_regions if r.get("page") == page_select]
+                page_layout = [r for r in layout_regions if r.get("page") == page_select]
+
+                col1, col2 = st.columns(2)
+                with col1:
+                    st.metric("OCR Regions", len(page_ocr))
+                with col2:
+                    st.metric("Layout Regions", len(page_layout))
+
+                st.info("Bounding box overlay visualization will be available in future updates")
+        else:
+            st.info("No page images available. Process a PDF document to see page images.")
+
+    with tab_export:
+        st.markdown("### Export Evidence Data")
+
+        export_cols = st.columns(3)
+
+        with export_cols[0]:
+            st.markdown("**OCR Regions JSON**")
+            if st.button("📥 Export OCR", use_container_width=True):
+                import json
+                ocr_json = json.dumps({
+                    "document_id": active_doc.doc_id,
+                    "filename": active_doc.filename,
+                    "ocr_regions": ocr_regions,
+                }, indent=2)
+                st.download_button(
+                    "Download JSON",
+                    ocr_json,
+                    file_name=f"{active_doc.doc_id}_ocr.json",
+                    mime="application/json"
+                )
+
+        with export_cols[1]:
+            st.markdown("**Layout Regions JSON**")
+            if st.button("📥 Export Layout", use_container_width=True):
+                import json
+                layout_json = json.dumps({
+                    "document_id": active_doc.doc_id,
+                    "filename": active_doc.filename,
+                    "layout_regions": layout_regions,
+                }, indent=2)
+                st.download_button(
+                    "Download JSON",
+                    layout_json,
+                    file_name=f"{active_doc.doc_id}_layout.json",
+                    mime="application/json"
+                )
+
+        with export_cols[2]:
+            st.markdown("**Full Text**")
+            st.download_button(
+                "📥 Export Text",
+                active_doc.raw_text,
+                file_name=f"{active_doc.doc_id}.txt",
+                mime="text/plain",
+                use_container_width=True
+            )
+
+        # Confidence distribution chart
+        st.markdown("---")
+        st.markdown("### Confidence Distribution")
+
+        if ocr_regions:
+            import pandas as pd
+
+            # Build distribution data
+            conf_bins = {"High (>80%)": 0, "Medium (60-80%)": 0, "Low (<60%)": 0}
+            for r in ocr_regions:
+                c = r.get("confidence", 0)
+                if c >= 0.8:
+                    conf_bins["High (>80%)"] += 1
+                elif c >= 0.6:
+                    conf_bins["Medium (60-80%)"] += 1
+                else:
+                    conf_bins["Low (<60%)"] += 1
+
+            df = pd.DataFrame({
+                "Confidence Level": list(conf_bins.keys()),
+                "Count": list(conf_bins.values())
+            })
+            st.bar_chart(df.set_index("Confidence Level"))
+
+    # Navigation
+    st.markdown("---")
+    st.markdown("### Actions")
+    nav_cols = st.columns(4)
+
+    with nav_cols[0]:
+        if st.button("💬 Query RAG", use_container_width=True):
+            st.switch_page("pages/2_💬_Interactive_RAG.py")
+    with nav_cols[1]:
+        if st.button("📄 Document Viewer", use_container_width=True):
+            st.switch_page("pages/5_📄_Document_Viewer.py")
+    with nav_cols[2]:
+        if st.button("📊 Compare", use_container_width=True):
+            st.switch_page("pages/3_📊_Document_Comparison.py")
+    with nav_cols[3]:
+        if st.button("🔬 Process New", use_container_width=True):
+            st.switch_page("pages/1_🔬_Live_Processing.py")
+
+else:
+    # No document selected
+    st.markdown("## No Document Selected")
+
+    st.markdown("""
+    ### Getting Started
+
+    1. Go to **Live Processing** to upload and process a document
+    2. Come back here to view OCR regions and evidence grounding
+    3. Use confidence filters to focus on high or low quality regions
+
+    Evidence viewer shows:
+    - OCR extracted text regions with confidence scores
+    - Layout detection results (titles, paragraphs, tables, etc.)
+    - Bounding box coordinates for each region
+    - Page images with region overlays
+    """)
+
+    col1, col2 = st.columns(2)
+    with col1:
+        if st.button("🔬 Go to Live Processing", type="primary", use_container_width=True):
+            st.switch_page("pages/1_🔬_Live_Processing.py")
+    with col2:
+        if st.button("📄 Go to Document Viewer", use_container_width=True):
+            st.switch_page("pages/5_📄_Document_Viewer.py")
+
+    # Legend
+    st.markdown("---")
+    st.markdown("### Confidence Color Legend")
+
+    legend_cols = st.columns(3)
+    with legend_cols[0]:
+        st.markdown("""
+        <div style="background: rgba(78,205,196,0.2); padding: 10px; border-radius: 8px; border-left: 4px solid #4ECDC4;">
+            <strong style="color: #4ECDC4;">High Confidence (>80%)</strong><br>
+            <span style="color: #8b949e;">Reliable extraction</span>
+        </div>
+        """, unsafe_allow_html=True)
+    with legend_cols[1]:
+        st.markdown("""
+        <div style="background: rgba(255,193,7,0.2); padding: 10px; border-radius: 8px; border-left: 4px solid #ffc107;">
+            <strong style="color: #ffc107;">Medium Confidence (60-80%)</strong><br>
+            <span style="color: #8b949e;">Review recommended</span>
+        </div>
+        """, unsafe_allow_html=True)
+    with legend_cols[2]:
+        st.markdown("""
+        <div style="background: rgba(220,53,69,0.2); padding: 10px; border-radius: 8px; border-left: 4px solid #dc3545;">
+            <strong style="color: #dc3545;">Low Confidence (<60%)</strong><br>
+            <span style="color: #8b949e;">Manual verification needed</span>
+        </div>
+        """, unsafe_allow_html=True)

demo/pages/5_📄_Document_Viewer.py

@@ -0,0 +1,565 @@
+"""
+Document Viewer - SPARKNET
+
+View and explore processed documents from the state manager.
+Provides visual chunk segmentation, OCR regions, and layout visualization.
+"""
+
+import streamlit as st
+import sys
+from pathlib import Path
+import time
+import hashlib
+import base64
+from typing import List, Dict, Any
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+sys.path.insert(0, str(PROJECT_ROOT / "demo"))
+
+# Import state manager and RAG config
+from state_manager import (
+    get_state_manager,
+    ProcessedDocument,
+    render_global_status_bar,
+)
+from rag_config import (
+    get_unified_rag_system,
+    get_store_stats,
+    get_indexed_documents,
+    get_chunks_for_document,
+    check_ollama,
+)
+
+st.set_page_config(
+    page_title="Document Viewer - SPARKNET",
+    page_icon="📄",
+    layout="wide"
+)
+
+# Custom CSS
+st.markdown("""
+<style>
+.chunk-card {
+    background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);
+    border-radius: 10px;
+    padding: 12px;
+    margin: 8px 0;
+    border-left: 4px solid #4ECDC4;
+}
+.chunk-header {
+    display: flex;
+    justify-content: space-between;
+    font-size: 11px;
+    color: #8b949e;
+    margin-bottom: 6px;
+}
+.chunk-text {
+    font-size: 13px;
+    line-height: 1.5;
+    color: #c9d1d9;
+    font-family: 'Monaco', 'Menlo', monospace;
+}
+.ocr-region {
+    background: #161b22;
+    border-radius: 6px;
+    padding: 8px;
+    margin: 4px 0;
+    border-left: 3px solid;
+}
+.layout-region {
+    display: inline-block;
+    padding: 4px 8px;
+    margin: 3px;
+    border-radius: 4px;
+    font-size: 11px;
+}
+.doc-card {
+    background: #0d1117;
+    border-radius: 10px;
+    padding: 15px;
+    margin: 10px 0;
+    border: 1px solid #30363d;
+    cursor: pointer;
+    transition: border-color 0.2s;
+}
+.doc-card:hover {
+    border-color: #4ECDC4;
+}
+.doc-card.active {
+    border-color: #4ECDC4;
+    border-width: 2px;
+}
+.metric-mini {
+    background: #161b22;
+    border-radius: 6px;
+    padding: 8px;
+    text-align: center;
+    margin: 4px;
+}
+.metric-mini .value {
+    font-size: 18px;
+    font-weight: bold;
+    color: #4ECDC4;
+}
+.metric-mini .label {
+    font-size: 10px;
+    color: #8b949e;
+    text-transform: uppercase;
+}
+.page-viewer {
+    background: #0d1117;
+    border-radius: 10px;
+    padding: 20px;
+    max-height: 600px;
+    overflow-y: auto;
+}
+.confidence-high { color: #4ECDC4; }
+.confidence-med { color: #ffc107; }
+.confidence-low { color: #dc3545; }
+</style>
+""", unsafe_allow_html=True)
+
+
+def get_chunk_color(index: int) -> str:
+    """Get distinct color for chunk visualization."""
+    colors = [
+        "#FF6B6B", "#4ECDC4", "#45B7D1", "#96CEB4",
+        "#FFEAA7", "#DDA0DD", "#98D8C8", "#F7DC6F",
+        "#BB8FCE", "#85C1E9", "#F8B500", "#00CED1"
+    ]
+    return colors[index % len(colors)]
+
+
+def get_confidence_class(conf: float) -> str:
+    """Get confidence CSS class."""
+    if conf >= 0.8:
+        return "confidence-high"
+    elif conf >= 0.6:
+        return "confidence-med"
+    return "confidence-low"
+
+
+def get_layout_color(layout_type: str) -> str:
+    """Get color for layout type."""
+    colors = {
+        "title": "#FF6B6B",
+        "heading": "#FF8E6B",
+        "paragraph": "#4ECDC4",
+        "text": "#45B7D1",
+        "list": "#96CEB4",
+        "table": "#FFEAA7",
+        "figure": "#DDA0DD",
+        "header": "#98D8C8",
+        "footer": "#8b949e",
+    }
+    return colors.get(layout_type.lower(), "#666")
+
+
+# Initialize state manager
+state_manager = get_state_manager()
+
+# Header
+st.markdown("# 📄 Document Viewer")
+st.markdown("Explore processed documents, chunks, OCR regions, and layout structure")
+
+# Global status bar
+render_global_status_bar()
+
+st.markdown("---")
+
+# Get all documents from state and RAG
+all_state_docs = state_manager.get_all_documents()
+rag_docs = get_indexed_documents()
+
+# Sidebar for document selection
+with st.sidebar:
+    st.markdown("## 📚 Documents")
+
+    # Processed documents from state manager
+    if all_state_docs:
+        st.markdown("### Recently Processed")
+        selected_doc_id = None
+
+        for doc in reversed(all_state_docs[-10:]):
+            is_active = state_manager.state.get("active_doc_id") == doc.doc_id
+            card_class = "doc-card active" if is_active else "doc-card"
+
+            if st.button(
+                f"📄 {doc.filename[:25]}...",
+                key=f"doc_{doc.doc_id}",
+                use_container_width=True,
+                type="primary" if is_active else "secondary"
+            ):
+                state_manager.set_active_document(doc.doc_id)
+                st.rerun()
+
+            # Mini stats
+            cols = st.columns(3)
+            cols[0].caption(f"📄 {doc.page_count}p")
+            cols[1].caption(f"📦 {len(doc.chunks)}")
+            if doc.indexed:
+                cols[2].caption("✓ Indexed")
+            st.markdown("---")
+    else:
+        st.info("No documents processed yet")
+        st.markdown("Go to **Live Processing** to process documents")
+
+    # RAG indexed documents
+    if rag_docs:
+        st.markdown("### 📊 RAG Index")
+        st.caption(f"{len(rag_docs)} documents indexed")
+        for doc in rag_docs[:5]:
+            st.caption(f"• {doc.get('document_id', 'unknown')[:20]}...")
+
+# Main content
+active_doc = state_manager.get_active_document()
+
+if active_doc:
+    # Document header
+    col1, col2 = st.columns([3, 1])
+
+    with col1:
+        st.markdown(f"## 📄 {active_doc.filename}")
+        st.caption(f"ID: `{active_doc.doc_id}` | Type: {active_doc.file_type} | Processed: {active_doc.created_at.strftime('%Y-%m-%d %H:%M')}")
+
+    with col2:
+        if active_doc.indexed:
+            st.success(f"✓ Indexed ({active_doc.indexed_chunks} chunks)")
+        else:
+            st.warning("Not indexed")
+
+    # Summary metrics
+    metric_cols = st.columns(6)
+    metric_cols[0].markdown(f"""
+    <div class="metric-mini">
+        <div class="value">{active_doc.page_count}</div>
+        <div class="label">Pages</div>
+    </div>
+    """, unsafe_allow_html=True)
+    metric_cols[1].markdown(f"""
+    <div class="metric-mini">
+        <div class="value">{len(active_doc.chunks)}</div>
+        <div class="label">Chunks</div>
+    </div>
+    """, unsafe_allow_html=True)
+    metric_cols[2].markdown(f"""
+    <div class="metric-mini">
+        <div class="value">{len(active_doc.ocr_regions)}</div>
+        <div class="label">OCR Regions</div>
+    </div>
+    """, unsafe_allow_html=True)
+    layout_count = len(active_doc.layout_data.get("regions", []))
+    metric_cols[3].markdown(f"""
+    <div class="metric-mini">
+        <div class="value">{layout_count}</div>
+        <div class="label">Layout Regions</div>
+    </div>
+    """, unsafe_allow_html=True)
+    metric_cols[4].markdown(f"""
+    <div class="metric-mini">
+        <div class="value">{len(active_doc.raw_text):,}</div>
+        <div class="label">Characters</div>
+    </div>
+    """, unsafe_allow_html=True)
+    metric_cols[5].markdown(f"""
+    <div class="metric-mini">
+        <div class="value">{active_doc.processing_time:.1f}s</div>
+        <div class="label">Process Time</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    st.markdown("---")
+
+    # Tabs for different views
+    tab_chunks, tab_text, tab_ocr, tab_layout, tab_pages = st.tabs([
+        "📦 Chunks",
+        "📝 Full Text",
+        "🔍 OCR Regions",
+        "🗺️ Layout",
+        "📄 Page Images"
+    ])
+
+    with tab_chunks:
+        st.markdown("### Document Chunks")
+
+        # Filter options
+        filter_cols = st.columns([2, 1, 1])
+        with filter_cols[0]:
+            search_term = st.text_input("Search in chunks", placeholder="Enter search term...")
+        with filter_cols[1]:
+            chunk_types = list(set(c.get("chunk_type", "text") for c in active_doc.chunks))
+            selected_type = st.selectbox("Filter by type", ["All"] + chunk_types)
+        with filter_cols[2]:
+            page_filter = st.selectbox("Filter by page", ["All"] + list(range(1, active_doc.page_count + 1)))
+
+        # Filter chunks
+        filtered_chunks = active_doc.chunks
+        if search_term:
+            filtered_chunks = [c for c in filtered_chunks if search_term.lower() in c.get("text", "").lower()]
+        if selected_type != "All":
+            filtered_chunks = [c for c in filtered_chunks if c.get("chunk_type") == selected_type]
+        if page_filter != "All":
+            filtered_chunks = [c for c in filtered_chunks if c.get("page", 0) + 1 == page_filter]
+
+        st.caption(f"Showing {len(filtered_chunks)} of {len(active_doc.chunks)} chunks")
+
+        # Display chunks
+        for i, chunk in enumerate(filtered_chunks[:30]):
+            chunk_type = chunk.get("chunk_type", "text")
+            conf = chunk.get("confidence", 0)
+            color = get_chunk_color(i)
+            conf_class = get_confidence_class(conf)
+
+            with st.expander(f"[{i+1}] {chunk_type.upper()} - {chunk.get('text', '')[:60]}...", expanded=(i == 0)):
+                st.markdown(f"""
+                <div class="chunk-card" style="border-left-color: {color};">
+                    <div class="chunk-header">
+                        <span>ID: <code>{chunk.get('chunk_id', 'N/A')}</code></span>
+                        <span>Page {chunk.get('page', 0) + 1}</span>
+                        <span class="{conf_class}">Confidence: {conf:.0%}</span>
+                    </div>
+                    <div class="chunk-text">{chunk.get('text', '')}</div>
+                </div>
+                """, unsafe_allow_html=True)
+
+                # Bounding box info
+                bbox = chunk.get("bbox")
+                if bbox:
+                    st.caption(f"Bbox: ({bbox[0]:.0f}, {bbox[1]:.0f}) - ({bbox[2]:.0f}, {bbox[3]:.0f})")
+
+        if len(filtered_chunks) > 30:
+            st.info(f"Showing 30 of {len(filtered_chunks)} matching chunks")
+
+    with tab_text:
+        st.markdown("### Extracted Text")
+
+        # Text display options
+        text_cols = st.columns([1, 1, 1])
+        with text_cols[0]:
+            show_page_markers = st.checkbox("Show page markers", value=True)
+        with text_cols[1]:
+            font_size = st.slider("Font size", 10, 18, 13)
+        with text_cols[2]:
+            max_chars = st.slider("Max characters", 5000, 50000, 20000, 1000)
+
+        text_to_display = active_doc.raw_text[:max_chars]
+        if len(active_doc.raw_text) > max_chars:
+            text_to_display += f"\n\n... [Truncated - {len(active_doc.raw_text) - max_chars:,} more characters]"
+
+        st.markdown(f"""
+        <div class="page-viewer" style="font-size: {font_size}px;">
+            <pre style="white-space: pre-wrap; font-family: monospace; margin: 0;">{text_to_display}</pre>
+        </div>
+        """, unsafe_allow_html=True)
+
+        # Download button
+        st.download_button(
+            "📥 Download Full Text",
+            active_doc.raw_text,
+            file_name=f"{active_doc.filename}.txt",
+            mime="text/plain"
+        )
+
+    with tab_ocr:
+        st.markdown("### OCR Regions")
+
+        if active_doc.ocr_regions:
+            # Group by page
+            by_page = {}
+            for region in active_doc.ocr_regions:
+                page = region.get("page", 0)
+                if page not in by_page:
+                    by_page[page] = []
+                by_page[page].append(region)
+
+            # Page selector
+            page_select = st.selectbox(
+                "Select page",
+                sorted(by_page.keys()),
+                format_func=lambda x: f"Page {x + 1} ({len(by_page.get(x, []))} regions)"
+            )
+
+            if page_select is not None and page_select in by_page:
+                page_regions = by_page[page_select]
+
+                # Summary
+                avg_conf = sum(r.get("confidence", 0) for r in page_regions) / len(page_regions) if page_regions else 0
+                conf_class = get_confidence_class(avg_conf)
+
+                st.markdown(f"**{len(page_regions)} regions** | Average confidence: <span class='{conf_class}'>{avg_conf:.0%}</span>", unsafe_allow_html=True)
+
+                # Filter by confidence
+                min_conf = st.slider("Minimum confidence", 0.0, 1.0, 0.5, 0.1)
+                filtered_regions = [r for r in page_regions if r.get("confidence", 0) >= min_conf]
+
+                for i, region in enumerate(filtered_regions[:50]):
+                    conf = region.get("confidence", 0)
+                    conf_class = get_confidence_class(conf)
+                    color = "#4ECDC4" if conf >= 0.8 else "#ffc107" if conf >= 0.6 else "#dc3545"
+
+                    st.markdown(f"""
+                    <div class="ocr-region" style="border-left-color: {color};">
+                        <div style="display: flex; justify-content: space-between; margin-bottom: 4px;">
+                            <span style="font-size: 11px; color: #8b949e;">Region {i+1}</span>
+                            <span class="{conf_class}" style="font-size: 11px;">{conf:.0%}</span>
+                        </div>
+                        <div style="font-family: monospace; font-size: 12px;">{region.get('text', '')}</div>
+                    </div>
+                    """, unsafe_allow_html=True)
+
+                if len(filtered_regions) > 50:
+                    st.info(f"Showing 50 of {len(filtered_regions)} regions")
+        else:
+            st.info("No OCR regions available for this document")
+            st.markdown("OCR regions are extracted during document processing with OCR enabled.")
+
+    with tab_layout:
+        st.markdown("### Layout Structure")
+
+        layout_regions = active_doc.layout_data.get("regions", [])
+
+        if layout_regions:
+            # Group by type
+            by_type = {}
+            for region in layout_regions:
+                rtype = region.get("type", "unknown")
+                if rtype not in by_type:
+                    by_type[rtype] = []
+                by_type[rtype].append(region)
+
+            # Type summary
+            st.markdown("**Detected region types:**")
+            type_cols = st.columns(min(len(by_type), 6))
+            for i, (rtype, regions) in enumerate(by_type.items()):
+                color = get_layout_color(rtype)
+                type_cols[i % 6].markdown(f"""
+                <div class="layout-region" style="background: {color}20; border: 1px solid {color};">
+                    <strong>{rtype.title()}</strong>: {len(regions)}
+                </div>
+                """, unsafe_allow_html=True)
+
+            st.markdown("---")
+
+            # Layout regions list
+            type_filter = st.selectbox("Filter by type", ["All"] + list(by_type.keys()))
+
+            filtered_layout = layout_regions
+            if type_filter != "All":
+                filtered_layout = by_type.get(type_filter, [])
+
+            for i, region in enumerate(filtered_layout[:30]):
+                rtype = region.get("type", "unknown")
+                conf = region.get("confidence", 0)
+                color = get_layout_color(rtype)
+                conf_class = get_confidence_class(conf)
+
+                st.markdown(f"""
+                <div style="background: #161b22; border-radius: 6px; padding: 10px; margin: 6px 0; border-left: 3px solid {color};">
+                    <div style="display: flex; justify-content: space-between;">
+                        <span><strong style="color: {color};">{rtype.upper()}</strong></span>
+                        <span>Page {region.get('page', 0) + 1}</span>
+                        <span class="{conf_class}">{conf:.0%}</span>
+                    </div>
+                </div>
+                """, unsafe_allow_html=True)
+
+            if len(filtered_layout) > 30:
+                st.info(f"Showing 30 of {len(filtered_layout)} regions")
+        else:
+            st.info("No layout regions available for this document")
+            st.markdown("Layout regions are extracted during document processing with layout detection enabled.")
+
+    with tab_pages:
+        st.markdown("### Page Images")
+
+        if active_doc.page_images:
+            page_idx = st.selectbox(
+                "Select page",
+                list(range(len(active_doc.page_images))),
+                format_func=lambda x: f"Page {x + 1}"
+            )
+
+            if page_idx is not None and page_idx < len(active_doc.page_images):
+                img_data = active_doc.page_images[page_idx]
+
+                # Display image
+                st.image(
+                    f"data:image/png;base64,{img_data}",
+                    caption=f"Page {page_idx + 1}",
+                    use_container_width=True
+                )
+
+                # Overlay options
+                st.markdown("**Overlay options:**")
+                overlay_cols = st.columns(3)
+                with overlay_cols[0]:
+                    show_chunks = st.checkbox("Show chunk boundaries", value=False)
+                with overlay_cols[1]:
+                    show_ocr = st.checkbox("Show OCR regions", value=False)
+                with overlay_cols[2]:
+                    show_layout = st.checkbox("Show layout regions", value=False)
+
+                if show_chunks or show_ocr or show_layout:
+                    st.info("Overlay visualization coming soon - requires image annotation support")
+        else:
+            st.info("No page images available for this document")
+            st.markdown("Page images are extracted from PDF documents during processing.")
+
+    # Navigation to other modules
+    st.markdown("---")
+    st.markdown("### 🔗 Actions")
+
+    nav_cols = st.columns(4)
+
+    with nav_cols[0]:
+        if st.button("💬 Ask Questions", use_container_width=True):
+            st.switch_page("pages/2_💬_Interactive_RAG.py")
+
+    with nav_cols[1]:
+        if st.button("🎯 View Evidence", use_container_width=True):
+            st.switch_page("pages/4_🎯_Evidence_Viewer.py")
+
+    with nav_cols[2]:
+        if st.button("📊 Compare Documents", use_container_width=True):
+            st.switch_page("pages/3_📊_Document_Comparison.py")
+
+    with nav_cols[3]:
+        if st.button("🔬 Process New", use_container_width=True):
+            st.switch_page("pages/1_🔬_Live_Processing.py")
+
+else:
+    # No active document
+    st.markdown("## No Document Selected")
+
+    col1, col2 = st.columns(2)
+
+    with col1:
+        st.markdown("""
+        ### Getting Started
+
+        1. Go to **Live Processing** to upload and process a document
+        2. Processed documents will appear in the sidebar
+        3. Click on a document to view its details
+
+        Or select a document from the sidebar if you've already processed some.
+        """)
+
+        if st.button("🔬 Go to Live Processing", type="primary", use_container_width=True):
+            st.switch_page("pages/1_🔬_Live_Processing.py")
+
+    with col2:
+        # Show RAG stats
+        stats = get_store_stats()
+        st.markdown("### RAG Index Status")
+        st.metric("Total Indexed Chunks", stats.get("total_chunks", 0))
+
+        if rag_docs:
+            st.markdown("**Indexed Documents:**")
+            for doc in rag_docs[:5]:
+                doc_id = doc.get("document_id", "unknown")
+                chunks = doc.get("chunk_count", 0)
+                st.caption(f"• {doc_id[:30]}... ({chunks} chunks)")
+
+            if len(rag_docs) > 5:
+                st.caption(f"... and {len(rag_docs) - 5} more")

demo/rag_config.py

@@ -0,0 +1,396 @@
+"""
+Unified RAG Configuration for SPARKNET Demo
+
+This module provides a single source of truth for RAG system configuration,
+ensuring all demo pages use the same vector store, embeddings, and models.
+"""
+
+import streamlit as st
+from pathlib import Path
+import sys
+
+PROJECT_ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+# Configuration constants
+OLLAMA_BASE_URL = "http://localhost:11434"
+VECTOR_STORE_PATH = "data/sparknet_unified_rag"
+COLLECTION_NAME = "sparknet_documents"
+
+# Model preferences (in order of preference)
+EMBEDDING_MODELS = ["nomic-embed-text", "mxbai-embed-large:latest", "mxbai-embed-large"]
+LLM_MODELS = ["llama3.2:latest", "llama3.1:8b", "mistral:latest", "qwen2.5:14b", "qwen2.5:32b"]
+
+
+def check_ollama():
+    """Check Ollama availability and get available models."""
+    try:
+        import httpx
+        with httpx.Client(timeout=5.0) as client:
+            resp = client.get(f"{OLLAMA_BASE_URL}/api/tags")
+            if resp.status_code == 200:
+                models = [m["name"] for m in resp.json().get("models", [])]
+                return True, models
+    except:
+        pass
+    return False, []
+
+
+def select_model(available_models: list, preferred_models: list) -> str:
+    """Select the best available model from preferences."""
+    for model in preferred_models:
+        if model in available_models:
+            return model
+    # Return first preference as fallback
+    return preferred_models[0] if preferred_models else "llama3.2:latest"
+
+
+@st.cache_resource
+def get_unified_rag_system():
+    """
+    Initialize and return the unified RAG system.
+
+    This is cached at the Streamlit level so all pages share the same instance.
+    """
+    try:
+        from src.rag.agentic import AgenticRAG, RAGConfig
+        from src.rag.store import get_vector_store, VectorStoreConfig, reset_vector_store
+        from src.rag.embeddings import get_embedding_adapter, EmbeddingConfig, reset_embedding_adapter
+
+        # Check Ollama
+        ollama_ok, available_models = check_ollama()
+        if not ollama_ok:
+            return {
+                "status": "error",
+                "error": "Ollama is not running. Please start Ollama first.",
+                "rag": None,
+                "store": None,
+                "embedder": None,
+            }
+
+        # Select models
+        embed_model = select_model(available_models, EMBEDDING_MODELS)
+        llm_model = select_model(available_models, LLM_MODELS)
+
+        # Reset singletons to ensure fresh config
+        reset_vector_store()
+        reset_embedding_adapter()
+
+        # Initialize embedding adapter
+        embed_config = EmbeddingConfig(
+            ollama_model=embed_model,
+            ollama_base_url=OLLAMA_BASE_URL,
+        )
+        embedder = get_embedding_adapter(config=embed_config)
+
+        # Initialize vector store
+        store_config = VectorStoreConfig(
+            persist_directory=VECTOR_STORE_PATH,
+            collection_name=COLLECTION_NAME,
+            similarity_threshold=0.0,  # No threshold - let reranker handle filtering
+        )
+        store = get_vector_store(config=store_config)
+
+        # Initialize RAG config
+        rag_config = RAGConfig(
+            model=llm_model,
+            base_url=OLLAMA_BASE_URL,
+            max_revision_attempts=1,
+            enable_query_planning=True,
+            enable_reranking=True,
+            enable_validation=True,
+            retrieval_top_k=10,
+            final_top_k=5,
+            min_confidence=0.3,
+            verbose=False,
+        )
+
+        # Initialize RAG system
+        rag = AgenticRAG(
+            config=rag_config,
+            vector_store=store,
+            embedding_adapter=embedder,
+        )
+
+        return {
+            "status": "ready",
+            "error": None,
+            "rag": rag,
+            "store": store,
+            "embedder": embedder,
+            "embed_model": embed_model,
+            "llm_model": llm_model,
+            "available_models": available_models,
+        }
+
+    except Exception as e:
+        import traceback
+        return {
+            "status": "error",
+            "error": f"{str(e)}\n{traceback.format_exc()}",
+            "rag": None,
+            "store": None,
+            "embedder": None,
+        }
+
+
+def get_store_stats():
+    """Get current vector store statistics."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return {"total_chunks": 0, "status": "error"}
+
+    try:
+        return {
+            "total_chunks": system["store"].count(),
+            "status": "ready",
+            "embed_model": system.get("embed_model", "unknown"),
+            "llm_model": system.get("llm_model", "unknown"),
+        }
+    except:
+        return {"total_chunks": 0, "status": "error"}
+
+
+def index_document(text: str, document_id: str, metadata: dict = None) -> dict:
+    """Index a document into the unified RAG system."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return {"success": False, "error": system["error"], "num_chunks": 0}
+
+    try:
+        num_chunks = system["rag"].index_text(
+            text=text,
+            document_id=document_id,
+            metadata=metadata or {},
+        )
+        return {"success": True, "num_chunks": num_chunks, "error": None}
+    except Exception as e:
+        return {"success": False, "error": str(e), "num_chunks": 0}
+
+
+def query_rag(question: str, filters: dict = None):
+    """Query the unified RAG system."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return None, system["error"]
+
+    try:
+        response = system["rag"].query(question, filters=filters)
+        return response, None
+    except Exception as e:
+        return None, str(e)
+
+
+def clear_index():
+    """Clear the vector store index."""
+    # Force reinitialization by clearing cache
+    get_unified_rag_system.clear()
+    return True
+
+
+def get_indexed_documents() -> list:
+    """Get list of indexed document IDs from vector store."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return []
+
+    try:
+        # Query ChromaDB for unique document IDs
+        store = system["store"]
+        collection = store._collection
+
+        # Get all metadata to extract unique document_ids
+        results = collection.get(include=["metadatas"])
+        if not results or not results.get("metadatas"):
+            return []
+
+        doc_ids = set()
+        doc_info = {}
+        for meta in results["metadatas"]:
+            doc_id = meta.get("document_id", "unknown")
+            if doc_id not in doc_info:
+                doc_info[doc_id] = {
+                    "document_id": doc_id,
+                    "source_path": meta.get("source_path", ""),
+                    "chunk_count": 0,
+                }
+            doc_info[doc_id]["chunk_count"] += 1
+
+        return list(doc_info.values())
+    except Exception as e:
+        return []
+
+
+def get_chunks_for_document(document_id: str) -> list:
+    """Get all chunks for a specific document."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return []
+
+    try:
+        store = system["store"]
+        collection = store._collection
+
+        # Query for chunks with this document_id
+        results = collection.get(
+            where={"document_id": document_id},
+            include=["documents", "metadatas"]
+        )
+
+        if not results or not results.get("ids"):
+            return []
+
+        chunks = []
+        for i, chunk_id in enumerate(results["ids"]):
+            chunks.append({
+                "chunk_id": chunk_id,
+                "text": results["documents"][i] if results.get("documents") else "",
+                "metadata": results["metadatas"][i] if results.get("metadatas") else {},
+            })
+
+        return chunks
+    except Exception as e:
+        return []
+
+
+def search_similar_chunks(query: str, top_k: int = 5, doc_filter: str = None):
+    """Search for similar chunks with optional document filter."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return []
+
+    try:
+        embedder = system["embedder"]
+        store = system["store"]
+
+        # Generate query embedding
+        query_embedding = embedder.embed_text(query)
+
+        # Build filter
+        filters = None
+        if doc_filter:
+            filters = {"document_id": doc_filter}
+
+        # Search
+        results = store.search(
+            query_embedding=query_embedding,
+            top_k=top_k,
+            filters=filters,
+        )
+
+        return [
+            {
+                "chunk_id": r.chunk_id,
+                "document_id": r.document_id,
+                "text": r.text,
+                "similarity": r.similarity,
+                "page": r.page,
+                "metadata": r.metadata,
+            }
+            for r in results
+        ]
+    except Exception as e:
+        return []
+
+
+def compute_document_similarity(doc_id_1: str, doc_id_2: str) -> dict:
+    """Compute semantic similarity between two documents."""
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return {"error": "RAG system not ready", "similarity": 0.0}
+
+    try:
+        # Get chunks for both documents
+        chunks_1 = get_chunks_for_document(doc_id_1)
+        chunks_2 = get_chunks_for_document(doc_id_2)
+
+        if not chunks_1 or not chunks_2:
+            return {"error": "One or both documents not found", "similarity": 0.0}
+
+        embedder = system["embedder"]
+
+        # Compute average embeddings for each document
+        def avg_embedding(chunks):
+            embeddings = []
+            for chunk in chunks[:10]:  # Limit to first 10 chunks
+                emb = embedder.embed_text(chunk["text"])
+                embeddings.append(emb)
+            if not embeddings:
+                return None
+            # Average
+            import numpy as np
+            return np.mean(embeddings, axis=0).tolist()
+
+        emb1 = avg_embedding(chunks_1)
+        emb2 = avg_embedding(chunks_2)
+
+        if emb1 is None or emb2 is None:
+            return {"error": "Could not compute embeddings", "similarity": 0.0}
+
+        # Compute cosine similarity
+        import numpy as np
+        emb1 = np.array(emb1)
+        emb2 = np.array(emb2)
+        similarity = np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2))
+
+        return {
+            "similarity": float(similarity),
+            "doc1_chunks": len(chunks_1),
+            "doc2_chunks": len(chunks_2),
+            "error": None,
+        }
+    except Exception as e:
+        return {"error": str(e), "similarity": 0.0}
+
+
+def auto_index_processed_document(doc_id: str, text: str, chunks: list, metadata: dict = None):
+    """
+    Auto-index a processed document with pre-computed chunks.
+
+    This is called after document processing completes to immediately
+    make the document available in RAG.
+    """
+    system = get_unified_rag_system()
+    if system["status"] != "ready":
+        return {"success": False, "error": "RAG system not ready", "num_chunks": 0}
+
+    try:
+        store = system["store"]
+        embedder = system["embedder"]
+
+        # Prepare chunks for indexing
+        chunk_dicts = []
+        embeddings = []
+
+        for i, chunk in enumerate(chunks):
+            chunk_text = chunk.get("text", chunk) if isinstance(chunk, dict) else chunk
+
+            if len(chunk_text.strip()) < 20:
+                continue
+
+            chunk_id = f"{doc_id}_chunk_{i}"
+            chunk_dict = {
+                "chunk_id": chunk_id,
+                "document_id": doc_id,
+                "text": chunk_text,
+                "page": chunk.get("page", 0) if isinstance(chunk, dict) else 0,
+                "chunk_type": "text",
+                "source_path": metadata.get("filename", "") if metadata else "",
+                "sequence_index": i,
+            }
+            chunk_dicts.append(chunk_dict)
+
+            # Generate embedding
+            embedding = embedder.embed_text(chunk_text)
+            embeddings.append(embedding)
+
+        if not chunk_dicts:
+            return {"success": False, "error": "No valid chunks to index", "num_chunks": 0}
+
+        # Add to store
+        store.add_chunks(chunk_dicts, embeddings)
+
+        return {"success": True, "num_chunks": len(chunk_dicts), "error": None}
+
+    except Exception as e:
+        return {"success": False, "error": str(e), "num_chunks": 0}

demo/requirements.txt

@@ -0,0 +1,19 @@
+# SPARKNET Demo Requirements
+# Run: pip install -r demo/requirements.txt
+
+# Streamlit
+streamlit>=1.28.0
+
+# Data handling
+pandas>=2.0.0
+numpy>=1.24.0
+
+# HTTP client (for Ollama checks)
+httpx>=0.25.0
+
+# Image handling (optional, for advanced features)
+Pillow>=10.0.0
+
+# Charts (optional)
+plotly>=5.18.0
+altair>=5.2.0

demo/state_manager.py

@@ -0,0 +1,833 @@
+"""
+Unified State Manager for SPARKNET Demo
+
+Enhanced state management for cross-module communication (Phase 1B):
+- Document processing state tracking
+- Indexed documents registry
+- Cross-module event system (pub/sub)
+- Real-time status updates
+- Evidence highlighting synchronization
+- Document selection synchronization
+- Query/response sharing between modules
+"""
+
+import streamlit as st
+from pathlib import Path
+from typing import Dict, List, Any, Optional, Callable, Set
+from dataclasses import dataclass, field
+from datetime import datetime
+from enum import Enum
+import hashlib
+import json
+import sys
+import time
+from threading import Lock
+
+PROJECT_ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+
+# ==============================================================================
+# Event System (Phase 1B Enhancement)
+# ==============================================================================
+
+class EventType(str, Enum):
+    """Cross-module event types for synchronization."""
+    DOCUMENT_SELECTED = "document_selected"
+    DOCUMENT_PROCESSED = "document_processed"
+    DOCUMENT_INDEXED = "document_indexed"
+    DOCUMENT_REMOVED = "document_removed"
+    CHUNK_SELECTED = "chunk_selected"
+    EVIDENCE_HIGHLIGHT = "evidence_highlight"
+    RAG_QUERY_STARTED = "rag_query_started"
+    RAG_QUERY_COMPLETED = "rag_query_completed"
+    PAGE_CHANGED = "page_changed"
+    PROCESSING_STARTED = "processing_started"
+    PROCESSING_COMPLETED = "processing_completed"
+    SYSTEM_STATUS_CHANGED = "system_status_changed"
+
+
+@dataclass
+class Event:
+    """Cross-module event for synchronization."""
+    event_type: EventType
+    source_module: str
+    payload: Dict[str, Any]
+    timestamp: datetime = field(default_factory=datetime.now)
+    event_id: str = field(default_factory=lambda: hashlib.md5(
+        f"{time.time()}".encode()
+    ).hexdigest()[:8])
+
+
+@dataclass
+class EvidenceHighlight:
+    """Evidence highlight for cross-module visualization."""
+    doc_id: str
+    chunk_id: str
+    page: int
+    bbox: tuple  # (x_min, y_min, x_max, y_max)
+    text_snippet: str
+    confidence: float
+    source_query: Optional[str] = None
+    highlight_color: str = "#FFE082"  # Amber highlight
+
+
+@dataclass
+class ProcessedDocument:
+    """Represents a processed document with all extracted data."""
+    doc_id: str
+    filename: str
+    file_type: str
+    raw_text: str
+    chunks: List[Dict[str, Any]]
+    page_count: int = 1
+    page_images: List[bytes] = field(default_factory=list)
+    ocr_regions: List[Dict[str, Any]] = field(default_factory=list)
+    layout_data: Dict[str, Any] = field(default_factory=dict)
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    indexed: bool = False
+    indexed_chunks: int = 0
+    processing_time: float = 0.0
+    created_at: datetime = field(default_factory=datetime.now)
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "doc_id": self.doc_id,
+            "filename": self.filename,
+            "file_type": self.file_type,
+            "text_length": len(self.raw_text),
+            "chunk_count": len(self.chunks),
+            "page_count": self.page_count,
+            "ocr_region_count": len(self.ocr_regions),
+            "indexed": self.indexed,
+            "indexed_chunks": self.indexed_chunks,
+            "processing_time": self.processing_time,
+            "created_at": self.created_at.isoformat(),
+        }
+
+
+@dataclass
+class ProcessingStatus:
+    """Tracks processing status for a document."""
+    doc_id: str
+    stage: str  # loading, ocr, chunking, embedding, indexing, complete, error
+    progress: float  # 0.0 - 1.0
+    message: str
+    started_at: datetime = field(default_factory=datetime.now)
+    completed_at: Optional[datetime] = None
+    error: Optional[str] = None
+
+
+class UnifiedStateManager:
+    """
+    Central state manager for SPARKNET demo.
+
+    Enhanced with Phase 1B features:
+    - Document processing state tracking
+    - Indexed documents registry
+    - Cross-module event system (pub/sub)
+    - Real-time status updates
+    - Evidence highlighting sync
+    - Query/response sharing
+    """
+
+    def __init__(self):
+        self._ensure_session_state()
+        self._event_handlers: Dict[EventType, List[Callable]] = {}
+
+    def _ensure_session_state(self):
+        """Initialize session state if not exists."""
+        if "unified_state" not in st.session_state:
+            st.session_state.unified_state = {
+                "documents": {},  # doc_id -> ProcessedDocument
+                "processing_status": {},  # doc_id -> ProcessingStatus
+                "indexed_doc_ids": set(),
+                "active_doc_id": None,
+                "active_page": 0,
+                "active_chunk_id": None,
+                "notifications": [],
+                "rag_ready": False,
+                "total_indexed_chunks": 0,
+                "last_update": datetime.now().isoformat(),
+                # Phase 1B: Cross-module sync
+                "event_queue": [],  # List of Event objects
+                "evidence_highlights": [],  # List of EvidenceHighlight
+                "last_rag_query": None,
+                "last_rag_response": None,
+                "selected_sources": [],  # Source chunks from RAG
+                "module_states": {},  # Per-module custom state
+                "sync_version": 0,  # Increment on any state change
+            }
+
+    @property
+    def state(self) -> Dict:
+        """Get the unified state dict."""
+        self._ensure_session_state()
+        return st.session_state.unified_state
+
+    # ==================== Document Management ====================
+
+    def add_document(self, doc: ProcessedDocument) -> str:
+        """Add a processed document to the state."""
+        self.state["documents"][doc.doc_id] = doc
+        self._notify(f"Document '{doc.filename}' added", "info")
+        self._update_timestamp()
+        return doc.doc_id
+
+    def get_document(self, doc_id: str) -> Optional[ProcessedDocument]:
+        """Get a document by ID."""
+        return self.state["documents"].get(doc_id)
+
+    def get_all_documents(self) -> List[ProcessedDocument]:
+        """Get all documents."""
+        return list(self.state["documents"].values())
+
+    def get_indexed_documents(self) -> List[ProcessedDocument]:
+        """Get only indexed documents."""
+        return [d for d in self.state["documents"].values() if d.indexed]
+
+    def remove_document(self, doc_id: str):
+        """Remove a document from state."""
+        if doc_id in self.state["documents"]:
+            doc = self.state["documents"].pop(doc_id)
+            self.state["indexed_doc_ids"].discard(doc_id)
+            self._notify(f"Document '{doc.filename}' removed", "warning")
+            self._update_timestamp()
+
+    def set_active_document(self, doc_id: Optional[str]):
+        """Set the currently active document."""
+        self.state["active_doc_id"] = doc_id
+        self._update_timestamp()
+
+    def get_active_document(self) -> Optional[ProcessedDocument]:
+        """Get the currently active document."""
+        if self.state["active_doc_id"]:
+            return self.get_document(self.state["active_doc_id"])
+        return None
+
+    # ==================== Processing Status ====================
+
+    def start_processing(self, doc_id: str, filename: str):
+        """Start processing a document."""
+        status = ProcessingStatus(
+            doc_id=doc_id,
+            stage="loading",
+            progress=0.0,
+            message=f"Loading {filename}..."
+        )
+        self.state["processing_status"][doc_id] = status
+        self._update_timestamp()
+
+    def update_processing(self, doc_id: str, stage: str, progress: float, message: str):
+        """Update processing status."""
+        if doc_id in self.state["processing_status"]:
+            status = self.state["processing_status"][doc_id]
+            status.stage = stage
+            status.progress = progress
+            status.message = message
+            self._update_timestamp()
+
+    def complete_processing(self, doc_id: str, success: bool = True, error: str = None):
+        """Mark processing as complete."""
+        if doc_id in self.state["processing_status"]:
+            status = self.state["processing_status"][doc_id]
+            status.stage = "complete" if success else "error"
+            status.progress = 1.0 if success else status.progress
+            status.completed_at = datetime.now()
+            status.error = error
+            status.message = "Processing complete!" if success else f"Error: {error}"
+
+            if success:
+                self._notify(f"Document processed successfully!", "success")
+            else:
+                self._notify(f"Processing failed: {error}", "error")
+
+            self._update_timestamp()
+
+    def get_processing_status(self, doc_id: str) -> Optional[ProcessingStatus]:
+        """Get processing status for a document."""
+        return self.state["processing_status"].get(doc_id)
+
+    def is_processing(self, doc_id: str) -> bool:
+        """Check if document is being processed."""
+        status = self.get_processing_status(doc_id)
+        return status is not None and status.stage not in ["complete", "error"]
+
+    # ==================== Indexing ====================
+
+    def mark_indexed(self, doc_id: str, chunk_count: int):
+        """Mark a document as indexed to RAG."""
+        if doc_id in self.state["documents"]:
+            doc = self.state["documents"][doc_id]
+            doc.indexed = True
+            doc.indexed_chunks = chunk_count
+            self.state["indexed_doc_ids"].add(doc_id)
+            self.state["total_indexed_chunks"] += chunk_count
+            self._notify(f"Indexed {chunk_count} chunks from '{doc.filename}'", "success")
+            self._update_timestamp()
+
+    def is_indexed(self, doc_id: str) -> bool:
+        """Check if document is indexed."""
+        return doc_id in self.state["indexed_doc_ids"]
+
+    def get_total_indexed_chunks(self) -> int:
+        """Get total number of indexed chunks."""
+        return self.state["total_indexed_chunks"]
+
+    # ==================== Notifications ====================
+
+    def _notify(self, message: str, level: str = "info"):
+        """Add a notification."""
+        self.state["notifications"].append({
+            "message": message,
+            "level": level,
+            "timestamp": datetime.now().isoformat(),
+        })
+        # Keep only last 50 notifications
+        if len(self.state["notifications"]) > 50:
+            self.state["notifications"] = self.state["notifications"][-50:]
+
+    def get_notifications(self, limit: int = 10) -> List[Dict]:
+        """Get recent notifications."""
+        return self.state["notifications"][-limit:]
+
+    def clear_notifications(self):
+        """Clear all notifications."""
+        self.state["notifications"] = []
+
+    # ==================== RAG Status ====================
+
+    def set_rag_ready(self, ready: bool):
+        """Set RAG system ready status."""
+        self.state["rag_ready"] = ready
+        self._update_timestamp()
+
+    def is_rag_ready(self) -> bool:
+        """Check if RAG is ready."""
+        return self.state["rag_ready"]
+
+    # ==================== Utilities ====================
+
+    def _update_timestamp(self):
+        """Update the last update timestamp."""
+        self.state["last_update"] = datetime.now().isoformat()
+        self.state["sync_version"] += 1
+
+    def get_summary(self) -> Dict[str, Any]:
+        """Get a summary of current state."""
+        return {
+            "total_documents": len(self.state["documents"]),
+            "indexed_documents": len(self.state["indexed_doc_ids"]),
+            "total_indexed_chunks": self.state["total_indexed_chunks"],
+            "active_doc_id": self.state["active_doc_id"],
+            "active_page": self.state.get("active_page", 0),
+            "rag_ready": self.state["rag_ready"],
+            "last_update": self.state["last_update"],
+            "sync_version": self.state.get("sync_version", 0),
+            "processing_count": sum(
+                1 for s in self.state["processing_status"].values()
+                if s.stage not in ["complete", "error"]
+            ),
+            "evidence_count": len(self.state.get("evidence_highlights", [])),
+        }
+
+    def reset(self):
+        """Reset all state."""
+        st.session_state.unified_state = {
+            "documents": {},
+            "processing_status": {},
+            "indexed_doc_ids": set(),
+            "active_doc_id": None,
+            "active_page": 0,
+            "active_chunk_id": None,
+            "notifications": [],
+            "rag_ready": False,
+            "total_indexed_chunks": 0,
+            "last_update": datetime.now().isoformat(),
+            "event_queue": [],
+            "evidence_highlights": [],
+            "last_rag_query": None,
+            "last_rag_response": None,
+            "selected_sources": [],
+            "module_states": {},
+            "sync_version": 0,
+        }
+
+    # ==================== Event System (Phase 1B) ====================
+
+    def publish_event(
+        self,
+        event_type: EventType,
+        source_module: str,
+        payload: Dict[str, Any]
+    ) -> Event:
+        """
+        Publish an event for cross-module synchronization.
+
+        Args:
+            event_type: Type of event
+            source_module: Name of module publishing the event
+            payload: Event data
+
+        Returns:
+            The created Event object
+        """
+        event = Event(
+            event_type=event_type,
+            source_module=source_module,
+            payload=payload
+        )
+
+        # Add to event queue
+        self.state["event_queue"].append(event)
+
+        # Keep only last 100 events
+        if len(self.state["event_queue"]) > 100:
+            self.state["event_queue"] = self.state["event_queue"][-100:]
+
+        # Call registered handlers
+        if event_type in self._event_handlers:
+            for handler in self._event_handlers[event_type]:
+                try:
+                    handler(event)
+                except Exception as e:
+                    self._notify(f"Event handler error: {e}", "error")
+
+        self._update_timestamp()
+        return event
+
+    def subscribe(self, event_type: EventType, handler: Callable[[Event], None]):
+        """
+        Subscribe to an event type.
+
+        Args:
+            event_type: Type of event to subscribe to
+            handler: Callback function to handle the event
+        """
+        if event_type not in self._event_handlers:
+            self._event_handlers[event_type] = []
+        self._event_handlers[event_type].append(handler)
+
+    def unsubscribe(self, event_type: EventType, handler: Callable[[Event], None]):
+        """Unsubscribe from an event type."""
+        if event_type in self._event_handlers:
+            self._event_handlers[event_type] = [
+                h for h in self._event_handlers[event_type] if h != handler
+            ]
+
+    def get_recent_events(
+        self,
+        event_type: Optional[EventType] = None,
+        limit: int = 10
+    ) -> List[Event]:
+        """Get recent events, optionally filtered by type."""
+        events = self.state.get("event_queue", [])
+
+        if event_type:
+            events = [e for e in events if e.event_type == event_type]
+
+        return events[-limit:]
+
+    # ==================== Evidence Highlighting (Phase 1B) ====================
+
+    def add_evidence_highlight(self, highlight: EvidenceHighlight):
+        """
+        Add an evidence highlight for cross-module visualization.
+
+        Used when RAG finds relevant evidence that should be displayed
+        in the Document Viewer or Evidence Viewer.
+        """
+        self.state["evidence_highlights"].append(highlight)
+
+        # Publish event for other modules
+        self.publish_event(
+            EventType.EVIDENCE_HIGHLIGHT,
+            source_module="rag",
+            payload={
+                "doc_id": highlight.doc_id,
+                "chunk_id": highlight.chunk_id,
+                "page": highlight.page,
+                "bbox": highlight.bbox,
+                "text_snippet": highlight.text_snippet[:100],
+            }
+        )
+
+        self._update_timestamp()
+
+    def clear_evidence_highlights(self, doc_id: Optional[str] = None):
+        """Clear evidence highlights, optionally for a specific document."""
+        if doc_id:
+            self.state["evidence_highlights"] = [
+                h for h in self.state["evidence_highlights"]
+                if h.doc_id != doc_id
+            ]
+        else:
+            self.state["evidence_highlights"] = []
+
+        self._update_timestamp()
+
+    def get_evidence_highlights(
+        self,
+        doc_id: Optional[str] = None,
+        page: Optional[int] = None
+    ) -> List[EvidenceHighlight]:
+        """Get evidence highlights, optionally filtered by doc_id and page."""
+        highlights = self.state.get("evidence_highlights", [])
+
+        if doc_id:
+            highlights = [h for h in highlights if h.doc_id == doc_id]
+
+        if page is not None:
+            highlights = [h for h in highlights if h.page == page]
+
+        return highlights
+
+    # ==================== Page/Chunk Selection (Phase 1B) ====================
+
+    def select_page(self, page: int, source_module: str = "unknown"):
+        """
+        Set the active page and notify other modules.
+
+        Used for synchronized scrolling between Document Viewer and Evidence Viewer.
+        """
+        old_page = self.state.get("active_page", 0)
+        self.state["active_page"] = page
+
+        if old_page != page:
+            self.publish_event(
+                EventType.PAGE_CHANGED,
+                source_module=source_module,
+                payload={"page": page, "previous_page": old_page}
+            )
+
+    def get_active_page(self) -> int:
+        """Get the currently active page."""
+        return self.state.get("active_page", 0)
+
+    def select_chunk(
+        self,
+        chunk_id: str,
+        doc_id: str,
+        source_module: str = "unknown"
+    ):
+        """
+        Select a chunk and navigate to its location.
+
+        Publishes event to trigger synchronized navigation.
+        """
+        self.state["active_chunk_id"] = chunk_id
+
+        # Get chunk details to navigate
+        doc = self.get_document(doc_id)
+        if doc:
+            for chunk in doc.chunks:
+                if chunk.get("chunk_id") == chunk_id:
+                    page = chunk.get("page", 0)
+                    self.select_page(page, source_module)
+
+                    self.publish_event(
+                        EventType.CHUNK_SELECTED,
+                        source_module=source_module,
+                        payload={
+                            "chunk_id": chunk_id,
+                            "doc_id": doc_id,
+                            "page": page,
+                            "bbox": chunk.get("bbox"),
+                        }
+                    )
+                    break
+
+    def get_active_chunk_id(self) -> Optional[str]:
+        """Get the currently selected chunk ID."""
+        return self.state.get("active_chunk_id")
+
+    # ==================== RAG Query Sync (Phase 1B) ====================
+
+    def store_rag_query(
+        self,
+        query: str,
+        response: Dict[str, Any],
+        sources: List[Dict[str, Any]]
+    ):
+        """
+        Store the last RAG query and response for cross-module access.
+
+        Allows Evidence Viewer to display sources from Interactive RAG.
+        """
+        self.state["last_rag_query"] = query
+        self.state["last_rag_response"] = response
+        self.state["selected_sources"] = sources
+
+        # Clear old highlights and add new ones from sources
+        self.clear_evidence_highlights()
+
+        for source in sources:
+            if all(k in source for k in ["doc_id", "chunk_id", "page"]):
+                bbox = source.get("bbox", (0, 0, 1, 1))
+                if isinstance(bbox, dict):
+                    bbox = (bbox.get("x_min", 0), bbox.get("y_min", 0),
+                            bbox.get("x_max", 1), bbox.get("y_max", 1))
+
+                highlight = EvidenceHighlight(
+                    doc_id=source["doc_id"],
+                    chunk_id=source["chunk_id"],
+                    page=source["page"],
+                    bbox=bbox,
+                    text_snippet=source.get("text", "")[:200],
+                    confidence=source.get("score", 0.0),
+                    source_query=query,
+                )
+                self.add_evidence_highlight(highlight)
+
+        self.publish_event(
+            EventType.RAG_QUERY_COMPLETED,
+            source_module="rag",
+            payload={
+                "query": query,
+                "source_count": len(sources),
+                "response_length": len(str(response)),
+            }
+        )
+
+        self._update_timestamp()
+
+    def get_last_rag_query(self) -> Optional[str]:
+        """Get the last RAG query."""
+        return self.state.get("last_rag_query")
+
+    def get_last_rag_response(self) -> Optional[Dict[str, Any]]:
+        """Get the last RAG response."""
+        return self.state.get("last_rag_response")
+
+    def get_selected_sources(self) -> List[Dict[str, Any]]:
+        """Get the sources from the last RAG query."""
+        return self.state.get("selected_sources", [])
+
+    # ==================== Module State (Phase 1B) ====================
+
+    def set_module_state(self, module_name: str, state: Dict[str, Any]):
+        """
+        Store custom state for a specific module.
+
+        Allows modules to persist their own state across reruns.
+        """
+        self.state["module_states"][module_name] = {
+            **state,
+            "updated_at": datetime.now().isoformat()
+        }
+
+    def get_module_state(self, module_name: str) -> Dict[str, Any]:
+        """Get custom state for a specific module."""
+        return self.state.get("module_states", {}).get(module_name, {})
+
+    def get_sync_version(self) -> int:
+        """
+        Get the current sync version.
+
+        Modules can use this to detect if state has changed since last check.
+        """
+        return self.state.get("sync_version", 0)
+
+
+def generate_doc_id(filename: str, content_hash: str = None) -> str:
+    """Generate a unique document ID."""
+    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+    base = f"{filename}_{timestamp}"
+    if content_hash:
+        base = f"{base}_{content_hash[:8]}"
+    return hashlib.md5(base.encode()).hexdigest()[:12]
+
+
+def get_state_manager() -> UnifiedStateManager:
+    """Get or create the unified state manager."""
+    if "state_manager_instance" not in st.session_state:
+        st.session_state.state_manager_instance = UnifiedStateManager()
+    return st.session_state.state_manager_instance
+
+
+# ==================== Global Status Bar Component ====================
+
+def render_global_status_bar():
+    """Render a global status bar showing system state."""
+    manager = get_state_manager()
+    summary = manager.get_summary()
+
+    # Import RAG config for additional status
+    try:
+        from rag_config import get_unified_rag_system, check_ollama
+        rag_system = get_unified_rag_system()
+        ollama_ok, models = check_ollama()
+        rag_status = rag_system["status"]
+        llm_model = rag_system.get("llm_model", "N/A")
+    except:
+        ollama_ok = False
+        rag_status = "error"
+        llm_model = "N/A"
+        models = []
+
+    # Status bar
+    cols = st.columns(6)
+
+    with cols[0]:
+        if ollama_ok:
+            st.success(f"Ollama ({len(models)})")
+        else:
+            st.error("Ollama Offline")
+
+    with cols[1]:
+        if rag_status == "ready":
+            st.success("RAG Ready")
+        else:
+            st.error("RAG Error")
+
+    with cols[2]:
+        st.info(f"{llm_model.split(':')[0]}")
+
+    with cols[3]:
+        st.info(f"{summary['total_documents']} Docs")
+
+    with cols[4]:
+        if summary['indexed_documents'] > 0:
+            st.success(f"{summary['total_indexed_chunks']} Chunks")
+        else:
+            st.warning("0 Chunks")
+
+    with cols[5]:
+        if summary['processing_count'] > 0:
+            st.warning(f"Processing...")
+        else:
+            st.info("Idle")
+
+
+def render_notifications():
+    """Render recent notifications."""
+    manager = get_state_manager()
+    notifications = manager.get_notifications(5)
+
+    if notifications:
+        for notif in reversed(notifications):
+            level = notif["level"]
+            msg = notif["message"]
+            if level == "success":
+                st.success(msg)
+            elif level == "error":
+                st.error(msg)
+            elif level == "warning":
+                st.warning(msg)
+            else:
+                st.info(msg)
+
+
+# ==================== Helper Components (Phase 1B) ====================
+
+def render_evidence_panel():
+    """
+    Render a panel showing current evidence highlights.
+
+    Can be used in any module to show sources from RAG queries.
+    """
+    manager = get_state_manager()
+    highlights = manager.get_evidence_highlights()
+
+    if not highlights:
+        st.info("No evidence highlights. Run a RAG query to see sources.")
+        return
+
+    st.subheader(f"Evidence Sources ({len(highlights)})")
+
+    for i, h in enumerate(highlights):
+        with st.expander(f"Source {i+1}: Page {h.page + 1} ({h.confidence:.0%})"):
+            st.markdown(f"**Document:** {h.doc_id}")
+            st.markdown(f"**Text:** {h.text_snippet}")
+
+            if h.source_query:
+                st.markdown(f"**Query:** _{h.source_query}_")
+
+            # Button to navigate to source
+            if st.button(f"View in Document", key=f"view_source_{i}"):
+                manager.set_active_document(h.doc_id)
+                manager.select_page(h.page, "evidence_panel")
+                manager.select_chunk(h.chunk_id, h.doc_id, "evidence_panel")
+                st.rerun()
+
+
+def render_sync_status():
+    """Render sync status indicator for debugging."""
+    manager = get_state_manager()
+    summary = manager.get_summary()
+
+    with st.expander("Sync Status", expanded=False):
+        st.json({
+            "sync_version": summary["sync_version"],
+            "active_doc": summary["active_doc_id"],
+            "active_page": summary["active_page"],
+            "evidence_count": summary["evidence_count"],
+            "last_update": summary["last_update"],
+        })
+
+        # Recent events
+        events = manager.get_recent_events(limit=5)
+        if events:
+            st.subheader("Recent Events")
+            for event in reversed(events):
+                st.text(f"{event.event_type.value}: {event.source_module}")
+
+
+def render_document_selector():
+    """
+    Render a document selector that syncs with state manager.
+
+    Returns the selected document ID.
+    """
+    manager = get_state_manager()
+    documents = manager.get_all_documents()
+
+    if not documents:
+        st.info("No documents uploaded. Upload a document to get started.")
+        return None
+
+    # Get current selection
+    active_doc_id = manager.state.get("active_doc_id")
+
+    # Create options
+    options = {doc.doc_id: f"{doc.filename} ({doc.indexed_chunks} chunks)" for doc in documents}
+    option_list = list(options.keys())
+
+    # Find current index
+    current_index = option_list.index(active_doc_id) if active_doc_id in option_list else 0
+
+    # Render selectbox
+    selected_id = st.selectbox(
+        "Select Document",
+        options=option_list,
+        format_func=lambda x: options[x],
+        index=current_index,
+        key="global_doc_selector"
+    )
+
+    # Update state if changed
+    if selected_id != active_doc_id:
+        manager.set_active_document(selected_id)
+        manager.publish_event(
+            EventType.DOCUMENT_SELECTED,
+            source_module="selector",
+            payload={"doc_id": selected_id}
+        )
+
+    return selected_id
+
+
+def create_sync_callback(module_name: str) -> Callable:
+    """
+    Create a rerun callback for a module.
+
+    Returns a function that can be used as an event handler
+    to trigger Streamlit rerun when relevant events occur.
+    """
+    def callback(event: Event):
+        # Only rerun if event is from a different module
+        if event.source_module != module_name:
+            # Store that we need to rerun
+            st.session_state[f"_{module_name}_needs_rerun"] = True
+
+    return callback

docker-compose.dev.yml

@@ -0,0 +1,66 @@
+version: '3.8'
+
+# SPARKNET Development Docker Compose
+# Lighter configuration for local development
+
+services:
+  sparknet-api:
+    build:
+      context: .
+      dockerfile: Dockerfile
+      target: development
+    container_name: sparknet-api-dev
+    ports:
+      - "8000:8000"
+    volumes:
+      - .:/app
+      - ./data:/app/data
+      - ./uploads:/app/uploads
+      - ./outputs:/app/outputs
+    environment:
+      - PYTHONPATH=/app
+      - OLLAMA_HOST=http://host.docker.internal:11434
+      - LOG_LEVEL=DEBUG
+      - SPARKNET_SECRET_KEY=dev-secret-key
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    networks:
+      - sparknet-dev-network
+    restart: unless-stopped
+
+  sparknet-demo:
+    build:
+      context: .
+      dockerfile: Dockerfile
+      target: development
+    container_name: sparknet-demo-dev
+    command: ["streamlit", "run", "demo/app.py", "--server.address", "0.0.0.0", "--server.port", "4000", "--server.runOnSave", "true"]
+    ports:
+      - "4000:4000"
+    volumes:
+      - .:/app
+      - ./data:/app/data
+      - ./uploads:/app/uploads
+    environment:
+      - PYTHONPATH=/app
+      - OLLAMA_HOST=http://host.docker.internal:11434
+      - API_URL=http://sparknet-api:8000
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    depends_on:
+      - sparknet-api
+    networks:
+      - sparknet-dev-network
+    restart: unless-stopped
+
+  redis:
+    image: redis:7-alpine
+    container_name: sparknet-redis-dev
+    ports:
+      - "6379:6379"
+    networks:
+      - sparknet-dev-network
+
+networks:
+  sparknet-dev-network:
+    driver: bridge

docker-compose.yml

@@ -0,0 +1,163 @@
+version: '3.8'
+
+# SPARKNET Docker Compose Configuration
+# Full stack deployment with all services
+
+services:
+  # ============== Main Application ==============
+  sparknet-api:
+    build:
+      context: .
+      dockerfile: Dockerfile
+      target: production
+    container_name: sparknet-api
+    ports:
+      - "8000:8000"
+    volumes:
+      - ./data:/app/data
+      - ./uploads:/app/uploads
+      - ./outputs:/app/outputs
+      - ./logs:/app/logs
+    environment:
+      - PYTHONPATH=/app
+      - OLLAMA_HOST=http://ollama:11434
+      - CHROMA_HOST=chromadb
+      - CHROMA_PORT=8000
+      - REDIS_URL=redis://redis:6379
+      - SPARKNET_SECRET_KEY=${SPARKNET_SECRET_KEY:-sparknet-docker-secret-key}
+      - LOG_LEVEL=INFO
+    depends_on:
+      ollama:
+        condition: service_healthy
+      chromadb:
+        condition: service_started
+      redis:
+        condition: service_healthy
+    networks:
+      - sparknet-network
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/api/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+
+  sparknet-demo:
+    build:
+      context: .
+      dockerfile: Dockerfile
+      target: production
+    container_name: sparknet-demo
+    command: ["streamlit", "run", "demo/app.py", "--server.address", "0.0.0.0", "--server.port", "4000"]
+    ports:
+      - "4000:4000"
+    volumes:
+      - ./data:/app/data
+      - ./uploads:/app/uploads
+      - ./outputs:/app/outputs
+    environment:
+      - PYTHONPATH=/app
+      - OLLAMA_HOST=http://ollama:11434
+      - CHROMA_HOST=chromadb
+      - CHROMA_PORT=8000
+      - API_URL=http://sparknet-api:8000
+    depends_on:
+      - sparknet-api
+    networks:
+      - sparknet-network
+    restart: unless-stopped
+
+  # ============== Ollama LLM Service ==============
+  ollama:
+    image: ollama/ollama:latest
+    container_name: sparknet-ollama
+    ports:
+      - "11434:11434"
+    volumes:
+      - ollama_data:/root/.ollama
+    environment:
+      - OLLAMA_KEEP_ALIVE=24h
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    networks:
+      - sparknet-network
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+      start_period: 120s
+
+  # ============== ChromaDB Vector Store ==============
+  chromadb:
+    image: chromadb/chroma:latest
+    container_name: sparknet-chromadb
+    ports:
+      - "8001:8000"
+    volumes:
+      - chroma_data:/chroma/chroma
+    environment:
+      - IS_PERSISTENT=TRUE
+      - PERSIST_DIRECTORY=/chroma/chroma
+      - ANONYMIZED_TELEMETRY=FALSE
+    networks:
+      - sparknet-network
+    restart: unless-stopped
+
+  # ============== Redis Cache ==============
+  redis:
+    image: redis:7-alpine
+    container_name: sparknet-redis
+    ports:
+      - "6379:6379"
+    volumes:
+      - redis_data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    networks:
+      - sparknet-network
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  # ============== Nginx Reverse Proxy (Optional) ==============
+  nginx:
+    image: nginx:alpine
+    container_name: sparknet-nginx
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./nginx/ssl:/etc/nginx/ssl:ro
+    depends_on:
+      - sparknet-api
+      - sparknet-demo
+    networks:
+      - sparknet-network
+    restart: unless-stopped
+    profiles:
+      - production
+
+# ============== Volumes ==============
+volumes:
+  ollama_data:
+    driver: local
+  chroma_data:
+    driver: local
+  redis_data:
+    driver: local
+
+# ============== Networks ==============
+networks:
+  sparknet-network:
+    driver: bridge

docs/CLOUD_ARCHITECTURE.md

@@ -0,0 +1,392 @@
+# SPARKNET Cloud Architecture
+
+This document outlines the cloud-ready architecture for deploying SPARKNET on AWS.
+
+## Overview
+
+SPARKNET is designed with a modular architecture that supports both local development and cloud deployment. The system can scale from a single developer machine to enterprise-grade cloud infrastructure.
+
+## Local Development Stack
+
+```
+┌─────────────────────────────────────────────────────┐
+│                    Local Machine                     │
+├─────────────────────────────────────────────────────┤
+│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │
+│  │   Ollama    │  │  ChromaDB   │  │  File I/O   │ │
+│  │   (LLM)     │  │  (Vector)   │  │  (Storage)  │ │
+│  └─────────────┘  └─────────────┘  └─────────────┘ │
+│          │               │               │          │
+│          └───────────────┼───────────────┘          │
+│                          │                          │
+│                 ┌────────┴────────┐                 │
+│                 │    SPARKNET     │                 │
+│                 │   Application   │                 │
+│                 └─────────────────┘                 │
+└─────────────────────────────────────────────────────┘
+```
+
+## AWS Cloud Architecture
+
+### Target Architecture
+
+```
+┌────────────────────────────────────────────────────────────────────┐
+│                           AWS Cloud                                 │
+├────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  ┌─────────────┐      ┌─────────────┐      ┌─────────────────────┐ │
+│  │   API GW    │──────│   Lambda    │──────│    Step Functions   │ │
+│  │  (REST)     │      │  (Compute)  │      │   (Orchestration)   │ │
+│  └─────────────┘      └─────────────┘      └─────────────────────┘ │
+│         │                    │                       │              │
+│         │                    │                       │              │
+│         ▼                    ▼                       ▼              │
+│  ┌─────────────┐      ┌─────────────┐      ┌─────────────────────┐ │
+│  │     S3      │      │   Bedrock   │      │   OpenSearch        │ │
+│  │  (Storage)  │      │   (LLM)     │      │   (Vector Store)    │ │
+│  └─────────────┘      └─────────────┘      └─────────────────────┘ │
+│                                                                     │
+│  ┌─────────────┐      ┌─────────────┐      ┌─────────────────────┐ │
+│  │  Textract   │      │   Titan     │      │     DynamoDB        │ │
+│  │   (OCR)     │      │ (Embeddings)│      │    (Metadata)       │ │
+│  └─────────────┘      └─────────────┘      └─────────────────────┘ │
+│                                                                     │
+└────────────────────────────────────────────────────────────────────┘
+```
+
+### Component Mapping
+
+| Local Component | AWS Service | Purpose |
+|----------------|-------------|---------|
+| File I/O | S3 | Document storage |
+| PaddleOCR/Tesseract | Textract | OCR extraction |
+| Ollama LLM | Bedrock (Claude/Titan) | Text generation |
+| Ollama Embeddings | Titan Embeddings | Vector embeddings |
+| ChromaDB | OpenSearch Serverless | Vector search |
+| SQLite (optional) | DynamoDB | Metadata storage |
+| Python Process | Lambda | Compute |
+| CLI | API Gateway | HTTP interface |
+
+## Migration Strategy
+
+### Phase 1: Storage Migration
+
+```python
+# Abstract storage interface
+class StorageAdapter:
+    def put(self, key: str, data: bytes) -> str: ...
+    def get(self, key: str) -> bytes: ...
+    def delete(self, key: str) -> bool: ...
+
+# Local implementation
+class LocalStorageAdapter(StorageAdapter):
+    def __init__(self, base_path: str):
+        self.base_path = Path(base_path)
+
+# S3 implementation
+class S3StorageAdapter(StorageAdapter):
+    def __init__(self, bucket: str):
+        self.client = boto3.client('s3')
+        self.bucket = bucket
+```
+
+### Phase 2: OCR Migration
+
+```python
+# Abstract OCR interface
+class OCREngine:
+    def recognize(self, image: np.ndarray) -> OCRResult: ...
+
+# Local: PaddleOCR
+class PaddleOCREngine(OCREngine): ...
+
+# Cloud: Textract
+class TextractEngine(OCREngine):
+    def __init__(self):
+        self.client = boto3.client('textract')
+
+    def recognize(self, image: np.ndarray) -> OCRResult:
+        response = self.client.detect_document_text(
+            Document={'Bytes': image_bytes}
+        )
+        return self._convert_response(response)
+```
+
+### Phase 3: LLM Migration
+
+```python
+# Abstract LLM interface
+class LLMAdapter:
+    def generate(self, prompt: str) -> str: ...
+
+# Local: Ollama
+class OllamaAdapter(LLMAdapter): ...
+
+# Cloud: Bedrock
+class BedrockAdapter(LLMAdapter):
+    def __init__(self, model_id: str = "anthropic.claude-3-sonnet"):
+        self.client = boto3.client('bedrock-runtime')
+        self.model_id = model_id
+
+    def generate(self, prompt: str) -> str:
+        response = self.client.invoke_model(
+            modelId=self.model_id,
+            body=json.dumps({"prompt": prompt})
+        )
+        return response['body']
+```
+
+### Phase 4: Vector Store Migration
+
+```python
+# Abstract vector store interface (already implemented)
+class VectorStore:
+    def add_chunks(self, chunks, embeddings): ...
+    def search(self, query_embedding, top_k): ...
+
+# Local: ChromaDB (already implemented)
+class ChromaVectorStore(VectorStore): ...
+
+# Cloud: OpenSearch
+class OpenSearchVectorStore(VectorStore):
+    def __init__(self, endpoint: str, index: str):
+        self.client = OpenSearch(hosts=[endpoint])
+        self.index = index
+
+    def search(self, query_embedding, top_k):
+        response = self.client.search(
+            index=self.index,
+            body={
+                "knn": {
+                    "embedding": {
+                        "vector": query_embedding,
+                        "k": top_k
+                    }
+                }
+            }
+        )
+        return self._convert_results(response)
+```
+
+## AWS Services Deep Dive
+
+### Amazon S3
+
+- **Purpose**: Document storage and processed results
+- **Structure**:
+  ```
+  s3://sparknet-documents/
+  ├── raw/                    # Original documents
+  │   └── {doc_id}/
+  │       └── document.pdf
+  ├── processed/              # Processed results
+  │   └── {doc_id}/
+  │       ├── metadata.json
+  │       ├── chunks.json
+  │       └── pages/
+  │           ├── page_0.png
+  │           └── page_1.png
+  └── cache/                  # Processing cache
+  ```
+
+### Amazon Textract
+
+- **Purpose**: OCR extraction with layout analysis
+- **Features**:
+  - Document text detection
+  - Table extraction
+  - Form extraction
+  - Handwriting recognition
+
+### Amazon Bedrock
+
+- **Purpose**: LLM inference
+- **Models**:
+  - Claude 3.5 Sonnet (primary)
+  - Titan Text (cost-effective)
+  - Titan Embeddings (vectors)
+
+### Amazon OpenSearch Serverless
+
+- **Purpose**: Vector search and retrieval
+- **Configuration**:
+  ```json
+  {
+    "index": "sparknet-vectors",
+    "settings": {
+      "index.knn": true,
+      "index.knn.space_type": "cosinesimil"
+    },
+    "mappings": {
+      "properties": {
+        "embedding": {
+          "type": "knn_vector",
+          "dimension": 1024
+        }
+      }
+    }
+  }
+  ```
+
+### AWS Lambda
+
+- **Purpose**: Serverless compute
+- **Functions**:
+  - `process-document`: Document processing pipeline
+  - `extract-fields`: Field extraction
+  - `rag-query`: RAG query handling
+  - `index-document`: Vector indexing
+
+### AWS Step Functions
+
+- **Purpose**: Workflow orchestration
+- **Workflow**:
+  ```json
+  {
+    "StartAt": "ProcessDocument",
+    "States": {
+      "ProcessDocument": {
+        "Type": "Task",
+        "Resource": "arn:aws:lambda:process-document",
+        "Next": "IndexChunks"
+      },
+      "IndexChunks": {
+        "Type": "Task",
+        "Resource": "arn:aws:lambda:index-document",
+        "End": true
+      }
+    }
+  }
+  ```
+
+## Cost Optimization
+
+### Tiered Processing
+
+| Tier | Use Case | Services | Cost |
+|------|----------|----------|------|
+| Basic | Simple OCR | Textract + Titan | $ |
+| Standard | Full pipeline | + Claude Haiku | $$ |
+| Premium | Complex analysis | + Claude Sonnet | $$$ |
+
+### Caching Strategy
+
+1. **Document Cache**: S3 with lifecycle policies
+2. **Embedding Cache**: ElastiCache (Redis)
+3. **Query Cache**: Lambda@Edge
+
+## Security
+
+### IAM Policies
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:PutObject"
+      ],
+      "Resource": "arn:aws:s3:::sparknet-documents/*"
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "textract:DetectDocumentText",
+        "textract:AnalyzeDocument"
+      ],
+      "Resource": "*"
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "bedrock:InvokeModel"
+      ],
+      "Resource": "arn:aws:bedrock:*::foundation-model/*"
+    }
+  ]
+}
+```
+
+### Data Encryption
+
+- S3: Server-side encryption (SSE-S3 or SSE-KMS)
+- OpenSearch: Encryption at rest
+- Lambda: Environment variable encryption
+
+## Deployment
+
+### Infrastructure as Code (Terraform)
+
+```hcl
+# S3 Bucket
+resource "aws_s3_bucket" "documents" {
+  bucket = "sparknet-documents"
+}
+
+# Lambda Function
+resource "aws_lambda_function" "processor" {
+  function_name = "sparknet-processor"
+  runtime       = "python3.11"
+  handler       = "handler.process"
+  memory_size   = 1024
+  timeout       = 300
+}
+
+# OpenSearch Serverless
+resource "aws_opensearchserverless_collection" "vectors" {
+  name = "sparknet-vectors"
+  type = "VECTORSEARCH"
+}
+```
+
+### CI/CD Pipeline
+
+```yaml
+# GitHub Actions
+name: Deploy SPARKNET
+
+on:
+  push:
+    branches: [main]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Deploy Lambda
+        run: |
+          aws lambda update-function-code \
+            --function-name sparknet-processor \
+            --zip-file fileb://package.zip
+```
+
+## Monitoring
+
+### CloudWatch Metrics
+
+- Lambda invocations and duration
+- S3 request counts
+- OpenSearch query latency
+- Bedrock token usage
+
+### Dashboards
+
+- Processing throughput
+- Error rates
+- Cost tracking
+- Vector store statistics
+
+## Next Steps
+
+1. **Implement Storage Abstraction**: Create S3 adapter
+2. **Add Textract Engine**: Implement AWS OCR
+3. **Create Bedrock Adapter**: LLM migration
+4. **Deploy OpenSearch**: Vector store setup
+5. **Build Lambda Functions**: Serverless compute
+6. **Setup Step Functions**: Workflow orchestration
+7. **Configure CI/CD**: Automated deployment

docs/DOCUMENT_INTELLIGENCE.md

@@ -0,0 +1,470 @@
+# SPARKNET Document Intelligence
+
+A vision-first agentic document understanding platform that goes beyond OCR, supports complex layouts, and produces LLM-ready, visually grounded outputs suitable for RAG and field extraction at scale.
+
+## Overview
+
+The Document Intelligence subsystem provides:
+
+- **Vision-First Understanding**: Treats documents as visual objects, not just text
+- **Semantic Chunking**: Classifies regions by type (text, table, figure, chart, form, etc.)
+- **Visual Grounding**: Every extraction includes evidence (page, bbox, snippet, confidence)
+- **Zero-Shot Capability**: Works across diverse document formats without training
+- **Schema-Driven Extraction**: Define fields using JSON Schema or Pydantic models
+- **Abstention Policy**: Never guesses - abstains when confidence is low
+- **Local-First**: All processing happens locally for privacy
+
+## Quick Start
+
+### Basic Parsing
+
+```python
+from src.document_intelligence import DocumentParser, ParserConfig
+
+# Configure parser
+config = ParserConfig(
+    render_dpi=200,
+    max_pages=10,
+    include_markdown=True,
+)
+
+parser = DocumentParser(config=config)
+result = parser.parse("document.pdf")
+
+print(f"Parsed {len(result.chunks)} chunks from {result.num_pages} pages")
+
+# Access chunks
+for chunk in result.chunks:
+    print(f"[Page {chunk.page}] {chunk.chunk_type.value}: {chunk.text[:100]}...")
+```
+
+### Field Extraction
+
+```python
+from src.document_intelligence import (
+    FieldExtractor,
+    ExtractionSchema,
+    create_invoice_schema,
+)
+
+# Use preset schema
+schema = create_invoice_schema()
+
+# Or create custom schema
+schema = ExtractionSchema(name="CustomSchema")
+schema.add_string_field("company_name", "Name of the company", required=True)
+schema.add_date_field("document_date", "Date on document")
+schema.add_currency_field("total_amount", "Total amount")
+
+# Extract fields
+extractor = FieldExtractor()
+extraction = extractor.extract(parse_result, schema)
+
+print("Extracted Data:")
+for key, value in extraction.data.items():
+    if key in extraction.abstained_fields:
+        print(f"  {key}: [ABSTAINED]")
+    else:
+        print(f"  {key}: {value}")
+
+print(f"Confidence: {extraction.overall_confidence:.2f}")
+```
+
+### Visual Grounding
+
+```python
+from src.document_intelligence import (
+    load_document,
+    RenderOptions,
+)
+from src.document_intelligence.grounding import (
+    crop_region,
+    create_annotated_image,
+    EvidenceBuilder,
+)
+
+# Load and render page
+loader, renderer = load_document("document.pdf")
+page_image = renderer.render_page(1, RenderOptions(dpi=200))
+
+# Create annotated visualization
+bboxes = [chunk.bbox for chunk in result.chunks if chunk.page == 1]
+labels = [chunk.chunk_type.value for chunk in result.chunks if chunk.page == 1]
+annotated = create_annotated_image(page_image, bboxes, labels)
+
+# Crop specific region
+crop = crop_region(page_image, chunk.bbox, padding_percent=0.02)
+```
+
+### Question Answering
+
+```python
+from src.document_intelligence.tools import get_tool
+
+qa_tool = get_tool("answer_question")
+result = qa_tool.execute(
+    parse_result=parse_result,
+    question="What is the total amount due?",
+)
+
+if result.success:
+    print(f"Answer: {result.data['answer']}")
+    print(f"Confidence: {result.data['confidence']:.2f}")
+
+    for ev in result.evidence:
+        print(f"  Evidence: Page {ev['page']}, {ev['snippet'][:50]}...")
+```
+
+## Architecture
+
+### Module Structure
+
+```
+src/document_intelligence/
+├── __init__.py           # Main exports
+├── chunks/               # Core data models
+│   ├── models.py         # BoundingBox, DocumentChunk, TableChunk, etc.
+│   └── __init__.py
+├── io/                   # Document loading
+│   ├── base.py           # Abstract interfaces
+│   ├── pdf.py            # PDF loading (PyMuPDF)
+│   ├── image.py          # Image loading (PIL)
+│   ├── cache.py          # Page caching
+│   └── __init__.py
+├── models/               # Model interfaces
+│   ├── base.py           # BaseModel, BatchableModel
+│   ├── ocr.py            # OCRModel interface
+│   ├── layout.py         # LayoutModel interface
+│   ├── table.py          # TableModel interface
+│   ├── chart.py          # ChartModel interface
+│   ├── vlm.py            # VisionLanguageModel interface
+│   └── __init__.py
+├── parsing/              # Document parsing
+│   ├── parser.py         # DocumentParser orchestrator
+│   ├── chunking.py       # Semantic chunking utilities
+│   └── __init__.py
+├── grounding/            # Visual evidence
+│   ├── evidence.py       # EvidenceBuilder, EvidenceTracker
+│   ├── crops.py          # Image cropping utilities
+│   └── __init__.py
+├── extraction/           # Field extraction
+│   ├── schema.py         # ExtractionSchema, FieldSpec
+│   ├── extractor.py      # FieldExtractor
+│   ├── validator.py      # ExtractionValidator
+│   └── __init__.py
+├── tools/                # Agent tools
+│   ├── document_tools.py # Tool implementations
+│   └── __init__.py
+├── validation/           # Result validation
+│   └── __init__.py
+└── agent_adapter.py      # Agent integration
+```
+
+### Data Models
+
+#### BoundingBox
+
+Represents a rectangular region in XYXY format:
+
+```python
+from src.document_intelligence.chunks import BoundingBox
+
+# Normalized coordinates (0-1)
+bbox = BoundingBox(
+    x_min=0.1, y_min=0.2,
+    x_max=0.9, y_max=0.3,
+    normalized=True
+)
+
+# Convert to pixels
+pixel_bbox = bbox.to_pixel(width=1000, height=800)
+
+# Calculate IoU
+overlap = bbox1.iou(bbox2)
+
+# Check containment
+is_inside = bbox.contains((0.5, 0.25))
+```
+
+#### DocumentChunk
+
+Base semantic chunk:
+
+```python
+from src.document_intelligence.chunks import DocumentChunk, ChunkType
+
+chunk = DocumentChunk(
+    chunk_id="abc123",
+    doc_id="doc001",
+    chunk_type=ChunkType.PARAGRAPH,
+    text="Content...",
+    page=1,
+    bbox=bbox,
+    confidence=0.95,
+    sequence_index=0,
+)
+```
+
+#### TableChunk
+
+Table with cell structure:
+
+```python
+from src.document_intelligence.chunks import TableChunk, TableCell
+
+# Access cells
+cell = table.get_cell(row=0, col=1)
+
+# Export formats
+csv_data = table.to_csv()
+markdown = table.to_markdown()
+json_data = table.to_structured_json()
+```
+
+#### EvidenceRef
+
+Links extractions to visual sources:
+
+```python
+from src.document_intelligence.chunks import EvidenceRef
+
+evidence = EvidenceRef(
+    chunk_id="chunk_001",
+    doc_id="doc_001",
+    page=1,
+    bbox=bbox,
+    source_type="text",
+    snippet="The total is $500",
+    confidence=0.9,
+    cell_id=None,  # For table cells
+    crop_path=None,  # Path to cropped image
+)
+```
+
+## CLI Commands
+
+```bash
+# Parse document
+sparknet docint parse document.pdf -o result.json
+sparknet docint parse document.pdf --format markdown
+
+# Extract fields
+sparknet docint extract invoice.pdf --preset invoice
+sparknet docint extract doc.pdf -f vendor_name -f total_amount
+sparknet docint extract doc.pdf --schema my_schema.json
+
+# Ask questions
+sparknet docint ask document.pdf "What is the contract value?"
+
+# Classify document
+sparknet docint classify document.pdf
+
+# Search content
+sparknet docint search document.pdf -q "payment terms"
+sparknet docint search document.pdf --type table
+
+# Visualize regions
+sparknet docint visualize document.pdf --page 1 --annotate
+```
+
+## Configuration
+
+### Parser Configuration
+
+```python
+from src.document_intelligence import ParserConfig
+
+config = ParserConfig(
+    # Rendering
+    render_dpi=200,          # DPI for page rasterization
+    max_pages=None,          # Limit pages (None = all)
+
+    # OCR
+    ocr_enabled=True,
+    ocr_languages=["en"],
+    ocr_min_confidence=0.5,
+
+    # Layout
+    layout_enabled=True,
+    reading_order_enabled=True,
+
+    # Specialized extraction
+    table_extraction_enabled=True,
+    chart_extraction_enabled=True,
+
+    # Chunking
+    merge_adjacent_text=True,
+    min_chunk_chars=10,
+    max_chunk_chars=4000,
+
+    # Output
+    include_markdown=True,
+    cache_enabled=True,
+)
+```
+
+### Extraction Configuration
+
+```python
+from src.document_intelligence import ExtractionConfig
+
+config = ExtractionConfig(
+    # Confidence
+    min_field_confidence=0.5,
+    min_overall_confidence=0.5,
+
+    # Abstention
+    abstain_on_low_confidence=True,
+    abstain_threshold=0.3,
+
+    # Search
+    search_all_chunks=True,
+    prefer_structured_sources=True,
+
+    # Validation
+    validate_extracted_values=True,
+    normalize_values=True,
+)
+```
+
+## Preset Schemas
+
+### Invoice
+
+```python
+from src.document_intelligence import create_invoice_schema
+
+schema = create_invoice_schema()
+# Fields: invoice_number, invoice_date, due_date, vendor_name, vendor_address,
+#         customer_name, customer_address, subtotal, tax_amount, total_amount,
+#         currency, payment_terms
+```
+
+### Receipt
+
+```python
+from src.document_intelligence import create_receipt_schema
+
+schema = create_receipt_schema()
+# Fields: merchant_name, merchant_address, transaction_date, transaction_time,
+#         subtotal, tax_amount, total_amount, payment_method, last_four_digits
+```
+
+### Contract
+
+```python
+from src.document_intelligence import create_contract_schema
+
+schema = create_contract_schema()
+# Fields: contract_title, effective_date, expiration_date, party_a_name,
+#         party_b_name, contract_value, governing_law, termination_clause
+```
+
+## Agent Integration
+
+```python
+from src.document_intelligence.agent_adapter import (
+    DocumentIntelligenceAdapter,
+    EnhancedDocumentAgent,
+    AgentConfig,
+)
+
+# Create adapter
+config = AgentConfig(
+    render_dpi=200,
+    min_confidence=0.5,
+    max_iterations=10,
+)
+
+# With existing LLM client
+agent = EnhancedDocumentAgent(
+    llm_client=ollama_client,
+    config=config,
+)
+
+# Load document
+await agent.load_document("document.pdf")
+
+# Extract with schema
+result = await agent.extract_fields(schema)
+
+# Answer questions
+answer, evidence = await agent.answer_question("What is the total?")
+
+# Classify
+classification = await agent.classify()
+```
+
+## Available Tools
+
+| Tool | Description |
+|------|-------------|
+| `parse_document` | Parse document into semantic chunks |
+| `extract_fields` | Schema-driven field extraction |
+| `search_chunks` | Search document content |
+| `get_chunk_details` | Get detailed chunk information |
+| `get_table_data` | Extract structured table data |
+| `answer_question` | Document Q&A |
+| `crop_region` | Extract visual regions |
+
+## Best Practices
+
+### 1. Always Check Confidence
+
+```python
+if extraction.overall_confidence < 0.7:
+    print("Low confidence - manual review recommended")
+
+for field, value in extraction.data.items():
+    if field in extraction.abstained_fields:
+        print(f"{field}: Needs manual verification")
+```
+
+### 2. Use Evidence for Verification
+
+```python
+for evidence in extraction.evidence:
+    print(f"Found on page {evidence.page}")
+    print(f"Location: {evidence.bbox.xyxy}")
+    print(f"Source text: {evidence.snippet}")
+```
+
+### 3. Handle Abstention Gracefully
+
+```python
+result = extractor.extract(parse_result, schema)
+
+for field in schema.get_required_fields():
+    if field.name in result.abstained_fields:
+        # Request human review
+        flag_for_review(field.name, parse_result.doc_id)
+```
+
+### 4. Validate Before Use
+
+```python
+from src.document_intelligence import ExtractionValidator
+
+validator = ExtractionValidator(min_confidence=0.7)
+validation = validator.validate(result, schema)
+
+if not validation.is_valid:
+    for issue in validation.issues:
+        print(f"[{issue.severity}] {issue.field_name}: {issue.message}")
+```
+
+## Dependencies
+
+- `pymupdf` - PDF loading and rendering
+- `pillow` - Image processing
+- `numpy` - Array operations
+- `pydantic` - Data validation
+
+Optional:
+- `paddleocr` - OCR engine
+- `tesseract` - Alternative OCR
+- `chromadb` - Vector storage for RAG
+
+## License
+
+MIT License - see LICENSE file for details.

docs/SPARKNET_Progress_Report.py

@@ -0,0 +1,1432 @@
+#!/usr/bin/env python3
+"""
+SPARKNET Progress Report & Future Work PDF Generator
+Generates a comprehensive stakeholder presentation document.
+"""
+
+from reportlab.lib import colors
+from reportlab.lib.pagesizes import A4, landscape
+from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle
+from reportlab.lib.units import inch, cm
+from reportlab.lib.enums import TA_CENTER, TA_LEFT, TA_JUSTIFY, TA_RIGHT
+from reportlab.platypus import (
+    SimpleDocTemplate, Paragraph, Spacer, Table, TableStyle,
+    PageBreak, Image, ListFlowable, ListItem, KeepTogether,
+    Flowable, HRFlowable
+)
+from reportlab.graphics.shapes import Drawing, Rect, String, Line, Polygon
+from reportlab.graphics.charts.barcharts import VerticalBarChart
+from reportlab.graphics.charts.piecharts import Pie
+from reportlab.graphics import renderPDF
+from reportlab.pdfgen import canvas
+from datetime import datetime
+import os
+
+# Color Scheme - Professional Blue Theme
+PRIMARY_BLUE = colors.HexColor('#1e3a5f')
+SECONDARY_BLUE = colors.HexColor('#2d5a87')
+ACCENT_BLUE = colors.HexColor('#4a90d9')
+LIGHT_BLUE = colors.HexColor('#e8f4fc')
+SUCCESS_GREEN = colors.HexColor('#28a745')
+WARNING_ORANGE = colors.HexColor('#fd7e14')
+DANGER_RED = colors.HexColor('#dc3545')
+GRAY_DARK = colors.HexColor('#343a40')
+GRAY_LIGHT = colors.HexColor('#f8f9fa')
+WHITE = colors.white
+
+
+class DiagramFlowable(Flowable):
+    """Custom flowable for drawing architecture diagrams."""
+
+    def __init__(self, width, height, diagram_type='architecture'):
+        Flowable.__init__(self)
+        self.width = width
+        self.height = height
+        self.diagram_type = diagram_type
+
+    def draw(self):
+        if self.diagram_type == 'architecture':
+            self._draw_architecture()
+        elif self.diagram_type == 'rag_pipeline':
+            self._draw_rag_pipeline()
+        elif self.diagram_type == 'document_pipeline':
+            self._draw_document_pipeline()
+        elif self.diagram_type == 'agent_interaction':
+            self._draw_agent_interaction()
+        elif self.diagram_type == 'data_flow':
+            self._draw_data_flow()
+
+    def _draw_box(self, x, y, w, h, text, fill_color, text_color=WHITE, font_size=9):
+        """Draw a rounded box with text."""
+        self.canv.setFillColor(fill_color)
+        self.canv.roundRect(x, y, w, h, 5, fill=1, stroke=0)
+        self.canv.setFillColor(text_color)
+        self.canv.setFont('Helvetica-Bold', font_size)
+        # Center text
+        text_width = self.canv.stringWidth(text, 'Helvetica-Bold', font_size)
+        self.canv.drawString(x + (w - text_width) / 2, y + h/2 - 3, text)
+
+    def _draw_arrow(self, x1, y1, x2, y2, color=GRAY_DARK):
+        """Draw an arrow from (x1,y1) to (x2,y2)."""
+        self.canv.setStrokeColor(color)
+        self.canv.setLineWidth(2)
+        self.canv.line(x1, y1, x2, y2)
+        # Arrow head
+        import math
+        angle = math.atan2(y2-y1, x2-x1)
+        arrow_len = 8
+        self.canv.line(x2, y2, x2 - arrow_len * math.cos(angle - 0.4), y2 - arrow_len * math.sin(angle - 0.4))
+        self.canv.line(x2, y2, x2 - arrow_len * math.cos(angle + 0.4), y2 - arrow_len * math.sin(angle + 0.4))
+
+    def _draw_architecture(self):
+        """Draw the high-level SPARKNET architecture."""
+        # Title
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 12)
+        self.canv.drawCentredString(self.width/2, self.height - 20, 'SPARKNET Architecture Overview')
+
+        # User Layer
+        self._draw_box(self.width/2 - 60, self.height - 70, 120, 35, 'User Interface', ACCENT_BLUE)
+
+        # Demo Layer
+        self.canv.setFillColor(LIGHT_BLUE)
+        self.canv.roundRect(30, self.height - 160, self.width - 60, 70, 8, fill=1, stroke=0)
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 10)
+        self.canv.drawString(40, self.height - 100, 'Streamlit Demo Application')
+
+        # Demo pages
+        pages = ['Live\nProcessing', 'Interactive\nRAG', 'Doc\nComparison', 'Evidence\nViewer', 'Doc\nViewer']
+        page_width = (self.width - 100) / 5
+        for i, page in enumerate(pages):
+            x = 45 + i * page_width
+            self._draw_box(x, self.height - 150, page_width - 10, 35, page.replace('\n', ' '), SECONDARY_BLUE, font_size=7)
+
+        # Arrow from UI to Demo
+        self._draw_arrow(self.width/2, self.height - 70, self.width/2, self.height - 90, ACCENT_BLUE)
+
+        # Core Services Layer
+        self.canv.setFillColor(LIGHT_BLUE)
+        self.canv.roundRect(30, self.height - 280, self.width - 60, 100, 8, fill=1, stroke=0)
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 10)
+        self.canv.drawString(40, self.height - 190, 'Core Services')
+
+        # Core boxes
+        self._draw_box(50, self.height - 230, 100, 30, 'Document Intel', PRIMARY_BLUE, font_size=8)
+        self._draw_box(170, self.height - 230, 100, 30, 'Multi-Agent RAG', PRIMARY_BLUE, font_size=8)
+        self._draw_box(290, self.height - 230, 100, 30, 'Vector Store', PRIMARY_BLUE, font_size=8)
+        self._draw_box(410, self.height - 230, 80, 30, 'LLM Layer', PRIMARY_BLUE, font_size=8)
+
+        # Sub-components
+        self._draw_box(50, self.height - 270, 100, 30, 'OCR + Layout', SECONDARY_BLUE, font_size=7)
+        self._draw_box(170, self.height - 270, 100, 30, '5 Agents', SECONDARY_BLUE, font_size=7)
+        self._draw_box(290, self.height - 270, 100, 30, 'ChromaDB', SECONDARY_BLUE, font_size=7)
+        self._draw_box(410, self.height - 270, 80, 30, 'Ollama', SECONDARY_BLUE, font_size=7)
+
+        # Arrow from Demo to Core
+        self._draw_arrow(self.width/2, self.height - 160, self.width/2, self.height - 180, ACCENT_BLUE)
+
+        # Storage Layer
+        self.canv.setFillColor(GRAY_LIGHT)
+        self.canv.roundRect(30, self.height - 340, self.width - 60, 45, 8, fill=1, stroke=0)
+        self.canv.setFillColor(GRAY_DARK)
+        self.canv.setFont('Helvetica-Bold', 10)
+        self.canv.drawString(40, self.height - 310, 'Persistent Storage')
+
+        self._draw_box(150, self.height - 335, 80, 25, 'Embeddings', GRAY_DARK, font_size=7)
+        self._draw_box(250, self.height - 335, 80, 25, 'Documents', GRAY_DARK, font_size=7)
+        self._draw_box(350, self.height - 335, 80, 25, 'Cache', GRAY_DARK, font_size=7)
+
+        # Arrow
+        self._draw_arrow(self.width/2, self.height - 280, self.width/2, self.height - 295, GRAY_DARK)
+
+    def _draw_rag_pipeline(self):
+        """Draw the Multi-Agent RAG Pipeline."""
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 12)
+        self.canv.drawCentredString(self.width/2, self.height - 20, 'Multi-Agent RAG Pipeline')
+
+        # Query input
+        self._draw_box(20, self.height - 70, 80, 30, 'User Query', ACCENT_BLUE, font_size=8)
+
+        # Agents in sequence
+        agents = [
+            ('QueryPlanner', PRIMARY_BLUE, 'Intent Classification\nQuery Decomposition'),
+            ('Retriever', SECONDARY_BLUE, 'Hybrid Search\nDense + Sparse'),
+            ('Reranker', SECONDARY_BLUE, 'Cross-Encoder\nMMR Diversity'),
+            ('Synthesizer', PRIMARY_BLUE, 'Answer Generation\nCitation Tracking'),
+            ('Critic', WARNING_ORANGE, 'Hallucination Check\nValidation'),
+        ]
+
+        x_start = 120
+        box_width = 80
+        spacing = 10
+
+        for i, (name, color, desc) in enumerate(agents):
+            x = x_start + i * (box_width + spacing)
+            self._draw_box(x, self.height - 70, box_width, 30, name, color, font_size=7)
+            # Description below
+            self.canv.setFillColor(GRAY_DARK)
+            self.canv.setFont('Helvetica', 6)
+            lines = desc.split('\n')
+            for j, line in enumerate(lines):
+                self.canv.drawCentredString(x + box_width/2, self.height - 85 - j*8, line)
+
+            # Arrow to next
+            if i < len(agents) - 1:
+                self._draw_arrow(x + box_width, self.height - 55, x + box_width + spacing, self.height - 55, GRAY_DARK)
+
+        # Arrow from query to first agent
+        self._draw_arrow(100, self.height - 55, 120, self.height - 55, ACCENT_BLUE)
+
+        # Revision loop
+        self.canv.setStrokeColor(WARNING_ORANGE)
+        self.canv.setLineWidth(1.5)
+        self.canv.setDash(3, 3)
+        # Draw curved line for revision
+        critic_x = x_start + 4 * (box_width + spacing) + box_width
+        synth_x = x_start + 3 * (box_width + spacing)
+        self.canv.line(critic_x - 40, self.height - 100, synth_x + 40, self.height - 100)
+        self.canv.setDash()
+
+        self.canv.setFillColor(WARNING_ORANGE)
+        self.canv.setFont('Helvetica-Oblique', 7)
+        self.canv.drawCentredString((critic_x + synth_x)/2, self.height - 115, 'Revision Loop (if validation fails)')
+
+        # Final output
+        self._draw_box(critic_x + 20, self.height - 70, 80, 30, 'Response', SUCCESS_GREEN, font_size=8)
+        self._draw_arrow(critic_x, self.height - 55, critic_x + 20, self.height - 55, SUCCESS_GREEN)
+
+        # State tracking bar
+        self.canv.setFillColor(LIGHT_BLUE)
+        self.canv.roundRect(20, self.height - 160, self.width - 40, 35, 5, fill=1, stroke=0)
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 8)
+        self.canv.drawString(30, self.height - 145, 'RAGState: Query → Plan → Retrieved Chunks → Reranked → Answer → Validation → Citations')
+
+    def _draw_document_pipeline(self):
+        """Draw Document Processing Pipeline."""
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 12)
+        self.canv.drawCentredString(self.width/2, self.height - 20, 'Document Processing Pipeline')
+
+        stages = [
+            ('Input', 'PDF/Image\nUpload', ACCENT_BLUE),
+            ('OCR', 'PaddleOCR\nTesseract', PRIMARY_BLUE),
+            ('Layout', 'Region\nDetection', PRIMARY_BLUE),
+            ('Reading\nOrder', 'Sequence\nReconstruction', SECONDARY_BLUE),
+            ('Chunking', 'Semantic\nSplitting', SECONDARY_BLUE),
+            ('Indexing', 'ChromaDB\nEmbedding', SUCCESS_GREEN),
+        ]
+
+        box_width = 70
+        box_height = 45
+        spacing = 15
+        total_width = len(stages) * box_width + (len(stages) - 1) * spacing
+        x_start = (self.width - total_width) / 2
+        y_pos = self.height - 90
+
+        for i, (name, desc, color) in enumerate(stages):
+            x = x_start + i * (box_width + spacing)
+            # Main box
+            self._draw_box(x, y_pos, box_width, box_height, name.replace('\n', ' '), color, font_size=8)
+            # Description
+            self.canv.setFillColor(GRAY_DARK)
+            self.canv.setFont('Helvetica', 6)
+            lines = desc.split('\n')
+            for j, line in enumerate(lines):
+                self.canv.drawCentredString(x + box_width/2, y_pos - 15 - j*8, line)
+
+            # Arrow
+            if i < len(stages) - 1:
+                self._draw_arrow(x + box_width, y_pos + box_height/2, x + box_width + spacing, y_pos + box_height/2)
+
+        # Output description
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 9)
+        self.canv.drawCentredString(self.width/2, self.height - 160, 'Output: ProcessedDocument with chunks, OCR regions, layout data, bounding boxes')
+
+    def _draw_agent_interaction(self):
+        """Draw Agent Interaction Diagram."""
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 12)
+        self.canv.drawCentredString(self.width/2, self.height - 20, 'Agent Interaction & Data Flow')
+
+        # Central orchestrator
+        center_x, center_y = self.width/2, self.height/2 - 20
+        self._draw_box(center_x - 50, center_y - 20, 100, 40, 'Orchestrator', PRIMARY_BLUE, font_size=9)
+
+        # Surrounding agents
+        import math
+        agents = [
+            ('QueryPlanner', -120, 60),
+            ('Retriever', 0, 90),
+            ('Reranker', 120, 60),
+            ('Synthesizer', 120, -60),
+            ('Critic', 0, -90),
+        ]
+
+        for name, dx, dy in agents:
+            x = center_x + dx - 45
+            y = center_y + dy - 15
+            self._draw_box(x, y, 90, 30, name, SECONDARY_BLUE, font_size=8)
+            # Arrow to/from orchestrator
+            if dy > 0:
+                self._draw_arrow(center_x, center_y + 20, center_x + dx*0.3, center_y + dy - 15, ACCENT_BLUE)
+            else:
+                self._draw_arrow(center_x + dx*0.3, center_y + dy + 15, center_x, center_y - 20, ACCENT_BLUE)
+
+        # External connections
+        # Vector Store
+        self._draw_box(30, center_y - 15, 70, 30, 'ChromaDB', SUCCESS_GREEN, font_size=8)
+        self._draw_arrow(100, center_y, center_x - 50, center_y, SUCCESS_GREEN)
+
+        # LLM
+        self._draw_box(self.width - 100, center_y - 15, 70, 30, 'Ollama LLM', WARNING_ORANGE, font_size=8)
+        self._draw_arrow(self.width - 100, center_y, center_x + 50, center_y, WARNING_ORANGE)
+
+    def _draw_data_flow(self):
+        """Draw Data Flow Diagram."""
+        self.canv.setFillColor(PRIMARY_BLUE)
+        self.canv.setFont('Helvetica-Bold', 12)
+        self.canv.drawCentredString(self.width/2, self.height - 20, 'End-to-End Data Flow')
+
+        # Vertical flow
+        items = [
+            ('Document Upload', ACCENT_BLUE, 'PDF, Images, Text files'),
+            ('Document Processor', PRIMARY_BLUE, 'OCR → Layout → Chunking'),
+            ('State Manager', SECONDARY_BLUE, 'ProcessedDocument storage'),
+            ('Embedder', SECONDARY_BLUE, 'mxbai-embed-large (1024d)'),
+            ('ChromaDB', SUCCESS_GREEN, 'Vector indexing & storage'),
+            ('RAG Query', WARNING_ORANGE, 'User question processing'),
+            ('Multi-Agent Pipeline', PRIMARY_BLUE, '5-agent collaboration'),
+            ('Response', SUCCESS_GREEN, 'Answer with citations'),
+        ]
+
+        box_height = 28
+        spacing = 8
+        total_height = len(items) * box_height + (len(items) - 1) * spacing
+        y_start = self.height - 50
+        box_width = 160
+        x_center = self.width / 2 - box_width / 2
+
+        for i, (name, color, desc) in enumerate(items):
+            y = y_start - i * (box_height + spacing)
+            self._draw_box(x_center, y - box_height, box_width, box_height, name, color, font_size=8)
+            # Description on right
+            self.canv.setFillColor(GRAY_DARK)
+            self.canv.setFont('Helvetica', 7)
+            self.canv.drawString(x_center + box_width + 15, y - box_height/2 - 3, desc)
+
+            # Arrow
+            if i < len(items) - 1:
+                self._draw_arrow(x_center + box_width/2, y - box_height, x_center + box_width/2, y - box_height - spacing + 2)
+
+
+def create_styles():
+    """Create custom paragraph styles."""
+    styles = getSampleStyleSheet()
+
+    # Title style
+    styles.add(ParagraphStyle(
+        name='MainTitle',
+        parent=styles['Title'],
+        fontSize=28,
+        textColor=PRIMARY_BLUE,
+        spaceAfter=30,
+        alignment=TA_CENTER,
+        fontName='Helvetica-Bold'
+    ))
+
+    # Subtitle
+    styles.add(ParagraphStyle(
+        name='Subtitle',
+        parent=styles['Normal'],
+        fontSize=16,
+        textColor=SECONDARY_BLUE,
+        spaceAfter=20,
+        alignment=TA_CENTER,
+        fontName='Helvetica'
+    ))
+
+    # Section Header
+    styles.add(ParagraphStyle(
+        name='SectionHeader',
+        parent=styles['Heading1'],
+        fontSize=18,
+        textColor=PRIMARY_BLUE,
+        spaceBefore=25,
+        spaceAfter=15,
+        fontName='Helvetica-Bold',
+        borderColor=ACCENT_BLUE,
+        borderWidth=2,
+        borderPadding=5,
+    ))
+
+    # Subsection Header
+    styles.add(ParagraphStyle(
+        name='SubsectionHeader',
+        parent=styles['Heading2'],
+        fontSize=14,
+        textColor=SECONDARY_BLUE,
+        spaceBefore=15,
+        spaceAfter=10,
+        fontName='Helvetica-Bold'
+    ))
+
+    # Body text
+    styles.add(ParagraphStyle(
+        name='CustomBody',
+        parent=styles['Normal'],
+        fontSize=10,
+        textColor=GRAY_DARK,
+        spaceAfter=8,
+        alignment=TA_JUSTIFY,
+        leading=14
+    ))
+
+    # Bullet style
+    styles.add(ParagraphStyle(
+        name='BulletText',
+        parent=styles['Normal'],
+        fontSize=10,
+        textColor=GRAY_DARK,
+        leftIndent=20,
+        spaceAfter=5,
+        leading=13
+    ))
+
+    # Caption
+    styles.add(ParagraphStyle(
+        name='Caption',
+        parent=styles['Normal'],
+        fontSize=9,
+        textColor=GRAY_DARK,
+        alignment=TA_CENTER,
+        spaceAfter=15,
+        fontName='Helvetica-Oblique'
+    ))
+
+    # Highlight box text
+    styles.add(ParagraphStyle(
+        name='HighlightText',
+        parent=styles['Normal'],
+        fontSize=10,
+        textColor=PRIMARY_BLUE,
+        spaceAfter=5,
+        fontName='Helvetica-Bold'
+    ))
+
+    return styles
+
+
+def create_highlight_box(text, styles, color=LIGHT_BLUE):
+    """Create a highlighted text box."""
+    data = [[Paragraph(text, styles['HighlightText'])]]
+    table = Table(data, colWidths=[450])
+    table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, -1), color),
+        ('BOX', (0, 0), (-1, -1), 1, ACCENT_BLUE),
+        ('PADDING', (0, 0), (-1, -1), 12),
+        ('VALIGN', (0, 0), (-1, -1), 'MIDDLE'),
+    ]))
+    return table
+
+
+def create_status_table(items, styles):
+    """Create a status table with colored indicators."""
+    data = [['Component', 'Status', 'Completion']]
+    for item, status, completion in items:
+        if status == 'Complete':
+            status_color = SUCCESS_GREEN
+        elif status == 'In Progress':
+            status_color = WARNING_ORANGE
+        else:
+            status_color = DANGER_RED
+        data.append([item, status, completion])
+
+    table = Table(data, colWidths=[250, 100, 100])
+    table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), PRIMARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 10),
+        ('ALIGN', (1, 0), (-1, -1), 'CENTER'),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 8),
+    ]))
+    return table
+
+
+def create_metrics_table(metrics, styles):
+    """Create a metrics display table."""
+    data = []
+    for metric, value, change in metrics:
+        data.append([metric, value, change])
+
+    table = Table(data, colWidths=[200, 150, 100])
+    table.setStyle(TableStyle([
+        ('FONTNAME', (0, 0), (0, -1), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 11),
+        ('TEXTCOLOR', (1, 0), (1, -1), PRIMARY_BLUE),
+        ('ALIGN', (1, 0), (-1, -1), 'CENTER'),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('PADDING', (0, 0), (-1, -1), 10),
+        ('ROWBACKGROUNDS', (0, 0), (-1, -1), [LIGHT_BLUE, WHITE]),
+    ]))
+    return table
+
+
+def generate_report():
+    """Generate the complete SPARKNET progress report PDF."""
+
+    filename = '/home/mhamdan/SPARKNET/docs/SPARKNET_Progress_Report.pdf'
+    os.makedirs(os.path.dirname(filename), exist_ok=True)
+
+    doc = SimpleDocTemplate(
+        filename,
+        pagesize=A4,
+        rightMargin=50,
+        leftMargin=50,
+        topMargin=60,
+        bottomMargin=60
+    )
+
+    styles = create_styles()
+    story = []
+
+    # ========== TITLE PAGE ==========
+    story.append(Spacer(1, 100))
+    story.append(Paragraph('SPARKNET', styles['MainTitle']))
+    story.append(Paragraph('Multi-Agentic Document Intelligence Framework', styles['Subtitle']))
+    story.append(Spacer(1, 30))
+    story.append(Paragraph('Progress Report & Future Roadmap', styles['Subtitle']))
+    story.append(Spacer(1, 50))
+
+    # Version info box
+    version_data = [
+        ['Version', '1.0.0-beta'],
+        ['Report Date', datetime.now().strftime('%B %d, %Y')],
+        ['Document Type', 'Stakeholder Progress Report'],
+        ['Classification', 'Internal / Confidential'],
+    ]
+    version_table = Table(version_data, colWidths=[150, 200])
+    version_table.setStyle(TableStyle([
+        ('FONTNAME', (0, 0), (0, -1), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 10),
+        ('TEXTCOLOR', (0, 0), (-1, -1), GRAY_DARK),
+        ('ALIGN', (0, 0), (-1, -1), 'CENTER'),
+        ('GRID', (0, 0), (-1, -1), 0.5, ACCENT_BLUE),
+        ('PADDING', (0, 0), (-1, -1), 8),
+        ('BACKGROUND', (0, 0), (-1, -1), LIGHT_BLUE),
+    ]))
+    story.append(version_table)
+
+    story.append(PageBreak())
+
+    # ========== TABLE OF CONTENTS ==========
+    story.append(Paragraph('Table of Contents', styles['SectionHeader']))
+    story.append(Spacer(1, 20))
+
+    toc_items = [
+        ('1. Executive Summary', '3'),
+        ('2. Project Overview', '4'),
+        ('3. Technical Architecture', '5'),
+        ('4. Component Deep Dive', '8'),
+        ('5. Current Progress & Achievements', '12'),
+        ('6. Gap Analysis', '14'),
+        ('7. Future Work & Roadmap', '17'),
+        ('8. Risk Assessment', '20'),
+        ('9. Resource Requirements', '21'),
+        ('10. Conclusion & Recommendations', '22'),
+    ]
+
+    toc_data = [[Paragraph(f'<b>{item}</b>', styles['CustomBody']), page] for item, page in toc_items]
+    toc_table = Table(toc_data, colWidths=[400, 50])
+    toc_table.setStyle(TableStyle([
+        ('FONTSIZE', (0, 0), (-1, -1), 11),
+        ('ALIGN', (1, 0), (1, -1), 'RIGHT'),
+        ('BOTTOMPADDING', (0, 0), (-1, -1), 8),
+        ('LINEBELOW', (0, 0), (-1, -2), 0.5, colors.lightgrey),
+    ]))
+    story.append(toc_table)
+
+    story.append(PageBreak())
+
+    # ========== 1. EXECUTIVE SUMMARY ==========
+    story.append(Paragraph('1. Executive Summary', styles['SectionHeader']))
+
+    story.append(Paragraph(
+        '''SPARKNET represents a next-generation document intelligence platform that combines
+        advanced OCR capabilities, sophisticated layout analysis, and a state-of-the-art
+        Multi-Agent Retrieval-Augmented Generation (RAG) system. This report provides a
+        comprehensive overview of the project's current state, technical achievements,
+        identified gaps, and the strategic roadmap for future development.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>Key Highlights</b>', styles['SubsectionHeader']))
+
+    highlights = [
+        '<b>Multi-Agent RAG Architecture:</b> Successfully implemented a 5-agent pipeline (QueryPlanner, Retriever, Reranker, Synthesizer, Critic) with self-correction capabilities.',
+        '<b>Document Processing Pipeline:</b> Complete end-to-end document processing with OCR, layout detection, and semantic chunking.',
+        '<b>Production-Ready Demo:</b> Fully functional Streamlit application with 5 interactive modules for document intelligence workflows.',
+        '<b>Hallucination Detection:</b> Built-in validation and criticism system to ensure factual accuracy of generated responses.',
+        '<b>Unified State Management:</b> Cross-module communication enabling seamless user experience across all application components.',
+    ]
+
+    for h in highlights:
+        story.append(Paragraph(f'• {h}', styles['BulletText']))
+
+    story.append(Spacer(1, 20))
+
+    # Key Metrics
+    story.append(Paragraph('<b>Current System Metrics</b>', styles['SubsectionHeader']))
+    metrics = [
+        ('RAG Pipeline Agents', '5 Specialized Agents', '✓ Complete'),
+        ('Document Formats Supported', 'PDF, Images', '2 formats'),
+        ('Vector Dimensions', '1024 (mxbai-embed-large)', 'Production'),
+        ('Demo Application Pages', '5 Interactive Modules', '✓ Complete'),
+        ('LLM Integration', 'Ollama (Local)', 'Self-hosted'),
+    ]
+    story.append(create_metrics_table(metrics, styles))
+
+    story.append(PageBreak())
+
+    # ========== 2. PROJECT OVERVIEW ==========
+    story.append(Paragraph('2. Project Overview', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>2.1 Vision & Objectives</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''SPARKNET aims to revolutionize document intelligence by providing an integrated
+        platform that can understand, process, and intelligently query complex documents.
+        The system leverages cutting-edge AI techniques including multi-agent collaboration,
+        hybrid retrieval, and sophisticated answer synthesis with built-in validation.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>Core Objectives:</b>', styles['CustomBody']))
+
+    objectives = [
+        '<b>Intelligent Document Understanding:</b> Extract and structure information from diverse document formats with high accuracy.',
+        '<b>Conversational Intelligence:</b> Enable natural language querying over document collections with citation-backed responses.',
+        '<b>Reliability & Trust:</b> Implement hallucination detection and self-correction to ensure factual accuracy.',
+        '<b>Scalability:</b> Design for enterprise-scale document processing and retrieval workloads.',
+        '<b>Extensibility:</b> Modular architecture allowing easy integration of new capabilities and models.',
+    ]
+
+    for obj in objectives:
+        story.append(Paragraph(f'• {obj}', styles['BulletText']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>2.2 Target Use Cases</b>', styles['SubsectionHeader']))
+
+    use_cases = [
+        ['Use Case', 'Description', 'Status'],
+        ['Legal Document Analysis', 'Contract review, clause extraction, compliance checking', 'Supported'],
+        ['Research Paper Synthesis', 'Multi-paper querying, citation tracking, summary generation', 'Supported'],
+        ['Technical Documentation', 'API docs, manuals, knowledge base querying', 'Supported'],
+        ['Financial Reports', 'Annual reports, SEC filings, financial data extraction', 'Planned'],
+        ['Medical Records', 'Clinical notes, diagnostic reports (HIPAA compliance needed)', 'Future'],
+    ]
+
+    uc_table = Table(use_cases, colWidths=[130, 230, 90])
+    uc_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), PRIMARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 6),
+        ('ALIGN', (2, 0), (2, -1), 'CENTER'),
+    ]))
+    story.append(uc_table)
+
+    story.append(PageBreak())
+
+    # ========== 3. TECHNICAL ARCHITECTURE ==========
+    story.append(Paragraph('3. Technical Architecture', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>3.1 High-Level Architecture</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''SPARKNET follows a layered microservices-inspired architecture with clear separation
+        of concerns. The system is organized into presentation, service, and persistence layers,
+        with a central orchestration mechanism coordinating multi-agent workflows.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+
+    # Architecture Diagram
+    arch_diagram = DiagramFlowable(500, 350, 'architecture')
+    story.append(arch_diagram)
+    story.append(Paragraph('Figure 1: SPARKNET High-Level Architecture', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>3.2 Multi-Agent RAG Pipeline</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The heart of SPARKNET is its Multi-Agent RAG system, which orchestrates five
+        specialized agents in a sophisticated pipeline with self-correction capabilities.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+
+    # RAG Pipeline Diagram
+    rag_diagram = DiagramFlowable(500, 180, 'rag_pipeline')
+    story.append(rag_diagram)
+    story.append(Paragraph('Figure 2: Multi-Agent RAG Pipeline with Revision Loop', styles['Caption']))
+
+    story.append(PageBreak())
+
+    story.append(Paragraph('<b>3.3 Document Processing Pipeline</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''Documents undergo a multi-stage processing pipeline that extracts text, identifies
+        layout structure, establishes reading order, and creates semantically coherent chunks
+        optimized for retrieval.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+
+    # Document Pipeline Diagram
+    doc_diagram = DiagramFlowable(500, 180, 'document_pipeline')
+    story.append(doc_diagram)
+    story.append(Paragraph('Figure 3: Document Processing Pipeline', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>3.4 Agent Interaction Model</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The orchestrator coordinates all agents, managing state transitions and ensuring
+        proper data flow between components. External services (Vector Store, LLM) are
+        accessed through well-defined interfaces.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+
+    # Agent Interaction Diagram
+    agent_diagram = DiagramFlowable(500, 250, 'agent_interaction')
+    story.append(agent_diagram)
+    story.append(Paragraph('Figure 4: Agent Interaction Model', styles['Caption']))
+
+    story.append(PageBreak())
+
+    story.append(Paragraph('<b>3.5 Data Flow Architecture</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The end-to-end data flow illustrates how documents are processed from upload
+        through indexing, and how queries are handled through the multi-agent pipeline
+        to produce validated, citation-backed responses.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+
+    # Data Flow Diagram
+    flow_diagram = DiagramFlowable(500, 320, 'data_flow')
+    story.append(flow_diagram)
+    story.append(Paragraph('Figure 5: End-to-End Data Flow', styles['Caption']))
+
+    story.append(PageBreak())
+
+    # ========== 4. COMPONENT DEEP DIVE ==========
+    story.append(Paragraph('4. Component Deep Dive', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>4.1 Query Planning Agent</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The QueryPlannerAgent is responsible for understanding user intent, classifying
+        query types, and decomposing complex queries into manageable sub-queries.''',
+        styles['CustomBody']
+    ))
+
+    # Query types table
+    query_types = [
+        ['Intent Type', 'Description', 'Example'],
+        ['FACTOID', 'Simple fact lookup', '"What is the revenue for Q4?"'],
+        ['COMPARISON', 'Multi-entity comparison', '"Compare product A vs B features"'],
+        ['AGGREGATION', 'Cross-document summary', '"Summarize all quarterly reports"'],
+        ['CAUSAL', 'Why/how explanations', '"Why did revenue decline?"'],
+        ['PROCEDURAL', 'Step-by-step instructions', '"How to configure the system?"'],
+        ['MULTI_HOP', 'Multi-step reasoning', '"Which supplier has the lowest cost for product X?"'],
+    ]
+
+    qt_table = Table(query_types, colWidths=[90, 180, 180])
+    qt_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SECONDARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 5),
+    ]))
+    story.append(qt_table)
+    story.append(Paragraph('Table 1: Supported Query Intent Types', styles['Caption']))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>4.2 Hybrid Retrieval System</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The RetrieverAgent implements a sophisticated hybrid search combining dense
+        semantic retrieval with sparse keyword matching, using Reciprocal Rank Fusion (RRF)
+        to merge results optimally.''',
+        styles['CustomBody']
+    ))
+
+    retrieval_features = [
+        '<b>Dense Retrieval:</b> Embedding-based semantic search using mxbai-embed-large (1024 dimensions)',
+        '<b>Sparse Retrieval:</b> BM25-style keyword matching for precise term matching',
+        '<b>RRF Fusion:</b> Combines rankings using formula: RRF = Σ(1 / (k + rank))',
+        '<b>Intent-Adaptive Weights:</b> Adjusts dense/sparse balance based on query type (e.g., 80/20 for definitions, 50/50 for comparisons)',
+    ]
+
+    for feat in retrieval_features:
+        story.append(Paragraph(f'• {feat}', styles['BulletText']))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>4.3 Cross-Encoder Reranking</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The RerankerAgent applies LLM-based cross-encoder scoring to refine retrieval
+        results, implementing deduplication and Maximal Marginal Relevance (MMR) for
+        diversity promotion.''',
+        styles['CustomBody']
+    ))
+
+    reranker_config = [
+        ['Parameter', 'Value', 'Purpose'],
+        ['top_k', '5', 'Final result count'],
+        ['min_relevance_score', '0.3', 'Quality threshold'],
+        ['dedup_threshold', '0.9', 'Similarity for duplicate detection'],
+        ['MMR lambda', '0.7', 'Relevance vs diversity balance'],
+    ]
+
+    rr_table = Table(reranker_config, colWidths=[140, 80, 230])
+    rr_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SECONDARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('PADDING', (0, 0), (-1, -1), 6),
+    ]))
+    story.append(rr_table)
+    story.append(Paragraph('Table 2: Reranker Configuration', styles['Caption']))
+
+    story.append(PageBreak())
+
+    story.append(Paragraph('<b>4.4 Answer Synthesis</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The SynthesizerAgent generates comprehensive answers with automatic citation
+        tracking, supporting multiple output formats and implementing intelligent abstention
+        when evidence is insufficient.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Paragraph('<b>Supported Answer Formats:</b>', styles['CustomBody']))
+    formats = ['PROSE - Flowing paragraph narrative', 'BULLET_POINTS - Enumerated key points',
+               'TABLE - Comparative tabular format', 'STEP_BY_STEP - Procedural instructions']
+    for fmt in formats:
+        story.append(Paragraph(f'• {fmt}', styles['BulletText']))
+
+    story.append(Paragraph('<b>Confidence Calculation:</b>', styles['CustomBody']))
+    story.append(Paragraph('confidence = 0.5 × source_relevance + 0.3 × source_count_factor + 0.2 × consistency', styles['BulletText']))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>4.5 Validation & Hallucination Detection</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The CriticAgent performs comprehensive validation including hallucination detection,
+        citation verification, and factual consistency checking. It can trigger revision
+        cycles when issues are detected.''',
+        styles['CustomBody']
+    ))
+
+    issue_types = [
+        ['Issue Type', 'Description', 'Severity'],
+        ['HALLUCINATION', 'Information not supported by sources', 'Critical'],
+        ['UNSUPPORTED_CLAIM', 'Statement without citation', 'High'],
+        ['INCORRECT_CITATION', 'Citation references wrong source', 'High'],
+        ['CONTRADICTION', 'Internal inconsistency in answer', 'Medium'],
+        ['INCOMPLETE', 'Missing important information', 'Medium'],
+        ['FACTUAL_ERROR', 'Verifiable factual mistake', 'Critical'],
+    ]
+
+    it_table = Table(issue_types, colWidths=[130, 230, 90])
+    it_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), WARNING_ORANGE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 5),
+    ]))
+    story.append(it_table)
+    story.append(Paragraph('Table 3: Validation Issue Types', styles['Caption']))
+
+    story.append(PageBreak())
+
+    story.append(Paragraph('<b>4.6 Document Processing Components</b>', styles['SubsectionHeader']))
+
+    story.append(Paragraph('<b>OCR Engines:</b>', styles['CustomBody']))
+    ocr_comparison = [
+        ['Feature', 'PaddleOCR', 'Tesseract'],
+        ['GPU Acceleration', '✓ Yes', '✗ No'],
+        ['Multi-language', '✓ 80+ languages', '✓ 100+ languages'],
+        ['Accuracy (Clean)', '~95%', '~90%'],
+        ['Accuracy (Complex)', '~85%', '~75%'],
+        ['Speed', 'Fast', 'Moderate'],
+        ['Confidence Scores', '✓ Per-region', '✓ Per-word'],
+    ]
+
+    ocr_table = Table(ocr_comparison, colWidths=[130, 160, 160])
+    ocr_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), PRIMARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('PADDING', (0, 0), (-1, -1), 5),
+    ]))
+    story.append(ocr_table)
+    story.append(Paragraph('Table 4: OCR Engine Comparison', styles['Caption']))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>Layout Detection:</b>', styles['CustomBody']))
+    layout_types = ['TEXT, TITLE, HEADING, PARAGRAPH - Text regions',
+                    'TABLE, FIGURE, CHART - Visual elements',
+                    'CAPTION, FOOTNOTE - Supplementary text',
+                    'HEADER, FOOTER - Page elements',
+                    'FORMULA - Mathematical expressions']
+    for lt in layout_types:
+        story.append(Paragraph(f'• {lt}', styles['BulletText']))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>Chunking Configuration:</b>', styles['CustomBody']))
+    chunk_config = [
+        ['Parameter', 'Default', 'Description'],
+        ['max_chunk_chars', '1000', 'Maximum characters per chunk'],
+        ['min_chunk_chars', '50', 'Minimum viable chunk size'],
+        ['overlap_chars', '100', 'Overlap between consecutive chunks'],
+        ['Strategy', 'Semantic', 'Respects layout boundaries'],
+    ]
+
+    cc_table = Table(chunk_config, colWidths=[120, 80, 250])
+    cc_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SECONDARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('PADDING', (0, 0), (-1, -1), 5),
+    ]))
+    story.append(cc_table)
+    story.append(Paragraph('Table 5: Chunking Configuration', styles['Caption']))
+
+    story.append(PageBreak())
+
+    # ========== 5. CURRENT PROGRESS ==========
+    story.append(Paragraph('5. Current Progress & Achievements', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>5.1 Development Milestones</b>', styles['SubsectionHeader']))
+
+    milestones = [
+        ['Milestone', 'Status', 'Completion'],
+        ['Core RAG Pipeline', 'Complete', '100%'],
+        ['5-Agent Architecture', 'Complete', '100%'],
+        ['Document Processing Pipeline', 'Complete', '100%'],
+        ['ChromaDB Integration', 'Complete', '100%'],
+        ['Ollama LLM Integration', 'Complete', '100%'],
+        ['Streamlit Demo Application', 'Complete', '100%'],
+        ['State Management System', 'Complete', '100%'],
+        ['Hallucination Detection', 'Complete', '100%'],
+        ['PDF Processing', 'Complete', '100%'],
+        ['Self-Correction Loop', 'Complete', '100%'],
+    ]
+
+    ms_table = Table(milestones, colWidths=[220, 120, 110])
+    ms_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), PRIMARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 6),
+        ('ALIGN', (1, 0), (-1, -1), 'CENTER'),
+    ]))
+    story.append(ms_table)
+    story.append(Paragraph('Table 6: Development Milestones', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>5.2 Demo Application Features</b>', styles['SubsectionHeader']))
+
+    demo_features = [
+        ['Page', 'Features', 'Status'],
+        ['Live Processing', 'Real-time document processing, progress tracking, auto-indexing', '✓ Complete'],
+        ['Interactive RAG', 'Query interface, document filtering, chunk preview, citations', '✓ Complete'],
+        ['Document Comparison', 'Semantic similarity, structure analysis, content diff', '✓ Complete'],
+        ['Evidence Viewer', 'Confidence coloring, bounding boxes, OCR regions, export', '✓ Complete'],
+        ['Document Viewer', 'Multi-tab view, chunk display, layout visualization', '✓ Complete'],
+    ]
+
+    df_table = Table(demo_features, colWidths=[110, 270, 70])
+    df_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SECONDARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 5),
+        ('ALIGN', (2, 0), (2, -1), 'CENTER'),
+    ]))
+    story.append(df_table)
+    story.append(Paragraph('Table 7: Demo Application Features', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>5.3 Technical Achievements</b>', styles['SubsectionHeader']))
+
+    achievements = [
+        '<b>Hybrid Retrieval:</b> Successfully combined dense and sparse retrieval with RRF fusion, achieving better recall than either method alone.',
+        '<b>Self-Correction:</b> Implemented revision loop allowing the system to automatically fix issues detected by the Critic agent.',
+        '<b>Citation Tracking:</b> Automatic citation generation with [N] notation linking answers to source documents.',
+        '<b>Confidence Scoring:</b> Multi-factor confidence calculation providing transparency into answer reliability.',
+        '<b>Streaming Support:</b> Real-time response streaming for improved user experience during long generations.',
+        '<b>Cross-Module Communication:</b> Unified state manager enabling seamless navigation between application modules.',
+    ]
+
+    for ach in achievements:
+        story.append(Paragraph(f'• {ach}', styles['BulletText']))
+
+    story.append(PageBreak())
+
+    # ========== 6. GAP ANALYSIS ==========
+    story.append(Paragraph('6. Gap Analysis', styles['SectionHeader']))
+
+    story.append(Paragraph(
+        '''This section identifies current limitations and gaps in the SPARKNET system
+        that represent opportunities for improvement and future development.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>6.1 Functional Gaps</b>', styles['SubsectionHeader']))
+
+    functional_gaps = [
+        ['Gap ID', 'Category', 'Description', 'Impact', 'Priority'],
+        ['FG-001', 'Document Support', 'Limited to PDF and images; no Word, Excel, PowerPoint support', 'High', 'P1'],
+        ['FG-002', 'Table Extraction', 'Table structure not preserved during chunking', 'High', 'P1'],
+        ['FG-003', 'Multi-modal', 'No image/chart understanding within documents', 'Medium', 'P2'],
+        ['FG-004', 'Languages', 'Primarily English; limited multi-language support', 'Medium', 'P2'],
+        ['FG-005', 'Batch Processing', 'No bulk document upload/processing capability', 'Medium', 'P2'],
+        ['FG-006', 'Document Updates', 'No incremental update; full reprocessing required', 'Medium', 'P2'],
+        ['FG-007', 'User Feedback', 'No mechanism to learn from user corrections', 'Low', 'P3'],
+    ]
+
+    fg_table = Table(functional_gaps, colWidths=[50, 85, 200, 55, 55])
+    fg_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), DANGER_RED),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 4),
+        ('ALIGN', (0, 0), (0, -1), 'CENTER'),
+        ('ALIGN', (3, 0), (-1, -1), 'CENTER'),
+    ]))
+    story.append(fg_table)
+    story.append(Paragraph('Table 8: Functional Gaps', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>6.2 Technical Gaps</b>', styles['SubsectionHeader']))
+
+    technical_gaps = [
+        ['Gap ID', 'Category', 'Description', 'Impact', 'Priority'],
+        ['TG-001', 'Scalability', 'Single-node architecture; no distributed processing', 'High', 'P1'],
+        ['TG-002', 'Authentication', 'No user authentication or access control', 'High', 'P1'],
+        ['TG-003', 'API', 'No REST API for external integration', 'High', 'P1'],
+        ['TG-004', 'Caching', 'Limited query result caching; redundant LLM calls', 'Medium', 'P2'],
+        ['TG-005', 'Monitoring', 'Basic logging only; no metrics/alerting system', 'Medium', 'P2'],
+        ['TG-006', 'Testing', 'Limited test coverage; no integration tests', 'Medium', 'P2'],
+        ['TG-007', 'Cloud Deploy', 'Not containerized; no Kubernetes manifests', 'Medium', 'P2'],
+        ['TG-008', 'GPU Sharing', 'Single GPU utilization; no multi-GPU support', 'Low', 'P3'],
+    ]
+
+    tg_table = Table(technical_gaps, colWidths=[50, 80, 205, 55, 55])
+    tg_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), WARNING_ORANGE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 4),
+        ('ALIGN', (0, 0), (0, -1), 'CENTER'),
+        ('ALIGN', (3, 0), (-1, -1), 'CENTER'),
+    ]))
+    story.append(tg_table)
+    story.append(Paragraph('Table 9: Technical Gaps', styles['Caption']))
+
+    story.append(PageBreak())
+
+    story.append(Paragraph('<b>6.3 Performance Gaps</b>', styles['SubsectionHeader']))
+
+    perf_gaps = [
+        ['Gap ID', 'Metric', 'Current', 'Target', 'Gap'],
+        ['PG-001', 'Query Latency (simple)', '3-5 seconds', '<2 seconds', '~2x improvement needed'],
+        ['PG-002', 'Query Latency (complex)', '10-20 seconds', '<5 seconds', '~3x improvement needed'],
+        ['PG-003', 'Document Processing', '30-60 sec/page', '<10 sec/page', '~4x improvement needed'],
+        ['PG-004', 'Concurrent Users', '1-5', '50+', 'Major scaling required'],
+        ['PG-005', 'Index Size', '10K chunks', '1M+ chunks', 'Architecture redesign'],
+        ['PG-006', 'Accuracy (hallucination)', '~85%', '>95%', '~10% improvement'],
+    ]
+
+    pg_table = Table(perf_gaps, colWidths=[50, 120, 90, 90, 100])
+    pg_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SECONDARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 4),
+        ('ALIGN', (0, 0), (-1, -1), 'CENTER'),
+    ]))
+    story.append(pg_table)
+    story.append(Paragraph('Table 10: Performance Gaps', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>6.4 Security & Compliance Gaps</b>', styles['SubsectionHeader']))
+
+    security_gaps = [
+        '<b>No Authentication:</b> Currently no user login or session management',
+        '<b>No Authorization:</b> Missing role-based access control (RBAC) for documents',
+        '<b>Data Encryption:</b> Documents and embeddings stored unencrypted at rest',
+        '<b>Audit Logging:</b> No comprehensive audit trail for compliance requirements',
+        '<b>PII Detection:</b> No automatic detection/redaction of personally identifiable information',
+        '<b>GDPR/HIPAA:</b> Not compliant with major data protection regulations',
+    ]
+
+    for sg in security_gaps:
+        story.append(Paragraph(f'• {sg}', styles['BulletText']))
+
+    story.append(PageBreak())
+
+    # ========== 7. FUTURE WORK & ROADMAP ==========
+    story.append(Paragraph('7. Future Work & Roadmap', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>7.1 Strategic Roadmap Overview</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''The SPARKNET roadmap is organized into three phases, each building upon the
+        previous to transform the current prototype into a production-ready enterprise
+        solution.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+
+    # Roadmap phases
+    roadmap = [
+        ['Phase', 'Timeline', 'Focus Areas', 'Key Deliverables'],
+        ['Phase 1:\nFoundation', 'Q1-Q2 2026',
+         'Stability, Core Features,\nBasic Security',
+         '• REST API\n• Authentication\n• Extended document formats\n• Basic containerization'],
+        ['Phase 2:\nScale', 'Q3-Q4 2026',
+         'Performance, Scalability,\nEnterprise Features',
+         '• Distributed processing\n• Advanced caching\n• Multi-tenancy\n• Monitoring & alerting'],
+        ['Phase 3:\nInnovation', 'Q1-Q2 2027',
+         'Advanced AI, Compliance,\nEcosystem',
+         '• Multi-modal understanding\n• Compliance frameworks\n• Plugin architecture\n• Advanced analytics'],
+    ]
+
+    rm_table = Table(roadmap, colWidths=[70, 80, 130, 170])
+    rm_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), PRIMARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [LIGHT_BLUE, WHITE]),
+        ('PADDING', (0, 0), (-1, -1), 6),
+        ('VALIGN', (0, 0), (-1, -1), 'TOP'),
+    ]))
+    story.append(rm_table)
+    story.append(Paragraph('Table 11: Strategic Roadmap', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>7.2 Phase 1: Foundation (Q1-Q2 2026)</b>', styles['SubsectionHeader']))
+
+    phase1_items = [
+        ['Item', 'Description', 'Effort', 'Dependencies'],
+        ['REST API Development', 'FastAPI-based API for all core functions', '4 weeks', 'None'],
+        ['User Authentication', 'JWT-based auth with OAuth2 support', '3 weeks', 'API'],
+        ['Document Format Extension', 'Add Word, Excel, PowerPoint support', '4 weeks', 'None'],
+        ['Table Extraction', 'Preserve table structure in processing', '3 weeks', 'None'],
+        ['Docker Containerization', 'Production-ready Docker images', '2 weeks', 'None'],
+        ['Basic CI/CD Pipeline', 'Automated testing and deployment', '2 weeks', 'Docker'],
+        ['Query Result Caching', 'Redis-based caching layer', '2 weeks', 'API'],
+        ['Unit Test Coverage', 'Achieve 80% code coverage', '3 weeks', 'Ongoing'],
+    ]
+
+    p1_table = Table(phase1_items, colWidths=[130, 180, 60, 80])
+    p1_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SUCCESS_GREEN),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 4),
+    ]))
+    story.append(p1_table)
+    story.append(Paragraph('Table 12: Phase 1 Deliverables', styles['Caption']))
+
+    story.append(PageBreak())
+
+    story.append(Paragraph('<b>7.3 Phase 2: Scale (Q3-Q4 2026)</b>', styles['SubsectionHeader']))
+
+    phase2_items = [
+        ['Item', 'Description', 'Effort', 'Dependencies'],
+        ['Distributed Processing', 'Celery/Ray for parallel document processing', '6 weeks', 'Phase 1'],
+        ['Vector Store Scaling', 'Milvus/Pinecone for large-scale indices', '4 weeks', 'Phase 1'],
+        ['Multi-tenancy', 'Organization-based data isolation', '4 weeks', 'Auth'],
+        ['Kubernetes Deployment', 'Full K8s manifests and Helm charts', '3 weeks', 'Docker'],
+        ['Monitoring Stack', 'Prometheus, Grafana, ELK integration', '3 weeks', 'K8s'],
+        ['Batch Processing', 'Bulk document upload and processing', '3 weeks', 'Distributed'],
+        ['Advanced Caching', 'Semantic caching for similar queries', '3 weeks', 'Cache'],
+        ['Performance Optimization', 'Achieve <2s simple query latency', '4 weeks', 'Caching'],
+    ]
+
+    p2_table = Table(phase2_items, colWidths=[130, 180, 60, 80])
+    p2_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), WARNING_ORANGE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 4),
+    ]))
+    story.append(p2_table)
+    story.append(Paragraph('Table 13: Phase 2 Deliverables', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>7.4 Phase 3: Innovation (Q1-Q2 2027)</b>', styles['SubsectionHeader']))
+
+    phase3_items = [
+        ['Item', 'Description', 'Effort', 'Dependencies'],
+        ['Multi-modal Understanding', 'GPT-4V/Claude Vision for image analysis', '6 weeks', 'Phase 2'],
+        ['Advanced Table QA', 'SQL-like queries over extracted tables', '4 weeks', 'Table Extract'],
+        ['PII Detection/Redaction', 'Automatic sensitive data handling', '4 weeks', 'None'],
+        ['Compliance Framework', 'GDPR, HIPAA, SOC2 compliance', '8 weeks', 'PII'],
+        ['Plugin Architecture', 'Extensible agent and tool system', '4 weeks', 'Phase 2'],
+        ['Analytics Dashboard', 'Usage analytics and insights', '3 weeks', 'Monitoring'],
+        ['Multi-language Support', 'Full support for top 10 languages', '4 weeks', 'None'],
+        ['Feedback Learning', 'Learn from user corrections', '4 weeks', 'Analytics'],
+    ]
+
+    p3_table = Table(phase3_items, colWidths=[130, 180, 60, 80])
+    p3_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), ACCENT_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 4),
+    ]))
+    story.append(p3_table)
+    story.append(Paragraph('Table 14: Phase 3 Deliverables', styles['Caption']))
+
+    story.append(PageBreak())
+
+    # ========== 8. RISK ASSESSMENT ==========
+    story.append(Paragraph('8. Risk Assessment', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>8.1 Technical Risks</b>', styles['SubsectionHeader']))
+
+    tech_risks = [
+        ['Risk', 'Probability', 'Impact', 'Mitigation'],
+        ['LLM API Changes', 'Medium', 'High', 'Abstract LLM interface; support multiple providers'],
+        ['Scaling Bottlenecks', 'High', 'High', 'Early load testing; phased rollout'],
+        ['Model Accuracy Plateau', 'Medium', 'Medium', 'Ensemble approaches; fine-tuning capability'],
+        ['Dependency Vulnerabilities', 'Medium', 'Medium', 'Regular dependency audits; Dependabot'],
+        ['Data Loss', 'Low', 'Critical', 'Automated backups; disaster recovery plan'],
+    ]
+
+    tr_table = Table(tech_risks, colWidths=[120, 70, 70, 190])
+    tr_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), DANGER_RED),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 5),
+        ('ALIGN', (1, 0), (2, -1), 'CENTER'),
+    ]))
+    story.append(tr_table)
+    story.append(Paragraph('Table 15: Technical Risks', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>8.2 Project Risks</b>', styles['SubsectionHeader']))
+
+    proj_risks = [
+        ['Risk', 'Probability', 'Impact', 'Mitigation'],
+        ['Scope Creep', 'High', 'Medium', 'Strict phase gates; change control process'],
+        ['Resource Constraints', 'Medium', 'High', 'Prioritized backlog; MVP focus'],
+        ['Timeline Slippage', 'Medium', 'Medium', 'Buffer time; parallel workstreams'],
+        ['Knowledge Silos', 'Medium', 'Medium', 'Documentation; pair programming; code reviews'],
+        ['Stakeholder Alignment', 'Low', 'High', 'Regular demos; feedback cycles'],
+    ]
+
+    pr_table = Table(proj_risks, colWidths=[120, 70, 70, 190])
+    pr_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), WARNING_ORANGE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 5),
+        ('ALIGN', (1, 0), (2, -1), 'CENTER'),
+    ]))
+    story.append(pr_table)
+    story.append(Paragraph('Table 16: Project Risks', styles['Caption']))
+
+    story.append(PageBreak())
+
+    # ========== 9. RESOURCE REQUIREMENTS ==========
+    story.append(Paragraph('9. Resource Requirements', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>9.1 Team Structure (Recommended)</b>', styles['SubsectionHeader']))
+
+    team = [
+        ['Role', 'Count', 'Phase 1', 'Phase 2', 'Phase 3'],
+        ['Senior ML Engineer', '2', '✓', '✓', '✓'],
+        ['Backend Developer', '2', '✓', '✓', '✓'],
+        ['Frontend Developer', '1', '✓', '✓', '✓'],
+        ['DevOps Engineer', '1', '✓', '✓', '✓'],
+        ['QA Engineer', '1', '—', '✓', '✓'],
+        ['Technical Lead', '1', '✓', '✓', '✓'],
+        ['Product Manager', '1', '✓', '✓', '✓'],
+    ]
+
+    team_table = Table(team, colWidths=[130, 60, 70, 70, 70])
+    team_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), PRIMARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 9),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 6),
+        ('ALIGN', (1, 0), (-1, -1), 'CENTER'),
+    ]))
+    story.append(team_table)
+    story.append(Paragraph('Table 17: Team Structure', styles['Caption']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>9.2 Infrastructure Requirements</b>', styles['SubsectionHeader']))
+
+    infra = [
+        ['Component', 'Development', 'Staging', 'Production'],
+        ['GPU Servers', '1x A100 40GB', '2x A100 40GB', '4x A100 80GB'],
+        ['CPU Servers', '4 vCPU, 16GB', '8 vCPU, 32GB', '16 vCPU, 64GB x3'],
+        ['Storage', '500GB SSD', '2TB SSD', '10TB SSD + S3'],
+        ['Vector DB', 'ChromaDB local', 'Milvus single', 'Milvus cluster'],
+        ['Cache', 'In-memory', 'Redis single', 'Redis cluster'],
+        ['Load Balancer', 'None', 'Nginx', 'AWS ALB / GCP LB'],
+    ]
+
+    infra_table = Table(infra, colWidths=[100, 120, 120, 110])
+    infra_table.setStyle(TableStyle([
+        ('BACKGROUND', (0, 0), (-1, 0), SECONDARY_BLUE),
+        ('TEXTCOLOR', (0, 0), (-1, 0), WHITE),
+        ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
+        ('FONTSIZE', (0, 0), (-1, -1), 8),
+        ('GRID', (0, 0), (-1, -1), 0.5, colors.grey),
+        ('ROWBACKGROUNDS', (0, 1), (-1, -1), [WHITE, GRAY_LIGHT]),
+        ('PADDING', (0, 0), (-1, -1), 5),
+    ]))
+    story.append(infra_table)
+    story.append(Paragraph('Table 18: Infrastructure Requirements', styles['Caption']))
+
+    story.append(PageBreak())
+
+    # ========== 10. CONCLUSION ==========
+    story.append(Paragraph('10. Conclusion & Recommendations', styles['SectionHeader']))
+
+    story.append(Paragraph('<b>10.1 Summary</b>', styles['SubsectionHeader']))
+    story.append(Paragraph(
+        '''SPARKNET has achieved significant progress as a proof-of-concept for multi-agentic
+        document intelligence. The core RAG pipeline is functional, demonstrating the viability
+        of the 5-agent architecture with self-correction capabilities. The system successfully
+        processes documents, performs hybrid retrieval, and generates citation-backed responses.''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 10))
+    story.append(Paragraph('<b>10.2 Key Recommendations</b>', styles['SubsectionHeader']))
+
+    recommendations = [
+        '<b>Prioritize API Development:</b> Enable external integrations and unlock enterprise adoption.',
+        '<b>Invest in Security:</b> Authentication and authorization are prerequisites for any production deployment.',
+        '<b>Focus on Performance:</b> Current latency is acceptable for demos but needs significant improvement for production use.',
+        '<b>Expand Document Support:</b> Office formats (Word, Excel, PowerPoint) are critical for enterprise adoption.',
+        '<b>Implement Monitoring:</b> Observability is essential for maintaining and scaling the system.',
+        '<b>Plan for Scale Early:</b> Architectural decisions made now will impact scalability; consider distributed architecture.',
+    ]
+
+    for rec in recommendations:
+        story.append(Paragraph(f'• {rec}', styles['BulletText']))
+
+    story.append(Spacer(1, 15))
+    story.append(Paragraph('<b>10.3 Immediate Next Steps</b>', styles['SubsectionHeader']))
+
+    next_steps = [
+        '1. Finalize Phase 1 scope and create detailed sprint plans',
+        '2. Set up development infrastructure and CI/CD pipeline',
+        '3. Begin REST API development (target: 4 weeks)',
+        '4. Initiate security assessment and authentication design',
+        '5. Start documentation and knowledge transfer activities',
+        '6. Schedule bi-weekly stakeholder demos for continuous feedback',
+    ]
+
+    for step in next_steps:
+        story.append(Paragraph(step, styles['BulletText']))
+
+    story.append(Spacer(1, 30))
+
+    # Final signature block
+    story.append(HRFlowable(width='100%', thickness=1, color=PRIMARY_BLUE))
+    story.append(Spacer(1, 15))
+
+    story.append(Paragraph(
+        f'''<b>Document prepared by:</b> SPARKNET Development Team<br/>
+        <b>Report Date:</b> {datetime.now().strftime('%B %d, %Y')}<br/>
+        <b>Version:</b> 1.0<br/>
+        <b>Classification:</b> Internal / Confidential''',
+        styles['CustomBody']
+    ))
+
+    story.append(Spacer(1, 20))
+    story.append(Paragraph(
+        '<i>This document contains confidential information intended for stakeholder review. '
+        'Please do not distribute without authorization.</i>',
+        styles['Caption']
+    ))
+
+    # Build PDF
+    doc.build(story)
+    print(f"Report generated: {filename}")
+    return filename
+
+
+if __name__ == '__main__':
+    generate_report()

examples/document_agent.py

@@ -0,0 +1,240 @@
+"""
+Example: DocumentAgent with ReAct-style Processing
+
+Demonstrates:
+1. Loading and processing documents
+2. Field extraction with evidence
+3. Document classification
+4. Question answering with grounding
+"""
+
+import asyncio
+from pathlib import Path
+from loguru import logger
+
+# Import DocumentAgent
+from src.agents.document_agent import (
+    DocumentAgent,
+    AgentConfig,
+)
+from src.document.schemas.extraction import (
+    ExtractionSchema,
+    FieldDefinition,
+)
+
+
+async def example_basic_agent():
+    """Basic agent usage."""
+    print("=" * 50)
+    print("Basic DocumentAgent Usage")
+    print("=" * 50)
+
+    # Create agent with custom config
+    config = AgentConfig(
+        default_model="llama3.2:3b",
+        max_iterations=10,
+        temperature=0.1,
+    )
+    agent = DocumentAgent(config)
+
+    # Load document
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print(f"Sample document not found: {sample_doc}")
+        print("Create a sample PDF at ./data/sample.pdf")
+        return
+
+    print(f"\nLoading document: {sample_doc}")
+    await agent.load_document(str(sample_doc))
+
+    print(f"Document loaded: {agent.document.metadata.filename}")
+    print(f"Pages: {agent.document.metadata.num_pages}")
+    print(f"Chunks: {len(agent.document.chunks)}")
+
+
+async def example_field_extraction():
+    """Extract structured fields with evidence."""
+    print("\n" + "=" * 50)
+    print("Field Extraction with Evidence")
+    print("=" * 50)
+
+    agent = DocumentAgent()
+
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print("Sample document not found")
+        return
+
+    await agent.load_document(str(sample_doc))
+
+    # Define extraction schema
+    schema = ExtractionSchema(
+        name="document_info",
+        description="Extract key document information",
+        fields=[
+            FieldDefinition(
+                name="title",
+                field_type="string",
+                description="Document title",
+                required=True,
+            ),
+            FieldDefinition(
+                name="author",
+                field_type="string",
+                description="Document author or organization",
+                required=False,
+            ),
+            FieldDefinition(
+                name="date",
+                field_type="string",
+                description="Document date",
+                required=False,
+            ),
+            FieldDefinition(
+                name="summary",
+                field_type="string",
+                description="Brief summary of document content",
+                required=True,
+            ),
+        ],
+    )
+
+    # Extract fields
+    print("\nExtracting fields...")
+    result = await agent.extract_fields(schema)
+
+    print(f"\nExtracted Fields:")
+    for field, value in result.fields.items():
+        print(f"  {field}: {value}")
+
+    print(f"\nConfidence: {result.confidence:.2f}")
+
+    if result.evidence:
+        print(f"\nEvidence ({len(result.evidence)} sources):")
+        for ev in result.evidence[:3]:
+            print(f"  - Page {ev.page + 1}: {ev.snippet[:80]}...")
+
+
+async def example_classification():
+    """Classify document type."""
+    print("\n" + "=" * 50)
+    print("Document Classification")
+    print("=" * 50)
+
+    agent = DocumentAgent()
+
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print("Sample document not found")
+        return
+
+    await agent.load_document(str(sample_doc))
+
+    # Classify
+    print("\nClassifying document...")
+    classification = await agent.classify()
+
+    print(f"\nDocument Type: {classification.document_type.value}")
+    print(f"Confidence: {classification.confidence:.2f}")
+    print(f"Reasoning: {classification.reasoning}")
+
+    if classification.metadata:
+        print(f"\nAdditional metadata:")
+        for key, value in classification.metadata.items():
+            print(f"  {key}: {value}")
+
+
+async def example_question_answering():
+    """Answer questions about document with evidence."""
+    print("\n" + "=" * 50)
+    print("Question Answering with Evidence")
+    print("=" * 50)
+
+    agent = DocumentAgent()
+
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print("Sample document not found")
+        return
+
+    await agent.load_document(str(sample_doc))
+
+    # Questions to ask
+    questions = [
+        "What is this document about?",
+        "What are the main findings or conclusions?",
+        "Are there any tables or figures? What do they show?",
+    ]
+
+    for question in questions:
+        print(f"\nQ: {question}")
+        print("-" * 40)
+
+        answer, evidence = await agent.answer_question(question)
+
+        print(f"A: {answer}")
+
+        if evidence:
+            print(f"\nEvidence:")
+            for ev in evidence[:2]:
+                print(f"  - Page {ev.page + 1} ({ev.source_type}): {ev.snippet[:60]}...")
+
+
+async def example_react_task():
+    """Run a complex task with ReAct-style reasoning."""
+    print("\n" + "=" * 50)
+    print("ReAct-style Task Execution")
+    print("=" * 50)
+
+    agent = DocumentAgent()
+
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print("Sample document not found")
+        return
+
+    await agent.load_document(str(sample_doc))
+
+    # Complex task
+    task = """
+    Analyze this document and provide:
+    1. A brief summary of the content
+    2. The document type and purpose
+    3. Any key data points or figures mentioned
+    4. Your confidence in the analysis
+    """
+
+    print(f"\nTask: {task}")
+    print("-" * 40)
+
+    # Run with trace
+    result, trace = await agent.run(task)
+
+    print(f"\nResult:\n{result}")
+
+    print(f"\n--- Agent Trace ---")
+    print(f"Steps: {len(trace.steps)}")
+    print(f"Tools used: {trace.tools_used}")
+    print(f"Total time: {trace.total_time:.2f}s")
+
+    # Show thinking process
+    print(f"\nReasoning trace:")
+    for i, step in enumerate(trace.steps[:5], 1):
+        print(f"\n[Step {i}] {step.action}")
+        if step.thought:
+            print(f"  Thought: {step.thought[:100]}...")
+        if step.observation:
+            print(f"  Observation: {step.observation[:100]}...")
+
+
+async def main():
+    """Run all examples."""
+    await example_basic_agent()
+    await example_field_extraction()
+    await example_classification()
+    await example_question_answering()
+    await example_react_task()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/document_intelligence_demo.py

@@ -0,0 +1,314 @@
+#!/usr/bin/env python3
+"""
+Document Intelligence Demo
+
+Demonstrates the capabilities of the SPARKNET document_intelligence subsystem:
+- Document parsing with OCR and layout detection
+- Schema-driven field extraction
+- Visual grounding with evidence
+- Question answering
+- Document classification
+"""
+
+import asyncio
+import json
+from pathlib import Path
+
+# Add project root to path
+import sys
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+
+def demo_parse_document(doc_path: str):
+    """Demo: Parse a document into semantic chunks."""
+    print("\n" + "=" * 60)
+    print("1. DOCUMENT PARSING")
+    print("=" * 60)
+
+    from src.document_intelligence import (
+        DocumentParser,
+        ParserConfig,
+    )
+
+    # Configure parser
+    config = ParserConfig(
+        render_dpi=200,
+        max_pages=5,  # Limit for demo
+        include_markdown=True,
+    )
+
+    parser = DocumentParser(config=config)
+
+    print(f"\nParsing: {doc_path}")
+    result = parser.parse(doc_path)
+
+    print(f"\nDocument ID: {result.doc_id}")
+    print(f"Filename: {result.filename}")
+    print(f"Pages: {result.num_pages}")
+    print(f"Chunks: {len(result.chunks)}")
+    print(f"Processing time: {result.processing_time_ms:.0f}ms")
+
+    # Show chunk summary by type
+    print("\nChunk types:")
+    by_type = {}
+    for chunk in result.chunks:
+        t = chunk.chunk_type.value
+        by_type[t] = by_type.get(t, 0) + 1
+
+    for t, count in sorted(by_type.items()):
+        print(f"  - {t}: {count}")
+
+    # Show first few chunks
+    print("\nFirst 3 chunks:")
+    for i, chunk in enumerate(result.chunks[:3]):
+        print(f"\n  [{i+1}] Type: {chunk.chunk_type.value}, Page: {chunk.page}")
+        print(f"      ID: {chunk.chunk_id}")
+        print(f"      Text: {chunk.text[:100]}...")
+        print(f"      BBox: {chunk.bbox.xyxy}")
+        print(f"      Confidence: {chunk.confidence:.2f}")
+
+    return result
+
+
+def demo_extract_fields(parse_result):
+    """Demo: Extract fields using a schema."""
+    print("\n" + "=" * 60)
+    print("2. SCHEMA-DRIVEN EXTRACTION")
+    print("=" * 60)
+
+    from src.document_intelligence import (
+        FieldExtractor,
+        ExtractionSchema,
+        FieldType,
+        ExtractionValidator,
+    )
+
+    # Create a custom schema
+    schema = ExtractionSchema(
+        name="DocumentInfo",
+        description="Basic document information",
+    )
+
+    schema.add_string_field("title", "Document title or heading", required=True)
+    schema.add_string_field("date", "Document date", required=False)
+    schema.add_string_field("author", "Author or organization name", required=False)
+    schema.add_string_field("reference_number", "Reference or ID number", required=False)
+
+    print(f"\nExtraction schema: {schema.name}")
+    print("Fields:")
+    for field in schema.fields:
+        req = "required" if field.required else "optional"
+        print(f"  - {field.name} ({field.field_type.value}, {req})")
+
+    # Extract fields
+    extractor = FieldExtractor()
+    result = extractor.extract(parse_result, schema)
+
+    print("\nExtracted data:")
+    for key, value in result.data.items():
+        status = " [ABSTAINED]" if key in result.abstained_fields else ""
+        print(f"  {key}: {value}{status}")
+
+    print(f"\nOverall confidence: {result.overall_confidence:.2f}")
+
+    # Show evidence
+    if result.evidence:
+        print("\nEvidence:")
+        for ev in result.evidence[:3]:
+            print(f"  - Page {ev.page}, Chunk {ev.chunk_id[:12]}...")
+            print(f"    Snippet: {ev.snippet[:80]}...")
+
+    # Validate
+    validator = ExtractionValidator()
+    validation = validator.validate(result, schema)
+
+    print(f"\nValidation: {'PASSED' if validation.is_valid else 'FAILED'}")
+    if validation.issues:
+        print("Issues:")
+        for issue in validation.issues[:3]:
+            print(f"  - [{issue.severity}] {issue.field_name}: {issue.message}")
+
+    return result
+
+
+def demo_search_and_qa(parse_result):
+    """Demo: Search and question answering."""
+    print("\n" + "=" * 60)
+    print("3. SEARCH AND Q&A")
+    print("=" * 60)
+
+    from src.document_intelligence.tools import get_tool
+
+    # Search demo
+    print("\nSearching for 'document'...")
+    search_tool = get_tool("search_chunks")
+    search_result = search_tool.execute(
+        parse_result=parse_result,
+        query="document",
+        top_k=5,
+    )
+
+    if search_result.success:
+        matches = search_result.data.get("results", [])
+        print(f"Found {len(matches)} matches:")
+        for i, match in enumerate(matches[:3], 1):
+            print(f"  {i}. Page {match['page']}, Type: {match['type']}")
+            print(f"     Score: {match['score']:.2f}")
+            print(f"     Text: {match['text'][:80]}...")
+
+    # Q&A demo
+    print("\nAsking: 'What is this document about?'")
+    qa_tool = get_tool("answer_question")
+    qa_result = qa_tool.execute(
+        parse_result=parse_result,
+        question="What is this document about?",
+    )
+
+    if qa_result.success:
+        print(f"Answer: {qa_result.data.get('answer', 'No answer')}")
+        print(f"Confidence: {qa_result.data.get('confidence', 0):.2f}")
+
+
+def demo_grounding(parse_result, doc_path: str):
+    """Demo: Visual grounding with crops."""
+    print("\n" + "=" * 60)
+    print("4. VISUAL GROUNDING")
+    print("=" * 60)
+
+    from src.document_intelligence import (
+        load_document,
+        RenderOptions,
+    )
+    from src.document_intelligence.grounding import (
+        EvidenceBuilder,
+        crop_region,
+        create_annotated_image,
+    )
+
+    # Load page image
+    loader, renderer = load_document(doc_path)
+    page_image = renderer.render_page(1, RenderOptions(dpi=200))
+    loader.close()
+
+    print(f"\nPage 1 image size: {page_image.shape}")
+
+    # Get chunks from page 1
+    page_chunks = [c for c in parse_result.chunks if c.page == 1]
+    print(f"Page 1 chunks: {len(page_chunks)}")
+
+    # Create evidence for first chunk
+    if page_chunks:
+        chunk = page_chunks[0]
+        evidence_builder = EvidenceBuilder()
+
+        evidence = evidence_builder.create_evidence(
+            chunk=chunk,
+            value=chunk.text[:50],
+            field_name="example_field",
+        )
+
+        print(f"\nEvidence created:")
+        print(f"  Chunk ID: {evidence.chunk_id}")
+        print(f"  Page: {evidence.page}")
+        print(f"  BBox: {evidence.bbox.xyxy}")
+        print(f"  Snippet: {evidence.snippet[:80]}...")
+
+        # Crop region
+        crop = crop_region(page_image, chunk.bbox)
+        print(f"  Crop size: {crop.shape}")
+
+    # Create annotated image (preview)
+    print("\nAnnotated image would include bounding boxes for all chunks.")
+    print("Use the CLI 'sparknet docint visualize' command to generate.")
+
+
+def demo_classification(parse_result):
+    """Demo: Document classification."""
+    print("\n" + "=" * 60)
+    print("5. DOCUMENT CLASSIFICATION")
+    print("=" * 60)
+
+    from src.document_intelligence.chunks import DocumentType
+
+    # Simple keyword-based classification
+    first_page = [c for c in parse_result.chunks if c.page == 1][:5]
+    content = " ".join(c.text for c in first_page).lower()
+
+    type_keywords = {
+        "invoice": ["invoice", "bill", "payment due", "amount due"],
+        "contract": ["agreement", "contract", "party", "whereas"],
+        "receipt": ["receipt", "paid", "transaction"],
+        "patent": ["patent", "claims", "invention"],
+        "report": ["report", "findings", "summary"],
+    }
+
+    detected_type = "other"
+    confidence = 0.3
+
+    for doc_type, keywords in type_keywords.items():
+        matches = sum(1 for k in keywords if k in content)
+        if matches >= 2:
+            detected_type = doc_type
+            confidence = min(0.95, 0.5 + matches * 0.15)
+            break
+
+    print(f"\nDetected type: {detected_type}")
+    print(f"Confidence: {confidence:.2f}")
+
+
+def main():
+    """Run all demos."""
+    print("=" * 60)
+    print("SPARKNET Document Intelligence Demo")
+    print("=" * 60)
+
+    # Check for sample document
+    sample_paths = [
+        Path("Dataset/Patent_1.pdf"),
+        Path("data/sample.pdf"),
+        Path("tests/fixtures/sample.pdf"),
+    ]
+
+    doc_path = None
+    for path in sample_paths:
+        if path.exists():
+            doc_path = str(path)
+            break
+
+    if not doc_path:
+        print("\nNo sample document found.")
+        print("Please provide a PDF file path as argument.")
+        print("\nUsage: python document_intelligence_demo.py [path/to/document.pdf]")
+
+        if len(sys.argv) > 1:
+            doc_path = sys.argv[1]
+        else:
+            return
+
+    print(f"\nUsing document: {doc_path}")
+
+    try:
+        # Run demos
+        parse_result = demo_parse_document(doc_path)
+        demo_extract_fields(parse_result)
+        demo_search_and_qa(parse_result)
+        demo_grounding(parse_result, doc_path)
+        demo_classification(parse_result)
+
+        print("\n" + "=" * 60)
+        print("Demo complete!")
+        print("=" * 60)
+
+    except ImportError as e:
+        print(f"\nImport error: {e}")
+        print("Make sure all dependencies are installed:")
+        print("  pip install pymupdf pillow numpy pydantic")
+
+    except Exception as e:
+        print(f"\nError: {e}")
+        import traceback
+        traceback.print_exc()
+
+
+if __name__ == "__main__":
+    main()

examples/document_processing.py

@@ -0,0 +1,133 @@
+"""
+Example: Document Processing Pipeline
+
+Demonstrates:
+1. Processing a PDF document
+2. Extracting text with OCR
+3. Layout detection
+4. Semantic chunking
+"""
+
+import asyncio
+from pathlib import Path
+from loguru import logger
+
+# Import document processing components
+from src.document.pipeline import (
+    PipelineConfig,
+    DocumentProcessor,
+    process_document,
+)
+from src.document.ocr import OCRConfig
+
+
+def example_basic_processing():
+    """Basic document processing example."""
+    print("=" * 50)
+    print("Basic Document Processing")
+    print("=" * 50)
+
+    # Configure pipeline
+    config = PipelineConfig(
+        ocr=OCRConfig(engine="paddleocr"),
+        render_dpi=300,
+        max_pages=5,  # Limit for demo
+    )
+
+    # Create processor
+    processor = DocumentProcessor(config)
+
+    # Process a sample document
+    # NOTE: Replace with actual document path
+    sample_doc = Path("./data/sample.pdf")
+
+    if not sample_doc.exists():
+        print(f"Sample document not found: {sample_doc}")
+        print("Create a sample PDF at ./data/sample.pdf to run this example")
+        return
+
+    # Process
+    result = processor.process(sample_doc)
+
+    # Display results
+    print(f"\nDocument: {result.metadata.filename}")
+    print(f"Pages: {result.metadata.num_pages}")
+    print(f"Chunks: {result.metadata.total_chunks}")
+    print(f"Characters: {result.metadata.total_characters}")
+    print(f"OCR Confidence: {result.metadata.ocr_confidence_avg:.2%}")
+
+    print("\n--- Sample Chunks ---")
+    for i, chunk in enumerate(result.chunks[:3]):
+        print(f"\n[Chunk {i+1}] Type: {chunk.chunk_type.value}, Page: {chunk.page+1}")
+        print(f"Text: {chunk.text[:200]}...")
+        print(f"BBox: ({chunk.bbox.x_min:.0f}, {chunk.bbox.y_min:.0f}) -> ({chunk.bbox.x_max:.0f}, {chunk.bbox.y_max:.0f})")
+
+
+def example_with_layout():
+    """Document processing with layout analysis."""
+    print("\n" + "=" * 50)
+    print("Document Processing with Layout Analysis")
+    print("=" * 50)
+
+    from src.document.layout import LayoutConfig, LayoutType
+
+    # Configure with layout detection
+    config = PipelineConfig(
+        ocr=OCRConfig(engine="paddleocr"),
+        layout=LayoutConfig(method="rule_based"),
+        include_layout_regions=True,
+    )
+
+    processor = DocumentProcessor(config)
+
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print("Sample document not found")
+        return
+
+    result = processor.process(sample_doc)
+
+    # Count layout types
+    layout_counts = {}
+    for region in result.layout_regions:
+        layout_type = region.layout_type.value
+        layout_counts[layout_type] = layout_counts.get(layout_type, 0) + 1
+
+    print(f"\nLayout Analysis:")
+    for layout_type, count in sorted(layout_counts.items()):
+        print(f"  {layout_type}: {count} regions")
+
+    # Show tables if found
+    tables = [r for r in result.layout_regions if r.layout_type == LayoutType.TABLE]
+    if tables:
+        print(f"\n--- Tables Found ({len(tables)}) ---")
+        for i, table in enumerate(tables[:2]):
+            print(f"\nTable {i+1}: Page {table.page+1}")
+            print(f"  Position: ({table.bbox.x_min:.0f}, {table.bbox.y_min:.0f})")
+            print(f"  Size: {table.bbox.width:.0f} x {table.bbox.height:.0f}")
+
+
+def example_convenience_function():
+    """Using the convenience function."""
+    print("\n" + "=" * 50)
+    print("Using Convenience Function")
+    print("=" * 50)
+
+    sample_doc = Path("./data/sample.pdf")
+    if not sample_doc.exists():
+        print("Sample document not found")
+        return
+
+    # Simple one-liner
+    result = process_document(sample_doc)
+
+    print(f"Processed: {result.metadata.filename}")
+    print(f"Chunks: {len(result.chunks)}")
+    print(f"\nFull text preview:")
+    print(result.full_text[:500] + "..." if len(result.full_text) > 500 else result.full_text)
+
+
+if __name__ == "__main__":
+    example_basic_processing()
+    example_with_layout()
+    example_convenience_function()

examples/document_rag_end_to_end.py

@@ -0,0 +1,359 @@
+#!/usr/bin/env python3
+"""
+Document Intelligence RAG End-to-End Example
+
+Demonstrates the complete RAG workflow:
+1. Parse documents into semantic chunks
+2. Index chunks into vector store
+3. Semantic retrieval with filters
+4. Grounded question answering with evidence
+5. Evidence visualization
+
+Requirements:
+- ChromaDB: pip install chromadb
+- Ollama running with nomic-embed-text model: ollama pull nomic-embed-text
+- PyMuPDF: pip install pymupdf
+"""
+
+import sys
+from pathlib import Path
+
+# Add project root to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+
+def check_dependencies():
+    """Check that required dependencies are available."""
+    missing = []
+
+    try:
+        import chromadb
+    except ImportError:
+        missing.append("chromadb")
+
+    try:
+        import fitz  # PyMuPDF
+    except ImportError:
+        missing.append("pymupdf")
+
+    if missing:
+        print("Missing dependencies:")
+        for dep in missing:
+            print(f"  - {dep}")
+        print("\nInstall with: pip install " + " ".join(missing))
+        return False
+
+    # Check Ollama
+    try:
+        import requests
+        response = requests.get("http://localhost:11434/api/tags", timeout=2)
+        if response.status_code != 200:
+            print("Warning: Ollama server not responding")
+            print("Start Ollama with: ollama serve")
+            print("Then pull the embedding model: ollama pull nomic-embed-text")
+    except:
+        print("Warning: Could not connect to Ollama server")
+        print("The example will still work but with mock embeddings")
+
+    return True
+
+
+def demo_parse_and_index(doc_paths: list):
+    """
+    Demo: Parse documents and index into vector store.
+
+    Args:
+        doc_paths: List of document file paths
+    """
+    print("\n" + "=" * 60)
+    print("STEP 1: PARSE AND INDEX DOCUMENTS")
+    print("=" * 60)
+
+    from src.document_intelligence import DocumentParser, ParserConfig
+    from src.document_intelligence.tools import get_rag_tool
+
+    # Get the index tool
+    index_tool = get_rag_tool("index_document")
+
+    results = []
+    for doc_path in doc_paths:
+        print(f"\nProcessing: {doc_path}")
+
+        # Parse document first (optional - tool can do this)
+        config = ParserConfig(render_dpi=200, max_pages=10)
+        parser = DocumentParser(config=config)
+
+        try:
+            parse_result = parser.parse(doc_path)
+            print(f"  Parsed: {len(parse_result.chunks)} chunks, {parse_result.num_pages} pages")
+
+            # Index the parse result
+            result = index_tool.execute(parse_result=parse_result)
+
+            if result.success:
+                print(f"  Indexed: {result.data['chunks_indexed']} chunks")
+                print(f"  Document ID: {result.data['document_id']}")
+                results.append({
+                    "path": doc_path,
+                    "doc_id": result.data['document_id'],
+                    "chunks": result.data['chunks_indexed'],
+                })
+            else:
+                print(f"  Error: {result.error}")
+
+        except Exception as e:
+            print(f"  Failed: {e}")
+
+    return results
+
+
+def demo_semantic_retrieval(query: str, document_id: str = None):
+    """
+    Demo: Semantic retrieval from vector store.
+
+    Args:
+        query: Search query
+        document_id: Optional document filter
+    """
+    print("\n" + "=" * 60)
+    print("STEP 2: SEMANTIC RETRIEVAL")
+    print("=" * 60)
+
+    from src.document_intelligence.tools import get_rag_tool
+
+    retrieve_tool = get_rag_tool("retrieve_chunks")
+
+    print(f"\nQuery: \"{query}\"")
+    if document_id:
+        print(f"Document filter: {document_id}")
+
+    result = retrieve_tool.execute(
+        query=query,
+        top_k=5,
+        document_id=document_id,
+        include_evidence=True,
+    )
+
+    if result.success:
+        chunks = result.data.get("chunks", [])
+        print(f"\nFound {len(chunks)} relevant chunks:\n")
+
+        for i, chunk in enumerate(chunks, 1):
+            print(f"{i}. [similarity={chunk['similarity']:.3f}]")
+            print(f"   Page {chunk.get('page', '?')}, Type: {chunk.get('chunk_type', 'unknown')}")
+            print(f"   Text: {chunk['text'][:150]}...")
+            print()
+
+        # Show evidence
+        if result.evidence:
+            print("Evidence references:")
+            for ev in result.evidence[:3]:
+                print(f"  - Chunk {ev['chunk_id'][:12]}... Page {ev.get('page', '?')}")
+
+        return chunks
+    else:
+        print(f"Error: {result.error}")
+        return []
+
+
+def demo_grounded_qa(question: str, document_id: str = None):
+    """
+    Demo: Grounded question answering with evidence.
+
+    Args:
+        question: Question to answer
+        document_id: Optional document filter
+    """
+    print("\n" + "=" * 60)
+    print("STEP 3: GROUNDED QUESTION ANSWERING")
+    print("=" * 60)
+
+    from src.document_intelligence.tools import get_rag_tool
+
+    qa_tool = get_rag_tool("rag_answer")
+
+    print(f"\nQuestion: \"{question}\"")
+
+    result = qa_tool.execute(
+        question=question,
+        document_id=document_id,
+        top_k=5,
+    )
+
+    if result.success:
+        data = result.data
+        print(f"\nAnswer: {data.get('answer', 'No answer')}")
+        print(f"Confidence: {data.get('confidence', 0):.2f}")
+
+        if data.get('abstained'):
+            print("Note: System abstained due to low confidence")
+
+        # Show citations if any
+        citations = data.get('citations', [])
+        if citations:
+            print("\nCitations:")
+            for cit in citations:
+                print(f"  [{cit['index']}] {cit.get('text', '')[:80]}...")
+
+        # Show evidence
+        if result.evidence:
+            print("\nEvidence locations:")
+            for ev in result.evidence:
+                print(f"  - Page {ev.get('page', '?')}: {ev.get('snippet', '')[:60]}...")
+
+        return data
+    else:
+        print(f"Error: {result.error}")
+        return None
+
+
+def demo_filtered_retrieval():
+    """
+    Demo: Retrieval with various filters.
+    """
+    print("\n" + "=" * 60)
+    print("STEP 4: FILTERED RETRIEVAL")
+    print("=" * 60)
+
+    from src.document_intelligence.tools import get_rag_tool
+
+    retrieve_tool = get_rag_tool("retrieve_chunks")
+
+    # Filter by chunk type
+    print("\n--- Retrieving only table chunks ---")
+    result = retrieve_tool.execute(
+        query="data values",
+        top_k=3,
+        chunk_types=["table"],
+    )
+
+    if result.success:
+        chunks = result.data.get("chunks", [])
+        print(f"Found {len(chunks)} table chunks")
+        for chunk in chunks:
+            print(f"  - Page {chunk.get('page', '?')}: {chunk['text'][:80]}...")
+
+    # Filter by page range
+    print("\n--- Retrieving from pages 1-3 only ---")
+    result = retrieve_tool.execute(
+        query="content",
+        top_k=3,
+        page_range=(1, 3),
+    )
+
+    if result.success:
+        chunks = result.data.get("chunks", [])
+        print(f"Found {len(chunks)} chunks from pages 1-3")
+        for chunk in chunks:
+            print(f"  - Page {chunk.get('page', '?')}: {chunk['text'][:80]}...")
+
+
+def demo_index_stats():
+    """
+    Demo: Show index statistics.
+    """
+    print("\n" + "=" * 60)
+    print("INDEX STATISTICS")
+    print("=" * 60)
+
+    from src.document_intelligence.tools import get_rag_tool
+
+    stats_tool = get_rag_tool("get_index_stats")
+    result = stats_tool.execute()
+
+    if result.success:
+        data = result.data
+        print(f"\nTotal chunks indexed: {data.get('total_chunks', 0)}")
+        print(f"Embedding model: {data.get('embedding_model', 'unknown')}")
+        print(f"Embedding dimension: {data.get('embedding_dimension', 'unknown')}")
+    else:
+        print(f"Error: {result.error}")
+
+
+def main():
+    """Run the complete RAG demo."""
+    print("=" * 60)
+    print("SPARKNET Document Intelligence RAG Demo")
+    print("=" * 60)
+
+    # Check dependencies
+    if not check_dependencies():
+        print("\nPlease install missing dependencies and try again.")
+        return
+
+    # Find sample documents
+    sample_paths = [
+        Path("Dataset/Patent_1.pdf"),
+        Path("data/sample.pdf"),
+        Path("tests/fixtures/sample.pdf"),
+    ]
+
+    doc_paths = []
+    for path in sample_paths:
+        if path.exists():
+            doc_paths.append(str(path))
+            break
+
+    if not doc_paths:
+        print("\nNo sample documents found.")
+        print("Please provide a PDF file path as argument.")
+        print("\nUsage: python document_rag_end_to_end.py [path/to/document.pdf]")
+
+        if len(sys.argv) > 1:
+            doc_paths = sys.argv[1:]
+        else:
+            return
+
+    print(f"\nUsing documents: {doc_paths}")
+
+    try:
+        # Step 1: Parse and index
+        indexed_docs = demo_parse_and_index(doc_paths)
+
+        if not indexed_docs:
+            print("\nNo documents were indexed. Exiting.")
+            return
+
+        # Get first document ID for filtering
+        first_doc_id = indexed_docs[0]["doc_id"]
+
+        # Step 2: Semantic retrieval
+        demo_semantic_retrieval(
+            query="main topic content",
+            document_id=first_doc_id,
+        )
+
+        # Step 3: Grounded Q&A
+        demo_grounded_qa(
+            question="What is this document about?",
+            document_id=first_doc_id,
+        )
+
+        # Step 4: Filtered retrieval
+        demo_filtered_retrieval()
+
+        # Show stats
+        demo_index_stats()
+
+        print("\n" + "=" * 60)
+        print("Demo complete!")
+        print("=" * 60)
+
+        print("\nNext steps:")
+        print("  1. Try the CLI: sparknet docint index your_document.pdf")
+        print("  2. Query the index: sparknet docint retrieve 'your query'")
+        print("  3. Ask questions: sparknet docint ask doc.pdf 'question' --use-rag")
+
+    except ImportError as e:
+        print(f"\nImport error: {e}")
+        print("Make sure all dependencies are installed:")
+        print("  pip install pymupdf pillow numpy pydantic chromadb")
+
+    except Exception as e:
+        print(f"\nError: {e}")
+        import traceback
+        traceback.print_exc()
+
+
+if __name__ == "__main__":
+    main()

examples/rag_pipeline.py

@@ -0,0 +1,192 @@
+"""
+Example: RAG Pipeline
+
+Demonstrates:
+1. Indexing documents into vector store
+2. Semantic search
+3. Question answering with citations
+"""
+
+from pathlib import Path
+from loguru import logger
+
+# Import RAG components
+from src.rag import (
+    VectorStoreConfig,
+    EmbeddingConfig,
+    RetrieverConfig,
+    GeneratorConfig,
+    get_document_indexer,
+    get_document_retriever,
+    get_grounded_generator,
+)
+
+
+def example_indexing():
+    """Index documents into vector store."""
+    print("=" * 50)
+    print("Document Indexing")
+    print("=" * 50)
+
+    # Get indexer
+    indexer = get_document_indexer()
+
+    # Index a document
+    sample_doc = Path("./data/sample.pdf")
+
+    if not sample_doc.exists():
+        print(f"Sample document not found: {sample_doc}")
+        print("Create a sample PDF at ./data/sample.pdf")
+        return False
+
+    # Index
+    result = indexer.index_document(sample_doc)
+
+    if result.success:
+        print(f"\nIndexed: {result.source_path}")
+        print(f"  Document ID: {result.document_id}")
+        print(f"  Chunks indexed: {result.num_chunks_indexed}")
+        print(f"  Chunks skipped: {result.num_chunks_skipped}")
+    else:
+        print(f"Indexing failed: {result.error}")
+        return False
+
+    # Show stats
+    stats = indexer.get_index_stats()
+    print(f"\nIndex Stats:")
+    print(f"  Total chunks: {stats['total_chunks']}")
+    print(f"  Documents: {stats['num_documents']}")
+    print(f"  Embedding model: {stats['embedding_model']}")
+
+    return True
+
+
+def example_search():
+    """Search indexed documents."""
+    print("\n" + "=" * 50)
+    print("Semantic Search")
+    print("=" * 50)
+
+    # Get retriever
+    retriever = get_document_retriever()
+
+    # Search queries
+    queries = [
+        "What is the main topic?",
+        "key findings",
+        "conclusions and recommendations",
+    ]
+
+    for query in queries:
+        print(f"\nQuery: '{query}'")
+
+        chunks = retriever.retrieve(query, top_k=3)
+
+        if not chunks:
+            print("  No results found")
+            continue
+
+        for i, chunk in enumerate(chunks, 1):
+            print(f"\n  [{i}] Similarity: {chunk.similarity:.3f}")
+            if chunk.page is not None:
+                print(f"      Page: {chunk.page + 1}")
+            print(f"      Text: {chunk.text[:150]}...")
+
+
+def example_question_answering():
+    """Answer questions using RAG."""
+    print("\n" + "=" * 50)
+    print("Question Answering with Citations")
+    print("=" * 50)
+
+    # Get generator
+    generator = get_grounded_generator()
+
+    # Questions
+    questions = [
+        "What is the main purpose of this document?",
+        "What are the key findings?",
+        "What recommendations are made?",
+    ]
+
+    for question in questions:
+        print(f"\nQuestion: {question}")
+        print("-" * 40)
+
+        result = generator.answer_question(question, top_k=5)
+
+        print(f"\nAnswer: {result.answer}")
+        print(f"\nConfidence: {result.confidence:.2f}")
+
+        if result.abstained:
+            print(f"Note: {result.abstain_reason}")
+
+        if result.citations:
+            print(f"\nCitations ({len(result.citations)}):")
+            for citation in result.citations:
+                page = f"Page {citation.page + 1}" if citation.page is not None else ""
+                print(f"  [{citation.index}] {page}: {citation.text_snippet[:60]}...")
+
+
+def example_filtered_search():
+    """Search with metadata filters."""
+    print("\n" + "=" * 50)
+    print("Filtered Search")
+    print("=" * 50)
+
+    retriever = get_document_retriever()
+
+    # Search only in tables
+    print("\nSearching for tables only...")
+    table_chunks = retriever.retrieve_tables("data values", top_k=3)
+
+    if table_chunks:
+        print(f"Found {len(table_chunks)} table chunks:")
+        for chunk in table_chunks:
+            print(f"  - Page {chunk.page + 1}: {chunk.text[:100]}...")
+    else:
+        print("No table chunks found")
+
+    # Search specific page range
+    print("\nSearching pages 1-3...")
+    page_chunks = retriever.retrieve_by_page(
+        "introduction",
+        page_range=(0, 2),
+        top_k=3,
+    )
+
+    if page_chunks:
+        print(f"Found {len(page_chunks)} chunks in pages 1-3:")
+        for chunk in page_chunks:
+            print(f"  - Page {chunk.page + 1}: {chunk.text[:100]}...")
+    else:
+        print("No chunks found in specified pages")
+
+
+def example_full_pipeline():
+    """Complete RAG pipeline demo."""
+    print("\n" + "=" * 50)
+    print("Full RAG Pipeline Demo")
+    print("=" * 50)
+
+    # Step 1: Index
+    print("\n[Step 1] Indexing documents...")
+    if not example_indexing():
+        return
+
+    # Step 2: Search
+    print("\n[Step 2] Testing search...")
+    example_search()
+
+    # Step 3: Q&A
+    print("\n[Step 3] Question answering...")
+    example_question_answering()
+
+    print("\n" + "=" * 50)
+    print("Pipeline demo complete!")
+    print("=" * 50)
+
+
+if __name__ == "__main__":
+    # Run full pipeline
+    example_full_pipeline()

nginx/nginx.conf

@@ -0,0 +1,254 @@
+# SPARKNET Production Nginx Configuration
+# Reverse proxy for API and Demo services
+
+user nginx;
+worker_processes auto;
+error_log /var/log/nginx/error.log warn;
+pid /var/run/nginx.pid;
+
+events {
+    worker_connections 1024;
+    use epoll;
+    multi_accept on;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+
+    # Logging format
+    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+                    '$status $body_bytes_sent "$http_referer" '
+                    '"$http_user_agent" "$http_x_forwarded_for" '
+                    'rt=$request_time uct="$upstream_connect_time" '
+                    'uht="$upstream_header_time" urt="$upstream_response_time"';
+
+    access_log /var/log/nginx/access.log main;
+
+    # Performance optimizations
+    sendfile on;
+    tcp_nopush on;
+    tcp_nodelay on;
+    keepalive_timeout 65;
+    types_hash_max_size 2048;
+
+    # Gzip compression
+    gzip on;
+    gzip_vary on;
+    gzip_proxied any;
+    gzip_comp_level 6;
+    gzip_types text/plain text/css text/xml application/json application/javascript
+               application/xml application/xml+rss text/javascript application/x-javascript;
+    gzip_min_length 1000;
+
+    # Rate limiting zones
+    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=30r/s;
+    limit_req_zone $binary_remote_addr zone=upload_limit:10m rate=5r/s;
+    limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
+
+    # Security headers map
+    map $sent_http_content_type $security_headers {
+        default "always";
+    }
+
+    # Upstream servers
+    upstream sparknet_api {
+        server sparknet-api:8000;
+        keepalive 32;
+    }
+
+    upstream sparknet_demo {
+        server sparknet-demo:4000;
+        keepalive 32;
+    }
+
+    # HTTP redirect to HTTPS (uncomment for production with SSL)
+    # server {
+    #     listen 80;
+    #     listen [::]:80;
+    #     server_name _;
+    #     return 301 https://$host$request_uri;
+    # }
+
+    # Main HTTP server (development/internal)
+    server {
+        listen 80;
+        listen [::]:80;
+        server_name _;
+
+        # Connection limits
+        limit_conn conn_limit 20;
+
+        # Security headers
+        add_header X-Frame-Options "SAMEORIGIN" always;
+        add_header X-Content-Type-Options "nosniff" always;
+        add_header X-XSS-Protection "1; mode=block" always;
+        add_header Referrer-Policy "strict-origin-when-cross-origin" always;
+
+        # Client body size for file uploads
+        client_max_body_size 100M;
+        client_body_buffer_size 128k;
+        client_body_timeout 300s;
+
+        # Proxy timeouts
+        proxy_connect_timeout 60s;
+        proxy_send_timeout 300s;
+        proxy_read_timeout 300s;
+
+        # Health check endpoint (no rate limiting)
+        location /api/health {
+            proxy_pass http://sparknet_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+        }
+
+        # API endpoints
+        location /api/ {
+            # Rate limiting
+            limit_req zone=api_limit burst=50 nodelay;
+
+            proxy_pass http://sparknet_api;
+            proxy_http_version 1.1;
+
+            # Headers
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Connection "";
+
+            # CORS headers (if not handled by FastAPI)
+            # add_header Access-Control-Allow-Origin "*" always;
+            # add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" always;
+            # add_header Access-Control-Allow-Headers "Authorization, Content-Type" always;
+
+            # Handle OPTIONS for CORS preflight
+            if ($request_method = 'OPTIONS') {
+                add_header Access-Control-Allow-Origin "*";
+                add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS";
+                add_header Access-Control-Allow-Headers "Authorization, Content-Type";
+                add_header Access-Control-Max-Age 3600;
+                add_header Content-Length 0;
+                add_header Content-Type text/plain;
+                return 204;
+            }
+        }
+
+        # Document upload endpoint (lower rate limit)
+        location /api/documents/upload {
+            limit_req zone=upload_limit burst=10 nodelay;
+
+            proxy_pass http://sparknet_api;
+            proxy_http_version 1.1;
+
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+
+            # Increased timeout for large uploads
+            proxy_connect_timeout 120s;
+            proxy_send_timeout 600s;
+            proxy_read_timeout 600s;
+        }
+
+        # RAG streaming endpoint (SSE support)
+        location /api/rag/query/stream {
+            proxy_pass http://sparknet_api;
+            proxy_http_version 1.1;
+
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Connection "";
+
+            # SSE-specific settings
+            proxy_buffering off;
+            proxy_cache off;
+            chunked_transfer_encoding off;
+            proxy_read_timeout 3600s;
+        }
+
+        # Streamlit Demo (with WebSocket support)
+        location / {
+            proxy_pass http://sparknet_demo;
+            proxy_http_version 1.1;
+
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+
+            # WebSocket support for Streamlit
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+
+            # Streamlit specific
+            proxy_read_timeout 86400;
+        }
+
+        # Streamlit WebSocket endpoint
+        location /_stcore/stream {
+            proxy_pass http://sparknet_demo;
+            proxy_http_version 1.1;
+
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+            # WebSocket
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+
+            proxy_read_timeout 86400;
+            proxy_buffering off;
+        }
+
+        # Streamlit static files
+        location /static {
+            proxy_pass http://sparknet_demo;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+
+            # Cache static assets
+            expires 1d;
+            add_header Cache-Control "public, immutable";
+        }
+
+        # Error pages
+        error_page 502 503 504 /50x.html;
+        location = /50x.html {
+            root /usr/share/nginx/html;
+            internal;
+        }
+    }
+
+    # HTTPS server (uncomment and configure for production)
+    # server {
+    #     listen 443 ssl http2;
+    #     listen [::]:443 ssl http2;
+    #     server_name sparknet.example.com;
+    #
+    #     # SSL configuration
+    #     ssl_certificate /etc/nginx/ssl/fullchain.pem;
+    #     ssl_certificate_key /etc/nginx/ssl/privkey.pem;
+    #     ssl_session_timeout 1d;
+    #     ssl_session_cache shared:SSL:50m;
+    #     ssl_session_tickets off;
+    #
+    #     # Modern SSL configuration
+    #     ssl_protocols TLSv1.2 TLSv1.3;
+    #     ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
+    #     ssl_prefer_server_ciphers off;
+    #
+    #     # HSTS
+    #     add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+    #
+    #     # Include same location blocks as HTTP server above
+    #     # ...
+    # }
+}

run_demo.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""
+SPARKNET Demo Launcher
+
+Cross-platform launcher for the Streamlit demo.
+
+Usage:
+    python run_demo.py [--port PORT]
+"""
+
+import subprocess
+import sys
+import os
+from pathlib import Path
+
+def check_dependencies():
+    """Check and install required dependencies."""
+    print("📦 Checking dependencies...")
+
+    try:
+        import streamlit
+        print(f"   ✅ Streamlit {streamlit.__version__}")
+    except ImportError:
+        print("   📥 Installing Streamlit...")
+        subprocess.run([sys.executable, "-m", "pip", "install", "streamlit"], check=True)
+
+    try:
+        import pandas
+        print(f"   ✅ Pandas {pandas.__version__}")
+    except ImportError:
+        print("   📥 Installing Pandas...")
+        subprocess.run([sys.executable, "-m", "pip", "install", "pandas"], check=True)
+
+    try:
+        import httpx
+        print(f"   ✅ httpx {httpx.__version__}")
+    except ImportError:
+        print("   📥 Installing httpx...")
+        subprocess.run([sys.executable, "-m", "pip", "install", "httpx"], check=True)
+
+
+def check_ollama():
+    """Check if Ollama is running."""
+    print("\n🔍 Checking Ollama status...")
+
+    try:
+        import httpx
+        with httpx.Client(timeout=2.0) as client:
+            response = client.get("http://localhost:11434/api/tags")
+            if response.status_code == 200:
+                data = response.json()
+                models = len(data.get("models", []))
+                print(f"   ✅ Ollama is running ({models} models)")
+                return True
+    except Exception:
+        pass
+
+    print("   ⚠️  Ollama not running (demo will use simulated responses)")
+    print("       Start with: ollama serve")
+    return False
+
+
+def main():
+    """Main entry point."""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="SPARKNET Demo Launcher")
+    parser.add_argument("--port", type=int, default=8501, help="Port to run on")
+    args = parser.parse_args()
+
+    print("=" * 50)
+    print("🔥 SPARKNET Demo Launcher")
+    print("=" * 50)
+    print()
+
+    # Get project root
+    project_root = Path(__file__).parent
+    demo_app = project_root / "demo" / "app.py"
+
+    if not demo_app.exists():
+        print(f"❌ Demo app not found: {demo_app}")
+        sys.exit(1)
+
+    # Check dependencies
+    check_dependencies()
+
+    # Check Ollama
+    check_ollama()
+
+    # Launch
+    print()
+    print(f"🚀 Launching SPARKNET Demo on port {args.port}...")
+    print(f"   URL: http://localhost:{args.port}")
+    print()
+    print("Press Ctrl+C to stop")
+    print("=" * 50)
+    print()
+
+    # Run Streamlit
+    os.chdir(project_root)
+    subprocess.run([
+        sys.executable, "-m", "streamlit", "run",
+        str(demo_app),
+        "--server.port", str(args.port),
+        "--server.headless", "true",
+    ])
+
+
+if __name__ == "__main__":
+    main()

run_demo.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+# SPARKNET Demo Launcher
+# Usage: ./run_demo.sh [port]
+
+set -e
+
+PORT=${1:-8501}
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+echo "🔥 SPARKNET Demo Launcher"
+echo "========================="
+echo ""
+
+# Check Python
+if ! command -v python3 &> /dev/null; then
+    echo "❌ Python3 not found. Please install Python 3.10+"
+    exit 1
+fi
+
+# Check Streamlit
+if ! python3 -c "import streamlit" &> /dev/null; then
+    echo "📦 Installing Streamlit..."
+    pip install streamlit
+fi
+
+# Check demo dependencies
+echo "📦 Checking dependencies..."
+pip install -q -r "$SCRIPT_DIR/demo/requirements.txt" 2>/dev/null || true
+
+# Check Ollama status
+echo ""
+echo "🔍 Checking Ollama status..."
+if curl -s http://localhost:11434/api/tags > /dev/null 2>&1; then
+    echo "✅ Ollama is running"
+    MODELS=$(curl -s http://localhost:11434/api/tags | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('models', [])))" 2>/dev/null || echo "?")
+    echo "   Models available: $MODELS"
+else
+    echo "⚠️  Ollama not running (demo will use simulated responses)"
+    echo "   Start with: ollama serve"
+fi
+
+# Launch demo
+echo ""
+echo "🚀 Launching SPARKNET Demo on port $PORT..."
+echo "   URL: http://localhost:$PORT"
+echo ""
+echo "Press Ctrl+C to stop"
+echo "========================="
+echo ""
+
+cd "$SCRIPT_DIR"
+streamlit run demo/app.py --server.port "$PORT" --server.headless true

scripts to get ideas from/ides.txt

@@ -0,0 +1,151 @@
+This introduces the fundamentals of document processing and how they connect to agentic AI workflows.
+
+The core problem is that modern organizations are overwhelmed with digital documents such as PDFs, scans, receipts, contracts, and reports. These documents are designed for human reading, not machine processing, which makes searching, analyzing, and automating information extremely difficult. Valuable data is often trapped inside unstructured formats, requiring manual reading and re-entry, which does not scale.
+
+The goal of document processing is to convert unstructured documents into structured, machine-readable data. Common output formats include JSON and Markdown or HTML. JSON is well suited for machines, APIs, databases, and analytics pipelines because it is hierarchical and easy to process programmatically. Markdown or HTML preserves layout elements such as headings, tables, and lists, making it ideal for human readers and large language models, especially in chat interfaces and retrieval-augmented generation systems.
+
+When documents are scanned or photographed, the system only sees image pixels. Optical Character Recognition (OCR) is required to convert those pixels into text. OCR typically involves two main steps: image preprocessing, such as deskewing, denoising, and contrast adjustment, followed by text recognition, where visual patterns are matched to characters. The output is editable or searchable text.
+
+However, OCR has important limitations. It does not understand document structure, meaning, or relationships between elements. It often produces a flat block of text and struggles with poor image quality, complex layouts, multi-column text, nested tables, handwriting, stamps, and stylized fonts. These weaknesses can lead to cascading errors during parsing and extraction. OCR provides perception, but not comprehension.
+
+A key distinction introduced in this lesson is that processing is not the same as understanding. OCR can read characters but cannot determine what is a header, a value, a total amount, or a table entry. To move from raw text to meaningful structured data, an additional cognitive layer is required.
+
+Agentic AI provides this missing layer. In document processing, an agent is an autonomous system that can perceive input, reason about goals, decide which tools to use, and act iteratively until the task is complete. In this context, OCR functions as the eyes, while the agent serves as the brain. Unlike rigid rule-based pipelines, agents can adapt to edge cases and unexpected document variations.
+
+An agentic document system is typically composed of three components. The brain, implemented using a large language model, handles reasoning, planning, and decision-making. The eyes, implemented through OCR, convert visual content into text. The hands are the tools the agent can use, such as APIs, database queries, file operations, and function calls. Together, these components allow the system to answer high-level requests, such as identifying the total amount on an invoice, without hardcoding every step.
+
+The lesson also introduces the ReAct framework, which describes how agents reason step by step. The agent alternates between thinking about what to do next, taking an action by calling a tool, observing the result, and then repeating the process. This loop enables adaptability, error correction, and transparency, since the agent’s reasoning and tool usage can be inspected.
+
+The lesson concludes with a practical lab. Learners build a simple document agent that combines OCR, parsing, and agentic reasoning to read documents and extract structured information. The lab follows a step-by-step approach, reinforcing the bottom-up journey from pixels, to text, to structure, and finally to reasoning.
+=========================================================================================================================
+
+This walkthrough demonstrates how OCR, rule-based methods, and LLM-based agentic reasoning work together in document processing, and where each approach succeeds or fails.
+
+OCR is first applied to extract raw text from documents, which works well for clean, printed invoices but produces unstructured, noisy text with no understanding of meaning. Simple rule-based approaches such as regular expressions are then used to extract values like tax and total, but they fail easily due to small OCR variations, ambiguous wording, or layout differences. This highlights how brittle traditional pipelines are when faced with real-world data.
+
+An agentic approach is then introduced, combining OCR as perception, an LLM as the reasoning component, and tools within a ReAct-style loop. The agent decides when to call OCR, interprets the extracted text semantically, and outputs structured JSON without relying on hardcoded rules. This allows correct extraction of values such as totals even when multiple similar terms (e.g., subtotal vs total) appear.
+
+More challenging examples show the limits of OCR and the strengths and weaknesses of LLM reasoning. Tables with complex layouts, handwriting, and low-quality receipts produce chaotic OCR outputs. The agent can often infer intent and recover partially correct information, but errors still occur when OCR inaccuracies distort the underlying data. In some cases, the LLM overcorrects or reasons from incorrect inputs, leading to plausible but wrong conclusions.
+
+The key takeaway is that OCR provides reading but not understanding, regex provides rules without meaning, and LLM-based agents introduce semantic reasoning that significantly improves robustness. However, reliable real-world document understanding still requires multiple components working together, including OCR, layout analysis, vision-language models, agentic workflows, and validation mechanisms.
+=========================================================================================================================
+
+OCR has evolved from rule-based, procedural computer vision systems to modern deep learning–based approaches. Early OCR systems, represented by Tesseract, relied heavily on handcrafted pipelines such as line detection, character segmentation, and shape matching. These systems work well for clean, printed, black-and-white text with regular layouts and can run efficiently on CPUs, but they struggle with real-world variability such as complex layouts, curved text, images, or noise.
+
+Around 2015, deep learning fundamentally changed OCR by introducing data-driven, end-to-end models. Modern OCR systems separate the problem into two modular stages: text detection (finding text regions) and text recognition (reading the text within those regions). PaddleOCR is a representative system from this era, using neural networks for both stages, specifically DBNet for detection and transformer-based models for recognition. This approach handles irregular layouts, curved or rotated text, and noisy real-world images far better than traditional methods, especially when accelerated with GPUs.
+
+While both Tesseract and PaddleOCR are open source and support many languages, they are best suited to different use cases. Tesseract is ideal for simple document scanning such as books with clean layouts, whereas PaddleOCR performs better on complex, real-world documents like receipts, signage, and mixed-layout content. Overall, these tools illustrate how OCR has shifted from rigid, rule-based pipelines to flexible, learnable systems that can be integrated into larger document intelligence and agentic workflows.
+
+=========================================================================================================================
+A modern OCR pipeline is set up using PaddleOCR along with image and visualization tools. PaddleOCR runs an end-to-end process that includes preprocessing and two deep learning stages: text detection, which finds text regions and returns bounding boxes, and text recognition, which reads the text in each region and outputs the text with confidence scores. Compared to earlier OCR, this pipeline provides localization and improved accuracy on messy inputs such as receipts, which makes downstream reasoning tasks like verifying totals more reliable when combined with an LLM agent.
+
+The same approach is tested on harder examples. On a complex table, PaddleOCR still makes errors such as misreading scientific notation, but an LLM agent can sometimes correct these issues using contextual reasoning and domain expectations. On handwriting, recognition improves over older OCR in some places, but key fields like names and several answers can still be misread, and the agent can only be as accurate as the OCR signal.
+
+New document types expose major weaknesses related to layout and reading order. For report pages containing charts, the OCR may extract axis numbers without recognizing the full chart as a unit, losing context. For multi-column articles, the text can be read across columns incorrectly, producing garbled output. To address this, a layout detection model is added to segment the document into labeled regions such as title, abstract, text blocks, table, chart, footer, and numbers. This improves structure and preserves reading order by keeping text within coherent regions, although errors remain, such as splitting a table into multiple parts or failing to separate headers from table content in bank statements.
+
+Overall, PaddleOCR significantly improves real-world OCR accuracy and adds bounding-box structure, and layout detection helps with region-level organization and reading order. However, these tools still fall short of full semantic document understanding, especially for complex layouts, tables, and small but important text.
+
+
+=========================================================================================================================
+Documents often have complex layouts, so extracting text and sending it directly to a language model can destroy structure and mix content such as columns, tables, captions, and figures. Layout detection addresses this by identifying and labeling page regions like paragraphs, tables, figures, headers, footers, and captions so downstream systems keep structure and target the right areas.
+
+Reading order is a separate problem: it determines the sequence a human would read content, especially in multi-column pages and documents with floating elements. Older heuristic methods (top-to-bottom, left-to-right rules) fail on real layouts. LayoutReader replaces rules with a learned model trained on a large reading-order dataset, using OCR bounding boxes and visual-spatial features to reconstruct a human-like token sequence.
+
+Even with correct reading order, OCR-only pipelines remain limited because OCR captures text but misses visual context such as charts, diagrams, and spatial relationships. Forms require linking labels to values and may need key-value models like LayoutLM and vision for elements like checkboxes. Tables need structure-preserving models such as Table Transformer, TableFormer, or TABLET to recover rows and columns and output usable formats like CSV/JSON/HTML. Handwriting often requires specialized ICR models trained on handwritten data. Multilingual documents add challenges like script detection and different reading directions.
+
+Vision-Language Models (VLMs) extend LLMs by adding a vision encoder and projector so they can reason over images plus text, enabling interpretation of visual elements. However, VLMs can still struggle with small text, nested layouts, multi-page structure, hallucinations, and weak grounding unless they are guided by layout structure.
+
+A practical hybrid approach combines layout detection and reading-order models for deterministic structure with VLMs for visually rich regions. An agent can orchestrate this workflow by using OCR plus bounding boxes, reordering text with LayoutReader, detecting regions (tables, charts, text blocks), and selectively sending cropped regions to specialized VLM-based tools for table and chart understanding based on the user’s question.
+
+
+
+=========================================================================================================================
+An agentic document intelligence pipeline combines OCR, reading-order reconstruction, layout detection, and vision-language model analysis for visual regions.
+
+Text extraction uses PaddleOCR to produce, for each detected text region, the recognized string, a confidence score, and polygon bounding boxes. Bounding boxes are visualized for verification and converted into a standardized XYXY format using structured data objects for cleaner downstream processing.
+
+Reading order is reconstructed with a LayoutReader model built on LayoutLMv3. OCR bounding boxes are normalized to the 0–1000 coordinate range expected by LayoutLM-style models, then the model predicts an ordering index for each region. Regions are sorted by this index to create a correctly sequenced text representation that can answer many questions without visual models.
+
+Layout detection uses PaddleOCR’s layout detector to segment the page into labeled regions such as text blocks, titles, tables, charts, and figures. Each region is assigned a unique ID, stored in structured objects, and visualized with labeled boxes and confidence scores.
+
+For tables and charts, regions are cropped from the original document and encoded in base64 to be sent to vision APIs. Cropping improves focus, reduces noise, and lowers cost, but localization can still be imperfect and requires careful prompt design.
+
+Two specialized tools are defined for vision-language model calls: one for chart interpretation and one for table extraction. Each tool uses a structured prompt with explicit fields and a JSON output template to produce machine-readable results. A shared multimodal-call utility packages the prompt plus the cropped image, and the tools are exposed to the agent via a tool interface.
+
+A tool-calling agent is created with a system context containing the ordered OCR text plus layout region IDs and types. For a given user question, the agent decides whether text alone is sufficient; if not, it selects the appropriate tool, analyzes the target region, and merges the tool output with the textual context into a final answer.
+
+
+=========================================================================================================================
+Agentic Document Extraction (ADE) is a unified, vision-first document intelligence system exposed through a single API that converts documents, images, presentations, and spreadsheets into structured Markdown and JSON.
+
+The system is designed around three core principles. Vision-first processing treats documents as visual objects where meaning comes from layout, structure, and spatial relationships rather than raw text tokens. A data-centric approach emphasizes training on highly curated, document-specific datasets, prioritizing data quality alongside model design. An agentic architecture enables planning, routing, execution, and verification steps to iteratively reach high-quality outputs.
+
+ADE replaces traditional pipelines built from OCR, layout analysis, and vision-language models with document-native vision transformers called DPTs (DPT-1, DPT-2, and DPT-2-mini). These models natively perform reading order reconstruction, layout detection, text recognition, and figure captioning within a single framework.
+
+The core architecture consists of document-native vision models at the foundation, intelligent parsing and routing agents that handle different content types such as text, tables, and figures through separate paths, and an application layer that delivers user-facing capabilities like key-value (field) extraction, document splitting, and content preparation for retrieval-augmented generation.
+
+Primary use cases include precise field extraction with traceability back to source regions, and preparation of complex documents for RAG systems that must preserve tables, figures, and structural context. ADE achieves state-of-the-art accuracy, exceeding human performance on the DocVQA benchmark, demonstrating strong performance on real scanned and handwritten documents.
+
+The platform is accessible through a visual interface, REST APIs, and Python or TypeScript libraries, enabling flexible integration into document 
+
+
+
+=========================================================================================================================
+Agentic Document Extraction (ADE) is used through an API-driven workflow to parse complex documents and extract structured, verifiable information using document-natiprocessing workflows at scale.ve vision models.
+
+The process begins by sending documents to a parsing API powered by Document Pretrained Transformers (DPT-2-latest or DPT-1-latest). The parser converts each document into structured JSON and Markdown, identifying semantically meaningful chunks such as text blocks, tables, figures, charts, logos, margins, and attestations. Each chunk and even individual table cells receive unique identifiers and bounding boxes, enabling precise visual grounding and traceability.
+
+Parsed outputs include:
+
+* Structured chunks with type, coordinates, and page references
+* Markdown representations of text, tables, and figures
+* Cell-level identifiers for tables, enabling fine-grained referencing
+* Rich visual descriptions for figures, charts, flowcharts, and illustrations
+
+A schema-based extraction step is then applied. A user-defined JSON schema specifies the required fields, including nested objects, numeric values, strings, and booleans. The extraction API combines the parsed document representation with this schema to return structured key-value pairs along with metadata linking each extracted value back to its exact source region or table cell.
+
+The system demonstrates robust performance across highly challenging document types:
+
+* Utility bills with mixed text, tables, and usage charts
+* Charts and flowcharts with implicit spatial relationships and arrows
+* Sparse tables, merged cells, and very large “mega tables” with thousands of values
+* Handwritten forms, checkboxes, circled answers, and medical annotations
+* Mathematical handwriting with symbols, equations, and square roots
+* Purely visual documents such as instruction manuals and infographics
+* Official documents containing stamps, curved text, and handwritten signatures
+
+ADE handles all of these cases through a single, consistent API without requiring custom OCR pipelines, layout rules, or manual model orchestration. The output supports downstream applications such as user interfaces, compliance workflows, analytics, and reliable field extraction with full visual traceability, even under extreme document variability and complexity.
+
+
+=========================================================================================================================
+A multi-document financial intake pipeline is built around LandingAI ADE to handle mixed uploads with unknown filenames and unknown document types.
+
+1. Batch parsing and page-level Markdown
+   Each uploaded file is sent to the Parse API using a DPT model. The response is requested as per-page Markdown so the first page can be used for fast identification while still keeping full parsed output available for extraction and grounding.
+
+2. Automatic document type classification
+   The Extract API is used to categorize each file by running a lightweight schema over the first-page Markdown. A Pydantic schema defines an enum of expected document types (for example investment statement, pay stub, bank statement, government ID, tax form) with rich descriptions to improve classification reliability. Pydantic is converted to JSON internally before extraction.
+
+3. Type-specific field extraction with dedicated schemas
+   For each identified document type, a separate Pydantic extraction schema is applied (ID fields, tax form fields, pay stub fields, bank statement fields, investment fields). The pipeline selects the schema dynamically based on the classified type, then calls Extract to return structured key-value pairs plus extraction metadata that links each value to chunk IDs for visual grounding.
+
+4. Grounding-focused visualization for review
+   Parsed outputs are rendered with bounding boxes to show detected chunks (text, tables, figures) and cell-level structure for tables. A second visualization focuses only on the specific fields requested by each schema, highlighting exactly where each extracted value came from, enabling fast human review.
+
+5. Consolidation into a structured summary table
+   All extracted fields across documents are aggregated into a single tabular summary (for example a Pandas DataFrame) with columns such as applicant folder, document name, detected type, field name, and field value. This replaces manual opening, searching, and retyping.
+
+6. Validation and consistency checks
+   Custom validation logic is applied across the extracted results, such as:
+
+* Cross-document name matching to detect inconsistent applicants across uploaded files
+* Recency checks by extracting years from dates and flagging outdated documents
+* Asset aggregation by summing balances across bank and investment statements, scalable to many accounts
+
+The result is an end-to-end workflow that parses heterogeneous documents, identifies their types, extracts structured fields with traceable grounding, produces a reviewer-friendly summary, and runs automated checks to surface inconsistencies and missing requirements.
+
+
+=========================================================================================================================
+
+
+

src/agents/document_agent.py

@@ -0,0 +1,661 @@
+"""
+DocumentAgent for SPARKNET
+
+A ReAct-style agent for document intelligence tasks:
+- Document parsing and extraction
+- Field extraction with grounding
+- Table and chart analysis
+- Document classification
+- Question answering over documents
+"""
+
+from typing import List, Dict, Any, Optional, Tuple
+from dataclasses import dataclass
+from enum import Enum
+import json
+import time
+from loguru import logger
+
+from .base_agent import BaseAgent, Task, Message
+from ..llm.langchain_ollama_client import LangChainOllamaClient
+from ..document.schemas.core import (
+    ProcessedDocument,
+    DocumentChunk,
+    EvidenceRef,
+    ExtractionResult,
+)
+from ..document.schemas.extraction import ExtractionSchema, ExtractedField
+from ..document.schemas.classification import DocumentClassification, DocumentType
+
+
+class AgentAction(str, Enum):
+    """Actions the DocumentAgent can take."""
+    THINK = "think"
+    USE_TOOL = "use_tool"
+    ANSWER = "answer"
+    ABSTAIN = "abstain"
+
+
+@dataclass
+class ThoughtAction:
+    """A thought-action pair in the ReAct loop."""
+    thought: str
+    action: AgentAction
+    tool_name: Optional[str] = None
+    tool_args: Optional[Dict[str, Any]] = None
+    observation: Optional[str] = None
+    evidence: Optional[List[EvidenceRef]] = None
+
+
+@dataclass
+class AgentTrace:
+    """Full trace of agent execution for inspection."""
+    task: str
+    steps: List[ThoughtAction]
+    final_answer: Optional[Any] = None
+    confidence: float = 0.0
+    total_time_ms: float = 0.0
+    success: bool = True
+    error: Optional[str] = None
+
+
+class DocumentAgent:
+    """
+    ReAct-style agent for document intelligence tasks.
+
+    Implements the Think -> Tool -> Observe -> Refine loop
+    with inspectable traces and grounded outputs.
+    """
+
+    # System prompt for ReAct reasoning
+    SYSTEM_PROMPT = """You are a document intelligence agent that analyzes documents
+and extracts information with evidence.
+
+You operate in a Think-Act-Observe loop:
+1. THINK: Analyze what you need to do and what information you have
+2. ACT: Choose a tool to use or provide an answer
+3. OBSERVE: Review the tool output and update your understanding
+
+Available tools:
+{tool_descriptions}
+
+CRITICAL RULES:
+- Every extraction MUST include evidence (page, bbox, text snippet)
+- If you cannot find evidence for a value, ABSTAIN rather than guess
+- Always cite the source of information with page numbers
+- For tables, analyze structure before extracting data
+- For charts, describe what you see before extracting values
+
+Output format for each step:
+THOUGHT: <your reasoning>
+ACTION: <tool_name or ANSWER or ABSTAIN>
+ACTION_INPUT: <JSON arguments for tool, or final answer>
+"""
+
+    # Available tools
+    TOOLS = {
+        "extract_text": {
+            "description": "Extract text from specific pages or regions",
+            "args": ["page_numbers", "region_bbox"],
+        },
+        "analyze_table": {
+            "description": "Analyze and extract structured data from a table region",
+            "args": ["page", "bbox", "expected_columns"],
+        },
+        "analyze_chart": {
+            "description": "Analyze a chart/graph and extract insights",
+            "args": ["page", "bbox"],
+        },
+        "extract_fields": {
+            "description": "Extract specific fields using a schema",
+            "args": ["schema", "context_chunks"],
+        },
+        "classify_document": {
+            "description": "Classify the document type",
+            "args": ["first_page_chunks"],
+        },
+        "search_text": {
+            "description": "Search for text patterns in the document",
+            "args": ["query", "page_range"],
+        },
+    }
+
+    def __init__(
+        self,
+        llm_client: LangChainOllamaClient,
+        memory_agent: Optional[Any] = None,
+        max_iterations: int = 10,
+        temperature: float = 0.3,
+    ):
+        """
+        Initialize DocumentAgent.
+
+        Args:
+            llm_client: LangChain Ollama client
+            memory_agent: Optional memory agent for context retrieval
+            max_iterations: Maximum ReAct iterations
+            temperature: LLM temperature for reasoning
+        """
+        self.llm_client = llm_client
+        self.memory_agent = memory_agent
+        self.max_iterations = max_iterations
+        self.temperature = temperature
+
+        # Current document context
+        self._current_document: Optional[ProcessedDocument] = None
+        self._page_images: Dict[int, Any] = {}
+
+        logger.info(f"Initialized DocumentAgent (max_iterations={max_iterations})")
+
+    def set_document(
+        self,
+        document: ProcessedDocument,
+        page_images: Optional[Dict[int, Any]] = None,
+    ):
+        """
+        Set the current document context.
+
+        Args:
+            document: Processed document
+            page_images: Optional dict of page number -> image array
+        """
+        self._current_document = document
+        self._page_images = page_images or {}
+        logger.info(f"Set document context: {document.metadata.document_id}")
+
+    async def run(
+        self,
+        task_description: str,
+        extraction_schema: Optional[ExtractionSchema] = None,
+    ) -> Tuple[Any, AgentTrace]:
+        """
+        Run the agent on a task.
+
+        Args:
+            task_description: Natural language task description
+            extraction_schema: Optional schema for structured extraction
+
+        Returns:
+            Tuple of (result, trace)
+        """
+        start_time = time.time()
+
+        if not self._current_document:
+            raise ValueError("No document set. Call set_document() first.")
+
+        trace = AgentTrace(task=task_description, steps=[])
+
+        try:
+            # Build initial context
+            context = self._build_context(extraction_schema)
+
+            # ReAct loop
+            result = None
+            for iteration in range(self.max_iterations):
+                logger.debug(f"ReAct iteration {iteration + 1}")
+
+                # Generate thought and action
+                step = await self._generate_step(task_description, context, trace.steps)
+                trace.steps.append(step)
+
+                # Check for terminal actions
+                if step.action == AgentAction.ANSWER:
+                    result = self._parse_answer(step.tool_args)
+                    trace.final_answer = result
+                    trace.confidence = self._calculate_confidence(trace.steps)
+                    break
+
+                elif step.action == AgentAction.ABSTAIN:
+                    trace.final_answer = {
+                        "abstained": True,
+                        "reason": step.thought,
+                    }
+                    trace.confidence = 0.0
+                    break
+
+                elif step.action == AgentAction.USE_TOOL:
+                    # Execute tool and get observation
+                    observation, evidence = await self._execute_tool(
+                        step.tool_name, step.tool_args
+                    )
+                    step.observation = observation
+                    step.evidence = evidence
+
+                    # Update context with observation
+                    context += f"\n\nObservation from {step.tool_name}:\n{observation}"
+
+            trace.success = True
+
+        except Exception as e:
+            logger.error(f"Agent execution failed: {e}")
+            trace.success = False
+            trace.error = str(e)
+
+        trace.total_time_ms = (time.time() - start_time) * 1000
+        return trace.final_answer, trace
+
+    async def extract_fields(
+        self,
+        schema: ExtractionSchema,
+    ) -> ExtractionResult:
+        """
+        Extract fields from the document using a schema.
+
+        Args:
+            schema: Extraction schema defining fields
+
+        Returns:
+            ExtractionResult with extracted data and evidence
+        """
+        task = f"Extract the following fields from this document: {', '.join(f.name for f in schema.fields)}"
+        result, trace = await self.run(task, schema)
+
+        # Build extraction result
+        data = {}
+        evidence = []
+        warnings = []
+        abstained = []
+
+        if isinstance(result, dict):
+            data = result.get("data", result)
+
+            # Collect evidence from trace
+            for step in trace.steps:
+                if step.evidence:
+                    evidence.extend(step.evidence)
+
+            # Check for abstained fields
+            for field in schema.fields:
+                if field.name not in data and field.required:
+                    abstained.append(field.name)
+                    warnings.append(
+                        f"Required field '{field.name}' not found with sufficient confidence"
+                    )
+
+        return ExtractionResult(
+            data=data,
+            evidence=evidence,
+            warnings=warnings,
+            confidence=trace.confidence,
+            abstained_fields=abstained,
+        )
+
+    async def classify(self) -> DocumentClassification:
+        """
+        Classify the document type.
+
+        Returns:
+            DocumentClassification with type and confidence
+        """
+        task = "Classify this document into one of the standard document types (contract, invoice, patent, research_paper, report, letter, form, etc.)"
+        result, trace = await self.run(task)
+
+        # Parse classification result
+        doc_type = DocumentType.UNKNOWN
+        confidence = 0.0
+
+        if isinstance(result, dict):
+            type_str = result.get("document_type", "unknown")
+            try:
+                doc_type = DocumentType(type_str.lower())
+            except ValueError:
+                doc_type = DocumentType.OTHER
+
+            confidence = result.get("confidence", trace.confidence)
+
+        return DocumentClassification(
+            document_id=self._current_document.metadata.document_id,
+            primary_type=doc_type,
+            primary_confidence=confidence,
+            evidence=[e for step in trace.steps if step.evidence for e in step.evidence],
+            method="llm",
+            is_confident=confidence >= 0.7,
+        )
+
+    async def answer_question(self, question: str) -> Tuple[str, List[EvidenceRef]]:
+        """
+        Answer a question about the document.
+
+        Args:
+            question: Natural language question
+
+        Returns:
+            Tuple of (answer, evidence)
+        """
+        task = f"Answer this question about the document: {question}"
+        result, trace = await self.run(task)
+
+        answer = ""
+        evidence = []
+
+        if isinstance(result, dict):
+            answer = result.get("answer", str(result))
+        elif isinstance(result, str):
+            answer = result
+
+        # Collect evidence
+        for step in trace.steps:
+            if step.evidence:
+                evidence.extend(step.evidence)
+
+        return answer, evidence
+
+    def _build_context(self, schema: Optional[ExtractionSchema] = None) -> str:
+        """Build initial context from document."""
+        doc = self._current_document
+        context_parts = [
+            f"Document: {doc.metadata.filename}",
+            f"Type: {doc.metadata.file_type}",
+            f"Pages: {doc.metadata.num_pages}",
+            f"Chunks: {len(doc.chunks)}",
+            "",
+            "Document content summary:",
+        ]
+
+        # Add first few chunks as context
+        for chunk in doc.chunks[:10]:
+            context_parts.append(
+                f"[Page {chunk.page + 1}, {chunk.chunk_type.value}]: {chunk.text[:200]}..."
+            )
+
+        if schema:
+            context_parts.append("")
+            context_parts.append("Extraction schema:")
+            for field in schema.fields:
+                req = "required" if field.required else "optional"
+                context_parts.append(f"- {field.name} ({field.type.value}, {req}): {field.description}")
+
+        return "\n".join(context_parts)
+
+    async def _generate_step(
+        self,
+        task: str,
+        context: str,
+        previous_steps: List[ThoughtAction],
+    ) -> ThoughtAction:
+        """Generate the next thought-action step."""
+        # Build prompt
+        tool_descriptions = "\n".join(
+            f"- {name}: {info['description']}"
+            for name, info in self.TOOLS.items()
+        )
+
+        system_prompt = self.SYSTEM_PROMPT.format(tool_descriptions=tool_descriptions)
+
+        messages = [{"role": "system", "content": system_prompt}]
+
+        # Add task and context
+        user_content = f"TASK: {task}\n\nCONTEXT:\n{context}"
+
+        # Add previous steps
+        if previous_steps:
+            user_content += "\n\nPREVIOUS STEPS:"
+            for i, step in enumerate(previous_steps, 1):
+                user_content += f"\n\nStep {i}:"
+                user_content += f"\nTHOUGHT: {step.thought}"
+                user_content += f"\nACTION: {step.action.value}"
+                if step.tool_name:
+                    user_content += f"\nTOOL: {step.tool_name}"
+                if step.observation:
+                    user_content += f"\nOBSERVATION: {step.observation[:500]}..."
+
+        user_content += "\n\nNow generate your next step:"
+        messages.append({"role": "user", "content": user_content})
+
+        # Generate response
+        llm = self.llm_client.get_llm(complexity="complex", temperature=self.temperature)
+
+        from langchain_core.messages import HumanMessage, SystemMessage
+        lc_messages = [
+            SystemMessage(content=system_prompt),
+            HumanMessage(content=user_content),
+        ]
+
+        response = await llm.ainvoke(lc_messages)
+        response_text = response.content
+
+        # Parse response
+        return self._parse_step(response_text)
+
+    def _parse_step(self, response: str) -> ThoughtAction:
+        """Parse LLM response into ThoughtAction."""
+        thought = ""
+        action = AgentAction.THINK
+        tool_name = None
+        tool_args = None
+
+        lines = response.strip().split("\n")
+        current_section = None
+
+        for line in lines:
+            line = line.strip()
+
+            if line.startswith("THOUGHT:"):
+                current_section = "thought"
+                thought = line[8:].strip()
+            elif line.startswith("ACTION:"):
+                current_section = "action"
+                action_str = line[7:].strip().lower()
+                if action_str == "answer":
+                    action = AgentAction.ANSWER
+                elif action_str == "abstain":
+                    action = AgentAction.ABSTAIN
+                elif action_str in self.TOOLS:
+                    action = AgentAction.USE_TOOL
+                    tool_name = action_str
+                else:
+                    action = AgentAction.USE_TOOL
+                    tool_name = action_str
+            elif line.startswith("ACTION_INPUT:"):
+                current_section = "input"
+                input_str = line[13:].strip()
+                try:
+                    tool_args = json.loads(input_str)
+                except json.JSONDecodeError:
+                    tool_args = {"raw": input_str}
+            elif current_section == "thought":
+                thought += " " + line
+            elif current_section == "input":
+                try:
+                    tool_args = json.loads(line)
+                except:
+                    pass
+
+        return ThoughtAction(
+            thought=thought,
+            action=action,
+            tool_name=tool_name,
+            tool_args=tool_args,
+        )
+
+    async def _execute_tool(
+        self,
+        tool_name: str,
+        tool_args: Optional[Dict[str, Any]],
+    ) -> Tuple[str, List[EvidenceRef]]:
+        """Execute a tool and return observation."""
+        if not tool_args:
+            tool_args = {}
+
+        doc = self._current_document
+        evidence = []
+
+        try:
+            if tool_name == "extract_text":
+                return self._tool_extract_text(tool_args)
+
+            elif tool_name == "analyze_table":
+                return await self._tool_analyze_table(tool_args)
+
+            elif tool_name == "analyze_chart":
+                return await self._tool_analyze_chart(tool_args)
+
+            elif tool_name == "extract_fields":
+                return await self._tool_extract_fields(tool_args)
+
+            elif tool_name == "classify_document":
+                return self._tool_classify_document(tool_args)
+
+            elif tool_name == "search_text":
+                return self._tool_search_text(tool_args)
+
+            else:
+                return f"Unknown tool: {tool_name}", []
+
+        except Exception as e:
+            logger.error(f"Tool {tool_name} failed: {e}")
+            return f"Error executing {tool_name}: {e}", []
+
+    def _tool_extract_text(self, args: Dict[str, Any]) -> Tuple[str, List[EvidenceRef]]:
+        """Extract text from pages or regions."""
+        doc = self._current_document
+        page_numbers = args.get("page_numbers", list(range(doc.metadata.num_pages)))
+
+        if isinstance(page_numbers, int):
+            page_numbers = [page_numbers]
+
+        texts = []
+        evidence = []
+
+        for page in page_numbers:
+            page_chunks = doc.get_page_chunks(page)
+            for chunk in page_chunks:
+                texts.append(f"[Page {page + 1}]: {chunk.text}")
+                evidence.append(EvidenceRef(
+                    chunk_id=chunk.chunk_id,
+                    page=chunk.page,
+                    bbox=chunk.bbox,
+                    source_type="text",
+                    snippet=chunk.text[:100],
+                    confidence=chunk.confidence,
+                ))
+
+        return "\n".join(texts[:20]), evidence[:10]
+
+    async def _tool_analyze_table(self, args: Dict[str, Any]) -> Tuple[str, List[EvidenceRef]]:
+        """Analyze a table region."""
+        page = args.get("page", 0)
+        doc = self._current_document
+
+        # Find table chunks
+        table_chunks = [c for c in doc.chunks if c.chunk_type.value == "table" and c.page == page]
+
+        if not table_chunks:
+            return "No table found on this page", []
+
+        # Use LLM to analyze table
+        table_text = table_chunks[0].text
+        llm = self.llm_client.get_llm(complexity="standard")
+
+        from langchain_core.messages import HumanMessage
+        prompt = f"Analyze this table and extract structured data as JSON:\n\n{table_text}"
+        response = await llm.ainvoke([HumanMessage(content=prompt)])
+
+        evidence = [EvidenceRef(
+            chunk_id=table_chunks[0].chunk_id,
+            page=page,
+            bbox=table_chunks[0].bbox,
+            source_type="table",
+            snippet=table_text[:200],
+            confidence=table_chunks[0].confidence,
+        )]
+
+        return response.content, evidence
+
+    async def _tool_analyze_chart(self, args: Dict[str, Any]) -> Tuple[str, List[EvidenceRef]]:
+        """Analyze a chart region."""
+        page = args.get("page", 0)
+        doc = self._current_document
+
+        # Find chart/figure chunks
+        chart_chunks = [
+            c for c in doc.chunks
+            if c.chunk_type.value in ("chart", "figure") and c.page == page
+        ]
+
+        if not chart_chunks:
+            return "No chart/figure found on this page", []
+
+        # If we have the image, use vision model
+        if page in self._page_images:
+            # TODO: Use vision model for chart analysis
+            pass
+
+        return f"Chart found on page {page + 1}: {chart_chunks[0].caption or 'No caption'}", []
+
+    async def _tool_extract_fields(self, args: Dict[str, Any]) -> Tuple[str, List[EvidenceRef]]:
+        """Extract specific fields."""
+        schema_dict = args.get("schema", {})
+        doc = self._current_document
+
+        # Build context from chunks
+        context = "\n".join(c.text for c in doc.chunks[:20])
+
+        # Use LLM to extract
+        llm = self.llm_client.get_llm(complexity="complex")
+
+        from langchain_core.messages import HumanMessage, SystemMessage
+        system = "Extract the requested fields from the document. Output JSON with field names as keys."
+        user = f"Fields to extract: {json.dumps(schema_dict)}\n\nDocument content:\n{context}"
+
+        response = await llm.ainvoke([
+            SystemMessage(content=system),
+            HumanMessage(content=user),
+        ])
+
+        return response.content, []
+
+    def _tool_classify_document(self, args: Dict[str, Any]) -> Tuple[str, List[EvidenceRef]]:
+        """Classify document type based on first page."""
+        doc = self._current_document
+        first_page_chunks = doc.get_page_chunks(0)
+        text = " ".join(c.text for c in first_page_chunks[:5])
+
+        return f"First page content for classification:\n{text[:500]}", []
+
+    def _tool_search_text(self, args: Dict[str, Any]) -> Tuple[str, List[EvidenceRef]]:
+        """Search for text in document."""
+        query = args.get("query", "").lower()
+        doc = self._current_document
+
+        matches = []
+        evidence = []
+
+        for chunk in doc.chunks:
+            if query in chunk.text.lower():
+                matches.append(f"[Page {chunk.page + 1}]: ...{chunk.text}...")
+                evidence.append(EvidenceRef(
+                    chunk_id=chunk.chunk_id,
+                    page=chunk.page,
+                    bbox=chunk.bbox,
+                    source_type="text",
+                    snippet=chunk.text[:100],
+                    confidence=chunk.confidence,
+                ))
+
+        if not matches:
+            return f"No matches found for '{query}'", []
+
+        return f"Found {len(matches)} matches:\n" + "\n".join(matches[:10]), evidence[:10]
+
+    def _parse_answer(self, answer_input: Optional[Dict[str, Any]]) -> Any:
+        """Parse the final answer from tool args."""
+        if not answer_input:
+            return None
+
+        if isinstance(answer_input, dict):
+            return answer_input
+
+        return {"answer": answer_input}
+
+    def _calculate_confidence(self, steps: List[ThoughtAction]) -> float:
+        """Calculate overall confidence from trace."""
+        if not steps:
+            return 0.0
+
+        # Average evidence confidence
+        all_evidence = [e for s in steps if s.evidence for e in s.evidence]
+        if all_evidence:
+            return sum(e.confidence for e in all_evidence) / len(all_evidence)
+
+        return 0.5  # Default moderate confidence

src/cli/__init__.py

@@ -0,0 +1,9 @@
+"""
+SPARKNET Command Line Interface
+
+Provides CLI commands for document intelligence and RAG operations.
+"""
+
+from .main import app, main
+
+__all__ = ["app", "main"]

src/cli/docint.py

@@ -0,0 +1,681 @@
+"""
+Document Intelligence CLI Commands
+
+CLI interface for the document_intelligence subsystem.
+"""
+
+import json
+import sys
+from pathlib import Path
+from typing import List, Optional
+
+import click
+
+
+@click.group(name="docint")
+def docint_cli():
+    """Document Intelligence commands."""
+    pass
+
+
+@docint_cli.command()
+@click.argument("path", type=click.Path(exists=True))
+@click.option("--output", "-o", type=click.Path(), help="Output JSON file")
+@click.option("--max-pages", type=int, help="Maximum pages to process")
+@click.option("--dpi", type=int, default=200, help="Render DPI (default: 200)")
+@click.option("--format", "output_format", type=click.Choice(["json", "markdown", "text"]),
+              default="json", help="Output format")
+def parse(path: str, output: Optional[str], max_pages: Optional[int],
+          dpi: int, output_format: str):
+    """
+    Parse a document into semantic chunks.
+
+    Example:
+        sparknet docint parse invoice.pdf -o result.json
+        sparknet docint parse document.pdf --format markdown
+    """
+    from src.document_intelligence import (
+        DocumentParser,
+        ParserConfig,
+    )
+
+    config = ParserConfig(
+        render_dpi=dpi,
+        max_pages=max_pages,
+    )
+
+    parser = DocumentParser(config=config)
+
+    click.echo(f"Parsing: {path}")
+
+    try:
+        result = parser.parse(path)
+
+        if output_format == "json":
+            output_data = {
+                "doc_id": result.doc_id,
+                "filename": result.filename,
+                "num_pages": result.num_pages,
+                "chunks": [
+                    {
+                        "chunk_id": c.chunk_id,
+                        "type": c.chunk_type.value,
+                        "text": c.text,
+                        "page": c.page,
+                        "bbox": c.bbox.xyxy,
+                        "confidence": c.confidence,
+                    }
+                    for c in result.chunks
+                ],
+                "processing_time_ms": result.processing_time_ms,
+            }
+
+            if output:
+                with open(output, "w") as f:
+                    json.dump(output_data, f, indent=2)
+                click.echo(f"Output written to: {output}")
+            else:
+                click.echo(json.dumps(output_data, indent=2))
+
+        elif output_format == "markdown":
+            if output:
+                with open(output, "w") as f:
+                    f.write(result.markdown_full)
+                click.echo(f"Markdown written to: {output}")
+            else:
+                click.echo(result.markdown_full)
+
+        else:  # text
+            for chunk in result.chunks:
+                click.echo(f"[Page {chunk.page}, {chunk.chunk_type.value}]")
+                click.echo(chunk.text)
+                click.echo()
+
+        click.echo(f"\nParsed {len(result.chunks)} chunks in {result.processing_time_ms:.0f}ms")
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command()
+@click.argument("path", type=click.Path(exists=True))
+@click.option("--field", "-f", multiple=True, help="Field to extract (can specify multiple)")
+@click.option("--schema", "-s", type=click.Path(exists=True), help="JSON schema file")
+@click.option("--preset", type=click.Choice(["invoice", "receipt", "contract"]),
+              help="Use preset schema")
+@click.option("--output", "-o", type=click.Path(), help="Output JSON file")
+def extract(path: str, field: tuple, schema: Optional[str], preset: Optional[str],
+            output: Optional[str]):
+    """
+    Extract fields from a document.
+
+    Example:
+        sparknet docint extract invoice.pdf --preset invoice
+        sparknet docint extract doc.pdf -f vendor_name -f total_amount
+        sparknet docint extract doc.pdf --schema my_schema.json
+    """
+    from src.document_intelligence import (
+        DocumentParser,
+        FieldExtractor,
+        ExtractionSchema,
+        FieldSpec,
+        FieldType,
+        create_invoice_schema,
+        create_receipt_schema,
+        create_contract_schema,
+    )
+
+    # Build schema
+    if preset:
+        if preset == "invoice":
+            extraction_schema = create_invoice_schema()
+        elif preset == "receipt":
+            extraction_schema = create_receipt_schema()
+        elif preset == "contract":
+            extraction_schema = create_contract_schema()
+    elif schema:
+        with open(schema) as f:
+            schema_dict = json.load(f)
+        extraction_schema = ExtractionSchema.from_json_schema(schema_dict)
+    elif field:
+        extraction_schema = ExtractionSchema(name="custom")
+        for f in field:
+            extraction_schema.add_string_field(f, required=True)
+    else:
+        click.echo("Error: Specify --field, --schema, or --preset", err=True)
+        sys.exit(1)
+
+    click.echo(f"Extracting from: {path}")
+    click.echo(f"Fields: {', '.join(f.name for f in extraction_schema.fields)}")
+
+    try:
+        # Parse document
+        parser = DocumentParser()
+        parse_result = parser.parse(path)
+
+        # Extract fields
+        extractor = FieldExtractor()
+        result = extractor.extract(parse_result, extraction_schema)
+
+        output_data = {
+            "doc_id": parse_result.doc_id,
+            "filename": parse_result.filename,
+            "extracted_data": result.data,
+            "confidence": result.overall_confidence,
+            "abstained_fields": result.abstained_fields,
+            "evidence": [
+                {
+                    "chunk_id": e.chunk_id,
+                    "page": e.page,
+                    "bbox": e.bbox.xyxy,
+                    "snippet": e.snippet,
+                }
+                for e in result.evidence
+            ],
+        }
+
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            click.echo(f"Output written to: {output}")
+        else:
+            click.echo("\nExtracted Data:")
+            for key, value in result.data.items():
+                status = "" if key not in result.abstained_fields else " [ABSTAINED]"
+                click.echo(f"  {key}: {value}{status}")
+
+            click.echo(f"\nConfidence: {result.overall_confidence:.2f}")
+
+            if result.abstained_fields:
+                click.echo(f"Abstained: {', '.join(result.abstained_fields)}")
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command()
+@click.argument("path", type=click.Path(exists=True))
+@click.argument("question")
+@click.option("--verbose", "-v", is_flag=True, help="Show evidence details")
+@click.option("--use-rag", is_flag=True, help="Use RAG for retrieval (requires indexed document)")
+@click.option("--document-id", "-d", help="Document ID for RAG retrieval")
+@click.option("--top-k", "-k", type=int, default=5, help="Number of chunks to consider")
+@click.option("--chunk-type", "-t", multiple=True, help="Filter by chunk type (can specify multiple)")
+@click.option("--page-start", type=int, help="Filter by page range start")
+@click.option("--page-end", type=int, help="Filter by page range end")
+def ask(path: str, question: str, verbose: bool, use_rag: bool,
+        document_id: Optional[str], top_k: int, chunk_type: tuple,
+        page_start: Optional[int], page_end: Optional[int]):
+    """
+    Ask a question about a document.
+
+    Example:
+        sparknet docint ask invoice.pdf "What is the total amount?"
+        sparknet docint ask doc.pdf "Find claims" --use-rag --top-k 10
+        sparknet docint ask doc.pdf "What tables show?" -t table --use-rag
+    """
+    from src.document_intelligence import DocumentParser
+
+    click.echo(f"Document: {path}")
+    click.echo(f"Question: {question}")
+
+    if use_rag:
+        click.echo("Mode: RAG (semantic retrieval)")
+    else:
+        click.echo("Mode: Keyword search")
+
+    click.echo()
+
+    try:
+        if use_rag:
+            # Use RAG-based answering
+            from src.document_intelligence.tools import get_rag_tool
+
+            tool = get_rag_tool("rag_answer")
+
+            # Build page range if specified
+            page_range = None
+            if page_start is not None and page_end is not None:
+                page_range = (page_start, page_end)
+
+            result = tool.execute(
+                question=question,
+                document_id=document_id,
+                top_k=top_k,
+                chunk_types=list(chunk_type) if chunk_type else None,
+                page_range=page_range,
+            )
+        else:
+            # Parse document and use keyword-based search
+            from src.document_intelligence.tools import get_tool
+
+            parser = DocumentParser()
+            parse_result = parser.parse(path)
+
+            tool = get_tool("answer_question")
+            result = tool.execute(
+                parse_result=parse_result,
+                question=question,
+                top_k=top_k,
+            )
+
+        if result.success:
+            data = result.data
+            click.echo(f"Answer: {data.get('answer', 'No answer found')}")
+            click.echo(f"Confidence: {data.get('confidence', 0):.2f}")
+
+            if data.get('abstained'):
+                click.echo("Note: The system abstained due to low confidence.")
+
+            if verbose and result.evidence:
+                click.echo("\nEvidence:")
+                for ev in result.evidence:
+                    click.echo(f"  - Page {ev.get('page', '?')}: {ev.get('snippet', '')[:100]}...")
+
+            if data.get('citations'):
+                click.echo("\nCitations:")
+                for cit in data['citations']:
+                    click.echo(f"  [{cit['index']}] {cit.get('text', '')[:80]}...")
+        else:
+            click.echo(f"Error: {result.error}", err=True)
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command()
+@click.argument("path", type=click.Path(exists=True))
+@click.option("--output", "-o", type=click.Path(), help="Output JSON file")
+def classify(path: str, output: Optional[str]):
+    """
+    Classify a document's type.
+
+    Example:
+        sparknet docint classify document.pdf
+    """
+    from src.document_intelligence import DocumentParser
+    from src.document_intelligence.chunks import DocumentType
+
+    click.echo(f"Classifying: {path}")
+
+    try:
+        # Parse document
+        parser = DocumentParser()
+        parse_result = parser.parse(path)
+
+        # Simple classification based on keywords
+        first_page_chunks = [c for c in parse_result.chunks if c.page == 1][:5]
+        content = " ".join(c.text[:200] for c in first_page_chunks).lower()
+
+        doc_type = "other"
+        confidence = 0.5
+
+        type_keywords = {
+            "invoice": ["invoice", "bill", "payment due", "amount due", "invoice number"],
+            "contract": ["agreement", "contract", "party", "whereas", "terms and conditions"],
+            "receipt": ["receipt", "paid", "transaction", "thank you for your purchase"],
+            "form": ["form", "fill in", "checkbox", "signature line"],
+            "letter": ["dear", "sincerely", "regards", "to whom it may concern"],
+            "report": ["report", "findings", "conclusion", "summary", "analysis"],
+            "patent": ["patent", "claims", "invention", "embodiment", "disclosed"],
+        }
+
+        for dtype, keywords in type_keywords.items():
+            matches = sum(1 for k in keywords if k in content)
+            if matches >= 2:
+                doc_type = dtype
+                confidence = min(0.95, 0.5 + matches * 0.15)
+                break
+
+        output_data = {
+            "doc_id": parse_result.doc_id,
+            "filename": parse_result.filename,
+            "document_type": doc_type,
+            "confidence": confidence,
+        }
+
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            click.echo(f"Output written to: {output}")
+        else:
+            click.echo(f"Type: {doc_type}")
+            click.echo(f"Confidence: {confidence:.2f}")
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command()
+@click.argument("path", type=click.Path(exists=True))
+@click.option("--query", "-q", help="Search query")
+@click.option("--type", "chunk_type", help="Filter by chunk type")
+@click.option("--top", "-k", type=int, default=10, help="Number of results")
+def search(path: str, query: Optional[str], chunk_type: Optional[str], top: int):
+    """
+    Search document content.
+
+    Example:
+        sparknet docint search document.pdf -q "payment terms"
+        sparknet docint search document.pdf --type table
+    """
+    from src.document_intelligence import DocumentParser
+    from src.document_intelligence.tools import get_tool
+
+    click.echo(f"Searching: {path}")
+
+    try:
+        # Parse document
+        parser = DocumentParser()
+        parse_result = parser.parse(path)
+
+        if query:
+            # Search by query
+            tool = get_tool("search_chunks")
+            result = tool.execute(
+                parse_result=parse_result,
+                query=query,
+                chunk_types=[chunk_type] if chunk_type else None,
+                top_k=top,
+            )
+
+            if result.success:
+                results = result.data.get("results", [])
+                click.echo(f"Found {len(results)} results:\n")
+
+                for i, r in enumerate(results, 1):
+                    click.echo(f"{i}. [Page {r['page']}, {r['type']}] (score: {r['score']:.2f})")
+                    click.echo(f"   {r['text'][:200]}...")
+                    click.echo()
+            else:
+                click.echo(f"Error: {result.error}", err=True)
+
+        elif chunk_type:
+            # Filter by type
+            matching = [c for c in parse_result.chunks if c.chunk_type.value == chunk_type]
+            click.echo(f"Found {len(matching)} {chunk_type} chunks:\n")
+
+            for i, chunk in enumerate(matching[:top], 1):
+                click.echo(f"{i}. [Page {chunk.page}] {chunk.chunk_id}")
+                click.echo(f"   {chunk.text[:200]}...")
+                click.echo()
+
+        else:
+            # List all chunks
+            click.echo(f"Total chunks: {len(parse_result.chunks)}\n")
+
+            # Group by type
+            by_type = {}
+            for chunk in parse_result.chunks:
+                t = chunk.chunk_type.value
+                by_type[t] = by_type.get(t, 0) + 1
+
+            click.echo("Chunk types:")
+            for t, count in sorted(by_type.items()):
+                click.echo(f"  {t}: {count}")
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command()
+@click.argument("path", type=click.Path(exists=True))
+@click.option("--page", "-p", type=int, default=1, help="Page number")
+@click.option("--output-dir", "-d", type=click.Path(), default="./crops",
+              help="Output directory for crops")
+@click.option("--annotate", "-a", is_flag=True, help="Create annotated page image")
+def visualize(path: str, page: int, output_dir: str, annotate: bool):
+    """
+    Visualize document regions.
+
+    Example:
+        sparknet docint visualize document.pdf --page 1 --annotate
+    """
+    from src.document_intelligence import (
+        DocumentParser,
+        load_document,
+        RenderOptions,
+    )
+    from src.document_intelligence.grounding import create_annotated_image, CropManager
+    from PIL import Image
+    import numpy as np
+
+    output_path = Path(output_dir)
+    output_path.mkdir(parents=True, exist_ok=True)
+
+    click.echo(f"Processing: {path}, page {page}")
+
+    try:
+        # Parse document
+        parser = DocumentParser()
+        parse_result = parser.parse(path)
+
+        # Load and render page
+        loader, renderer = load_document(path)
+        page_image = renderer.render_page(page, RenderOptions(dpi=200))
+        loader.close()
+
+        # Get page chunks
+        page_chunks = [c for c in parse_result.chunks if c.page == page]
+
+        if annotate:
+            # Create annotated image
+            bboxes = [c.bbox for c in page_chunks]
+            labels = [f"{c.chunk_type.value[:10]}" for c in page_chunks]
+
+            annotated = create_annotated_image(page_image, bboxes, labels)
+
+            output_file = output_path / f"annotated_page_{page}.png"
+            Image.fromarray(annotated).save(output_file)
+            click.echo(f"Saved annotated image: {output_file}")
+
+        else:
+            # Save individual crops
+            crop_manager = CropManager(output_path)
+
+            for chunk in page_chunks:
+                crop_path = crop_manager.save_crop(
+                    page_image,
+                    parse_result.doc_id,
+                    page,
+                    chunk.bbox,
+                )
+                click.echo(f"Saved crop: {crop_path}")
+
+        click.echo(f"\nProcessed {len(page_chunks)} chunks from page {page}")
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command()
+@click.argument("paths", nargs=-1, type=click.Path(exists=True), required=True)
+@click.option("--max-pages", type=int, help="Maximum pages to process per document")
+@click.option("--batch-size", type=int, default=32, help="Embedding batch size")
+@click.option("--min-length", type=int, default=10, help="Minimum chunk text length")
+def index(paths: tuple, max_pages: Optional[int], batch_size: int, min_length: int):
+    """
+    Index documents into the vector store for RAG.
+
+    Example:
+        sparknet docint index document.pdf
+        sparknet docint index *.pdf --max-pages 50
+        sparknet docint index doc1.pdf doc2.pdf doc3.pdf
+    """
+    from src.document_intelligence.tools import get_rag_tool
+
+    click.echo(f"Indexing {len(paths)} document(s)...")
+    click.echo()
+
+    try:
+        tool = get_rag_tool("index_document")
+
+        total_indexed = 0
+        total_skipped = 0
+        errors = []
+
+        for path in paths:
+            click.echo(f"Processing: {path}")
+
+            result = tool.execute(
+                path=path,
+                max_pages=max_pages,
+            )
+
+            if result.success:
+                data = result.data
+                indexed = data.get("chunks_indexed", 0)
+                skipped = data.get("chunks_skipped", 0)
+                total_indexed += indexed
+                total_skipped += skipped
+                click.echo(f"  Indexed: {indexed} chunks, Skipped: {skipped}")
+                click.echo(f"  Document ID: {data.get('document_id', 'unknown')}")
+            else:
+                errors.append((path, result.error))
+                click.echo(f"  Error: {result.error}", err=True)
+
+        click.echo()
+        click.echo("=" * 40)
+        click.echo(f"Total documents: {len(paths)}")
+        click.echo(f"Total chunks indexed: {total_indexed}")
+        click.echo(f"Total chunks skipped: {total_skipped}")
+
+        if errors:
+            click.echo(f"Errors: {len(errors)}")
+            for path, err in errors:
+                click.echo(f"  - {path}: {err}")
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command(name="index-stats")
+def index_stats():
+    """
+    Show statistics about the vector store index.
+
+    Example:
+        sparknet docint index-stats
+    """
+    from src.document_intelligence.tools import get_rag_tool
+
+    try:
+        tool = get_rag_tool("get_index_stats")
+        result = tool.execute()
+
+        if result.success:
+            data = result.data
+            click.echo("Vector Store Statistics:")
+            click.echo(f"  Total chunks: {data.get('total_chunks', 0)}")
+            click.echo(f"  Embedding model: {data.get('embedding_model', 'unknown')}")
+            click.echo(f"  Embedding dimension: {data.get('embedding_dimension', 'unknown')}")
+        else:
+            click.echo(f"Error: {result.error}", err=True)
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command(name="delete-index")
+@click.argument("document_id")
+@click.option("--yes", "-y", is_flag=True, help="Skip confirmation prompt")
+def delete_index(document_id: str, yes: bool):
+    """
+    Delete a document from the vector store index.
+
+    Example:
+        sparknet docint delete-index doc_abc123
+    """
+    from src.document_intelligence.tools import get_rag_tool
+
+    if not yes:
+        click.confirm(f"Delete document '{document_id}' from index?", abort=True)
+
+    try:
+        tool = get_rag_tool("delete_document")
+        result = tool.execute(document_id=document_id)
+
+        if result.success:
+            data = result.data
+            click.echo(f"Deleted {data.get('chunks_deleted', 0)} chunks for document: {document_id}")
+        else:
+            click.echo(f"Error: {result.error}", err=True)
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+@docint_cli.command(name="retrieve")
+@click.argument("query")
+@click.option("--top-k", "-k", type=int, default=5, help="Number of results")
+@click.option("--document-id", "-d", help="Filter by document ID")
+@click.option("--chunk-type", "-t", multiple=True, help="Filter by chunk type")
+@click.option("--page-start", type=int, help="Filter by page range start")
+@click.option("--page-end", type=int, help="Filter by page range end")
+@click.option("--verbose", "-v", is_flag=True, help="Show full chunk text")
+def retrieve(query: str, top_k: int, document_id: Optional[str],
+             chunk_type: tuple, page_start: Optional[int],
+             page_end: Optional[int], verbose: bool):
+    """
+    Retrieve relevant chunks from the vector store.
+
+    Example:
+        sparknet docint retrieve "payment terms"
+        sparknet docint retrieve "claims" -d doc_abc123 -t paragraph -k 10
+    """
+    from src.document_intelligence.tools import get_rag_tool
+
+    click.echo(f"Query: {query}")
+    click.echo()
+
+    try:
+        tool = get_rag_tool("retrieve_chunks")
+
+        page_range = None
+        if page_start is not None and page_end is not None:
+            page_range = (page_start, page_end)
+
+        result = tool.execute(
+            query=query,
+            top_k=top_k,
+            document_id=document_id,
+            chunk_types=list(chunk_type) if chunk_type else None,
+            page_range=page_range,
+        )
+
+        if result.success:
+            data = result.data
+            chunks = data.get("chunks", [])
+            click.echo(f"Found {len(chunks)} results:\n")
+
+            for i, chunk in enumerate(chunks, 1):
+                click.echo(f"{i}. [sim={chunk['similarity']:.3f}] Page {chunk.get('page', '?')}, {chunk.get('chunk_type', 'text')}")
+                click.echo(f"   Document: {chunk['document_id']}")
+
+                text = chunk['text']
+                if verbose:
+                    click.echo(f"   Text: {text}")
+                else:
+                    click.echo(f"   Text: {text[:150]}...")
+                click.echo()
+        else:
+            click.echo(f"Error: {result.error}", err=True)
+
+    except Exception as e:
+        click.echo(f"Error: {e}", err=True)
+        sys.exit(1)
+
+
+# Register with main CLI
+def register_commands(cli):
+    """Register docint commands with main CLI."""
+    cli.add_command(docint_cli)

src/cli/document.py

@@ -0,0 +1,322 @@
+"""
+Document Processing CLI Commands
+
+Commands:
+    sparknet document parse <file>     - Parse and extract text from document
+    sparknet document extract <file>   - Extract structured fields
+    sparknet document classify <file>  - Classify document type
+    sparknet document analyze <file>   - Full document analysis
+"""
+
+import typer
+from typing import Optional, List
+from pathlib import Path
+import json
+import sys
+
+# Create document sub-app
+document_app = typer.Typer(
+    name="document",
+    help="Document processing commands",
+)
+
+
+@document_app.command("parse")
+def parse_document(
+    file_path: Path = typer.Argument(..., help="Path to document file"),
+    output: Optional[Path] = typer.Option(None, "--output", "-o", help="Output JSON file"),
+    ocr_engine: str = typer.Option("paddleocr", "--ocr", help="OCR engine: paddleocr, tesseract"),
+    dpi: int = typer.Option(300, "--dpi", help="Rendering DPI for PDFs"),
+    max_pages: Optional[int] = typer.Option(None, "--max-pages", help="Maximum pages to process"),
+    include_images: bool = typer.Option(False, "--images", help="Include cropped region images"),
+):
+    """
+    Parse a document and extract text with layout information.
+
+    Example:
+        sparknet document parse invoice.pdf -o result.json
+    """
+    from loguru import logger
+
+    if not file_path.exists():
+        typer.echo(f"Error: File not found: {file_path}", err=True)
+        raise typer.Exit(1)
+
+    typer.echo(f"Parsing document: {file_path}")
+
+    try:
+        from ..document.pipeline import (
+            PipelineConfig,
+            get_document_processor,
+        )
+        from ..document.ocr import OCRConfig
+
+        # Build config
+        ocr_config = OCRConfig(engine=ocr_engine)
+        config = PipelineConfig(
+            ocr=ocr_config,
+            render_dpi=dpi,
+            max_pages=max_pages,
+        )
+
+        # Process document
+        processor = get_document_processor(config)
+        result = processor.process(str(file_path))
+
+        # Format output
+        output_data = {
+            "document_id": result.metadata.document_id,
+            "filename": result.metadata.filename,
+            "num_pages": result.metadata.num_pages,
+            "total_chunks": result.metadata.total_chunks,
+            "total_characters": result.metadata.total_characters,
+            "ocr_confidence": result.metadata.ocr_confidence_avg,
+            "chunks": [
+                {
+                    "chunk_id": c.chunk_id,
+                    "type": c.chunk_type.value,
+                    "page": c.page,
+                    "text": c.text[:500] + "..." if len(c.text) > 500 else c.text,
+                    "confidence": c.confidence,
+                    "bbox": {
+                        "x_min": c.bbox.x_min,
+                        "y_min": c.bbox.y_min,
+                        "x_max": c.bbox.x_max,
+                        "y_max": c.bbox.y_max,
+                    },
+                }
+                for c in result.chunks
+            ],
+            "full_text": result.full_text[:2000] + "..." if len(result.full_text) > 2000 else result.full_text,
+        }
+
+        # Output
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            typer.echo(f"Results written to: {output}")
+        else:
+            typer.echo(json.dumps(output_data, indent=2))
+
+        typer.echo(f"\nProcessed {result.metadata.num_pages} pages, {len(result.chunks)} chunks")
+
+    except ImportError as e:
+        typer.echo(f"Error: Missing dependency - {e}", err=True)
+        raise typer.Exit(1)
+    except Exception as e:
+        typer.echo(f"Error processing document: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@document_app.command("extract")
+def extract_fields(
+    file_path: Path = typer.Argument(..., help="Path to document file"),
+    schema: Optional[Path] = typer.Option(None, "--schema", "-s", help="Extraction schema YAML file"),
+    fields: Optional[List[str]] = typer.Option(None, "--field", "-f", help="Fields to extract (can use multiple)"),
+    output: Optional[Path] = typer.Option(None, "--output", "-o", help="Output JSON file"),
+    validate: bool = typer.Option(True, "--validate/--no-validate", help="Validate extraction"),
+):
+    """
+    Extract structured fields from a document.
+
+    Example:
+        sparknet document extract invoice.pdf -f "invoice_number" -f "total_amount"
+        sparknet document extract contract.pdf --schema contract_schema.yaml
+    """
+    from loguru import logger
+
+    if not file_path.exists():
+        typer.echo(f"Error: File not found: {file_path}", err=True)
+        raise typer.Exit(1)
+
+    if not schema and not fields:
+        typer.echo("Error: Provide --schema or --field options", err=True)
+        raise typer.Exit(1)
+
+    typer.echo(f"Extracting fields from: {file_path}")
+
+    try:
+        from ..document.schemas.extraction import ExtractionSchema, FieldDefinition
+        from ..agents.document_agent import DocumentAgent
+
+        # Build extraction schema
+        if schema:
+            import yaml
+            with open(schema) as f:
+                schema_data = yaml.safe_load(f)
+            extraction_schema = ExtractionSchema(**schema_data)
+        else:
+            # Build from field names
+            field_defs = [
+                FieldDefinition(
+                    name=f,
+                    field_type="string",
+                    required=True,
+                )
+                for f in fields
+            ]
+            extraction_schema = ExtractionSchema(
+                name="cli_extraction",
+                fields=field_defs,
+            )
+
+        # Run extraction with agent
+        import asyncio
+        agent = DocumentAgent()
+        asyncio.run(agent.load_document(str(file_path)))
+        result = asyncio.run(agent.extract_fields(extraction_schema))
+
+        # Format output
+        output_data = {
+            "document": str(file_path),
+            "fields": result.fields,
+            "confidence": result.confidence,
+            "evidence": [
+                {
+                    "chunk_id": e.chunk_id,
+                    "page": e.page,
+                    "snippet": e.snippet,
+                }
+                for e in result.evidence
+            ] if result.evidence else [],
+        }
+
+        # Validate if requested
+        if validate and result.fields:
+            from ..document.validation import get_extraction_critic
+            critic = get_extraction_critic()
+
+            evidence_chunks = [
+                {"text": e.snippet, "page": e.page, "chunk_id": e.chunk_id}
+                for e in result.evidence
+            ] if result.evidence else []
+
+            validation = critic.validate_extraction(result.fields, evidence_chunks)
+            output_data["validation"] = {
+                "status": validation.overall_status.value,
+                "confidence": validation.overall_confidence,
+                "should_accept": validation.should_accept,
+                "abstain_reason": validation.abstain_reason,
+            }
+
+        # Output
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            typer.echo(f"Results written to: {output}")
+        else:
+            typer.echo(json.dumps(output_data, indent=2))
+
+    except ImportError as e:
+        typer.echo(f"Error: Missing dependency - {e}", err=True)
+        raise typer.Exit(1)
+    except Exception as e:
+        typer.echo(f"Error extracting fields: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@document_app.command("classify")
+def classify_document(
+    file_path: Path = typer.Argument(..., help="Path to document file"),
+    output: Optional[Path] = typer.Option(None, "--output", "-o", help="Output JSON file"),
+):
+    """
+    Classify document type.
+
+    Example:
+        sparknet document classify document.pdf
+    """
+    from loguru import logger
+
+    if not file_path.exists():
+        typer.echo(f"Error: File not found: {file_path}", err=True)
+        raise typer.Exit(1)
+
+    typer.echo(f"Classifying document: {file_path}")
+
+    try:
+        from ..agents.document_agent import DocumentAgent
+        import asyncio
+
+        agent = DocumentAgent()
+        asyncio.run(agent.load_document(str(file_path)))
+        classification = asyncio.run(agent.classify())
+
+        output_data = {
+            "document": str(file_path),
+            "document_type": classification.document_type.value,
+            "confidence": classification.confidence,
+            "reasoning": classification.reasoning,
+            "metadata": classification.metadata,
+        }
+
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            typer.echo(f"Results written to: {output}")
+        else:
+            typer.echo(json.dumps(output_data, indent=2))
+
+    except Exception as e:
+        typer.echo(f"Error classifying document: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@document_app.command("ask")
+def ask_document(
+    file_path: Path = typer.Argument(..., help="Path to document file"),
+    question: str = typer.Argument(..., help="Question to ask about the document"),
+    output: Optional[Path] = typer.Option(None, "--output", "-o", help="Output JSON file"),
+):
+    """
+    Ask a question about a document.
+
+    Example:
+        sparknet document ask invoice.pdf "What is the total amount?"
+    """
+    from loguru import logger
+
+    if not file_path.exists():
+        typer.echo(f"Error: File not found: {file_path}", err=True)
+        raise typer.Exit(1)
+
+    typer.echo(f"Processing question for: {file_path}")
+
+    try:
+        from ..agents.document_agent import DocumentAgent
+        import asyncio
+
+        agent = DocumentAgent()
+        asyncio.run(agent.load_document(str(file_path)))
+        answer, evidence = asyncio.run(agent.answer_question(question))
+
+        output_data = {
+            "document": str(file_path),
+            "question": question,
+            "answer": answer,
+            "evidence": [
+                {
+                    "chunk_id": e.chunk_id,
+                    "page": e.page,
+                    "snippet": e.snippet,
+                    "confidence": e.confidence,
+                }
+                for e in evidence
+            ] if evidence else [],
+        }
+
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            typer.echo(f"Results written to: {output}")
+        else:
+            typer.echo(f"\nQuestion: {question}")
+            typer.echo(f"\nAnswer: {answer}")
+            if evidence:
+                typer.echo(f"\nEvidence ({len(evidence)} sources):")
+                for e in evidence[:3]:
+                    typer.echo(f"  - Page {e.page + 1}: {e.snippet[:100]}...")
+
+    except Exception as e:
+        typer.echo(f"Error processing question: {e}", err=True)
+        raise typer.Exit(1)

src/cli/main.py

@@ -0,0 +1,110 @@
+"""
+SPARKNET CLI Main Entry Point
+
+Usage:
+    sparknet document parse <file>
+    sparknet document extract <file> --schema <schema.yaml>
+    sparknet rag index <file>
+    sparknet rag ask <question>
+"""
+
+import typer
+from typing import Optional
+from pathlib import Path
+import json
+import sys
+
+from .document import document_app
+from .rag import rag_app
+
+# Create main app
+app = typer.Typer(
+    name="sparknet",
+    help="SPARKNET Document Intelligence CLI",
+    add_completion=False,
+)
+
+# Register sub-commands
+app.add_typer(document_app, name="document", help="Document processing commands")
+app.add_typer(rag_app, name="rag", help="RAG and retrieval commands")
+
+
+@app.command()
+def version():
+    """Show SPARKNET version."""
+    typer.echo("SPARKNET Document Intelligence v0.1.0")
+
+
+@app.command()
+def info():
+    """Show system information and configuration."""
+    from loguru import logger
+    import platform
+
+    typer.echo("SPARKNET Document Intelligence")
+    typer.echo("=" * 40)
+    typer.echo(f"Python: {platform.python_version()}")
+    typer.echo(f"Platform: {platform.system()} {platform.release()}")
+    typer.echo()
+
+    # Check component availability
+    typer.echo("Components:")
+
+    # OCR
+    try:
+        from paddleocr import PaddleOCR
+        typer.echo("  [✓] PaddleOCR")
+    except ImportError:
+        typer.echo("  [✗] PaddleOCR (install with: pip install paddleocr)")
+
+    try:
+        import pytesseract
+        typer.echo("  [✓] Tesseract")
+    except ImportError:
+        typer.echo("  [✗] Tesseract (install with: pip install pytesseract)")
+
+    # Vector Store
+    try:
+        import chromadb
+        typer.echo("  [✓] ChromaDB")
+    except ImportError:
+        typer.echo("  [✗] ChromaDB (install with: pip install chromadb)")
+
+    # Ollama
+    try:
+        import httpx
+        with httpx.Client(timeout=2.0) as client:
+            resp = client.get("http://localhost:11434/api/tags")
+            if resp.status_code == 200:
+                models = resp.json().get("models", [])
+                typer.echo(f"  [✓] Ollama ({len(models)} models)")
+            else:
+                typer.echo("  [✗] Ollama (not responding)")
+    except Exception:
+        typer.echo("  [✗] Ollama (not running)")
+
+
+@app.callback()
+def main_callback(
+    verbose: bool = typer.Option(False, "--verbose", "-v", help="Enable verbose output"),
+    quiet: bool = typer.Option(False, "--quiet", "-q", help="Suppress output"),
+):
+    """SPARKNET Document Intelligence CLI."""
+    from loguru import logger
+    import sys
+
+    # Configure logging
+    logger.remove()
+    if verbose:
+        logger.add(sys.stderr, level="DEBUG")
+    elif not quiet:
+        logger.add(sys.stderr, level="INFO")
+
+
+def main():
+    """Main entry point."""
+    app()
+
+
+if __name__ == "__main__":
+    main()

src/cli/rag.py

@@ -0,0 +1,314 @@
+"""
+RAG CLI Commands
+
+Commands:
+    sparknet rag index <file>      - Index document for retrieval
+    sparknet rag search <query>    - Search indexed documents
+    sparknet rag ask <question>    - Answer question using RAG
+    sparknet rag status            - Show index status
+"""
+
+import typer
+from typing import Optional, List
+from pathlib import Path
+import json
+import sys
+
+# Create RAG sub-app
+rag_app = typer.Typer(
+    name="rag",
+    help="RAG and retrieval commands",
+)
+
+
+@rag_app.command("index")
+def index_document(
+    files: List[Path] = typer.Argument(..., help="Document file(s) to index"),
+    collection: str = typer.Option("sparknet_documents", "--collection", "-c", help="Collection name"),
+    embedding_model: str = typer.Option("nomic-embed-text", "--model", "-m", help="Embedding model"),
+):
+    """
+    Index document(s) for RAG retrieval.
+
+    Example:
+        sparknet rag index document.pdf
+        sparknet rag index *.pdf --collection contracts
+    """
+    from loguru import logger
+
+    # Validate files
+    valid_files = []
+    for f in files:
+        if f.exists():
+            valid_files.append(f)
+        else:
+            typer.echo(f"Warning: File not found, skipping: {f}", err=True)
+
+    if not valid_files:
+        typer.echo("Error: No valid files to index", err=True)
+        raise typer.Exit(1)
+
+    typer.echo(f"Indexing {len(valid_files)} document(s)...")
+
+    try:
+        from ..rag import (
+            VectorStoreConfig,
+            EmbeddingConfig,
+            get_document_indexer,
+        )
+
+        # Configure
+        store_config = VectorStoreConfig(collection_name=collection)
+        embed_config = EmbeddingConfig(ollama_model=embedding_model)
+
+        # Get indexer
+        indexer = get_document_indexer()
+
+        # Index documents
+        results = indexer.index_batch([str(f) for f in valid_files])
+
+        # Summary
+        successful = sum(1 for r in results if r.success)
+        total_chunks = sum(r.num_chunks_indexed for r in results)
+
+        typer.echo(f"\nIndexing complete:")
+        typer.echo(f"  Documents: {successful}/{len(results)} successful")
+        typer.echo(f"  Chunks indexed: {total_chunks}")
+
+        for r in results:
+            status = "✓" if r.success else "✗"
+            typer.echo(f"  [{status}] {r.source_path}: {r.num_chunks_indexed} chunks")
+            if r.error:
+                typer.echo(f"      Error: {r.error}")
+
+    except ImportError as e:
+        typer.echo(f"Error: Missing dependency - {e}", err=True)
+        raise typer.Exit(1)
+    except Exception as e:
+        typer.echo(f"Error indexing documents: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@rag_app.command("search")
+def search_documents(
+    query: str = typer.Argument(..., help="Search query"),
+    top_k: int = typer.Option(5, "--top", "-k", help="Number of results"),
+    collection: str = typer.Option("sparknet_documents", "--collection", "-c", help="Collection name"),
+    document_id: Optional[str] = typer.Option(None, "--document", "-d", help="Filter by document ID"),
+    chunk_type: Optional[str] = typer.Option(None, "--type", "-t", help="Filter by chunk type"),
+    output: Optional[Path] = typer.Option(None, "--output", "-o", help="Output JSON file"),
+):
+    """
+    Search indexed documents.
+
+    Example:
+        sparknet rag search "payment terms" --top 10
+        sparknet rag search "table data" --type table
+    """
+    typer.echo(f"Searching: {query}")
+
+    try:
+        from ..rag import get_document_retriever, RetrieverConfig
+
+        # Configure
+        config = RetrieverConfig(default_top_k=top_k)
+        retriever = get_document_retriever(config)
+
+        # Build filters
+        filters = {}
+        if document_id:
+            filters["document_id"] = document_id
+        if chunk_type:
+            filters["chunk_type"] = chunk_type
+
+        # Search
+        chunks = retriever.retrieve(query, top_k=top_k, filters=filters if filters else None)
+
+        if not chunks:
+            typer.echo("No results found.")
+            return
+
+        # Format output
+        output_data = {
+            "query": query,
+            "num_results": len(chunks),
+            "results": [
+                {
+                    "chunk_id": c.chunk_id,
+                    "document_id": c.document_id,
+                    "page": c.page,
+                    "chunk_type": c.chunk_type,
+                    "similarity": c.similarity,
+                    "text": c.text[:500] + "..." if len(c.text) > 500 else c.text,
+                }
+                for c in chunks
+            ],
+        }
+
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            typer.echo(f"Results written to: {output}")
+        else:
+            typer.echo(f"\nFound {len(chunks)} results:\n")
+            for i, c in enumerate(chunks, 1):
+                typer.echo(f"[{i}] Similarity: {c.similarity:.3f}")
+                if c.page is not None:
+                    typer.echo(f"    Page: {c.page + 1}, Type: {c.chunk_type or 'text'}")
+                typer.echo(f"    {c.text[:200]}...")
+                typer.echo()
+
+    except Exception as e:
+        typer.echo(f"Error searching: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@rag_app.command("ask")
+def ask_question(
+    question: str = typer.Argument(..., help="Question to answer"),
+    top_k: int = typer.Option(5, "--top", "-k", help="Number of context chunks"),
+    collection: str = typer.Option("sparknet_documents", "--collection", "-c", help="Collection name"),
+    document_id: Optional[str] = typer.Option(None, "--document", "-d", help="Filter by document ID"),
+    output: Optional[Path] = typer.Option(None, "--output", "-o", help="Output JSON file"),
+    show_evidence: bool = typer.Option(True, "--evidence/--no-evidence", help="Show evidence sources"),
+):
+    """
+    Answer a question using RAG.
+
+    Example:
+        sparknet rag ask "What are the payment terms?"
+        sparknet rag ask "What is the contract value?" --document contract123
+    """
+    typer.echo(f"Question: {question}")
+    typer.echo("Processing...")
+
+    try:
+        from ..rag import get_grounded_generator, GeneratorConfig
+
+        # Configure
+        config = GeneratorConfig()
+        generator = get_grounded_generator(config)
+
+        # Build filters
+        filters = {"document_id": document_id} if document_id else None
+
+        # Generate answer
+        result = generator.answer_question(question, top_k=top_k, filters=filters)
+
+        # Format output
+        output_data = {
+            "question": question,
+            "answer": result.answer,
+            "confidence": result.confidence,
+            "abstained": result.abstained,
+            "abstain_reason": result.abstain_reason,
+            "citations": [
+                {
+                    "index": c.index,
+                    "page": c.page,
+                    "snippet": c.text_snippet,
+                    "confidence": c.confidence,
+                }
+                for c in result.citations
+            ],
+            "num_chunks_used": result.num_chunks_used,
+        }
+
+        if output:
+            with open(output, "w") as f:
+                json.dump(output_data, f, indent=2)
+            typer.echo(f"Results written to: {output}")
+        else:
+            typer.echo(f"\nAnswer: {result.answer}")
+            typer.echo(f"\nConfidence: {result.confidence:.2f}")
+
+            if result.abstained:
+                typer.echo(f"Note: {result.abstain_reason}")
+
+            if show_evidence and result.citations:
+                typer.echo(f"\nSources ({len(result.citations)}):")
+                for c in result.citations:
+                    page_info = f"Page {c.page + 1}" if c.page is not None else ""
+                    typer.echo(f"  [{c.index}] {page_info}: {c.text_snippet[:80]}...")
+
+    except Exception as e:
+        typer.echo(f"Error generating answer: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@rag_app.command("status")
+def show_status(
+    collection: str = typer.Option("sparknet_documents", "--collection", "-c", help="Collection name"),
+):
+    """
+    Show RAG index status.
+
+    Example:
+        sparknet rag status
+        sparknet rag status --collection contracts
+    """
+    typer.echo("RAG Index Status")
+    typer.echo("=" * 40)
+
+    try:
+        from ..rag import get_vector_store, VectorStoreConfig
+
+        config = VectorStoreConfig(collection_name=collection)
+        store = get_vector_store(config)
+
+        # Get stats
+        total_chunks = store.count()
+
+        typer.echo(f"Collection: {collection}")
+        typer.echo(f"Total chunks: {total_chunks}")
+
+        # List documents
+        if hasattr(store, 'list_documents'):
+            doc_ids = store.list_documents()
+            typer.echo(f"Documents indexed: {len(doc_ids)}")
+
+            if doc_ids:
+                typer.echo("\nDocuments:")
+                for doc_id in doc_ids[:10]:
+                    chunk_count = store.count(doc_id)
+                    typer.echo(f"  - {doc_id}: {chunk_count} chunks")
+
+                if len(doc_ids) > 10:
+                    typer.echo(f"  ... and {len(doc_ids) - 10} more")
+
+    except Exception as e:
+        typer.echo(f"Error getting status: {e}", err=True)
+        raise typer.Exit(1)
+
+
+@rag_app.command("delete")
+def delete_document(
+    document_id: str = typer.Argument(..., help="Document ID to delete"),
+    collection: str = typer.Option("sparknet_documents", "--collection", "-c", help="Collection name"),
+    force: bool = typer.Option(False, "--force", "-f", help="Skip confirmation"),
+):
+    """
+    Delete a document from the index.
+
+    Example:
+        sparknet rag delete doc123
+        sparknet rag delete doc123 --force
+    """
+    if not force:
+        confirm = typer.confirm(f"Delete document '{document_id}' from index?")
+        if not confirm:
+            typer.echo("Cancelled.")
+            return
+
+    try:
+        from ..rag import get_vector_store, VectorStoreConfig
+
+        config = VectorStoreConfig(collection_name=collection)
+        store = get_vector_store(config)
+
+        deleted = store.delete_document(document_id)
+        typer.echo(f"Deleted {deleted} chunks for document: {document_id}")
+
+    except Exception as e:
+        typer.echo(f"Error deleting document: {e}", err=True)
+        raise typer.Exit(1)

src/document/__init__.py

@@ -0,0 +1,75 @@
+"""
+SPARKNET Document Intelligence Subsystem
+
+A comprehensive document processing pipeline for:
+- OCR with PaddleOCR and Tesseract
+- Layout detection and reading order reconstruction
+- Semantic chunking with grounding evidence
+- Document classification and field extraction
+- Extraction validation with Critic/Verifier
+
+Principles:
+- Processing is not understanding: OCR alone is insufficient
+- Every extraction includes evidence pointers (bbox, page, chunk_id)
+- Modular, pluggable components with clean interfaces
+- Abstain with evidence when confidence is low
+"""
+
+from .schemas.core import (
+    BoundingBox,
+    OCRRegion,
+    LayoutRegion,
+    LayoutType,
+    DocumentChunk,
+    ChunkType,
+    EvidenceRef,
+    ExtractionResult,
+    DocumentMetadata,
+    ProcessedDocument,
+)
+
+from .pipeline import (
+    PipelineConfig,
+    DocumentProcessor,
+    get_document_processor,
+    process_document,
+)
+
+from .validation import (
+    CriticConfig,
+    ValidationResult,
+    ExtractionCritic,
+    get_extraction_critic,
+    VerifierConfig,
+    VerificationResult,
+    EvidenceVerifier,
+    get_evidence_verifier,
+)
+
+__all__ = [
+    # Core schemas
+    "BoundingBox",
+    "OCRRegion",
+    "LayoutRegion",
+    "LayoutType",
+    "DocumentChunk",
+    "ChunkType",
+    "EvidenceRef",
+    "ExtractionResult",
+    "DocumentMetadata",
+    "ProcessedDocument",
+    # Pipeline
+    "PipelineConfig",
+    "DocumentProcessor",
+    "get_document_processor",
+    "process_document",
+    # Validation
+    "CriticConfig",
+    "ValidationResult",
+    "ExtractionCritic",
+    "get_extraction_critic",
+    "VerifierConfig",
+    "VerificationResult",
+    "EvidenceVerifier",
+    "get_evidence_verifier",
+]

src/document/chunking/__init__.py

@@ -0,0 +1,19 @@
+"""
+Document Chunking Module
+
+Creates semantic chunks from document content for retrieval and processing.
+"""
+
+from .chunker import (
+    ChunkerConfig,
+    DocumentChunker,
+    SemanticChunker,
+    get_document_chunker,
+)
+
+__all__ = [
+    "ChunkerConfig",
+    "DocumentChunker",
+    "SemanticChunker",
+    "get_document_chunker",
+]

src/document/chunking/chunker.py

@@ -0,0 +1,944 @@
+"""
+Document Chunker Implementation
+
+Creates semantic chunks from document content with bounding box tracking.
+Includes TableAwareChunker for preserving table structure in markdown format.
+"""
+
+import uuid
+import time
+import re
+from typing import List, Optional, Dict, Any, Tuple
+from dataclasses import dataclass
+from pydantic import BaseModel, Field
+from loguru import logger
+from collections import defaultdict
+
+from ..schemas.core import (
+    BoundingBox,
+    DocumentChunk,
+    ChunkType,
+    LayoutRegion,
+    LayoutType,
+    OCRRegion,
+)
+
+
+class ChunkerConfig(BaseModel):
+    """Configuration for document chunking."""
+    # Chunk size limits
+    max_chunk_chars: int = Field(
+        default=1000,
+        ge=100,
+        description="Maximum characters per chunk"
+    )
+    min_chunk_chars: int = Field(
+        default=50,
+        ge=10,
+        description="Minimum characters per chunk"
+    )
+    overlap_chars: int = Field(
+        default=100,
+        ge=0,
+        description="Character overlap between chunks"
+    )
+
+    # Chunking strategy
+    strategy: str = Field(
+        default="semantic",
+        description="Chunking strategy: semantic, fixed, or layout"
+    )
+    respect_layout: bool = Field(
+        default=True,
+        description="Respect layout region boundaries"
+    )
+    merge_small_regions: bool = Field(
+        default=True,
+        description="Merge small adjacent regions"
+    )
+
+    # Special element handling
+    chunk_tables: bool = Field(
+        default=True,
+        description="Create separate chunks for tables"
+    )
+    chunk_figures: bool = Field(
+        default=True,
+        description="Create separate chunks for figures"
+    )
+    include_captions: bool = Field(
+        default=True,
+        description="Include captions with figures/tables"
+    )
+
+    # Sentence handling
+    split_on_sentences: bool = Field(
+        default=True,
+        description="Split on sentence boundaries when possible"
+    )
+
+    # Table-aware chunking (FG-002)
+    preserve_table_structure: bool = Field(
+        default=True,
+        description="Preserve table structure as markdown with structured data"
+    )
+    table_row_threshold: float = Field(
+        default=10.0,
+        description="Y-coordinate threshold for grouping cells into rows"
+    )
+    table_col_threshold: float = Field(
+        default=20.0,
+        description="X-coordinate threshold for grouping cells into columns"
+    )
+    detect_table_headers: bool = Field(
+        default=True,
+        description="Attempt to detect and mark header rows"
+    )
+
+
+# Map layout types to chunk types
+LAYOUT_TO_CHUNK_TYPE = {
+    LayoutType.TEXT: ChunkType.TEXT,
+    LayoutType.TITLE: ChunkType.TITLE,
+    LayoutType.HEADING: ChunkType.HEADING,
+    LayoutType.PARAGRAPH: ChunkType.PARAGRAPH,
+    LayoutType.LIST: ChunkType.LIST_ITEM,
+    LayoutType.TABLE: ChunkType.TABLE,
+    LayoutType.FIGURE: ChunkType.FIGURE,
+    LayoutType.CHART: ChunkType.CHART,
+    LayoutType.FORMULA: ChunkType.FORMULA,
+    LayoutType.CAPTION: ChunkType.CAPTION,
+    LayoutType.FOOTNOTE: ChunkType.FOOTNOTE,
+    LayoutType.HEADER: ChunkType.HEADER,
+    LayoutType.FOOTER: ChunkType.FOOTER,
+}
+
+
+class DocumentChunker:
+    """Base class for document chunkers."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        self.config = config or ChunkerConfig()
+
+    def create_chunks(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout_regions: Optional[List[LayoutRegion]] = None,
+        document_id: str = "",
+        source_path: Optional[str] = None,
+    ) -> List[DocumentChunk]:
+        """
+        Create chunks from OCR and layout regions.
+
+        Args:
+            ocr_regions: OCR text regions
+            layout_regions: Optional layout regions
+            document_id: Parent document ID
+            source_path: Source file path
+
+        Returns:
+            List of DocumentChunk
+        """
+        raise NotImplementedError
+
+
+class SemanticChunker(DocumentChunker):
+    """
+    Semantic chunker that respects document structure.
+
+    Creates chunks based on:
+    - Layout region boundaries
+    - Semantic coherence (paragraphs, sections)
+    - Size constraints with overlap
+    """
+
+    def create_chunks(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout_regions: Optional[List[LayoutRegion]] = None,
+        document_id: str = "",
+        source_path: Optional[str] = None,
+    ) -> List[DocumentChunk]:
+        """Create semantic chunks from document content."""
+        if not ocr_regions:
+            return []
+
+        start_time = time.time()
+        chunks = []
+        chunk_index = 0
+
+        if layout_regions and self.config.respect_layout:
+            # Use layout regions to guide chunking
+            chunks = self._chunk_by_layout(
+                ocr_regions, layout_regions, document_id, source_path
+            )
+        else:
+            # Fall back to text-based chunking
+            chunks = self._chunk_by_text(
+                ocr_regions, document_id, source_path
+            )
+
+        # Assign sequence indices
+        for i, chunk in enumerate(chunks):
+            chunk.sequence_index = i
+
+        logger.debug(
+            f"Created {len(chunks)} chunks in "
+            f"{(time.time() - start_time) * 1000:.1f}ms"
+        )
+
+        return chunks
+
+    def _chunk_by_layout(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout_regions: List[LayoutRegion],
+        document_id: str,
+        source_path: Optional[str],
+    ) -> List[DocumentChunk]:
+        """Create chunks based on layout regions."""
+        chunks = []
+
+        # Sort layout regions by reading order
+        sorted_layouts = sorted(
+            layout_regions,
+            key=lambda r: (r.reading_order or 0, r.bbox.y_min, r.bbox.x_min)
+        )
+
+        for layout in sorted_layouts:
+            # Get OCR regions within this layout region
+            contained_ocr = self._get_contained_ocr(ocr_regions, layout)
+
+            if not contained_ocr:
+                continue
+
+            # Determine chunk type
+            chunk_type = LAYOUT_TO_CHUNK_TYPE.get(layout.type, ChunkType.TEXT)
+
+            # Handle special types differently
+            if layout.type == LayoutType.TABLE and self.config.chunk_tables:
+                chunk = self._create_table_chunk(
+                    contained_ocr, layout, document_id, source_path
+                )
+                chunks.append(chunk)
+
+            elif layout.type in (LayoutType.FIGURE, LayoutType.CHART) and self.config.chunk_figures:
+                chunk = self._create_figure_chunk(
+                    contained_ocr, layout, document_id, source_path
+                )
+                chunks.append(chunk)
+
+            else:
+                # Regular text chunk - may need splitting
+                text_chunks = self._create_text_chunks(
+                    contained_ocr, layout, chunk_type, document_id, source_path
+                )
+                chunks.extend(text_chunks)
+
+        return chunks
+
+    def _chunk_by_text(
+        self,
+        ocr_regions: List[OCRRegion],
+        document_id: str,
+        source_path: Optional[str],
+    ) -> List[DocumentChunk]:
+        """Create chunks from text without layout guidance."""
+        chunks = []
+
+        # Sort by reading order (y then x)
+        sorted_regions = sorted(
+            ocr_regions,
+            key=lambda r: (r.page, r.bbox.y_min, r.bbox.x_min)
+        )
+
+        # Group by page
+        pages: Dict[int, List[OCRRegion]] = {}
+        for r in sorted_regions:
+            if r.page not in pages:
+                pages[r.page] = []
+            pages[r.page].append(r)
+
+        # Process each page
+        for page_num in sorted(pages.keys()):
+            page_regions = pages[page_num]
+            page_chunks = self._split_text_regions(
+                page_regions, document_id, source_path, page_num
+            )
+            chunks.extend(page_chunks)
+
+        return chunks
+
+    def _get_contained_ocr(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout: LayoutRegion,
+    ) -> List[OCRRegion]:
+        """Get OCR regions contained within a layout region."""
+        contained = []
+        for ocr in ocr_regions:
+            if ocr.page == layout.page:
+                # Check if OCR region overlaps significantly with layout
+                iou = layout.bbox.iou(ocr.bbox)
+                if iou > 0.3 or layout.bbox.contains(ocr.bbox):
+                    contained.append(ocr)
+        return contained
+
+    def _create_text_chunks(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout: LayoutRegion,
+        chunk_type: ChunkType,
+        document_id: str,
+        source_path: Optional[str],
+    ) -> List[DocumentChunk]:
+        """Create text chunks from OCR regions, splitting if needed."""
+        chunks = []
+
+        # Combine text
+        text = " ".join(r.text for r in ocr_regions)
+
+        # Calculate average confidence
+        avg_conf = sum(r.confidence for r in ocr_regions) / len(ocr_regions)
+
+        # Check if splitting is needed
+        if len(text) <= self.config.max_chunk_chars:
+            # Single chunk
+            chunk = DocumentChunk(
+                chunk_id=f"{document_id}_{uuid.uuid4().hex[:8]}",
+                chunk_type=chunk_type,
+                text=text,
+                bbox=layout.bbox,
+                page=layout.page,
+                document_id=document_id,
+                source_path=source_path,
+                sequence_index=0,
+                confidence=avg_conf,
+            )
+            chunks.append(chunk)
+        else:
+            # Split into multiple chunks
+            split_chunks = self._split_text(
+                text, layout.bbox, layout.page, chunk_type,
+                document_id, source_path, avg_conf
+            )
+            chunks.extend(split_chunks)
+
+        return chunks
+
+    def _split_text(
+        self,
+        text: str,
+        bbox: BoundingBox,
+        page: int,
+        chunk_type: ChunkType,
+        document_id: str,
+        source_path: Optional[str],
+        confidence: float,
+    ) -> List[DocumentChunk]:
+        """Split long text into multiple chunks with overlap."""
+        chunks = []
+        max_chars = self.config.max_chunk_chars
+        overlap = self.config.overlap_chars
+
+        # Split on sentences if enabled
+        if self.config.split_on_sentences:
+            sentences = self._split_sentences(text)
+        else:
+            sentences = [text]
+
+        current_text = ""
+        for sentence in sentences:
+            if len(current_text) + len(sentence) > max_chars and current_text:
+                # Create chunk
+                chunk = DocumentChunk(
+                    chunk_id=f"{document_id}_{uuid.uuid4().hex[:8]}",
+                    chunk_type=chunk_type,
+                    text=current_text.strip(),
+                    bbox=bbox,
+                    page=page,
+                    document_id=document_id,
+                    source_path=source_path,
+                    sequence_index=len(chunks),
+                    confidence=confidence,
+                )
+                chunks.append(chunk)
+
+                # Start new chunk with overlap
+                if overlap > 0:
+                    overlap_text = current_text[-overlap:] if len(current_text) > overlap else current_text
+                    current_text = overlap_text + " " + sentence
+                else:
+                    current_text = sentence
+            else:
+                current_text += " " + sentence if current_text else sentence
+
+        # Don't forget the last chunk
+        if current_text.strip():
+            chunk = DocumentChunk(
+                chunk_id=f"{document_id}_{uuid.uuid4().hex[:8]}",
+                chunk_type=chunk_type,
+                text=current_text.strip(),
+                bbox=bbox,
+                page=page,
+                document_id=document_id,
+                source_path=source_path,
+                sequence_index=len(chunks),
+                confidence=confidence,
+            )
+            chunks.append(chunk)
+
+        return chunks
+
+    def _split_sentences(self, text: str) -> List[str]:
+        """Split text into sentences."""
+        # Simple sentence splitting
+        import re
+        sentences = re.split(r'(?<=[.!?])\s+', text)
+        return [s.strip() for s in sentences if s.strip()]
+
+    def _create_table_chunk(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout: LayoutRegion,
+        document_id: str,
+        source_path: Optional[str],
+    ) -> DocumentChunk:
+        """
+        Create a chunk for table content with structure preservation.
+
+        Enhanced table handling (FG-002):
+        - Reconstructs table structure from OCR regions
+        - Generates markdown table representation
+        - Stores structured data for SQL-like queries
+        - Detects and marks header rows
+        """
+        if not ocr_regions:
+            return DocumentChunk(
+                chunk_id=f"{document_id}_table_{uuid.uuid4().hex[:8]}",
+                chunk_type=ChunkType.TABLE,
+                text="[Empty Table]",
+                bbox=layout.bbox,
+                page=layout.page,
+                document_id=document_id,
+                source_path=source_path,
+                sequence_index=0,
+                confidence=0.0,
+                extra=layout.extra or {},
+            )
+
+        avg_conf = sum(r.confidence for r in ocr_regions) / len(ocr_regions)
+
+        # Check if we should preserve table structure
+        if not self.config.preserve_table_structure:
+            # Fall back to simple pipe-separated format
+            text = " | ".join(r.text for r in ocr_regions)
+            return DocumentChunk(
+                chunk_id=f"{document_id}_table_{uuid.uuid4().hex[:8]}",
+                chunk_type=ChunkType.TABLE,
+                text=text,
+                bbox=layout.bbox,
+                page=layout.page,
+                document_id=document_id,
+                source_path=source_path,
+                sequence_index=0,
+                confidence=avg_conf,
+                extra=layout.extra or {},
+            )
+
+        # Reconstruct table structure from spatial positions
+        table_data = self._reconstruct_table_structure(ocr_regions)
+
+        # Generate markdown representation
+        markdown_table = self._table_to_markdown(
+            table_data["rows"],
+            table_data["headers"],
+            table_data["has_header"]
+        )
+
+        # Create rich metadata for structured queries
+        table_extra = {
+            **(layout.extra or {}),
+            "table_structure": {
+                "row_count": table_data["row_count"],
+                "col_count": table_data["col_count"],
+                "has_header": table_data["has_header"],
+                "headers": table_data["headers"],
+                "cells": table_data["cells"],  # 2D list of cell values
+                "cell_positions": table_data["cell_positions"],  # For highlighting
+            },
+            "format": "markdown",
+            "searchable_text": table_data["searchable_text"],
+        }
+
+        return DocumentChunk(
+            chunk_id=f"{document_id}_table_{uuid.uuid4().hex[:8]}",
+            chunk_type=ChunkType.TABLE,
+            text=markdown_table,
+            bbox=layout.bbox,
+            page=layout.page,
+            document_id=document_id,
+            source_path=source_path,
+            sequence_index=0,
+            confidence=avg_conf,
+            extra=table_extra,
+        )
+
+    def _reconstruct_table_structure(
+        self,
+        ocr_regions: List[OCRRegion],
+    ) -> Dict[str, Any]:
+        """
+        Reconstruct table structure from OCR regions based on spatial positions.
+
+        Groups OCR regions into rows and columns by analyzing their bounding boxes.
+        Returns structured table data for markdown generation and queries.
+        """
+        if not ocr_regions:
+            return {
+                "rows": [],
+                "headers": [],
+                "has_header": False,
+                "row_count": 0,
+                "col_count": 0,
+                "cells": [],
+                "cell_positions": [],
+                "searchable_text": "",
+            }
+
+        # Sort regions by vertical position (y_min) then horizontal (x_min)
+        sorted_regions = sorted(
+            ocr_regions,
+            key=lambda r: (r.bbox.y_min, r.bbox.x_min)
+        )
+
+        # Group into rows based on y-coordinate proximity
+        row_threshold = self.config.table_row_threshold
+        rows: List[List[OCRRegion]] = []
+        current_row: List[OCRRegion] = []
+        current_y = None
+
+        for region in sorted_regions:
+            if current_y is None:
+                current_y = region.bbox.y_min
+                current_row.append(region)
+            elif abs(region.bbox.y_min - current_y) <= row_threshold:
+                current_row.append(region)
+            else:
+                if current_row:
+                    # Sort row by x position
+                    current_row.sort(key=lambda r: r.bbox.x_min)
+                    rows.append(current_row)
+                current_row = [region]
+                current_y = region.bbox.y_min
+
+        # Don't forget the last row
+        if current_row:
+            current_row.sort(key=lambda r: r.bbox.x_min)
+            rows.append(current_row)
+
+        # Determine column structure
+        # Find consistent column boundaries across all rows
+        col_positions = self._detect_column_positions(rows)
+        num_cols = len(col_positions) if col_positions else max(len(row) for row in rows)
+
+        # Build structured cell data
+        cells: List[List[str]] = []
+        cell_positions: List[List[Dict[str, Any]]] = []
+
+        for row in rows:
+            row_cells = self._assign_cells_to_columns(row, col_positions, num_cols)
+            cells.append([cell["text"] for cell in row_cells])
+            cell_positions.append([{
+                "text": cell["text"],
+                "bbox": cell["bbox"],
+                "confidence": cell["confidence"]
+            } for cell in row_cells])
+
+        # Detect header row
+        has_header = False
+        headers: List[str] = []
+
+        if self.config.detect_table_headers and len(cells) > 0:
+            has_header, headers = self._detect_header_row(cells, rows)
+
+        # Build searchable text (for vector embedding)
+        searchable_parts = []
+        for i, row in enumerate(cells):
+            if has_header and i == 0:
+                searchable_parts.append("Headers: " + ", ".join(row))
+            else:
+                if has_header and headers:
+                    # Include header context for each value
+                    for j, cell in enumerate(row):
+                        if j < len(headers) and headers[j]:
+                            searchable_parts.append(f"{headers[j]}: {cell}")
+                        else:
+                            searchable_parts.append(cell)
+                else:
+                    searchable_parts.extend(row)
+
+        return {
+            "rows": cells,
+            "headers": headers,
+            "has_header": has_header,
+            "row_count": len(cells),
+            "col_count": num_cols,
+            "cells": cells,
+            "cell_positions": cell_positions,
+            "searchable_text": " | ".join(searchable_parts),
+        }
+
+    def _detect_column_positions(
+        self,
+        rows: List[List[OCRRegion]],
+    ) -> List[Tuple[float, float]]:
+        """
+        Detect consistent column boundaries from table rows.
+
+        Returns list of (x_start, x_end) tuples for each column.
+        """
+        if not rows:
+            return []
+
+        col_threshold = self.config.table_col_threshold
+
+        # Collect all x positions
+        all_x_starts = []
+        for row in rows:
+            for region in row:
+                all_x_starts.append(region.bbox.x_min)
+
+        if not all_x_starts:
+            return []
+
+        # Cluster x positions into columns
+        all_x_starts.sort()
+        columns = []
+        current_col_start = all_x_starts[0]
+        current_col_regions = [all_x_starts[0]]
+
+        for x in all_x_starts[1:]:
+            if x - current_col_regions[-1] <= col_threshold:
+                current_col_regions.append(x)
+            else:
+                # Calculate column boundary
+                col_center = sum(current_col_regions) / len(current_col_regions)
+                columns.append(col_center)
+                current_col_regions = [x]
+
+        # Last column
+        if current_col_regions:
+            col_center = sum(current_col_regions) / len(current_col_regions)
+            columns.append(col_center)
+
+        # Convert to column ranges
+        col_ranges = []
+        for i, col_x in enumerate(columns):
+            x_start = col_x - col_threshold
+            if i < len(columns) - 1:
+                x_end = (col_x + columns[i + 1]) / 2
+            else:
+                x_end = col_x + col_threshold * 3  # Extend last column
+            col_ranges.append((x_start, x_end))
+
+        return col_ranges
+
+    def _assign_cells_to_columns(
+        self,
+        row_regions: List[OCRRegion],
+        col_positions: List[Tuple[float, float]],
+        num_cols: int,
+    ) -> List[Dict[str, Any]]:
+        """
+        Assign OCR regions in a row to their respective columns.
+        Handles merged cells and missing cells.
+        """
+        # Initialize empty cells for each column
+        row_cells = [
+            {"text": "", "bbox": None, "confidence": 0.0}
+            for _ in range(num_cols)
+        ]
+
+        if not col_positions:
+            # No column positions detected, just use order
+            for i, region in enumerate(row_regions):
+                if i < num_cols:
+                    row_cells[i] = {
+                        "text": region.text.strip(),
+                        "bbox": region.bbox.to_xyxy(),
+                        "confidence": region.confidence,
+                    }
+            return row_cells
+
+        # Assign regions to columns based on x position
+        for region in row_regions:
+            region_x = region.bbox.x_min
+            assigned = False
+
+            for col_idx, (x_start, x_end) in enumerate(col_positions):
+                if x_start <= region_x <= x_end:
+                    # Append to existing cell (handle multi-line cells)
+                    if row_cells[col_idx]["text"]:
+                        row_cells[col_idx]["text"] += " " + region.text.strip()
+                    else:
+                        row_cells[col_idx]["text"] = region.text.strip()
+                    row_cells[col_idx]["bbox"] = region.bbox.to_xyxy()
+                    row_cells[col_idx]["confidence"] = max(
+                        row_cells[col_idx]["confidence"],
+                        region.confidence
+                    )
+                    assigned = True
+                    break
+
+            # If not assigned, put in nearest column
+            if not assigned:
+                min_dist = float("inf")
+                nearest_col = 0
+                for col_idx, (x_start, x_end) in enumerate(col_positions):
+                    col_center = (x_start + x_end) / 2
+                    dist = abs(region_x - col_center)
+                    if dist < min_dist:
+                        min_dist = dist
+                        nearest_col = col_idx
+
+                if row_cells[nearest_col]["text"]:
+                    row_cells[nearest_col]["text"] += " " + region.text.strip()
+                else:
+                    row_cells[nearest_col]["text"] = region.text.strip()
+                row_cells[nearest_col]["bbox"] = region.bbox.to_xyxy()
+                row_cells[nearest_col]["confidence"] = region.confidence
+
+        return row_cells
+
+    def _detect_header_row(
+        self,
+        cells: List[List[str]],
+        rows: List[List[OCRRegion]],
+    ) -> Tuple[bool, List[str]]:
+        """
+        Detect if the first row is a header row.
+
+        Heuristics used:
+        - First row contains non-numeric text
+        - First row text is shorter (labels vs data)
+        - First row has distinct formatting (if available)
+        """
+        if not cells or len(cells) < 2:
+            return False, []
+
+        first_row = cells[0]
+        other_rows = cells[1:]
+
+        # Check if first row is mostly non-numeric
+        first_row_numeric_count = sum(
+            1 for cell in first_row
+            if cell and self._is_numeric(cell)
+        )
+        first_row_text_ratio = (len(first_row) - first_row_numeric_count) / max(len(first_row), 1)
+
+        # Check if other rows are more numeric
+        other_numeric_ratios = []
+        for row in other_rows:
+            if row:
+                numeric_count = sum(1 for cell in row if cell and self._is_numeric(cell))
+                other_numeric_ratios.append(numeric_count / max(len(row), 1))
+
+        avg_other_numeric = sum(other_numeric_ratios) / max(len(other_numeric_ratios), 1)
+
+        # Header detection: first row is text-heavy, others are more numeric
+        is_header = (
+            first_row_text_ratio > 0.5 and
+            (avg_other_numeric > first_row_text_ratio * 0.5 or first_row_text_ratio > 0.8)
+        )
+
+        # Also consider: shorter cell lengths in first row (labels are usually shorter)
+        first_row_avg_len = sum(len(cell) for cell in first_row) / max(len(first_row), 1)
+        other_avg_lens = [
+            sum(len(cell) for cell in row) / max(len(row), 1)
+            for row in other_rows
+        ]
+        avg_other_len = sum(other_avg_lens) / max(len(other_avg_lens), 1)
+
+        if first_row_avg_len < avg_other_len * 0.8:
+            is_header = True
+
+        return is_header, first_row if is_header else []
+
+    def _is_numeric(self, text: str) -> bool:
+        """Check if text is primarily numeric (including currency, percentages)."""
+        cleaned = re.sub(r'[$€£¥%,.\s\-+()]', '', text)
+        return cleaned.isdigit() if cleaned else False
+
+    def _table_to_markdown(
+        self,
+        rows: List[List[str]],
+        headers: List[str],
+        has_header: bool,
+    ) -> str:
+        """
+        Convert table data to markdown format.
+
+        Creates a properly formatted markdown table with:
+        - Header row (if detected)
+        - Separator row
+        - Data rows
+        """
+        if not rows:
+            return "[Empty Table]"
+
+        # Determine column count
+        num_cols = max(len(row) for row in rows) if rows else 0
+        if num_cols == 0:
+            return "[Empty Table]"
+
+        # Normalize all rows to same column count
+        normalized_rows = []
+        for row in rows:
+            normalized = row + [""] * (num_cols - len(row))
+            normalized_rows.append(normalized)
+
+        # Build markdown lines
+        md_lines = []
+
+        if has_header and headers:
+            # Use detected headers
+            header_line = "| " + " | ".join(headers + [""] * (num_cols - len(headers))) + " |"
+            separator = "| " + " | ".join(["---"] * num_cols) + " |"
+            md_lines.append(header_line)
+            md_lines.append(separator)
+            data_rows = normalized_rows[1:]
+        else:
+            # No header - create generic headers
+            generic_headers = [f"Col{i+1}" for i in range(num_cols)]
+            header_line = "| " + " | ".join(generic_headers) + " |"
+            separator = "| " + " | ".join(["---"] * num_cols) + " |"
+            md_lines.append(header_line)
+            md_lines.append(separator)
+            data_rows = normalized_rows
+
+        # Add data rows
+        for row in data_rows:
+            # Escape pipe characters in cell content
+            escaped_row = [cell.replace("|", "\\|") for cell in row]
+            row_line = "| " + " | ".join(escaped_row) + " |"
+            md_lines.append(row_line)
+
+        return "\n".join(md_lines)
+
+    def _create_figure_chunk(
+        self,
+        ocr_regions: List[OCRRegion],
+        layout: LayoutRegion,
+        document_id: str,
+        source_path: Optional[str],
+    ) -> DocumentChunk:
+        """Create a chunk for figure/chart content."""
+        # For figures, text is usually caption
+        text = " ".join(r.text for r in ocr_regions) if ocr_regions else "[Figure]"
+        avg_conf = sum(r.confidence for r in ocr_regions) / len(ocr_regions) if ocr_regions else 0.5
+
+        chunk_type = ChunkType.CHART if layout.type == LayoutType.CHART else ChunkType.FIGURE
+
+        return DocumentChunk(
+            chunk_id=f"{document_id}_{chunk_type.value}_{uuid.uuid4().hex[:8]}",
+            chunk_type=chunk_type,
+            text=text,
+            bbox=layout.bbox,
+            page=layout.page,
+            document_id=document_id,
+            source_path=source_path,
+            sequence_index=0,
+            confidence=avg_conf,
+            caption=text if ocr_regions else None,
+        )
+
+    def _split_text_regions(
+        self,
+        ocr_regions: List[OCRRegion],
+        document_id: str,
+        source_path: Optional[str],
+        page_num: int,
+    ) -> List[DocumentChunk]:
+        """Split OCR regions into chunks without layout guidance."""
+        if not ocr_regions:
+            return []
+
+        chunks = []
+        current_text = ""
+        current_regions = []
+
+        for region in ocr_regions:
+            if len(current_text) + len(region.text) > self.config.max_chunk_chars:
+                if current_regions:
+                    # Create chunk from accumulated regions
+                    chunk = self._create_chunk_from_regions(
+                        current_regions, document_id, source_path, page_num, len(chunks)
+                    )
+                    chunks.append(chunk)
+
+                current_text = region.text
+                current_regions = [region]
+            else:
+                current_text += " " + region.text
+                current_regions.append(region)
+
+        # Final chunk
+        if current_regions:
+            chunk = self._create_chunk_from_regions(
+                current_regions, document_id, source_path, page_num, len(chunks)
+            )
+            chunks.append(chunk)
+
+        return chunks
+
+    def _create_chunk_from_regions(
+        self,
+        regions: List[OCRRegion],
+        document_id: str,
+        source_path: Optional[str],
+        page_num: int,
+        sequence_index: int,
+    ) -> DocumentChunk:
+        """Create a chunk from a list of OCR regions."""
+        text = " ".join(r.text for r in regions)
+        avg_conf = sum(r.confidence for r in regions) / len(regions)
+
+        # Compute bounding box
+        x_min = min(r.bbox.x_min for r in regions)
+        y_min = min(r.bbox.y_min for r in regions)
+        x_max = max(r.bbox.x_max for r in regions)
+        y_max = max(r.bbox.y_max for r in regions)
+
+        bbox = BoundingBox(
+            x_min=x_min, y_min=y_min,
+            x_max=x_max, y_max=y_max,
+            normalized=False,
+        )
+
+        return DocumentChunk(
+            chunk_id=f"{document_id}_{uuid.uuid4().hex[:8]}",
+            chunk_type=ChunkType.TEXT,
+            text=text,
+            bbox=bbox,
+            page=page_num,
+            document_id=document_id,
+            source_path=source_path,
+            sequence_index=sequence_index,
+            confidence=avg_conf,
+        )
+
+
+# Factory
+_document_chunker: Optional[DocumentChunker] = None
+
+
+def get_document_chunker(
+    config: Optional[ChunkerConfig] = None,
+) -> DocumentChunker:
+    """Get or create singleton document chunker."""
+    global _document_chunker
+    if _document_chunker is None:
+        config = config or ChunkerConfig()
+        _document_chunker = SemanticChunker(config)
+    return _document_chunker

src/document/grounding/__init__.py

@@ -0,0 +1,21 @@
+"""
+Document Grounding Module
+
+Provides evidence packaging and visual grounding for extracted information.
+"""
+
+from .evidence import (
+    GroundingConfig,
+    EvidenceBuilder,
+    create_evidence_ref,
+    crop_region_image,
+    encode_image_base64,
+)
+
+__all__ = [
+    "GroundingConfig",
+    "EvidenceBuilder",
+    "create_evidence_ref",
+    "crop_region_image",
+    "encode_image_base64",
+]

src/document/grounding/evidence.py

@@ -0,0 +1,365 @@
+"""
+Evidence Builder for Document Grounding
+
+Creates evidence references for extracted information.
+Handles image cropping and base64 encoding.
+"""
+
+import base64
+import io
+from typing import List, Optional, Dict, Any, Tuple
+from pydantic import BaseModel, Field
+import numpy as np
+from PIL import Image
+from loguru import logger
+
+from ..schemas.core import (
+    BoundingBox,
+    DocumentChunk,
+    EvidenceRef,
+    OCRRegion,
+)
+
+
+class GroundingConfig(BaseModel):
+    """Configuration for grounding and evidence generation."""
+    # Image cropping
+    include_images: bool = Field(
+        default=True,
+        description="Include cropped images in evidence"
+    )
+    crop_padding: int = Field(
+        default=10,
+        ge=0,
+        description="Padding around crop regions in pixels"
+    )
+    max_image_size: int = Field(
+        default=512,
+        ge=64,
+        description="Maximum dimension for cropped images"
+    )
+    image_format: str = Field(
+        default="PNG",
+        description="Image format for encoding (PNG/JPEG)"
+    )
+    image_quality: int = Field(
+        default=85,
+        ge=1,
+        le=100,
+        description="JPEG quality if using JPEG format"
+    )
+
+    # Snippet settings
+    max_snippet_length: int = Field(
+        default=200,
+        ge=50,
+        description="Maximum length of text snippets"
+    )
+    include_context: bool = Field(
+        default=True,
+        description="Include surrounding context in snippets"
+    )
+
+
+def crop_region_image(
+    image: np.ndarray,
+    bbox: BoundingBox,
+    padding: int = 10,
+    max_size: Optional[int] = None,
+) -> np.ndarray:
+    """
+    Crop a region from an image.
+
+    Args:
+        image: Source image (RGB, HWC format)
+        bbox: Bounding box to crop
+        padding: Padding around the crop
+        max_size: Maximum dimension (will resize if larger)
+
+    Returns:
+        Cropped image as numpy array
+    """
+    height, width = image.shape[:2]
+
+    # Get coordinates with padding
+    x1 = max(0, int(bbox.x_min) - padding)
+    y1 = max(0, int(bbox.y_min) - padding)
+    x2 = min(width, int(bbox.x_max) + padding)
+    y2 = min(height, int(bbox.y_max) + padding)
+
+    # Crop
+    cropped = image[y1:y2, x1:x2]
+
+    # Resize if needed
+    if max_size and max(cropped.shape[:2]) > max_size:
+        pil_img = Image.fromarray(cropped)
+        pil_img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
+        cropped = np.array(pil_img)
+
+    return cropped
+
+
+def encode_image_base64(
+    image: np.ndarray,
+    format: str = "PNG",
+    quality: int = 85,
+) -> str:
+    """
+    Encode image to base64 string.
+
+    Args:
+        image: Image as numpy array
+        format: Image format (PNG/JPEG)
+        quality: JPEG quality if applicable
+
+    Returns:
+        Base64-encoded string
+    """
+    pil_img = Image.fromarray(image)
+
+    # Convert to RGB if needed
+    if pil_img.mode != "RGB":
+        pil_img = pil_img.convert("RGB")
+
+    # Encode
+    buffer = io.BytesIO()
+    if format.upper() == "JPEG":
+        pil_img.save(buffer, format="JPEG", quality=quality)
+    else:
+        pil_img.save(buffer, format="PNG")
+
+    buffer.seek(0)
+    encoded = base64.b64encode(buffer.read()).decode("utf-8")
+
+    return encoded
+
+
+def create_evidence_ref(
+    chunk: DocumentChunk,
+    source_type: str = "text",
+    snippet: Optional[str] = None,
+    confidence: float = 1.0,
+    image: Optional[np.ndarray] = None,
+    config: Optional[GroundingConfig] = None,
+) -> EvidenceRef:
+    """
+    Create an evidence reference from a document chunk.
+
+    Args:
+        chunk: Source chunk
+        source_type: Type of source (text/table/figure)
+        snippet: Optional specific snippet (defaults to chunk text)
+        confidence: Confidence score
+        image: Optional page image for cropping
+        config: Grounding configuration
+
+    Returns:
+        EvidenceRef instance
+    """
+    config = config or GroundingConfig()
+
+    # Create snippet
+    if snippet is None:
+        snippet = chunk.text[:config.max_snippet_length]
+        if len(chunk.text) > config.max_snippet_length:
+            snippet += "..."
+
+    # Create base evidence
+    evidence = EvidenceRef(
+        chunk_id=chunk.chunk_id,
+        page=chunk.page,
+        bbox=chunk.bbox,
+        source_type=source_type,
+        snippet=snippet,
+        confidence=confidence,
+    )
+
+    # Add image if available and configured
+    if image is not None and config.include_images:
+        try:
+            cropped = crop_region_image(
+                image,
+                chunk.bbox,
+                padding=config.crop_padding,
+                max_size=config.max_image_size,
+            )
+            evidence.image_base64 = encode_image_base64(
+                cropped,
+                format=config.image_format,
+                quality=config.image_quality,
+            )
+        except Exception as e:
+            logger.warning(f"Failed to crop evidence image: {e}")
+
+    return evidence
+
+
+class EvidenceBuilder:
+    """
+    Builder for creating evidence references.
+
+    Handles:
+    - Evidence from chunks
+    - Evidence from OCR regions
+    - Evidence aggregation
+    - Image cropping and encoding
+    """
+
+    def __init__(self, config: Optional[GroundingConfig] = None):
+        """Initialize evidence builder."""
+        self.config = config or GroundingConfig()
+
+    def from_chunk(
+        self,
+        chunk: DocumentChunk,
+        image: Optional[np.ndarray] = None,
+        additional_context: Optional[str] = None,
+    ) -> EvidenceRef:
+        """
+        Create evidence reference from a chunk.
+
+        Args:
+            chunk: Source chunk
+            image: Optional page image for visual evidence
+            additional_context: Optional additional context
+
+        Returns:
+            EvidenceRef
+        """
+        # Determine source type
+        source_type = chunk.chunk_type.value
+
+        # Build snippet with optional context
+        snippet = chunk.text[:self.config.max_snippet_length]
+        if additional_context:
+            snippet = f"{additional_context}\n{snippet}"
+        if len(chunk.text) > self.config.max_snippet_length:
+            snippet += "..."
+
+        return create_evidence_ref(
+            chunk=chunk,
+            source_type=source_type,
+            snippet=snippet,
+            confidence=chunk.confidence,
+            image=image,
+            config=self.config,
+        )
+
+    def from_ocr_region(
+        self,
+        region: OCRRegion,
+        chunk_id: str,
+        document_id: str,
+        image: Optional[np.ndarray] = None,
+    ) -> EvidenceRef:
+        """
+        Create evidence reference from an OCR region.
+
+        Args:
+            region: OCR region
+            chunk_id: ID to assign
+            document_id: Parent document ID
+            image: Optional page image
+
+        Returns:
+            EvidenceRef
+        """
+        # Create a temporary chunk for the evidence
+        from ..schemas.core import DocumentChunk, ChunkType
+
+        chunk = DocumentChunk(
+            chunk_id=chunk_id,
+            chunk_type=ChunkType.TEXT,
+            text=region.text,
+            bbox=region.bbox,
+            page=region.page,
+            document_id=document_id,
+            source_path=None,
+            sequence_index=0,
+            confidence=region.confidence,
+        )
+
+        return self.from_chunk(chunk, image)
+
+    def aggregate_evidence(
+        self,
+        evidence_list: List[EvidenceRef],
+        combine_snippets: bool = True,
+    ) -> List[EvidenceRef]:
+        """
+        Aggregate and deduplicate evidence references.
+
+        Args:
+            evidence_list: List of evidence references
+            combine_snippets: Whether to combine snippets from same chunk
+
+        Returns:
+            Deduplicated evidence list
+        """
+        if not evidence_list:
+            return []
+
+        # Group by chunk_id
+        by_chunk: Dict[str, List[EvidenceRef]] = {}
+        for ev in evidence_list:
+            if ev.chunk_id not in by_chunk:
+                by_chunk[ev.chunk_id] = []
+            by_chunk[ev.chunk_id].append(ev)
+
+        # Combine or select best
+        result = []
+        for chunk_id, evidences in by_chunk.items():
+            if len(evidences) == 1:
+                result.append(evidences[0])
+            else:
+                # Take highest confidence, combine snippets
+                best = max(evidences, key=lambda e: e.confidence)
+                if combine_snippets:
+                    all_snippets = list(set(e.snippet for e in evidences))
+                    combined = " ... ".join(all_snippets[:3])
+                    best = EvidenceRef(
+                        chunk_id=best.chunk_id,
+                        page=best.page,
+                        bbox=best.bbox,
+                        source_type=best.source_type,
+                        snippet=combined[:self.config.max_snippet_length],
+                        confidence=best.confidence,
+                        image_base64=best.image_base64,
+                    )
+                result.append(best)
+
+        # Sort by page and position
+        result.sort(key=lambda e: (e.page, e.bbox.y_min, e.bbox.x_min))
+
+        return result
+
+    def create_grounding_context(
+        self,
+        evidence_list: List[EvidenceRef],
+        include_images: bool = False,
+    ) -> str:
+        """
+        Create a text context from evidence for LLM prompting.
+
+        Args:
+            evidence_list: Evidence references
+            include_images: Whether to include image markers
+
+        Returns:
+            Formatted context string
+        """
+        if not evidence_list:
+            return ""
+
+        lines = ["Evidence from document:"]
+        for i, ev in enumerate(evidence_list, 1):
+            lines.append(
+                f"\n[{i}] Page {ev.page + 1}, {ev.source_type} "
+                f"(confidence: {ev.confidence:.2f}):"
+            )
+            lines.append(f'   "{ev.snippet}"')
+
+            if include_images and ev.image_base64:
+                lines.append("   [Image available]")
+
+        return "\n".join(lines)

src/document/io/__init__.py

@@ -0,0 +1,28 @@
+"""
+Document I/O Module
+
+Handles loading, rendering, and caching of PDF and image documents.
+"""
+
+from .loader import (
+    DocumentLoader,
+    load_document,
+    load_pdf,
+    load_image,
+    render_page,
+)
+
+from .cache import (
+    DocumentCache,
+    get_document_cache,
+)
+
+__all__ = [
+    "DocumentLoader",
+    "load_document",
+    "load_pdf",
+    "load_image",
+    "render_page",
+    "DocumentCache",
+    "get_document_cache",
+]

src/document/io/cache.py

@@ -0,0 +1,268 @@
+"""
+Document Cache
+
+Caches rendered page images and document metadata for performance.
+"""
+
+import hashlib
+import os
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from dataclasses import dataclass
+from datetime import datetime, timedelta
+from loguru import logger
+
+import numpy as np
+from PIL import Image
+
+from cachetools import TTLCache, LRUCache
+
+
+@dataclass
+class CacheEntry:
+    """A cached page image entry."""
+    document_id: str
+    page_number: int
+    dpi: int
+    image: np.ndarray
+    created_at: datetime
+    size_bytes: int
+
+
+class DocumentCache:
+    """
+    In-memory cache for rendered document pages.
+    Uses LRU eviction with optional disk persistence.
+    """
+
+    def __init__(
+        self,
+        max_pages: int = 100,
+        max_memory_mb: int = 1024,
+        ttl_seconds: int = 3600,
+        disk_cache_dir: Optional[str] = None,
+    ):
+        """
+        Initialize document cache.
+
+        Args:
+            max_pages: Maximum number of pages to cache in memory
+            max_memory_mb: Maximum memory usage in MB
+            ttl_seconds: Time-to-live for cache entries
+            disk_cache_dir: Optional directory for disk caching
+        """
+        self.max_pages = max_pages
+        self.max_memory_mb = max_memory_mb
+        self.ttl_seconds = ttl_seconds
+        self.disk_cache_dir = disk_cache_dir
+
+        # In-memory cache
+        self._cache: TTLCache = TTLCache(maxsize=max_pages, ttl=ttl_seconds)
+
+        # Memory tracking
+        self._memory_used_bytes = 0
+
+        # Statistics
+        self._hits = 0
+        self._misses = 0
+
+        # Initialize disk cache if enabled
+        if disk_cache_dir:
+            self._disk_cache_path = Path(disk_cache_dir)
+            self._disk_cache_path.mkdir(parents=True, exist_ok=True)
+        else:
+            self._disk_cache_path = None
+
+        logger.debug(f"Initialized DocumentCache (max_pages={max_pages}, max_memory={max_memory_mb}MB)")
+
+    def _make_key(self, document_id: str, page_number: int, dpi: int) -> str:
+        """Generate cache key."""
+        return f"{document_id}:p{page_number}:d{dpi}"
+
+    def get(
+        self,
+        document_id: str,
+        page_number: int,
+        dpi: int = 300,
+    ) -> Optional[np.ndarray]:
+        """
+        Get a cached page image.
+
+        Args:
+            document_id: Document identifier
+            page_number: Page number
+            dpi: Rendering DPI
+
+        Returns:
+            Cached image array or None
+        """
+        key = self._make_key(document_id, page_number, dpi)
+
+        # Check in-memory cache
+        entry = self._cache.get(key)
+        if entry is not None:
+            self._hits += 1
+            return entry.image
+
+        # Check disk cache
+        if self._disk_cache_path:
+            disk_path = self._disk_cache_path / f"{key}.npy"
+            if disk_path.exists():
+                try:
+                    image = np.load(disk_path)
+                    # Promote to memory cache
+                    self._put_memory(key, document_id, page_number, dpi, image)
+                    self._hits += 1
+                    return image
+                except Exception as e:
+                    logger.warning(f"Failed to load from disk cache: {e}")
+
+        self._misses += 1
+        return None
+
+    def put(
+        self,
+        document_id: str,
+        page_number: int,
+        dpi: int,
+        image: np.ndarray,
+        persist_to_disk: bool = False,
+    ):
+        """
+        Cache a page image.
+
+        Args:
+            document_id: Document identifier
+            page_number: Page number
+            dpi: Rendering DPI
+            image: Page image as numpy array
+            persist_to_disk: Whether to persist to disk
+        """
+        key = self._make_key(document_id, page_number, dpi)
+
+        # Put in memory cache
+        self._put_memory(key, document_id, page_number, dpi, image)
+
+        # Optionally persist to disk
+        if persist_to_disk and self._disk_cache_path:
+            self._put_disk(key, image)
+
+    def _put_memory(
+        self,
+        key: str,
+        document_id: str,
+        page_number: int,
+        dpi: int,
+        image: np.ndarray,
+    ):
+        """Put entry in memory cache."""
+        size_bytes = image.nbytes
+
+        # Check memory limit
+        max_bytes = self.max_memory_mb * 1024 * 1024
+        if self._memory_used_bytes + size_bytes > max_bytes:
+            # Evict oldest entries until we have space
+            self._evict_to_fit(size_bytes)
+
+        entry = CacheEntry(
+            document_id=document_id,
+            page_number=page_number,
+            dpi=dpi,
+            image=image,
+            created_at=datetime.utcnow(),
+            size_bytes=size_bytes,
+        )
+
+        self._cache[key] = entry
+        self._memory_used_bytes += size_bytes
+
+    def _put_disk(self, key: str, image: np.ndarray):
+        """Persist entry to disk cache."""
+        if not self._disk_cache_path:
+            return
+
+        try:
+            disk_path = self._disk_cache_path / f"{key}.npy"
+            np.save(disk_path, image)
+        except Exception as e:
+            logger.warning(f"Failed to write to disk cache: {e}")
+
+    def _evict_to_fit(self, needed_bytes: int):
+        """Evict entries to fit new entry."""
+        max_bytes = self.max_memory_mb * 1024 * 1024
+        target = max_bytes - needed_bytes
+
+        # Get entries sorted by creation time (oldest first)
+        entries = list(self._cache.items())
+
+        for key, entry in entries:
+            if self._memory_used_bytes <= target:
+                break
+            self._memory_used_bytes -= entry.size_bytes
+            del self._cache[key]
+
+    def invalidate(self, document_id: str, page_number: Optional[int] = None):
+        """
+        Invalidate cache entries for a document.
+
+        Args:
+            document_id: Document to invalidate
+            page_number: Optional specific page (None = all pages)
+        """
+        keys_to_remove = []
+
+        for key in self._cache.keys():
+            if key.startswith(f"{document_id}:"):
+                if page_number is None or f":p{page_number}:" in key:
+                    keys_to_remove.append(key)
+
+        for key in keys_to_remove:
+            entry = self._cache.pop(key, None)
+            if entry:
+                self._memory_used_bytes -= entry.size_bytes
+
+        # Also remove from disk cache
+        if self._disk_cache_path:
+            for key in keys_to_remove:
+                disk_path = self._disk_cache_path / f"{key}.npy"
+                if disk_path.exists():
+                    disk_path.unlink()
+
+    def clear(self):
+        """Clear all cache entries."""
+        self._cache.clear()
+        self._memory_used_bytes = 0
+
+        # Clear disk cache
+        if self._disk_cache_path:
+            for f in self._disk_cache_path.glob("*.npy"):
+                f.unlink()
+
+        logger.info("Document cache cleared")
+
+    @property
+    def stats(self) -> Dict:
+        """Get cache statistics."""
+        total = self._hits + self._misses
+        hit_rate = (self._hits / total * 100) if total > 0 else 0
+
+        return {
+            "hits": self._hits,
+            "misses": self._misses,
+            "hit_rate": f"{hit_rate:.1f}%",
+            "entries": len(self._cache),
+            "memory_used_mb": self._memory_used_bytes / (1024 * 1024),
+            "max_memory_mb": self.max_memory_mb,
+        }
+
+
+# Global cache instance
+_document_cache: Optional[DocumentCache] = None
+
+
+def get_document_cache() -> DocumentCache:
+    """Get or create the global document cache."""
+    global _document_cache
+    if _document_cache is None:
+        _document_cache = DocumentCache()
+    return _document_cache