ragtrace 0.2.0
Observability and tracing for Retrieval-Augmented Generation pipelines
Requires Python: <4.0,>=3.9

# RAGTrace 📊

> **Observability for RAG pipelines** - Trace, inspect, and optimize Retrieval-Augmented Generation systems with ease.

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

## ✨ What is RAGTrace?

RAGTrace is a lightweight observability layer for RAG (Retrieval-Augmented Generation) systems that captures and visualizes every step of your pipeline:

- 🔍 **Event Capture** - Automatically intercepts retrieval, prompt, and generation events
- 💰 **Cost Tracking** - Accurate token counting and cost estimation per query
- 📊 **Interactive Web UI** - Modern timeline view with charts, filters, and event inspection
- 🔧 **CLI Tool** - Developer-friendly command-line interface
- 🌐 **REST API** - Query and analyze sessions programmatically
- 🧪 **Regression Testing** - Snapshot and compare RAG outputs

**Think of it as OpenTelemetry, but specifically for RAG pipelines.**

## 🚀 Quick Start

### Installation

```bash
# Clone the repository
git clone https://github.com/yourusername/ragtrace.git
cd ragtrace

# Install dependencies (using pip)
pip install -e .

# Or using Poetry
poetry install
poetry shell

# Initialize database
ragtrace init
```

### Basic Usage

```python
from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from ragtrace import RagTracer

# Your existing RAG setup
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_texts(["Your documents here..."], embeddings)
llm = ChatOpenAI(model="gpt-3.5-turbo")

# Add RAGTrace - just one line!
tracer = RagTracer(auto_save=True)

# Create chain with tracer
chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    callbacks=[tracer]  # ← Automatic capture!
)

# Run your query (automatically captured)
result = chain.run("What is RAG?")
```

### View Results

```bash
# View latest session in CLI
ragtrace show last

# List all sessions
ragtrace list

# Export to JSON
ragtrace export <session-id> > session.json

# Start API server (for Web UI)
ragtrace run

# Or start both servers for full Web UI experience
uvicorn api.main:app --port 8000 &  # API server
python ui/serve.py                   # UI server → http://localhost:3000
```

## 🌐 Web UI

RAGTrace includes a modern web interface for visualizing and analyzing your RAG pipelines:

### Start the Servers

```bash
# Terminal 1: Start API server
cd ragtrace
source venv/bin/activate
uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

# Terminal 2: Start UI server
cd ragtrace
python ui/serve.py
```

Then open **http://localhost:3000** in your browser.

### Web UI Features

- **📋 Sessions View** - Browse all captured RAG sessions with search and filtering
- **📊 Timeline View** - Interactive timeline showing retrieval → prompt → generation flow
- **📈 Performance Charts** - Waterfall chart for event durations, cost breakdown by component
- **🔍 Event Inspector** - Click any event to see full details including tokens, costs, and data
- **🎯 Smart Filters** - Filter events by type, duration, and cost
- **📤 Export Tools** - Export session data as JSON or CSV, copy to clipboard

### Quick UI Guide

1. **Sessions Tab**: View all RAG sessions, sorted by recent activity
2. **Timeline Tab**: Click on any session to see detailed event timeline
3. **Click Events**: Click timeline events to inspect full details
4. **Apply Filters**: Use dropdowns to filter by event type or performance metrics
5. **Export Data**: Use export buttons to download session data


## 📊 Example Output

```
╭─ Session: d4f3a8b2-... ─────────────────────────────────╮
│ Query: What is RAG?                                      │
│ Model: gpt-3.5-turbo                                     │
│ Cost: $0.00360                                           │
│ Duration: 1,850ms                                        │
╰──────────────────────────────────────────────────────────╯

┏━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Event       ┃ Duration   ┃ Cost       ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Retrieval   │ 150ms      │ $0.00001   │
│ Prompt      │ 0ms        │ $0.00000   │
│ Generation  │ 1,700ms    │ $0.00359   │
└─────────────┴────────────┴────────────┘
```

## 🎯 Features

### ✅ Core Features

- **Automatic Event Capture** - Works with LangChain callbacks
- **Cost Tracking** - Uses tiktoken for accurate token counting
- **Timeline Visualization** - See your RAG pipeline in action
- **Session Management** - Store and retrieve debugging sessions
- **CLI Tool** - Rich formatted terminal output
- **REST API** - FastAPI server with OpenAPI docs
- **Web UI** - Interactive timeline with charts and event inspection
- **JSON/CSV Export** - Export sessions for analysis
- **Snapshot Testing** - Save and compare pipeline outputs

### 🎨 Web UI Features

- **Sessions Dashboard** - View all sessions with search and sort
- **Interactive Timeline** - Click events to inspect details
- **Performance Waterfall** - Visualize event durations side-by-side
- **Cost Breakdown Chart** - See spending by component (embeddings, prompts, generation)
- **Smart Filters** - Filter by event type, duration, or cost
- **Real-time Updates** - WebSocket support for live session monitoring
- **Export Tools** - Download as JSON, CSV, or copy to clipboard

### 🎨 CLI Commands

```bash
ragtrace init              # Initialize database
ragtrace list              # List recent sessions
ragtrace show [id]         # View session details
ragtrace show last         # View latest  session
ragtrace export <id>       # Export to JSON
ragtrace clear             # Clear all data
ragtrace snapshot save     # Save snapshot
ragtrace snapshot list     # List snapshots
ragtrace run               # Start API server
```

### 🌐 API Endpoints

```
POST   /api/sessions                      # Create session
GET    /api/sessions                      # List sessions
GET    /api/sessions/{id}                 # Get session
PATCH  /api/sessions/{id}                 # Update session
DELETE /api/sessions/{id}                 # Delete session
POST   /api/sessions/{id}/events          # Add event
GET    /api/sessions/{id}/costs           # Get costs
POST   /api/snapshots                     # Create snapshot
GET    /api/snapshots                     # List snapshots
GET    /api/snapshots/{id1}/compare/{id2} # Compare snapshots
```

Visit `http://localhost:8000/docs` after running `ragtrace run` for interactive API documentation.

## 🏗️ Architecture

```
ragtrace/
├── core/              # Core business logic
│   ├── models.py      # Pydantic data models
│   ├── storage.py     # SQLite database layer
│   ├── cost.py        # Token counting & cost calculation
│   └── capture.py     # Event aggregation
├── langchain/         # LangChain integration
│   └── middleware.py  # Callback handler
├── api/               # REST API (FastAPI)
│   ├── main.py        # FastAPI application
│   └── routes.py      # API endpoints
├── ui/                # Web UI (HTML/CSS/JS)
│   ├── index.html     # Main UI page
│   ├── app.js         # Frontend application logic
│   ├── styles.css     # UI styling
│   └── serve.py       # Development server
├── cli/               # Command-line interface
│   └── main.py        # Click CLI commands
├── examples/          # Usage examples
│   └── simple_rag.py  # Complete working example
└── tests/             # Test suite
    ├── test_cost.py   # Cost calculation tests
    ├── test_storage.py # Database tests
    └── test_capture.py # Capture logic tests
```

## 📋 Requirements

- **Python**: 3.11+ (3.12 recommended)
- **Dependencies**: FastAPI, LangChain, tiktoken, Rich, Click
- **OpenAI API Key**: Required for examples (not for core functionality)

## 🛠️ Development

### Setup Development Environment

```bash
# Clone repository
git clone https://github.com/yourusername/ragtrace.git
cd ragtrace

# Install with development dependencies
poetry install
poetry shell

# Run tests
pytest

# Run with coverage
pytest --cov=core --cov=langchain --cov=api --cov=cli
```

### Running Tests

```bash
# All tests
pytest

# Specific test file
pytest tests/test_cost.py -v

# With coverage report
pytest --cov=core tests/
```

### Code Quality

```bash
# Format code
black .

# Lint
ruff check .

# Type checking
mypy .
```

## 📖 Examples

Check out the `examples/` directory for complete working examples:

- **`simple_rag.py`** - Basic RAG pipeline with debugger
- **`with_sources.py`** - RAG with source tracking (coming soon)

## 🔬 Use Cases

### 1. Debug Failed Queries
```bash
# See exactly why your RAG pipeline failed
ragtrace show last
```

### 2. Track Costs
```bash
# Monitor spending per query
ragtrace list --sort-by cost
```

### 3. Identify Retrieval Issues
```python
# Check which documents were retrieved
session = tracer.get_latest_session()
print(session.retrieval_event.chunks)
```

### 4. Regression Testing
```bash
# Save baseline
ragtrace snapshot save "v1-baseline"

# Compare after changes
ragtrace snapshot compare <snapshot-id-1> <snapshot-id-2>
```

## 🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Built with [FastAPI](https://fastapi.tiangolo.com/)
- Integrated with [LangChain](https://python.langchain.com/)
- Token counting via [tiktoken](https://github.com/openai/tiktoken)
- Beautiful CLI with [Rich](https://rich.readthedocs.io/)

## 📞 Support

- **Issues**: [GitHub Issues](https://github.com/yourusername/ragtrace/issues)
- **Documentation**: See [examples/](examples/) directory
- **API Docs**: Run `ragtrace run` and visit `http://localhost:8000/docs`

## 🗺️ Roadmap

### v0.2.0 (Current)
- [x] Web UI for timeline visualization
- [x] Interactive event inspection
- [x] Performance charts (waterfall, cost breakdown)
- [x] Export tools (JSON, CSV)
- [ ] Advanced regression testing
- [ ] LlamaIndex integration
- [ ] Prompt versioning

### v0.3.0 (Planned)
- [ ] Agent tracing support
- [ ] Cost optimization suggestions
- [ ] Quality scoring
- [ ] Team collaboration features

### v1.0.0
- [ ] Cloud mode
- [ ] Advanced analytics
- [ ] Alert system
- [ ] Multi-framework support

## ⭐ Star History

If you find RAGTrace useful, please consider giving it a star! ⭐

---

**Built with ❤️ for RAG developers**