semvec/pypi-packages huggingface dataset:
rag-sentinel 0.1.8
RAG Evaluation Framework using Ragas metrics and MLflow tracking
License: MIT
Requires Python: >=3.9
Keywords: rag, evaluation, ragas, mlflow, llm, ai

# RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

## Installation

### 1. Create Virtual Environment

```bash
# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

### 2. Install Package

```bash
pip install rag-sentinel
```

## Quick Start

### 1. Initialize Project

```bash
rag-sentinel init
```

This creates:
- `.env` - LLM/Embeddings API keys
- `config.ini` - App settings and authentication
- `rag_eval_config.yaml` - Master configuration
- `test_dataset.csv` - Sample test dataset

### 2. Configure

**Edit `.env` file:**
- Add your LLM API keys (Azure OpenAI, OpenAI, or Ollama credentials)
- Set API endpoints and deployment names

**Edit `config.ini` file:**
- Set your RAG backend URL in `app_url`
- Set API paths for context and answer endpoints (`context_api_path`, `answer_api_path`)
- Configure authentication (cookie, bearer token, or API key)
- Set MLflow tracking URI (default: `http://127.0.0.1:5000`)

**Edit `test_dataset.csv` file:**
- Add your test queries in format: `query,ground_truth,chat_id`
- Example: `What is RAG?,RAG stands for Retrieval-Augmented Generation,1`

For detailed configuration help, see the comments in each config file.

### 3. Validate & Run

```bash
# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run
```

Results will be available in the MLflow UI at the configured tracking URI.

## CLI Commands

```bash
# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel
```

## Evaluation Categories

Set `category` in `config.ini` to choose evaluation type:

### Simple (RAGAS Quality Metrics)
```ini
category = simple
```
- **Faithfulness** - Factual consistency of answer with context
- **Answer Relevancy** - How relevant the answer is to the question
- **Context Precision** - Quality of retrieved context
- **Answer Correctness** - Comparison against ground truth

### Guardrail (Security Metrics)
```ini
category = guardrail
```
- **Toxicity Score** - Detects toxic content in responses
- **Bias Score** - Detects biased content in responses

## Performance Metrics

Logged for all evaluation categories:
- **Avg Response Time** - Average API response time (ms)
- **P90 Latency** - 90th percentile latency (ms)
- **Queries Per Second** - Throughput (QPS)

## License

MIT