aligneval 0.3.0
Production-ready Python package for evaluating LLM and RAG systems using GPT-4o/GPT-4.1 as a judge
Requires Python: >=3.10
Keywords: llm, rag, evaluation, gpt-4, testing, ai, machine-learning

# AlignEval

**Production-ready Python package for evaluating LLM and RAG systems using GPT-4o/GPT-4.1 as a judge.**

AlignEval provides comprehensive evaluation metrics, autonomous agents, synthetic data generation, and multiple integration options (API, CLI, adapters) for seamless incorporation into development workflows. Built on 2025 research best practices including Chain-of-Thought prompting, structured outputs, and position bias mitigation.

## Features

### 🎯 Comprehensive Metrics
- **General Metrics**: Accuracy, Helpfulness, Relevance, Clarity, Safety
- **RAG-Specific Metrics**: Faithfulness, Answer Relevancy, Context Precision, Context Recall, Groundedness
- All metrics return 1-5 scores with normalized 0.0-1.0 values and detailed reasoning

### 🤖 Autonomous Agent
- LangGraph-based agent for end-to-end evaluation workflows
- Automatic metric selection based on test case characteristics
- Actionable recommendations for improving low-scoring responses

### 🔄 Multiple Integration Options
- **Python API**: Direct integration with type-safe Pydantic models
- **REST API**: FastAPI server for microservice architectures
- **CLI**: Rich terminal interface for quick evaluations
- **Adapters**: Native support for LangChain, LangGraph, and external REST APIs

### 📊 Advanced Features
- Batch evaluation with progress tracking
- Async support for high-throughput scenarios
- Synthetic test case generation from domain descriptions or documents
- HTML and JSON report generation
- Position bias mitigation
- Comprehensive error handling with retry logic

## Installation

### Requirements
- Python 3.10, 3.11, or 3.12
- OpenAI API key with access to GPT-4o or GPT-4.1

### Install from PyPI

```bash
pip install aligneval
```

### Install from source

```bash
git clone https://github.com/aligneval/aligneval.git
cd aligneval
pip install -e .
```

### Development installation

```bash
pip install -e ".[dev]"
```

## Quick Start

### 1. Set up environment variables

Create a `.env` file in your project root:

```bash
ALIGNEVAL_OPENAI_API_KEY=your-openai-api-key-here
ALIGNEVAL_JUDGE_MODEL=gpt-4o  # or gpt-4.1
ALIGNEVAL_TEMPERATURE=0.0
ALIGNEVAL_MAX_TOKENS=1000
ALIGNEVAL_TIMEOUT=60
ALIGNEVAL_MAX_RETRIES=3
ALIGNEVAL_MAX_CONCURRENT_REQUESTS=10
ALIGNEVAL_LOG_LEVEL=INFO
ALIGNEVAL_LOG_FILE=aligneval.log  # Optional: log to file
```

Or set environment variables directly:

```bash
export ALIGNEVAL_OPENAI_API_KEY=your-openai-api-key-here
```

### 2. Basic evaluation

```python
from aligneval.core.test_case import LLMTestCase
from aligneval.metrics.accuracy import AccuracyMetric
from aligneval.metrics.helpfulness import HelpfulnessMetric
from aligneval.evaluator.evaluator import AlignEvaluator

# Create a test case
test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris.",
    expected_output="Paris"
)

# Create evaluator with metrics
evaluator = AlignEvaluator(metrics=[
    AccuracyMetric(),
    HelpfulnessMetric()
])

# Evaluate
result = evaluator.evaluate(test_case)

print(f"Overall Score: {result.overall_score:.2f}")
for metric_result in result.metric_results:
    print(f"{metric_result.metric_name}: {metric_result.normalized_score:.2f}")
    print(f"Reasoning: {metric_result.reasoning}")
```

### 3. RAG evaluation

```python
from aligneval.core.test_case import LLMTestCase
from aligneval.metrics.faithfulness import FaithfulnessMetric
from aligneval.metrics.answer_relevancy import AnswerRelevancyMetric
from aligneval.evaluator.rag_evaluator import RAGEvaluator

# Create a RAG test case
test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris, which is also its largest city.",
    retrieval_context=[
        "Paris is the capital and largest city of France.",
        "France is a country in Western Europe."
    ]
)

# Create RAG evaluator
evaluator = RAGEvaluator(metrics=[
    FaithfulnessMetric(),
    AnswerRelevancyMetric()
])

# Evaluate with diagnostics
result = evaluator.evaluate(test_case)

print(f"Overall Score: {result.overall_score:.2f}")
print(f"Retrieval Score: {result.diagnostics.retrieval_score:.2f}")
print(f"Generation Score: {result.diagnostics.generation_score:.2f}")
print(f"Bottleneck: {result.diagnostics.bottleneck}")
print(f"Recommendations: {result.diagnostics.recommendations}")
```

### 4. Batch evaluation

```python
from aligneval.core.dataset import EvaluationDataset
from aligneval.core.test_case import LLMTestCase
from aligneval.evaluator.evaluator import AlignEvaluator
from aligneval.metrics.accuracy import AccuracyMetric

# Create dataset
dataset = EvaluationDataset(name="my_test_suite")
dataset.add_test_case(LLMTestCase(
    input="What is 2+2?",
    actual_output="4",
    expected_output="4"
))
dataset.add_test_case(LLMTestCase(
    input="What is the capital of Spain?",
    actual_output="Madrid",
    expected_output="Madrid"
))

# Batch evaluate with progress bar
evaluator = AlignEvaluator(metrics=[AccuracyMetric()])
results = evaluator.evaluate_dataset(dataset)

print(f"Evaluated {len(results)} test cases")
passed = sum(1 for r in results if r.passed)
print(f"Passed: {passed}/{len(results)}")
```

### 5. Using the CLI

```bash
# Evaluate a single test case
aligneval evaluate \
  --input "What is the capital of France?" \
  --output "Paris" \
  --metrics accuracy helpfulness

# Batch evaluate from JSON file
aligneval batch \
  --input dataset.json \
  --metrics accuracy relevance clarity \
  --output results.json

# Generate synthetic test cases
aligneval generate \
  --domain "customer support chatbot" \
  --num-cases 10 \
  --output synthetic_dataset.json

# Generate HTML report
aligneval report \
  --input results.json \
  --output report.html
```

### 6. Using the REST API

Start the API server:

```bash
# Simple start
uvicorn aligneval.api.main:app --reload

# Or use the convenience script
python run_api.py --reload

# Production with multiple workers
uvicorn aligneval.api.main:app --host 0.0.0.0 --port 8000 --workers 4
```

The API will be available at:
- API: http://localhost:8000
- Interactive docs: http://localhost:8000/docs
- Health check: http://localhost:8000/api/v1/health

Make requests:

```bash
# List available metrics
curl http://localhost:8000/api/v1/metrics

# Evaluate single test case
curl -X POST http://localhost:8000/api/v1/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "test_case": {
      "input": "What is the capital of France?",
      "actual_output": "The capital of France is Paris.",
      "expected_output": "Paris"
    },
    "metrics": ["Accuracy", "Clarity"]
  }'

# Batch evaluation
curl -X POST http://localhost:8000/api/v1/evaluate/batch \
  -H "Content-Type: application/json" \
  -d '{
    "test_cases": [
      {
        "input": "What is 2+2?",
        "actual_output": "4",
        "expected_output": "4"
      },
      {
        "input": "What is the largest planet?",
        "actual_output": "Jupiter",
        "expected_output": "Jupiter"
      }
    ],
    "metrics": ["Accuracy"],
    "max_concurrent": 5
  }'
```

See `aligneval/api/README.md` for complete API documentation.

### 7. Async evaluation for high throughput

```python
import asyncio
from aligneval.core.dataset import EvaluationDataset
from aligneval.evaluator.evaluator import AlignEvaluator
from aligneval.metrics.accuracy import AccuracyMetric

async def evaluate_async():
    # Create dataset
    dataset = EvaluationDataset(name="async_test")
    # ... add test cases ...
    
    # Async batch evaluation with concurrency control
    evaluator = AlignEvaluator(metrics=[AccuracyMetric()])
    results = await evaluator.evaluate_dataset_async(
        dataset,
        max_concurrent=5  # Control API rate limits
    )
    
    return results

# Run async evaluation
results = asyncio.run(evaluate_async())
```

## Configuration

AlignEval uses environment variables for configuration. All variables are prefixed with `ALIGNEVAL_`:

| Variable | Default | Description |
|----------|---------|-------------|
| `ALIGNEVAL_OPENAI_API_KEY` | *required* | OpenAI API key |
| `ALIGNEVAL_JUDGE_MODEL` | `gpt-4o` | Judge model (`gpt-4o` or `gpt-4.1`) |
| `ALIGNEVAL_TEMPERATURE` | `0.0` | Temperature for judge LLM (0.0-2.0) |
| `ALIGNEVAL_MAX_TOKENS` | `1000` | Maximum tokens for responses |
| `ALIGNEVAL_TIMEOUT` | `60` | API call timeout in seconds |
| `ALIGNEVAL_MAX_RETRIES` | `3` | Maximum retry attempts for failed calls |
| `ALIGNEVAL_RETRY_DELAY` | `1.0` | Initial retry delay (uses exponential backoff) |
| `ALIGNEVAL_MAX_CONCURRENT_REQUESTS` | `10` | Max concurrent requests for async batch evaluation |
| `ALIGNEVAL_ENABLE_POSITION_BIAS_MITIGATION` | `true` | Enable output randomization |
| `ALIGNEVAL_LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) |
| `ALIGNEVAL_LOG_FILE` | `None` | Optional log file path |

## ⚡ Performance Optimization

AlignEval uses OpenAI's GPT models as judge LLMs, which introduces API latency. Here's how to optimize performance:

### Speed Comparison

| Method | 100 Test Cases (5 metrics) | Speed |
|--------|----------------------------|-------|
| Sync (sequential) | ~16 minutes | 1x baseline |
| Async (10 concurrent) | ~1.7 minutes | 10x faster ⚡ |
| Async (20 concurrent) | ~50 seconds | 20x faster ⚡⚡ |
| Async + gpt-4o-mini | ~25 seconds | 40x faster ⚡⚡⚡ |

### Quick Wins

**1. Use Async for Batch Evaluation (10x faster!)**

```python
import asyncio
from aligneval import AlignEvaluator, EvaluationDataset
from aligneval.metrics import AccuracyMetric, RelevanceMetric

evaluator = AlignEvaluator([AccuracyMetric(), RelevanceMetric()])
dataset = EvaluationDataset.from_json("test_cases.json")

# Async with concurrency control
async def main():
    results = await evaluator.evaluate_dataset_async(
        dataset,
        max_concurrent=20  # Adjust based on rate limits
    )

asyncio.run(main())
```

**2. Use Fewer Metrics (5x faster)**

```python
# ❌ Slow: 10 metrics = 10 API calls per test case
evaluator = AlignEvaluator(get_all_metrics())

# ✅ Fast: 2 metrics = 2 API calls per test case
evaluator = AlignEvaluator([AccuracyMetric(), RelevanceMetric()])
```

**3. Use Faster Model (2-3x faster)**

```bash
# Use gpt-4o-mini instead of gpt-4o
export ALIGNEVAL_JUDGE_MODEL="gpt-4o-mini"
```

**4. Increase Concurrency (2-5x faster)**

```bash
# Default: 10 concurrent requests
export ALIGNEVAL_MAX_CONCURRENT_REQUESTS=10

# Faster: 20-50 concurrent (check your OpenAI rate limits!)
export ALIGNEVAL_MAX_CONCURRENT_REQUESTS=20
```

### Performance Demo

Run the performance comparison:

```bash
python examples/performance_demo.py
```

For detailed optimization strategies, see [PERFORMANCE_OPTIMIZATION_GUIDE.md](./PERFORMANCE_OPTIMIZATION_GUIDE.md).

## Available Metrics

### General Metrics

- **AccuracyMetric**: Evaluates factual correctness
- **HelpfulnessMetric**: Evaluates utility and usefulness
- **RelevanceMetric**: Evaluates pertinence to the query
- **ClarityMetric**: Evaluates understandability and coherence
- **SafetyMetric**: Evaluates harmlessness and absence of toxic content

### RAG-Specific Metrics

- **FaithfulnessMetric**: Evaluates if response is grounded in context
- **AnswerRelevancyMetric**: Evaluates if response addresses the query
- **ContextPrecisionMetric**: Evaluates if retrieved context is relevant
- **ContextRecallMetric**: Evaluates if all necessary information was retrieved
- **GroundednessMetric**: Evaluates factual consistency with context

All metrics return:
- `raw_score`: Integer from 1-5
- `normalized_score`: Float from 0.0-1.0 (calculated as `(raw_score - 1) / 4.0`)
- `reasoning`: Chain-of-thought explanation from the judge LLM
- `metadata`: Additional information about the evaluation

## Examples

See the [examples/](./examples) directory for complete working examples:

- `evaluator_demo.py`: Basic single and batch evaluation
- `rag_evaluator_demo.py`: RAG system evaluation with diagnostics
- `async_evaluation.py`: High-throughput async evaluation
- `custom_metrics.py`: Creating custom metrics
- `report_generation.py`: Generating HTML and JSON reports

## Documentation

- [Full Documentation](https://aligneval.readthedocs.io)
- [API Reference](https://aligneval.readthedocs.io/api)
- [Examples](./examples)
- [Contributing Guide](./CONTRIBUTING.md)

## Architecture

AlignEval follows a layered architecture:

```
┌─────────────────────────────────────────────────────────┐
│                    Integration Layer                     │
│  (FastAPI API, Click CLI, LangChain/LangGraph Adapters) │
└─────────────────────────────────────────────────────────┘
                            │
┌─────────────────────────────────────────────────────────┐
│                   Application Layer                      │
│  (AlignEvaluator, RAGEvaluator, Pipeline, Agent)       │
└─────────────────────────────────────────────────────────┘
                            │
┌─────────────────────────────────────────────────────────┐
│                      Metrics Layer                       │
│     (BaseMetric, 5 General Metrics, 5 RAG Metrics)     │
└─────────────────────────────────────────────────────────┘
                            │
┌─────────────────────────────────────────────────────────┐
│                       Core Layer                         │
│    (LLMTestCase, EvaluationDataset, Settings, Config)  │
└─────────────────────────────────────────────────────────┘
```

## Technology Stack

- **Core**: Python 3.10+, Pydantic 2.5+
- **LLM**: OpenAI GPT-4o/GPT-4.1
- **Orchestration**: LangChain 0.3.x, LangGraph 0.2.x
- **API**: FastAPI 0.115.x, Uvicorn
- **CLI**: Click 8.x, Rich 13.x
- **Testing**: pytest 7.x, pytest-asyncio, hypothesis

## Contributing

We welcome contributions! Please see our [Contributing Guide](./CONTRIBUTING.md) for details.

## License

MIT License - see [LICENSE](./LICENSE) file for details.

## Citation

If you use AlignEval in your research, please cite:

```bibtex
@software{aligneval2025,
  title = {AlignEval: Production-ready LLM and RAG Evaluation Framework},
  author = {AlignEval Team},
  year = {2025},
  url = {https://github.com/aligneval/aligneval}
}
```

## Support

- [GitHub Issues](https://github.com/aligneval/aligneval/issues)
- [Documentation](https://aligneval.readthedocs.io)
- [Discord Community](https://discord.gg/aligneval)

## Acknowledgments

Built on research best practices from:
- MT-Bench: Multi-turn benchmark for LLM evaluation
- G-Eval: LLM-based evaluation with Chain-of-Thought
- RAGAS: RAG assessment framework