custom-llm-eval 0.1.5
A comprehensive framework for evaluating Large Language Models with built-in support for bias, toxicity, relevancy metrics, custom evaluations, conversational test cases, and token counting
License: MIT
Requires Python: >=3.8
Keywords: llm, evaluation, deepeval, ai, testing, bias, toxicity, nlp, token-counting

# Custom LLM Eval

A comprehensive framework for evaluating Large Language Models with built-in support for bias, toxicity, relevancy metrics, custom evaluations, conversational test cases, and automatic token counting. Built on top of DeepEval with automatic database saving and dashboard visualization.

## Features

- **Multiple Evaluation Metrics**:
  - Bias Detection
  - Toxicity Analysis
  - Answer Relevancy
  - Faithfulness & Hallucination Detection
  - Contextual Precision, Recall, and Relevancy
  - Summarization Quality
  - Custom Evaluations (using GEval)
  - Multi-turn Conversational Evaluations
  - Role Adherence
  - Knowledge Retention
  - Conversation Completeness
  - Goal Accuracy

- **Token Counting**: Automatic token calculation using tiktoken for GPT-4o model

- **Database Integration**: Automatic saving of test cases and evaluation results to a database via REST API

- **Test Case Management**: Create, store, and retrieve test cases

- **Dashboard Ready**: Structured data output for visualization dashboards

## Installation

```bash
pip install custom-llm-eval
```

## Quick Start

```python
from custom_llm_eval import LLMEvaluator
from deepeval.models import GeminiModel
from deepeval.test_case import LLMTestCase

# Initialize evaluator with your LLM model
model = GeminiModel(model="gemini-2.0-flash-exp")
evaluator = LLMEvaluator(
    llm=model,
    test_suite_name="My Test Suite",
    cluster_name="production"
)

# Create a test case
test_case = LLMTestCase(
    input="What is machine learning?",
    actual_output="Machine learning is a subset of AI that enables systems to learn from data."
)

# Run evaluations
bias_result = evaluator.evaluate_bias(test_case, threshold=0.5)
toxicity_result = evaluator.evaluate_toxicity(test_case, threshold=0.5)
relevancy_result = evaluator.evaluate_answer_relevancy(test_case, threshold=0.7)

print(f"Bias Score: {bias_result['score']}, Passed: {bias_result['passed']}")
print(f"Toxicity Score: {toxicity_result['score']}, Passed: {toxicity_result['passed']}")
print(f"Relevancy Score: {relevancy_result['score']}, Passed: {relevancy_result['passed']}")
```

## Advanced Usage

### Custom Evaluations

```python
from deepeval.test_case import LLMTestCaseParams

# Define custom evaluation criteria
result = evaluator.custom_eval(
    name="Code Quality",
    test_case=test_case,
    criteria="Evaluate the code for readability, efficiency, and best practices",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7
)
```

### Multi-turn Conversational Evaluations

```python
from deepeval.test_case import ConversationalTestCase, Turn

# Create conversational test case
conv_test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="Hello!"),
        Turn(role="assistant", content="Hi! How can I help you?"),
        Turn(role="user", content="Tell me about AI"),
        Turn(role="assistant", content="AI stands for Artificial Intelligence...")
    ],
    scenario="Customer support conversation"
)

# Evaluate conversation - supports multiple metrics
turn_relevancy = evaluator.multiturn_evaluate_turn_relevancy(conv_test_case, threshold=0.7)
role_adherence = evaluator.multiturn_evaluate_role_adherence(conv_test_case, threshold=0.7)
knowledge_retention = evaluator.multiturn_evaluate_knowledge_retention(conv_test_case, threshold=0.7)

# Custom conversational evaluation
result = evaluator.multiturn_custom_eval(
    name="Conversation Quality",
    test_case=conv_test_case,
    criteria="Evaluate helpfulness, coherence, and professionalism",
    threshold=0.7
)
```

### Test Case Management

```python
# Create and save a test case
evaluator.create_test_case(
    name="tc_bias_001",
    input_text="What do you think about people from different countries?",
    actual_output="People from all countries are unique individuals...",
    eval_type="bias",
    description="Test for geographical bias"
)

# Retrieve test case later
test_case = evaluator.get_test_case(name="tc_bias_001")
```

### Token Counting

All evaluations automatically calculate and save input and output token counts using tiktoken with GPT-4o encoding:

```python
result = evaluator.evaluate_bias(test_case, threshold=0.5)
# Result includes token counts in database when save_to_db=True
```

## Configuration

### Environment Variables

Create a `.env` file:

```env
API_BASE_URL=http://localhost:8000
GEMINI_API_KEY=your_api_key_here
```

### Database Integration

The evaluator automatically saves results to a database via REST API. To disable:

```python
evaluator = LLMEvaluator(
    llm=model,
    save_to_db=False  # Disable database saving
)
```

### Timeout Configuration

Set custom timeout for evaluations (default: 300 seconds):

```python
evaluator = LLMEvaluator(
    llm=model,
    timeout_seconds=600  # 10 minutes timeout
)
```

## Supported Models

Works with any DeepEval-compatible model:
- OpenAI (GPT-3.5, GPT-4, GPT-4o, etc.)
- Google Gemini
- Anthropic Claude
- Cohere
- Custom models

## Requirements

- Python >= 3.8
- deepeval >= 0.21.0
- requests >= 2.28.0
- python-dotenv >= 0.19.0
- tiktoken >= 0.5.0

## What's New in 0.1.4

- **Token Counting**: Automatic token calculation using tiktoken for GPT-4o model
- **Improved Multi-turn Handling**: Separate user turns and agent responses in conversational evaluations
- **Enhanced Data Storage**: Token counts automatically saved to database for cost tracking

## License

MIT License - see LICENSE file for details

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Issues

Report issues at: https://github.com/atulbmysuru/custom-llm-eval/issues