evalgate 0.3.0
Deterministic LLM/RAG evals as a PR check
License: MIT
Requires Python: >=3.10

# AOTP Ventures EvalGate

**EvalGate** runs deterministic LLM/RAG evals as a PR check. It compares your repo's generated outputs against **fixtures**, validates **formatting**, **label accuracy**, **latency/cost budgets**, and can use **LLMs as judges** for complex criteria. It posts a readable summary on the PR. Default is **local-only** (no telemetry).

- ✅ Deterministic checks (schema/labels/latency/cost)
- 🧠 LLM-based evaluation for complex criteria
- 🧪 Regression vs `main` baseline
- 🔒 Local-only by default; optional “metrics-only” later
- 🧰 Zero infra — a composite GitHub Action + tiny CLI

## Quick Start

### 1. Install and Initialize
```bash
# Initialize EvalGate in your project
uvx --from evalgate evalgate init

# This creates:
# - .github/evalgate.yml (configuration)
# - eval/fixtures/ (test data with expected outputs)
# - eval/schemas/ (JSON schemas for validation)
```

### 2. Generate Your Model's Outputs
```bash
# Run your model/system to generate outputs for the fixtures
# (Replace with your actual prediction script)
python scripts/predict.py --in eval/fixtures --out .evalgate/outputs
```

### 3. Run Evaluation
```bash
# Run the evaluation suite
uvx --from evalgate evalgate run --config .github/evalgate.yml

# View results summary
uvx --from evalgate evalgate report --summary --artifact .evalgate/results.json
# Inside GitHub Actions, add --check-run to publish results as a check run
```

### 4. Update Baseline (optional)
When your fixtures or model outputs change, update the stored baseline results. This runs the evals and commits the results to the git ref specified by `baseline.ref` (default `origin/main`).

```bash
uvx --from evalgate evalgate baseline update --config .github/evalgate.yml
```

Pull requests will be compared against these baseline results.

## Conversation Fixtures

When working with chat-based models, fixtures can describe full conversations.
Each conversation fixture contains a list of `messages`, where every message has
a `role` (such as `system`, `user`, `assistant`, or `tool`) and `content`.
Assistant messages may optionally include `tool_calls` describing functions the
assistant wants to invoke.

Example conversation fixture:

```json
{
  "messages": [
    { "role": "user", "content": "Hello!" },
    {
      "role": "assistant",
      "content": "Hi there!",
      "tool_calls": [
        { "name": "search", "arguments": { "query": "Hello!" } }
      ]
    }
  ]
}
```

## LLM as Judge

EvalGate can use LLMs to evaluate outputs for complex criteria beyond simple schema validation.

### 1. Install with LLM support
```bash
# Install with LLM dependencies
pip install evalgate[llm]
# Or with uv
uvx --from evalgate[llm] evalgate
```

### 2. Create a prompt template
Create an evaluation prompt in `eval/prompts/quality_judge.txt`:
```
You are evaluating the quality of customer support responses.

INPUT:
{input}

EXPECTED:
{expected}

OUTPUT:
{output}

Rate from 0.0 to 1.0 based on accuracy, helpfulness, and tone.
Score: [your score]
```

### 3. Configure your evaluator
In `.github/evalgate.yml`:
```yaml
evaluators:
  # Your existing evaluators...
  - name: content_quality
    type: llm
    provider: openai  # or anthropic, azure, local
    model: gpt-4      # or other models
    prompt_path: eval/prompts/quality_judge.txt
    api_key_env_var: OPENAI_API_KEY
    weight: 0.3
    min_score: 0.75  # fail if score < 0.75
```

`min_score` enforces a minimum evaluator score; the run fails if the score drops below this threshold.

### 4. Set your API key
```bash
export OPENAI_API_KEY=your_api_key_here
```

### 5. Run evaluation
```bash
evalgate run --config .github/evalgate.yml
```

LLM judge responses are cached in `.evalgate/cache.json`. Reuse cached results to avoid repeat API calls or clear them with:

```bash
evalgate run --config .github/evalgate.yml --clear-cache
```

### GitHub Actions Integration with API Keys

Add your API keys as repository secrets in GitHub, then use them in your workflow:

```yaml
# Optional: Validate API key is set (fail fast with clear message)
- name: Validate OpenAI API Key  
  run: |
    if [ -z "${{ secrets.OPENAI_API_KEY }}" ]; then
      echo "❌ OPENAI_API_KEY secret is not set"
      echo "Please add your OpenAI API key as a repository secret named 'OPENAI_API_KEY'"
      echo "Go to: Settings > Secrets and variables > Actions > New repository secret"
      exit 1
    fi

- name: Run EvalGate with LLM judge
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: uvx --from evalgate[llm] evalgate run --config .github/evalgate.yml
```

Or with the composite action:

```yaml
- uses: aotp-ventures/evalgate@main
  with:
    config: .github/evalgate.yml
    openai_api_key: ${{ secrets.OPENAI_API_KEY }}
    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    check_run: true
```

## Tool Usage Logs

Model outputs can record tool invocations to enable deterministic evaluation of agent behavior. Each output may include a `tool_calls` array with call `name` and `args` in the order executed:

```json
{
  "output": "...",
  "tool_calls": [
    {"name": "search", "args": {"query": "foo"}},
    {"name": "lookup", "args": {"id": 1}}
  ]
}
```

The `tool_usage` evaluator compares these logs against expected sequences via the `expected_tool_calls` config field.

## GitHub Actions Integration

### Option 1: Use the Composite Action
Add this to your `.github/workflows/` directory:

```yaml
name: EvalGate
on: [pull_request]

jobs:
  evalgate:
    runs-on: ubuntu-latest
    outputs:
      total_score: ${{ steps.evalgate.outputs.total_score }}
      passed: ${{ steps.evalgate.outputs.passed }}
    permissions:
      contents: read
      checks: write
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }

      # Generate your model outputs
      - name: Generate outputs
        run: python scripts/predict.py --in eval/fixtures --out .evalgate/outputs

      # Run EvalGate
      - uses: aotp-ventures/evalgate@main
        id: evalgate
        with:
          config: .github/evalgate.yml
          check_run: true
  build:
    needs: evalgate
    if: ${{ needs.evalgate.outputs.passed == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "EvalGate score ${{ needs.evalgate.outputs.total_score }}"
          
```


> **Note:** The workflow requires `checks: write` permission to publish the check run.

### Option 2: Direct Integration
Or integrate directly in your existing workflow:

```yaml
- name: Install uv
  run: |
    curl -LsSf https://astral.sh/uv/install.sh | sh
    echo "$HOME/.local/bin" >> $GITHUB_PATH

- name: Run EvalGate
  id: evalgate
  run: |
    uvx --from evalgate evalgate run --config .github/evalgate.yml
    total_score=$(jq -r '.overall' .evalgate/results.json)
    passed=$(jq -r '.gate.passed' .evalgate/results.json)
    echo "total_score=$total_score" >> "$GITHUB_OUTPUT"
    echo "passed=$passed" >> "$GITHUB_OUTPUT"

- name: EvalGate Summary
  if: always()
  run: uvx --from evalgate evalgate report --summary --artifact ./.evalgate/results.json

- name: Continue if passed
  if: ${{ steps.evalgate.outputs.passed == 'true' }}
  run: echo "EvalGate score ${{ steps.evalgate.outputs.total_score }}"

# Optional: Upload detailed results for debugging
- name: Upload EvalGate Results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: evalgate-results
    path: .evalgate/results.json
    retention-days: 30
```
```

## Conversation Flow Evaluator

Validate multi-turn conversations by checking the final message and turn count.

```yaml
evaluators:
  - name: convo_flow
    type: conversation
    expected_final_field: content
    max_turns: 5
    weight: 0.2
```

Each output must provide a `messages` array. The evaluator compares the last
message's `content` against the fixture's `expected.content` and fails if the
conversation exceeds `max_turns`.

## Writing a custom evaluator

EvalGate supports custom evaluators through the plugin registry introduced in Issue 1. This lets you package and share evaluation logic as reusable plugins.

```python
from evalgate.plugins import registry
from evalgate.evaluators import BaseEvaluator

@registry.evaluator("my_custom")
class MyCustomEvaluator(BaseEvaluator):
    def evaluate(self, outputs, fixtures, **kwargs):
        score = 1.0  # your scoring logic here
        violations: list[str] = []
        return score, violations
```

After registering, reference the evaluator in your configuration:

```yaml
evaluators:
  - name: my_custom
    type: my_custom
    weight: 0.5
```

## Refreshing your baseline

EvalGate compares pull requests against a baseline stored on your main branch. When your model's expected outputs change, refresh the baseline so future PRs compare against the new results:

```bash
# Generate fresh outputs and update baseline
python scripts/predict.py --in eval/fixtures --out .evalgate/outputs
uvx --from evalgate evalgate run --config .github/evalgate.yml
git add .evalgate/results.json
git commit -m "Refresh eval baseline"
```

Merge the commit into `main`, and subsequent runs will use the updated baseline for regression checks.