README.md

metadata

title: EvalArena
emoji: 🥇
colorFrom: pink
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: true
license: cc-by-nc-4.0
short_description: An AI Judge Evaluation Platform
sdk_version: 5.19.0

EvalArena

An AI Judge Evaluation Platform

About

EvalArena is a platform that allows users to compare and rate different AI evaluation models (judges). The platform uses a competitive ELO rating system to rank different judge models based on human preferences.

Project Structure

After refactoring, the project now has a cleaner structure:

EvalArena/
│
├── src/                    # Source code
│   ├── app.py              # Application logic
│   ├── config.py           # Constants and configuration
│   ├── data_manager.py     # Dataset loading and management
│   ├── judge.py            # Judge evaluation functionality
│   └── ui.py               # Gradio UI components
│
├── data/                   # Data directory for CSV files
├── models.jsonl            # Model definitions
├── main.py                 # Entry point
└── requirements.txt        # Dependencies

Setup

1. Clone the repository
2. Install dependencies:

   pip install -r requirements.txt
   

3. Create a .env file with any API keys:

   OPENAI_API_KEY=your_key_here
   ANTHROPIC_API_KEY=your_key_here
   QUALIFIRE_API_KEY=your_qualifire_key_here
   

Running

Run the application using:

python main.py

This will start the Gradio web interface where you can:

• Select test types (grounding, hallucinations, safety, etc.)
• Get random examples
• See evaluations from two random judge models
• Select which judge provided a better evaluation
• View the leaderboard of judges ranked by ELO score

Features

• Multiple test types (prompt injections, safety, grounding, hallucinations, policy)
• ELO-based competitive rating system
• Support for various model providers (OpenAI, Anthropic, Together AI)
• Detailed evaluations with scoring criteria
• Persistent leaderboard

Overview

This application allows users to:

1. View AI-generated outputs based on input prompts
2. Compare evaluations from two different AI judges
3. Select the better evaluation
4. Build a leaderboard of judges ranked by ELO score

Features

• Blind Comparison: Judge identities are hidden until after selection
• ELO Rating System: Calculates judge rankings based on user preferences
• Leaderboard: Track performance of different evaluation models
• Sample Examples: Includes pre-loaded examples for immediate testing

Setup

Prerequisites

• Python 3.6+
• Required packages: gradio, pandas, numpy

Installation

1. Clone this repository:

git clone https://github.com/yourusername/eval-arena.git
cd eval-arena

2. Install dependencies:

pip install -r requirements.txt

3. Run the application:

python app.py

4. Open your browser and navigate to the URL displayed in the terminal (typically http://127.0.0.1:7860)

Usage

1. Get Random Example: Click to load a random input/output pair
2. Get Judge Evaluations: View two anonymous evaluations of the output
3. Select Better Evaluation: Choose which evaluation you prefer
4. See Results: Learn which judges you compared and update the leaderboard
5. Leaderboard Tab: View current rankings of all judges

Extending the Application

Adding New Examples

Add new examples in JSON format to the data/examples directory:

{
  "id": "example_id",
  "input": "Your input prompt",
  "output": "AI-generated output to evaluate"
}

Adding New Judges

Add new judges in JSON format to the data/judges directory:

{
  "id": "judge_id",
  "name": "Judge Name",
  "description": "Description of judge's evaluation approach"
}

Integrating Real Models

For production use, modify the get_random_judges_evaluations function to call actual AI evaluation models instead of using the simulated evaluations.

License

MIT

Citation

If you use this platform in your research, please cite:

@software{ai_eval_arena,
  author = {Your Name},
  title = {AI Evaluation Judge Arena},
  year = {2023},
  url = {https://github.com/yourusername/eval-arena}
}

Start the configuration

Most of the variables to change for a default leaderboard are in src/env.py (replace the path for your leaderboard) and src/about.py (for tasks).

Results files should have the following format and be stored as json files:

{
    "config": {
        "model_dtype": "torch.float16", # or torch.bfloat16 or 8bit or 4bit
        "model_name": "path of the model on the hub: org/model",
        "model_sha": "revision on the hub",
    },
    "results": {
        "task_name": {
            "metric_name": score,
        },
        "task_name2": {
            "metric_name": score,
        }
    }
}

Request files are created automatically by this tool.

If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.

Code logic for more complex edits

You'll find

• the main table' columns names and properties in src/display/utils.py
• the logic to read all results and request files, then convert them in dataframe lines, in src/leaderboard/read_evals.py, and src/populate.py
• the logic to allow or filter submissions in src/submission/submit.py and src/submission/check_validity.py