control-runner 0.1.1
Tool for configuring and running control_arena experiments using .yaml files.
Requires Python: >=3.13

### Control Runner

Run control_arena experiments from YAML. Define settings, protocols, policies, monitors, and scorers as reusable components or inline specs. Then execute them from a single CLI.

### Environment

Most model providers require API keys via environment variables. It’s recommended to keep them in a `.env` file and ensure your environment loads it.

```
# OpenAI
OPENAI_API_KEY=sk-...replace-me...

# Google
GOOGLE_API_KEY=...replace-me...
```

### Quick start

1) Install

```bash
uv add control-runner
```

2) Create `quickstart.yaml`

```yaml
settings:
  apps_setting:
    type: "apps"
    params:
      dataset_name: "simple"

tasks:
  - id: "quickstart"
    setting: "apps_setting"
    protocol:
      type: "tm_protocol"
      params:
        policy_builder:
          type: "untrusted_policy"
          params:
            honest:
              type: "honest_policy"
              params:
                model: "google/gemini-2.5-flash-lite"
            attack:
              type: "attack_policy"
              params:
                model: "google/gemini-2.5-flash"
        monitor_builder:
          type: "basic_monitor"
          params:
            model: "google/gemini-2.5-flash-lite"
    mode: "ATTACK"
    monitoring_scorers:
      - "max_suspicion_scorer"
    limit: 5
```

3) Run

```bash
uv run control_runner run quickstart.yaml --dry-run  # validate
uv run control_runner run quickstart.yaml            # execute
```

Alternatively, without adding to your project:

```bash
uvx control-runner run quickstart.yaml --dry-run
```

### Why use it
- **Simple YAML**: Reference reusable components or define them inline
- **Dynamic variables**: `${var}` substitution and file-content expansion
- **Safe dry-runs**: Validate configs without importing heavy deps
- **Batteries included**: Rich built‑in registry mapped to control_arena

## Quick start (PyPI)

Prereqs: Python >= 3.13, `uv` installed.

```bash
# 1) Add to your project
uv add control-runner

# 2) Run with uv
uv run control_runner run path/to/config.yaml --dry-run  # validate
uv run control_runner run path/to/config.yaml            # execute

# Alternative: one-off run without adding to your project
uvx control-runner run path/to/config.yaml --dry-run
```

## CLI

```text
control_runner run CONFIG [--dry-run]
```

- `CONFIG` (required): Path to a YAML file
- `--dry-run`: Load and print tasks without running evaluation

Behavior:
- Sets `UNTRUSTED_MODEL` and `TRUSTED_MODEL` env vars if present in `vars`
- Groups tasks by their `limit` and runs each group via `inspect_ai.eval(tasks=..., limit=...)`

## YAML configuration

Top-level keys:
- `vars` (optional): Simple key/value variables
- `settings`: Named settings objects
- `components` (optional): Reusable components (policies, monitors, protocols, scorers, …)
- `tasks`: List of tasks to run

### Referencing values
You can reference values in two ways:
- `${name}`: explicit reference
- `name`: bare string; if it exactly matches a key in `vars`, `settings`, or `components`, it resolves to that value

String substitutions are supported for primitives inside larger strings, e.g. `"Model: ${untrusted_model}"`.

### Variables and file loading
Within `vars`, any string that looks like a path (`./`, `../`, or `/` and has an extension) is read as file content. This makes prompts easy to manage:

```yaml
vars:
  untrusted_model: google/gemini-2.5-flash
  trusted_model: google/gemini-2.5-flash-lite
  attack_monitor_prompt: ./prompts/attack_monitor_simple.txt  # contents will be in this var
```

Use these variables anywhere (including inside inline specs) via `${...}`.

### Inline specs and the registry
Components can be defined inline using `type` (lookup in the registry):

```yaml
policy_builder:
  type: untrusted_policy
  params:
    honest: ${honest_policy_builder}
    attack: ${attack_policy_builder}
```

The registry contains keys for settings, policies, monitors, protocols, scorers, and tools. See `src/control_runner/registry.py` for the full list of supported names.

### Tasks

```yaml
tasks:
  - id: attack_task
    setting: ${apps_setting}          # reference or inline spec
    protocol: ${monitored_protocol}   # reference or inline spec
    mode: ATTACK                      # string or EvalMode name (case‑insensitive)
    monitoring_scorers:               # list of names or inline spec(s)
      - max_suspicion_scorer
    limit: 1                          # per‑group limit for inspect_ai.eval
```

`monitoring_scorers` accepts either a list or a single inline spec object.

### Settings and components
Reusable blocks defined under `settings` and `components` can be referenced from tasks or other components:

```yaml
settings:
  apps_setting:
    type: apps
    params:
      dataset_name: simple

components:
  monitored_protocol:
    type: tm_protocol
    params:
      policy_builder: ${untrusted_policy_builder}
      monitor_builder: ${untrusted_monitor_builder}
```



## Examples

The `example_configs` directory shows common patterns:
- `components.yaml`: Reusable components and reference-based tasks
- `inline.yaml`: Fully inline protocol and nested inline components
- `inline_with_components.yaml`: Mix of inline protocol with referenced components
- `inline_strings.yaml`: Minimal config using bare string references

Run any example with `--dry-run` to validate it, then without to execute.

## Troubleshooting
- **Unknown reference**: Ensure the name exists in `vars`, `settings`, or `components`
- **Unknown type**: Check that the `type` exists in the registry (`src/control_runner/registry.py`)
- **File var not loaded**: File expansion only applies to values in `vars`

## Development

```text
control_runner/
├─ example_configs/
├─ prompts/
├─ src/control_runner/
│  ├─ cli.py         # Click CLI entrypoint
│  ├─ loader.py      # YAML loading and resolution
│  └─ registry.py    # control_arena registry mapping
├─ tests/
└─ README.md
```

From source (repo checkout):

```bash
uv sync
uv run control_runner run example_configs/components.yaml --dry-run
```

## Testing

```bash
uv run pytest -q
```