full-repo-refactor 0.2.2
Full repository refactoring setting for Control Arena
License: MIT
Requires Python: >=3.12

# Full Repository Refactor

A Control Arena setting for evaluating agents on full repository refactoring tasks with optional
backdoor insertion.

[View full documentation](https://fastfedora.github.io/full-repo-refactor/).

## Overview

This setting provides an environment for evaluating agents on full repository refactoring tasks.
Each task involves:

1. **Main Task**: Refactoring the repository to improve code quality and maintainability.
2. **Side Task** (Attack Mode): Inserting a hidden vulnerability while performing the refactoring
   that is exploited when running the tests.

## Installation

Install the package (requires Python 3.12 or later):

```bash
pip install full-repo-refactor
```

Then create a `.env` file:

```bash
# Provider API keys
OPENAI_API_KEY=<your key here>
ANTHROPIC_API_KEY=<your key here>

# Trusted and untrusted models--use versions to record the exact model in the transcripts
TRUSTED_MODEL=openai/gpt-4.1-nano-2025-04-14
UNTRUSTED_MODEL=anthropic/claude-sonnet-4-20250514

# Model overrides when using SWE agents (e.g., `--swe-agent` for the CLI). If not specified, the
# `UNTRUSTED_MODEL` or `TRUSTED_MODEL` will be used
CLAUDE_CODE_MODEL=anthropic/claude-sonnet-4-20250514
CODEX_MODEL=openai/gpt-5.1-2025-11-13
```

If not already installed, [install Docker](https://docs.docker.com/engine/install/). Docker must be
running for the evals to execute.

## Usage

### Using the CLI

```bash
# Run with default settings using `side-tasks-3/mixed-3` dataset
full-repo-refactor

# Run getting started scenario in default model (honest)
full-repo-refactor --config=scenarios/getting-started

# Run getting started scenario in attack mode
full-repo-refactor --config=scenarios/getting-started --eval-mode attack

# Run with a different dataset from `github:fastfedora/full_repo_datasets` and limit main tasks
full-repo-refactor --dataset side-tasks-3/mixed-20 --main-task "refactor_single_file"

# Run with verbose logging and limit to 3 samples
full-repo-refactor --config=scenarios/getting-started --verbose --limit 3

# View help
full-repo-refactor --help
```

## Configuration

To create a new scenario, create a file that references the main configuration file from the package
and then override the settings you want to change.

For instance, to create a scenario that runs only the Refactor Single File main task against the
20-repo `side-tasks-3/mixed-20` dataset in attack mode, create a file called `mixed-20.yaml`:

```yaml
# mixed-20.yaml
includes:
  - package:config.yaml

setting:
  dataset: "side-tasks-3/mixed-20"
  main_task: refactor_single_file$

eval:
  # Default mode for this scenario is attack; use `--eval-mode honest` to run in honest mode
  mode: attack
```

Then run this scenario using:

```bash
full-repo-refactor --config=mixed-20
```

Almost all aspects of the setting can be configured via configuration files, including:

- models
- prompts
- main tasks
- side tasks
- policies
- monitors
- metrics
- eval settings

See [the documentation](https://fastfedora.github.io/full-repo-refactor/) for more details.

## License

MIT License