Repository: harbor-framework/harbor
Issue: Feature proposal: Multi-Turn Trial support for SCBench
Root cause: Unimplemented Feature Gap

We recently released [SlopCodeBench (SCBench)](https://github.com/SprocketLab/slop-code-bench) and wanted to add it to Harbor as an adapter. To do this, we would also need to extend Harbor's functionality to support the multi-turn/iterative nature of our benchmark. 

SCBench sets up problems in the benchmark as _checkpoints_, where the agent is given a specification, asked to implement it, and then given the following checkpoint's specification to extend its current solution. Our current setup for running this is:
1. Create a temp workspace for the agent w/ any required files - we call them static assets, but they are essentially the `environment` dir in harbor tasks. The only special thing we do with them is use them in testing and rematerialize them to a new directory.
2. The agent is given the checkpoint specification with language placeholders filled in, and it generates a solution to the specification.
    a. Launching the Docker/local runtime is the responsibility of the agent in our case. We dont do anything here.
3. We then kill that docker container, create a "snapshot" of the working dir, excluding venv/static asset directories.
4. We then copy over this snapshot to a clean temp dir and our test files up to the current checkpoint. Launch a new Docker container, reinstall the solution's dependencies and run the tests
    a. Not leaking the tests to the agents nor giving the agent _any_ way to cheat was our top priority, given our multi-step setup led to this paranoia. 
    b. It is slow but works
5. Depending on the result of tests and the configuration setup, we either continue and repeat from step 2 with the next checkpoint, or stop.

We completely reworked our testing system to work with Harbor's pytest style. So now the big hurdle left is to get the multi-step inference to work. But that would require some substantial features to make it compatible. Hence, I am creating this issue because I don't want to make these changes without any discussion of their design. The most critical factor for the rest of the implementation is the trial approach. The options I see are:

**Option A: Step Loop in Trial**
- Update trial's `run` method to handle multiple steps
- Same container, sequential execution -- This would be a  _significant risk of leakage_ given test information could leave artifacts over in the workspace directory. 
- Seems like a pretty small overall code change

**Option B: Step as Sub-Trial**
- Trial can spawn a child trial for each step
- Separate container for each child trial, would require re-installing agent at each sub-trial
- Would be more structural changes that would require more verification and testing.
- Would make it easier to add in future parallelization of independent steps for future benchmarks like ours

**Option C: Orchestrator-Level**
- Requires adding in graph/dependency tracking so that the orchestrator would handle scheduling the multiple steps per task.
- Trials would remain unchanged and only handle single steps. 
- It is probably the most intensive implementation of the three, but would provide the best security guarantees.

My opinion would be that **B** is the optimal one given the tradeoffs. My biggest concern is security of leakage becasue some of the problems involve caching/other local modifications that could leak to the agent. So I want to keep the workspace vs eval space as far away as possible.

Tasks would look like:
```
task-name/
├── task.toml
├── steps/                  # per-step instruction files
│   ├── step_1.md
│   └── step_2.md
├── environment/
│   └── Dockerfile       # We use a constant docker across all of our tasks, so this would be trivial
│   └── ...                     # We would add all static assets here than update our tests to use them.
└── tests/
    ├── pytest.ini          # markers config
    ├── conftest.py         # fixtures
    ├── test_checkpoint_1.py
    ├── test_checkpoint_2.py
    └── checkpoint_data/    # test fixtures/data if needed
```

In terms of saving, I was thinking a pretty simple result structure:
```
results/{task_id}/
├── trial_result.json           # Aggregated result with all steps
├── step_1/
│   ├── trajectory.json         # ATIF format
│   ├── test_results.json       # CTRF format
│   └── workspace_snapshot/     # Optional state capture
├── step_2/
│   └── ...
└── summary.json                # Quick overview metrics
```

As I was writing this, I guess we could also create a tarball of the workspace prior to eval, then remake it after. But there are so many things that could change there. There is _also_ Docker commit, but that doesn't preserve volumes. 


Hi @gabeorlanski,

I'm currently adapting BFCL (Berkeley Function Call Leaderboard) to Harbor and I have a similar multi-turn requirement:
- 800 multi-turn tasks with 3 conversational turns each
- Each turn depends on previous conversation history
- Similar state-preservation needs as SCBench

I've completed the single-turn categories (3,865 tasks) and are waiting for Harbor's multi-turn support before implementing the multi-turn portion.

**Question:** Would you like collaboration on this feature, or should we both wait for the Harbor team's guidance on the design approach before proceeding?

Happy to help test or validate once a direction is chosen!

Best,
Ternura

Hi @Ternura143 , collaboration sounds great! I do think, given the implications the different choices could have on the overall contracts in harbour, we should wait until the Harbor team gives guidance before starting anything. 

Sounds good, that makes sense. Happy to wait for guidance from the Harbor team before moving forward.

@Ternura143 sorry for the delay, i just went ahead and did it these past two days. I am staggering the PRs so as not to overburden reviewers due to its size

It's okay, you're doing a really great job! Thanks for your hard work @gabeorlanski !