zhiminy/EvalEng huggingface dataset:
Repository: harbor-framework/harbor
Issue: Feature Request: Support arbitrary tasks instead of datasets
Root cause: Unimplemented Feature Gap

Is there any way we could have arbitrary task support + optional verification ? 

I've been looking for something to spin up lots of coding agents on isolated environments for a long time, and none of them have had the great infra that harbor does. However, it'd be really nice to not have it bound to the dataset case. 

Is this something that the team or community is interested in? Happy to discuss design and PR .   

I feel this could be a very useful dev feature for anyone building their own terminal tasks. It allows people to spin up a basic environment and a prompt and observe how agents behave under that setup, without needing to write verification yet.

@darinkishore can you say more about what you have in mind?

@digitsisyph we do have the `--disable-verification` flag that allows you to skip verification.

@alexgshaw 

It's currently hard to orchestrate and control large amounts of Claude code/codex agents. This is a non-worktree based solution that lets them work in parallel. It is by FAR the best one I've seen. 

I want to have lots of tasks for agents, and let them spin up commits. I want them to be able to recieve follow up messages. 

This is different from training/eval infra though, i'd want the ability to bring in custom coding agent configurations, define tasks one off, et al.

Some concrete requests:
- folder/repo based tasks, eg:


@RepoTaskConfig(
    repo_url="https://github.com/user/project",
    git_ref="main",
    instruction="Add tests for all functions",
)

which would then autogen task directories,

git aware environments that expose FS changes as well (so u dont have to replay ATIF)

ability to continue where the agents have left off, so multiple user messages in one interaction.


note: this doesn't have to be exposed via CLI (or at least for my use cases), I can't figure out how the UX would look like. 

But it's be immensely useful for me if this could work as something like a library—especially for programming large scale ai systems w/a fuckton of coding agents. core infra 


obviously big feature creep on that post; I want to fork and I'd MUCH rather PR. I think u could add support for most of this with some surgical tweaks. happy to put together something for u guys to look at with more concrete design sketches. 

@darinkishore any chance you'd be willing to hop on a call to chat about this? Here's [my calendar](https://cal.com/alexgshaw/harbor-feature-brainstorm) if so!
Repository: harbor-framework/harbor
Issue: Feature request: Installation of local agents
Root cause: Unimplemented Feature Gap

All installed agent examples are using remote repositories like git or pakage manager to be installed on installation script of the agent. There is no way to make a local wheel file of the agent and then using this in job execution.

Check out https://github.com/laude-institute/harbor/pull/28

@li-boxuan when will be added?

@quancore This is not the highest priority right now but I'd like to hear more about your use case.

it is very iteresting that it is not the high priority. We have an agent where we would need multi-container nevironment with env vars and some artifacts. Also it is very hard to use local agent wheel in agent installation script so we have to commit everything everytime and install the agent from remote

I see, I do agree a local agent (that's not published/released publicly) doesn't really fit into today's design, while the mounted agent approach proposed in #28 *might* satisfy your use case.


Mounted agent approach is promising, may I ask when PR will be finlized and merge?

@li-boxuan Can I continue working on the  **`/nix` mounting agent** based on https://github.com/laude-institute/harbor/pull/28?
I am a NixOS user and am quite familiar with Nix-related mechanisms. I've been quite free lately, so I can prioritize implementing this feature.

> [@li-boxuan](https://github.com/li-boxuan) Can I continue working on the **`/nix` mounting agent** based on [#28](https://github.com/laude-institute/harbor/pull/28)? I am a NixOS user and am quite familiar with Nix-related mechanisms. I've been quite free lately, so I can prioritize implementing this feature.

Yes please, you are more than welcome to!
Repository: harbor-framework/harbor
Issue: Running tasks with daytona seems flaky, timeouts commonly happen
Root cause: External Dependency Breakage

<img width="1744" height="1402" alt="Image" src="https://github.com/user-attachments/assets/11a58960-17e3-4c18-82ca-fe236a116ad4" />
i'm noticing a lot of these errors when i try to spin up a bunch of containers in parallel with Harbor

from daytona team:

> "We haven’t seen this outside the harbor use case
> So seem something specific to how they spin them up"


Tasks run reliably in docker locally, but this issue happens persistently for daytona. Some daytona tasks work fine, some time out. No real rhyme or reason to it from what i've seen. Including 10 second sleeps before spawning containers did not help either.

👋 Thanks for posting! I'm seeing failures running the following command for harbor main branch version 60676c3:
```
harbor run -d terminal-bench@2.0 -e daytona -n 32 --max-retries 2 --force-build
```
due to the follow pseudo stacktrace:
```
RuntimeError: Session is closed
  at aiohttp/client.py:529 in _request()
  ↓
Caught and wrapped by daytona/_utils/errors.py:65
  ↓
DaytonaError: Failed to create sandbox: Session is closed
  ↓
Re-raised through tenacity retry mechanism
  ↓
Propagates to Trial.run() → marked as failed trial
```

so I uploaded the full `results.json`, `config.json`, and `exception.txt` files here with the details:
https://github.com/danielzayas/harbor/tree/main/jobs/2025-11-09__60676c3__oracle__daytona-0-113-1__session-closed

Closing bc we resolved this offline. The issue was a bad version of the daytona sdk that has since been updated.

Thank you for taking a look @alexgshaw ! daytona has been pinned to version 0.112.2 since your commit https://github.com/laude-institute/harbor/commit/2fe6f9ff481f2fa53a79a15c1fd46249dc4f020e on 2025-11-09, so I assume that's the version update you're referring to. Lmk if I misunderstood. 

Today, 2025-11-13 I checked out harbor at head of main bfcb3de4d2e1ef7c295f2d49250332ce9e1b5aea. I verified that version 0.112.2 is installed in my venv and ran:
```
harbor run -a oracle -d terminal-bench@2.0 -e daytona -n 16 --max-retries 2
```

84 out of 89 trials failed due to "Connection reset by peer":

- [jobs/2025-11-13__5fd4eb8__oracle__daytona](https://github.com/danielzayas/harbor/tree/main/jobs/2025-11-13__5fd4eb8__oracle__daytona)
- [FAILURE_ANALYSIS.md](https://github.com/danielzayas/harbor/blob/main/jobs/2025-11-13__5fd4eb8__oracle__daytona/FAILURE_ANALYSIS.md)

Should I be using a different daytona library version (not version 0.112.2) or is that the version you're referring to? 


Continuing the covo in Discord instead of Gtihub: https://discord.com/channels/1360039261361012928/1437233631650119741/1438654049154240672

<img width="1216" height="743" alt="Image" src="https://github.com/user-attachments/assets/db659d10-d094-4e99-88cd-3b7160bb23e0" />
Repository: harbor-framework/harbor
Issue: Error handling on non-critical paths
Root cause: Validation Gap

I was using Sandboxes + Docker to run terminal-bench v1.5 tasks. A task's cleanup failed:

> Command: docker compose -p password-recovery__zkybimr -f /home/ec2-user/sandboxes/src/sandboxes/environments/docker/docker-compose-build.yaml down --rmi all --volumes --remove-orphans. Return code: 1. Stdout:  Container password-recovery__zkybimr-main-1  Stopping\n Container password-recovery__zkybimr-main-1  Error while Stopping\nError response from daemon: cannot stop container: 07c7a55bb1357f1ecc8862f33a8cb7f513fbc2037b08500764fcb6c93b5af8f7: tried to kill container, but did not receive an exit event\n. Stderr: None. ')])>

and subsequently, the whole run terminates.
Repository: harbor-framework/harbor
Issue: Send CTRL+C upon agent timeout
Root cause: Unimplemented Feature Gap

I am working on making openhands gracefully shutdown (i.e. saving trajectory before exiting) upon receiving a SIGINT signal. I tested in terminal-bench that if harness sends a CTRL+C signal upon timeout, openhands (unreleased version) could properly save the trajectory. See https://github.com/laude-institute/terminal-bench/pull/1299 Sandboxes should be pretty similar.

Some other agents might have similar graceful shutdown behavior. At least I think it's very doable with mini-swe-agent.



Need to confirm but looks like sandboxes doesn't need this.

In terminal-bench harness we need to do this because agents run in Tmux, and are thus not cancelled by the agent coroutine when timeout happens.

Okay I am fairly confident this is not needed in sandboxes.

Reopening; this is less of a bug in sandboxes (unlike terminal-bench harness, where agents could continue running upon timeout), but still something we could consider to give agents a signal before we abruptly kill them.
Repository: harbor-framework/harbor
Issue: Alternative way to install agents
Root cause: Architectural Constraint

Currently we install agents on the fly. This is very time consuming and a waste of resources. For 200 tasks x 3 agents (I am just making numbers up), we need 600 installation (via pypi or npm).

An alternative way is to bake agent installation into task images. This is not ideal because it means we have number of images = M (tasks) x N (agents), and it doesn't scale well.

Another alternative is to build agent binaries separately and mounting them to task containers. System-level compatibility might be troublesome, but worth trying.

We currently cannot include https://github.com/laude-institute/terminal-bench/tree/main/tasks/extract-safely in t-bench because it's unfair to installed agents, but if we use the pre-install or mounting approach then it won't be a problem

Gemini researched the three aforementioned approaches and recommended the mounting approach with nix to build the agents' environments. I am working on a POC and will submit a PR soon.