zhiminy/EvalEng huggingface dataset:
Repository: harbor-framework/harbor
Issue: Support LiteLLM Proxy and Custom Anthropic API Endpoints in Claude Code Agent
Root cause: Architectural Constraint

### Problem

Users running LiteLLM proxies or custom Anthropic API endpoints cannot configure the Claude Code agent to use them. Currently, the agent only passes through specific environment variables (`ANTHROPIC_API_KEY`, `CLAUDE_CODE_OAUTH_TOKEN`, `MAX_THINKING_TOKENS`), which means users need code changes for each new configuration option.

Common use cases that are blocked:
- LiteLLM proxy deployments requiring `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_DEFAULT_SONNET_MODEL`
- Any future Anthropic CLI configuration options

### Proposed Solution

Pass through all `ANTHROPIC_*` environment variables from the Harbor environment to the Claude CLI, similar to how the OpenHands agent handles `OPENHANDS_*` variables.

#### Implementation:
```python
# Pass through all ANTHROPIC_* environment variables
# This supports LiteLLM proxies, custom API endpoints, and any future Anthropic configs
for key, value in os.environ.items():
  if key.startswith("ANTHROPIC_") and value:
	  env[key] = value
```

This approach:
- ✅ Future-proof: New Anthropic CLI environment variables automatically work
- ✅ Simpler: No need to manually add each variable
- ✅ Flexible: Supports any configuration the Claude CLI might need
- ✅ Consistent: Follows the pattern already used by OpenHands agent (see src/harbor/agents/installed/openhands.py:851-853)

#### Usage Example

Users can now configure LiteLLM proxy support:

```bash
export ANTHROPIC_BASE_URL="https://your-litellm-proxy.com"
export ANTHROPIC_AUTH_TOKEN="your-proxy-auth-token"
export ANTHROPIC_DEFAULT_SONNET_MODEL="gcp/claude-sonnet-4-5"

harbor run --dataset terminal-bench@2.0 --agent claude-code --n-concurrent 4
```

Or any other Anthropic-related environment variables that the Claude CLI supports.

#### Precedent

The OpenHands agent already uses this pattern:
```python
# src/harbor/agents/installed/openhands.py:851-853
for key, value in os.environ.items():
  if key.startswith("OPENHANDS_"):
	  env[key.replace("OPENHANDS_", "")] = value
```
	  
#### Backward Compatibility

✅ Fully backward compatible. Existing configurations continue to work unchanged. The Harbor --model flag still takes precedence and overrides ANTHROPIC_MODEL when specified.

#### Files Changed

- src/harbor/agents/installed/claude_code.py - Update create_run_agent_commands() method

Let me know if you would rather me submit a PR directly rather than open an issue.
Repository: harbor-framework/harbor
Issue: PODMAN not supported
Root cause: Unimplemented Feature Gap

# Issue:
When running harbor it defaults to use incompatible flags.

```
$ harbor run --agent oracle --path
<redacted>
docker-compose: error: unrecognized arguments: --rmi
Error: executing /usr/bin/docker-compose -p 9a7dcc7f-0a93-4239-8cda-112b22f7__zu9nqwp -f /home/nuci3/.local/share/uv/tools/harbor/lib/python3.13/site-packages/harbor/environments/docker/docker-compose-build.yaml down --rmi all --volumes --remove-orphans: exit status 2
```

# Expected Results:
harbor executes as documented.

# Additional Details:
## OS
```
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=25.10
DISTRIB_CODENAME=questing
DISTRIB_DESCRIPTION="Ubuntu 25.10"
PRETTY_NAME="Ubuntu 25.10"
NAME="Ubuntu"
VERSION_ID="25.10"
VERSION="25.10 (Questing Quokka)"
VERSION_CODENAME=questing
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=questing
LOGO=ubuntu-logo
```

## Runtime
```
podman-compose version 1.3.0
podman version 5.4.2

crun version 1.21
commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL

buildah version 1.39.3 (image-spec 1.1.0, runtime-spec 1.2.0)
tini version 0.2.1_catatonit

netavark version 1.14.0-2
conmon version 2.1.12-4

```

I've also using Podman instead of Docker, and I spent one day to do a adaptation because docker-compose not very compatible with podman-compose. But it's based on v0.1.32, not the latest. So if you or someone else want a code, please at me, and I will reorganize and upload the codebase.

it would be nice to get this as a PR @HairlessVillager 

Hey folks, I've just opened a PR to address this: #1432

It introduces a dedicated `PodmanEnvironment` for local execution as a built-in alternative to Docker. Instead of just a naive wrapper, it fully handles `podman` and `podman-compose` lifecycles, dynamic container resolution, and includes comprehensive unit tests.

Let me know what you think or if you'd like any changes during review!
Repository: harbor-framework/harbor
Issue: BUG: where does 4096 come from?
Root cause: Validation Gap

in this error message: https://github.com/laude-institute/harbor/blob/027b6b52331aab576bddd0a1dab3e76752e8e33e/src/harbor/agents/terminus_2/terminus_2.py#L935C1-L940C14

Thanks for reporting. This is the default value used in litellm but agree we should use litellm library to get the actual output token count limit.
Repository: harbor-framework/harbor
Issue: Syntax Error in Registry.json
Root cause: Validation Gap

I think there is an syntax typo in registry json file https://raw.githubusercontent.com/laude-institute/harbor/main/registry.json here.

When I ran harbor dataset list, I ran into this issue:
`JSONDecodeError: Expecting ',' delimiter: line 11377 column 5 (char 466521)`

It would be great to have this file tested in CI before release, or at least locked to a release. Currently an error here takes out any evals using Harbor's default registry url even if the release hasn't changed.

In the meantime, here's a workaround using the last commit `harbor run -d terminal-bench@2.0 --registry-url https://raw.githubusercontent.com/laude-institute/harbor/9f398835a353/registry.json`
Repository: harbor-framework/harbor
Issue: Cannot run endpoints hosted on Azure OpenAI
Root cause: Validation Gap

Thanks for releasing this framework! I would like to be able to run Harbor with models hosted Azure OpenAI.

Command: `harbor run -d terminal-bench@2.0 -m "azure/gpt-5-mini" -a "mini-swe-agent" -t multi-source-data-merger`

Expected outcome: agent runs

Outcome:

```bash
$ harbor run -d terminal-bench@2.0 -m "azure/gpt-5-mini" -a "mini-swe-agent" -t multi-source-data-merger
# Console output
Mini-swe-agent trajectory file jobs/2025-12-22__20-20-00/multi-source-data-merger__kH3onQT/agent/mini-swe-agent.trajectory.json does not exist
  1/1 Mean: 0.000 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:01:05 0:00:00
Results written to jobs/2025-12-22__20-20-00/result.json
     mini-swe-agent (gpt-5-mini) on terminal-bench      
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Metric                 ┃ Value                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Agent                  │ mini-swe-agent (gpt-5-mini) │
│ Dataset                │ terminal-bench              │
│ Trials                 │ 0                           │
│ Errors                 │ 1                           │
│                        │                             │
│ Mean                   │ 0.000                       │
│                        │                             │
│ Exception Distribution │                             │
│   ValueError           │ 1                           │
└────────────────────────┴─────────────────────────────┘

# Inside job.log:
Trial multi-source-data-merger__kH3onQT failed: Unable to determine API key for model azure/gpt-5-mini: Unset API variable for model azure/gpt-5-mini. Please set AZURE_API_KEY or MSWEA_API_KEY environment variable. Please set MSWEA_API_KEY environment variable as fallback
Not retrying trial because the maximum number of retries has been reached
```

I made a PR with the fix at #269. Would appreciate merging or letting me know what needs to change in order to merge so I + others who want this functionality can stay using upstream. Thanks!
Repository: harbor-framework/harbor
Issue: `claude-code` agent fails on Alpine-based task environments
Root cause: Architectural Constraint

## Description

The `claude-code` agent's install script assumes a Debian/Ubuntu environment and fails on Alpine Linux containers because it uses `apt-get`, which doesn't exist on Alpine.

## Steps to Reproduce

```bash
uv run harbor jobs start -d compilebench@1.0 -a claude-code -m claude-sonnet-4-20250514 -t "coreutils-old-version-alpine"
```

## Expected Behavior

Agent installs successfully and attempts the task.

## Actual Behavior

Agent installation fails:

```
/installed-agent/install.sh: line 3: apt-get: command not found
/installed-agent/install.sh: line 4: apt-get: command not found
...
nvm: install v22.21.1 failed!
/installed-agent/install.sh: line 11: npm: command not found
```

The task completes with `reward = 0.0` because the agent never ran.

## Root Cause

The `claude-code` agent's `install.sh` uses Debian-specific commands:

```bash
apt-get update
apt-get install -y curl
```

Alpine Linux uses `apk` instead.

## Suggested Fix

Add distro detection to the install script:

```bash
if command -v apk &> /dev/null; then
    # Alpine
    apk add --no-cache curl bash nodejs npm
elif command -v apt-get &> /dev/null; then
    # Debian/Ubuntu
    apt-get update && apt-get install -y curl
    # ... existing nvm/node setup
fi
```

## Affected Datasets

Any dataset with Alpine-based task containers, including:
- CompileBench: `coreutils-old-version-alpine`, `coreutils-static-alpine`

## Environment

- Harbor version: latest (installed via `uv tool install harbor`)
- Agent: `claude-code`
- Task: `compilebench@1.0 / coreutils-old-version-alpine`



Supporting `alpine` is essential because it is a widely used base image due to its smaller size. Moreover, we find some models overfitting just for `ubuntu`. Still wonder if there is a more generic fix for that, though it may require architectural changes.
Repository: harbor-framework/harbor
Issue: registry.json is invalid JSON after USACO adapter commit
Root cause: Validation Gap

## Bug

The `registry.json` file is invalid JSON after commit b972d81 (Adding USACO Adapter).

## Error

```
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 7161 column 9 (char 366068)
```

## Location

Line 7160-7164 shows the issue:

```json
                "path": "datasets/aime/aime_ii-15"
              }
        "name": "compilebench",
```

Missing `}],` to close the previous dataset before starting `"name": "compilebench"`.

## Impact

All `harbor run` and `harbor datasets list` commands fail.

## Verification

```bash
curl -s "https://raw.githubusercontent.com/laude-institute/harbor/main/registry.json" | python3 -m json.tool
# Fails with JSONDecodeError
```

This is fixed now, thanks for the catch.
Repository: harbor-framework/harbor
Issue: environment.exec(): bash: cannot set terminal process group (-1): Not a tty bash: no job control in this shell
Root cause: Interface Contract Mismatch

https://github.com/laude-institute/harbor/blob/2ad21b07db051721064f2ebc8830b2bd4205281c/src/harbor/environments/docker/docker.py#L261

https://github.com/laude-institute/harbor/blob/2ad21b07db051721064f2ebc8830b2bd4205281c/src/harbor/environments/docker/docker.py#L135
 

`_run_docker_compose_command` sets `stdin` to `DEVNULL`, while `environment.exec()` passes the `-ic` flags to `bash`. This causes `bash` to fail when trying to allocate a TTY, which in turn produces a warning:

```
bash: cannot set terminal process group (-1): Not a tty
bash: no job control in this shell
```

Given that `exec` does not actually support an interactive shell, it would be better to change `-ic` to `-c`.


Can you try the change and lmk if it works? iirc we added the `-i` to ensure the startup files (e.g. `.bashrc`) get sourced 

https://chatgpt.com/share/694212c0-3890-800a-9696-7cebd26df7a4
```bash 
bash -c 'source ~/.bashrc; your_command'
```

I’m not sure whether specific tasks depend on `.bashrc`, but if they do strongly depend on it, maybe that should be explicitly stated in the task instructions. In that case, it might be better to let the agent decide on its own whether to source `~/.bashrc`.

Alternatively, `environment.exec` could behave more like `docker exec` and execute the command directly, instead of automatically spawning a shell to run it. If a task needs a shell, the agent could explicitly prepend `bash -c` to the command passed to `exec`.



I think there are a few options to consider:

1. Change the implementation of `environment.exec()` from `bash -ic "cmd"` to `bash -c "source ~/.bashrc; cmd"`.
2. Read `task.toml` to decide whether to use `bash -ic` or `bash -c`.
3. Only provide a basic interface, similar to `docker exec`, and let the agent decide whether to run the command via `bash -c`.
Repository: harbor-framework/harbor
Issue: Integrate Tinker + Harbor for RL

For the integration, should we add a new `harbor train` command? Or a new `src/training` folder with prebuilt RL training integrations. This seems cleaner, given we'll have many training integrations (Tinker, SkyRL-train, SkyRL-tx, vanilla PyTorch, etc.). 

See WIP for Tinker/SkyRL-tx/Prime-rl integration [here](https://github.com/abundant-ai/harbortrainer).

I'm also working on Prime-RL + Harbor for RL training
Repository: harbor-framework/harbor
Issue: Revert litellm hack once upstream fix is released

Revert https://github.com/laude-institute/harbor/pull/181 once litellm fix is in-place and used by openhands' new version(s)

litellm PR: https://github.com/BerriAI/litellm/pull/17284

@li-boxuan this has been fixed upstream, can we revert now?

It's fixed upstream in litellm but not picked up by latest openhands (1.0.0) yet
Repository: harbor-framework/harbor
Issue: migrate tasks from tb 1.0 to harbor fails
Root cause: External Dependency Breakage

I used the migration tool to migrate tb 1.0 tasks to harbor/tb 2.0
```
harbor tasks migrate -i terminal-bench-core==0.1.1 -o terminal-bench-core==0.1.1_migrated
```

but when running the tasks in harbor, it fails with the following error:

`invalid mount config for type "volume": invalid mount path: ':' mount path must be absolute`

it seems these environment variables are in the resulting task Dockerfile but are not specified the migration tool nor by harbor when running the task:
```
    volumes:
      - ${T_BENCH_TASK_LOGS_PATH}:${T_BENCH_CONTAINER_LOGS_PATH}
```


just tried with a recent commit 62870f0b5b2f8d2a400a5c0eb915c7c7ce1ae361 and it works.  i was previously using an installation from pypi or uv.
Repository: harbor-framework/harbor
Issue: Feature Request: Add support for AI IDE
Root cause: Unimplemented Feature Gap

### Feature Request: Add Support for Evaluating AI IDEs

**Is your feature request related to a problem? Please describe.**

Currently, Harbor evaluation framework only supports CLI-based agents and does not provide benchmarks or evaluation capabilities for AI-powered IDEs such as Cursor, Antigravity, Qoder, and Trae. This limits our ability to assess and compare how these AI IDEs perform on real-world development tasks.

**Describe the solution you'd like**

Extend Harbor's evaluation framework to support benchmarking AI IDEs.
This would include:

 **IDE / Plugins:**
   - Cursor
   - Antigravity
   - Qoder
   - Trae support
   - VS Code extension (as many AI IDEs are VS Code-based)

You can somewhat easily implement this feature, but it really depends on each extension's registered commands (the quick way to do it using vscode extension test runner)

I can share our solution which we use on azad.bot

> You can somewhat easily implement this feature, but it really depends on each extension's registered commands (the quick way to do it using vscode extension test runner)
> 
> I can share our solution which we use on azad.bot

@matannahmani hello, I am also working on testing AI IDE on such benchmarks. Could you pls share your solution? any git repo or blog?
Repository: harbor-framework/harbor
Issue: Feature Request: Harbor as a LLM proxy
Root cause: Unimplemented Feature Gap

Using Harbor with custom inference engines (vLLM/SGLang) running on local GPU cluster + agents running on remote cloud cluster (e.g. Daytona) is a common use case for evaluation and RL. It's non-trivial for agents to communicate with LLM without setting up a proxy server that has a public endpoint. As @penfever points out, [Pinggy](https://pinggy.io/) might be a handy solution but it's not free. Would be nice if Harbor could act as a LLM proxy, exposing a public endpoint for agents to interact with from the remote sandboxes as illustrated below:

<img width="1024" height="559" alt="Image" src="https://github.com/user-attachments/assets/03d0c57d-79ef-4a88-82d2-806d35812737" />

Note that this is also a blocker for https://github.com/laude-institute/harbor/pull/127

LiteLLM provides an LLM proxy and could be used purely in that role—as an external dependency rather than an internal module. In this setup, all LLM configuration would be handled by the external proxy and managed as part of the environment abstraction.
Repository: harbor-framework/harbor
Issue: Feature Request: Terminus-2 to support interleaved thinking
Root cause: Unimplemented Feature Gap

Some models like kimi-k2-thinking [1] and minimax-m2 [2] work best when the entire reasoning content history is included in subsequent LLM calls.

We could send back reasoning_content in Terminus-2 if configured.

Example of how to do it with litellm can be found in [OpenHands](https://github.com/OpenHands/software-agent-sdk/blob/c196a24148b53485127a53b1de0ba4262013343c/openhands-sdk/openhands/sdk/llm/message.py#L295-L297)

[1] https://platform.moonshot.ai/docs/guide/use-kimi-k2-thinking-model#usage-notes
[2] https://x.com/MiniMax__AI/status/1985375617622454566

We should do an evaluation to verify its performance improvement.
Repository: harbor-framework/harbor
Issue: Question: bug in summarization fallback logic in Terminus 2 agent
Root cause: Algorithmic Error

Thanks for building an awesome framework!

There seems to be a bug in the summarization fallback logic in the Terminus 2 agent. 

If we hit `ContextLengthExceededError` in the `_query_llm` function, the _summarize function is called [here](https://github.com/laude-institute/harbor/blob/main/src/harbor/agents/terminus_2/terminus_2.py#L893-L895). If the summarizing call fails, the error handling will return the original instruction [here](https://github.com/laude-institute/harbor/blob/main/src/harbor/agents/terminus_2/terminus_2.py#L459-L461) leaving `chat._messages` unchanged. Then, the agent wrongly assumes that summarization succeeded, which can lead to an infinite loop of failed summarizations until reaching agent timeout. [Fallbacks 2 and 3](https://github.com/laude-institute/harbor/blob/main/src/harbor/agents/terminus_2/terminus_2.py#L904-L928) would not be run in this case.

Good catch, thank you! I'll fix it.
Repository: harbor-framework/harbor
Issue: Docker environment fails when using relative path '.' - trial ID violates Docker Compose naming
Root cause: Validation Gap

## Problem

When running Harbor with a Docker environment using a relative path `.` from inside a task directory, the generated trial ID starts with `__` (double underscores), which causes Docker Compose to fail.

## Error Message

```
invalid project name "__9mmlzw7": must consist only of lowercase alphanumeric characters, hyphens, and underscores as well as start with a letter or number
```

## Reproduction Steps

1. Navigate into a task directory:
```bash
cd tasks/my-task
```

2. Run Harbor with relative path:
```bash
harbor run -p "." -a "oracle"
```

3. Observe the failure with trial IDs like: `__9mmLzW7`, `__4gCKmTe`, `__tqBwyAc`

## Root Cause

In `src/harbor/environments/docker/docker.py` line 150, the `session_id` is used directly as the Docker Compose project name:

```python
self.session_id.lower().replace(".", "-"),
```

When using `-p "."`, Harbor doesn't derive a task name, so the trial ID is just the random suffix prefixed with `__`, violating Docker Compose's naming requirements (must start with letter/number).

## Workaround

Run Harbor from the project root using the full task path:

```bash
harbor run -p "tasks/my-task" -a "oracle"
```

This generates trial IDs like `my-task__ABC123` which start with a letter and work correctly.

## Environment

- Harbor version: v0.1.18
- Docker Compose: latest
- OS: macOS


This is fixed now
Repository: harbor-framework/harbor
Issue: Question: Streaming on logs
Root cause: Documentation Deficiency

I am not able to see agent stdout log in streamed fashion before agent execution finished?


Are you using docker or a cloud sandbox?

I am using docker

if your docker containers are local, the logs directories are mounted so any files that are being continuously written to should be visible even in a partial state. let me know if you have a more specific example of what you are hoping to see
Repository: harbor-framework/harbor
Issue: Flaky test_deterministic_openhands.py due to openhands' non-determinism
Root cause: Algorithmic Error

`test_deterministic_openhands.py` sometimes fails due to trajectory mismatch. This mismatch seems to come from openhands' own non-determinism.

```
            {
  -             'message': '',
  +             'message': 'Added workspace context',
                'source': 'system',
                'step_id': 4,
            },
            {
  -             'message': 'Added workspace context',
  +             'message': '',
                'source': 'system',
                'step_id': 5,
            },
```
Repository: harbor-framework/harbor
Issue: Feature Request: Interactive Trajectory Viewer
Root cause: Unimplemented Feature Gap

Now to inspect trajectories we need to use a text editor or asciinema, which is a bit rough. Other agent frameworks already have nicer UIs for this, like [swe-agent’s inspector](https://swe-agent.com/latest/usage/inspector/) and [OpenHands/trajectory-visualizer](https://github.com/OpenHands/trajectory-visualizer).

It’d be great if Harbor could generate a self-contained HTML viewer from `trajectory.json` files, for example:

```bash
harbor inspect trajectory1.json trajectory2.json -o trajectories.html
```

The generated HTML would be standalone and openable in any browser, with things like:

* showing all steps (system / user / agent)
* step details with messages, tool calls, observations, and metrics
* metrics summary for total cost, token usage, etc.

This would make debugging a lot easier. I’m happy to work on this if it sounds good.

C.C. @harshraj172 who might have started working on that

@digitsisyph im one of the maintainers of Opik and we have a native Harbor intergration https://www.comet.com/docs/opik/integrations/harbor and we are 100% open-source. You can automatically send all your trajectories and traces into Opik and visualize and drilldown, can also apply online evals for particular turns and do annotations etc.