zhiyuanhusf/tb2-eval-progress

TB2 evaluation progress

Terminal-Bench 2.0 evaluations for Qwen3.6-27B and Ornith-1.0-35B checkpoints using Harbor terminus-2.

Snapshot date: 2026-07-14 UTC.

Reproduction contents

This repository now includes:

• The patched sf-agent source used for evaluation.
• The 89 Terminal-Bench 2.0 task definitions, tests, instructions, and Dockerfiles.
• Evaluation, serving, node-preparation, and result-summary scripts.
• The tmax dependency files and the exact upstream tmax commit.
• The Harbor prebuilt-image compose override.
• A detailed partial-reward and RL-effect analysis in ANALYSIS.md.

Large model weights and the 89 prebuilt Docker image tarballs are not duplicated here.

Installation

Prerequisites:

• Linux, Docker, and an eight-GPU node.
• Python 3.12 and uv.
• A GPU-compatible PyTorch/vLLM installation (the original environment used vLLM 0.19.1).
• Qwen3.6-27B and/or Ornith-1.0-35B weights.
• The 89 prebuilt image tarballs, or locally built images from the included task Dockerfiles.

git clone https://huggingface.co/zhiyuanhusf/tb2-eval-progress
cd tb2-eval-progress
bash scripts/install.sh
cp env.example ~/.gemma4_env
# Edit every cluster-specific path in ~/.gemma4_env and scripts/*.sh.

The tmax source is pinned to:

https://github.com/hamishivi/tmax
1cce58a2627c39e41ab2d3033ddbb082787da23a

Before each Harbor run on a reused node:

bash scripts/node_prep.sh

Example merged-checkpoint evaluation:

MODE=branch \
KIND=qwen \
TAG=qwen-step-220 \
REPO=zhiyuanhusf/tmax-qwen36-27b-run2-merged \
BRANCH=step_220 \
GPUS=0,1,2,3,4,5,6,7 \
DATASET=terminal-bench@2.0 \
bash scripts/eval_harbor.sh

Summarize a Harbor result directory:

python scripts/summarize_results.py /path/to/results/<job-name>

Important configuration distinction

Two configurations were tested and must not be compared as if they were identical:

v1: generous-resource configuration

• --n-concurrent 8
• --override-cpus 24
• --override-memory-mb 49152
• --agent-timeout-multiplier 12
• temperature=1.0, k=1

This configuration is implemented by scripts/eval_harbor_qwenbase.sh. It produced a Qwen base score close to the reported 59.5% card result, but several tasks ran for 8–14 hours or became stuck. Runs were therefore recorded with 85–88 of 89 verifier results.

Native-resource configuration

• Task CPU, memory, and timeout from each task.toml
• --n-concurrent 20
• --agent-timeout-multiplier 1
• temperature=1.0, k=1

This is the current default in scripts/eval_harbor.sh. It completes much faster, but Qwen base is approximately 45.5%, significantly below the 59.5% card result. The native limits cause failures on compute-heavy tasks, so this configuration is useful as a controlled fast comparison but should not be described as card-aligned.

See results/progress.tsv for the current snapshot.

Current interpretation

• Under v1, Qwen base and RL checkpoints are close (roughly 58–60%), so the measured RL gain is small.
• Under native limits, Qwen RL checkpoints score above the Qwen base, but the absolute baseline is too low. This improvement is configuration-dependent and is not yet sufficient evidence for a card-level RL gain.
• Ornith later checkpoints decline under native limits; the full checkpoint curve is still being evaluated.
• Single-sample (k=1, temperature 1.0) task outcomes are noisy. A final claim should use repeated runs (k>=3, preferably k=5) under one fixed, validated configuration.

See ANALYSIS.md for the sf-agent partial-reward tables, paired task transitions, Harbor configuration sensitivity, and failure-trajectory analysis.

Failure-trajectory findings

• Most failed trajectories call mark_task_complete despite verifier failure.
• Qwen failures commonly stop after roughly 25–27 steps, below the 100-step cap.
• Many failures are executable solutions with incorrect output or semantics, rather than infrastructure crashes.
• About one quarter of failed tasks are near misses with at least 70% of verifier subtests passing.
• Across matched Qwen base/step100/step140 tasks, RL produced both stable wins and stable regressions; the net pass@1 gain was small under v1.

Operational notes

• vLLM requires the matching bundled CUDA libraries; the serving environment used unset LD_LIBRARY_PATH.
• Merged checkpoints are loaded as language-model-only Qwen VLM architectures using --hf-overrides and copied base config.json.
• Docker images are loaded from 89 local tarballs and retagged by scripts/node_prep.sh.
• Re-run node_prep.sh before every new Harbor job on a reused node. Previous runs may remove upstream image tags.
• Prune stale Docker networks when a recycled node reports all predefined address pools have been fully subnetted.
• The Harbor prebuilt compose file was locally configured with pull_policy: never to avoid Docker Hub pulls.

Files

• scripts/eval_harbor.sh: generic local/Hugging Face checkpoint evaluator.
• scripts/eval_harbor_qwenbase.sh: original v1 generous-resource Qwen base evaluator.
• scripts/eval_merged.sh: earlier sf-agent merged-checkpoint evaluator.
• scripts/node_prep.sh: local image loading, retagging, network setup.
• scripts/install.sh: installs the pinned tmax and bundled sf-agent source.
• scripts/summarize_results.py: reports accuracy, reward mean, and verifier-subtest partial mean.
• src/sf-agent/: patched sf-agent source and all 89 TB2 task definitions.
• vendor/tmax/: tmax dependency metadata and original evaluation notes.
• patches/: Harbor compose override used for local-only prebuilt images.
• ANALYSIS.md: partial-reward, RL-effect, and trajectory-failure analysis.
• hold_1node.sbatch: four-day, eight-GPU Slurm hold job.
• results/progress.tsv: live snapshot at publication time.

Paths are cluster-specific and must be edited for another environment.