kfallah/Qwen3.5-9B-tb2-opd-lora

Qwen3.5-9B TerminalBench-2 on-policy distillation LoRA

Improves Qwen3.5-9B on TerminalBench-2 from 21.6% to 27.5%, and cuts turns per episode from 53.6 to 29.4, matching its 3x larger teacher's 28.5.

A rank-32 LoRA distilled on-policy from a Qwen/Qwen3.6-27B teacher. The student drives harbor's terminus-2 agent through real multi-turn agentic episodes in E2B sandboxes; the teacher scores the student's exact sampled tokens; the per-token gap teacher_logprob - student_logprob is the training signal. Token-exact throughout: sampled ids are never re-encoded.

Trained with world-model-optimizer.

Results

17-task TerminalBench-2 holdout x 3 attempts = 51 trials per arm. Complete denominators, zero infrastructure failures, zero context overflows.

arm	solve rate	turns / episode
teacher Qwen3.6-27B	49.0% (25/51)	28.5
student before	21.6% (11/51)	53.6
student after (this adapter)	27.5% (14/51)	29.4

+5.9 points of solve rate, closing 21% of the gap to a teacher three times its size, after 4 training steps and roughly 64 optimizer updates.

It also learned the teacher's working shape. The teacher solves TerminalBench-2 in fewer, longer turns; the untrained student in many more, shorter ones. After training the student sits at 29.4 turns against the teacher's 28.5, within a turn. On-policy distillation moved the student's policy, not just its token statistics, and that is the effect that shows up most clearly.

Statistical footing, stated so you can weigh it: the paired per-task delta is +0.059 with a 95% CI of [+0.000, +0.157], bootstrapped over tasks. 14 of the 17 holdout tasks sat at floor or ceiling in both arms, so roughly 3 tasks carry the measurement and the interval is wide by construction. The promotion gate at 0.7x teacher (34.3%) was not reached.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-9B")
model = PeftModel.from_pretrained(base, "kfallah/Qwen3.5-9B-tb2-opd-lora")

Or serve with vLLM: --enable-lora --lora-modules tb2=<path>.

Details

Rank 32, alpha 32, targeting attention and MLP projections plus the unembedding. lr 5e-5, ppo loss with a raw uncentered unclipped per-token advantage, 4 steps x 16 optimizer substeps. Renderer wmo/qwen3_5_strip_history on both sides with terminus-2 compaction on.

Training was stopped at 4 of 15 planned steps: generation length grew to 5.33x its step-0 value and entropy rose 0.513 -> 0.862, overshooting the teacher's verbosity and dropping 22 of 64 episodes to context overflow. This is the checkpoint at that point, not a converged model.