Tev-27B

🌐 Project Page  |  📄 Paper (arXiv)  |  🤗 Tev-320B

Tev-27B is a long-horizon terminal agent model trained from Qwen3.8-27B with reinforcement learning on Tev-Gym, using progress-potential reward shaping (PPRS). It is the main model of the paper Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards.

Highlights

• Agents' Last Exam, computing & math split (ALE-CM). Pass@5 rises from 0.17 to 0.28 (+65% over Qwen3.8-27B), matching GPT-6 Astra and Claude Opus 5 at Pass@5. Mean graded score (S@1) rises from 0.33 to 0.40.
• Zero-shot transfer to longer horizons. No Terminal-Bench task is used in training. Pass@1 rises by +56% on the Terminal-Bench 3.0 "CM" subset (0.09 → 0.14) and 2.5× on the Terminal-Bench 4.0 "CM" subset (0.04 → 0.10), ahead of GLM 5.2 and Gemini 3.7 Flash.
• Mid-horizon coding is kept and improved. On Terminal-Bench 2.1, Pass@1 reaches 0.794 (base 0.764), on par with Claude Opus 4.8 (0.789).

How Tev works

Open models struggle on long-horizon tasks for two reasons: there is almost no long-horizon training data, and terminal-only rewards are too sparse to learn from. Tev addresses both.

Tev-Gym: latent-world-first environment synthesis. Each task is built backward from a latent world, a complete, correct, hidden specification of the task domain, its workflow (a DAG of stages), and the expected state after every step. Agent-facing inputs are rendered from that world and corrupted with controlled, invertible defects that the agent must repair. Because the correct answer exists before the task, the latent world serves as an exact answer key. It grades the final submission and also every intermediate state. Tev-Gym contains 1,170 verified long-horizon environments across 7 domains (software engineering, data engineering, AI/CS research, mathematics and operations research, security and forensics, infrastructure administration, and quantum computing). A frontier agent needs a median of 12.4 minutes per rollout on these tasks, about 6× longer than on the most time-intensive prior agentic corpus.

PPRS: dense progress rewards. After every turn, the agent's workspace is scored against the latent world's known-correct intermediate states, giving a progress potential Φ(s) ∈ [0, 1]. The rollout reward is

R(τ) = 1[τ passes the strict verifier] + λ · Φ(s_T),   λ = 0.25

applied with per-turn credit (return-to-go, γ = 1) and a per-turn-index group baseline. Under terminal-only rewards, GRPO-style training throws away every group in which all rollouts fail, and on long-horizon tasks that is most groups. Φ separates those rollouts by how far they got. At 27B, terminal-only reward produced almost entirely zero-advantage groups and did not complete a single optimizer step, while PPRS trained to completion.

Model details

Base model	Qwen/Qwen3.8-27B
Parameters	~27B (dense, bf16)
Training	RL with DPPO + PPRS on the full Tev-Gym corpus
Agent setting	Single bash tool in a per-task container; up to 64 tool calls, 65,536 response tokens per rollout (16,384 per turn)
Training compute	~4 days on 4 nodes of B200 GPUs (or 8 nodes of A100 GPUs)
Weights	Language model + lm_head only (see note below)
License	CC BY-NC 4.0, non-commercial use only

Note on the weights. The checkpoint contains only the text model and lm_head (12 bf16 safetensors shards). It has no vision tower and no MTP heads, although config.json is the base model's vision-language config (Qwen3_5ForConditionalGeneration). Serve it as a language-only model and do not enable MTP speculative decoding.

Training hyperparameters

Setting	Value
Objective	Group-relative policy gradient with centered advantages (DPPO loss)
Trust region	Total-variation divergence bound 0.1 on the policy ratio; no KL anchor
Rollouts per task (G)	24
Unique tasks per step	8 (192 rollouts per step)
Active sampling	Zero-variance groups discarded and refilled
Optimizer	AdamW, learning rate 1e-6, constant schedule, one epoch per batch
Sampling temperature	1.0
Prompt / response budget	2,048 prompt tokens; 65,536 response tokens
Per-turn budget	16,384 tokens; at most 64 tool calls per rollout
Tools	A single bash tool in a per-task container (python:3.13-slim base image)
Reward	Terminal pass + λΦ, λ = 0.25; per-turn credit, return-to-go with γ = 1, per-turn-index group baseline
Precision	bf16 with fp32 LM head; ZeRO-3
Learner / inference split	16 learner GPUs + 48 inference engines (64-GPU run)
Seed	42

Evaluation

Every benchmark uses 5 independent rollouts per task. S@1 is the mean graded score, P@1 the fraction of rollouts that fully solve the task, and S@5 / P@5 are their best-of-5 counterparts.

ALE-CM: Agents' Last Exam, computing & math split (18 tasks)

On ALE-CM, the open models (base and Tev) run in the single-bash harness used in training, with 64 turns and 16,384 tokens per turn. Frontier models run in their native agent harnesses (Codex, Claude Code, Gemini CLI), the setting most favorable to them.

Model	S@1	S@5	P@1	P@5
GPT-6 Astra (Codex)	0.62	0.63	0.28	0.28
Claude Fable 5 (Claude Code)	0.58	0.63	0.29	0.33
Claude Opus 5 (Claude Code)	0.58	0.60	0.28	0.28
GPT-5.6 sol (Codex)	0.58	0.60	0.28	0.28
Claude Opus 4.8 (Claude Code)	0.52	0.55	0.22	0.22
Gemini 3.7 Flash (Gemini CLI)	0.52	0.54	0.22	0.22
Qwen3.8-2.4T-A95B	0.32	0.46	0.14	0.28
DeepSeek-V4-pro	0.29	0.41	0.11	0.22
Qwen3.8-27B (base)	0.33	0.43	0.12	0.17
Tev-27B (ours)	0.40	0.54	0.16	0.28

By category, Tev-27B's largest gain is on Math & Science, where P@5 doubles from 0.25 to 0.50.

Terminal-Bench 3.0 and 4.0, "CM" subsets (zero-shot)

The computing-and-math subsets contain 22 TB 3.0 tasks and the 18 of them that remain in TB 4.0. On the leaderboard models, subset scores track full-benchmark scores closely (Pearson r = 0.96 for TB 3.0 P@1 and 0.97 for TB 4.0 P@1).

Model	TB 3.0 P@1	TB 3.0 P@5	TB 4.0 P@1	TB 4.0 P@5
Claude Opus 5	0.53	0.64	0.52	0.72
Claude Fable 5	0.41	0.64	0.46	0.61
Claude Opus 4.8	0.19	0.50	0.27	0.50
Gemini 3.7 Flash	0.07	0.18	0.09	0.17
GLM 5.2	0.04	0.18	0.03	0.13
Qwen3.8-27B (base)	0.09	0.24	0.04	0.19
Tev-27B (ours)	0.14	0.27	0.10	0.25

Terminal-Bench 2.1 (89 tasks, zero-shot, Terminus-2 agent)

Model	Pass@1	Pass@5
GPT-6 Astra (Codex, high)	0.874	0.944
Claude Fable 5 (Claude Code, xhigh)	0.838	0.933
Claude Opus 4.8 (Claude Code, high)	0.789	0.944
Claude Sonnet 5 (Claude Code, high)	0.746	0.921
Qwen3.8-27B (base)	0.764	0.864
Tev-27B (ours)	0.794	0.884

Frontier rows come from the official leaderboards or from our runs in each model's native harness. See the paper for per-category results, full tables, and the evaluation protocol.

Usage

Tev-27B is a terminal agent. It works best in a loop that gives it a single bash tool and executes its commands in a sandbox, the setting it was trained in.

Serve with vLLM

vllm serve gordonhu/Tev-27B \
  --served-model-name tev-27b \
  --language_model_only \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --max-model-len 131072

Call it as an agent

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "bash",
        "description": "Execute a bash command. Each command runs in a new subshell.",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string", "description": "The bash command to execute."}},
            "required": ["command"],
        },
    },
}]

messages = [
    {"role": "system", "content": (
        "You are a helpful coding assistant. You have access to a bash terminal. "
        "When you are confident your solution is correct, submit by running: "
        "`echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`"
    )},
    {"role": "user", "content": "Read /task/prompt.md and complete the task described there."},
]

response = client.chat.completions.create(
    model="tev-27b", messages=messages, tools=tools,
    temperature=0.7, top_p=0.95, max_tokens=16384,
)
print(response.choices[0].message.tool_calls)
# Run each `bash` call in your sandbox, append the output as a "tool" message, and repeat
# until the model submits or the turn budget (64 in our evaluations) runs out.

Intended use and limitations

• Research use only. Tev-27B is released for non-commercial research on long-horizon agents (see License).
• Run it in a sandbox. The model executes shell commands autonomously. Run it in an isolated container with no access to credentials, production systems, or data you cannot afford to lose.
• Gap to frontier models. Tev-27B still trails the strongest frontier agents on the hardest long-horizon benchmarks (for example, 0.10 vs. 0.52 P@1 for Claude Opus 5 on TB 4.0-CM).
• Domain focus. Training covers terminal-based computing-and-math workflows in synthetic environments. Results may not transfer to GUI, web, or multimodal agent tasks, and the vision encoder of the base model is not included.
• Dual use. Like any more capable coding agent, the model can be misused. The security-flavored tasks in Tev-Gym are analysis and forensics exercises over synthetic artifacts.

License

Tev-27B is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. You may use, share, and adapt the model for non-commercial purposes only, with attribution. Commercial use is not permitted.

Tev-27B is a derivative of Qwen3.8-27B, which is licensed under the Apache License 2.0. A copy of that license is included as LICENSE-Qwen3.8-27B. The weights were modified from the original by reinforcement-learning fine-tuning.

Citation

@misc{hu2026tev,
  title  = {{Tev}: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards},
  author = {Hu, Wenbo and Pramanick, Shraman and Singh, Krishna Kumar and
            Peng, Nanyun and Chang, Kai-Wei and Lee, Yong Jae},
  year   = {2026},
  eprint = {2610.18868},
  archivePrefix = {arXiv}
}

Notice

The model weights held in this repo are a reproduced version, not official model weights trained at Adobe.