zhw1111/skillhelix-14b-v1521

Skill-HELIX 14B (v15.21)

Full-parameter SFT of Qwen2.5-14B-Instruct for the AppWorld interactive-coding agent benchmark, trained on 504 skill-augmented full-conversation trajectories (87 train tasks × 6 sampling temperatures, hole-repaired to near-complete grid).

Results (AppWorld test_normal, T=0, single attempt, 50-step cap)

Metric	Score
Task Goal Completion (TGC)	51.2% (86/168)
Scenario Goal Completion (SGC)	30.4% (17/56)
d1 / d2 / d3	78.9% / 58.3% / 20.6%

Reference points under the same protocol: SAGE-14B baseline 35.7% TGC; a 32B sibling of this recipe reaches 61.9% TGC.

What makes it different

• <think> + <code> structured steps with terse reasoning (median think length ≈ 159 chars);
• Scenario function inheritance: functions that verifiably worked in an earlier sibling task are re-offered with a real worked invocation example — first-call success of inherited functions ≈ 94% in eval;
• Retrieved experience bank with citation behavior trained into the model (cites entry ids like [wf_004] before applying them);
• Frozen byte-identical prompt renderer across SFT and evaluation.

Usage

Serve with vLLM (context 32k native; long-context eval used yarn ×2):

vllm serve <this-repo> --max-model-len 32768

The model expects the Skill-HELIX prompt contract (supervisor header, function policy, retrieved-experience block, scenario position). It is a research artifact, not a general assistant.

Provenance

Training data was teacher-distilled on AppWorld train scenarios only; train/dev/test scenario splits are disjoint; no evaluator internals or ground truth appear in prompts. Full run records, manifests and fingerprints are archived by the authors.