paper-harness-and-weights-1.5b

Recipe: recipes/papers/harness-and-weights · Collection: Papers, replicated

Two levers on one task set: the harness (instructions, skills file, a Python tool) and the weights (GRPO with LoRA). Four cells, paired by task: neither, harness only, weights only, both. The harness was picked on the holdout before any training.

Result

2026-09-21, one H100, 79 held-out tasks in six families the training never showed, 4 rollouts a task, 40 GRPO steps per trained arm.

Cell	harness	model	pass@1	95% CI
neither	00_baseline	base	0.00	[0.00, 0.01]
harness	01_skills	base	0.09	[0.05, 0.13]
weights	00_baseline	trained	0.03	[0.01, 0.04]
both	01_skills	trained	0.09	[0.06, 0.12]

Picking the harness on the holdout moved the base +0.08 [+0.05, +0.12]. Training the weights under the bare harness moved it +0.03. Training under the chosen harness landed where the harness alone did on pass@1 and raised pass@4 from 0.09 to 0.30. One training seed per arm, so the trained pairs are unresolved. The recipe README has the full attribution table and both hack scans.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	both: trained under the 01_skills harness, 2026-09-21 run
weights	weights: trained under the bare 00_baseline harness, 2026-09-21 run

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "while-ai/paper-harness-and-weights-1.5b")  # the headline arm
model = PeftModel.from_pretrained(base, "while-ai/paper-harness-and-weights-1.5b", subfolder="weights")  # another arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/harness-and-weights
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.