digest-sft3

SFT pass on HuggingFaceTB/SmolLM2-135M-Instruct from digest-finetune, trained on a diff-aware dataset (prompts include file stats + truncated patches on sparse-commit days, matching what production git-digest actually shows the model — the prior sft2 checkpoint was trained on commit-message-only prompts, a mismatch with production input).

Training

6 epochs over 99 examples, final training loss 1.944, mean token accuracy 0.74. The last step is published; there is no checkpoint selection (see Replaced weights below for why).

Evaluation

Mean reward 0.6724 over all 10 held-out days, scored by scripts/eval_reward.py, greedy decoding, max_new_tokens=768. 1 of 10 days truncated at the cap; no day scored 0.000.

Reward ladder

Every row scored by the same reward.py, the same 10 held-out days, greedy, max_new_tokens=768. This is the only table on this card where the numbers are comparable to each other.

	model	reward
	untrained base	0.0000	░░░░░░░░░░░░░░░░░░░░░░░░
	digest-sft2	0.2100	█████░░░░░░░░░░░░░░░░░░░
	digest-sft3	0.6724	████████████████░░░░░░░░

per-model detail (truncation + zero-scoring days)

model	mean	days	truncated at cap	scored 0.000
untrained base	0.0000	10	7/10	10/10
digest-sft2	0.2100	10	5/10	7/10
digest-sft3	0.6724	10	1/10	0/10

Raw generations for every row are published alongside these weights, one file per model: eval/ — each log carries the full completion and its per-component score, so every number above can be checked against the text that produced it.

What changed between sft2 and sft3

Both the data and the scorer were rewritten. Prompts are now diff-aware — they carry file stats and truncated patches, matching what production git-digest actually sends — where digest-sft2 only ever saw commit messages. reward.py was then hardened by adversarial fuzzing and teacher calibration, closing four scoring holes (fabricated summaries, orphan prose, binary coverage credit, and omission costing less than a weak section) and adding penalties for truncation, self-repetition and summary echo.

Earlier published totals for these checkpoints (base 0.3040, sft2 0.4990) came from evaluate.py at max_new=400 with repetition_penalty=1.08, under the pre-hardening formula. They differ from the table above in three ways at once — scorer, token cap, and repetition penalty — so no single delta can be attributed to the reward rewrite, and they are not restated here as a comparison. The ladder above replaces them outright: it is the only measurement where all three checkpoints share settings.

What the rescoring does show is that the untrained base earns 0.0000 under the current scorer, on all 10 days. It emits **Summary** instead of ## Summary, invents sections outside the schema, fabricates its content, and loops the same two paragraphs until it hits the token cap. Any nonzero credit for that was the old formula's error.

Decoding

The ladder above is greedy. Sampled at temperature=0.8, top_p=0.95, digest-sft3 scores 0.904 with 0% truncation — but on 3 days × 8 rollouts, not the full 10, so it is kept out of the table rather than compared against numbers measured differently. Under sampling, coverage and grounding are maxed (1.000 across all 24 rollouts) and the only remaining cost is a summary-echo penalty on ~29% of generations.

The looping failure described in earlier versions of this card is a greedy-decoding artifact, not a policy defect: the day that loops to the token cap under greedy scores 8/8 perfect when sampled. Prefer temperature=0.8 in production.

Scope

• The ladder is 10 held-out days per model. That is a small eval set; treat differences of a few hundredths as noise, and the base-vs-trained gap as real.
• The untrained base scores 0.0000 on a task it was never trained for, given diff-aware prompts it has never seen, against a rubric written for this digest schema. It is the right number for "which of these should I use" and the wrong number for "how good is SmolLM2".
• The temp-0.8 result is 3 days × 8 rollouts, not 10 days. It is the weakest-evidence number on this card.

Replaced weights

An earlier upload under this name claimed mean reward 1.0000. That number was wrong twice over and those weights have been replaced:

1. It came from argmax reward over 52 checkpoints scored on the same 10-row eval set. With n=10 and 52 noisy candidates the winner is selected for luck on that exact set.
2. The reward function it maximised did not penalise truncation or self-repetition, so a completion that looped one fragment to the token cap scored a clean 1.0. Fixed in e1e71f2 (truncation, trigram-repetition and summary-echo penalties); the same run rescores to 0.6724.

The 52 checkpoint-* folders have been removed — they were the artifacts of that selection procedure. They remain in this repo's git history.

License

Weights: Apache-2.0 (inherited from the base model). Training code: MIT — usr-wwelsh/digest-finetune.