Post
2836
Loss went from 2.35 to 0.27 in 50 steps. Clean, textbook convergence curve.
Held-out score: 0/10 before fine-tuning. 0/10 after.
Ran a before/after LoRA fine-tune on IFM/K2-Horizon-0.9B (Apache 2.0, released this week) on a binary fabrication-detection gate — entirely on a free CPU tier, no GPU. The training loss says it learned something real. The eval says it learned nothing that generalizes.
Looked at the actual raw outputs instead of trusting the score. Both before and after, the model never once emits TRUE or FALSE — it just continues the system prompt as text: "The user is asking me..." before, "The user is asking for..." after. Fine-tuning moved the failure string by two words. It didn't touch the failure.
Most likely cause: a 0.9B model that doesn't follow "answer in one word" untrained won't learn to from 50 gradient steps on 194 rows — it'll memorize the training set's surface form (which is exactly what the loss curve shows) without the instruction-following ever kicking in. The fix isn't more steps, it's checking base instruction-following before spending the run at all.
Writing up a clean negative is worth exactly as much as a clean positive if you actually open the raw generations instead of stopping at the number. Full run, scripts, and raw JSON:
SoulInPsyAbstract/sipa-os-governance
Held-out score: 0/10 before fine-tuning. 0/10 after.
Ran a before/after LoRA fine-tune on IFM/K2-Horizon-0.9B (Apache 2.0, released this week) on a binary fabrication-detection gate — entirely on a free CPU tier, no GPU. The training loss says it learned something real. The eval says it learned nothing that generalizes.
Looked at the actual raw outputs instead of trusting the score. Both before and after, the model never once emits TRUE or FALSE — it just continues the system prompt as text: "The user is asking me..." before, "The user is asking for..." after. Fine-tuning moved the failure string by two words. It didn't touch the failure.
Most likely cause: a 0.9B model that doesn't follow "answer in one word" untrained won't learn to from 50 gradient steps on 194 rows — it'll memorize the training set's surface form (which is exactly what the loss curve shows) without the instruction-following ever kicking in. The fix isn't more steps, it's checking base instruction-following before spending the run at all.
Writing up a clean negative is worth exactly as much as a clean positive if you actually open the raw generations instead of stopping at the number. Full run, scripts, and raw JSON:
SoulInPsyAbstract/sipa-os-governance
