enseven/lfm-2.5-think-code

LFM2.5-1.2B-Thinking — KodCode LoRA fine-tune (documented negative result)

LoRA fine-tune (r=32, α=64, attention + MLP modules, 1 epoch, LR 1e-4, bf16, 4096-token packing) of LiquidAI/LFM2.5-1.2B-Thinking on KodCode-V1-SFT-R1 — 70% code-only r1_solution targets, 30% chain-of-thought — formatted for the LFM2.5 chat template (dataset).

Evaluation result: the fine-tune made the model worse at code

Sealed HumanEval+ benchmark (128 never-before-used tasks, greedy decoding, one sample per task, EvalPlus v0.1.10, pinned llama.cpp runtime, restricted offline containers):

Row	plus pass@1 (scored)	plus rate (absence = 0)	tokens / task	extraction yield
Base model (BF16 GGUF)	0.5545 (56/101)	0.4375	≈1,654	0.789
This adapter (merged BF16)	0.4048 (51/126)	0.3984	≈148	0.984
Q8_0 of this adapter	0.4173 (53/127)	0.4141	≈128	0.992
Q6_K of this adapter	0.3889 (49/126)	0.3828	≈151	0.984

Paired on the 99 tasks scored for both models: the fine-tune passes 49 versus the base model's 56 (−7.1 pp; 16 gained, 23 lost). "Absence = 0" counts unextractable outputs as failures; the scorer's pass@1 silently drops them from the denominator, which flatters a broken model — both views are reported.

Mechanism. The fine-tune learned to skip the base model's reasoning traces: output length collapsed ~11× (≈1,654 → ≈148 tokens per task), which is also why extraction yield and decode speed improved. Those discarded reasoning tokens were doing the work. A 33-task pilot before the sealed run showed the same direction.

Recommendation

• For deployment on HumanEval-style code generation, use the base model.
• GGUF artifacts used in this evaluation (including the base-model baseline): enseven/lfm-2.5-think-code-GGUF.
• This adapter is published as a documented negative result. Do not build on it expecting coding gains. The next iteration would preserve the reasoning path and execution-verify training targets — see the repository's evaluation notes.

Provenance

• Evaluated adapter revision: e01354fecc52e62b9ca86399da10e9e40ebf51e9 (unchanged since training).
• Benchmark: EvalPlus HumanEval+ v0.1.10; frozen prompt lfm-code-v1; per-task extraction-failure policy (failures score zero, never repaired or retried); all generation and scoring inside restricted offline containers.
• Full evidence chain (frozen suites, raw generations, extraction and scoring runs, manifests): GitHub — lfm2.5-finetune-code (reports/e3/, reports/e4/).
• Attached summary reports: pilot-report.json, e4-report.json.

License

CC BY-NC 4.0 — inherited from the KodCode-V1-SFT-R1 training data. The MIT license in the GitHub repository applies to its code, not to this adapter.