underdog-lab-smollm2-360m-lora

A LoRA adapter over SmolLM2-360M-Instruct for structured factor extraction from football match scenario text.

This adapter was evaluated against a pre-declared gate and did not pass. It is not the production path. It is published because the negative result is the useful part — the gate, the frozen test set, and the numbers below are the artifact. See Evaluation.

Model details

• Developed by: Osama Moftah (sammoftah)
• Model type: LoRA adapter (PEFT) for a causal LM
• Base model: HuggingFaceTB/SmolLM2-360M-Instruct
• Language: English
• License: Apache-2.0
• Adapter size: 3.3 MB (adapter_model.safetensors)

Sources

• Training data: sammoftah/underdog-lab-scenarios
• Project: Underdog Lab — scenario extraction for a World Cup 2026 forecaster
• Write-up: https://moftah.dev/work/underdog-eval-gate

What it does

Given a free-text match scenario, the model emits a structured record of factors — the things in the text that should move a forecast. Each factor carries a type, the team it applies to, a severity and certainty in [0, 1], and the span of evidence it was drawn from. The schema also has explicit slots for unsupported_claims and ambiguities, so the model is asked to mark what it could not ground rather than quietly inventing it.

Intended use

Research and replication of the evaluation below. Not intended for production use — the base model plus a deterministic fallback outperforms it on the metrics that matter (see below).

Out of scope

Anything requiring calibrated severity or certainty values. The central failure of this adapter is that it emits near-zero severity and certainty, which a downstream zero-weight check correctly flags as unusable.

Training

Method	LoRA (PEFT 0.19.1)
Rank r	16
lora_alpha	32
lora_dropout	0.05
Target modules	q_proj, v_proj
Bias	none
Task type	CAUSAL_LM
Train / validation	861 / 56 examples

Evaluation

Benchmarked 2026-06-13 against a frozen 98-example test set (data/scenarios/test.jsonl), base vs. tuned, both quantised to Q8_0 and served through the same runtime. Thresholds were declared before the run.

Metric	Base	Tuned	Δ	Threshold
Factor micro-F1	0.0261	0.0270	+0.0010	≥ +0.15 gain	❌
Team attribution accuracy	0.0430	0.0323	−0.0108	no regression	❌
Fallback rate	0.0918	0.1735	+0.0817	≤ +0.02	❌
Schema validity	1.000	1.000	0.000	≥ 0.99	✅
Median latency	3675.6 ms	2995.4 ms	−680.2 ms	—	—

Result: 3 of 4 checks failed. Decision: NO-SHIP.

The adapter got marginally better at classifying factor type and materially worse at everything else. It emits near-zero severity and certainty, which trips the zero-weight hallucination check, which is why the fallback rate nearly doubled. It is faster, and speed was not one of the gates.

The production path remains base model + deterministic fallback.

Why this is published

A fine-tune that fails a gate is only a wasted run if you throw away the gate. The threshold table, the frozen test set, and the base-vs-tuned comparison are reusable; the weights are the least interesting output. Publishing the adapter alongside the report is what makes the negative result checkable by someone else.

Limitations and bias

• Trained on synthetic and semi-synthetic football scenarios; it will not transfer to other domains.
• Severity and certainty outputs are not calibrated and should not be read as probabilities.
• Micro-F1 is low in absolute terms for both base and tuned models — this task is not solved at 360M parameters. The comparison is valid; the ceiling is low.
• Evaluated in English only.

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "HuggingFaceTB/SmolLM2-360M-Instruct"
model = AutoModelForCausalLM.from_pretrained(base)
model = PeftModel.from_pretrained(model, "sammoftah/underdog-lab-smollm2-360m-lora")
tok = AutoTokenizer.from_pretrained(base)

Citation

@misc{moftah2026underdoglora,
  author = {Moftah, Osama},
  title  = {underdog-lab-smollm2-360m-lora: a LoRA adapter and the eval gate that blocked it},
  year   = {2026},
  url    = {https://huggingface.co/sammoftah/underdog-lab-smollm2-360m-lora}
}

Contact

moftah.dev · github.com/OsamaMoftah