gemma-4-E2B-it-danish (v1): an honestly documented negative result

This is Gemma 4 E2B (instruct) fine-tuned on 74,000 Danish instruction-transformation examples. It was built to test a simple hypothesis: can careful, light-touch LoRA fine-tuning on synthetic Danish data improve the strongest laptop-class open model's Danish performance?

The answer was no, and this repo documents exactly why. We publish the model, the adapter, and the complete evaluation because the Danish NLP community benefits more from honest numbers than from silence about failed experiments.

TL;DR

• Fluent Danish, strong stylistic instruction-following in qualitative use
• Regresses vs. stock google/gemma-4-E2B-it on 7 of 10 EuroEval Danish datasets
• Ties on sentiment classification and NER
• Largest losses on grammar judgment (ScaLA), Danish idioms, and common-sense reasoning
• Key lesson: validation loss improved monotonically (0.72 → 0.56) while actual capabilities decayed. Loss on narrow data measures dataset obedience, not quality.

If you want the best small Danish model, use google/gemma-4-E2B-it (stock). Use this repo if you're studying SFT-induced regression, distillation data design, or fine-tuning dynamics of heavily optimized instruct checkpoints.

Evaluation

Full EuroEval Danish suite (v17.6.0, few-shot, validation split), both models evaluated back-to-back on the same pod and same vLLM version, so the comparison is as close to apples-to-apples as this setup allows. Primary metric per dataset shown; the raw results JSONL files are in the companion dataset repo.

Dataset	Task	This model	Stock E2B-it	Δ
AngryTweets	sentiment	50.5	50.4	tie
DANSK	NER	51.1	50.6	tie
ScaLA-da	grammar judgment	15.9	27.0	−11.1
MultiWikiQA-da	reading comprehension	70.1	72.3	−2.2
Nordjylland News	summarization	36.5	37.0	−0.5
Danske Talemåder	idioms	35.1	48.4	−13.3
Danish Citizen Tests	knowledge	48.8	53.3	−4.5
HellaSwag-da	common sense	21.7	28.0	−6.3
IFEval-da	instruction following	71.6	78.8	−7.2
VALEU-da	values	6.6	0.2	noise

What happened, and why we think it happened

Training data: 59k Danish + 15k English→Danish instruction-transformation examples (simplify, paraphrase, summarize to exact lengths, extract facts, rewrite in register), synthesized with google/gemma-4-31B-it by schneiderkamplab (Apache-2.0). High-quality, but narrow: one task family, one instruction dialect.

Recipe (deliberately conservative): QLoRA r=16 on the instruct checkpoint, loss masked to assistant turns, effective batch 16, LR 1e-4 cosine, one epoch, 20% English mix against catastrophic forgetting. Trained with unsloth on a single L40S in ~5 hours (≈ $4 of compute).

Our reading of the regression pattern:

1. Knowledge/reasoning losses (idioms −13, common sense −6, citizenship −4.5): the signature of mild catastrophic forgetting. The English mix protected general task ability (sentiment/NER held) but not Danish-specific knowledge.
2. Grammar judgment (−11): our data trains the model to helpfully process messy Danish (spoken transcripts, archaic literature) rather than flag it. We trained an accommodating editor; ScaLA rewards a strict proofreader.
3. Instruction following (−7): the model learned our narrow instruction dialect at the cost of general compliance. IFEval's diverse constraints exposed it.
4. The training curves showed none of this. Val loss fell smoothly throughout. If you fine-tune modern instruct models, evaluate checkpoints on downstream benchmarks during training. Loss will lie to you.

Danish Foundation Models' munin-gemma4-e4b, a heavier continued-pretraining recipe on the larger sibling, regressed by even more (EuroEval rank ~2.60 vs stock E4B's ~1.86). Two independent recipes, same direction. Gemma 4's stock instruction tuning appears to sit at a sharp optimum that naive SFT of any weight moves away from.

Contents

• Merged 16-bit model (root): drop-in usable with transformers/vLLM
• adapter/: the LoRA adapter alone (~120MB), for composing or further research
• Training checkpoints (steps 250–3500) in the companion repo, useful for studying regression vs. training steps

Usage

from transformers import pipeline

pipe = pipeline("text-generation", model="deahmed/gemma-4-E2B-it-danish",
                device_map="auto", torch_dtype="bfloat16")
messages = [{"role": "user", "content": "Omskriv teksten, så et barn på 10 år kan forstå den: ..."}]
print(pipe(messages, max_new_tokens=400)[0]["generated_text"][-1]["content"])

Use of this model is subject to the upstream Gemma license terms via the base model.

Acknowledgments

• EuroEval (Alexandra Institute): the benchmark that made honest measurement possible
• schneiderkamplab: open synthetic Danish data
• unsloth: training on a hobbyist budget
• Google: releasing Gemma 4 openly

Part of an ongoing project on offline Danish AI assistants. A v2 with diverse, knowledge-bearing Danish instruction data is planned. Questions/collaboration: open a discussion on this repo.