hafsteinn/icelandic-modernbert-large

Icelandic ModernBERT, size large

Question this run answers. How good does a ModernBERT-architecture encoder trained only on Icelandic get at this size when it sees 4x the Chinchilla-optimal number of tokens (80 tokens per parameter)? It is one rung of a size ladder (xs, s, m, base, large) trained with one recipe, so the rungs can be compared with each other and with existing Icelandic encoders (IceBERT, ELECTRA/ConvBERT-IGC, XLM-R, mmBERT, ScandiBERT) under one fine-tuning protocol.

Release. Released under the MIT licence with the paper, as one rung of its scratch-trained size ladder (xs to large, at 80 and 160 tokens per parameter); the training corpus is not released, because not every source states a licence that permits redistribution. Parts of the training data carry licences that require attribution (CC BY 4.0: IGC; CC BY-SA: Wikipedia, Wikisource) or come from web crawls with mixed rights (FineWeb-2, FinePDFs, HPLT 3.0, MaCoCu, MADLAD-400; IC3-v2 states no licence). The IGC restricted-licence parts (News2, Books) were not used. MIM-GOLD-NER, used only for evaluation, has a research-only licence, so no model fine-tuned on it is released.

Model

Architecture	ModernBERT (Warner et al. 2024): pre-norm, GeGLU, RoPE (global theta 160k, local 10k), alternating attention (global every 3rd layer, local window 128), no biases, tied embeddings
Shape	28 layers, hidden 1024, GeGLU inner 2624, 16 heads (Ettin/ModernBERT ladder)
Parameters	377.8M total = 33.6M embedding + 344.3M non-embedding
Vocabulary	32,768 byte-level BPE trained on our Icelandic corpus (NFC, digits split)
Context	trained at 1,024 tokens, last 10% of tokens at 8,192
Loads with	transformers AutoModelForMaskedLM (model_type modernbert)

Fill-mask. Write the mask token right after the previous word, with no space: Reykjavík er höfuðborg[MASK]. The tokenizer keeps each word's leading space inside the word's token, so a space typed before the mask becomes a token of its own. In training such a lone space comes mostly before a number, because digits are split off, so with the space the model predicts digits.

Training

Tokens seen	30.23B = 80.0 tokens/parameter (4x Chinchilla, counting all parameters as Hoffmann et al.); 88 per non-embedding parameter
Steps	28,827 at 1,048,576 tokens per step
Objective	MLM, 30% masking (80/10/10), 15% in the decay phase; loss on masked tokens only
Optimiser	AdamW (0.9, 0.98), eps 1e-6, decoupled weight decay 1e-05, grad clip 1.0, bf16 autocast
Schedule	warmup 907M tokens, constant peak LR 0.0004, 1-sqrt decay to 0 over the last 15%
Hardware	MareNostrum 5 accelerated partition (4x H100 64 GB per node), DDP, FlexAttention with document masks (no cross-document attention)

Data

Deduplicated (exact + MinHash 0.75 across all sources, curated sources kept first), quality- and language-filtered, and decontaminated against every evaluation set (13-gram and whole-sentence matching). Per-source epochs below are for the 1,024-token phase.

source	docs	tokens (unique, after filtering)	epochs
dynaword	63,034	0.059B	5.99
finepdfs	230,722	1.862B	3.00
fineweb2	1,977,502	1.614B	3.00
hplt3	1,715,238	1.376B	3.00
ic3v2	964,100	0.364B	3.00
igc	2,012,883	2.012B	3.00
macocu	2,083,595	1.173B	3.00
madlad	604,825	0.559B	3.00
wiki	8,843	0.001B	5.99
all	9,660,742	9.02B

Sources: IGC-2024 open part (arnastofnun/IGC-2024, CC BY 4.0); the non-IGC subsets of danish-foundation-models/icelandic-dynaword (government gazette and Rafbókavefur under Icelandic copyright law's exemptions; Wikipedia, Wikisource, Wikibooks, wiki comments, radio talks CC BY-SA; sagas CC BY; Gutenberg); Icelandic Wikipedia (wikimedia/wikipedia 20231101.is); MaCoCu-is 2.0 (CC0); FineWeb-2 and FinePDFs isl_Latn (ODC-By); HPLT 3.0 isl_Latn; MADLAD-400 is, clean split (ODC-By); IC3-v2 (Miðeind; licence not stated, Common Crawl derived). IcePaHC excluded (evaluation source); restricted IGC parts (News2, Books) not used.

Results

Held-out MLM (15% masking) loss per source at the end of training: {"dynaword": 0.442, "finepdfs": 1.171, "fineweb2": 1.051, "hplt3": 1.073, "ic3v2": 0.724, "igc": 0.738, "macocu": 0.964, "madlad": 0.879, "wiki": 0.589}.

Fine-tuning (same protocol for every model: learning rate picked on dev from a 3-value grid with seed 0, then 5 seeds at that rate; best-dev epoch; 95% CI from a hierarchical bootstrap over seeds and test items):

task (metric)	this model	95% CI	seeds	IceBERT (reference)	difference vs IceBERT [95% CI]
iceec (mcc)	50.0	47.5 to 52.5	5	47.1	+2.9 [+0.8, +5.0]
ner (f1)	94.5	93.8 to 95.1	5	94.1	+0.3 [-0.1, +0.8]
pos (acc)	95.5	95.1 to 95.8	5	95.4	+0.1 [-0.1, +0.3]
qa (f1)	56.0	54.6 to 57.4	5	57.5	-1.4 [-2.3, -0.6]
scala (mcc)	73.4	69.4 to 77.1	5	73.6	-0.1 [-3.6, +3.5]

Tasks: NER = MIM-GOLD-NER (IceBERT split, entity micro F1); POS = UD Icelandic-Modern + GC (UPOS accuracy); ScaLA = acceptability built from the same UD splits (MCC); IceEC = error vs no error (MCC); QA = NQiI + RUQuAD-1 (token F1).

Caveats

• One pretraining seed per size; differences between sizes include pretraining-seed noise that the CIs do not cover.
• The 4x Chinchilla budget is a decoder-derived heuristic; no compute-optimal law for MLM encoders is established.
• Larger sizes repeat the data several times (see epochs); returns from repetition fall off beyond about 4 epochs (Muennighoff et al. 2023).
• IGC JSONL documents were not used for the 8,192-token phase because their paragraph order is not guaranteed.
• AdamW with gradient clipping replaces ModernBERT's StableAdamW; global RoPE theta is 160k from the start.
• The fine-tuning learning rate comes from a 3-value grid per size class. For large only, the grid was extended upward on NER, POS and QA, because the chosen rate had sat at the top edge. The extension used the same seeds and the same dev-based selection (7 Oct). POS moved to 8e-5 (95.3 -> 95.5) and QA moved to 6e-5 (55.6 -> 56.0); NER was unchanged. The baselines kept their 3-value grids.
• Fine-tuning used bf16 autocast; ScandiBERT gave non-finite losses in bf16 and was fine-tuned in fp32 (flagged per run).

Full numbers: the accompanying paper (in preparation). The fine-tuning results above are the first round; the paper extends every learning-rate grid until the chosen rate is not at an edge, so its numbers differ slightly. Export parity check (ours vs transformers logits): see export_check.json.