dmbert-230m-mc4

A Dutch ModernBERT encoder pretrained from scratch with a masked-language-modelling objective on Dutch web text, using JAX/Flax on TPU. It has 22 layers, hidden size 1024 and ~230M parameters, and uses a Dutch tokenizer with a 32k vocabulary.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="yhavinga/dmbert-230m-mc4")
# Write <mask> directly after the preceding word, without a space.
for r in fill("Amsterdam is de<mask> van Nederland.", top_k=5):
    print(r["token_str"], round(r["score"], 3))

This is a pretrained base model: fine-tune it for classification, NER, retrieval etc.

Model details

Architecture	ModernBERT (alternating local/global attention, RoPE, GeGLU)
Layers / hidden / heads / intermediate	22 / 1024 / 16 / 1536
Parameters	230,110,592 (~230M). The Hub's parameter counter shows ~263M because the tied decoder weight is stored a second time in the file.
Attention	local window 128, global attention every 3rd layer
Max positions	8192
Tokenizer	yhavinga/dutch-llama-tokenizer (32,000 tokens + <mask>, padded to 32,128)

Training

• Data: yhavinga/mc4_nl_cleaned (config full)
• Steps: 2,000,000 (full schedule)
• Batch size: 8 per device, TPU v4-32
• Max sequence length: 1024
• Learning rate: 3e-5, 20,000 warmup steps, cosine decay to 1e-6
• Weight decay: 0.01
• Code: JAX/Flax, scripts/train_minimal.py in mbert-jax

Evaluation

Fine-tuned for 3 epochs on two tasks of the DUMB benchmark (test accuracy):

DBRD (sentiment)	SICK-NL (NLI)
0.9460	0.7981

Single run per model; differences of a few tenths of a point are within noise.

Related models

Same architecture family, different data: dmbert-144m-mc4, dmbert-230m-mc4, dmbert-230m-culturax, dmbert-230m-fineweb2.

Acknowledgements

Cloud TPUs were provided by Google's TPU Research Cloud (TRC).

Citation

@misc{dmbert_230m_mc4,
  title={dmbert-230m-mc4: a Dutch ModernBERT},
  author={Yeb Havinga},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/yhavinga/dmbert-230m-mc4}
}