dmbert-144m-mc4

A Dutch ModernBERT encoder pretrained from scratch with a masked-language-modelling objective on Dutch web text, using JAX/Flax on TPU. It has 22 layers, hidden size 512 and 143,645,056 (~144M) parameters, and uses a Dutch tokenizer with a 32k vocabulary.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="yhavinga/dmbert-144m-mc4")
# Write <mask> directly after the preceding word, without a space.
for r in fill("Amsterdam is de<mask> van Nederland.", top_k=5):
    print(r["token_str"], round(r["score"], 3))

This is a pretrained base model: fine-tune it for classification, NER, retrieval etc.

Model details

Architecture	ModernBERT (alternating local/global attention, RoPE, GeGLU)
Layers / hidden / heads / intermediate	22 / 512 / 8 / 3072
Parameters	143,645,056 (~144M). The Hub's parameter counter shows ~160M because the tied decoder weight is stored a second time in the file.
Attention	local window 128, global attention every 3rd layer
Max positions	8192
Tokenizer	yhavinga/dutch-llama-tokenizer (32,000 tokens + <mask>, padded to 32,128)

Training

• Data: yhavinga/mc4_nl_cleaned (config full)
• Steps: 1,350,000 (checkpoint of a run scheduled for 2,000,000 steps; the run crashed at step 1,350,040)
• Batch size: 8 per device, TPU v4-32
• Learning rate: 1.41e-4, 20,000 warmup steps, cosine decay to 1e-6
• Weight decay: 0.01
• Code: JAX/Flax, scripts/train_minimal.py in mbert-jax

Related models

Same architecture family, different data: dmbert-144m-mc4, dmbert-230m-mc4, dmbert-230m-culturax, dmbert-230m-fineweb2.

Acknowledgements

Cloud TPUs were provided by Google's TPU Research Cloud (TRC).

Citation

@misc{dmbert_144m_mc4,
  title={dmbert-144m-mc4: a Dutch ModernBERT},
  author={Yeb Havinga},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/yhavinga/dmbert-144m-mc4}
}