RobBERT-2026: A Fully EU AI Act-Compliant Dutch Language Model

RobBERT-2026 is the 2026 release of the Dutch RobBERT model, and the first member of the family trained entirely on provenance-clean, permissibly-licensed and synthetic data, with no undifferentiated web crawl in the trainng set. Where earlier RobBERT models (and most Dutch encoders) are pretrained on OSCAR/CommonCrawl, RobBERT-2026 is built to be compliant by construction with the data-provenance and transparency expectations of the EU AI Act, while staying competitive as a general-purpose Dutch encoder.

It is also the first RobBERT built on the modern NeoBERT encoder architecture rather than RoBERTa: pretrained from scratch with a masked-language-modeling objective on ~27B tokens of compliant Dutch text. The pretraining corpus combines Common Corpus NL (63%), the GPT-NL public corpus (25%), synthetic Dutch data (8%; seeded by Belgian staatsblad and wikipedia) and case law from The Netherlands (4%). Every source permissibly licensed or synthetic, with no web-crawl lineage of any kind: not even provenance-defensible derivatives such as a CC-licensed CommonCrawl subset or machine-translated FineWeb-Edu.

More in-depth information about the RobBERT family can be found on the RobBERT website, in the original RobBERT paper, and in the RobBERT GitHub repository. The RobBERT-2026 paper is forthcoming.

Model details


Architecture	NeoBERT-base (28 layers, hidden 768, 12 heads, SwiGLU, RMSNorm, RoPE)
Parameters	275M
Sequence length	512 (pretrained) — the RoPE table ships for 8192, see note below
Objective	Masked language modeling (20% masking)
Tokenizer	Byte-level BPE, 50k vocab
Pretraining data	~27B tokens (13,000 steps at ~2.1M tokens/step), no web-crawl lineage
License of data	Per-document permissive / synthetic (EU AI Act–compliant)

Results

Evaluated on DUMB (test split; fine-tuned for 2 epochs at lr 5e-5, bf16, identical recipe for both models):

Task	Metric	RobBERT-2023-base	RobBERT-2026
QA (SQuAD-NL)	F1	69.3	69.6
QA (SQuAD-NL)	EM	46.7	65.5
Sentiment (DBRD)	acc	93.5	93.3
NLI (SICK-NL)	acc	85.9	82.4

RobBERT-2026 matches RobBERT-2023 on question-answering F1 and is far ahead on exact match.

We evaluate on DUMB rather than EuroEval because EuroEval's squad-nl pipeline has an offset bug (GH issue)

How to use

RobBERT-2026 uses the NeoBERT architecture and ships its modeling code with the repository, so load it with trust_remote_code=True:

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("pdelobelle/robbert-2026")
model = AutoModelForMaskedLM.from_pretrained("pdelobelle/robbert-2026", trust_remote_code=True)

For a downstream task, swap in AutoModelForSequenceClassification (a NeoBERTForSequenceClassification head is included) and fine-tune as you would any HuggingFace encoder.

No xformers required. The modeling code uses xformers.ops.SwiGLU when xformers is installed and importable, and otherwise falls back to a pure-torch SwiGLU with the identical packed weight layout — verified to produce bit-identical outputs. Attention already uses PyTorch's scaled_dot_product_attention. Installing xformers is optional and only affects speed.

Context length. The model was pretrained at sequence length 512. The released RoPE table is precomputed to 8192 positions so longer sequences run without modification, but positions beyond 512 are untrained extrapolation and quality there is not validated. Note also that the attention mask is materialised as a dense [batch, heads, L, L] tensor, so memory grows quadratically with length.

Why compliant data?

The EU AI Act raises the bar on data governance and transparency for models placed on the EU market. Most Dutch language models, including previous RobBERT releases, are pretrained on large web crawls (OSCAR, CommonCrawl) whose per-document licensing and provenance are unclear. RobBERT-2026 instead restricts pretraining to sources that are individually permissibly licensed or synthetically generated, so the training corpus is auditable and free of undifferentiated web crawl. We view the (modest) quality cost of this restriction as the price of provenance, and release the model, tokenizer and data recipe so others can build on a clean-provenance base.

Comparison of available Dutch BERT models

(this model) pdelobelle/robbert-2026: the first fully EU AI Act–compliant Dutch encoder, NeoBERT architecture, trained from scratch on provenance-clean data. Use it when data provenance/licensing matters.
DTAI-KULeuven/robbert-2023-dutch-large / -base: RobBERT-2023 on OSCAR-2023 with a new tokenizer (Tik-to-Tok); strong general-purpose Dutch models.
DTAI-KULeuven/robbert-2022-dutch-base: RobBERT further pre-trained on OSCAR-2022.
pdelobelle/robbert-v2-dutch-base: the long-standing RoBERTa-based RobBERT, trained on OSCAR.
DTAI-KULeuven/robbertje-1-gb-merged: a distilled, smaller/faster RobBERT.

There is also GroNLP/bert-base-dutch-cased ("BERTje"), an older BERT-based Dutch model.

Name origin of RobBERT

Like most BERT-like models, RobBERT has BERT in its name. When we queried the original RobBERT model to name itself, it consistently called itself RobBERT — fitting, since it is a very Dutch name and echoes its root architecture. As "rob" is the Dutch word for a seal, the RobBERT logo is a seal dressed up like Bert from Sesame Street.

Credits and citation

The RobBERT models are created by Pieter Delobelle, Thomas Winters, Bettina Berendt and François Remy. A paper describing RobBERT-2026 is forthcoming; in the meantime please cite this release and the earlier models:

@misc{delobelle2026robbert2026,
    title        = {RobBERT-2026},
    author       = {Delobelle, Pieter},
    year         = {2026},
    howpublished = {\url{https://huggingface.co/pdelobelle/robbert-2026}}
}

@inproceedings{delobelle2020robbert,
    title = "{R}ob{BERT}: a {D}utch {R}o{BERT}a-based {L}anguage {M}odel",
    author = "Delobelle, Pieter and Winters, Thomas and Berendt, Bettina",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov, year = "2020",
    url = "https://arxiv.org/abs/2001.06286"
}

@inproceedings{delobelle2022robbert2022,
    title = {RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use},
    author = {Delobelle, Pieter and Winters, Thomas and Berendt, Bettina},
    year = {2022}, url = {https://arxiv.org/abs/2211.08192}
}