pdelobelle/robbert-2026-retrieval

RobBERT-2026 Retrieval

A Dutch dense retriever fine-tuned from RobBERT-2026, the first EU AI Act–compliant Dutch encoder (NeoBERT architecture, pretrained from scratch on provenance-clean, permissibly-licensed and synthetic data, meaning no web-crawl sourced data at all). It is contrastively fine-tuned on pdelobelle/cc-nl-retrieval (synthetic queries over Common Corpus NL) plus the CC-BY-SA NFI symmetric-similarity sets.

Usage

Trained with query: / document: prefixes; these are stored as the model's prompts, so use prompt_name and they are applied automatically:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("pdelobelle/robbert-2026-retrieval", trust_remote_code=True)

docs = [
    "Het Atomium is een bouwwerk in Brussel, opgericht voor de Wereldtentoonstelling van 1958.",
    "Fonske, voluit Fons Sapientiae, is een beeld in Leuven van een student die water uit een boek over zijn hoofd giet.",
    "Manneken Pis is een klein bronzen standbeeldje in het centrum van Brussel.",
    "De Grote Markt van Antwerpen is een historisch plein met het stadhuis.",
]
d = model.encode(docs, prompt_name="document")

for query in ["Waar ligt het Atomium?", "Wat stelt het beeld Fonske in Leuven voor?"]:
    q = model.encode([query], prompt_name="query")
    sims = model.similarity(q, d)[0]
    best = int(sims.argmax())
    print(f"{query} -> {docs[best]} ({sims[best]:.2f})")

# Waar ligt het Atomium?                        -> Het Atomium ... (0.56)
# Wat stelt het beeld Fonske in Leuven voor?    -> Fonske ...      (0.65)

Prefixes are required. If you encode without them (or swap them), retrieval quality drops sharply.

Training

• Base: pdelobelle/robbert-2026 (NeoBERT-base, 275M, trust_remote_code).
• Stage-2 recipe: CachedMultipleNegativesRankingLoss on asymmetric (query, positive, 4×hard-negative) triples from cc-nl-retrieval, mixed with three NFI symmetric-similarity sets (CC-BY-SA) for symmetric coverage: simplewiki-translated-nl, stackexchange-duplicate-questions-translated-nl, wiki-atomic-edits-translated-nl.
• Config: CLS pooling, query: /document: prefixes, max sequence length 512, cosine similarity, 1 epoch.

Evaluation

Zero-shot on BEIR-NL (Banar et al., 2025; machine-translated Dutch BEIR), nDCG@10.

Compared against similarly-sized multilingual dense retrievers (numbers from Banar et al., 2025):

	nfcorpus	scifact	arguana	scidocs	NQ	FEVER
domain, for robbert-2026-retrieval	out	out	out	out	in	in
multilingual-e5-base (278M)	24.17	67.23	47.06	10.53	36.06†	58.52†
robbert-2026-retrieval (275M)	17.83	40.74	33.72	9.01	20.26	54.56
LaBSE (470M)	13.54	39.07	39.15	6.32	11.24	12.51
mContriever	13.36	37.89	39.60	4.93	10.50	21.51

The domain row is relative to robbert-2026-retrieval, which is trained on Wikipedia + Common Corpus NL: the biomedical/scientific/argument sets are out-of-domain (its hardest footing), NQ and FEVER are in-domain (Wikipedia). The other models have their own, broader training — the † marks cells that Banar et al. (2025) flag as likely contaminated (the model was probably trained on that dataset). On the in-domain sets it clearly beats LaBSE and mContriever and is within range of multilingual-e5-base (whose NQ/FEVER are flagged as contaminated).

This is the tradeoff for clean provenance: it trails the newest multilingual models like mE5, trained on far more, and often web/in-domain data. But is competitive with older multilingual embedders such as LaBSE and mContriever, beating both on nfcorpus and scidocs.

Compliance

Both the base model and this retriever avoid undifferentiated web crawl end to end: RobBERT-2026 is pretrained only on permissibly-licensed / synthetic Dutch text, and this model is fine-tuned only on synthetic queries over Common Corpus NL plus the CC-BY-SA NFI sets. See the base model card for the full data recipe and the EU AI Act rationale.

Licensing. Because the NFI fine-tuning sets are CC-BY-SA-4.0, this retriever is released under CC-BY-SA-4.0 (ShareAlike) and attributes them (see below). The base RobBERT-2026 is unaffected and releaed under MIT license.

Citation

To cite this release:

@misc{delobelle2026robbert2026retrieval,
    title        = {RobBERT-2026 Retrieval},
    author       = {Delobelle, Pieter},
    year         = {2026},
    howpublished = {\url{https://huggingface.co/pdelobelle/robbert-2026-retrieval}}
}

RobBERT lineage:

@inproceedings{delobelle2020robbert,
    title = "{R}ob{BERT}: a {D}utch {R}o{BERT}a-based {L}anguage {M}odel",
    author = "Delobelle, Pieter and Winters, Thomas and Berendt, Bettina",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    year = "2020",
    url = "https://arxiv.org/abs/2001.06286"
}

@article{remy2023tiktotok,
    title = {Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation},
    author = {Remy, Fran{\c{c}}ois and Delobelle, Pieter and Berendt, Bettina and Demuynck, Kris and Demeester, Thomas},
    journal = {arXiv preprint arXiv:2310.03477},
    year = {2023}
}