ModernCamemBERT-bio-v2-base

ModernCamemBERT-bio-v2-base is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of ModernCamemBERT-bio-base with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 70.0 micro-F1 at 150M parameters. A larger version is available as ModernCamemBERT-bio-v2-large.

Authors

Rian Touchent and Eric de la Clergerie
Sorbonne Université, Inria Paris

Overview

ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Instead of plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The aim is to inject structured coding knowledge without adding a task-specific head. This is the base-size document specialist.

Architecture	ModernBERT
Parameters	150M
Context length	8,192 tokens
Language	French
Base model	almanach/ModernCamemBERT-bio-base

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "rntc/ModernCamemBERT-bio-v2-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = "Le patient présente une [MASK] aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))

Evaluation

Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.

Model	FrACCO-30	FrACCO-100	CANTEMIST	MORFITT	MedDialog-FR	Avg
ModernCamemBERT-bio-v2-large	82.0	67.8	74.6	74.3	65.2	72.8
ModernCamemBERT-bio-v2-base	78.0	63.7	70.9	73.5	63.7	70.0
ModernCamemBERT-bio-base	74.2	61.1	71.2	72.9	63.7	68.6
DoctoModernBERT	74.0	58.3	69.3	73.9	64.7	68.0
DrBERT	54.2	39.6	41.2	71.0	64.1	54.0
DoctoBERT	51.6	34.7	36.2	73.5	62.1	51.6
CamemBERT-bio	44.0	22.3	17.7	70.9	45.8	40.1

At base size, ModernCamemBERT-bio-v2-base leads the average and scores 1.4 points above its ModernCamemBERT-bio-base starting point. ModernCamemBERT-bio-v2-large scores 2.8 points higher at 350M parameters.

Environmental impact

The OntoBook phase reported here ran for about 1.3 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.04 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.

License

MIT

Citation

@inproceedings{touchent:hal-05697506,
  TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
  AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
  URL = {https://hal.science/hal-05697506},
  BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
  ADDRESS = {Palma de Mallorca, Spain},
  YEAR = {2026},
  MONTH = May,
  PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
  HAL_ID = {hal-05697506},
  HAL_VERSION = {v1},
}

@misc{touchent2026causallanguagemodelingdetour,
  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
  author={Rian Touchent and Eric de la Clergerie},
  year={2026},
  eprint={2605.12438},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2605.12438},
}