ModernCamemBERT-bio-v2-large

ModernCamemBERT-bio-v2-large is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of ModernCamemBERT-bio-large with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 72.8 micro-F1, the best score we measured for coding and long-document classification at this size.

Authors

Rian Touchent and Eric de la Clergerie
Sorbonne Université, Inria Paris

Overview

ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Rather than plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The goal is to inject structured coding knowledge without adding any task-specific head. This model is our document specialist.

Architecture	ModernBERT
Parameters	350M
Context length	8,192 tokens
Language	French
Base model	almanach/ModernCamemBERT-bio-large

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "rntc/ModernCamemBERT-bio-v2-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = "Le patient présente une [MASK] aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))

Evaluation

Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.

Model	FrACCO-30	FrACCO-100	CANTEMIST	MORFITT	MedDialog-FR	Avg
ModernCamemBERT-bio-v2-large	82.0	67.8	74.6	74.3	65.2	72.8
ModernCamemBERT-bio-v2-base	78.0	63.7	70.9	73.5	63.7	70.0
ModernCamemBERT-bio-base	74.2	61.1	71.2	72.9	63.7	68.6
DoctoModernBERT	74.0	58.3	69.3	73.9	64.7	68.0
DrBERT	54.2	39.6	41.2	71.0	64.1	54.0
DoctoBERT	51.6	34.7	36.2	73.5	62.1	51.6
CamemBERT-bio	44.0	22.3	17.7	70.9	45.8	40.1

ModernCamemBERT-bio-v2-large has the best average and the best score on each coding task. It scores 4.2 points above ModernCamemBERT-bio-base, the strongest publicly available French biomedical encoder we evaluated.

Environmental impact

The OntoBook phase reported here ran for about 3.4 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.10 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.

License

MIT

Citation

@inproceedings{touchent:hal-05697506,
  TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
  AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
  URL = {https://hal.science/hal-05697506},
  BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
  ADDRESS = {Palma de Mallorca, Spain},
  YEAR = {2026},
  MONTH = May,
  PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
  HAL_ID = {hal-05697506},
  HAL_VERSION = {v1},
}

@misc{touchent2026causallanguagemodelingdetour,
  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
  author={Rian Touchent and Eric de la Clergerie},
  year={2026},
  eprint={2605.12438},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2605.12438},
}