oliviercaron/camembertav2-adverse-event-fr

CamemBERTav2 — French Adverse-Event Mention Classifier

A binary classifier that detects whether a French text mentions an adverse event attributed to a vaccine or medicine. Fine-tuned from CamemBERTav2-base (DeBERTa-v2 architecture, Antoun et al. 2024) using a weak-supervision pipeline: LLM labels, targeted human audit, and self-training.

Designed as a reusable building block for French-language pharmacovigilance pipelines. The model, annotation guide, and evaluation protocol are released as commons for the clinical NLP community.

Intended uses

• Screening/triage of French health-related text for adverse-event mentions
• Pre-filter in NER pipelines (e.g., before entity extraction with GLiNER)
• Research on weak supervision, LLM-as-annotator, and low-resource health NLP

Not intended for: clinical decision-making, automated pharmacovigilance reporting, or use as a diagnostic tool. Predictions require human review.

Performance

Human gold standard (primary evaluation)

300 French tweets, double-coded by two independent annotators (95.0% agreement, Cohen's kappa = 0.900), adjudicated. Disjoint from all training data. Stratified by a pharmacovigilance lexicon: 176 lexicon matches, 124 non-matches. This set oversamples positives (151/149 split); corpus-wide metrics would differ.

Metric	Value
F1	0.954
Precision	0.942
Recall	0.967
ROC-AUC	0.970
Accuracy	95.3% (286/300)

Model-human agreement (94.0%–96.3%) bracketed human-human agreement (95.0%). This indicates comparable agreement, not superior performance.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "oliviercaron/camembertav2-adverse-event-fr"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

texts = [
    "Forte fièvre et douleurs musculaires 48h après ma 2e dose Pfizer",
    "Le vaccin est efficace à 95% contre les formes graves",
]

inputs = tokenizer(texts, truncation=True, padding=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
probs = torch.softmax(logits, dim=-1)[:, 1].numpy()

for text, p in zip(texts, probs):
    label = "adverse event" if p >= 0.5 else "no adverse event"
    print(f"[{p:.3f}] {label}: {text[:80]}")

Training details

Data

Source	Count	Positives	Notes
GPT-5.4 Nano labels	996	75	2 FP corrected after audit of 25 positives
Prior human NER annotations	145	125	Entity annotations converted to binary
Seed subtotal	1,141	200
Self-training positives (p >= 0.9)	2,176	2,176	Preliminary classifier on keyword-filtered corpus
Self-training hard negatives (p < 0.1)	2,768	0	Counterbalances keyword bias
Full pool	6,085	2,376
Held out for development	-229		20% of seeds
Training set	5,856	~2,335	~98% automatic labels

LLM annotation

Four LLMs annotated a stratified 1,000-tweet sample (400 random, 300 vaccine-enriched, 300 symptom-enriched) from 50.2 million French COVID-19 tweets. Three converged (positive rates 7.8–10.1%, pairwise kappa 0.694–0.749); Gemini 2.0 Flash was excluded (23.2% positive rate). GPT-5.4 Nano was retained after auditing 25 positive labels. Total annotation cost: ~US$0.35.

Model	Positives	Negatives	Retained
Gemini 2.5 Flash Lite	101	899	No
Gemini 2.0 Flash	232	766	Excluded
GPT-4.1 Nano	98	901	No
GPT-5.4 Nano	78	922	Yes

Hyperparameters (Optuna, 26 trials)

Parameter	Value
Learning rate	4.97e-5
Weight decay	0.0367
Warmup ratio	0.1934
Epochs	8
Batch size	16
Max length	256
Dropout	0.0527
Loss	Weighted cross-entropy (scale 0.3055)
Decision threshold	0.5 (fixed, not tuned on test)

Training: 21 minutes on one NVIDIA RTX 3070 (8 GB VRAM), FP16 mixed precision. Seed: 42.

Training code

The training/ folder contains the scripts used to build the dataset and train the model:

Script	Purpose
PIPELINE.md	Full pipeline documentation with ASCII diagrams
build_classifier_dataset.py	Assembles the training dataset from LLM batch results
preannotate_batch.py	Annotates 1,000 tweets via Gemini and GPT batch APIs
mine_corpus.py	Self-training: scores 5,000 candidates, retains high-confidence pseudo-labels
optuna_search.py	Hyperparameter search (26 Optuna trials)
train_production_model.py	Trains the final model with the best hyperparameters
honest_eval.py	Evaluates on the held-out test set (fixed threshold, bootstrap CI)

Note: Some scripts contain hardcoded paths that must be adapted to your environment. The corpus parquet file (50.2M tweets) is not distributed.

Limitations

1. Domain: trained on French COVID-19 tweets (2020–2021). Performance on other domains, time periods, or languages is untested.
2. Enriched evaluation: the gold set oversamples positives (~50/50); corpus-wide precision and recall would differ from the reported values. Only 1 of 124 out-of-lexicon tweets in the gold was positive.
3. Self-training circularity: 84% of training labels are pseudo-labels from the model's own preliminary version. Systematic blind spots may be amplified.
4. Not clinical text: trained and evaluated on public social-media text. Transfer to de-identified clinical notes remains untested.
5. Teacher audit scope: only 25 of ~78 GPT-5.4 Nano positive labels were audited; negative labels were not audited.

Ethical considerations

All tweets were public at collection time. Processing relies on GDPR Article 89 (research derogation). No individual is identified in the model output. The model detects mentions of adverse events as reported by social-media users; these are not confirmed medical events, diagnoses, or causal evidence.

Annotation guide

The frozen English annotation guide used for the gold standard is included in this repository as annotation_guide.md. It defines the task (ADR/AEFI detection), provides 11 decision rules, 14 reference examples, and describes the double-coding procedure.

Citation

If you use this model, please cite:

@inproceedings{caron2026weak,
  title     = {From {LLM} Weak Supervision to Human Gold:
               A Compact French Adverse-Event Mention Classifier},
  author    = {Caron, Olivier and Gimenez, Lucile and Benavent, Christophe},
  booktitle = {Building Commons for Clinical LLMs},
  year      = {2026},
  address   = {Paris, France}
}

Acknowledgments

The base model is CamemBERTav2 by Antoun et al. (2024).