ptanwar/pubmedbert-chia-ner

PubMedBERT/BiomedBERT fine-tuned for CHIA clinical trial eligibility NER

Fine-tuned microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext for named entity recognition on clinical trial eligibility criteria (CHIA corpus), as part of a course NLP project comparing fine-tuned biomedical transformers vs. GPT-4 prompting.

• Code / full writeup: https://github.com/jatinpsingh/NER_Clinical_Trial_Eligibility/tree/main/pmb
• Base model: PubMedBERT-base (renamed BiomedBERT), Gu et al. 2021
• Task: token classification, 31 labels (O + 15 entity types x B/I)
• Data: CHIA eligibility criteria, team's shared fixed split (10,006 train / 1,240 val / 1,163 test sentences -- fold 0 of a 10-fold CV scheme, see below)

Hyperparameters (matching Li et al. 2022, Table 3)

Learning rate	5e-5
Batch size	8
Epochs	10
Max sequence length	256
Adam epsilon	1e-8

Results

This checkpoint's own fold (entity-level, test set):

	Precision	Recall	F1
Strict (exact span match)	0.639	0.674	0.656
Relaxed (type + overlap match)	0.750	0.791	0.770

Full 10-fold cross-validation (mean +/- std across all 10 folds -- the number directly comparable to Li et al. 2022's own 10-fold-averaged reporting; the weights hosted in this repo are from one of these 10 folds, not a checkpoint averaged across them):

	Precision	Recall	F1
Strict	0.657 +/- 0.013	0.682 +/- 0.021	0.669 +/- 0.013
Relaxed	0.758 +/- 0.014	0.787 +/- 0.024	0.772 +/- 0.015

Both this checkpoint's own score and the 10-fold mean exceed Li et al. 2022's published PubMedBERT numbers on Chia (0.622 strict / 0.744 relaxed).

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ptanwar/pubmedbert-chia-ner")
model = AutoModelForTokenClassification.from_pretrained("ptanwar/pubmedbert-chia-ner")

Known limitations

• The hosted weights are one fold's model, not a checkpoint averaged/ensembled across the 10-fold CV run -- treat the single-fold numbers above as this specific checkpoint's performance, and the 10-fold mean +/- std as the more rigorous estimate of the approach's true performance.
• Weakest on rare/ambiguous types (Mood, Observation, Reference_point) -- see the full writeup for error analysis.