rucolaes/nih-adrd-bioclinical-modernbert

NIH AD/ADRD Classifier (BioClinical-ModernBERT)

A binary text classifier that predicts whether a research project's title + abstract belongs to NIH's Alzheimer's Disease / Alzheimer's Disease-Related Dementias (AD/ADRD) research portfolio. Fine-tuned from BioClinical-ModernBERT on NIH's own portfolio labels — not a general "neurodegenerative disease" classifier.

A PubMedBERT-based sibling model was trained on the identical dataset for comparison; see the Evaluation section below for how they compare head to head.

Model description

• Base model: thomas-sounack/BioClinical-ModernBERT-base — ModernBERT architecture, pretrained on 50.7B tokens of PubMed/PMC text plus 2.8B tokens of clinical notes
• Task: binary sequence classification, input = (title, abstract) as a tokenizer pair (proper segment encoding, not a manually concatenated string)
• Max sequence length: 1024 tokens (chosen to cover roughly the 99th percentile of title+abstract length — median NIH title+abstract is ~650-700 tokens, already past BERT-family models' 512-token limit, which was the original motivation for testing a longer-context architecture at all)
• Output: softmax probability of the positive (AD/ADRD) class

Training data

Identical dataset and label definition as the PubMedBERT sibling model - see its model card for the full detail on label construction, deduplication, grouped splitting, and hard-negative cohorts. Source: NIH ExPORTER, fiscal years 2018-2024.

Split	Rows	Positive	Rate
train	125,961	11,451	9.1%
val	42,431	2,411	5.7%
test	42,034	2,513	6.0%

Training procedure

• Loss: class-weighted cross-entropy (weight = train-set negative:positive ratio)
• Optimizer: AdamW, lr 2e-5, weight decay 0.01, warmup ratio 0.1
• 3 epochs, batch size 16 (train) / 32 (eval), fp16, attn_implementation="sdpa"
• Model selection: best checkpoint by validation PR-AUC
• Hardware: single A10G GPU (~4.5 hours - notably slower than the PubMedBERT sibling, since sequences here are up to 2x longer)

Evaluation

Held-out test set (natural class distribution, never used for training or model selection):

Model	PR-AUC	F1 (threshold 0.5)
PubMedBERT sibling	0.9596	0.929
This model	0.9610	0.932

Trained and evaluated on the identical dataset, so this is a clean head-to-head: this model edges out the PubMedBERT sibling on the NIH metric that's the actual training objective - the first point in this project's development where the longer context showed a measurable benefit.

Cross-domain check (informal): applied to EU/Horizon (CORDIS) project text and compared against an independent LLM-based classification (Claude + GPT, matching disease scope). Agreement was "almost perfect" by the standard interpretation scale (Cohen's κ ≈ 0.82-0.85 depending on which LLM), edging out the PubMedBERT sibling on precision (fewest false positives of any model version tested in this project) at the cost of slightly more misses. Not a validated cross-domain benchmark - treat out-of-domain predictions as a signal to review, not a final decision.

On threshold choice: as with the PubMedBERT sibling, the raw softmax output is not a calibrated posterior probability (negative downsampling + inverse-frequency class weighting shift the effective training prevalence well above NIH's true ~6% base rate). Use PR-AUC/ROC-AUC for ranking; pick an operating threshold from a validation set matching your deployment distribution rather than assuming 0.5.

Intended use and limitations

• Trained and evaluated on NIH-style grant abstracts. See the cross-domain note above.
• Known residual failure mode: like the PubMedBERT sibling, occasional false positives on text with surface-level token overlap with AD-associated vocabulary but no real disease content. The specific cases each model gets wrong only partially overlap - this model fixed some cases (e.g. a generic "AI/Data/Robotics" infrastructure project) that the PubMedBERT sibling did not, and vice versa, suggesting an ensemble of the two catches more than either alone.
• Designed as a high-recall prefilter / triage signal, not a sole final-decision system - best used alongside expert or LLM review of flagged candidates, particularly for anything outside the strict NIH AD/ADRD definition it was trained on.

How to use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO_ID = "rucolaes/nih-adrd-bioclinical-modernbert"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForSequenceClassification.from_pretrained(REPO_ID, attn_implementation="sdpa").eval()

title = "..."
abstract = "..."
inputs = tokenizer(title, abstract, truncation=True, max_length=1024, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
prob_adrd = torch.softmax(logits, dim=1)[0, 1].item()