omareng/TrialJev-FineTunedLLM

TrialJev comparison: LLM fine-tuned to write the verdict

Research preview. Not for clinical decisions, trial enrollment or patient care. TrialJev is independent research that adopts the design of the open-weight MedJev and kev releases (Apache-2.0). It is not affiliated with or endorsed by TypeSafe AI, the developer of Jev.

This repository holds a LoRA adapter trained the conventional generative way (full-vocabulary cross-entropy on the answer tokens) to write "Eligible" or "Excluded" and stop; this is a comparison model, not the proposed method. Accuracy below is the written verdict. Each sub-folder is one selected checkpoint (lowest development loss), trained on TREC Clinical Trials 2021 and evaluated on 6,975 eligible-or-excluded pairs from 50 unseen TREC 2022 patients (always predicting "eligible" scores 56.47%).

Checkpoints

Sub-folder	Configuration	Backbone (revision)	Step	Accuracy	Macro-F1	AUROC	Brier	ECE
qwen3.5-0.8b-fresh-seed0	0.8B, fresh, seed 0	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	800	74.05%	73.0%	0.802	n/a	n/a
qwen3.5-9b-seed0	9B, seed 0	Qwen/Qwen3.5-9B-Base (revision 68c46c4b3498877f3ef123c856ecfde50c39f404)	400	79.50%	78.5%	0.864	n/a	n/a
qwen3.5-9b-seed1	9B, seed 1	Qwen/Qwen3.5-9B-Base (revision 68c46c4b3498877f3ef123c856ecfde50c39f404)	400	79.84%	79.0%	0.862	n/a	n/a
qwen3.5-9b-seed2	9B, seed 2	Qwen/Qwen3.5-9B-Base (revision 68c46c4b3498877f3ef123c856ecfde50c39f404)	200	75.68%	75.7%	0.860	n/a	n/a
qwen3.5-27b	Qwen3.5-27B, seed 0	Qwen/Qwen3.5-27B (revision fc05daec18b0a78c049392ed2e771dde82bdf654)	200	79.87%	79.8%	0.879	n/a	n/a
qwen3.8-27b	Qwen3.8-27B, seed 0	Qwen/Qwen3.8-27B (revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0)	200	79.93%	79.9%	0.888	n/a	n/a

All metrics are on the 6,975 eligible-or-excluded TREC 2022 test pairs and are the numbers reported in the manuscript for that exact checkpoint: accuracy and macro-F1 of the model's decision, AUROC from P(eligible), the binary Brier score and a 10-bin expected calibration error (lower is better for both). For the fine-tuned LLM the accuracy and macro-F1 are those of the written verdict (an unparsable answer counts as wrong) and the AUROC comes from the first-token two-way score; Brier and ECE are not reported because a written verdict carries no probability. (The TrialJev numbers use the canonical two-pass evaluator; 27B and Qwen3.8-27B are single seeds, and 95% intervals are about plus or minus 3 points because the 50 test patients are the independent units). The backbone weights are not included: the adapter loads on the upstream model at the listed revision. Seed-0 checkpoints of the 9B and 27B models did not record their backbone revision; the revision shown is the only snapshot in the training cache. Every sub-folder also holds training_config.json, training_metrics.json and the training log, and SHA256SUMS.txt lists the checksum of every file.

How it works

The same backbone, LoRA settings, data, optimizer schedule and development-loss checkpoint selection as TrialJev, but the training target is the answer word followed by the end-of-sequence token and the loss covers the whole vocabulary on the answer tokens. At inference the model decodes greedily until the end token.

Usage

from huggingface_hub import snapshot_download
local = snapshot_download("omareng/TrialJev-FineTunedLLM", allow_patterns=["qwen3.8-27b/*"])      # one sub-folder, not the whole repository
path = f"{local}/qwen3.8-27b"

Evaluate with the code repository (https://github.com/oibrahi1/TrialJev): python scripts/generate_baselines.py run --condition sft-generate --base <backbone> --base_revision <revision> --adapter <path> --data <dir>/test.jsonl --per_patient 0. The prompt is the zero-shot word prompt of the manuscript (Appendix D); the model writes " Eligible" or " Excluded" and stops.

Training

Setting	Value
Training data	TREC Clinical Trials 2021: 60 training patients, 9,302 eligible-or-excluded pairs (development: 15 patients, 2,233 pairs)
Adapter	LoRA rank 16, alpha 32, dropout 0.05 on the attention, MLP and Gated DeltaNet projections
Optimization	AdamW, learning rate 1e-4 (one-cycle, 5% warm-up), weight decay 0.01, gradient clipping 1.0, 3 epochs (1,746 steps), effective batch 16
Inputs	patient note capped at 512 tokens, state plus trial text capped at 4,096 tokens
Selection	lowest development loss (checked every 100 steps at 0.8B, every 200 at 9B and above); the test set was scored once
Precision	fp32 backbone at 0.8B, bf16 at 9B and above

Training data

TREC Clinical Trials 2021 patient cases and ClinicalTrials.gov records. The processed data are not redistributed because they derive from TREC topics and registry text; the code repository explains how to rebuild them and lists SHA-256 hashes. No institutional patient records were used.

Limitations and intended use

Research use only. One benchmark of 50 synthetic patients; pairs that assessors had already judged relevant (not retrieval); synthetic, short patient notes; no explanation or rationale is produced; the scores are not calibrated probabilities and a cutoff should be chosen on development data; one seed at 27B. head.pt is a PyTorch pickle holding the pointer-head tensors and small metadata; load it only from this repository. See the manuscript and docs/RESULTS.md in the code repository.

Citation

@misc{ibrahim2026trialjev,
  title  = {TrialJev: Open-Weight Single-Pass Eligibility Scoring for Patient--Trial Screening},
  author = {Ibrahim, Omar and Hamblin, Grant and Haibe-Kains, Benjamin},
  year   = {2026},
  url    = {https://github.com/oibrahi1/TrialJev},
  note   = {Preprint; arXiv identifier to be added at posting}
}

Please also cite the work this builds on: MedJev (https://github.com/JunMa11/MedJev) and kev (https://github.com/jaredpalmer/kev).

License

Apache-2.0 for the adapters and heads. The backbones are released by Alibaba Cloud under their own licenses; the TREC data belong to NIST/TREC and are not included.

Framework versions

Large models: torch 2.14.0+cu130, transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, flash-linear-attention 0.5.2 (H100 NVL, bf16 backbone). 0.8B models: torch 2.14.0, transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, flash-linear-attention 0.5.2 (RTX 6000 Ada, fp32 backbone)