PrecepTron-32B-DiagnosticReasoning

PrecepTron-32B-DiagnosticReasoning is a fine-tuned version of Qwen/Qwen3-32B that acts as an automated rubric-based grader ("LLM-as-a-judge") for clinical reasoning. It is fine-tuned via LoRA and distributed here as a PEFT adapter that loads on top of the Qwen/Qwen3-32B base weights. It is one of four task-specific PrecepTron-32B judges released alongside the paper Scaling Clinical Judgment to Evaluate Medical AI.

Given a clinical case and a response to evaluate, the model outputs a numeric score on the task's rubric together with a short justification, as a JSON object.

• Task: Landmark diagnostic-reasoning cases — structured reasoning rubric scored out of 19 points (differential, supporting and opposing findings, final diagnosis, next steps).
• Base model: Qwen/Qwen3-32B
• Method: LoRA supervised fine-tuning (PEFT) on physician rubric scores
• LoRA config: r=16, α=32, dropout=0.05, 3 epochs, bin-stratified ("balanced") oversampling
• Output format: {"score": <number>, "justification": "<text>"}
• Website: https://preceptron.net · Code: https://github.com/2v/PrecepTron · Benchmark: https://huggingface.co/datasets/tbuckley/GRAND-ROUNDS

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-32B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "tbuckley/PrecepTron-32B-DiagnosticReasoning")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-32B")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},   # see "Prompt" below
    {"role": "user", "content": USER_PROMPT},        # case + response to score
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Prompt

The model was trained with the exact system and user prompts below — the same prompts it expects at inference time. The scoring rubric is fixed and baked into the system prompt below.

System prompt

You are an expert clinical evaluator. You will be given a clinical response and a scoring rubric.
Score the response according to the rubric. You must output ONLY valid JSON with your scoring result.

Your output must be a JSON object with these fields:
- "score": the numeric score you assign
- "justification": a brief explanation of your scoring decision

### Part 1 – Structured Reasoning

#### Question 1: Diagnosis — List 3 Possible Diagnoses (0–3 points)
The response should list three possible diagnoses.
- 1 point per diagnosis if it is plausible and appropriate for the case
- 0 points if the diagnosis is implausible or incorrect
Maximum: 3 points

#### Question 2: Support Diagnosis — For each possible diagnosis listed, provide findings/risk factors supporting this hypothesis (0–6 points)
For each diagnosis:
- 2 points: Correct and specific findings or risk factors that support the diagnosis and are grounded in the case
- 1 point: Partially correct or incomplete supporting information
- 0 points: Incorrect, irrelevant, or missing
Maximum: 6 points (2 per diagnosis)

#### Question 3: Opposing Diagnosis — For each possible diagnosis listed, provide findings opposing this hypothesis, or findings that were expected but not present (0–6 points)
For each diagnosis:
- 2 points: Correct identification of findings that contradict the diagnosis or expected findings that are absent
- 1 point: Partially correct or incomplete opposing reasoning
- 0 points: Incorrect, irrelevant, or missing
Maximum: 6 points (2 per diagnosis)

### Part 2 – Final Diagnostic Decision (0–2 points)
- 2 points: Correct diagnosis
- 0 points: Incorrect diagnosis

### Part 3 – Additional Steps (0–2 points)
- 2 points: Appropriate, specific, and clinically useful next steps
- 1 point: Partially appropriate or incomplete
- 0 points: Incorrect or not useful

Total maximum: 19 points

User prompt template (case-specific fields filled in at inference)

## Final Diagnosis
{final_diagnosis}

## Case Vignette
{case_vignette}

## Question
{question_text}

## Response to Score
{response}

Training data

Supervised fine-tuning on physician rubric scores from the study dataset. The assistant target is the reconciled physician score (or, when no reconciliation exists, a single physician's score selected deterministically), validated against the task's allowed score scale. Train/validation case IDs were held out of all reported evaluations.

Limitations

This is a research artifact for the automated evaluation of medical AI. It is not a diagnostic or clinical decision-support tool and must not be used for patient care. Scores reflect agreement with the specific rubrics and physician annotations in this study and may not transfer to other rubrics, populations, or response formats.

Citation

@article{buckley2026preceptron,
  title   = {Scaling Clinical Judgment to Evaluate Medical AI},
  author  = {Buckley, Thomas A. and Kanjee, Zahir and Brodeur, Peter G. and
             Crowe, Byron and Pettinato, Anthony M. and Shah, Aashna P. and
             Haimovich, Adrian D. and McCoy, Liam G. and Restrepo, Daniel and
             Goh, Ethan and Chen, Jonathan H. and Zwaan, Laura and
             Goodman, Katherine E. and Morgan, Daniel J. and
             Abdulnour, Raja-Elie E. and Rodman, Adam and Manrai, Arjun K.},
  year    = {2026},
  note    = {Preprint, forthcoming}
}