PrecepTron-32B-Management

PrecepTron-32B-Management is a fine-tuned version of Qwen/Qwen3-32B that acts as an automated rubric-based grader ("LLM-as-a-judge") for clinical reasoning. It is fine-tuned via LoRA and distributed here as a PEFT adapter that loads on top of the Qwen/Qwen3-32B base weights. It is one of four task-specific PrecepTron-32B judges released alongside the paper Scaling Clinical Judgment to Evaluate Medical AI.

Given a clinical case and a response to evaluate, the model outputs a numeric score on the task's rubric together with a short justification, as a JSON object.

• Task: Grey Matters management-reasoning cases. The scoring rubric is question-specific and is supplied to the judge at inference time (see the user prompt below).
• Base model: Qwen/Qwen3-32B
• Method: LoRA supervised fine-tuning (PEFT) on physician rubric scores
• LoRA config: r=16, α=32, dropout=0.05, 3 epochs, bin-stratified ("balanced") oversampling
• Output format: {"score": <number>, "justification": "<text>"}
• Website: https://preceptron.net · Code: https://github.com/2v/PrecepTron · Benchmark: https://huggingface.co/datasets/tbuckley/GRAND-ROUNDS

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-32B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "tbuckley/PrecepTron-32B-Management")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-32B")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},   # see "Prompt" below
    {"role": "user", "content": USER_PROMPT},        # case + response to score
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Prompt

The model was trained with the exact system and user prompts below — the same prompts it expects at inference time. The scoring rubric is not fixed — it varies per question and is passed to the model at inference time via the {rubric} field of the user prompt. The system prompt below is the generic evaluator instruction.

System prompt

You are an expert clinical evaluator. You will be given a clinical response and a scoring rubric.
Score the response according to the rubric. You must output ONLY valid JSON with your scoring result.

Your output must be a JSON object with these fields:
- "score": the numeric score you assign
- "justification": a brief explanation of your scoring decision

Example output:
{
  "score": 3,
  "justification": "The response demonstrates..."
}

User prompt template (case-specific fields filled in at inference)

## Case Vignette
{case_vignette}

## Question
{question_text}

## Response to Score
{response}

## Scoring Rubric
{rubric}

Training data

Supervised fine-tuning on physician rubric scores from the study dataset. The assistant target is the reconciled physician score (or, when no reconciliation exists, a single physician's score selected deterministically), validated against the task's allowed score scale. Train/validation case IDs were held out of all reported evaluations.

Limitations

This is a research artifact for the automated evaluation of medical AI. It is not a diagnostic or clinical decision-support tool and must not be used for patient care. Scores reflect agreement with the specific rubrics and physician annotations in this study and may not transfer to other rubrics, populations, or response formats.

Citation

@article{buckley2026preceptron,
  title   = {Scaling Clinical Judgment to Evaluate Medical AI},
  author  = {Buckley, Thomas A. and Kanjee, Zahir and Brodeur, Peter G. and
             Crowe, Byron and Pettinato, Anthony M. and Shah, Aashna P. and
             Haimovich, Adrian D. and McCoy, Liam G. and Restrepo, Daniel and
             Goh, Ethan and Chen, Jonathan H. and Zwaan, Laura and
             Goodman, Katherine E. and Morgan, Daniel J. and
             Abdulnour, Raja-Elie E. and Rodman, Adam and Manrai, Arjun K.},
  year    = {2026},
  note    = {Preprint, forthcoming}
}