Do Large Language Models Know When Not to Answer in Medical QA ? (2025)

Abstract: Uncertainty awareness is essential for large language models (LLMs), particularly in safety-critical domains such as medicine where erroneous or hallucinatory outputs can cause harm. Yet most evaluations remain centered on accuracy, offering limited insight into model confidence and its relation to abstention. In this work, we present preliminary experiments that combine conformal prediction with abstention-augmented and perturbed variants of medical QA datasets. Our early results suggest a positive link between uncertainty estimates and abstention decisions, with this effect amplified under higher difficulty and adversarial perturbations. These findings highlight abstention as a practical handle for probing model reliability in medical QA.

## introduction uncertainty is a defining feature of human language: ambiguity, underspecification, and incomplete information are the rule rather than the exception. nevertheless, most nlp evaluation continues to assume that such ambiguities must be resolved, with accuracy as the dominant metric. this assumption becomes especially problematic in high-stakes domains such as medicine, law, and finance @xcite @xcite @xcite , where overconfident but incorrect answers can cause harm. recent advances show that large language models (llms) can achieve near-expert performance on many tasks @xcite , but their reliability hinges not only on being right when confident, but also on knowing when not to answer. in medical qa, for instance, users frequently pose ambiguous or even unanswerable * equal contribution, alphabetical order † co-corresponding authors queries @xcite , where calibrated abstention could prevent hallucinations and unsafe recommendations @xcite . existing benchmarks such as medqa, medqa-cs, and medmcqa @xcite @xcite mainly measure accuracy, leaving open the question of whether models can represent and act on their own uncertainty. at the same time, broader efforts in uncertainty quantification (uq) for llms, such as lm-polygraph @xcite , have begun to systematize estimation methods and provide unified implementations, while work in medical text analysis @xcite highlights selective prediction as a practical approach to safety in diagnosis. these directions reinforce the importance of studying abstentionaware evaluation in medical qa, where ambiguity and incomplete context are unavoidable. we use medical multiple-choice qa as a controlled proxy for clinical decision making: its finite option space yields precise uncertainty sets and abstention rules, and the resulting signals about when to answer or defer carry over to broader medical nlp tasks. in this work, we present ongoing work on abstention and uncertainty in medical multiple-choice qa. we combine conformal prediction @xcite with adversarial perturbations and abstention-augmented questions to probe how models behave under ambiguity. our preliminary findings suggest a consistent positive association between uncertainty and abstention: when given the explicit option to abstain, models tend to signal higher uncertainty, with effects amplified on more difficult and perturbed questions. we take these results as tentative evidence that abstention can serve as a conservative and responsible mechanism for handling uncertainty in medical qa with llms. abstention, refusal, and calibration in llms abstention, understood as deferring under uncertainty, spans from classic classification to modern llms @xcite @xcite . although some benchmarks add explicit abstain or "cannot answer" options, standardized mcqa evaluation, especially for proprietary models, remains scarce @xcite . existing approaches such as verbalized uncertainty, prompting, finetuning, and post-hoc rejection often show limited calibration or generalization @xcite @xcite @xcite . in medicine, selective prediction has been studied as a practical strategy for low-confidence cases, with recent work introducing huq-2, a hybrid method that combines aleatoric and epistemic uncertainty across tasks like mortality prediction, icd coding, and mental health detection @xcite @xcite . these studies show abstention reduces overconfident errors and even supports label-level abstention in multi-label settings. yet applications to medical qa remain limited, motivating our study. our qa focus complements classification-centric selective prediction by converting uncertainty into explicit answer-or-abstain decisions that generalize to defer or retrieve policies in clinical nlp. reasoning, prompting, and hallucination in llms reasoning-tuned models and chain-ofthought (cot) prompting improve accuracy in math, science, and clinical qa @xcite @xcite @xcite . yet accuracycentric evaluation neglects overconfidence and answer-at-all-costs behavior, compounding hallucination risks @xcite @xcite . current benchmarks such as abstentionbench, co-conot, and abstain-qa mainly emphasize opendomain settings, seldom probing abstention under adversarial or perturbed mcqa or scaling effects @xcite @xcite @xcite . we analyze how prompting and scale interact with abstention reliability in clinical mcqa. ## methodology our approach focuses on medical multiple-choice question answering (mcqa) tasks, consistent with the evaluation structure of the open medical-llm leaderboard. 1 the mcq format is especially suitable for uncertainty analysis via conformal prediction, which requires a well-defined output label space y (for more details, see appendix a). ## experiments we evaluate a broad set of both open-source and closed-source llms, spanning multiple architectural families and model scales. this diverse selection allows us to assess the generality of abstention and uncertainty behaviors across different llm paradigms. section b provides a comprehensive list of the models used for the study. under each experimental condition, models are prompted to output a single answer token (the selected option), and accuracy is computed by comparing this token with the gold label. the logit corresponding to the emitted token, together with the logits for the remaining candidate choices, is then extracted to compute conformal-prediction scores. for closed-source gpt-family models, these scores are derived from the api-exposed top-logprobs. ## results and discussion comparison of aps and lac distributions as shown in fig. 2 (a), aps produces tighter, lowervariance set-size distributions than lac across both datasets, suggesting more stable thresholding. under ap conditions, aps distributions also crowd near the upper limit, indicating that prediction sets frequently expand to include most options. this compactness carries over when conditioning on correctness (fig. 2(b) ), where aps remains less variable, though still skewed toward larger sets for incorrect answers. together, these patterns suggest that aps offers more consistency in how uncertainty maps to abstention. by contrast, lac produces broader set-size distributions (fig. 2(a) ), with a wider gap between correct and incorrect cases and heavier right tails (fig. 2(b) ). this separation is particularly visible in medqa, where lac more distinctly highlights error-prone instances. while less stable for thresholding, lac may therefore be more useful in contexts where surfacing likely mistakes -for example, for human review or triage. ## conclusion in this study, we asked how item difficulty shapes model uncertainty and abstention, and how two set-size signals: lac and aps, serving as uncertainty proxies across prompting style (cot vs. no-cot), demonstration count (zero-vs. few-shot), and input perturbations. we find a strong, positive uncertainty-abstention relationship and a consistently negative association between both aps/lac and accuracy. averaged over datasets, increasing difficulty does not materially change aggregate uncertainty or abstention. practically, aps is a reliable gate for abstention across conditions, while lac is a robust indicator of accuracy risk whose coupling to abstention weakens with cot: especially on easier items. aps produces tighter, more stable distributions, whereas lac yields clearer separation between correct and incorrect answers, suggesting complementary strengths in thresholding versus error triage.