Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions (2025)

Abstract: LLMs have demonstrated impressive performance in answering medical questions, such as achieving passing scores on medical licensing examinations. However, medical board exams or general clinical questions do not capture the complexity of realistic clinical cases. Moreover, the lack of reference explanations means we cannot easily evaluate the reasoning of model decisions, a crucial component of supporting doctors in making complex medical decisions. To address these challenges, we construct two new datasets: JAMA Clinical Challenge and Medbullets. JAMA Clinical Challenge consists of questions based on challenging clinical cases, while Medbullets comprises simulated clinical questions. Both datasets are structured as multiple-choice question-answering tasks, accompanied by expert-written explanations. We evaluate seven LLMs on the two datasets using various prompts. Experiments demonstrate that our datasets are harder than previous benchmarks. In-depth automatic and human evaluations of model-generated explanations provide insights into the promise and deficiency of LLMs for explainable medical QA.

## introduction medical question answering (qa) involves synthesizing relevant information and knowledge and then reasoning about how to apply them to the current situation. these characteristics, presented in many large language models (llms) @xcite @xcite , have been demonstrated in the case of answering medical questions, with some claims that llms can achieve passing scores on medical board exams @xcite . these results have been achieved with both domain-specific llms, such as med-palm @xcite and med-itron @xcite , as well as generalpurpose llms, such as @xcite , on the united states medical licensing examination (usmle) questions @xcite . while high accuracy on medical licensing exams sounds impressive, it does not mean that these models can answer complex medical questions or assist doctors with clinical decisions. board exam questions @xcite and general medical questions @xcite rely on textbook knowledge, a task for which llms are well suited since they learn primarily from medical texts. in contrast, doctors need support with clinical presentations that differ from textbook definitions and require blending established knowledge with clinical experience @xcite . additionally, doctors seek explanations, beyond predictions, that elucidate reasons for a decision to better understand complex clinical cases @xcite . models should be evaluated on their ability to correctly explain complex medical decisions in addition to making them. moving the evaluation of the medical capabilities of llms toward more realistic and challenging clinical settings requires supportive benchmarks. there have been some attempts in this regard, but they are limited in scope and the datasets are generally small @xcite @xcite . some benchmarks evaluate model capabilities in understanding clinical texts or following clinical instructions, without necessitating complex reasoning over synthesized information and knowledge for clinical decisions @xcite @xcite . while recent benchmarks include questions derived from real-world conversations or closely aligned with real-world practice @xcite @xcite , they do not provide explanations. other bench-figure 1 : examples from the jama clinical challenge and medbullets datasets respectively. each question is paired with 4/5 answer choices and a discussion/explanation. the correct answers are highlighted in green boxes. marks include explanations for medical exam questions @xcite . to the best of our knowledge, our datasets are the first to feature challenging real-world clinical questions accompanied by high-quality, expert-written explanations. we address this evaluation gap through the construction of two datasets: jama clinical challenge and medbullets (figure 1 ). both datasets contain high-quality explanations written by human experts-a distinctive feature lacking in previous benchmarks. jama clinical challenge consists of 1524 clinical cases collected from the jama network clinical challenge archive. these articles are summaries of actual challenging clinical cases framed as challenging questions. each article contains a long case description followed by a question, four answer choices, and a discussion (explanation) 2 explaining the correct (highlighted in green) and incorrect answers. the questions cover a wide range of medical topics. we expect these questions to be especially challenging given their length and complexity. medbullets comprises 308 usmle step 2&3 style questions collected from open-access tweets on x (formerly twitter) since april 2022. the difficulty is comparable to that of step 2&3 exams, which emulate common clinical scenarios but are shorter and (perhaps) less challenging than jama. each question is paired with a case description, five answer choices, and an explanation of correct and incorrect answers. although existing benchmarks have included the same type of questions (e.g., @xcite ), our data contains answer explanations and is more challenging due to its recency, which has led to a significant drop 2 for consistency, we will use "explanation" in this paper. we evaluate seven llms, spanning closedsource and open-source, general-purpose and domain-specific models: gpt-3.5 @xcite , @xcite , palm 2 @xcite , llama 2 @xcite , llama 3 (meta, 2024), medalpaca @xcite , and meerkat @xcite . we find our datasets to be harder (lower accuracy) compared to previous benchmarks, highlighting the challenges posed by the new tasks. in-context learning and prompting strategies yield marginal improvements, leaving potential areas for improvement in future work. indepth automatic and human evaluations of modelgenerated explanations highlight the limitations of llms in explaining complex medical decisionmaking. moreover, the weak correlation between human and automatic scores underscores the necessity of developing metrics that can support future research on explainable medical qa. overall, these datasets represent more challenging goals for model predictions and a new challenge in producing meaningful explanations for medical decisions. ## datasets we present two medical qa datasets with explanations, both in english. ## experimental setup we evaluate a range of llms on the datasets to determine if they are more challenging than previous benchmarks. we also evaluate the ability of models to produce explanations. we start by describing the llms in our experiments ( §3.1), followed by prompting strategies ( §3.2), and evaluation metrics ( §3.3). more details are in appendix b. ## results and discussion we measure the prediction accuracy of the seven llms on medqa and our two datasets with the @xmath0.1), few-shot prompts (@xmath1, §4.2) and cot reasoning (@xmath2, §4.3). we evaluate model-generated explanations under xy * →r prompting via both automatic and human evaluations ( §4.4). ## related work the emergence of llms has significantly transformed the medical domain, particularly in the realm of medical qa tasks. for example, generalpurpose gpt-4 (openai, 2023) has succeeded on medical board examination questions, such as the usmle @xcite . additionally, many llms have been fine-tuned or adapted to the medical domain, deriving their domain-specific variants, such as med-palm (2) @xcite , medalpaca @xcite , meditron @xcite . these models have been evaluated on medical qa benchmarks, which mostly consist of board exam questions @xcite @xcite or general medical questions @xcite @xcite . answering these questions, based on textbook knowledge or online resources, may be well-suited for llms, which learn from extensive texts @xcite @xcite . other datasets constructed to facilitate research on clinical qa also fail to capture the complexity of realistic clinical situations @xcite @xcite . some attempts have been made to examine llms on challenging clinical cases, but either the scope (restricted to a specific domain) or scale is limited @xcite @xcite @xcite . another line of work creates benchmarks to evaluate llms in handling long clinical texts @xcite , following clinical instructions @xcite , or performing multiple tasks across multiple domains @xcite . the tasks include classification and text-to-text generation (e.g., summarization, translation). differently, our focus is on multiple-choice qa tasks, assessing the abilities of llms in applying medical reasoning to infer correct answers. recent benchmarks collect questions derived from real-world conversations or closely aligned with real-world practice @xcite @xcite . however, these datasets do not include medical explanations. while medm-cqa @xcite provides explanations, the quality is not satisfactory. explaincpe's explanations are for chinese pharmacist examination @xcite . medexqa provides explanations for questions based on mock tests or online exams @xcite . our datasets include questions based on challenging real-world clinical cases, accompanied by high-quality, expert-written explanations. ## conclusion we introduce two challenging medical qa datasets, jama clinical challenge and medbullets, accompanied by expert-written explanations. we evaluate seven llms' ability in answering and explaining these medical questions. we find that these datasets are harder than previous benchmarks. since these are more reflective of complex clinical cases, they represent a new challenge for medical llm research. we assess model-generated explanations using various automatic metrics and human evaluation. while llms produce promising explanations, they also exhibit deficiencies such as irrelevance and errors. additionally, existing automatic evaluations do not correlate well with human judgements, suggesting future work to define a suite of evaluation metrics suitable for explainable medical qa.