Using LLM s to Grade Clinical Reasoning for Medical Students in Virtual Patient Dialogues (2025)

Abstract: This paper presents an evaluation of the use of large language models (LLMs) for grading clinical reasoning during rheumatology medical history virtual patient (VP) simulations. The study explores the feasibility of using state-of-the-art LLMs, including both general-purpose models, with various prompting strategies such as zero-shot, analysis-first, and chain-of-thought prompting, as well as reasoning models. The performance of these models in grading transcribed dialogues from VP simulations conducted on a Furhat robot was evaluated against human expert annotations. Human experts initially achieved a 65% inter-rater agreement, which resulted in a pooled Cohen’s Kappa of 0.71 and 82.3% correctness. The best LLM, o3-mini, achieved a pooled Kappa of 0.68 and 81.5% correctness, with response times under 30 seconds, compared to approximately 6 minutes for human grading. These results indicate the possibility that automatic assessments can approach human reliability under controlled simulation conditions while delivering time and cost efficiencies.

## introduction in recent years, the integration of new technologies into medical education has reshaped traditional pedagogical approaches by offering innovative solutions that improve learning outcomes. among these, social robots and large language models (llms) have emerged as promising tools, delivering immersive, interactive experiences and enhancing virtual patients (vps). vps simulate clinical scenarios, enabling students to engage in diagnostic and therapeutic decision-making processes @xcite , while social robots use advanced technologies to facilitate intuitive interactions @xcite . this dual approach fosters clinical reasoning (cr) skills and meets the growing demand for complementary educational resources. cr, i.e., "the application of knowledge to collect and integrate information from various sources to arrive at a diagnosis and management plan" @xcite , is an important part of medical education and widely recognized across health professions @xcite . high-quality cr assessment ensures that medical schools prepare competent learners, which is vital since faulty cr is a major contributor to diagnostic errors affecting patient safety @xcite . consequently, strategies to enhance cr education and assessment have become a focus in health professions research. however, because cr relies heavily on interaction, physical and economic constraints limit hands-on training. thus, innovative methods are needed to boost communication skills and adaptability in complex clinical settings. llms can transform these methods by simulating realistic patient-doctor interactions @xcite and creating personalized, interactive experiences that offer more realistic vp interactions @xcite . as noted in @xcite , reviews show that vp systems improve cr and knowledge acquisition compared to no intervention, though their benefits may be mod-est relative to conventional methods @xcite @xcite . in this paper, we implement and evaluate an assessment model using llms to support medical students in learning cr in rheumatology and to provide objective performance assessments. to validate the assessment model we benchmark its grading capabilities with practising rheumatologists as shown in figure 1 . our research question is: ## methodology our primary objective was to evaluate the use of llms for objectively grading the cr skills of medical students during a rheumatology medical history simulation. this was conducted in close collaboration with karolinska institute in stockholm, sweden, as part of a clinical teaching course in rheumatology. in the course, a vp system designed to simulate realistic clinical scenarios to assess cr was used, featuring five different patient cases @xcite . ## results this section presents the quantitative results of both human evaluations and automated dialogue assessment llms. the grading was performed on 30 dialogues comprising a total of 701 questions. in tables 2-6 the best performances of each metric are highlighted in bold. figure 6: reasoning model o3-mini correct grading of specific question 4.1 human inter-grader agreement and consensus building in the initial grading round, the graders achieved an average agreement of 65%. in the second round, where a senior grader reassessed questions with conflicting responses, consensus was reached for 92.6% of the questions (with only 52 questions remaining in disagreement). finally, after a thorough review in which the senior graders considered the responses of the other graders and cast a decisive vote, all questions ultimately reached consensus. the final consensus has the following distribution ## discussion below we integrate our observations on performance, efficiency, and the future directions for this assessment approach. ## conclusion under controlled vp simulation conditions, llmbased assessment achieved near-human reliability, suggesting that they could offer a provisional yet efficient way to assess medical students' cr skills in rheumatology history-taking. the o3-mini model achieved a cohen's kappa of 0.68, nearly matching the human expert consensus of 0.71, a 81.8 percent correct grades compared to 82.5 by human experts and graded dialogues in under 30 seconds compared to roughly 6 minutes for experts, substantially reducing grading time. although challenges remain in consistently assessing ambiguous responses and accounting for human biases, these findings highlight the potential for scalable, objective assessments amid the constraints of traditional cr training. future work should refine grading instructions and validate these tools in other clinical settings, as well as in live clinical encounters, to further enhance medical educational outcomes and patient care.