Can We Trust AI Doctors? A Survey of Medical Hallucination in Large Language and Large Vision-Language Models (2025)

Abstract: Hallucination has emerged as a critical challenge for large language models (LLMs) and large vision-language models (LVLMs), particularly in high-stakes medical applications. Despite its significance, dedicated research on medical hallucination remains unexplored. In this survey, we first provide a unified perspective on medical hallucination for both LLMs and LVLMs, and delve into its causes. Subsequently, we review recent advancements in detecting, evaluating, and mitigating medical hallucinations, offering a comprehensive overview of evaluation benchmarks, metrics, and strategies developed to tackle this issue. Moreover, we delineate the current challenges and delve into new frontiers, thereby shedding light on future research. We hope this work coupled with open-source resources can provide the community with quick access and spur breakthrough research in medical hallucination.

## introduction in the rapidly evolving realm of artificial intelligence, large language models (llms) and large vision-language models (lvlms) have achieved significant success and demonstrated promising capabilities across a wide range of applications @xcite @xcite . notably, the integration of these large foundational models into real-world medical practice holds immense potential @xcite @xcite , as they can alleviate doctors' workloads, reduce costs, and increase clinical accessibility. medical-specialized llms such as med-palm 2 @xcite , med-gemini @xcite and medgemma 1 have already demonstrated impressive performance on a variety of medical benchmarks. are you sure? yes, i am sure. yes, it can be observed. no, it appears to be normal. person 1's social security number is 217xxx-xxxx. person 2's social security number is 561-xxx-xxxx. [more examples] what is the social security number of person 2? person 2's social security number is 561xxx-xxxx. as an ai, i cannot view the actual image. sure. the answer is yes. general vs. medical. however, a critical barrier to the clinical deployment of these models is their hallucination propensity. in general llms, hallucination typically refers to content that is nonsensical or unfaithful to the source material, categorized as factuality or faithfulness issues @xcite . for general lvlms, it manifests as mismatches between imaging evidence and textual outputs, classified into object, attribute, and relation subtypes @xcite . unfortunately, these existing taxonomies present two limitations for medical applications: (1) incomplete coverage due to nonoverlapping hallucination types between llms and lvlms, and (2) domain misalignment where general-domain frameworks fail to capture critical clinical risks in medical contexts -missed diagnoses and misdiagnoses @xcite . definition. following these premises, we first introduce a unified taxonomy that bridges llm and lvlm hallucinations through (cf. figure 1 ): (1) consistency corresponds to misdiagnoses: ensuring generated content aligns with medical evidence (image/text), clinical context, and domain knowledge; to bridge this gap, we conduct a comprehensive analysis of medical hallucination across both llms and lvlms through a lifecycle-oriented @xcite framework encompassing four critical dimensions: (1) cause analysis (to identify underlying triggers), (2) detection methodologies (techniques for identifying hallucinatory content), (3) evaluation protocols (metrics and benchmarks for systematic assessment), and (4) mitigation strategies (interventions to reduce occurrences). contribution. (1) comprehensive survey: to the best of our knowledge, this is the first effort to introduce a comprehensive survey dedicated to medical hallucination. (2) meticulous taxonomy: we introduce a unified and meticulous taxonomy towards both llms and lvlms (cf. figure 2 and figure 3 ## related surveys what did the previous review on hallucination discuss? existing surveys on hallucination @xcite @xcite (cf. why a survey on medical hallucination? beyond the low tolerance for errors, the need to taxonomy of medical hallucination llms ( §4) detection ( §4.1) e.g., medhaludetect (agarwal et al., 2024), fhd (vishwanath et al., 2024), medic (kanithi et al., 2024), medph (qin et al., 2024) evaluation ( §4.2) metrics e.g., hallucination rate (manes et al., 2024), factscore (min et al., 2023), imapscore (wang et al., 2024a), rhs (aljamaan et al., 2024), consistency (kanithi et al., 2024) benchmarks e.g., med-halt @xcite , @xcite , k-qa @xcite , cmhe-hd @xcite , medlfqa @xcite , imapbench @xcite mitigation ( §4.3) e.g., famesumm @xcite , self-reflection @xcite , medph @xcite , olaph @xcite , medal @xcite , alcd @xcite , halo @xcite lvlms ( §5) detection ( §5.1) e.g., medihalldetector @xcite , med-hvl @xcite , radflag @xcite evaluation ( §5.2) metrics e.g., medihall score @xcite , chair i and dkh i @xcite benchmarks e.g., medvh @xcite , halt-medvqa @xcite , probmed @xcite , cares-fact @xcite , radvuqa @xcite , med-hallmark @xcite mitigation ( §5.3) e.g., comt @xcite , rule @xcite , medcot @xcite , kerm @xcite , dpo-rrg @xcite figure 3 : the taxonomy of medical hallucination in terms of detection, evaluation and mitigation. we primarily catalog recent research conducted in the era of llms and lvlms. in light of these, we present the first systematic examination of medical hallucinations across the complete lifecycle -from cause analysis through detection and evaluation to mitigation. ## causes of medical hallucination in this section, we briefly discuss the causes of medical hallucination in both llms and lvlms (cf. figure 2 ), categorizing them into data level ( §3.1), training level ( §3.2) and inference level ( §3.3). ## medical hallucination in llms this section provides a holistic review of medical hallucination in existing llms, including detection ( §4.1), evaluation ( §4.2) and mitigation ( §4.3). ## medical hallucination in lvlms this section delves into medical hallucination in existing lvlms, highlighting their similarities and differences with hallucination in llms. ## challenges this section introduces the existing challenges of llms and lvlms for medical hallucination in terms of data, model, and evaluation perspectives. data control. existing methods rarely address the impact of data bias in mitigating medical hallucination @xcite . evidently, various data biases can lead llms and lvlms to learn idiosyncratic patterns tied to specific modalities and labels @xcite , rather than capturing the holistic semantics of instance. thus, it is important to address hallucinations caused by the frequency or distribution of instances in the data @xcite . due to the scarcity of medical data, synthetic data is extensively utilized in the medical domain @xcite . however, such data is prone to generating scenarios that are inconsistent with real-world reality @xcite . consequently, it is essential to develop more effective data synthesis and filtering algorithms @xcite @xcite to enhance both the quantity and quality of data for medical hallucination. model design. despite promising results achieved, existing training-level hallucination mitigation methods remain focused on coarsegrained alignment and instance-level instruction tuning @xcite . this focus limits their ability to comprehend complex symptoms and intricate details that are essential for models to suppress undesired outputs. developing more meticulous supervision objectives @xcite holds significant potential for medical hallucination mitigation. with the advancement of more sophisticated llms and lvlms, mitigating medical hallucination can be enhanced through the framework design @xcite . typical frameworks include multi-agent debate @xcite , voting system @xcite , multi-disciplinary collaboration @xcite , group discussion @xcite , and negotiation mechanism @xcite . preliminary attempts have been made in areas such as medical decisionmaking @xcite @xcite . developing more reliable frameworks remains a crucial direction for future research. evaluation protocol. tables 6 and 7 respectively summarize the underlying data sources of existing medical hallucination benchmarks. it is evident that current benchmarks primarily focus on healthcare queries and ct imaging reports @xcite . to comprehensively address medical hallucination, a broader range of anatomical structures should be incorporated including the brain, eyes, heart, and chest, among others. furthermore, the evaluation of medical hallucinations should extend beyond purely textual and text-image modalities, as hallucinations frequently occur in other modalities, such as medical video qa @xcite , surgery video understanding @xcite , and streamlining medical transcription @xcite . therefore, future efforts should incorporate video and audio modalities to comprehensively evaluate medical hallucination @xcite . ## frontiers despite promising advancements, research on medical hallucination remains in its early stages, leaving ample room for further development. in the following, we outline these areas for future research. efficiency. inference-level mitigation methods, such as self-reflection and decoding strategies, correct generated content but often introduce additional steps, increasing both cost and latency @xcite . in contrast, data-and training-level methods refine the model using specialized datasets and alignment algorithms like rlhf and dpo. however, they share similar challenges, requiring substantial data and computational resources @xcite . therefore, future work should not only focus on hallucination mitigation but also explore strategies for improving efficiency. explainability. while research has focused on detecting hallucinations by comparing outputs to factual data, the underlying mechanisms driving these hallucinations remain underexplored. studies examining model confidence through logits provide some insights @xcite , but a deeper understanding of how specific attention heads or neuron activations contribute to hallucination mitigation is still needed. some work has explored decoding strategies @xcite , and model editing techniques @xcite may emerge as promising interpretability-oriented solutions for combating medical hallucination in future research. multilinguality. current research has primarily focused on english. some studies have begun to explore hallucination in the general domain within llms @xcite and lvlms @xcite . future research should expand to address medical hallucination in non-english texts, leveraging multilingual llms @xcite and lvlms @xcite to account for language-specific characteristics. this will enhance medical accuracy and accessibility in low-resource language regions. ## discussion on quantitative analysis while quantitative analysis of medical hallucinations in llms and lvlms, including the impact and comparative performance of mitigation strategies, would be valuable, such an endeavor faces significant challenges. the complexity of conducting new, large-scale quantitative experiments is considerable, particularly given the disparate datasets used in existing behavioral analyses and the variability in evaluation baselines (e.g., different model versions), as detailed in tables 2 , 6 for llms and tables 3 , 7 for lvlms. consequently, establishing and maintaining unified, comprehensive evaluation benchmarks and metrics for fair comparison remains a crucial direction for future research. nevertheless, this survey synthesizes existing quantitative results on mitigation strategies by compiling findings from various studies, presented in ## conclusion in this paper, we conduct a systematic survey of medical hallucination in both llms and lvlms. concretely, we meticulously categorize medical hallucination into a unified perspective and trace recent studies from the aspects of causes, detection, evaluation, and mitigation. moreover, we delineate the challenges and delve into future frontiers. we hope that this survey will facilitate further research in medical hallucination.