Towards Mitigating LLM Hallucination via Self Reflection (2023)

Abstract: Large language models (LLMs) have shown promise for generative and knowledge-intensive tasks including question-answering (QA) tasks. However, the practical deployment still faces challenges, notably the issue of “hallucination”, where models generate plausible-sounding but unfaithful or nonsensical information. This issue becomes particularly critical in the medical domain due to the uncommon professional concepts and potential social risks involved. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets. Our investigation centers on the identification and comprehension of common problematic answers, with a specific emphasis on hallucination. To tackle this challenge, we present an interactive self-reflection methodology that incorporates knowledge acquisition and answer generation. Through this feedback process, our approach steadily enhances the factuality, consistency, and entailment of the generated answers. Consequently, we harness the interactivity and multitasking ability of LLMs and produce progressively more precise and accurate answers. Experimental results on both automatic and human evaluation demonstrate the superiority of our approach in hallucination reduction compared to baselines.

## introduction large language models (llms) have shown promise for generative and knowledge-intensive tasks which require world or domain knowledge @xcite . one representative task is generative question-answering (gqa) which provides relevant information in response to queries @xcite @xcite and can also serve to probe the capabilities of language models @xcite . how-what causes noonan syndrome? noonan syndrome is caused by a mutation in the pten gene. this gene is responsible for regulating cell growth and division, and when it is mutated, it can lead to the development of noonan syndrome. noonan syndrome is caused by mutations in one of several different genes, including ptpn11, sos1, kras, nras, raf1, braf, or mek1. llm reference figure 1 : an example of hallucinated generative qa foot_1 . pten mutation is not typically associated with noonan syndrome foot_2 . thus, despite the plausible sounding answer, it is a hallucinated one. we highlight the hallucinated parts in pink and underline them. ever, despite their potential and recent advancements, these models face a concerning issue, "hallucination", a phenomenon where the model generates plausible-sounding but unfaithful or nonsensical information @xcite . in the medical domain, the challenges posed by hallucination are particularly critical, as inaccurate or misleading information can have severe consequences for patient care @xcite . furthermore, uncommon professional concepts complicate medical gqa task @xcite . for the gqa example in figure 1 , pten (phosphatase and tensin homolog) mutation is not typically associated with noonan syndrome, so the answer sounds plausible but is hallucinated. therefore, it is crucial to understand and mitigate these hallucinations to ensure the reliability and safety of ai-enabled medical services @xcite . parallel to retrieving external relevant knowledge @xcite @xcite , some current works @xcite @xcite @xcite explore leveraging the parametric knowledge in llms and tap into their potential for knowledge-intensive tasks. gqa works in other domains @xcite underscore the importance of addressing hallucination and improving faithfulness. however, the current understanding of the extent of hallucination in medical answers generated by llms remains unclear, and there is a need to explore the potential for further improvement in this aspect. to fill the gap, this study investigates hallucination in the context of medical gqa systems, particularly in general llms like vicuna @xcite , alpaca-lora @xcite , @xcite , and medical llms like medalpaca @xcite , and robinmedical @xcite on popular medical datasets: pubmedqa @xcite , medquad @xcite , mediqa2019 @xcite , livemedqa2017 @xcite , mash-qa @xcite . we evaluate the incidence of hallucinations, explore the potential cause, and propose a strategy to mitigate this issue. through a comprehensive analysis of the recent models, we hope to increase the understanding of hallucination in these systems and provide a road map towards more reliable ai-assisted healthcare. current research works @xcite @xcite @xcite highlight a gap between surface realization and inherent knowledge in nlg tasks. models can realize they are generating something hallucinated in some way. to reduce this gap and mitigate hallucinations in medical gqa, we devise an iterative, introspective process that leverages the multi-turn interactivity and multitask ability of llms. our self-reflective methodology initiates the generation of pertinent background knowledge for a given question, followed by a factuality evaluation. upon detection of discrepancies, the model is urged to self-correct, utilizing its inherent reflective capacities to refine the knowledge. this cyclical process is repeated until a satisfactory level of factuality is achieved. in the answering stage, we employ a similar generation-score-refine strategy to ensure consistency between the generated answer and the background knowledge. additionally, an entailment evaluation is conducted between the answer and the question. if the generated answer fails to meet the standard, the process returns to the initial stage, repeating the cycle. our method fosters a dynamic interaction between the system and its knowledge, enhancing the model's ability to provide accurate, reliable, and factually grounded responses in healthcare settings. the experimental results of our method showcase its effectiveness across llms with varying parameters, including 7b and 175b, on all five datasets. this robust performance highlights the generalizability and scalability of our approach, further validating its efficacy in the context of medical question-answering tasks. our method explores extracting and digging knowledge from llms, leveraging their full potential, and strives to approach their upper performance bound. our method can also combine other techniques such as utilizing external knowledge and more powerful llms in the future to contribute to the development of robust application systems. in summary, the major contributions of this work are threefold: the iterative feedback loop process generates, scores, and refines knowledge and answers until they reach satisfactory levels, enhancing the accuracy and reliability of the answers. ## analysis of hallucination in this section, we directly ask llms medical questions from five datasets leveraging their zero-shot capability. we then comprehensively evaluate and analyze the generated answers, with a focus on examining the occurrence of hallucination. ## conclusion and future work hallucinations in generation tasks pose significant challenges to ai's accountability and trustworthiness. we investigate this problem thoroughly and systematically in the context of medical gqa in general and domain-specific llms. to address this challenge, we propose an iterative self-reflection method by adopting a generate-score-refine strategy on background knowledge and answers. our method is empirically proven effective, generalizable, and scalable in reducing hallucinations. in future work, we will investigate underlying causes of hallucination, examine this phenomenon in other generation tasks and extend our method to address challenges associated with these tasks.