Zero-Resource Hallucination Prevention for Large Language Models (2024)

Abstract: The prevalent use of large language models (LLMs) in various domains has drawn attention to the issue of “hallucination”, which refers to instances where LLMs generate factually inaccurate or ungrounded information. Existing techniques usually identify hallucinations post-generation that cannot prevent their occurrence and suffer from inconsistent performance due to the influence of the instruction format and model style. In this paper, we introduce a novel pre-detection self-evaluation technique, referred to as SELF-FAMILIARITY, which focuses on evaluating the model’s familiarity with the concepts present in the input instruction and withholding the generation of response in case of unfamiliar concepts under the zero-resource setting, where external ground-truth or background information is not available. We also propose a new dataset Concept-7 focusing on the hallucinations caused by limited inner knowledge. We validate SELF-FAMILIARITY across four different large language models, demonstrating consistently superior performance compared to existing techniques. Our findings propose a significant shift towards preemptive strategies for hallucination mitigation in LLM assistants, promising improvements in reliability, applicability, and interpretability.

## introduction the widespread adoption of large language models (llms) has generated significant interest in their applications across diverse use cases, e.g., healthcare and medicine. however, a major challenge hindering their full potential is the issue of hallucination, where the models produce inaccurate or fabricated information, leading to a substantial gap in their reliability and trustworthiness and often resulting in the generation of fact-conflicting 1 code and data can be found in https://github.com/ soap117/self-evaluation user: i am writing a paper about the drug skytrofa. gpt-3.5: that sounds interesting! skytrofa (molgradex) is a medication developed for the treatment of autoimmune pulmonary alveolar proteinosis (apap). it's a rare lung disorder where a type of protein called surfactant accumulates in the air sacs of the lungs, leading to breathing difficulties. outcomes @xcite . figure 1 shows an example to demonstrate the problem when a user queries chatgpt about a newly developed drug "skytrofa" that aims to treat short stature due to inadequate secretion of endogenous growth hormone. 2 we can observe that chatgpt generates inaccurate responses on essential factors of this drug. recently, several approaches have been proposed to detect (and even correct) the hallucination responses generated by llms in open conversations, which can be roughly classified into two categories: the first category of methods hinges on a third-party knowledge base, combined with the chain-of-thought (cot) technique, to identify and rectify potential hallucination responses @xcite @xcite @xcite . the other research direction focuses on designing parameter-based methods @xcite , which mainly utilize specific metrics such as perplexity to evaluate the correctness of responses. however, all the proposed methods exhibit significant inadequacies in addressing hallucinations due to a lack of internal knowledge. firstly, the methods mentioned primarily focus on the postdetection of hallucinatory responses. these approaches require checking the responses to determine the hallucination level and lack the capability to prevent the generation of such responses in the future, thus diminishing reliability. meanwhile, humans can easily avoid this problem by recognizing inadequate knowledge. besides, the performance of existing methods is heavily influenced by instructional techniques and model styles, leading to challenges in maintaining robustness across diverse conversational scenarios. this complexity hinders the establishment of a clear threshold for distinguishing hallucinatory responses. for instance, binary queries often elicit brief response sequences, resulting in metrics that significantly diverge from those obtained from more extensive response sequences. therefore, a proactive, preventative approach, similar to that of humans, is essential for the practical and efficient application of artificial intelligence (ai) language assistants in preventing hallucinations. designing such an effective prevention method faces several challenges. firstly, the proposed approach must navigate a zero-resource environment, precluding any reliance on external knowledge gleaned from search engines. neglecting this imperative compromises the method's universality and applicability, rendering it unsuitable for situations with budgetary constraints or contexts lacking external access. consequently, a profound comprehension of the language model's intrinsic knowledge becomes essential. furthermore, the task of ensuring robustness is of paramount significance. the envisaged system must exhibit resilience against diverse instruction types, contextual variations, and model styles. given the openended and dynamic nature of human language, achieving consistent performance and unwavering resilience across a wide array of scenarios presents an undeniably formidable challenge. to tackle these challenges simultaneously, we propose a novel zero-resource, pre-detection method named self-familiarity, illustrated in figure 2 . this approach mimics human selfassessment capabilities by refraining from discussing unfamiliar concepts, thereby reducing the risk of creating hallucinated information. initially, our method extracts and processes concept entities from the instruction during the concept extraction stage. following this, the concept guessing stage individually examines the extracted concepts through prompt engineering to obtain each concept's familiarity score. lastly, during the aggregation stage, the familiarity scores from different concepts are combined to generate the final instruction-level familiarity score. compared to existing methods, our algorithm presents the following advantages. primarily, self-familiarity integrates the strengths of both cot techniques and parameter-based methods. like the cot methods, our algorithm can offer constructive responses by identifying concepts unfamiliar to the model. yet, our algorithm solely employs prompt engineering, eliminating the need for the model to possess strong inference abilities and avoiding their shortcomings while combining their advantages. additionally, our algorithm remains unaffected by instruction style and type and is proactive and preventative, thereby increasing its reliability and robustness. finally, it does not require any outside knowledge. we assessed our method across four large language models using a newly proposed predetection hallucinatory instruction classification dataset, concept-7, which focuses on the hallucinations caused by limited inner knowledge. experimental results show that the proposed self-familiarity foot_1 consistently outperforms other methods across all models in addressing this type of hallucination, demonstrating its huge application value. ## related work to the best of our knowledge, no existing work has been devoted to preventing hallucinated responses in open conversations by analyzing the understanding of concepts within the instruction under the zero-resource setting. consequently, the contexts we address are distinct from those of previous studies. ## methodology the aim of our algorithm is to evaluate if the target instruction p t is a potential hallucinatory instruction by checking the familiarity of the language model with the concepts that exist in the p t under the zero-resource setting. our method, as depicted in figure 2 , comprises three major steps: (1) concept extraction, (2) concept guessing, and (3) aggregation. the details of each step will be elucidated in the subsequent sections. ## experiments in this section, we introduce the experimental settings and results. due to space limitations, we include the implementation details, dataset creation, and additional experiment results in appendix. ## conclusion we introduced a novel pre-detection method for potential hallucination instruction, which we refer to as self-familiarity. our approach leverages concept guessing to assess the model's quality of concept explanation, thereby determining the model's level of understanding. self-familiarity consistently achieves state-of-theart results in the pre-detection of hallucinatory instruction across distinct language models using only self-evaluation under the zero-resource setting. additionally, our method demonstrates superior interpretability by identifying the concept that led to the hallucination. this unique feature enables the integration of our method with post-detection and correction techniques, enhancing its versatility. in future work, we plan to investigate how to evaluate the understanding of more granular sub-concepts to further refine the current algorithm.