Learning to Trust Your Feelings: Leveraging Self-awareness in LLM s for Hallucination Mitigation (2024)

Abstract: We evaluate the ability of Large Language Models (LLMs) to discern and express their internal knowledge state, a key factor in countering factual hallucination and ensuring reliable application of LLMs. We observe a robust self-awareness of internal knowledge state in LLMs, evidenced by over 85% accuracy in knowledge state probing. However, LLMs often fail to faithfully express their internal knowledge during generation, leading to factual hallucinations. We develop an automated hallucination annotation tool, DreamCatcher, which merges knowledge probing and consistency checking methods to rank factual preference data. Using knowledge preference as reward, We propose a Reinforcement Learning from Knowledge Feedback (RLKF) training framework, leveraging reinforcement learning to enhance the factuality and honesty of LLMs. Our experiments across multiple models show that RLKF training effectively enhances the ability of models to utilize their internal knowledge state, boosting performance in a variety of knowledge-based and honesty-related tasks.

## introduction large language models (llms), including notable examples such as gpt-3 @xcite , llama @xcite al. (2023a), touvron et al. (2023b) ), and palm @xcite , have emerged as a transformative tool in diverse fields due to their robust capabilities in various tasks. however, despite this significant progress and success, an inherent challenge continues to persist: their tendency to "hallucinate", i.e., generate content misaligned with actual facts. this issue is particularly problematic in critical applications, such as clinical or legal scenarios, where the reliability and accuracy of generated content is vital. therefore, mitigating hallucinations in llms is a * work done in idea crucial step toward enhancing their practical application scope and improving the overall trust in these emerging technologies. hallucinations of llms can be categorized into three types @xcite : input conflict, context conflict, and factual conflict. this paper focus on the issue of fact-conflicting hallucination, where llm produces fluent and seemingly plausible content, but conflicts with real-world facts, pose risks of misleading users and compromise the models' fact-based reasoning. commonly used hallucination mitigation methods, such as retrieval augmentation generation (rag), address fact-conflict hallucination of llm by bringing in external knowledge, but at the cost of introducing a retrieval system. in this paper, we propose to mitigate the factual hallucination problem from the perspective of enhancing the model's utilization of internal knowledge. previous works @xcite , @xcite ) have shown that llms have the capability to discern the validity of factual statements, supported further by @xcite suggesting these models' capacity to assess their ability in responding to specific questions. nevertheless, the universality and extent of models' self-awareness of their internal knowledge remains an open question. in light of this, we conducted exploratory experiments to probe the knowledge state of various models across different scales, employing linear probes to ascertain the accuracy of models' self-awareness regarding their internal knowledge states. the results revealed that all models under analysis demonstrated proficient accuracy in recognizing whether they have the internal knowledge about certain facts. however, during generation, such accurate judgments do not translate into honest output; instead, in the absence of specific internal knowledge, models often manifest a tendency towards hallucinations. therefore, to mitigate factual hallucinations, it is crucial that models leverage their selfawareness of internal knowledge states. we propose a training framework named reinforcement learning from knowledge feedback (rlkf) to improve the factuality and honesty of llm with reinforcement learning using factual preferences as reward. through the hallucination annotation method dreamcatchera blend of knowledge probing and consistencybased judgments -we rank the knowledge-based question-answering (qa) data adhering to a preference hierarchy delineated as: f @xmath0. this factual preference data is then utilized to train the reward model which is deployed to optimize the large language model via proximal policy optimisation (ppo) algorithm. the primary contributions of this paper are articulated as follows: ## problem setup hallucination, in the context of large language models, refers to a set of inconsistencies in model generation. the central focus of this paper is exploring the fact-conflict hallucination which is defined as the inconsistency between the generated content and the established facts. although the definition provides a description of the generation results, the causes underlying this phenomenon are multifaceted. in general, llms encode factual knowledge into parameters during training and utilize this internal knowledge during inference. however, llms do not always honestly express the knowledge in its parameters, which is one of the major causes of fact-conflict hallucination. for a given question that requires factual knowledge, the model output can be classified into one of four states, depending on the model's internal knowledge and its honesty. these states are illustrated in figure 1 : state 1: the model has relevant internal knowledge and expresses it faithfully. state 2: despite having the relevant internal knowledge, the model fails to express it honestly. this discrepancy could be due to various factors such as the decoding strategy @xcite , hallucination snowballing @xcite , or misalignment issues @xcite . state 3: the model lacks the necessary internal knowledge but honestly expresses unawareness. state 4: the model lacks the necessary internal knowledge and produces a hallucinated response. outputs in state 2 and state 4 are both considered forms of hallucination, despite the differing conditions of internal knowledge. in the upper section of figure 1 , the model's outputs are devoid of hallucinations, honestly mirroring its internal knowledge. here, state 1 stands out as the most desirable state, where the model both possesses and faithfully outputs the relevant knowledge. many efforts have been deployed to transition model toward state 1. retrieval-augmented generation (rag) attempts to bypass the lack of internal knowledge by providing knowledge via context, thereby enabling the model to transition from state 3/4 to state 1. on another front, certain strategies, like those of li et al. (2023b) and chuang et al. (2023), seek to move the model from state 2 to state 1 by intervening the model's internal representation or the decoding process during inference. while these methods improve the model's capacity to express existing internal knowledge, they disregard scenarios where the model lacks relevant internal knowledge. also, interference at inference time can potentially lead to unpredictable effects on other types of tasks. without the introduction of external knowledge, the mitigation of the model's fact-conflict hallucination correspond to an upward movement of the state in figure 1 . in essence, this symbolizes the enhancement of the model's capacity to accurately express its internal knowledge state. a critical question, then, is how to discern the internal knowledge state of llms? ## knowledge state probing this section delves into the complexities of discerning a model's internal knowledge state. it comprises two perspectives. the first, an external perspective, discuss how to determine if a model possesses specific knowledge based on the model generations; the second perspective, an internal view, questions if it is possible to determine whether a model possesses specific knowledge by its internal activation. for the following pilot experiments, we select three families of models with different sizes: llama2-chat @xcite (13b and 7b); qwen-chat @xcite (14b and 7b); ziyareader @xcite . as for data, we randomly select passages from chinese and english wikipedia and instruct gpt3.5 to generate a knowledge-related questionanswer pair. the answer generated by gpt3.5 based on the original wikipedia is considered as the correct answer. we refer to the qa pairs obtained by this method as wiki-qa in this paper. examples of instructions and corresponding output are shown in appendix a.1. ## method from the above knowledge-probing experiments, we discover that llms are capable of evaluating their own knowledge states in response to specific knowledge-based questions. this implies that llms demonstrate a self-awareness of their knowledge state, which does not consistently translate into their generation. frequently, when faced with questions outside of internal knowledge, llms tends to generate hallucinations. additionally, even with questions within internal knowledge, llms may potentially generate incorrect responses due to other influences. one possible explanation could be that llms did not learn to generate with respect to the internal knowledge state during model training. instead, the fine-tuning process often requires the model to generate seemingly reasonable answers to all factual questions. we therefore emphasize on enhancing the model's utilization of internal knowledge state so that the model can choose to rely on internal knowledge to answer or honestly express its lack of relevant knowledge. 2 consequently, we propose the rlkf (reinforce learning from knowledge feedback) training framework. this introduces model knowledge state assessments into the reinforcement learning feedback mechanism, enhancing model honesty and factuality. the rlkf training process shares similarities with the standard rlhf (reinforce learning from human feedback), and can integrate smoothly with the existing rlhf framework, but reduces data collection costs by substituting human labeling with automatic knowledge annotation. the rlkf training framework consists of the following components, as shown in figure 3 . ## experiments in the following experiments, we chose three different models of varying sizes: llama2-chat (13b and 7b); qwen-chat (14b and 7b); and ziya-reader (13b), which is consistent with the choice of models for the knowledge-probing experiments detailed in section 3. ## related work hallucination in large language models (llms) has been the focal point of research, spanning its causes, detection, and mitigation. our work relates to all three aspects. causes of hallucination: studies have linked llm hallucination to various causes. @xcite ascribes it to memorization of training data, indicating a direct correlation between the training data and the resultant hallucination. other works such as schulman (2023) pinpoint improper model fine-tuning as contributive, and @xcite argues that rlhf induce model "sycophancy" which in turn degrades honesty. other studies link hallucinations to the generation process. for example, @xcite suggests that sampling-induced randomness could be responsible. one perspective provided by @xcite proposes that "lower-level" prior layer information might overshadow factual information from subsequent layers. furthermore, some works relate hallucinations to the overconfidence of llms @xcite . hallucination detecting: in terms of detecting hallucination, the consistency of multiple generations has been recognized as an effective indicator. selfcheckgpt @xcite capitalizes on the consistent nature of internal knowledgebased generations compared to the variable nature of hallucination, propose several consistency checks to identify hallucinations. the idea is echoed by @xcite , who suggest evaluating the generation consistency of generated references to spot hallucination. similarly, @xcite proposes a metric involving the calculation of sentence-level entailment between response pairs as a measure of hallucination. employing large language models (llms) to recognize their own hallucinations has been suggested in @xcite , suggesting that discrimination is more accurate than generation for llms (g-d gap). this notion is furthered by @xcite and @xcite by directly prompting llms to assess the validity of their own output. another approach examines the factualness of statements by analyzing the model's internal representation. studies @xcite and @xcite identify a "factualness" direction in the model's internal representation, with @xcite showcasing a high accuracy attention head through linear probing, and @xcite locating factualness direction through consistency of facts. additionally, @xcite trains the model to predict the probability that it knows. base on these works, we shifts focus onto the model's self-evaluation of knowledge state. hallucination mitigation: the common ap-proach of hallucination mitigation involves enhancing the model with additional information. @xcite propose the use of larger models to provide additional information when hallucinations is detected. some research efforts focus on the optimization of decoding strategies to address hallucinations. @xcite suggests that contrastive decoding can augment the factualness of model generation. @xcite enhances factualness by adjusting the output of attention heads along the direction of factualness during inference. our work seeks to optimizes the utilization of the model's internal knowledge state, in line with the direction proposed by @xcite leveraging reinforcement learning to tackle hallucinations. ## conclusion in our research, we thoroughly explore the capability of large language models (llms) to discern and express their internal knowledge, a key factor in mitigating factual hallucinations and ensuring reliable applications. our research, manifested through a series of knowledge probing experiments, identifies the model's self-awareness of its knowledge state. we released the open-source tool dreamcatcher which scores and annotates the degree of hallucination in the llm's response to knowledge-oriented question and rank responses based on their factuality. we further validated our findings through the reinforcement learning from knowledge feedback (rlkf) training framework. utilizing dream-catcher to annotate factual preference data, we train a reward model and leveraging reinforcement learning to enhances llm's factuality and truthfulness. our results indicate rlkf's effectiveness in improving the model's utilization of its internal knowledge state, enhancing its performance in various knowledge and honesty related tasks. we posit that rlkf is a promising solution to address llm's hallucination issues and, combined with rlhf, offers significant potential for enhancing the model's overall capabilities.