Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations (2024)

Abstract: State-of-the-art language models (LMs) sometimes generate that misalign with world knowledge. To explore the mechanistic causes of these hallucinations, we create diagnostic datasets with subject-relation queries and adapt interpretability methods to trace hallucinations through internal model representations. We discover two general and distinct mechanistic causes of hallucinations shared across LMs (Llama-2, Pythia, GPT-J): 1) : insufficient subject attribute knowledge in lower layer MLPs, and 2) : failure to select the correct object attribute in upper layer attention heads. We also found these two internal mechanistic causes of hallucinations are reflected in external manifestations. Based on insights from our mechanistic analysis, we propose a novel hallucination mitigation method through targeted restoration of the LM’s internal fact recall pipeline, demonstrating superior performance compared to baselines.

## introduction language models (lms) serve as repositories of substantial knowledge @xcite @xcite through their parametric knowledge gained from pre-training. however, they are susceptible to generating "hallucinations" that contain factual errors. at the level of logit predictions, these hallucinations often display a pattern similar to factual generations. for example, lms have been observed to produce seemingly confident completions that are, in reality, hallucinations @xcite . to understand how hallucinations differ from factual outputs and whether they are uniformly generated or equally challenging to fix, thorough analysis tools that monitoring the information flow are required, extending beyond merely last-layer predictions @xcite . however, research on understanding the internal mechanisms of hallucination generation is limited. most efforts on detecting and mitigating hallucinations @xcite @xcite treat the lm as a black box, devising methods based on external features like predictive uncertainty @xcite and logical consistency @xcite . these approaches provide little insight into the internal mechanisms of factual errors and have been shown to be unreliable with often contradictory signals @xcite . in contrast, interpretability research, which examines the internal mechanisms of transformers in white-box settings, enables the identification of components that contribute to accurate factual predictions. for example, existing work has identified several critical model "components" (e.g., attention heads, feedforward layers) related to knowledge flow that are essential for answering questions accurately @xcite @xcite . however, it remains unclear whether the results of mechanistic interpretability on factual predictions can generalize to hallucinations. specifically, it is unknown which model components deviate from normal functioning to cause hallucinations. localizing the source of non-factual hallucination in lms may help us design targeted and efficient methods to mitigate hallucinations without significantly impacting utility (e.g., by editing a small set of model weights identified as causing hallucinations, without affecting other parts that are important for information flow). in this study, we employ mechanistic interpretability @xcite to investigate the origins and manifestations of non-factual hallucinations in lms. to address the lack of datasets for non-figure 1 : our main finding of two non-factual hallucination mechanisms. left (a): the knowledge enrichment hallucinations are caused by lacking general knowledge of the subject retrieved from early and middle layer mlps -in these cases, the subjects tend to be relatively unknown and the incorrect answer is often nonsensical. right (b): the answer extraction hallucinations are caused by the failure of middle and late layer self-attention heads to identify the most relevant object to the given subject and relation -in these cases, the subjects are often more strongly associated with the hallucinating answers than the with the true answers. figure 1 illustrates two distinct scenarios where the identified hallucinating components exhibit different behaviors. in some instances, lower-layer mlps function normally, successfully retrieving semantic attributes about queried entities, while upper-layer attention heads struggle to distinguish the most relevant attributes that lead to the correct answer. in other cases, the model fails to execute its fact-recalling pipeline at the beginning, extracting no useful information from lower-layer mlps. we also observe that these two hallucination mechanisms have varying external manifestations, distinguishable by their levels of subject-object association strengths, robustness to input perturbations, and model predictive uncertainty. moreover, we demonstrate that the mechanistic insights gained from our analyses can be leveraged to develop an effective method to reduce lm hallucinations on multiple open-domain question answering datasets. our research offers the first mechanistic explanation and mitigation of lm factual errors, fostering future research on model explainability and transparency. hallucinations. language models are susceptible to generating hallucinations that can be unfaithful (i.e. deviating from the source input provided by users) or non-factual (i.e. contradicting established world knowledge) @xcite @xcite . here, we focus on the latter type of hallucination. existing studies aimed at detecting or mitigating hallucinations leverage features such as internal activation patterns @xcite , predictive confidence @xcite , and generation consistency @xcite @xcite . ## mechanistic analysis of hallucinations we wish to know which "broken" lm components are causing the model to produce a factually incorrect answer. recent studies have shown that given a subject-relation query, an lm predicts a factual object answer via two steps @xcite @xcite knowledge enrichment step, the model retrieves from mlp sublayers many relevant semantic attributes of the subject and propagates them to the last query token position; and 2) during the answer extraction step, the self-attention modules select the most relevant object entity among the previously retrieved attributes. we postulates that an lm hallucinates if any one of these two steps get compromised during inference, and perform a series of mechanistic interpretability analyses to pinpoint the malfunctioning model components. ## mechanistic hallucination mitigation in this section, we propose a novel mechanistic hallucination mitigation (mhm) method that draws inspiration from our mechanistic analysis, and demonstrate that it can improve lm factuality in open-domain question answering. method given a question x and its true answer y (we take the first token for answers with multiple tokens), we wish to fix the model's imperfect knowledge enrichment and answer extraction modules in the fact-recalling pipeline when it generates an incorrect answer y ′ . we do so by encourage the lm to retrieve more information of y from mlps, and to suppress the information propagation of t ′ from self-attention heads. in particular, during model inference with input x, we take the intermediate mlp outputs m (l) i and attention head outputs a (l) i within a specific layer range @xmath0, and then project them directly to the language modeling head layer. let log p a (y|x) be the resulting log-likelihoods of y in the projected distribution, we fine-tune the lm to optimize the following objective function: in practice, we find that l mhm can be combined with the regular negative log likelihood (nll) loss of lm fine-tuning to achieve the best factuality: ## conclusion we conducted various interpretability analyses on non-factual hallucinations made by language models. we show