Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (2024)

Abstract: When asked to summarize articles or answer questions given a passage, large language models (LLMs) can hallucinate details and respond with unsubstantiated answers that are inaccurate with respect to the input context. This paper describes a simple approach for detecting such **contextual hallucinations**. We hypothesize that contextual hallucinations are related to the extent to which an LLM attends to information in the provided context versus its own generations. Based on this intuition, we propose a simple hallucination detection model whose input features are given by the ratio of attention weights on the context versus newly generated tokens (for each attention head). We find that a linear classifier based on these _lookback ratio_ features is as effective as a richer detector that utilizes the entire hidden states of an LLM or a text-based entailment model. The lookback ratio-based detector—**Lookback Lens**—is found to transfer across tasks and even models, allowing a detector that is trained on a 7B model to be applied (without retraining) to a larger 13B model. We further apply this detector to mitigate contextual hallucinations, and find that a simple classifier-guided decoding approach is able to reduce the amount of hallucination, for example by 9.6% in the XSum summarization task.

## introduction despite the utility and impressive capabilities of large language models (llms), their tendency to generate hallucinations, i.e., content that deviates from facts or contextually relevant information @xcite , presents a significant challenge in their deployment. in this work, we focus on the scenarios where the model is provided with the correct facts within the input context but still fails to generate accurate outputs, a phenomenon we term contextual hallucination. despite the simplicity of 1 source code: github.com/voidism/lookback-lens this setup, llms struggle with contextual hallucinations, frequently producing errors in tasks such as summarization and document-based question answering (e.g., most prior studies that propose methods to combat hallucination focus on the scenario without any input context, where the hallucinations arise from the llms' parametric knowledge. these works detect and mitigate hallucinations by generally using the llm's representations, such as hidden states @xcite , mlp outputs @xcite , attention block outputs @xcite and attention head outputs @xcite @xcite . in contrast, the provided contextual information plays a key role in detecting contextual hallucinations. insofar as attention (more so than other model internals) provides a humanmeaningful measure of how much weight is given to the context during generation, this motivates the use of signals from the attention maps for hallucination detection and mitigation. to leverage signals from attention maps, we start by hypothesizing that contextual hallucinations are related to the extent to which an llm attends to the provided contextual information. concretely, we propose a simple feature called lookback ratio, which is computed as the ratio of attention weights on the given context versus the newly generated tokens. at each time step, we calculate this lookback ratio for each attention head, and train a linear classifier, which we call the lookback lens, to detect contextual hallucinations based on the lookback ratio features, as illustrated in figure 1 . the lookback lens performs on par with, and sometimes even surpasses, more complex feature-based detectors that utilize hidden states from llms or textwe extract attention weights and calculate the lookback ratios for all layers and all heads. we train a linear classifier on the concatenated features to predict truthfulness of the generation. ## cross-model transfer one benefit of using the lookback ratio to capture higher-level model patterns for hallucination detection is its potential to better transfer across models. a classifier trained with one model's lookback ratio could potentially be applied to another model without retraining, provided correlation between the target model's attention pattern and that of the original model. here, we show that we can transfer a lookback lens trained on attention maps from llama-2-7b-chat to llama-2-13b-chat without any retraining. since the total numbers of attention heads are different in 7b and 13b models, and there is no obvious one-to-one mapping between the heads, we use a linear regression model to map the heads from the 13b model to the heads in 7b model. concretely, we have 1024 heads in 7b and 1600 heads in 13b. we extract the averaged lookback ratio per head for all the |d| training examples, resulting in a 1024 × |d| matrix and a 1600 × |d| matrix. 2 we then fit a linear regression model to map the heads to reconstruct the 7b heads from 13b heads. after applying the linear transformation to the lookback ratio from 13b, the transformed the detection results are shown in given promising results on detection tasks, we apply cross-model transfer to lookback lens guided decoding. we conduct the same-task transfer setting: nq-train (7b) to nq (13b), and cnn/dm (7b) to xsum (13b). in ## discussions and ablations in this section, we further conduct various experiments and ablation studies on the lookback lens and its corresponding classifier guided decoding. ## conclusion we introduce the lookback lens, a lightweight classifier designed to detect contextual hallucinations by utilizing the lookback ratio, which is computed solely from attention weights. this classifier not only effectively identifies contextual hallucinations but also mitigates them through lookback lens guided decoding from the llm. remarkably, the method is transferable across various tasks, and even across models after mapping their attention heads. this research opens up new possibilities for leveraging attention map information to combat hallucinations in large language models.