Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language Models (2024)

Abstract: Large Language Models (LLMs) have shown impressive capabilities but still suffer from the issue of hallucinations. A significant type of this issue is the false premise hallucination, which we define as the phenomenon when LLMs generate hallucinated text when confronted with false premise questions. In this paper, we perform a comprehensive analysis of the false premise hallucination and elucidate its internal working mechanism: a small subset of attention heads (which we designate as false premise heads) disturb the knowledge extraction process, leading to the occurrence of false premise hallucination. Based on our analysis, we propose FAITH ( F alse premise A ttention head constra I ining for mi T igating H allucinations), a novel and effective method to mitigate false premise hallucinations. It constrains the false premise attention heads during the model inference process. Impressively, extensive experiments demonstrate that constraining only approximately 1% of the attention heads in the model yields a notable increase of nearly 20% of model performance.

## introduction large language models (llms) have shown impressive capabilities @xcite @xcite and achieved remarkable success in many tasks @xcite @xcite . however, they often generate texts that are seemingly plausible but deviate from factual knowledge sources @xcite , which is a severe problem known as hallucination. to address this, many studies focus on detecting @xcite and mitigating hallucinations @xcite @xcite . however, few of them pay attention to a particular type of hallucination: false premise hallucination. we define the question contains the false premise that "albert einstein was awarded the nobel prize of physics in 1920" whereas in fact he was awarded the prize in 1921. we find that the presence of false premise attention heads contributes to the hallucinated response. our method can effectively mitigate the false premise hallucination. in this paper, we conduct a comprehensive analysis of false premise hallucination and elucidate its internal working mechanism. firstly, prior to the analysis, we propose an automatic dataset construction pipeline for the evaluation of false premise hallucination and create two representative and easyto-evaluate datasets based on it. then, we investigate the external manifestation of false premise hallucination and observe that llms exhibit more inherent uncertainty when generating hallucinated answers. subsequently, to reveal the source of the uncertainty, we delve into the internal information flow during the hallucination occurrences. we discover that knowledge about the subject stored in model parameters is disturbed in the shallow layers of the model, particularly around the false object mentioned in the question. furthermore, as many studies @xcite indicate that self-attention layers transfer the factual knowledge stored in the multi-layer perception (mlp) layers, we explore the internals of the self-attention layers and investigate the influence of each individual attention head on specific pieces of factual knowledge. we find out that a small set of attention heads consistently exert great influence on the factual knowledge across almost all the samples and we name them as false premise heads. as depicted in figure 1 , the false premise heads predominantly reside in the shallow layers, functioning around the false object mentioned in the question. experimental results demonstrate that the presence of false premise heads disturb the extraction of the factual knowledge about the subject, leading to false premise hallucinations. based on our analysis, we propose a novel method termed faith (false premise attention head constraining for mitigating hallucinations) to mitigate hallucinations. it localizes the false premise attention heads for a group of false premise questions and subsequently constrains the function of these attention heads during the model inference process. extensive experiments demonstrate the effectiveness of our method comparing with the baseline methods. our primary contributions can be summarized as follows: in this section, we briefly describe the transformer architecture @xcite in autoregressive, decoder-only language models from the perspective of residual stream @xcite . given the context {t 1 , t 2 , ..., t n } consisting of n tokens, the transformer architecture starts with a combination of token embeddings and position embeddings x 0 ∈ r n ×d , where d is the model dimension. it marks the start of the residual stream, with a series of residual layers that read from the stream and write back their processed results. each residual layer is comprised of a self-attention layer and a mlp layer. the information update of each residual layer can be expressed as: ## dataset construction in this section, we describe our proposed automatic dataset construction pipeline for the evaluation of false premise hallucination and provide the details of our constructed dataset. to prevent the memorization of the question @xcite and facilitate the incorporation of the evolving new knowledge, we propose an automatic dataset construction pipeline, which can be divided into the following three stages: ## hallucination analyze in this section, we conduct a comprehensive analysis of false premise hallucinations from the surface and delves deeper into the model step by step. ## hallucination mitigation based on our previous in-depth analysis, in this section, we introduce faith, a novel method aimed at mitigating false premise hallucinations. ## conclusion in this paper, we conduct a comprehensive analysis of an important type of hallucination: false premise hallucination. our analysis begins at the surface of the model and gradually delves deeper into it, ultimately revealing the presence of false premise attention heads. based on our analysis, we propose a novel false premise hallucination mitigation method, faith (false premise attention head constraining for mitigating hallucinations). extensive experiments demonstrate the effectiveness of our method comparing with the baselines and the promising nature of our revealed internal working mechanism of false premise hallucination.