On Large Language Models’ Hallucination with Regard to Known Facts (2024)

Abstract: Large language models are successful in answering factoid questions but are also prone to hallucination.We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics, an area not previously covered in studies on hallucinations.We are able to conduct this analysis via two key ideas.First, we identify the factual questions that query the same triplet knowledge but result in different answers. The difference between the model behaviors on the correct and incorrect outputs hence suggests the patterns when hallucinations happen.Second, to measure the pattern, we utilize mappings from the residual streams to vocabulary space.We reveal the different dynamics of the output token probabilities along the depths of layers between the correct and hallucinated cases. In hallucinated cases, the output token’s information rarely demonstrates abrupt increases and consistent superiority in the later stages of the model.Leveraging the dynamic curve as a feature, we build a classifier capable of accurately detecting hallucinatory predictions with an 88% success rate. Our study shed light on understanding the reasons for LLMs’ hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.

## introduction large language models (llms) have shown great potential to acquire extensive knowledge and apply it in various tasks @xcite @xcite . despite their proficiency in generating coherent and contextually relevant text, these models frequently manifest 'factual hallucinations,' significantly undermining their reliability in practical applications @xcite 2023; @xcite . factual hallucination is one of the least noticeable types of erroneous output, as the model often expresses fabricated content with a confident tone. there are two potential sources of factual hallucinations. the first arises from insufficient knowledge within the model's parameters, resulting in inaccurate responses based on a combination of false information. consequently, the model should express uncertainty or refer to external knowledge bases when faced with such limitations. studies propose self-assessment methods for models to rectify this form of hallucination @xcite @xcite . the second scenario occurs when the model's parameters memorize relevant knowledge but lack generalization ability. prompt engineering or prefix tuning can partially alleviate these factual hallucinations, enhancing the model's performance on specific tasks @xcite . however, the mechanism behind the model's hallucination of previously memorized knowledge remains puzzling. it is challenging to ascertain what the model does not know @xcite , but if the model provides the correct answer in response to a specific knowledge query, it can be inferred that the model has memorized relevant information. hence, this study focuses on hallucination for known facts. specifically, under the task of object completion for triplet knowledge, we aim to observe the behavioral characteristics of the model when there is a failure in recalling parameterized knowledge. we define a known fact hallucination as when a model, queried with different prompts for the same knowledge triplet, produces both correct and incorrect outputs. incorrect outputs may include uncertain responses, irrelevant information, or incorrect entities. in this paper, we investigate the dynamic inference characteristics of parameterized factual knowledge recall when llm exhibits known fact hallucinations. to achieve this, we curated an information extraction dataset and filtered specific modelselected data where the accuracy varies for the same knowledge under different queries. subsequently, we compare the inference dynamics under different scenarios so as to understand and identify such hallucinations. our primary findings are as follows: ## related work the process of knowledge recall in the model is intricate. from the perspective of grokking @xcite , information with good generalization tends to occupy more condensed positions within the parameters. prior research on locating and extracting triplet knowledge within the model indirectly supports this notion. when performing linear mappings for relations within specific layers concerning triplet information, fewer than half of the relations achieved satisfactory results @xcite . these relations mainly encompass common knowledge and verbal triplets, which are high-frequency occurrences within the training data. moreover, when pinpointing the attention head for subject-attribute mapping, only 30% of the knowledge could be localized to a single attention head @xcite . consequently, errors occur in the model's lack of generalization regarding known knowledge, involving complex mechanisms in the reasoning process. therefore, gaining access to model's hidden states in a broader perspective during inference process becomes imperative. several related studies aim to dissect internal knowledge extraction mechanisms. causal tracing identifies influential model components during inference @xcite . but the addition of gaussian noise to the input may cause artificial behavior in models, revealing a gap between causal tracing and the intricacies of natural language processing @xcite . probing techniques employ mapping functions to detect model states or representations @xcite , connecting the model's latent space with humanunderstandable representations such as the truthfulness of a statement @xcite @xcite . some work explains the model's behavior at the token level @xcite @xcite . however, these methods do not delve into the knowledge recall dynamics. regarding the exploration of internal model knowledge, numerous studies have analyzed the model from the perspective of residual streams, providing insightful breakdowns of the roles of various modules at each layer and the knowledge extraction process @xcite @xcite @xcite . they highlight two key processes in knowledge extraction: subject enrichment and knowledge extraction. additionally, they delve into the storage function of multi-layer perception layers in knowledge and the information transmission role of attention layers. however, these studies primarily investigate mechanisms when the model successfully recalls knowledge. our study extends their findings, examining the internal mechanisms when the model generates known fact hallucinations. ## experimental setup we focus on the recall process of the object in triplet knowledge (s, r, o). while previous studies have partially deciphered model's successful recall process of triplet knowledge @xcite , our curiosity lies in understanding the inference process during instances of known fact hallucination. what differences are in the dynamic change of hidden states throughout the residual stream comparing successful knowledge recalls and the failed ones? therefore, our experiments collect knowledge query data specifically for this scenario and test them on widely used llama model. then we use various tools to interpret and identify language model's dynamic inference processes when it makes known fact hallucination. ## conclusion we have analyzed the scenario where llms hallucinate on known facts. using our dataset based on triplet knowledge, we make comparative observations of the model's reasoning dynamics across various outputs. we show that the cause of hallucinations lies in the failure of factual recall. the failure may result from a bias in subject parsing, leading to inadequate extraction of object-related information at higher levels. this information then competes with hallucinatory information flow and gets suppressed by mlps. based on the distinct differences observed in the dynamics of reasoning, we train a well-performing classifier to determine the presence of hallucinations in model outputs. leveraging the findings of this study, future work could explore the impact of query formulation on known knowledge recall and methods to mitigate such hallucinations. our discoveries offer a novel perspective on observing knowledge hallucination: viewing the reasoning process of language models as a dynamic system where the internal state variations influence the ultimate output. this dynamic can be observed to analyze and determine the nature of the output. subsequent research could generalize this perspective to broader forms of knowledge outputs and more complex reasoning tasks.