LLM Internal States Reveal Hallucination Risk Faced With a Query (2024)

Abstract: The hallucination problem of Large Language Models (LLMs) significantly limits their reliability and trustworthiness. Humans have a self-awareness process that allows us to recognize what we don’t know when faced with queries. Inspired by this, our paper investigates whether LLMs can estimate their own hallucination risk before response generation. We analyze the internal mechanisms of LLMs broadly both in terms of training data sources and across 15 diverse Natural Language Generation (NLG) tasks, spanning over 700 datasets. Our empirical analysis reveals two key insights: (1) LLM internal states indicate whether they have seen the query in training data or not; and (2) LLM internal states show they are likely to hallucinate or not regarding the query. Our study explores particular neurons, activation layers, and tokens that play a crucial role in the LLM perception of uncertainty and hallucination risk. By a probing estimator, we leverage LLM self-assessment, achieving an average hallucination estimation accuracy of 84.32% at run time.

## introduction humans have an awareness of the scope and limit of their own knowledge @xcite @xcite , as illustrated in fig. 1 . this cognitive self-awareness ability in humans introduces hesitation in us before we respond to queries or make decisions in scenarios where we know we don't know @xcite @xcite . however, llm-based ai assistants lack this cognitive uncertainty estimation. consequently, they tend to be overconfident and may produce plausible-sounding but unfaithful or nonsensical contents called hallucination or confabulation @xcite @xcite . this problem limits their applications in numerous real-world scenarios and undermines user trustworthiness. previous research @xcite @xcite have explored the internal states of language models that capture contextual and semantic information learned from training data @xcite @xcite . nevertheless, internal states of language models sometimes exhibit limited generalization on unseen data and their representation effectiveness can be undermined by flawed training data or modeling issues @xcite @xcite @xcite . notably, recent works have shown that the llm's internal states can potentially detect hallucinations in texts @xcite @xcite . however, these works examine texts not exclusively produced by the same llms whose internal states are analyzed, highlighting the necessity for further investigation into the llm self-awareness and how their internal states correlate with their uncertainty and own hallucination occurrence. our work takes a step further by investigating whether llm internal states have some indication of hallucination risk given queries and whether it can be reliably estimated even before the actual response generation (fig. 1 ). we conduct a comprehensive analysis of llms internal mechanisms in terms of training data sources and across 15 diverse nlg tasks that extend beyond the qa task @xcite and span over 700 datasets. we explore particular neurons, different activation layers, and tokens that play a crucial role in the llm perception of uncertainty and hallucination risk. employing a probing estimator @xcite on the internal states associated with the queries, we validate their self-awareness and ability to indicate uncertainty in two aspects: (1) whether they have seen the query in training data, achieving an accuracy of 80.28%. (2) whether they are likely to hallucinate regarding the query, achieving an average estimation accuracy of 84.32% across 15 nlg tasks. we propose that understanding these representations could offer a proactive approach to estimating uncertainty, potentially serving as an early indicator for the necessity of retrieval augmentation @xcite or as an early warning system. ## hallucination and training data the sources of hallucination in llms can be traced back to data and modeling @xcite . factors tied to data encompass unseen knowledge, task-specific innate divergence, noisy training data, etc. from the modeling perspective, hallucinations can be traced to the model architecture, alignment tax, teacher-forced maximum likelihood estimation (mle) training, etc. a common situation where hallucinations from data occur is when llms attempt to provide information on unseen queries that are not included in their training set, rather than refusing to reply. previous works @xcite @xcite explore to identify the unseen data based on various indicators such as text similarity, perplexity. we investigate the capability of llms to recognize whether they have seen the query in training data via novel analysis of their internal states. to facilitate analysis, we craft two sets of queries by collecting news from periods before and after the release of the llm we analyze to represent unseen and seen data, respectively. however, in the real-world scenario, it's impractical to definitively categorize data as entirely seen or unseen due to the inability to access the vast training data of llm. thus, we expand the preliminary insights and further investigate llms' self-awareness of recognizing whether models are likely to hallucinate regarding the query. it's important to note that the hallucinations are sourceagnostic, meaning they can result from both unseen and seen data. the latter can still trigger hallucinations due to deficiencies in modeling. to facilitate analysis, we construct data by using llm to directly generate responses to queries across diverse nlg tasks and then label the hallucination level in the responses. ## methodology this section begins with an introduction to the problem formulation of uncertainty estimation faced with queries in § 3.1. we construct datasets in § 3.2 focusing on two dimensions: (1) the distinction between queries seen and unseen in the training data; (2) the likelihood of hallucination risk faced with the queries. to validate the efficacy of internal state representation in hallucination estimation, we visualize the neurons for perception extracted from a specified llm layer ( § 3.3) and then leverage the probing classifier technique @xcite on top of internal states associated with the last token of queries ( § 3.4). ## conclusion inspired by human self-awareness, this work demonstrates the latent capacity of llms to selfassess and estimate hallucination risks prior to response generation. we conduct a comprehensive analysis of the internal states of llms both in terms of training data sources and across 15 nlg tasks with over 700 datasets. employing a probing estimator on the internal states associated with the queries, we assess their self-awareness and ability to indicate uncertainty in two aspects: (1) recognizing whether they have seen the query in training data, achieving an accuracy of 80.28% foot_3 . (2) recognizing whether they are likely to hallucinate when faced with the query. the results demonstrate that internal state-based self-assessment outperforms ppl-based and prompt-based baselines, with an average estimation accuracy of 84.32% across all tested datasets. in addition, we explore the role of particular neurons in uncertainty and hallucination perception and reveal a positive correlation between the depth of activation layers in an llm and its predictive accuracy. the consistency of internal states across different models suggests a potential for zero-shot transfer, but model-specific estimation is the optimal strategy. challenges of generalizing these findings across different tasks are noted, despite observing promising generalizations within the same nlg tasks. for future work, we aim to refine our methodology to enhance the robustness and generalization across various nlg tasks in the field of hallucination risk assessment. in addition, we will involve a broader spectrum of llms to extend the applicability of our findings.