I nternal I nspector I^2 : Robust Confidence Estimation in LLM s through Internal States (2024)

Abstract: Despite their vast capabilities, Large Language Models (LLMs) often struggle with generating reliable outputs, frequently producing high-confidence inaccuracies known as hallucinations. Addressing this challenge, our research introduces InternalInspector, a novel framework designed to enhance confidence estimation in LLMs by leveraging contrastive learning on internal states including attention states, feed-forward states, and activation states of all layers. Unlike existing methods that primarily focus on the final activation state, InternalInspector conducts a comprehensive analysis across all internal states of every layer to accurately identify both correct and incorrect prediction processes. By benchmarking InternalInspector against existing confidence estimation methods across various natural language understanding and generation tasks, including factual question answering, commonsense reasoning, and reading comprehension, InternalInspector achieves significantly higher accuracy in aligning the estimated confidence scores with the correctness of the LLM’s predictions and lower calibration error. Furthermore, InternalInspector excels at HaluEval, a hallucination detection benchmark, outperforming other internal-based confidence estimation methods in this task.

## introduction large language models (llms) have demonstrated remarkable capabilities across a wide range of tasks, from reasoning to question answering @xcite @xcite . despite these advancements, llms still face significant challenges in hallucinating facts @xcite @xcite and providing robust confidence estimates for their predictions @xcite @xcite 1. this results in llms delivering confident but incorrect information, undermining their reliability and affecting their potential applications in real-world scenarios. therefore, a well-established confidence estimator is essential for users to determine when to trust the outputs of llms and identify hallucinations in the outputs, thereby enhancing the practicality and trustworthiness of llms. recent research suggests that llms exhibit a degree of self-awareness regarding the truthfulness of the generated statements @xcite . studies have investigated the relationship between the accuracy of llm outputs and the characteristics of their final activation state @xcite . although these findings are promising, they primarily focus on easy true/false factual question-answering tasks. furthermore, relying solely on the final hidden states offers a limited perspective on the intricate internal dynamics of llms. the generation process within llms involves a sophisticated interplay of internal modules, including attention mechanisms and feed-forward networks (ffns), which are critical for shaping the generated responses. recent studies have demonstrated that these internal modules play a crucial role in encoding and recalling the factual and linguistic knowledge essential for accurate predictions in llms @xcite @xcite @xcite . it has also been shown that hallucinations in llm outputs primarily originate from these internal modules @xcite . these studies further lead us to the question: can the dynamics of these internal modules potentially indicate the confidence of llms in their predictions? in response, we first provide a theoretical foundation that highlights the crucial role of internal states in robust and accurate confidence estimation. we further propose internalinspector (i 2 ), a simple yet robust confidence estimation method that leverages the internal states of llms to assess the truthfulness of generated statements across various tasks. specifically, internalinspector employs contrastive learning @xcite upon an encoder, such as a convolutional neural network (o'shea and nash, 2015) or a transformer @xcite , to learn meaningful feature representations from the internal states of an llm such as llama-2-7b @xcite . a binary classifier is trained simultaneously on top of these feature representations to estimate a confidence score for each llm prediction based on its correctness, either correct or incorrect. we evaluate internalinspector together with various existing confidence estimation methods, including logit-based, self-evaluation, and other internal-based approaches on several natural language understanding and generation tasks, including factual question answering, commonsense reasoning, and reading comprehension. experimental results demonstrate that internalinspector significantly enhances the alignment between accuracy and confidence scores, achieving up to 20.4% improvement in accuracy and 8.9% in expected calibration error (ece) across the various evaluation tasks. furthermore, internalinspector excels at identifying hallucinations in generated outputs, notably existing internal-based confidence estimation methods on the halueval benchmark @xcite . we also investigate the importance of different types of internal states in confidence estimation and showcase that attention states are particularly meaningful for tasks that require deep contextual understanding, such as open-book question answering and reading comprehension, while feed-forward states are more crucial for tasks centered on factual information, aligning with the recent research finding that the feed-forward networks (ffns) within transformer blocks are functioning as keyvalue memories to encode and retrieve factual and semantic knowledge @xcite . in summary, our contributions are as follows: ## related work confidence estimation for llms we summarize existing confidence estimation methods for llms into four categories: (1) logit-based methods @xcite @xcite utilize output probability distributions or entropy to directly measure confidence. however, they mainly reflect the probability distribution over possible tokens (vocabulary space) @xcite @xcite . probing probing utilizes linear classifiers as tools to analyze and understand the representations within the intermediate layers of neural networks @xcite . building upon the foundational concept of probing with linear classifiers, we advances this methodology by utilizing contrastive learning to examine the different internal states of llms. this approach allows us to investigate the model's confidence about its predictions, scrutinizing attention, feed-forward, and activation states across all layers, providing a more comprehensive view of how internal representations evolve and interact to influence the overall task performance of the model. ## conclusion in this work, we propose internalinspector, a simple yet robust confidence estimation method utilizing the internal states of llms, including attention, feed-forward, and activation states across all layers. experimental results underscore internalinspector's superior performance, which consistently outperforms baseline methods in a variety of natural language processing tasks, including factual question answering, commonsense reasoning, and reading comprehension. further analysis shows that internalinspectordemonstrates strong generalization capabilities within intra-domain scenarios. additionally, internalinspector outperforms other internal-state-based confidence estimation methods in halueval, suggesting its potential in hallucination detection.