Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models (2024)

Abstract: Hallucinations in large language models (LLMs) refer to the phenomenon of LLMs producing responses that are coherent yet factually inaccurate. This issue undermines the effectiveness of LLMs in practical applications, necessitating research into detecting and mitigating hallucinations of LLMs. Previous studies have mainly concentrated on post-processing techniques for hallucination detection, which tend to be computationally intensive and limited in effectiveness due to their separation from the LLM’s inference process. To overcome these limitations, we introduce MIND, an unsupervised training framework that leverages the internal states of LLMs for real-time hallucination detection without requiring manual annotations. Additionally, we present HELM, a new benchmark for evaluating hallucination detection across multiple LLMs, featuring diverse LLM outputs and the internal states of LLMs during their inference process. Our experiments demonstrate that MIND outperforms existing state-of-the-art methods in hallucination detection.

## introduction in recent years, large language models (llms) have demonstrated remarkable performance in a variety of natural language processing (nlp) applications @xcite @xcite @xcite . however, the widespread adoption of llms has highlighted a critical problem, i.e., hallucination. hallucination refers to the cases where llms generate responses that are logically coherent but factually incorrect or misleading @xcite @xcite . such flaw hurts the effectiveness and robustness of llms in real-world nlp applications, underlining the pressing necessity for research on detecting and mitigating hallucinations in llms. existing studies on hallucination detection for llm mainly focus on how to identify possible factrelated errors in llm's outputs @xcite @xcite . for example, wikibio gpt3 @xcite , a well-known benchmark for hallucination detection, hired human annotators to annotate a couple of true and false response. hallucination detection methods are then evaluated based on their accuracy in predicting the truthfulness of the answers. based on this paradigm, considerable hallucination detection methods have been proposed with the goal of taking a piece of text as input and predicting whether there are hallucinations in the inputs @xcite @xcite . since they detect hallucinations after llms finish the inference process, we refer to these hallucination detection methods as postprocessing methods. unfortunately, the post-processing methods widely used in existing studies are suboptimal for the applications of hallucination detection models for llms in practice. first, existing postprocessing methods often suffer from extreme computation costs and high latency. in order to identify hallucinations in input text without ground truth references (otherwise the task would downgrade to a simple fact verification task), hallucination detection models need to be powerful and knowledgeable on their own. sota detection methods are often implemented with llms (e.g., chat-gpt, llama, opt) directly @xcite @xcite , which makes the cost of hallucination detection on par or even larger than the inference process of many llms. second, post-processing methods are intrinsically limited in model capacities. as post-processing methods detect hallucinations independently with the inference process of llms, they can't analyze how hallucinations are generated from scratches in each llm. some studies tried to bypass this issue through the construction of proxy models @xcite . however, such proxy models must be trained with extensive human annotations, otherwise they cannot capture the unique characteristics of each llm. such manual annotation data are expensive to collect and not preferable considering the rapid developments of llm techniques. to address these limitations, we propose a novel reference-free, unsupervised training framework mind, i.e., unsupervised modeling of internal states for hallucination detection of large language models. we highlight mind with the following advantages: (1) unsupervised. in contrast to previous works, mind is an unsupervised framework that directly extracts pseudo-training data from wikipedia. it doesn't require any manual annotation for the training of the hallucination detector. (2) real-time. compared with the existing post-processing methods, mind is a realtime hallucination detection framework designed to reduce computational overhead and detection latency. with a simple multi-layer perceptron model built upon the contextual embeddings of each token in llm's inference process, mind can conduct the hallucination detection in a real-time process. further, to facilitate future research and to improve the reproducibility of this paper, we introduce a new benchmark for llm hallucination detection named helm: hallucination detection evaluation for multiple llms. in contrast to previous benchmarks that only provide the text generated based on hand-crafted heuristics @xcite or a single llm @xcite , helm provides not only the texts produced by six different llms (together with human-annotated hallucination labels) but also the contextualized embeddings, self-attentions and hidden-layer activations recorded in the inference process of each llm, which could serve as the snapshots of each llm's internal states during their inference process. in summary, the contributions of our paper are as follows: ## problem formulation hallucination detection can be defined as a binary classification problem. the objective is to judge whether the given output from an llm is a hallucination (false or misleading information) or a non-hallucination (accurate and relevant information). this diverges from traditional fact verification tasks. in contrast to traditional fact verification tasks that mainly assess the factual accuracy of text, hallucination detection in llms goes beyond simply evaluating the truthfulness of the information. it focuses more on the comprehensive analysis of the characteristics intrinsic to the generative models. examples of hallucination detection methods include the consistency of multi-responses to the same question @xcite , the llms' confidence in its generated content @xcite , the internal states of llms during the text generation process @xcite , etc. ## methodology in this section, we introduce the details of our proposed framework mind, i.e., unsupervised modeling of internal-states for hallucination detection of large language models. the mind framework consist of two steps: automatic training data generation and hallucination classifier training. ## the helm benchmark in this section, we introduce the details of our proposed new benchmark helm: hallucination detection evaluation for multiple llms. ## experimental results we conduct experiments to verify our our proposed hallucination detection method: mind. specifically, this section studies the following research questions (rq): ## conclusions and future works in this paper, we introduce mind, a novel unsupervised approach leveraging the internal states of large language models (llms) for real-time hallucination detection. moreover, we propose helm, a comprehensive benchmark for hallucination detection, incorporating outputs from six diverse llms along with their internal states during text generation.