LLM Factoscope: Uncovering LLM s’ Factual Discernment through Measuring Inner States (2024)

Abstract: Large Language Models (LLMs) have revolutionized various domains with extensive knowledge and creative capabilities. However, a critical issue with LLMs is their tendency to produce outputs that diverge from factual reality. This phenomenon is particularly concerning in sensitive applications such as medical consultation and legal advice, where accuracy is paramount. Inspired by human lie detectors using physiological responses, we introduce the LLM Factoscope, a novel Siamese network-based model that leverages the inner states of LLMs for factual detection. Our investigation reveals distinguishable patterns in LLMs’ inner states when generating factual versus non-factual content. We demonstrate its effectiveness across various architectures, achieving over 96% accuracy on our custom-collected factual detection dataset. Our work opens a new avenue for utilizing LLMs’ inner states for factual detection and encourages further exploration into LLMs’ inner workings for enhanced reliability and transparency.

## introduction large language models (llms) have gained immense popularity, revolutionizing various domains with their remarkable creative capabilities and vast knowledge repositories. these models reshape fields like natural language processing @xcite , content generation @xcite , and more. however, despite their advanced abilities, a growing concern surrounds their propensity for "hallucination" -the generation of outputs that deviate from factual reality @xcite . in critical applications like medical consultation @xcite , legal advice @xcite , and educational tutoring @xcite , factual llm outputs are not just desirable but essential, as non-factual outputs from these models could potentially lead to negative outcomes for users, affecting their health, legal standing, or educational understanding. recognizing this, llm-generated content's factual detection has emerged as an area of paramount importance @xcite . current research predominantly relies on cross-referencing llm outputs with external databases @xcite . while effective, this approach necessitates extensive external knowledge bases and sophisticated cross-referencing algorithms, introducing more complexity and dependency. this raises a compelling question: could we possibly exclude external resources but only leverage the inner states of llms for factual detection? inspired by human lie detectors, we explore establishing a lie detector for llms. humans show specific physiological changes when making statements that contradict their knowledge or beliefs, enabling lie detection through these changes. we assume that exposure to a wide range of world knowledge during llms' training establishes their knowledge or beliefs. moreover, training datasets usually have more factual than non-factual sources @xcite . this assumption makes establishing a lie detector for llms possible. to achieve this, we need to address two essential questions: (1) what might the "physiological" indicators for llms be, similar to heart rate and eye movements in humans? (2) how can we leverage these indicators in llms to establish the lie detector? however, the lack of interpretability in llms makes it impossible for humans to understand model parameters or hidden states, rendering the solutions to these questions challenging. we address the above two challenges through heuristic and data-driven methods, respectively. for the first challenge, we initially use activation maps as static features to represent the "physiological" states of the llm. based on our observations, factual-related activations show higher intensity when the llm produces factual outputs, making the activation maps representative to serve as fea-tures for the lie detector. subsequently, we use final output ranks, top-k output indices, and top-k output probabilities across layers as dynamic features to represent the llm's decision-making processes. we note that factual outputs by llms show more stable final output ranks, greater semantic similarity in top-k output indices, and larger top-k output probability differences in later layers (see section 3). for the second challenge, due to the intricate relationship between the features and the factuality of outputs, we employ a data-driven approach to extract underlying principles from these features. consequently, we design a systematic and automated factual detection pipeline, llm factoscope, that includes the factual datasets collection, the inner states collection, and the design of the factual detection model (see section 4). in our experiments, we empirically demonstrate the effectiveness of the llm factoscope across various llm architectures, including gpt2-xl-1.5b, llama2-7b, vicuna-7b, stablelm-7b, llama2-13b, and vicuna-13b. the llm factoscope achieves an accuracy rate exceeding 96% in factual detection. additionally, we extensively examine the model's generalization capabilities and conduct ablation studies to understand the impact of different components and parameters on the llm factoscope's performance. our work paves a new path for utilizing inner states from llms for factual detection, sparking further exploration and analysis of llms' inner states for enhanced model understanding and reliability. our contributions are as follows: (1) we identify effective inner states closely associated with the factual accuracy of the content generated by llms, discovered through observation. (2) we introduce the llm factoscope, a novel and extendable pipeline for automating the detection of factual accuracy in llm outputs, facilitating the incorporation of new datasets with ease. (3) our model has been empirically validated across various llm architectures, demonstrating over 96% accuracy on our custom-collected factual detection dataset. datasets and code are released for further research: https://github.com/jenniferho97/llm_factoscope . ## observations we present an analysis from both static and dynamic perspectives. statically, we examine activation maps, while dynamically, we use the unembedding matrix to decode and get the probability distribution over the vocabulary humans understand. at the static level, we focus on activation maps. figure 1 shows the top 1% neurons with the highest activations when the gpt2-xl-1.5b model outputs factual information, resulting in 64 neurons per layer from its 6,400 neurons per layer. specifically, we test with a labeled factual dataset, sorting the activation values of factual outputs in descending order and retaining the locations of the top 1% neurons with the highest activations for each data. we calculate the frequency of these neurons and then select those with the highest frequency of appearing in the top 1% activations. these neurons, which are most frequently highly activated when the llm outputs factual information, are displayed as factual-related neurons. figure 1(a) demonstrates that when the llm provides factual information, the factual-related neurons activate more intensely. this increased activation is likely due to the strengthening of these connections from training on factual data. conversely, figure 1(b) shows a case of non-factual information with less activation in factual-related neurons. this difference suggests that activation maps can potentially serve as indicators for factual accuracy assessment. from the dynamic perspective, we observe the evolution of the final output ranks across layers, depicted in figure 2 . for instance, when the llm is queried about the director of "the shining" and initially outputs "stanley," we observe its rank across layers, where rank 1 represents the highest probability. in factual outputs, the final output rank ascends to a higher position after layer 22, showing the llm's growing confidence to finalize its output in the later stages of the network. in contrast, non-factual outputs show rank fluctuations after layer 35, reflecting the model's uncertainty. further, we observe the top-k output indices and ## llm factoscope in this section, we begin with an overview of our pipeline. then, we introduce the data collection for factual detection and preprocessing steps for effective model training. lastly, we present the model architecture of the llm factoscope. ## conclusion we discover effective inner states and provide observation for llm factual detection. then, we develop a factual detection pipeline, the llm factoscope, which leverages these inner states to detect the factuality of llm outputs. the llm factoscope consistently demonstrated high factual detection accuracy, surpassing 96% in our customcollected datasets. our research not only provides a novel method for llms' factual detection but also opens new avenues for future explorations into the llms' inner states. by paving the way for enhanced model understanding and reliability, the llm factoscope sets a foundation for more transparent, accountable, and trustworthy use of llms in critical applications. 9 limitations. our research assumes llms' training corpora are predominantly factual, despite the potential inclusion of some non-factual content. our pipeline remains effective as long as factual outweighs non-factual data. when creating our dataset, we generate various prompts and responses to simulate factual inquiries. despite our efforts, the diversity and complexity of real-world knowledge may not be fully represented in our custom dataset. however, owners of llms providing factual detection services likely have access to more comprehensive and meticulously vetted datasets. such resources could potentially enhance the performance and accuracy of llm factoscope. the llm factoscope shows generalizability in our evaluations, effectively detecting factuality across various contexts. however, it may encounter challenges when dealing with data types that significantly deviate from those in our training dataset. following our pipeline to fine-tune the model with new, more representative datasets could further refine its accuracy and broaden its applicability.