The Internal State of an LLM Knows When It’s Lying (2023)

Abstract: While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. In this paper, we provide evidence that the LLM’s internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates. Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71% to 83% accuracy labeling which sentences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier’s performance and approaches based on the probability assigned to the sentence by the LLM. We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.

## introduction large language models (llms) have recently demonstrated remarkable success in a broad range of tasks @xcite @xcite . however, when composing a response, llms tend to hallucinate facts and provide inaccurate information @xcite . furthermore, they seem to provide this incorrect information using confident and compelling language. the combination of a broad body of knowledge, along with the provision pluto is the second largest smallest dwarf planet in our solar system. celestial body in the solar system that has ever been classified as a planet. only most figure 1: a tree diagram that demonstrates how generating words one at a time and committing to them may result in generating inaccurate information. we believe that in order to perform well, an llm must have some internal notion as to whether a sentence is true or false, as this information is required for generating (or predicting) following tokens. for example, consider an llm generating the following false information "the sun orbits the earth." after stating this incorrect fact, the llm is more likely to attempt to correct itself by saying that this is a misconception from the past. but after stating a true fact, for example "the earth orbits the sun," it is more likely to focus on other planets that orbit the sun. therefore, we hypothesize that the truth or falsehood of a statement should be represented by, and therefore extractable from, the llm's internal state. interestingly, retrospectively "understanding" that a statement that an llm has just generated is false does not entail that the llm will not generate it in the first place. we identify three reasons for such behavior. the first reason is that an llm generates a token at a time, and it "commits" to each token generated. therefore, even if maximizing the likelihood of each token given the previous tokens, the overall likelihood of the complete statement may be low. for example, consider a statement about pluto. the statement begins with the common words "pluto is the", then, since pluto used to be the smallest planet the word "smallest" may be a very plausible choice. once the sentence is "pluto is the smallest", completing it correctly is very challenging, and when prompted to complete the sentence, gpt-4 (march 23rd version) completes it incorrectly: "pluto is the smallest dwarf planet in our solar system." in fact, pluto is the second largest dwarf planet in our solar system (after eris). one plausible completion of the sentence correctly is "pluto is the smallest celestial body in the solar system that has ever been classified as a planet." (see figure 1 ). consider the following additional example: "tiztoutine is a town in africa located in the republic of niger." indeed, tiztoutine is a town in africa, and many countries' names in africa begin with "the republic of". however, tiztoutine is located in morocco, which is not a republic, so once the llm commits to "the republic of", it cannot complete the sentence using "morocco", but completes it with "niger". in addition, committing to a word at a time may lead the llm to be required to complete a sentence that it simply does not know how to complete. for example, when describing a city, it may predict that the next words should describe the city's population. therefore, it may include in a sentence "tiztoutine has a population of", but the population size is not present in the dataset, so it must complete the sentence with a pure guess. the second reason for an llm to provide false information is that at times, there may be many ways to complete a sentence correctly, but fewer ways to complete it incorrectly. therefore, it might be that a single incorrect completion may have a higher likelihood than any of the correct completions (when considered separately). finally, since it is common for an llm to not use the maximal probability for the next word, but to sample according to the distribution over the words, it may sample words that result in false information. in this paper we present our statement accuracy prediction, based on language model activations (saplma). saplma is a simple yet powerful method for detecting whether a statement generated by an llm is truthful or not. namely, we build a classifier that receives as input the activation values of the hidden layers of an llm. the classifier determines for each statement generated by the llm if it is true or false. importantly, the classifier is trained on out-of-distribution data, which allows us to focus specifically on whether the llm has an internal representation of a statement being true or false, regardless of the statement's topic. in order to train saplma we created a dataset of true and false statements from 6 different topics. each statement is fed to the llm, and its hidden layers' values are recorded. the classifier is then trained to predict whether a statement is true or false only based on the hidden layer's values. importantly, our classifier is not tested on the topics it is trained, but on a held-out topic. we believe that this is an important measure, as it requires saplma to extract the llm's internal belief, rather than learning how information must be aligned to be classified as true. we show that saplma, which leverages the llm's internal states, results in a better performance than prompting the llm to explicitly state whether a statement is true or false. specifically, saplma reaches accuracy levels of between 60% to 80% on specific topics, while few-shot prompting achieves only slightly above random performance, with no more than a 56% accuracy level. of course there will be some relationship between the truth/falsehood of a sentence, and the probability assigned to that sentence by a welltrained llm. but the probability assigned by an llm to a given statement depends heavily on the frequency of the tokens in the statement as well as its length. therefore, sentence probabilities provide only a weak signal of the truth/falsehood of the sentence. at minimum, they must be normalized to become a useful signal of the veracity of a statement. as we discuss later, saplma classifications of truth/falsehood significantly outperform simple sentence probability. in one test described later in more detail, we show that saplma performs well on a set of llm-generated sentences that contain 50% true, and 50% false statements. we further discuss the relationship between statement probability and veracity in the discussion section. saplma employs a simple and relatively shallow feedforward neural network as its classifier, which requires very little computational power at inference time. therefore, it can be computed alongside the llm's output. we propose for saplma to supplement an llm presenting information to users. if saplma detects that the llm "believes" that a statement that it has just generated is false, the llm can mark it as such. this could raise human trust in the llm responses. alternatively, the llm may merely delete the incorrect statement and generate a new one instead. to summarize, the contribution of this paper is twofold. in this section we provide an overview of previous research on llm hallucination, accuracy, and methods for detecting false information, and we discuss datasets used to that end. many works have focused on hallucination in machine translation @xcite . for example, dale et al. @xcite consider hallucinations as translations that are detached from the source, hence they propose a method that evaluates the percentage of the source contribution to a generated translation. if this contribution is low, they assume that the translation is detached from the source and is thus considered to be hallucinated. their method improves detection accuracy hallucinations. the authors also propose to use multilingual embeddings, and compare the similarity between the embeddings of the source sentence and the target sentence. if this similarity is low, the target sentence is considered to be hallucinated. the authors show that the latter method works better. however, their approach is very different than ours, as we do not assume any pair of source and target sentences. in addition, while we also use the internal states of the model, we do so by using the hidden states to descriminate between statements that the llm "believes" are true and those that are false. furtheremore, we focus on detecting false statements rather than hallucination, as defined by their work. other works have focused on hallucination in text summarization @xcite . @xcite propose a benchmark for factuality metrics of text summarization. their benchmark is developed by gathering summaries from several summarization methods and requested humans to annotate their errors. the authors analyze these anotation and the proportion of the different factual error of the summarization methods. we note that most works that consider hallucination do so with relation to a given input (e.g., a passage) that the model operates on. for example, a summarization model that outputs information that does not appear in the provided article, is considered hallucinate it, regardless if the information is factually true. however, in this work we consider a different problem--the veracity of the output of an llm, without respect to a specific input. some methods for reducing hallucination assume that the llm is a black box @xcite . this approach uses different methods for prompting the llm, possibly by posting multiple queries for achieving better performance. some methods that can be used for detecting false statements may include repeated queries and measuring the discrepancy among them. we note that instead of asking the llm to answer the same query multiple times, it is possible to request the llm to rephrase the query (without changing its meaning or any possible answer) and then asking it to answer each rephrased question. other methods finetune the llm, using human feedback, reinforcement learning, or both @xcite . @xcite propose a method to improve llm-generated content using reinforcement learning from human feedback. their approach focuses on fine-tuning the llm with a reward model based on human judgments, aiming to encourage the generation of better content. however, fine tuning, in general, may cause a model to not perform as well on other tasks @xcite . in this paper, we take an intermediate approach, that is, we assume access to the model parameters, but do not fine-tune or modify them. another approach that can be applied to our settings is presented by @xcite , named contrast-consistent search (ccs). however, ccs requires rephrasing a statement into a question, evaluating the llm on two different version of the prompt, and requires training data from the same dataset (topic) as the test set. these limitations render it unsuitable for running in practice on statements generated by an llm. in addition, ccs increases the accuracy by only approximately 4% over the 0-shot llm query, while our approach demonstrates a nearly 20% increase over the 0-shot llm a dataset commonly used for training and finetuning llms is the wizard-of-wikipedia @xcite . the wizard-of-wikipedia dataset includes interactions between a human apprentice and a human wizard. the human wizard receives relevant wikipedia articles, which should be used to select a relevant sentence and compose the response. the goal is to replace the wizard with a learned agent (such as an llm). another highly relevant dataset is fever @xcite . the fever dataset is designed for developing models that receive as input a claim and a passage, and must determine whether the passage supports the claim, refutes it, or does not provide enough information to support or refute it. while the fever dataset is highly relevant, it does not provide simple sentence that are clearly true or false independently of a provided passage. in addition, the fever dataset is not partitioned into different topics as the true-false dataset provided in this paper. in conclusion, while several approaches have been proposed to address the problem of hallucination and inaccuracy in automatically generated content, our work is unique in its focus on utilizing the llm's hidden layer activations to determine the veracity of generated statements. our method offers the potential for more general applicability in real-world scenarios, operating alongside an llm, without the need for fine-tuning or task-specific modifications. ## the true-false dataset the work presented in this paper requires a dataset of true and false statements. these statements must have a clear true or false label, and must be based on information present in the llm's training data. furthermore, since our approach intends to reveal that the hidden states of an llm have a notion of a statement being true or false, the dataset must cover several disjoint topics, such that a classifier can be trained on the llm's activations of some topics while being tested on another. unfortunately, we could not find any such dataset and therefore, compose the true-false dataset. our true-false dataset covers the following topics: "cities", "inventions", "chemical elements", "animals", "companies", and "scientific facts". for the first 5 topics, we used the following method to compose the dataset. we used a reliable source foot_0 that included a table with several properties for each instance. for example, for the "chemical elements" we used a table that included, for each element, its name, atomic number, symbol, standard state, group block, and a unique property (e.g., hydrogen, 1, h, gas, nonmetal, the most abundant element in the universe). for each element we composed true statement using the element name and one of its properties (e.g., "the atomic number of hydrogen is 1"). then, we randomly selected a different row for composing a false statement (e.g., "the atomic number of hydrogen is 34"). if the value in the different row is identical to the value in the current row, we resample a different row until we obtain a value that is different. this process was repeated for the all topics except the "scientific facts". for the "scientific facts" topic, we asked chatgpt (feb 13 version) to provide "scientific facts that are well known to humans" (e.g. "the sky is often cloudy when it's going to rain"). we then asked chatgpt to provide the opposite of each statement such that it becomes a false statement (e.g., "the sky is often clear when it's going to rain"). the statements provided by chatgpt were manually curated, and verified by two human annotators. the classification of 48 facts were questioned by at least one of the annotators; these facts were removed from the dataset. the truefalse dataset comprises 6,084 sentences, including 1,458 sentences for "cities", 876 for "inventions", 930 for "chemical elements", 1,008 for "animals", 1,200 for "companies", and 612 for "scientific facts". the following are some examples of true statements from the dataset: ## saplma in this section, we present our statement accuracy prediction, based on language model activations (saplma), a method designed to determine the truthfulness of statements generated by an llm using the values in its hidden layers. our general hypothesis is that the values in the hidden layers of an llm contain information on whether the llm "believes" that a statement is true or false. however, it is unclear which layer should be the best candidate for retaining such information. while the last hidden layer should contain such information, it is primarily focused on generating the next token. conversely, layers closer to the input are likely focused on extracting lower-level information from the input. therefore, we use several hidden layers as candidates. we use two different llms: facebook opt-6.7b @xcite and llama2 @xcite ; both composed of 32 layers. for each llm, we compose five different models, each using activations from a different layer. namely, we use the last hidden layer, the 28th layer (which is the 4th before last), the 24th layer (which is the 8th before last), the 20th layer (which is the 12th before last), and the middle layer (which is the 16th layer). we note that each layer is composed of 4096 hidden units. saplma's classifier employs a feedforward neural network, featuring three hidden layers with decreasing numbers of hidden units (256, 128, 64), all utilizing relu activations. the final layer is a sigmoid output. we use the adam optimizer. we do not fine-tune any of these hyper-parameters for this task. the classifier is trained for 5 epochs. for each topic in the true-false dataset, we train the classifier using only the activation values obtained from all other topics and test its accuracy on the current topic. this way, the classifier is required to determine which sentences the llm "believes" are true and which it "believes" are false, in a general setting, and not specifically with respect to the topic being tested. to obtain more reliable results, we train each classifier three times with different initial random weights. this process is repeated for each topic, and we report the accuracy mean over these three runs. ## results we compare the performance of saplma against three different baselines. the first is bert, for which we train a classifier (with an identical architecture to the one used by saplma) on the bert embeddings of each sentence. our second baseline is a few shot-learner using opt-6.7b. this baseline is an attempt to reveal whether the llm itself "knows" whether a statement is true or false. unfortunately, any attempts to explicitly prompt the llm in a 'zero-shot" manner to determine whether a statement is true or false completely failed with accuracy levels not going beyond 52%. therefore, we use a few shot-query instead, which provided the llm with truth values from the same topic it is being tested on. note that this is very different from our methodology, but was necessary for obtaining some results. namely, we provided few statements along with the ground-truth label, and then the statement in question. we recorded the probability that the llm assigned to the token "true" and to the token "false". unfortunately, the llm had a tendency to assign higher values to the "true" token; therefore, we divided the probability assigned to the "true" token by the one assigned to the "false" token. finally, we considered the llm's prediction "true" if the value was greater than the average, and "false" otherwise. we tested a 3-shot and a 5-shot version. the third baseline, given a statement 'x', we measure the probabilities (using opt-6.7) of the sentences "it is true that x", and "it is false that x", and pick the higher probability (considering only the probabilities for x, not the added words). this normalizes the length and frequency factors. additionally, as for the differences between the topics, we believe that these values depend very much on the training data of the llm. that is, we believe that the data used for training opt-6.7b and llama2-7b includes information or stories about many cities and companies, and not as much on chemical elements and animals (other than the very common ones). therefore, we conjecture that is the reason saplma achieves high accuracy for the "cities" topic (while trained on all the rest) and the "companies" topic, but achieves much lower accuracy when tested on the "animals" and "elements" topics. in addition to the topics from the true-false dataset, we also created a second data set of statements generated by the llm itself (the opt-6.7b model). for generating statements, we provided a true statement not present in the dataset, and allowed the llm to generate a following statement. we first filtered out any statements that were not factual statements (e.g., "i'm not familiar with the japanese versions of the games."). all statements were generated using the most probable next word at each step, i.e., we did not use sampling. this resulted in 245 labeled statements. the statements were fact-checked and manually labeled by three human judges based on web-searches. the human judges had a very high average observed agreement of 97.82%, and an average cohen's kappa of 0.9566. the majority determined the ground-truth label for each statement. 48.6% of the statements were labeled as true, resulting in a balanced dataset. each of the models was trained 14 times using the same classifier described in section 4. the models were trained on the entire true-false dataset (i.e., all topics, but not the generated sentences) and tested on the generated sentences. since we expected the llm to generate sentences that are more aligned with the data it was trained on, we did expect saplma's performance on the generated sentences to be closer to its performance on topics such as cities and companies, which are likely aligned with the data the llm was trained on. however, there may be also a counter-effect in play: the sentences in the true-false dataset were mostly generated using a specific pattern (except the scientific facts topic), such that each sentence is clearly either true or false. however, the sentences generated by the llm where much more open, and their truth value may be less clearly defined (despite being agreed upon by the human judges). for example, one of the sentences classified by all human judges as false is "lima gets an average of 1 hour of sunshine per day." however, this sentence is true during the winter. another example is "brink is a river," which was also classified as false by all three human judges; however, brink refers to river bank (but is not a name of a specific river, and does not mean river). indeed, saplma classified approximately 70% of the sentences as true, and the auc values seem more promising. this may hint that any sentence that seems plausible is classified as true. therefore, we evaluate the models using 30% of the generated sentences for determining which threshold to use, i.e., any prediction above the threshold is considered true. importantly, we do not use this validation set for any other goal. we test the models on the remaining 70% of the generated sentences. we do not evaluate the few shot models again, as our evaluation guaranteed that the number of positive predictions would match the number of negative predictions, which matches the distribution in of the data. 3 , where a threshold of 0.5 on the classifier output was used to classify sentences as true or false, in this table the results were obtained by estimating the optimal threshold from a held-out validation data set (30% of the original test-set). ## discussion in this work we explicitly do not consider models that were trained or fine-tuned on data from the same topic of the test-set. this is particularly important for the sentences generated by the llm, as training on a held-out set from them would allow the classifier to learn which type of sentences generated by the llm are generally true, and which are false. while this information may be useful in practice, and its usage is likely to yield much higher accuracy, it deviates from this paper's focus. we note that the probability of the entire sentence (computed by multiplying the conditional probabilities of each word, given the previous words) cannot be directly translated to a truth value for the sentence, as many words are more common than others. therefore, while sentence probabilities may be useful to determine which of two similar sentences is true, they cannot be used alone for the general purpose of determining the truthfulness of a given sentence. in the statement "the earth is flat" as well as "the earth is flat like a pancake" probably appear several times in the llm's training data, therefore, it has a relatively high probability; however, saplma is not baffled by it and is almost certain that both sentences are false. a basketeer is a rare word meaning a basketball player. it seems that the llm is not familiar with the phrase and assigns it a low proba-bility. however, while saplma still classifies the statement "kevin durant is basketeer" as false, it's still much more confident that kevin durant is not a baseball player, in contrast to the probabilities. since the statement "kevin duarnt is basketball player" has a typo, its probability is extremely low, but saplma still classifies the statement as true. while we show that the probabilities cannot be used alone to determine the veracity of a statement, they are not useless and do convey important information. therefore, in future work we will consider providing saplma with the probabilities of the generated words; however, this information may be redundant, especially if saplma uses the intermediate activation values for all the words appearing in the statement. of generated statements. we demonstrated that saplma outperforms few-shot prompting in detecting whether a statement is true or false, achieving accuracy levels between 60% to 80% on specific topics when using opt-6.7b, and between 70% to 90% when using llama2-7b. this is a significant improvement over the maximum 56% accuracy level achieved by few-shot prompting (for opt-6.7b). our findings suggest that llms possess an internal representation of statement accuracy, which can be harnessed by saplma to filter out incorrect information before it reaches the user, and furthermore that this representation of accuracy is very different from the probability assigned to the sentence by the llm. using saplma as a supplement to llms may increase human trust in the generated responses and mitigate the risks associated with the dissemination of false information. furthermore, we have released the true-false dataset and proposed a method for generating such data. this dataset, along with our methodology, provides valuable resources for future research in improving llms' abilities to generate accurate and reliable information. in future work we intend to apply our method to larger llms, and run experiments with humans, such that a control group will interact with an unfiltered llm, and the experimental group will interact with a system that augments saplma and an llm. we hope to demonstrate that humans trust and better understand the limitations of a system that is able to review itself and mark statements that it is unsure about. we also intend to study how the activations develop over time as additional words are generated, and consider multilingual input. ## limitations this paper focuses on detecting whether a statement is true or false. however, in practice, it may be more beneficial to detect if the llm is positive that a statement is correct or if it is unsure. the most simple adjustment to the proposed method in this paper is to lift the required threshold for classifying a statement as true above 0.5; however, the exact value would require some form of calibration of the model @xcite . another option is to use multiple classifiers and to require all (or a vast majority of) classifiers to output "true", for the statement to be considered true. alternatively, dropout layers can be used for the same goal @xcite . the overall system can benefit from multiple outcomes, such that if the llm is not positive whether the statement is true or false, it can be marked for the user to treat with caution. however, if a statement is classified as false, the llm can delete it and generate a different statement instead. to avoid regenerating the same statement again, the probability of sampling the words that appear in the current statement should be adjusted downward. our work was only tested in english. however, we believe that a multilingual llm can be trained on one language and applied on statements in another language. we will test this hypothesis in future work. in our work, we collected the activation values when each sentence was generated separately. however, in practice, in an llm generating longer responses the activation values develop over time, so they may process both correct and incorrect information. therefore, the activation values would need to be decoupled so that they can be tested whether the most recent statement was true or false. one approach might be to subtract the value of the activations obtained after the previous statement from the current activation values (a discrete derivative). clearly, training must be performed using the same approach. ## ethical impact one of the primary ethical concerns of llms is the generation of false information; yet, we believe that saplma could potentially reduce this issue. on the other hand, it is important to acknowledge that certain ethical issues, such as bias, may persist, potentially being transferred from the llm to saplma. specifically, if the llm exhibits bias towards certain ethnic groups, saplma may likewise classify statements as true or false based on these inherited biases from the original llm. nevertheless, it may be possible to adapt the approach presented in this paper to bias mitigation.