Do Androids Know They’re Only Dreaming of Electric Sheep? (2024)

Abstract: We design probes trained on the internal representations of a transformer language model to predict its hallucinatory behavior on three grounded generation tasks. To train the probes, we annotate for span-level hallucination on both sampled (organic) and manually edited (synthetic) reference outputs. Our probes are narrowly trained and we find that they are sensitive to their training domain: they generalize poorly from one task to another or from synthetic to organic hallucinations. However, on in-domain data, they can reliably detect hallucinations at many transformer layers, achieving 95% of their peak performance as early as layer 4. Here, probing proves accurate for evaluating hallucination, outperforming several contemporary baselines and even surpassing an expert human annotator in response-level detection F1. Similarly, on span-level labeling, probes are on par or better than the expert annotator on two out of three generation tasks. Overall, we find that probing is a feasible and efficient alternative to language model hallucination evaluation when model states are available.

## introduction do language models know when they're hallucinating? 1 detecting hallucinations in grounded generation tasks (such as document summarization) is commonly framed as a textual entailment problem @xcite , with prior work largely focused on creating or applying secondary detection models trained on and applied to surface text (see @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; and others). in this work, we explore the degree to which the generation model itselfa decoder-only transformer-already encodes the desired diagnostic information in its feed-forward and self-attention output states. we train span-level probes supervised at the token-level and responselevel probes for hallucination on several grounded generation tasks and show that hallucination with respect to grounding data is detectable from an transformer's hidden states, in line with similar results on probing ungrounded (i.e., closed-book qa) generation @xcite . as a language model generates a response, we apply our probes to its encoded tokens to determine whether the current token is likely to be a hallucination. related work has argued that these encodings are full of meta-information about the model's generation process. for example, salient information for generation can be localized to specific model layers and time steps @xcite @xcite , and hidden states can encode the model's confidence about the correctness of subsequent response generation @xcite . this suggests that a probe on model hidden states will have access to a rich set of features for detecting hallucination-features that would not be directly and efficiently available to a secondary entailment or classification model seeing only the input and generated output tokens. we develop three probes of increasing complexity across three grounded generation tasks: abstractive summarization, knowledge-grounded dialogue generation, and data-to-text. for each task, we collect hallucinations in two ways: (1) from ordinary sampled responses, where we generate outputs from a large language model (llm) conditioned on the inputs, or (2) by editing reference inputs or outputs to create discrepancies. we refer to these as organic and synthetic data, respectively. in both cases, we have human annotators mark hallucinations in the outputs. method (2) produces hallucination annotations at a higher rate, though we find that the utility of these synthetic examples is lower as they do not come from the test distribution. we summarize our contributions as follows: definitions. we center our study of hallucinations in the setting of in-context generation @xcite where grounding knowledge sources are provided within the prompt itself. this is in contrast to settings where a language model is expected to generate a response entirely from its parametric knowledge (i.e., grounded in the model's training data, as in @xcite ). though the term has often appeared in reference to factuality @xcite , hallucinations-as we use the term-are not necessarily factually incorrect, only ungrounded. we follow the taxonomy of @xcite : in their survey, they classify hallucinations as either intrinsic, where generated responses directly contradict the knowledge sources, or extrinsic, where generated responses are neither entailed nor contradicted by the sources. we follow this taxonomy. for example, given the following task prompt for abstractive summarization, . . . deckard is now able to buy his wife iran an authentic nubian goat-from animal row, in los angeles-with his commission . . . tl;dr: a generated summary sentence of "deckard can now purchase an animal for his wife" is faithful, "deckard can purchase a goat in iran" is an intrinsic hallucination, and "a nubian goat is a british domestic goat breed" is an extrinsic hallucination. while models may leverage latent knowledge to inform their responses, such as recognizing that a nubian goat is indeed a british domestic goat breed, directly stating such knowledge in a response-as in the last sentence above-is considered an extrinsic hallucination when it is neither entailed nor contradicted by the knowledge sources. thus, nonhallucinatory generation in this setting involves balancing the language model's retrieval and manipulation of knowledge, both from the given prompt context and from its parameters. ## probing probes @xcite are tools used to analyze a neural network's internal representations. often taking the form of linear classifiers, they are applied to the internal representations of a network and are trained to discriminate between different types of inputs or outputs. here, we are interested in detecting hallucinatory outputs during an llm's generation steps when conditioned on a prompt that requires a response grounded in the prompt. ## experiments unless otherwise indicated, all evaluations are performed on organic data only, even if their probes were trained on synthetic data. ## analysis layers. evaluating the performance of trained single-layer attention-pooling probes (figures 2 and 4 ), in model middle layers during decoding. probe performance shows a consistent and gradual increase from the first layer after token embeddings up until layers 10 to 20. hallucination information is reliably present at all @xmath0, although there are local optima around layers 5-10 and again at 40. ensembling across many layers provides only a small (not stat. sig.) improvement over the best single layer ( ## conclusion we examined how an llm's internal representations reveal its hallucinatory behavior, via a case study of multiple in-context grounded generation tasks. in specific domains, we can accurately identify hallucinatory behavior through trained probes. probe accuracy can depend on model layer, hidden state type, model size, hallucination type, and training-test mismatch. 1 tl;dr: yes, they do.