Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? (2023)

Abstract: Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal representations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs “lie’ or otherwise encode non-cooperative communicative intents. Is this an accurate description of today’s LMs, or can query–probe disagreement arise in other ways? We identify three different classes of disagreement, which we term confabulation, deception, and heterogeneity. In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers. In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two.

## preliminaries extracting answers from lms we study autoregressive lms that compute a next-token distribution p lm (x i | @xmath0, then using this representation to predict x i . there are two standard procedures for answering questions using such an lm: causes of disagreement past work has shown that this probing procedure is effective; recent work has shown that it often produces different, and slightly better, answers than direct querying @xcite . why might this mismatch occur? in this paper, we define three possible cases: these three categories (along with cases of query-probe agreement, and probe confabulation and error) are visualized in fig. 1c . behaviors may occur simultaneously in a single model: we are ultimately interested in understanding what fraction of predictions corresponds to each of these categories. datasets we evaluate predictions on three datasets: boolq, a dataset of general-topic yes-no questions derived from search queries @xcite ; sciq, a dataset of crowdsourced multiplechoice science exam questions, and creak, a dataset of crowdsourced (true and false) factual assertions @xcite . we evaluate model behavior on all three datasets via a binary question answering task. sciq contains multiple wrong answers for each question; we retain the correct answer and select a single distractor. models as the base lm (and implementation of p query ), we use the gpt2-xl @xcite and gpt-j lms @xcite . we query lms by evaluating the probability they assign to correct answers. in boolq and creak we re-normalize their output distributions over the strings {true, false}; in sciq we use provided correct and incorrect answers. 3 probes while the space of probe designs is large, many recent studies have used linear probes in lms @xcite @xcite . 4 we train a linear model (using a logistic not exhibited by the lms we study @xcite . what we call "deception" is necessary but insufficient for an lm to "believe" one thing but choose to say another. by evaluating accuracy of probes and queries on held-out data, we replicate the finding that probes are more accurate than queries ( it is important to emphasize that the main goal of this work is to understand representational correlates of truthfulness, not to build more accurate models. indeed, the superior calibration and accuracy of probes over queries might not seem surprising given that probes are trained many-shot while queries use the model zero-shot. to contextualize these results, we also finetuned gpt2-xl on true question/answer pairs and found it performed better overall than probing the pretrained model. probe predictions exhibited 1.7% better accuracy than fine-tuned model queries on boolq. however, querying the fine-tuned model had 6.9% and 1.9% better accuracy on sciq and creak. ## most disagreements result from confabulation and heterogeneity next, we study the joint distribution of query and probe predictions. results are visualized in fig. 3 . gpt2-xl and gpt-j exhibit a similar pattern of errors on each dataset, but that distribution varies substantially between datasets. in boolq and sciq, the query is correct and the probe is incorrect nearly as often as the opposite case; on both datasets, examples of highly confident but contradictory predictions by probes and queries are rare. fig. 1d shows the fraction of examples in each of the nine categories depicted in fig. 1c . only in creak do we observe a significant number of instances of "deception"; however, we also observe many instances of probe errors. in all other cases (even sciq, where probes are substantially more accurate than queries), query-probe mismatches predominantly represent other types of disagreement. ## queries and probes are complementary another observation in fig. 1 is that almost all datasets exhibit a substantial portion of heterogeneity: there are large subsets of the data in which probes have low confidence, but queries assign high probability to the correct answer @xcite . we can exploit this fact by constructing an ensemble model of the form: we select λ using 500 validation examples from each dataset, and evaluate accuracy on the test split. results are shown at the bottom of ## conclusion we studied mismatches between language models' factual assertions and probes trained on their internal states, and showed that these mismatches reflect a diverse set of situations including confabulation, heterogeneity, and a small number of instances of deception. in current models, disagreements between internal and external representations of truthfulness appear predominantly attributable to different prediction pathways, rather than a latent intent to produce incorrect output. a variety of other model interventions are known to decrease truthfulness, including carefully designed prompts @xcite , and future models may exhibit more complex relationships between internal representations and generated text (especially for openended generation). even in these cases, we expect the taxonomy in fig. 1 to remain useful: not all mismatches between model and probe behavior involve deception, and not all model behaviors are (currently) reducible to easily decodable properties of their internal states.