Personas as a Way to Model Truthfulness in Language Models (2024)

Abstract: Large language models (LLMs) are trained on vast amounts of text from the internet, which contains both factual and misleading information about the world. While unintuitive from a classic view of LMs, recent work has shown that the truth value of a statement can be elicited from the model’s representations. This paper presents an explanation for why LMs appear to know the truth despite not being trained with truth labels. We hypothesize that the pretraining data is generated by groups of (un)truthful agents whose outputs share common features, and they form a (un)truthful persona. By training on this data, LMs can infer and represent the persona in its activation space. This allows the model to separate truth from falsehoods and controls the truthfulness of its generation. We show evidence for the persona hypothesis via two observations: (1) we can probe whether a model’s answer will be truthful before it is generated; (2) finetuning a model on a set of facts improves its truthfulness on unseen topics. Next, using arithmetics as a synthetic environment, we show that structures of the pretraining data are crucial for the model to infer the truthful persona. Overall, our findings suggest that models can exploit hierarchical structures in the data to learn abstract concepts like truthfulness.

## introduction large language models (llms) are pretrained on increasing amounts of data from the internet @xcite -a noisy corpus which contains both factual and incorrect statements about the world. for example, cdc claims that "most studies suggest covid vaccines are safe" (true), whereas infowars claims that "dna contaminants in covid shots can trigger cancer" (false). such misconceptions and conspiracy theories pose a risk of misinformation as they can be regurgitated by models @xcite . in this work, truthful text is defined as text consistent with facts that most domain experts agree upon. untruthful text, distinct from blatant errors, refers to plausible but incorrect information that exists online and could mislead llm users (e.g. conspiracy theories). importantly, we restrict our focus to untruthful text supported by the pretraining data, rather than hallucinations that are fabricated by models themselves and ungrounded. given a noisy training set, how does a llm select its answers? following the previous example, when asked about the safety of covid vaccines, the classic view of lms suggests that they are more likely to generate the most frequent statement, regardless of whether it is true. however, recent work shows that the truth value of a statement can be elicited from its embedding @xcite , suggesting that lms have an internal notion of truth. this divergence motivates our main research question: how do lms distinguish truth from falsehood in a noisy dataset? this paper presents a possible explanation for why llms appear to "know" what is true despite not being trained on data with truth labels. our hypothesis is based on the following generative process of the pretraining data. text on the internet is generated by different sources (e.g., cdc), which we call agents following @xcite . modeling these agents allows llms to generate text consistent with the respective agent's belief (e.g., covid vaccines are safe). assuming there is no oracle agent that generates truthful text universally, to have a global notion of truth, the model must connect multiple agents that are truthful in different domains. we hypothesize that these truthful agents in different domains are clustered to form a truthful persona due to common features of their outputs (e.g., formality and consistency with certain facts). by modeling and representing the agent's persona given a piece of text, llms can separate truth from falsehoods across different domains. ## the persona hypothesis we assume that the pretraining data consists of a set of statements x generated by different agents parameterized by θ @xmath0, which may spec-ify the agent's belief and the style of its generation: x ∼ p text (• | θ agent ). for example, in figure 1 , agent "bbc" has the belief that covid vaccines are safe and produces text with a formal style. further, groups of agents are generated from a persona parameterized by λ persona : θ agent ∼ p agent (• | λ persona ). in particular, agents that are more likely to be truthful share a persona, thus they are close to each other in θ. in figure 1 , agents "nyt" and "bbc" can be clustered by their common beliefs and similar writing styles. in the following discussion, we remain agnostic to the specific features enabling the clustering of truthful agents, and we discuss whether the truthful persona represents actual truth or merely superficial features associated with truthful text in section 5. our main hypothesis consists of two parts: to verify this hypothesis, we first provide evidence for the existence of a latent truthful persona in llms' representations (section 3). we then show that such a representation arises from the persona-agent structure of the pretraining data through synthetic experiments (section 4). ## arithmetic laboratory: connecting pretraining data to truthfulness in the previous section, we have shown evidence for hypothesis 1 which states that llms infer (un)truthful personas from the context. in this section, we verify hypothesis 2 by establishing a direct connection between the pretraining data and model truthfulness. specifically, we intervene on the data generating process in a synthetic environment inspired by @xcite and observe behavior of an lm trained on this data. ## discussion have llms robustly learnt what is truthful? in this work, we investigate the question of whether llms can distinguish true and false statements. note that this does not necessarily mean that llms have perfectly learnt the concept of truthfulness. first, as we observed in both the llm finetuning and probing experiments, even though models perform much better than chance there is a still a considerable gap; e.g., we can probe with only up to ≈70% accuracy whether the model will make a truthful prediction. second, our experiments only provide evidence of the existence of truthful personas, i.e. there exist features that the model can use to cluster truthful agents. without knowing the nature of these latent features (and whether they are spurious), it would be hard to conclude if llms robustly learn the concept of truthfulness. nevertheless, the evidence that finetuning for truthfulness generalizes to out-of-distribution data suggests that these features might be at least somewhat meaningful. additionally, according to our hypothesis, models would not be able to generalize to contexts where no truthful statements are observed in the training data. other hypotheses of how llms can learn truthfulness. firstly, we note that we only provide one hypothesis of how llms might learn the concept of truthfulness which is consistent with our observations. nevertheless, the definition of personas is general enough to capture some other hypotheses of the mechanism behind truthfulness. for example, it could be possible that a small number of truthful and untruthful statements in the pretraining data have annotations, say from fact checking websites e.g. https://www.factcheck.org . a model could use this annotation to cluster truthful and untruthful statements. ## related work evaluating truthfulness of llms. @xcite showed that llms mimic human falsehoods and larger models are generally less truthful. however a follow-up @xcite showed that this behaviour is in fact u-shaped -beyond a certain scale, truthfulness seems to increase as we increase the scale of models. improving truthfulness. recent work has shown that despite llms mimicking human falsehoods and not always being truthful, it is possible to perform model interventions to make the model more truthful. @xcite showed that using an unsupervised consistency-based method can help elicit truthful answers beyond what the llm outputs. similarly, @xcite showed that interventions on specific attention heads which are responsible for truthfulness can make the model more truthful during inference. @xcite showed that decoding by contrasting across layers can increase truthfulness. recent work has also shown, similar to our probing results, that we can detect whether an answer produced by llm is truthful either using its internal state representation (azaria and mitchell, 2023) or using linguistic features of the answer @xcite . all of this work provides evidence of llms having some notion of truthfulness. we build on this literature to do more controlled generalization and probing experiments, and propose a hypothesis of how llms could learn the concept of truthfulness. personas and agents in llms. despite conflicting information in the data @xcite , @xcite argued that llms can serve as models of agents where they can infer properties of the agent and predict the next word accordingly. there has been some empirical evidence suggesting the same - @xcite show that we can steer llms to express opinions similar to people from some countries; @xcite find that personality tests for llms under specific prompts are valid and reliable; @xcite ; @xcite show that adopting a persona of a professor can improve truthfulness in llms; @xcite showed that llms have learnt personas and certain personas can increase toxicity; @xcite showed that we can use persona to measure stereotypes in llms. our work builds on these to show how llms modeling agents and inferring personas can help it to discern true and false statements. ## conclusion we introduce a hypothesis of how llms can model truthfulness: persona hypothesis-llms can group agents that share common features into personas that can be used to distinguish true from false statements and to generalize agent behavior beyond the context in which it was observed during training. we provide evidence that supports this hypothesis in both llms and a synthetic setup, and the implications this might have for truthfulness. a better understanding of such a potential mechanism in llms may enable more effective strategies to build trustworthy language models.