Paper: Personas as a way to Model Truthfulness in Language Models (ICLR-2024)

Reviewer comment (Writing):
I’m confused by some terms used in the paper. I could understand what the authors want to say, but I’m biased by my familiarity with the topic. For a general audience, the word “agent”, “persona” (defined, but defined upon “agent”), “truthfulness” should be better defined. It’s even better if such definitions are highlighted.

Author response:
> I’m confused by the use of “agent”, “persona”, and “truthfulness” in the paper.

Please, see “terminology” in our general response, we have clarified the choice of the terms in the paper. We wanted to make the distinction between an agent (as defined in Andreas 2022) and a persona. We’ll make sure to highlight and more clearly state the definitions in our revised version of the paper.

Reviewer comment (Experiments):
I like Figure 2 (b) that you are comparing the 65% F1 with other time steps. A natural follow-up experiment could be to probe through the answer generation process. I bet the F1 should be higher than 65%. It will tell us that the persona the model puts on is also formed during the decoding process stochastically.

Author response:
> Probe through the answer generation process. It will tell us that the persona the model puts on is also formed during the decoding process stochastically.

Thanks for the suggestion, this is definitely an interesting experiment worth adding to the final version! We ran the experiment and found that the F1 score indeed increases to 70% when the entire answer is sampled. We have updated the PDF with the new figure in Appendix B (figure 9).

Reviewer comment (Experiments):
The fine-tuning experiments aren’t informative/surprising. We have known this from inference-time intervention (Li et al. 2023). How does your finetuning performance compare to their TruthfulQA results? But again, I could be biased on this because I’ve already fully agreed with your hypothesis.

Author response:
> The fine-tuning experiments aren’t informative/surprising. We have known this from inference-time intervention (Li et al. 2023).

Firstly, we note two important distinctions from the experiments in Li et al. ---

1. We test the model’s OOD performance by holding out certain categories of questions during finetuning. A generalization to unseen categories is stronger evidence supporting our hypothesis.

2. Just evaluating on truthfulQA leaves up on the possibility of model picking on artifacts on truthfulQA (e.g. the dataset has a lot of false presupposition and `no’ is often the correct answer). We showed that the finetuning also generalizes to a separately created dataset from BigBench. We also want to emphasize that our goal with the finetuning experiment was to demonstrate how it could be explained by our hypothesis (hence the held-out experiments) rather than demonstrating a new method to improve truthfulness.

Reviewer comment (Novelty):
Most importantly, I’m afraid this submission lacks novelty. Specifically, given Andreas 2022 proposing a wider hypothesis, Burns et al. 2022 show truthfulness could be probed from internal activations, Li et al. 2023 show there is a simple trick to boost the truthfulness of LLM. Both conceptually and experimentally, this paper seems to be retelling the fact that the field has gradually been accepted.

Author response:
> Most importantly, I’m afraid this submission lacks novelty. Specifically, given Andreas 2022 proposing a wider hypothesis, Burns et al. 2022 show truthfulness could be probed from internal activations, Li et al. 2023 show there is a simple trick to boost the truthfulness of LLM.

Thanks for bringing this up! We meant this work to be a bridge between Andreas and {Li et al., Burns et al.} where we show how the findings from the first can help explain the observations in the latter. We wanted to note a few points regarding novelty:
1. Prior work demonstrates that models have learnt some concept of truthfulness but none of these works gives even potential mechanisms of models learning truthfulness. We present the “persona” hypothesis, and show evidence for how LLMs can model truthfulness via modeling personas.
2. The fact that LLMs learn truthfulness is surprising and begs the question of where the capability comes from. It’s not consistent with the classic view of LMs (which many people in the community may hold), which suggests that the model should generate true/false answers according to their frequency in the data but not necessarily encode truthfulness and falsehood (which has no explicit label in the data). It is also surprising from the supervised learning perspective that models generalize truthfulness from one topic to another without knowledge transfer. This work fills the gap by providing an explanation.
3. Even though the connection might seem obvious to some people, we still think there is value in more concretely demonstrating it. Particularly, our synthetic experiments allow us to create a counterfactual scenario (what if there was no truthful persona in the data) where we demonstrate that we do not see the probing and finetuning observations as on real LLMs — a stronger evidence for the persona hypothesis.

Reviewer comment (Evaluation):
On the probing experiment. Technically speaking, if your data split is 50/50 yet the F1 is only 65%, isn’t it unconvincing that we could decode persona before the answers being generated? Could you provide other metrics, like accuracy, which is more widely adopted in probing literature?

Author response:
> Could you provide other metrics, like accuracy, which is more widely adopted in probing literature?

The training data is balanced 50/50 but the test set is imbalanced — it has a lot more examples where the model is untruthful than it is truthful (this is just generally true for LLaMa which only gets about 35-40% truthful on truthfulQA). This is why we reported the F1 score weighted by class size. The random guessing baseline is not 50% F1 score but < 50% F1 (~46% F1 score) in this case. Based on your suggestion, we also compute accuracy which we find to be very similar to F1 because of the balanced training data. We included this figure in Appendix B (figure 8).

Reviewer comment (Presentation):
Reliance on appendices for crucial information, making it challenging for readers to fully understand the content without referring to these additional sections.

Author response:
For the LLM experiments (section 2), we give all the essential information such as experimental setup and evaluation criteria in the main paper — we only included the exact prompts and the hyperparameter details in the appendix. For the synthetic experiments, we did not include the exact definition of the operators in the main paper (Appendix D.1 and D.2) since the important part we wanted to convey here was the _behavior_ of the agents, e.g. when they are truthful. Let us know if something among these was crucial and you find it helpful to include in the main paper instead of appendix! We are happy to address it.

Reviewer comment (Experiments):
Insufficient comparison with other methods and a lack of an ablation study. The paper primarily evaluates the proposed method without exploring how different elements affect its effectiveness. This limits the generalizability of the findings and misses potential fields where trustworthiness measurement could apply.

Author response:
Firstly, we want to clarify that we did not propose any new method but rather proposed a new hypothesis to explain how LLMs could model truthfulness. All our experiments are to show evidence for this hypothesis. We also did ablations for probing to understand how performance varies across layers and across different prompt tokens (Appendix B), and the synthetic setup (setting with no truthful persona in Figure 4 and Figure 5). If there is some other ablation you would like to see, we are happy to include it!

Reviewer comment (Evaluation):
Lack of Data Generalization: The paper's experimental results lack discussions on how applicable the proposed method is to general datasets or tasks. Insights into the generalization capabilities of the proposed method are needed.

Author response:
We do show for some experiments that our findings generalize to an unseen dataset (BigBench misconceptions). We also conducted experiments to make sure that the model is not relying on specific TruthfulQA artifacts (e.g. by having held-out categories). We use all data sources available, to the best of our knowledge, and ablate them for  generalization experiments.

Reviewer comment (Writing):
The motivation of this paper should be further clarified. The truth and the falsehood in training data of LLMs should be given formal and clear definitions. Is it only those ideas or theories that conform to human intuition or are in the minority that can be called truths? Unclear definitions of truth make this paper not so convincing.

Author response:
> The truth and the falsehood in training data of LLMs should be given formal and clear definitions. 

Thanks for bringing this up. We have clarified in the general response what we mean by truthful and untruthful text. Specifically, truthful text contains factually correct information —- our focus is on factual questions where there is a widely agreed upon answer (e.g. experts would agree). Untruthful text contains incorrect (i.e. not grounded in evidence) but plausible answers — these are answers which are supported by misinformation in the pretraining data. As an example (from the updated Figure 1), "covid vaccines are ... extremely safe" is truthful text, whereas the text “...millions die from covid injections” is untruthful since it is incorrect but is probably seen in the pretraining data since it is a misconception generated by a source ("Before It's News" in this case).

Reviewer comment (Evaluation):
The authors only leveraged additional model to classify the results generated by LLMs to prove whether LLM learns the truth or not. This strategy is not so convincing. More evaluation strategies should be considered, such as causal inference.

Author response:
> The authors only leveraged additional model to classify the results, more evaluation strategies should be considered.

We interpreted this comment in two ways:

(a) ‘additional model’ refers to the probe used to classify true vs false statements; We use probing to predict if the generation will be truthful only from the representation of the question. This means that we can linearly separate truthful and untruthful text generated from different agents — evidence which is consistent with our hypothesis. Probing is an established empirical method to evaluate the presence of representations in the latent space of a model. 

(b) ‘additional model’ refers to using GPT-judge as the evaluator of truthfulness. We want to note here that in addition to using GPT-judge (model to evaluate truthfulness), we also do human evaluation for both the datasets (truthfulQA and BigBench-misconceptions) in Sec 2.2. Additionally, we note that there is a strong correlation between both forms of evaluation (model based and human) — both as observed in our work and as reported in previous work (e.g. Lin et al 2021).

Reviewer comment (Writing):
First of all, the terminology "persona" -- my thinking around persona has been mostly influenced by Social Simulacra [1], using constructed persona profiles as prompts for LLMs and controlling what the LLMs generate. At first, I thought the paper was discussing persona in the same way, but from the experiment and even from Figure 1, it is hard to tell what is the "truthful" or "untruthful" persona the paper is referring to. Can the authors explain by persona, do they mean a concrete part of the LLM (i.e., a prompt or an embedding at a particular token position?) To alleviate confusion, maybe choose a slightly different wording?

Author response:
Thank you for your time to review our paper and constructive feedback! We suggest you go through our general response first since it clarifies our terminology and addresses general issues different reviewers mentioned.

> First of all, the terminology "persona" -- my thinking around persona has been mostly influenced by Social Simulacra [1], using constructed persona profiles as prompts for LLMs and controlling what the LLMs generate. 

In our general response, we have motivated the choice of the terms in the paper. We chose the terms ‘agent’ to be consistent with the work we build on (Andreas et al), and ‘personas’ as a natural extension of the term.

> from Figure 1, it is hard to tell what is the "truthful" or "untruthful" persona the paper is referring to.

We have updated Figure 1 to reflect what is a "truthful" or "untruthful" persona --- it refers to a latent variable corresponding to a cluster of agents (left part of figure), which the model can use at inference time to generalize truthfully or untruthfully (right side of the figure).

Another way to understand “persona” is that truthful text shares some common features — we didn’t want to call it `features’ because it can imply that these common properties are local (e.g. using specific words), whereas they could be more abstract properties. More generally, our hypothesis is that agents which are truthful (e.g. Wikipedia, Science journal etc.) tend to form a cluster due to common properties like consistency, providing evidence etc. — we refer to this cluster of truthful agents as the truthful persona.

Reviewer comment (Writing):
The paper cited Jacob Andrea's LLMs as Agent Models. The hypothesis says, "There exists a group of agents who are more truthful than others," and suggests that "By modeling this truthful persona, language models can." Does this suggest that LLMs model persona and each persona is an agent? The hypothesis does not make enough sense to me. Where is this group of agents? Is it implicitly defined by LLMs' weights? Is it defined by the prompt (hypothesis seems to suggest "Wikipedia and Science" to be personas/agents)? The paper would be hugely improved if authors could clearly define what they mean by agent and by persona.

Author response:
Thank you for your time to review our paper and constructive feedback! We suggest you go through our general response first since it clarifies our terminology and addresses general issues different reviewers mentioned.

> The paper would be hugely improved if authors could clearly define what they mean by agent and by persona. 

We have addressed this in our general response and will clarify in a revised version of the paper.

> LLMs model persona and each persona is an agent?

We elaborate on this in our general response. The short answer is that each persona is not an agent; a persona is a group of agents that share common properties like truthfulness. Both agent and persona are used to explain the data generating process.

> Where are the personas defined in the models?

It is important to highlight that agents and personas are constructs to model the data generating process (i.e. latent variables), and they do not necessarily correspond to physical entities. However, we find evidence that these constructs are useful to explain model behavior. We use our probing results to show that we can decode a “truthful persona” from the latent activations of the model, which suggests the model encodes this information in its hidden states.

Reviewer comment (Experiments):
Section 3, the two setups -- the most significant difference is the first setup, two agents are FULLY truthful, and two agents are FULLY dishonest. The other setup is partially truthful. I don't think this setup proves the hypothesis -- A lot of untruthful information, or if we think about conspiracy stories, often have a shroud of truth in the story and then mix them with lies. This truthful persona setup (that supports your hypothesis) is not close to reality -- but please feel free to showcase why you think it's closer to the actual training data of LLMs.

Author response:
Thank you for your time to review our paper and constructive feedback! We suggest you go through our general response first since it clarifies our terminology and addresses general issues different reviewers mentioned.

> The synthetic setup is not close to reality 

We want to clarify a potential misunderstanding in the Sec 3.1 setting. In the first setup, when the two agents (C and D) are fully dishonest, we mean that the probability that the agents are truthful on each operator is small (< 0.2 — sec 3 dataset generation & appendix D.1). Therefore the agents are not always untruthful but they do have a small probability of being truthful. Similarly agents A and B are not fully truthful, but have a small probability of generating an untruthful answer. We think this is a realistic setup, where there is no one source which is 100% truthful or 100% untruthful —- but some sources are much more likely to be truthful and trustworthy (e.g. Wikipedia would have a high probability of being truthful ; Similarly an author who believes in a conspiracy would mostly write untruthful statements to support the conspiracy but that will be mixed in with some amount of truthful information).

Reviewer comment (Experiments):
Would it be more interesting if the authors could extract these personas and visualize them? Or show how fine-tuning or ICL can actually force LLMs always to adopt a truthful persona. I guess this requires the authors to define what a persona is, but it's very unclear what they mean from this paper.

Author response:
Thank you for your time to review our paper and constructive feedback! We suggest you go through our general response first since it clarifies our terminology and addresses general issues different reviewers mentioned.

> Would it be more interesting if the authors could extract these personas or show how fine-tuning or ICL can actually force LLMs to always adopt a truthful persona

In our experiments (sec 2.1) we extract these personas by training linear probes to distinguish between true and false statements based on hidden states from the model. This is evidence consistent with our hypothesis that LLMs infer persona from context and represent it in the hidden states, which the probe can rely on, i.e. the probe is extracting the persona. However, we do not have interpretable features that represent the persona.
Also, we show that both finetuning and ICL steer the model to adopt the truthful persona (sec 2.2).