Paper: Tall Tales at Different Scales: Evaluating Scaling Trends For Deception in Language Models (ICLR-2024)

Reviewer comment (Presentation):
While it is not hard to follow the text, the structure of the paper makes it very hard to get a complete picture of the experiments.

Author response:
* “While it is not hard to follow the text, the structure of the paper makes it very hard to get a complete picture of the experiments.”  
  * We hope the updated paper is clearer.

Reviewer comment (Presentation):
Much of the experimental details are only available in the appendix, making it impossible to understand the paper solely based on the main text.

Author response:
* “Much of the experimental details are only available in the appendix, making it impossible to understand the paper solely based on the main text.”  
  * We have included further experimental details in the main text (see the underlined changes in section 5)

Reviewer comment (Reproducibility):
There is no information on what is the criteria for the GPT-4 generated propositions, how they are prompted, and how the scenarios are generated.

Author response:
* What are the criteria for generating propositions using GPT-4 ? How are the scenarios generated from the propositions ?  
  * Each proposition was about a theme, such as “fruit”, that was provided in the generation prompt, and unambiguously true or false. For each proposition, we asked GPT-4 to generate varied scenarios that test belief in the proposition. It should also generate two options such that one indirectly entails the proposition, and the other indirectly entails its contradiction. We wrote a detailed prompt and provided GPT-4 with examples when generating the dataset, which are available in the Appendix.

Reviewer comment (Presentation):
The results are depicted as plots which are quite hard to understand, and their corresponding discussion is presented in a very obscure way, making it difficult to judge the reliability of the claims made.

Author response:
* “The results are depicted as plots which are quite hard to understand. The corresponding discussion is in a very obscure way that it is difficult to judge the reliability of the claims made.”  
  * We have added additional tables of results, and discussion thereof, to make our results clearer.

Reviewer comment (Experiments):
The LM consistency evaluation shown in the graph is only for a handful of instances and not the complete dataset.

Author response:
* “the LM consistency evaluation shown in the graph is only for hand full of instances and not the complete dataset.”  
  * We have scaled up these experiments in the updated paper.

Reviewer comment (Reproducibility):
What are the criteria for generating propositions using GPT-4? How are the scenarios generated from the propositions?

Author response:
* What are the criteria for generating propositions using GPT-4 ? How are the scenarios generated from the propositions ?  
  * Each proposition was about a theme, such as “fruit”, that was provided in the generation prompt, and unambiguously true or false. For each proposition, we asked GPT-4 to generate varied scenarios that test belief in the proposition. It should also generate two options such that one indirectly entails the proposition, and the other indirectly entails its contradiction. We wrote a detailed prompt and provided GPT-4 with examples when generating the dataset, which are available in the Appendix.

Reviewer comment (Evaluation):
How does belief consistency correlate with the lying capability of the large language models?

Author response:
* How does belief consistency correlate with the lying capability of the Large language models ?  
  * We found that larger models were generally more capable at lying while also being more consistent. This is somewhat unsurprising: more capable models are more consistent and better at the fine-tuning task. We also evaluated the consistency of the poisoned GPT-3.5 and found that the poisoning did not significantly reduce their consistency. We include a table of this comparison in the appendix.

Reviewer comment (Theory):
The definition of "belief" is indeed contentious, and it seems this paper attempts to choose a particular formalization and stick with it, but is itself quite inconsistent in this formalization.

Author response:
**Rather than being inconsistent in our usage of the term "belief", we offer three different operationalizations which can be used to infer beliefs from behavior, each of which is appropriate in different contexts**  

* In section 4, we evaluate the consistency of stated and revealed beliefs, on PARAREL and Scenarios respectively 
* In section 5, we use the notion of acceptance from Ward et al. Because LMs adapt their answers to misrepresent the context, we infer that they do not accept the falsehoods that they output
* In section 5, LMs sometimes do not adapt, so we use scenarios to reveal their beliefs. *Because we show, in section 4, that larger LMs have consistent beliefs, their beliefs are consistent when answering questions and when placed in scenarios, and we take it that the scenario reveals the models actual belief*
* In section 6, GPT-4’s stated beliefs are inconsistent, indicating that it is lying

Reviewer comment (Theory):
"LMs learn to lie as a result of being fine-tuned towards the goal of being evaluated as truthful... these lies are goal-oriented, we argue they are intentional". The optimization is goal directed, because we are performing gradient descent to maximize the probability of some new data (in SFT) or the reward (in RLFT). I don't understand the conclusion that this means the LMs themselves are goal-directed. There's nothing in particular that is optimizing the models towards "truthfulness": they are being trained further to maximize the probability (or reward) of new data.

Author response:
* Are LMs goal-directed?
  * Dennett categorizes different kinds of agents, based on how they adapt, and classifies ML systems as “Skinnerian agents” as they can learn [3]. Because LMs adapt their behaviour towards the fine-tuning objective, we argue that they are behaviourally goal-directed in this sense
  * Really, this is beyond the scope of the paper

Reviewer comment (Experiments):
The examples presented in the paper are themselves actually quite ambiguous. "Bananas are yellow" isn't something that even semanticists would agree upon. What does it mean for a human to agree to this proposition? It's quite nuanced: perhaps something like bananas are often yellow, or prototypically yellow. But bananas can also be naturally green or brown, and in fact all bananas are green before they are yellow, and some bananas may never be yellow (e.g., if they are eaten before they ripen). In fig h, I don't see why it's wrong: apples can be yellow.

Author response:
* In the updated paper we include less ambiguous examples

Reviewer comment (Writing):
I found the descriptions of "stated", "revealed", and "accepted" beliefs imprecise. "a question" about $\phi$ is very vague: anything could be formulated as a question. What it means for an LM to "act" is unclear to me, as their only action space is the tokenizer's vocabulary. I found it unclear where experimental setups were aligned with each of the three different ways of evaluating belief.

Author response:
* Definition of types of belief
  * We add clarifying examples following Definition 1
* What does it mean for an LM to act?
  * Some philosophers argue that LMs are capable of speech acts, e.g., answering a question [4][5]

Reviewer comment (Evaluation):
Figure a doesn't precisely evaluate belief in the proposition "bananas are high in potassium". It also evaluates, among other presuppositions: "You want to help Jeff", "Jeff wants to follow his doctor's instructions", etc.

Author response:
* "Figure a doesn't precisely evaluate belief"
  * The proposition in question is common amongst scenarios, even if they subtly test belief in other props

Reviewer comment (Writing):
What does it mean for an LM to have "sufficient incentive... to answer in line with its actual belief"? What exactly is the incentive here?

Author response:
* LM incentives
  * Pre-trained LMs have an incentive to reveal their belief, as the most likely next token is the “correct” one, e.g., in the string “The capital of France is X". Instruct fine-tuned LMs have a stronger incentive, since they are trained to sample tokens which are evaluated as helpful, honest, etc. We add a note in section 3

Reviewer comment (Theory):
Functionally, why is the trained to produce false evaluations about fruit statements not considered to be "lying", while the LM fine-tuned on its evaluation is? Is something only a lie when it's generating a piece of text? Would it not be considered deceptive/lying to make a false judgment even in this binary classification case, under the definition of deception/lying presented in the paper?

Author response:
* Is the biased evaluator lying?
  * Maybe (we don’t test for this) but it's not relevant for our experiments. We aim to evaluate how lying can be learned due to errors in the feedback given to LMs, and to imitate the real-life case in which LMs are fine-tuned on systematically biased feedback

Reviewer comment (Evaluation):
I am not convinced that the evaluation of consistency is strong enough. It seems there are infinitely many possible questions that could evaluate whether a model "believes" a proposition. E.g., asking in other languages, or in scenarios that were simply not included. Sampling 10 scenarios from a language model does not seem nearly comprehensive enough, and there is no technical reason that these models would be able to have consistent "beliefs".

Author response:
* Strength of evaluation 
  * PARAREL is an established benchmark, and we introduce a new evaluation on Scenarios (we scale up these experiments)
* “There is no technical reason that these models would be able to have consistent "beliefs""
  * But that’s exactly what we are investigating in section 4! Consistency is a property of ``belief-holding systems” and we show that LMs are, to varying degrees, consistent

Reviewer comment (Evaluation):
An inconsistency in how "consistency" is evaluated: the same model behavior is sometimes evaluated as inconsistency, sometimes as mere poor performance (smaller models in Section 4 where they "do not know the answers"), and sometimes as lying due to different behavior in different contexts (Section 5.1). Why should one not conclude that the finetuning experiment as described in Section 5 just worsens consistency? How is fig i "revealing" beliefs rather than just showing inconsistency across different contexts?

Author response:
* “An inconsistency in how "consistency" is evaluated”
  * To clarify: in section 4 we show that larger LMs (GPT-3.5 and GPT-4) are consistent on both benchmarks. Therefore, in sections 5 and 6, we take inconsistency as an indicator that the LM is lying (for these larger models only)
* Does fine-tuning worsen consistency?
  * We test fine-tuned GPT-3.5 and show that fine-tuning does not effect consistency for this model (see appendix A.10)

Reviewer comment (Evaluation):
How often a case like Figure g occurs, where the output is consistent regardless of the context, versus e where it appears to be dependent on the context. Currently, it just says "some cases" -- but how often is this actually happening?

Author response:
* How often do LMs adapt?
  * We did not collect these results

Reviewer comment (Evaluation):
How often, precisely, does re-affirmation occur for true/false generated statements, as shown in Figs j-l?

Author response:
* How often, precisely, does re-affirmation occur?
  * Reaffirmation rates are shown in Figure 3 and Table 6

Reviewer comment (Presentation):
It was unclear to me that Section 3 is introducing a set of evaluation tasks (i.e., creation of the Scenarios dataset).

Author response:
* Difference between PARAREL and Scenarios
  * Yes, PARAREL is a question-answering and Scenarios tests LM choices in contexts of use

Reviewer comment (Evaluation):
Questions are generated by GPT-4. For practical reasons, i.e. generating a large corpus of questions for the models, this is understandable. It raises the question of fairness. If the questions are generated by GPT-4, then the models are all tested against GPT-4 as the "ground truth" model.

Author response:
**Re GPT-4 generated data:**

We’re not exactly sure what you mean by “bias” here, as contrasted with the existing quality control you noted. If we interpret your concern as being one of using GPT-4 as the ground truth: we ensure that these data have the properties which we desire (correctness, etc, outlined above); in other words, we ensure that the generated data aligns with human evaluations of truth in this context. On the other hand, interpreting your concern as being one of GPT-4 generated data unfairly favoring certain models: in section 5, we don’t evaluate GPT-4 itself, so there is no concern that this LM in particular is advantaged; furthermore, we care about relative performance of the models and scaling trends, so our results are independent of whether GPT-4 generated questions are harder for smaller models. In section 4 we evaluate consistency on the existing PARAREL benchmark, in addition to our GPT-4 generated Scenarios. We observe the same trends on both these data sets, indicating that the GPT-4 generated data serves its purpose. 

You’re right that crowd-sourcing data can be a good alternative; hopefully our above response convinces you that using GPT-4 generated data (because it is cheaper and less time-consuming) doesn’t compromise the fairness of the data. This approach (“Model-Written evaluations”) was introduced by Perez et al [1].  We will add a clarifying note in the paper.

Reviewer comment (Theory):
The section about training a system how to lie is very interesting, but I am not sure if I would classify this as lying. The model is fine-tuned, e.g. trained to believe that a banana is green by reinforcement of a biased judge. I don't see how this is lying, if the system was trained to relate bananas to green.

Author response:
**Regarding the definition of lying:**

In the background (at the bottom of page 3) we give operational definitions of lying and deception which we use throughout the paper. We say “An LM is lying if its stated belief is false and is inconsistent with its accepted or revealed belief.” As discussed in the introduction, this follows Pacchiardi et al [2], who define lying in the context of LMs as “outputting false statements despite “knowing” the truth in a demonstrable sense.” We explicitly contrast lying with deception, by excluding the condition of intention to deceive. As discussed, a proper evaluation of LM intentions is out of scope.  

We hope it’s clear that our definition of lying excludes cases where the model is wrong / mistaken (because we require that it is shown not to believe the proposition in question). “Hallucination” is itself a vague term, it’s possible that there is some overlap with our definition of lying. In our GPT-4 examples, it’s more important that the model reaffirms the fabricated information, when we show that this is inconsistent with what it believes to be the case. 

**Regarding the fine-tuning:**

We agree that in your intuitive analogy this would not be lying. However, we think there is some confusion here. The fine-tuning task is to answer questions about the context (a short piece of text). The LMs learn to answer questions (either truthfully or not) about the context, and **we only claim that a model is lying if we demonstrate that its answer is inconsistent with its accepted or revealed belief.** In contrast, your analogy assumes belief in the “taught” proposition.

Furthermore, as discussed in the global response, we evaluate LMs on an unseen test set. **They learn to lie, in general, about facts not seen during training.** So the fine-tuning does not just get the models to internalize incorrect facts. In terms of your analogy, it’s more like the model was taught to say “bananas are red” and additionally learned to claim that London is the capital of France.

Reviewer comment (Reproducibility):
Please address my concern about generating the questions with GPT-4.

Author response:
**Regarding using GPT-4 to generate data**

Following the procedure from Perez et al ([1]), we conducted extensive data quality control, detailed in appendix A4. For example, when extending Multi-RC with GPT-4 generated questions about fruit, we first manually inspect 100 randomly sampled data points and verify that 1) the generated answer is correct, 2) the context is relevant to the question, 3) the question can be unambiguously answered from the context. Second, we created the evaluation set to ensure there are no duplicate examples between the fine-tuning and evaluation sets. Finally, we performed K-means clustering of vector embeddings of the data to check that the data are well distributed, and there are no duplicates within each cluster.