Paper: Language Models Struggle to Explain Themselves (ICLR-2024)

Reviewer comment (Evaluation):
The freeform articulation test appears brittle to me. Even though an articulation does not match the ground truth rule designed in the test, if the articulated rule could also explain the same set of examples, then it would not be correct to label this as an inaccuracy, in my opinion. There may be multiple correct strategies, and the model may be using it. This comment is related to W2.

Author response:
> “The freeform articulation test appears brittle to me. Even though an articulation does not match the ground truth rule designed in the test, if the articulated rule could also explain the same set of examples, then it would not be correct to label this as an inaccuracy, in my opinion. There may be multiple correct strategies, and the model may be using it. This comment is related to W2.”

> I would be curious to see whether the articulated explanations that are labeled as incorrect could also explain the provided examples. Specifically, what fraction of the ~30% of freeform evaluations provided by GPT-4 that are not aligned with the ground truth rule could also explain the provided in-context examples? This could be an easy test that could be evaluated by another LLM in the loop (give the articulation, and ask whether it could explain the in-context example). Alternatively, this could also be evaluated by hand since it will be a few tens of examples.

Just to reiterate the point made above: indeed, we agree this is possible and have accounted for it in both the multiple choice and freeform articulation tasks. In practice, we didn’t find a single instance of this happening in hundreds of manually reviewed freeform articulations, and avoided this edge case entirely by construction in the multiple choice articulation task.

Reviewer comment (Experiments):
The above two weaknesses are also made more brittle by the low sample size ($<256$). This also makes the finetuning results somewhat brittle – authors finetune models with billions of parameters with tens of examples, and this alone may contribute to why finetuning does not improve articulation accuracy.

Author response:
> The above two weaknesses are also made more brittle by the low sample size (< 256). This also makes the fine tuning results somewhat brittle – authors finetune models with billions of parameters with tens of examples, and this alone may contribute to why finetuning does not improve articulation accuracy.

We think this is an important misunderstanding. We trained GPT-3 on 100’s of classification and articulation demonstrations, not 10’s. We also ensured our approach was consistent with OpenAI’s best practices for fine-tuning [1]. However, the training set _is_ generated by 10’s of rule functions which is a limitation (see the “Increase the variety of articulations” dot point in Section 3 and “Finetuning on freeform articulations” in Section 4).

[1] https://platform.openai.com/docs/guides/legacy-fine-tuning/classification

Reviewer comment (Experiments):
Authors decided to introduce their test bed by evaluating black-box LMs. However, models with white box access (e.g., Pythia [https://arxiv.org/abs/2304.01373]) or Llama would provide more insights, e.g., by investigating input attribution. But even with only black-box access, input feature attribution can be generated, e.g., using perturbation methods or Contrastive Input Erasure [https://arxiv.org/pdf/2202.10419.pdf]. Further open LMs would provide greater reproducibility.

Author response:
We agree the corresponding experiments on white box models would be interesting. We plan on releasing our dataset so that such experiments can be run by others in the community. In this work, we elected to first evaluate OpenAI models to determine where the current SOTA articulation capabilities were. 

However, in response to similar comments by other reviewers, we are in the process of running some evaluations on the Llama series of models, results for which should be ready in the next few days. We will share the results of these in a top level comment.

We also take the point that white box methods could be illuminating. We too thought so, and discussed some potential avenues in this vein in the future work sections. We chose not to explore these due to time constraints, and these experiments falling out of the scope of our key contributions. Thanks for the comment on perturbation methods - these too would be interesting to explore, and we have added this to our future work section.

Reviewer comment (Presentation):
Tables and figures exceed the paper margins.

Author response:
Thanks for raising this point. We have fixed all of these issues.

Reviewer comment (Evaluation):
The paper states, "We evaluate to what extent LLMs are capable of producing faithful explanations of their own behavior via finetuning." However, the faithfulness of the explanation is largely ignored in the present evaluation. See e.g. https://arxiv.org/pdf/1911.03429.pdf.

Author response:
Thanks for suggesting this paper (https://arxiv.org/pdf/1911.03429.pdf) which we have added to related work. This paper studies faithfulness in rationales, a specific form of explanation. A rationale is considered faithful if it actually informs model predictions. One way to measure is to manipulate an input to remove the rationale (“comprehensiveness”).  


We understand “faithfulness” in a similar way. An articulation of a rule is faithful if it actually informs the model’s classifications on held-out examples. If a model articulates a rule R but is shown to classify held-out examples consistent with a distinct rule S, then this articulation is necessarily an unfaithful explanation. This is what we actually find for GPT-3. Specifically, GPT-3 produces freeform and multiple choice articulations of rules that do not match its classification behavior on held-out examples. Thus, GPT-3 fails to be faithful, even after finetuning.

GPT-4, by contrast, achieves significantly higher accuracy in articulating rules that match its classification behavior (~70% top-3 freeform articulation accuracy for GPT-4 vs ~10% for GPT-3). This suggests that GPT-4 may be “capable of producing faithful explanations”, although it still fails on ~30% of rules. As we discuss in the limitations section, there is a question of whether even 100% accurate articulation of rules is sufficient condition for faithfulness. It could be that the articulated rule describes the model’s behavior but does not causally influence its predictions. There are various approaches to testing this (e.g. manipulating internal representations in whitebox models). We think these are important questions for future work in this area. 

We hope we have clarified how our results relate to faithfulness. We have amended the paper to include these points about faithfulness in the introduction, conclusion and results sections.

Reviewer comment (Presentation):
Sec 2.2.1 seems to be the only subsection of 2.2. It also seems to be misplaced. It rather fits the 2.4. Same for 2.1. In Section 2, the authors could distinguish between the test-bed, including the tasks, and the dataset.

Author response:
Thanks for this. We have added a paragraph at the start of section 2 explaining on a high level the tasks and datasets.

Reviewer comment (Writing):
"Curie showed the first signs of life." Try to avoid such phrases.

Author response:
Thanks. We have also fixed all of these stylistic issues.

Reviewer comment (Presentation):
Remove figure references in the description of the main finding in the introduction since they interrupt the reading flow. Place Figure 2 as the first figure since it summarizes the methodology of the paper quite well and is more helpful than Figure 1 at the beginning of the paper. Current Figure 1 should be placed in the results.

Author response:
Thanks. We have also fixed all of these stylistic issues.

Reviewer comment (Presentation):
Captions should be left aligned.

Author response:
Thanks. We have also fixed all of these stylistic issues.

Reviewer comment (Evaluation):
While the insights provided in the paper are interesting, it is not clear to me what the benefit of self-explaining capabilities as defined in the paper is. As you briefly discuss in the limitation section, if a model can self-explain itself as the present paper defines it, it is still unclear if these explanations are faithful, i.e., corresponding to the reasons used in the model's internal process. Same for the opposite case. If it can not self-explain as defined in the paper, it does not prove that it is actually not using the right reasons. It would be interesting to see a relation between, e.g., the quality of feature attribution-based explanations and the investigated self-explain capabilities of a model.

Author response:
Thanks for pointing this out. Self-explanation is a desirable property for models to have, in general. As we noted above, we view high accuracy on our benchmark to be a necessary, but not sufficient test of language model self explanation. We chose such an approach as procedurally generated black box behavioral evaluations provide a cheap, fast and scalable test of model capabilities, while still being informative. 

We agree that studying the relation between explanation capabilities and feature attribution-based methods would be informative. We have listed this as a possible area of future work.

Reviewer comment (Theory):
The major concern is that the authors claim or speculate that GPT-4 has over 175 billion parameters. However, even during the paper review period, the exact size of GPT-4 remains unclear.

Author response:
Thanks for noting this. This is a good point and such speculation is unnecessary. We’ve removed the assumption that GPT-4’s parameter count is greater than 175B as part of the above change.

Reviewer comment (Evaluation):
Even if it is confirmed that GPT-4 has over 175 billion parameters, this alone does not lead to the conclusion that "articulation accuracy increases with model size." The reason is that the differences between the models shown in Figure 1 are not only in their number of parameters, but also in other significant factors like training data. Simply comparing model sizes does not account for these other variables that likely also impact articulation accuracy.

Author response:
Agreed; this is a good point and is well-taken. We’ve replaced the claim about articulation accuracy increasing with model size with a more nuanced claim which acknowledges that multiple factors other than model size (e.g. the training set or model architecture) play a role in the the increased general capabilities of GPT-4, and of the observed increase in articulation accuracy. The motivation behind studying a series of models in this manner was to establish whether articulation capabilities would arise by default with scale.

Reviewer comment (Evaluation):
The ArticulateRules dataset is relatively small and simple, focusing on text classification tasks based on simple syntactic rules. Performance on this limited dataset may not reflect abilities on more complex real-world tasks.

Author response:
Thank you for raising this point. We agree that high articulation accuracy on ArticulateRules does not correspond to high articulation accuracy on complex real-world tasks. However, we would like to clarify that we’re not claiming that an increase in articulation accuracy on ArticulateRules implies an increase in articulation accuracy on more complex real-world tasks, and this is not the goal of our work. Rather, we’re claiming that high articulation accuracy on this evaluation is a necessary but not sufficient condition for an LLM to be able to articulate its behavior on more complex real-world tasks.

> ”The ArticulateRules dataset is relatively small and simple…”

Thank you for pointing out the need for more exposition regarding our specific dataset construction. We’ve updated the paper to provide more clarity about the design decisions around the size of the dataset and the simplicity of the rules it includes.

** Size **

The size of the dataset is constrained by compute costs. We assume that a cost of $10’s or $100’s to reproduce our in-context evaluation results is affordable for most labs, and similarly a cost of $1,000’s for our finetuning results. Therefore, if the smallest variant of ArticulateRules exceeded these budgets, it would risk being too costly to run or build upon in practice.

We determined the minimum size of ArticulateRules in the following way: (1) we determined the minimum number of few-shot examples to include in the prompt in order for a human to be able to articulate the rule (we found this to be 16 few-shot examples; more details in Appendix 2.3), (2) we determined the number of examples per rule function needed to have an acceptably-low standard error of the mean for in-context tasks (we found this to be roughly >= 20 and landed on 32 examples per rule function; Figure 11 uses n=15 and has a large SEM, compared to Figure 5 which uses n=32 and has an acceptable SEM to be able to draw conclusions), (3) we then used the maximum number of rule functions given these constraints.

Larger dataset variants are also provided for labs with larger compute budgets. Also, it’s worth noting that the dataset size is easily increased: (1) it is procedurally generated, meaning that larger datasets can be generated as needed, and (2) rule functions are constructed from more primitive ones using AND, OR and NOT operators, meaning it’s straightforward to register more complex rule functions as needed.

** Simplicity **

We chose tasks where we could: (1) closely approximate the model’s behavior with a known rule (and therefore be able to determine if an articulation was correct or not), and (2) get an accurate picture of the limits of present-day LLMs’ articulation capabilities.

The binary text classification tasks used in ArticulateRules fit both of these criteria. Using easier tasks where LLMs achieved 100% freeform articulation accuracy wouldn’t have given an accurate picture of their limitations. Using harder tasks where LLMs get 0% accuracy would have been equally as uninformative. We tailored these tasks so that present-day LLMs are capable of articulating only a small fraction of the rules in the dataset, which gives a picture of their current articulation capabilities and its limitations, and allows us to track increases in articulation capabilities over time.

We’ve added these details to Appendix B.6 since we agree that the size of the dataset and the complexity of the tasks it includes requires justification.

Reviewer comment (Experiments):
Only one main model architecture (GPT-3/4) is tested in detail. Other model types may have different explanation capabilities that are not explored.

Author response:
Thanks for raising this concern. Our decision to focus on GPT-3/4 was primarily driven by time and cost constraints, however we acknowledge that evaluating LLMs with different architectures would be a valuable experiment to run. In particular, conducting a finetuning evaluation of GPT-3 traded-off against evaluating a wider range of LLMs in-context. We decided that approximating the upper bound of articulation capabilities of GPT-3 (the largest model available with finetuning access) was the more informative experiment to run.

In response to this feedback, we’ve also started evaluating Llama 2 in-context and are planning to have results in the next few days, results of which we will post in a top-level comment.