Paper: Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations (ICLR-2024)

Reviewer comment (Experiments):
There is a human study in the paper that does not mention whether the study has an IRB approval. Citing ICLR Code of Ethics `Where human subjects are involved in the research process (e.g., in direct experiments, or as annotators), the need for ethical approvals from an appropriate ethical review board should be assessed and reported.` **How to address: I recommend the authors to apply for an IRB approval from their respective institution to resolve this concern. Ideally this should have done prior to data collection, but I am leaving this to the judgment of Ethics Reviewers / ACs.**

Author response:
Q: Did you apply for IRB?

A: Yes we did! Our IRB was approved on 05/09/2023 prior to data collection. We did not submit the certificate due to anonymity constraints, but will include it if the paper is accepted.

Reviewer comment (Writing):
The terminological choice of "precision" looks very much like "faithfulness" (See Q1), which created significant confusion. The definition of precision in Section 3.2 is referred to as faithfulness in many other contexts (as in the cited related works). Unless there is a clear reason, it is unclear why a new name is being invented for it.

Author response:
Q: How is simulation precision different from “faithfulness”? Why “invent a new name” for simulation precision if it is just faithfulness?

A: We did not invent the term “simulation precision”. “Simulation precision” [1,2] and “faithfulness” [3,4,5] are two different terms __defined by prior work__, and our paper __explained their differences in Section 2 Paragraph 2__ and followed the use of their terminology. Both terms measure if an output predictor can guess the model’s output based on its explanation. The difference here is simulation precision requires the output predictor to be humans while faithfulness does not. Because of this difference, raw model weights are faithful because running the model weights as the output predictor yields perfect accuracy, but have zero simulation precision because humans cannot read model weights directly.

We focus on simulation precision because explanations need to be human-readable in order for humans to build mental models. Finally, to help clarify on the contribution of our paper, neither counterfactual simulatability or counterfactual faithfulness has been explored before for natural language explanations.

Reviewer comment (Evaluation):
Beyond the relative inference that a more capable model (GPT-4) provides `better` (under the paper's criteria) explanations than a less capable model (GPT-3), it is unclear whether the scores are good or not in an absolute sense, and thus whether the explanations are good or not (Q3). The absence of comparisons to interpretable numbers makes results hard to interpret. **How to address: Provide baseline explanations, e.g., by asking the same set of questions to human users, collecting their explanations, and comparing them with those from the models.**

Author response:
Q: How to understand the precision numbers “in an absolute sense”? Are human-written explanations more precise?

A: Following your advice, we __ran a new evaluation__ to measure the precision of human-written explanations. Just like how we scored LLM explanations, we asked a human annotator to write explanations for the same set of questions, used GPT-4 to generate counterfactual questions, asked the human annotator to answer the counterfactual questions, and scored how often the human annotator’s answer to the counterfactuals is consistent with the simulator’s answer. __Human-written explanations achieved a simulation precision of 95.0, much higher than the precision of LLM-generated explanations (80.6) with p-value < 0.001__. We’ve added this result to the pdf (Section 5.2.1 last paragraph).

Reviewer comment (Writing):
There is confusion about the proposed terminology. The authors argue that precision means `they should lead to mental models that are consistent with the model’s behavior,` but this sounds similar to faithfulness. In related works, faithfulness is described as `It is different from faithfulness, which measures whether an explanation is consistent with the model’s own decision process.` The reviewer personally finds "precision" to have connotations around explaining what could be explained and nothing more. Clarification is needed on why this is not simply faithfulness.

Author response:
Q: How is simulation precision different from “faithfulness”? Why “invent a new name” for simulation precision if it is just faithfulness?

A: We did not invent the term “simulation precision”. “Simulation precision” [1,2] and “faithfulness” [3,4,5] are two different terms __defined by prior work__, and our paper __explained their differences in Section 2 Paragraph 2__ and followed the use of their terminology. Both terms measure if an output predictor can guess the model’s output based on its explanation. The difference here is simulation precision requires the output predictor to be humans while faithfulness does not. Because of this difference, raw model weights are faithful because running the model weights as the output predictor yields perfect accuracy, but have zero simulation precision because humans cannot read model weights directly.

We focus on simulation precision because explanations need to be human-readable in order for humans to build mental models. Finally, to help clarify on the contribution of our paper, neither counterfactual simulatability or counterfactual faithfulness has been explored before for natural language explanations.

Reviewer comment (Theory):
The authors empirically argue that generality and precision do not seem correlated, but the reviewer does not necessarily agree and is unclear why an explanation must be general as long as it is precise. A highly local explanation could be extremely precise. For instance, using the formalization in the paper, $|C|$ could be large but $|C^*|$ small (even 0 or 1), e.g., if the model uses the answer to explain the answer. The reviewer questions whether these two desiderata are indeed independent and suggests that if they are not, the paper should include a discussion on this, particularly in Section 5.2.2, where independence is suggested.

Author response:
Q: Intuitively “one could make an extremely local explanation that is extremely precise”. What is reported in Table 5? Is it Spearman or Pearson? Are the numbers statistically significant?

A: We agree with your intuition that an extremely precise explanation can lead to low generality, which motivates our experiments to measure the correlation between generality and precision to see if there is any trade-off (Section 5.2.2 Paragraph 3). Table 5 shows near-zero Spearman correlations between precision and generality, and we observe similar near-zero correlations for Pearson (StrategyQA - BLEU: 0.023, Cosine: 0.003, Jaccard: -0.001; SHP - BLEU: 0.048, Cosine: 0.008, Jaccard: 0.013). We observe statistically significant correlations (p-value < 0.05) only on <3% of the examples where we calculated the correlations. Given these numbers, we conclude that there is no significant trade-off between precision and generality.

Reviewer comment (Evaluation):
It is unclear how one can claim that 80% precision is "not precise enough" (as per the claim `explanations have low precision (80% for binary classification)` in the introduction). The number seems nontrivial. Questions arise around why a higher number is expected, whether there are statistics on how precise human explanations typically are, and whether humans could even surpass this rate.

Author response:
Q: Why is 80% precision “not precise enough”?

A: Explanations are especially useful in high-stakes domains, where 80% precision is insufficient. Plus, human-written explanations have a precision of 95.0, indicating there could be a large room for improvement for LLM generations.

Reviewer comment (Evaluation):
The contents of Table 5 are unclear. Questions include: What is being reported? Is it the correlation coefficient? Is it Spearman or Pearson? What are the p-values? Are the numbers statistically significant?

Author response:
Q: It would be good to indicate in the caption of Table 5 what the numbers are.

A: We have updated the caption of the table in the pdf.

Reviewer comment (Presentation):
The inline citations are confusing. Replacing \cite calls with \citep calls could improve clarity in reading.

Author response:
Q: There are some issues with the citation format and two typos.

A: Thanks for catching these! We corrected the citation format and typos in the pdf.

Reviewer comment (Writing):
There is a typographical error: “Muslin countries” should be “Muslim countries” (Page 1).

Author response:
Q: There are some issues with the citation format and two typos.

A: Thanks for catching these! We corrected the citation format and typos in the pdf.

Reviewer comment (Experiments):
The evaluation of counterfactual simulatability is limited to classification tasks, and there is a need to extend it to more complex generation tasks.

Author response:
Q: How can we evaluate counterfactual simulatability on “more complex generation tasks”?

A: That’s an important but also challenging research question,  our paper discussed possible solutions in Section 6 Paragraph 1 as future work. The key difference between classification tasks and generation tasks is that multiple answers can be correct in generation tasks, so it is harder to define what it means for a human to “correctly” simulate the model’s output. One possible solution we proposed in the paper is contrastive simulation [5,6,7], where a human simulator is shown the model’s output mixed with fake outputs (distractors) and selects which output is from the model based on the model’s explanation.

Reviewer comment (Evaluation):
The human simulation task is complex and subjective, leading to only moderate agreement among human annotators, raising concerns about the reliability of human evaluation. That's extended to LLMs as well.

Author response:
Q: The human simulation task is complex and subjective, “leading to only moderate agreement among human annotators”. Is the human evaluation still “reliable”? Could you elaborate more on the "Forced" strategy you used for the sanity checks?

A: To address the moderate agreement rate, we ran a sanity check experiment in Section 5.1 to check whether our evaluation procedure can distinguish between explanation systems of different quality under the simulation noise. We constructed a baseline system “Forced” where we forced the model to generate a Post-Hoc explanation conditioned on the answer it assigns a lower score to, and compared its precision score to the “Normal” system where the model generates a Post-Hoc explanation conditioned on the answer it assigns a higher score to. “Normal” outperforms “Forced” significantly by +45.2 precision points (38.2 vs. 83.4; p-value < 1e-16), which verifies that our evaluation procedure can discriminate between explanation systems of different quality despite the simulation noise.

Regarding the fact that the IAA is only moderate: prior work also observed that __simulation is known to be inherently subjective__, and claimed that “variance in explanation ratings is quite high, relative to their scale” when they evaluated simulatability with humans [4]. However, despite the noisy simulation, our work and [4] are still able to __draw reliable conclusions that are statistically significant__ even from noisy human evaluation.

Reviewer comment (Theory):
The study assumes that language models have some form of "knowledge", which is not actually the case. Language models don't "know" or "understand" information in the way humans do - they generate responses based on patterns they've learned from training data.

Author response:
Q: The study assumes that language models have some form of "knowledge", which is not actually the case.

A: We do not assume that LLMs have any form of “knowledge”. Our evaluation procedure is behavioral, which only relies on examining the input-output behavior of the model.

Reviewer comment (Presentation):
Some citations do not follow the formatting instructions (When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{}... Otherwise, the citation should be in parenthesis using \citep{})

Author response:
Q: Some citations do not follow the formatting instructions.

A: Thanks for pointing this out! We have corrected the citation format in the pdf.

Reviewer comment (Evaluation):
The discussion of LLM simulators as proxies for human simulators is interesting, but it would be beneficial to provide more insights into the limitations of using LLMs in this role. What are the potential shortcomings or biases that LLM simulators may introduce? How well do they capture the full range of human reasoning and decision-making?

Author response:
Q: The discussion of LLM simulators as proxies for human simulators is interesting, but it would be beneficial to provide more insights into the limitations of using LLMs in this role.

A: Inspired by [8], one possible future direction is to study if GPT-4 can simulate each annotator even better when given a few annotations from that annotator to learn annotator-specific pattern/bias. Using one LLM simulator for each individual and aggregating the outputs from different simulators may further improve IAA.

Reviewer comment (Reproducibility):
Could you elaborate more on the "Forced" strategy you used for the sanity checks?

Author response:
Q: The human simulation task is complex and subjective, “leading to only moderate agreement among human annotators”. Is the human evaluation still “reliable”? Could you elaborate more on the "Forced" strategy you used for the sanity checks?

A: To address the moderate agreement rate, we ran a sanity check experiment in Section 5.1 to check whether our evaluation procedure can distinguish between explanation systems of different quality under the simulation noise. We constructed a baseline system “Forced” where we forced the model to generate a Post-Hoc explanation conditioned on the answer it assigns a lower score to, and compared its precision score to the “Normal” system where the model generates a Post-Hoc explanation conditioned on the answer it assigns a higher score to. “Normal” outperforms “Forced” significantly by +45.2 precision points (38.2 vs. 83.4; p-value < 1e-16), which verifies that our evaluation procedure can discriminate between explanation systems of different quality despite the simulation noise.

Regarding the fact that the IAA is only moderate: prior work also observed that __simulation is known to be inherently subjective__, and claimed that “variance in explanation ratings is quite high, relative to their scale” when they evaluated simulatability with humans [4]. However, despite the noisy simulation, our work and [4] are still able to __draw reliable conclusions that are statistically significant__ even from noisy human evaluation.

Reviewer comment (Experiments):
Could the authors elaborate on how the counterfactual questions were generated? Did they follow specific patterns or were they entirely randomly created?

Author response:
Q: Could the authors elaborate on how the counterfactual questions were generated?

A: We use LLMs to generate counterfactuals (Figure 2). To generate counterfactuals for each explanation from the QA model, we show the generator LLM the explanation and prompt it to generate ten follow-up questions for which it can confidently guess the QA model’s answer based on the shown explanation. We show the prompts we use in Appendix B. We show that our LLM-based generator outperforms the PolyJuice baseline on diversity (Section 5.1 Paragraph 4).

Reviewer comment (Evaluation):
Your evaluation method assumes that the knowledge contained in the explanation is the only information used by the model to make its decisions. How would your method account for scenarios where the model uses additional information not contained in the explanation to make its decision?

Author response:
Q: How would your metric account for scenarios where the model uses additional information not contained in the explanation to make its decision?

A: It depends on what information of the decision process is missing in the explanation. Explanations more specific than the model’s decision process (e.g., the model explains “animals with wings can fly” but uses the decision process “all animals can fly”) are penalized with a lower generality score. Conversely, explanations more generic than the decision process (e.g., the model uses the decision process “animals with wings can fly” but the explanation misses the modifier “with wings”) are penalized with a lower precision score.

Reviewer comment (Evaluation):
Within the same context, if GPT-4 exhibits a level of agreement with humans akin to that among humans themselves, it does not inherently establish its competence as an accurate representation of human cognition. An effective approximation should inherently exhibit a strong alignment with human behavioral patterns. In simpler terms, the findings merely attest to GPT-4's ability to replicate the same range of diversity observed among humans in their interactions with one another.

Author response:
Q: Re: GPT-4 as a proxy of human simulators, even though “GPT-4 exhibits a level of agreement with humans akin to that among humans themselves”, did you check if GPT-4 “exhibit[s] a strong alignment with human behavioral patterns”?

A: Following your advice, we conducted __an additional evaluation__ to test if GPT-4 has similar behavioral patterns as human simulators. Specifically, we studied if GPT-4 has higher agreement with humans on counterfactuals where human-human agreement is high. We measured the correlation between human-GPT-4 agreement and human-human agreement across 1532 counterfactual questions, and observed a strong correlation of Pearson coefficient +0.398 (p-value = 1e-58). This result shows that the GPT-4 simulator has some similar behavioral patterns as human simulators. We’ve added this result to the pdf (Section 5.1 Paragraph 3).

To strike a balance between scientific rigor and economic feasibility, our work __followed the practice from the prior literature__ to use LLMs for automatic evaluation [5,6,7], supported by the fact that LLMs do exhibit similar behavioral patterns as humans in many situations [8,9,10]; in fact, __we consider it necessary to use automatic evaluation to maintain rigor as well__, since using GPT-4 as the simulator for part of the experiments is necessary in order to be able to do all the robustness checks (e.g., making sure that our metric is sensible and can distinguish between good and poor explanation systems in Section 5.1 Paragraph 2) given a fixed budget. 

Finally, our paper also __used real actual human subjects__ for all experiments on StrategyQA (Section 5.2) and showed that all conclusions are equally valid.

Reviewer comment (Evaluation):
The conclusion of Table 3 is not correct. As shown in the table, GPT-mix demonstrates a higher similarity score compared with other models, therefore the conclusion should be that GPT-mix generates more relevant counterfactuals but not more diverse ones.

Author response:
Q: Table 3 shows that GPT-mix “demonstrates a higher similarity score compared with other models”, so the conclusion should be that GPT-mix does not generate “more diverse ones”.

A: We’d like to clarify some potential misunderstandings. The numbers in Table 3 are diversity scores, not similarity scores. We define diversity in Section 3.1 as one minus the expected similarity between any two simulatable counterfactuals, so Table 3 shows that GPT-mix generates more diverse ones.

Reviewer comment (Writing):
Typos: The 2nd paragraph of Section 3.1: "Expectation" doesn't match with the formula listed below.

Author response:
Q: Typo: The “expectation” in 2nd paragraph of Section 3.1 doesn’t match with the formula listed below.

A: Good catch! The expectation formula should state that x’ \neq x’’. We have updated this in the pdf.