Paper: ALMANACS: A Simulatability Benchmark for Language Model Explainability (ICLR-2024)

Reviewer comment (Experiments):
Only safety-relevant scenarios are included and this is not general.

Author response:
> Only safety-relevant scenarios are included and this is not general.

Our work contains 12 different scenario topics, and it has a total of 180,000 training examples and 18,000 test examples.

We are confused how this many different scenario topics and dataset examples is not general enough for ICLR. There are other natural language benchmarks published in ICLR that contain fewer topics and dataset examples. For instance:
- “WikiWhy: Answering and Explaining Cause-and-Effect Questions”. ICLR 2023 (notable top 5%). The dataset has 9,406 “why” question-answer-rationale triplets spanning 11 topics.
- “ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning”. ICLR 2020. The dataset has 6,138 questions (and no topic categories).

We hope that our work is judged consistently by the same standards as other papers.

Reviewer comment (Experiments):
The choice of language models for evaluation versus being explained may affect results. More analysis of this factor could be useful.

Author response:
> The choice of language models for evaluation versus being explained may affect results.

To test if our choice of GPT-4 for evaluation affected the results, **we ran additional experiments with two additional language models for evaluation. The results are in Appendix J.** We find that no combination of predictor/explanation, for either model being explained, performs significantly better than using GPT-4 for evaluation with no explanations provided, or than the logistic regression baseline. Consistent with our main results, this indicates that none of these predictors are able to extract helpful information from the explanations beyond what is present in the model’s answers.

Reviewer comment (Evaluation):
Automated evaluation using a language model proxy for humans has limitations vs. human studies. Direct comparisons would be needed to validate the benchmark.

Author response:
> Automated evaluation using a language model proxy for humans has limitations vs. human studies.

This is an important point; human studies are a valuable direction for future work. Nevertheless, even if an automated evaluation isn’t totally consistent with a human study, we still think automated evaluation has a valuable role to play. Because automated evaluation is an order of magnitude faster than human evaluation, it can provide quick, coarse-grained feedback throughout the interpretability researcher’s development cycle. A tool like ALMANACS is the difference between waiting weeks or months for the results of a human study, versus getting results of an automated benchmark in just a few hours.

Reviewer comment (Experiments):
Testing on more model sizes, scaling effects, and model families instead of just focusing on flan-alpaca-gpt4-xl and vicuna-7b-v1.3.

Author response:
> Testing on more model sizes, scaling effects, and model families instead of just focusing on flan-alpaca-gpt4-xl and vicuna-7b-v1.3.

**We have added scaling experiments to the paper, in Appendix G.** These experiments test models of a **variety of families with sizes ranging from 0.77B to 20B parameters on the advanced_ai_risk section of ALMANACS**. The benchmark difficulty, as measured by how well logistic regression is able to predict model behavior, is shown in Figure 10. We see a general trend that the behavior of larger models is harder to predict. The trend is more pronounced when we compare benchmark difficulty with model performance on a related task, with more capable models having behavior that is more difficult to predict.

Reviewer comment (Novelty):
Missing references for explanation distillation: Li et al. Explanations from Large Language Models Make Small Reasoners Better. 2022.

Author response:
> Missing references for explanation distillation:
>
> (1) Li et al. Explanations from Large Language Models Make Small Reasoners Better. 2022.

Thank you. We have added this reference to the paper.

Reviewer comment (Evaluation):
The observation that explanation techniques might not always heighten performance on unseen data is compelling, but the paper would benefit from a deeper analysis and discussion on the possible reasons behind this phenomenon.

Author response:
Thank you for this excellent suggestion. **In Appendix H, we have added additional analysis and discussion of possible reasons why the explanation methods are not improving performance.** Specifically, we provide examples of the explanation methods from our dataset, and we give a qualitative analysis of how much information is contained in the explanations. One insight is that to us, the authors of the study, the tokens with high salience scores do not intuitively suggest how the model will behave on new inputs. For example, the tokens are sometimes the question words, such as “would” and “choose”, rather than the details of the scenario. Another observation is that the language model’s rationalizations often discuss the specifics of the input scenario without discussing possible variations of the scenario. This might be limiting how much the rationalizations help the predictor at test-time, when the predictor is presented with scenario variations.

Reviewer comment (Writing):
The term "simulatability" appears to be inconsistently defined, leading to confusion. The initial definition of simulatability is "how well the explanations improve behavior prediction on new inputs". Subsequently, it seems to change to a definition centered on distribution distance, KLDiv or TVDist.

Author response:
It seems our original submission did not clearly present the KLDiv and TVDist metrics. “How well the explanations improve behavior prediction on new inputs” is indeed the definition of “simulatability” that we use throughout the paper. The key idea is that we are interested in distributions of behavior; we define “behavior” as the yes-no probability distribution that the language model gives to the scenario. We also have the predictor give a yes-no probability distribution as its prediction. To measure how good the predictor is, we therefore used the distribution distances KLDiv and TVDist.

Past work has considered binarized model output (taking just the most likely answer, either “yes” or “no”). This allows measuring simulatability with accuracy / F1 score. However, binarizing the model output is throwing away information. We choose to keep the entire distribution to create a more challenging benchmark, where the predictor must distinguish between models which answer “yes” 51% of the time versus models which answer “yes” 99% of the time.

**We have added a discussion of why we use distribution distances to Section 2.3: Evaluation Metrics.**

**In Appendix B, we also added a third quantitative metric, Spearman’s rank correlation coefficient**. We calculate the Spearman correlation between the predictor’s probability of “yes” and the language model’s probability of “yes” across all questions in a topic. This provides another way, different from distribution distance, to measure simulatability. In Appendix B, we see that Spearman correlation results are consistent with the KLDiv and TVDist results: no explanation method does substantially better than the explanation-free control.

Reviewer comment (Evaluation):
The paper doesn't provide a convincing argument for why simulatability is a good metric for language model explainability. Many factors can influence predictor outputs. Given that GPT-4 operates as a black box, it's hard to say GPT-4 predict solely to the presence or absence of explanations without providing additional constraints. This point is underscored by results from the NoExpl vs Expl comparison in Table 1, which indicates low KLDiv scores, hinting that GPT-4's predictions might be independent of input explanations.

Author response:
It’s important to know if GPT-4 has the ability to understand input explanations and change its predictions accordingly. To test this, in Section 4, we provide GPT-4 with explanations for a synthetic linear model. The explanations are hand-crafted to contain prediction-relevant information. **In Figure 4, we see that providing GPT-4 with the “qualitative” and “weights” explanations improves performance compared to the “NoExpl” no-explanation control.** Therefore, we have examples where we know that GPT-4’s predictions depend on the input explanations.

Reviewer comment (Writing):
Could the authors provide the precise definition of "simulatability"?

Author response:
It seems our original submission did not clearly present the KLDiv and TVDist metrics. “How well the explanations improve behavior prediction on new inputs” is indeed the definition of “simulatability” that we use throughout the paper. The key idea is that we are interested in distributions of behavior; we define “behavior” as the yes-no probability distribution that the language model gives to the scenario. We also have the predictor give a yes-no probability distribution as its prediction. To measure how good the predictor is, we therefore used the distribution distances KLDiv and TVDist.

Past work has considered binarized model output (taking just the most likely answer, either “yes” or “no”). This allows measuring simulatability with accuracy / F1 score. However, binarizing the model output is throwing away information. We choose to keep the entire distribution to create a more challenging benchmark, where the predictor must distinguish between models which answer “yes” 51% of the time versus models which answer “yes” 99% of the time.

**We have added a discussion of why we use distribution distances to Section 2.3: Evaluation Metrics.**

**In Appendix B, we also added a third quantitative metric, Spearman’s rank correlation coefficient**. We calculate the Spearman correlation between the predictor’s probability of “yes” and the language model’s probability of “yes” across all questions in a topic. This provides another way, different from distribution distance, to measure simulatability. In Appendix B, we see that Spearman correlation results are consistent with the KLDiv and TVDist results: no explanation method does substantially better than the explanation-free control.

Reviewer comment (Experiments):
The choice of GloVe embeddings for demonstration retrieval appears outdated. Have the authors considered more recent sentence embeddings, such as SimCSE?

Author response:
**We have added experiments with more recent sentence embeddings to Appendix I, including SimCSE and SentenceBERT.** With these new embeddings, we find that the baseline methods have improved performance. Therefore, **we have updated the main results tables (Tables 1, 2, and 3) to use the more recent embeddings.** Our overall pattern of results remains the same: even with new embeddings, no explanation method provides a substantial benefit over the no-explanation control.

Reviewer comment (Evaluation):
There seems to be a discrepancy between the comparison results for NoExpl in Figure 4 and Table 1 (PredictAverage vs NoExpl). PredictAverage outperforms NoExpl in Figure 4 but does not in Table 1. Could this be clarified?

Author response:
Thanks for bringing this to our attention.

PredictAverage depends only on the model being explained (predicting the average probability over the dataset). NoExpl, on the other hand, is GPT-4’s prediction when prompted with few-shot examples of the model’s behavior. We interpret this result as indicating that GPT-4 is better at in-context learning of other language models’ behavior than in-context learning of a synthetic linear model. **We have added this observation to the paper in the “No-explanation predictions” paragraph of Section 5: Results.**

Reviewer comment (Evaluation):
The paper only evaluates the explanation methods based on Kullback-Leibler divergence (KLDIV) and total variation distance (TVDIST). While these metrics provide insights into the performance of the methods, they may not capture all aspects of explanation quality.

Author response:
In Appendix B, we have added a third metric, Spearman’s rank correlation coefficient. We calculate the Spearman correlation between the predictor’s probability of “yes” and the language model’s probability of “yes.” This provides another way, different from distribution distance, to measure simulatability. In Appendix B, we see that Spearman correlation results are consistent with the KLDiv and TVDist results: no explanation method does substantially better than the explanation-free control.

Reviewer comment (Evaluation):
The paper acknowledges that the automated evaluation using language models may not be consistent with human evaluation. Human studies are still needed to validate the results and determine if humans can succeed where language models fail.

Author response:
This is an important point; human studies are a valuable direction for future work. Nevertheless, even if an automated evaluation isn’t totally consistent with a human study, we still think automated evaluation has a valuable role to play. Because automated evaluation is an order of magnitude faster than human evaluation, it can provide quick, coarse-grained feedback throughout the interpretability researcher’s development cycle. A tool like ALMANACS is the difference between waiting weeks or months for the results of a human study, versus getting results of an automated benchmark in just a few hours.

Reviewer comment (Experiments):
The paper evaluates only three explanation methods (counterfactual, rationalization, and salience-based). While these methods are commonly used in explainability research, there may be other methods that could be valuable to include in the benchmark.

Author response:
We have added an additional explanation method, Integrated Gradients. Integrated Gradients is a feature attribution method that is axiomatically motivated; it satisfies sensitivity and implementation invariance, and it is the unique path method that is symmetry preserving. (For details, see “Axiomatic Attribution for Deep Networks” by Sundararajan et al.) Integrated gradients was also one of the best-performing methods in Pruthi et al.’s explanation evaluation paper, “Evaluating Explanations: How much do explanations from the teacher aid students?”

In Table 1, we see that the Integrated Gradients results are consistent with the results for other explanation methods: Integrated Gradients does not improve simulatability over the no-explanation control.