Paper: Explaining black box text modules in natural language with language models (ICLR-2024)

Reviewer comment (Experiments):
There is no ablation study on the corpus size and the effect on explanation scores. Ideally the authors could explore the outcome using for example 10k, 20k, 40k, 80k, .... all n-grams.

Author response:
| 1. Corpus-size ablation

Thanks for this suggestion, we have added a new Fig A2 which uses the WikiText corpus (which contains 9,822k unique trigrams) and randomly subsamples its ngrams. Performance improves sharply from 10k to 50k and plateaus around 100k.

Reviewer comment (Evaluation):
The scoring step is suspect and introduces an unnecessary confound in terms of procedures. It seems incorrect to evaluate \( E(f(\text{Text}^{+})-f(\text{Text}^{-})) \). In this case, the language model is being asked to generate phrases unrelated to an explanation, so the final score depends not only on how good the positives are, but also on how "good" the negatives are. A more correct step might be \( E(f(\text{Text}^{+})-f(\text{Text}^{\text{all}})) \). This issue is less problematic if used internally, but is problematic when used as a point of comparison in Table 4, Figure 3, Table 6, fMRI experiments, Table A4, and Table A8.

Author response:
| 2. Scoring step, E[f(Text+)-f(Text-)] vs E[f(Text+)-f(Text^all).

Thanks, we agree using Text- can add more noise to the procedure and have added this note in the methods:

“The unrelated synthetic text should be neutral text that omits the relevant explanation, but may introduce bias into the scoring if the helper LLM improperly generates negative synthetic texts. Instead of synthetic texts, a large set of neutral texts may be used for Text−, e.g. samples from a generic corpus.”

Additionally we find that the empirical difference between using Text- and Text^All is minimal. Specifically, we add Fig A5, which shows results for different modules when evaluated on E[f(Text+)], E[f(Text+)-f(Text-)]. It also shows E[f(Text^All)] (the off-diagonal elements of Fig A5 left subpanel). We find that in this setting, there is essentially no difference between E[f(Text^All)] and E[f(Text-)] (the difference between the two is 0.02$\sigma_f \pm 0.1\sigma_f$). 

As LLMs improve, we do believe E[f(Text-)] will be a better choice than E[f(Text^All)], as Text- can correctly omit samples that should be in Text+, but otherwise stay neutral.

Reviewer comment (Evaluation):
It is questionable that the authors use manual inspection to evaluate their method. The paper and supplemental materials do not mention using Amazon Mechanical Turk or Prolific, nor provide details on this human study. It appears the experiment may have been conducted by the authors themselves, which is prone to bias. A more accurate approach would be to use a different text embedding model (e.g., bge-large) to compute the n-way cosine similarities among the 54 models and the 54 explanations, then compute the argmax agreement (whether the n-th explanation generated by SASC is cosine similarity maximized by the n-th model), or use cosine distance when n-way classification is not appropriate.

Author response:
| 4. Replacing manual inspection with text-embedding model

We have taken additional steps to mitigate the potential bias of our manual inspection:

First and foremost, we have added BERT score, a more objective similarity measure (similar to the cosine distance you mention), along with manual-inspection accuracy in Table 1 and Table 2. We find that the two metrics, when averaged over the 54 explanations, have a perfect rank correlation, i.e. every increase in average accuracy corresponds to an increase in average BERT score. Note that we compute BERT scores using the fairly capable deberta model recommended by the BERT-score repo (microsoft/deberta-xlarge-mnli).

We also add Fig A4 showing the similarities computed in the Default setting using the bge-large model as you suggest. A clear diagonal pattern is visible, as one would expect based on the other evaluations. Taking the 54-class classification accuracy using the argmax on these similarities results in an accuracy of 81.5%, lower than the manual inspection accuracy of 88.3%. This is not too surprising as many datasets have very similar groundtruth explanations, e.g. math/statistics, and a human may mark both as accurate. When we compute the top-2 accuracy of the bge-large similarities, the accuracy rises to 88.9%, now slightly higher than the manual inspection accuracy.

We also add more information on the manual inspection, showing manual classifications in Table A6-A8 and showing regexes that guide the manual inspection in Table A3.

Reviewer comment (Reproducibility):
The clarity of some specific methods is poor, with missing details in both the main text and supplemental. Specifically:  

a. The ngram summarization baseline used in Table 1 is unclear. References (Kadar et al. and Na et al.) do not actually perform ngram summarization: the first uses an omission-based approach, and the second uses a parsing + text replication approach.  

b. The method for obtaining the Spearman rank correlation SEM is not described. Given the low correlation, details such as whether the SEM is computed via bootstrap and whether a t-statistic could be reported instead should be provided.

Author response:
| 5. Method clarity

Apologies for the confusion. We have cleaned up some of the writing; in particular, we have added a “Baselines and evaluation metrics” header in Section 3. The first paragraph of this section describes the baseline.

Indeed, neither Kadar et al. nor Na et al. perform ngram summarization, but these references are the closest we can find to giving natural-language explanations of black-box text modules by summarizing ngrams; we updated their description in the related work:

| 5b . Spearman rank sem

The sem (Sec A.4.2) is computed across the rank correlation for the 1,500 voxels, so it is simply the standard deviation of the 1500 correlations (one for each voxel) divided by $\sqrt{1500}$. This can be translated into a t-statistic by simply dividing the mean by this sem, yielding $t=0.033/0.002=16.5$.

Nevertheless, we agree the correlation is quite low when making predictions as described in A.4.2; in a followup study we are seeing much stronger results when testing these explanations on unseen data in a more natural way (which requires followup fMRI experiments).

Reviewer comment (Experiments):
The proposed method may only work on semantic selectivity, and it is unclear if it can work on low-level text patterns. While this limitation may be acceptable, it should be more clearly discussed in the paper.

Author response:
| 6. Limiting to semantic selectivity

We agree and have added this sentence to the Discussion: “Additionally, due to its reliance on ngrams, SASC fails to capture low-level text patterns or patterns requiring long context, e.g. patterns based on position in a sequence”

Reviewer comment (Evaluation):
How do the authors decide whether an explanation is correct or not? For example, for Table 3, why ‘facts’ is considered to be equal for ‘information or knowledge’, while ‘language’ is considered to be unequal to ‘ungrammatical’? What is the criteria to decide if an explanation is correct or not?

Author response:
The criteria is judgement by manual inspection. As this may be subject to bias, we have added BERT scores to Table 2 in addition to Table 1 and explicitly show the judgements made (e.g. Table A6, A7, A8). We find that the average BERT scores have a perfect rank correlation with average accuracy, i.e. every increase in average accuracy also yields an increase in average BERT score. We have added this quote to the methods section to clarify this:

“We evaluate similarity of the recovered explanation and the groundtruth explanation in two ways: (1) Accuracy: verifying whether the ground truth is essentially equivalent to the recovered explanation via manual inspection and (2) BERT-score (Zhang et al., 2019)4. We find that these two metrics, when averaged over the datasets studied here, have a perfect rank correlation, i.e. every increase in average accuracy corresponds to an increase in average BERT score.”

Reviewer comment (Evaluation):
It would be great if the authors could give some hints in scenarios where the proposed SASC algorithm does not have satisfactory results. Whether the reason is due to the limitation of the ngrams or the limitations of the helper LLMs.

Author response:
Empirically, we find that when using GPT-3, the helper LLM is rarely the source of explanation errors (smaller LLMs do introduce errors, see Fig A1). Rather, the source of error is usually SASC’s reliance on ngrams. This is the case for explanations that may require long context (e.g. a module that responds to subjectivity), or that depend on positional information. We have added this comment to the Discussion to clarify this: “Additionally, due to its reliance on ngrams, SASC fails to capture low-level text patterns or patterns requiring long context, e.g. patterns based on position in a sequence.”

Reviewer comment (Writing):
It would be great if the authors could clarify in more detail what does ‘transformer factors’ mean for BERT in Section 4. Additionally, what are the ground truth for these transformer factors? How is the SASC win percentage: 61% calculated?

Author response:
We have added some more details on transformer factors to the main text and section A.3. Here are the relevant quotes from the main text:

Transformer factors: “we choose to interpret transformer factors, following a previous study that suggests that they are amenable to interpretation (Yun et al., 2021). Transformer factors learn a transformation of activations across layers via dictionary learning (details in Appendix A.3; corpus used is the WikiText dataset (Merity et al., 2016)). Each transformer factor is a module that takes as input a text sequence and yields a scalar dictionary coefficient, after averaging over the input’s sequence length. There are 1,500 factors, and their coefficients vary for each of BERT’s 13 encoding layers.”
Groundtruth: “In the absence of ground truth explanations, we evaluated the explanations by (i) comparing them to human-given explanations and (ii) checking their relevance to downstream tasks.” 
Win percentage: “Win percentage shows how often the SASC explanation yields a higher explanation score than the human explanation.”

Reviewer comment (Experiments):
What GPT-3 is used as the helper LLM, rather than a more advanced model, such as GPT-3.5?

Author response:
Thanks for this suggestion. We have a limited budget and GPT-3 was cheap and sufficiently effective for this task. GPT-3 was also easier to compare to smaller models, such as text-babbage-001 & text-curie-001 (Fig A1). Before camera ready, we will re-run the main experiments with the best available closed-source model (GPT-4 or better).

Reviewer comment (Evaluation):
Another interesting finding in Figure 3 is that, as the BERT layer increases, the explanation score tends to decrease. Could the authors give some hints on why this happens?

Author response:
Indeed this is interesting. We find that this is a consistent trend across interpretability works where higher layers tend to represent more abstract concepts which are harder to succinctly explain.

Reviewer comment (Experiments):
The paper does not compare SASC to other state-of-the-art methods for generating natural language explanations for black box models. It would be useful to compare SASC to other methods, such as LIME and SHAP, to determine how it performs in comparison.

Author response:
Apologies for any confusion caused by the writing. SASC and post-hoc interpretation methods such as LIME/SHAP have different goals:
|             | SASC                                                 | LIME/SHAP                                                  |
| ----------- | ---------------------------------------------------- | ---------------------------------------------------------- |
| Output      | Natural-language explanation                         | Feature importance scores                                  |
| Granularity | Module                                               | Individual prediction                                      |
| Goal        | Describe what elicits strongest response from module | Describe contributions of different inputs to a prediction |

They are useful in different scenarios. If one wants to compare them directly, we can take the highest-scoring inputs for each prediction and then try to summarize them into natural language. This is what our ngram summarization baseline in Table 2 aims to do, and we find that it performs decently, but not as well as SASC.

We have edited the writing to clarify this point in the first paragraph of section 2 and added this note in the related work section:


“Besides natural language explanations, some works explain individual prediction via feature importances (e.g. LIME (Ribeiro et al., 2016)/SHAP (Lundberg et al., 2019)), feature-interaction importances (Morris et al., 2023; Singh et al., 2019; Tsang et al., 2017), or extractive rationales (Zaidan & Eisner, 2008; Sha et al., 2021). These works are not directly comparable to SASC, as they work at the prediction-level and do not produce a natural-language explanation.”

Reviewer comment (Theory):
In SASC design, the first step is to input the n-grams from the reference corpus into the text module, which could contain many instances. How to ensure the efficiency of the proposed method remains undiscussed. It would be better for authors to provide a detailed analysis of the computational complexity of SASC. It would be useful to provide information on the computational requirements of the proposed method and potential ways to optimize it.

Author response:
Thanks for this comment. We have added this paragraph on computational complexity as the third paragraph in the methods section:

“The computational bottleneck of SASC is computing f ’s response to the corpus ngrams. This computation requires two choices: the corpus underlying the extracted ngrams, and the length of ngrams to extract. Using a larger corpus/higher order ngrams can make SASC more accurate, but the computational cost grows linearly with the unique number of ngrams in the corpus. The corpus should be large enough to include relevant ngrams, as the corpus limits what generated explanations are possible (e.g. it is difficult to recover mathematical explanations from a corpus that contains no math). To speed up computation, ngrams can be subsampled from the corpus.”

Additionally, we have added new experiments in Fig A2 showing computational tradeoffs between the number of ngrams and the performance; we find that increasing the number of ngrams beyond 100k no longer improves performance.

Reviewer comment (Experiments):
The design of synthetic scoring is heuristic and without any theoretical analysis to support this design. And most importantly, there is no ablation study to verify the effectiveness of this model design.

Author response:
Indeed it is difficult to develop rigorous theory for an LLM-based method like SASC. As a result, we do perform ablations in Table 1 and Fig 2 showing the importance of the synthetic scoring: Table 1 compares SASC to ngram summarization without synthetic scoring, finding that the synthetic scoring yields an improvement of >16% accuracy on average. Moreover, Fig 2 shows that accuracy at recovering the ground truth explanation increases as a function of the synthetic score. Thus, selecting the best explanation based on synthetic score improves performance.

We have made these ablations clearer in the updated text (see the new “Baselines and evaluation metrics paragraph” in Section 3).

Reviewer comment (Experiments):
Can you provide a comparison of SASC to other state-of-the-art methods for generating natural language explanations for black box models? This would help readers understand how SASC performs in comparison to other methods.

Author response:
Apologies for any confusion caused by the writing. SASC and post-hoc interpretation methods such as LIME/SHAP have different goals:
|             | SASC                                                 | LIME/SHAP                                                  |
| ----------- | ---------------------------------------------------- | ---------------------------------------------------------- |
| Output      | Natural-language explanation                         | Feature importance scores                                  |
| Granularity | Module                                               | Individual prediction                                      |
| Goal        | Describe what elicits strongest response from module | Describe contributions of different inputs to a prediction |

They are useful in different scenarios. If one wants to compare them directly, we can take the highest-scoring inputs for each prediction and then try to summarize them into natural language. This is what our ngram summarization baseline in Table 2 aims to do, and we find that it performs decently, but not as well as SASC.

We have edited the writing to clarify this point in the first paragraph of section 2 and added this note in the related work section:


“Besides natural language explanations, some works explain individual prediction via feature importances (e.g. LIME (Ribeiro et al., 2016)/SHAP (Lundberg et al., 2019)), feature-interaction importances (Morris et al., 2023; Singh et al., 2019; Tsang et al., 2017), or extractive rationales (Zaidan & Eisner, 2008; Sha et al., 2021). These works are not directly comparable to SASC, as they work at the prediction-level and do not produce a natural-language explanation.”

Reviewer comment (Reproducibility):
Can you provide more information on the computational complexity of SASC? Specifically, how long does it take to generate explanations for a given module and what are the computational requirements of the proposed method? This would help readers understand the practical limitations of SASC and potential ways to optimize it.

Author response:
Thanks for this comment. We have added this paragraph on computational complexity as the third paragraph in the methods section:

“The computational bottleneck of SASC is computing f ’s response to the corpus ngrams. This computation requires two choices: the corpus underlying the extracted ngrams, and the length of ngrams to extract. Using a larger corpus/higher order ngrams can make SASC more accurate, but the computational cost grows linearly with the unique number of ngrams in the corpus. The corpus should be large enough to include relevant ngrams, as the corpus limits what generated explanations are possible (e.g. it is difficult to recover mathematical explanations from a corpus that contains no math). To speed up computation, ngrams can be subsampled from the corpus.”

Additionally, we have added new experiments in Fig A2 showing computational tradeoffs between the number of ngrams and the performance; we find that increasing the number of ngrams beyond 100k no longer improves performance.