Paper: Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR-2024)

Reviewer comment (Writing):
The description of the proposed method in Section 3 is confusing and not easy to understand. The authors should rephrase Section 3 with a general description of the proposed causal model.

Author response:
We thank the reviewer for the feedback and advice. 
As recommended by reviewer GEWc, we have introduced two subsections to the appendix including a detailed description of the causal estimation pipeline (G.1) and the evaluation pipeline of explanation methods (G.2).

Reviewer comment (Theory):
About Eq.(2), the authors use the difference in the model's prediction before and after the treatment as the treatment effect, which, in the reviewer's opinion, is not robust when the model's output confidence is flat (or the uncertainty is high). This will affect the method's performance on small models like BERT.

Author response:
**Regarding Eq.2:**

This is a smart take. Eq.2 defines an estimator for the ICaCE, which is the causal effect of a concept on the classifier prediction. The estimator is the difference between predictions of an example and its approximated counterfactual (before and after the treatment). Notice that if the expectation of the approximation error is zero, then the estimator is unbiased and measures accurately the ICaCE (see our proof in Appendix A).

We agree that if the outputs of the classifier are flat (or have a high uncertainty), then the difference is small (or has a small norm in case it is a vector). However, so is the real ICaCE - it is not a limitation or weakness of the estimator, but the natural behaviour of the causal effect (see Goyal et.al., 2020)
For example, when the classifier is random and assigns the same values for every example, then the causal effect would be 0 (i.e., the concept has no effect on the prediction), the same as the estimator. Nevertheless, we agree that when the uncertainty of the model is high, the causal effect is low, and the estimator and the results might be noisy. For this reason, we conducted multiple experiments and explained various models of different sizes and architectures.

Reviewer comment (Writing):
The reviewer cannot find an accurate definition of "causal model." They ask whether the authors use the representation generated by a language representation model optimized with Eq.(5) with generated counterfactual and matched examples, then calculate the matching value in Eq.(4), and use this value to calculate Eq.(2) as a causal model.

Author response:
1. Your description of the estimation of the causal effect is accurate. We use the term “causal model” as a notation for “the causal representation model optimizing Eq.6”. Given an example, we first extract the representations of this example using the causal model. We then also extract the representations of the candidate set (this is done once at training since the candidate set does not change). We then find a match using Eq.4. After finding the match, we estimate the causal effect using Eq.2. 
Aspiring by your questions, we will include a page in the appendix and thoroughly detail the matching, estimation, and evaluation pipelines.

Reviewer comment (Theory):
The reviewer asks how the generative approach works, and if the assumption in Point 3 is true, whether the causal model includes the generative approach.

Author response:
2. According to the generative approach, we prompt an LLM to generate a counterfactual, and then use Eq.2 to measure the causal effect. 
When we write “causal model” we mean the causal representation model, which is used for matching (see the first question). 
Notice that we do use an LLM to generate counterfactuals for training the causal model. However, we do not generate any counterfactual at inference time when finding a match.

Reviewer comment (Reproducibility):
The reviewer asks which version of ChatGPT (GPT-3.5 or GPT-4) the authors used in the experiments.

Author response:
3. For CEBaB: We use GPT-3.5-turbo for generating all the counterfactuals, for both the generative approach (zero-shot and few-shot, Table 2) and for generating CFs for training the causal model (matching approach). 
For the new Stance Detection dataset: We use GPT-4 for constructing the new dataset (see section 5.1 and Appendix D), but as with CEBaB, we use GPT-3.5-turbo for the generative approach and for training the causal model that we benchmark with the new dataset. We will clarify this in the final version.

Reviewer comment (Evaluation):
The first method proposed, Counterfactual Generation, is computationally expensive and may be infeasible in scenarios requiring real-time explanations.

Author response:
**Regarding the cost of counterfactual generation:**

We agree with this comment and we actually discuss this issue in section 3 and Appendix B. Regardless, this approach may be used in high-stakes applications where accuracy is crucial. For other cases, we develop our efficient matching method, which is much faster and computationally cheaper. We would also like to refer the reviewer to the fact that the costs of strong LLMs like GPT-4 are becoming constantly cheaper (e.g. the recent announcement by OpenAI, which cut these costs by a half).

Reviewer comment (Experiments):
Although Matching is faster than CF Generation, it might not be as accurate for all situations, especially when the matching candidate set does not sufficiently represent the input data. It would be great if the authors performed some ablation studies (reducing the quality of the matches in the candidate sets to show how much the performance degrades).

Author response:
**Regarding ablations and the quality of the candidate set:**

The performance of matching methods indeed depends on the quality of the candidate set. Notice that we conducted extensive ablation experiments to explore the quality of the matching method as a function of the candidate set quality. While these experiments are in the appendix, they are referred to from the main paper.

1. In the ablation study (Appendix C, referred to from pages 5 and 7), we evaluated different models across three types of candidate sets: (a) the standard candidate set (randomly sampled from the training set), (b) the standard set augmented by ground-truth CFs, and (c) the standard set augmented by misspecified CFs (designed to challenge the matching model). The findings demonstrate that our full objective is the most robust across these variations, showcasing the adaptability of our method. This experiment underscores a key aspect of our paper: causality is crucial for learning faithful explanations.

2. Using our novel dataset (section 5.1 and Appendix D), we explored scenarios where the candidate set consists of out-of-distribution texts from topics that substantially differ from those of the primary examples (e.g., abortion vs climate change). Here, our method proved to be the most effective among matching methods, indicating its robustness even in less ideal conditions.

3. The first row of Table 2 (main results) shows our method's performance when the candidate set is augmented by ground-truth CFs. In this setup, our matching method surpasses even the generative models, illustrating its exceptional effectiveness under optimal conditions.

Reviewer comment (Experiments):
It's unclear how these techniques would perform on models trained on very niche tasks, which could inherently limit the possible counterfactuals, especially where such attributes may be hard to define beforehand. Both counterfactual generation and matching approaches assume that we have a set of attributes for which we wish to examine whether a model is paying attention to those. However, how does this approach work for open ended tasks (which is where LLMs are primarily being used) or tasks with a large number of classes, where generating counterfactuals (or finding matches) may be inherently difficult?

Author response:
**Regarding niche tasks with unknown concepts:**

This paper focuses on model interpretability via causal effect estimation (i.e., providing an accurate estimate of how a change of a given variable impacts the prediction of the model). In the causal effect estimation setup, we assume the existence of a causal graph (typically provided by domain experts), outlining the concepts and their interrelations. This assumption might require some effort of concept and causal graph specification, but it is a **requirement** for **accurate causal estimation** (see Shpitser and Pearl 2008 and Feder et.al. 2021 in the references comment). Moreover, as we theoretically show in the main theorem of the paper, a lack of information about the candidate concepts and the causal graph may result in an unfaithful explanation. Hence, we consider the predefined concepts a feature, rather than a limitation of our approach.

This aligns with prevalent practices in causal inference in NLP (see references comment). 
Additionally, pre-defined concepts (properties, attributes, aspects) are extensively used in the NLP model interpretability literature; for example, almost every probing, fairness, and bias detection paper assumes their existence (see references comment). We recognize the significance of this discussion, and we plan to include it in the final version of the paper. 

**Regarding open-ended tasks:**

This is a great question. Our framework is not applicable to open-ended tasks (and generation tasks) since the causal effect estimators assume a classifier. We are not aware of any work that aims to address causal model explanations for non-classification tasks, and we consider this an important topic for future research.

Reviewer comment (Theory):
It's unclear how these methods would handle situations with multilayered complexities, such as nested counterfactuals, where counterfactual changes to one concept might trigger changes to other related concepts. The paper also does not extensively address scenarios where counterfactual approximations could result in impossibilities or logical contradictions, potentially limiting the breadth of their application.

Author response:
**Regarding nested counterfactuals:**

If  “nested counterfactuals” refers to an intervention on a concept that is a parent of a mediator (e.g., given A->B->C, a change in A causes a change in B that changes C), then unfortunately, CEBaB - which is the only causal-driven benchmark for explanation methods - does not support such structures. Notice, however, that this work is the first step for further research, and the ideas proposed by the reviewer indeed provide great directions for future work. We believe that this paper laid the foundation for this kind of research: we show theoretically and empirically that faithful explanations must be causal.

Moreover, we demonstrate that LLMs can be utilized for constructing benchmarks for causal explanation methods that support your ideas: complex causal graphs, many classes, and nested CFs.

Reviewer comment (Writing):
Could you explain the difference between Random Match and Approx again? It's not that clear from the paper.

Author response:
**Regarding your question (difference between Random match and Approx):**

The difference between the two methods is that in the approx method, using concept classifiers trained on the training set, we first identify only candidates that share the same value of concepts as the examples and then randomly select a match from them. 
For example, if the intervention is Food (Positive -> Negative), and the remaining three values of the confounders are Ambiance (Negative), Noise (Negative), and Service (Positive) - We first find all candidates with values of Food (Negative), Ambiance (Negative), Noise (Negative), and Service (Positive), and then we randomly select a match from them. In Random Match, we do not perform the identification step, and only randomly select a match from the candidate set (in our example: texts with Food (Negative)).

Reviewer comment (Novelty):
The paper mostly follows the setting of the work of CEBaB, including causal analysis, the approximated counterfactual method, and the evaluation. While some theoretical analysis is provided in Section 3.1, it mainly argues why the approximated counterfactual method which is initially proposed in the CEBaB is better than others. Underlining this, the first concern is that the paper's contribution appears to be limited to the proposal of two LLM-based approximated counterfactual methods that perform better in CEBaB's causal framework. Given the powerful ability of LLM, using it can better generate counterfactual examples (that only change one concept of the input while keeping other aspects unchanged) is not very surprising.

Author response:
**Regarding the contribution of the work (first concern):**

First, we use the term “approximated counterfactual explanation method” as an umbrella term for any explanation in our paper that is based on CFs. The Approx method introduced in CEBaB is just an example of such a method and is a naive adaptation of low-dimensional matching that is abundant in the literature on causal analysis. It only looks for examples with similar concept values, completely ignoring the semantics of the text, which is very different from our approaches. 

Second, the theoretical analysis in Section 3.1 explains why any approximated CF explanation method is order-faithful, not only the naive Approx baseline from CEBaB, and underscores the importance of the causal approach to the explanations.

We would like to list the contributions of the paper, which we believe go well beyond the reviewer’s statement:

1. While the idea of causal explanations being integral to model faithfulness has been discussed in prior works, including CEBaB, our paper is the first to provide a theoretical framework that unifies the concepts of faithfulness and causal explanation. Specifically, we prove that CFs (which are a fundamental concept in causal inference) guarantee faithfulness, unlike other model explanation techniques, such as probing, concept removal, adversarial approaches, etc.

2. In line with recent advancements in the capabilities of LLMs, we are the first to show that LLM-generated CFs provide the SOTA NLP model explanation.

3. Moreover, we demonstrate empirically that utilizing multiple CFs enhances the quality of the causal estimator across various explanation methodologies. This empirical validation is the first of its kind and is both important and useful.

4. This is the first paper that thoroughly explores matching techniques for causal model explanation in NLP. Specifically, we enhance the matching paradigm originating in  the causal literature such that it considers the rich representation of the text, and not only the observed concepts. This key contribution lays the groundwork for future research in causal representation learning from text.

5. Finally, recognizing the gap in benchmarks for causal model explanations, we construct a new interventional benchmark. This demonstration paves the way for future research to explore causal model explanations for a broader range of NLP problems and more complex setups.

In essence, our main contribution lies in theoretically and empirically establishing that faithful explanation methods must be causal and propose CFs as the central mechanism to achieve this. We believe that this insight is pivotal for the field of model interpretability.

Reviewer comment (Novelty):
The second concern is the limited applicability of the proposed matching method. The use of the matching method under the CEBaB setting requires pre-defined or pre-identified concepts/factors, such as Food (F), Service (S), Ambiance (A), and Noise (N) in restaurant reviews. However, these concepts may not always be available or readily identified in many real-world NLP scenarios. Given that the paper exclusively focuses on using LLMs to generate counterfactual examples in this particular setting, its broader applicability and contribution are questioned.

Author response:
**Regarding the pre-defined concepts:**

This paper focuses on model interpretability via causal effect estimation (i.e., providing an accurate estimation of how a change of a given variable impacts the prediction of the model). This differs from other model interpretability problems, such as finding concepts that  may correlate with the prediction. 

Accordingly, in the causal effect estimation setup, we assume the existence of a causal graph (typically provided by domain experts), outlining the concepts and their interrelations. This assumption might require some effort of concept and causal graph specification, but it is a **requirement** for **accurate causal estimation** (see Shpitser and Pearl 2008 and Feder et.al. 2021 in the references comment). Moreover, as we theoretically show in the main theorem of the paper, a lack of information about the candidate concepts and the causal graph may result in an unfaithful explanation. Hence, we consider the predefined concepts a feature, rather than a limitation of our approach.

This aligns with prevalent practices in causal inference in NLP (Feder et al., 2022, see references comment).  Additionally, pre-defined concepts (properties, attributes, aspects) are extensively used in the NLP model interpretability literature; for example, almost every probing, fairness, and bias detection paper assumes their existence (see references comment). We recognize the importance of this discussion, and we plan to include it in the final version of the paper.

**Regarding concept annotations:**

Although we used concept annotations in our experiments, we believe they are not required and can be easily inferred by an LLM.  
Moreover, we would like to point to experiments from our ablation study (Appendix C, referred to on pages 5 and 7), which offers valuable insights into the applicability of our matching approach. In Table 4, rows 3 and 8 present experiments where components that rely on pre-identified concepts are omitted. 
Row 3 presents an experiment where misspecified CFs are not filtered. This filtering relies on concept classifiers, which are trained with concept annotations. 

Row 8 presents an experiment without the matches (X_M) and misspecified matches (X_~M) sets, which are part of the optimization objective and are constructed using concept annotations. We found that these omissions had minimal performance impact. However, this changes markedly when the quality of the candidate set changes. When the candidate set is augmented with misspecified CFs (“confusing examples”), models trained without these components show a noticeable decline in performance compared to the full model. This experiment underscores a key observation of our paper: a definition of the causal mechanism is crucial for learning faithful explanations, and without pre-defined concepts, explanation methods might default to mere correlational insights, which, while potentially effective, lack robustness. In challenging scenarios, such methods are prone to failure, as demonstrated in our main theorem: A non-causal method can be faithful for a given problem, but small changes to the causal mechanism may make it unfaithful. 

**Second concern:**

It is noteworthy that in some fields, causal explanations are crucial, making the effort to construct a causal graph a worthwhile endeavour. Consider an NLP model that recommends medical treatments based on symptoms described in the medical record. In that case, the clinician can not rely on correlational interpretations of the model recommendations. 

There are a few preliminary works that evaluate the capabilities of LLMs to construct a causal graph, for example, Kiciman et al. 2023, Arsenyan et al. 2023 and Tu et al. 2023 (see references  below). While these papers do not rely on training data but on world knowledge acquired by LLMs, constructing a causal graph from training data (observational data) using an LLM is a promising direction. In the very recent paper of Ludan et al. 2023, the authors augment the prompt of an LLM with examples from the training data and ask which concepts might impact the label. Notice, however, that this is not a causal approach (since no causal structure is discovered, only concepts) and it also does not attempt to explain model predictions (the authors develop an interpretable model based on the discovered concepts). Nevertheless, this demonstrates the promising capabilities of LLMs for such tasks. Following future research and technological advances, it is reasonable to assume that the reliance on human experts for concept and causal graph discovery will reduce, and this process will gradually become more and more automatic. Hopefully, our paper will contribute to the effort of unifying causality and model interpretability and facilitate important research in the directions suggested by the reviewer.

Reviewer comment (Evaluation):
As indicated by the results in Table 2, the matching method (causal model) demonstrates only a slight improvement over the Approx baseline, especially when K=10. While the matching method is the most novel part of the paper IMO, the fact that its performance isn't significantly superior to the Approx baseline brings its practical importance into question.

Author response:
**Regarding the results of the matching method:**

We argue that while the matching method is the main methodological contribution of this paper, the contributions of the paper go well beyond this contribution (please see the above comment about contributions). We would like to highlight the following aspects of the comparison between our matching method and the Approx baseline:

1. In Table 2: Our matching method consistently outperforms the Approx baseline for every metric and every model. Moreover, each number in the table is an average of 12 interventions, and while this is not shown in the table, we confirm that our method is superior in each of them. 

2. Our matching method is also substantially faster (7-28 times faster, see Appendix B) and does not require concept classifiers like Approx.

3. Our matching method surpasses the generative models in scenarios where the candidate set includes ground-truth CFs (as shown in the first row of Table 2).
 
4. The comparative analysis presented in Figures 2 and 3 illustrates the superiority of our matching method over the Approx baseline. Our method accurately ranks candidates (the lower the rank, the bigger the estimation error is) - unlike other matching baselines.

5. Regarding the comment about K matches: This technique improves all the methods we examined (generative and matching), which is also an important finding of our paper. Moreover, the Approx baseline samples a match from a subset of candidates agreeing on all the evaluated example's concept values. Therefore, it might not be applicable if this subset is empty or small, which could happen when there is a large number of concepts or when some concept values have a particularly low probability (although in CEBaB this did not happen).

Reviewer comment (Experiments):
The results are only on one dataset CEBaB. The experimental section would be more convincing if more experiments were done on a wider range of datasets.

Author response:
**Regarding using only the CEBaB dataset:**

Our choice of CEBaB was driven by its unique status as the **only** non-synthetic interventional dataset tailored for benchmarking concept-level explanation methods in NLP. While automatic techniques for evaluating explanations exist and can be applied to non-interventional datasets, they predominantly assess correlation rather than causation (as discussed in our Introduction) and, therefore, are unsuitable for this paper.

Nevertheless, recognizing this limitation, we have not limited our investigation to CEBaB only. As detailed in subsection 5.1 and Appendix D, we introduced a new benchmark focusing on Stance Detection. This extension serves to validate our findings in diverse contexts, demonstrating the replicability of our approach.

Additionally, our work underscores a novel contribution in utilizing LLMs for generating Counterfactuals (CFs). This, as we demonstrated, can facilitate the creation of new interventional datasets, significantly advancing the research in causal explanations and benchmark construction. We plan to highlight these aspects more prominently in the revised manuscript.

Reviewer comment (Theory):
The concepts are pre-defined, which can be a limiting factor to the comprehensiveness of the Counterfactuals generated.

Author response:
**Regarding pre-defined concepts:**

This paper focuses on model interpretability via causal effect estimation (i.e., providing an accurate estimation of how a change of a given variable impacts the prediction of the model). This differs from other model interpretability problems, such as finding concepts that  may correlate with the prediction. 

Accordingly, in the causal effect estimation setup, we assume the existence of a causal graph (typically provided by domain experts), outlining the concepts and their interrelations. This assumption might require some effort of concept and causal graph specification, but it is a **requirement** for **accurate causal estimation** (see Shpitser and Pearl 2008 and Feder et.al. 2021 in the references comment). Moreover, as we theoretically show in the main theorem of the paper, a lack of information about the candidate concepts and the causal graph may result in an unfaithful explanation. Hence, we consider the predefined concepts a feature, rather than a limitation of our approach.

This aligns with prevalent practices in causal inference in NLP (Feder et al., 2022, see references comment).  Additionally, pre-defined concepts (properties, attributes, aspects) are extensively used in the NLP model interpretability literature; for example, almost every probing, fairness, and bias detection paper assumes their existence (see references comment). We recognize the importance of this discussion, and we plan to include it in the final version of the paper.

Reviewer comment (Reproducibility):
If the matching candidate set doesn't exist, do you generate them given the concepts?

Author response:
**Regarding your question:**

As mentioned by the reviewer, we could generate the candidate set using LLMs, although this is not what we did in practice. We constructed the candidate sets as follows:
For the CEBaB setup, the candidate set is randomly sampled from the training set. Notice that this candidate set is used only at inference time. 
For the novel Stance Detection setup, we rely on texts from the SemEval16 dataset but generate the entire training, dev, test, and candidate sets. 

The only assumptions we make in this paper are about the pre-defined concepts and the causal graph, and the existence of training and dev sets. The CFs we use for training the models are generated by an LLM, making our matching approach more general compared to the alternative of using human-written CFs.