Paper: An Axiomatic Approach to Model-Agnostic Concept Explanations (ICLR-2024)

Reviewer comment (Theory):
It seems that the proposed method is only applicable to datasets with pre-defined concept annotations c(x).

Author response:
Note that the proposed method is not only applicable to datasets with pre-defined concept annotations. In Section 5.3 we discussed how to automatically label concepts.

Reviewer comment (Experiments):
Though authors show some connections between the proposed method and two existing methods, there still lacks baseline comparisons in the experiment part. For example, why not compare the proposed method with existing methods (such as [cite 1,2]) for model improvement.

Author response:
Thank you for the references you provided, we will add them in future versions of the paper. However, please note that in this paper, we adopt a black-box approach and intentionally avoid limiting ourselves to neural networks. In the related work section, specifically at the end of the paragraph discussing "concept-based explanations", we discussed papers where concepts constitute part of the model’s hidden representation, similar to [cite 1] and [cite 2].

Reviewer comment (Experiments):
As typical concept explanations usually provide visualizations of extracted concepts. It will help a lot if authors also provide such visualizations, and compare their method with existing concept explanations from this perspective.

Author response:
We appreciate the reviewer's perspective on the value of visualizations in evaluating XAI methods and understand their importance in certain contexts. However, we hold a different view in this instance. We believe that our approach, which focuses on more quantifiable and objective metrics, offers a complementary perspective to the prevalent use of visualizations. This approach, we argue, adds to the diversity of evaluation methods in the field and strengthens the paper by providing an alternative lens through which XAI methods can be assessed.

Reviewer comment (Theory):
Some assertions within the paper are inaccurate. Specifically, the authors incorrectly categorize SHAP and LIME as model-specific techniques that necessitate white-box access to the model. Contrary to these claims, SHAP and LIME are model-agnostic and can operate on black-box models. They generate explanations by probing the model with varying input combinations to obtain SHAP and LIME scores.

Author response:
**SHAP and LIME**. In our discussion of these methods in the related section, we highlight their model-agnostic nature, please refer to Section 2 paragraph “model-agnostic explanations”.

Reviewer comment (Theory):
The terminology used in the problem formulation section lacks precision. The term "measure" is ambiguously defined—it is unclear whether it refers to a "score" or "measurement," or to a "measure" in the context of measure theory. A more mathematically rigorous exposition of the problem setting would be beneficial.

Author response:
**Problem formulation**. We formally defined the problem in section 3, stating that the objective is to find $M$, a function that maps the following parameters to $\mathbb{R}$: (i) the model $h$, (ii) the concept $c$, and (iii) the a priori probability $p(\cdot)$ over the examples.

Reviewer comment (Theory):
The assertion that the proposed approach subsumes TCAV is not entirely accurate. According to Theorem 4.2, it approximates TCAV in an asymptotic sense rather than subsuming it exactly.

Author response:
**“Assertion that the proposed approach subsumes TCAV…”** we aim to establish connections between our approach and TCAV, rather than make such assertion. We will revise our presentation to reflect this point more clearly.

Reviewer comment (Experiments):
There is an absence of comparative results with baseline methods. Unless overlooked, the paper does not present any performance metrics juxtaposed with at least TCAV and completeness-aware explanations. Such comparative analysis is essential for rigorously assessing the technical and empirical merits of the proposed approach. Without it, evaluating the method's substantive contribution becomes challenging.

Author response:
**Comparative results.** Our goal in the paper is not to claim that our approach is better than TCAV or completeness-aware methods. This is why we chose not to compare them empirically. One exception to this rule is that we mentioned our method's faster computational speed compared to those approaches and provided an explanation for it  at the end of sections 4.1 and 4.2. However, we can also test it empirically. Thank you for your suggestion.

Reviewer comment (Writing):
**Presentation issues**. The notation is confusing. Concept is denoted with c(x) and is used to denote either set of concepts (i.e. a concept vector) or a scalar score interchangeably causing much confusion, see (1), (2). Yeh et.al. 2020 should be explained in more details in Sec 4.1. h is overloaded to map from example or representations to the label. (2) uses y to enumerate over concept values, which can be confusing.

Author response:
**Presentation issues**.Thank you for your suggestion regarding notation. We will incorporate it into future versions of the paper.

Reviewer comment (Evaluation):
**Faithfulness is not established**. (This is less a weakness and more a show-stopper). The premise of an explanation is its faithfulness, which surprisingly is not one of the axioms. All their estimators at best measure the correlation in activation of concept and the model-to-be-explained. They need to establish the faithfulness of their estimator with the evaluation suite of TCAV or Yeh et.al. to first establish theirs as an explanation. Imagine a model trained on a dataset with two co-occurring concepts but the model may only exploit one of the concepts for prediction (say because one concept is way easier than the other), however none of their estimators can bring out the asymmetric concept importance.

Author response:
**Faithfulness**. Thank you for providing an example to illustrate your point. It's important to note that ANY black-box explainer that only uses examples from the given distribution, which is the focus of our work, cannot detect the case of two different-impact co-occurring concepts.

Reviewer comment (Evaluation):
**Unconvincing evaluation**. The paper evaluates on a gamble of tasks. The prediction tasks considered are to predict *cat*, *chair*, *motorbike*, *bottle* in Figure 2, 3, 4 respectively. Some rationale on choice of tasks is expected, and some evaluation using established benchmarks is also expected.

Author response:
**Evaluation**. We used the a-Pascal dataset with its 20 classes, this dataset is acknowledged in the CV community. See [1,2,3]. The structure of this dataset also suits our task at hand.

Reviewer comment (Experiments):
**Computational efficiency not presented.** They claim computational efficiency but do not present any empirical evidence supporting the claim.

Author response:
**Computational efficiency**. At the end of sections 4.1 and 4.2, we elaborate on why these methods are more efficient. Thank you for suggesting an empirical examination of this aspect as well.

Reviewer comment (Evaluation):
**Sufficiency vs necessary based score**. The paper discusses concept scores based on sufficiency or necessity, but do not discuss their practical implications. When is one better than the other?

Author response:
**Sufficiency vs necessary based score.** It's not a matter of one being better than the other. Rather, it's about understanding how a concept influences a prediction. We've explored two types of influence: necessary and sufficient. There isn't a definitive choice between them; the selection depends on your specific application.

Reviewer comment (Experiments):
The concern regarding faithfulness with the paper is severe because their estimated importance scores seem model-agnostic. TCAV, for instance, accounts the effect of concept interventions on the representation to measure the importance. Nevertheless, experiments proving faithfulness are much needed.

Author response:
**Faithfulness**. Thank you for providing an example to illustrate your point. It's important to note that ANY black-box explainer that only uses examples from the given distribution, which is the focus of our work, cannot detect the case of two different-impact co-occurring concepts.

Reviewer comment (Evaluation):
The evaluations in Section 5.2 appear to focus on images that were predicted to be a given class. For different models and different optimizers, this set of images could be different. Given that the ground truth labels of the images do not seem to have been taken into account, it is unclear if the presence of "spurious" concepts observed is due to the model relying on spurious correlations for images where the ground truth class is the same as the predicted class being considered, or due to the model simply misclassifying images of other classes as the predicted class being considered. This would be important to properly evaluate the proposed approach.

Author response:
Spurious correlation refers to the scenario wherein a model learns to depend on features that lack a causal relationship with the target variable but exhibit strong correlation within the training data. Our measure identifies concepts on which a model relies, enabling human judgment regarding their relevance to accurate predictions—thereby detecting spurious correlations. While dissecting the reasoning behind spurious correlation, as you suggested, could be interesting, it extends beyond the scope of the current research.

Reviewer comment (Experiments):
Given the efficiency and equivalence arguments between the proposed approach and prior work (completeness-aware and TCAV, Section 4), it would be helpful to have an experimental comparison between these approaches in the application settings in Section 5, both in terms of performance and computational efficiency.

Author response:
At the end of sections 4.1 and 4.2, we elaborate on why these methods are more efficient. Thank you for suggesting an empirical examination of this aspect as well.

Reviewer comment (Theory):
The formulation of TCAV as an instantiation of the proposed framework seems to only apply to explanations at the last layer, while the TCAV formulation as originally proposed is more general. If this is the case, then this should be made clear.

Author response:
We apologize for any confusion caused. We will make sure to provide clarification in future versions of the paper.