Paper: AttributionLab: Faithfulness of Feature Attribution Under Controllable Environments (ICLR-2024)

Reviewer comment (Evaluation):
Faithfulness is a complicated question. The implication in this paper is that without AttributionLab one might not have ground truth feature importance scores. This paper argues that it is therefore hard to measure faithfulness, but leaves me wondering if there is value in defining the measure at all. Perhaps this weakness is really an issue in the field at large and not the responsibility of this paper.

Author response:
The reviewer's point about the abstract nature of "faithfulness" in feature attribution is indeed thought-provoking. This concept varies in interpretation among different researchers, leading to a spectrum of understandings. As highlighted in our Related Work section, the existing formalizations of faithfulness in the literature come with their own set of issues. A common complication is that these definitions often intertwine with other intricate concepts, such as out-of-distribution (OOD) effects.  In contrast, using our framework, we can materialize the abstract notion of alignment with the model, because we know how the model uses each feature. This approach simplifies the assessment of faithfulness by clearly separating it from other confounding factors, including OOD effects.

Reviewer comment (Evaluation):
The Disagreement Problem (Krishna et al. 2022) is cited but not discussed much. In light of this work, knowing that feature attribution methods tend to disagree, what level of faithfulness in the AttributionLab do the authors ascribe to "sufficient" for use in practice? In other words -- we have existing work that shows that two methods won't find the same feature importance scores, so less than 100% faithful must be tolerable -- do the authors have intuition or recommendations for practitioners on faithfulness ranges that are good/bad?

Author response:
We appreciate the reviewer for raising this point. It is important to note that our AttributionLab is primarily intended as a sanity check or debugging tool, rather than a benchmarking platform. Consequently, if an attribution method exhibits less than 100% faithfulness in our tests, it implies that the method may have inherent limitations. However, this does not automatically render the attribution method unsuitable for real-world applications. As we show in the paper, we can see how and why the attribution is not faithful to the model. For instance, our observations with DeepSHAP indicate that while it accurately attributes to the contributing features, it fails to distinguish between positively and negatively contributing features effectively.

Reviewer comment (Experiments):
With respect to IG and Occlusion, an interesting point is made on the importance of the baseline value for these methods. I understand how a 'good' choice is made in AttributionLab and how we can conclude that the baseline matters. Is there a method for picking real baselines on real data? On real tabular data? or natural language data?

Author response:
Thank you for bringing this to our attention. Indeed, our observations emphasize the significant impact of baseline choice on the effectiveness of attribution methods, particularly when the baseline is known. Our work underscores the crucial link between the choice of baseline and the model's learned distribution. To select an appropriate baseline, one must have prior knowledge of the model's learned distribution, which AttributionLab facilitates due to its designed model and data.

However, in many existing studies, the baseline is chosen based solely on the data. Common practices range from using simple black pixels (value 0) to average pixel values. Some approaches have employed more sophisticated techniques, such as inpainting the input when an object is removed, to predict the baseline for the excised portion. But these methods primarily consider only the data. Consider a model trained on CIFAR-10, which is not encoded to recognize a broader range of natural images beyond its 10 classes. If you input an image of a car on a dirt road into both a CLIP model and a CIFAR-trained model, the appropriate baselines would differ. For the CLIP model, using the dirt value as a baseline might be more appropriate, as it's likely been trained on images of both dirt and cars. However, for the CIFAR-trained model, the dirt represents an OOD value. Thus, a proper baseline should be within the distribution that the model has learned, which in itself is a challenging problem.

Within AttributionLab, we can accurately determine baseline values because we fully understand the model's internal computation process. In real-world scenarios, we can improve the likelihood of choosing the right baselines by considering the specific task at hand. For example, for a model trained on chest X-rays, a good baseline might be the background values of chest X-rays (close to black), as the model is extensively trained on images characterized by stark black-and-white contrast.

Reviewer comment (Experiments):
It would be interesting to see the framework evaluation on a larger scale that supports different architectures, tasks and problem domains.

Author response:
**Attribution is still not solved for CNNs and MLPs** The focus of the literature has primarily been on addressing the attribution problem within CNNs and MLPs, a challenge that continues to be unresolved. Furthermore, MLPs and CNNs are still widely used in recent large-scale models. For instance, MLPs are used for encoding Key, Query, and Value in transformers, while CNNs are used interspersed with attention in the UNet in Stable Diffusion. Therefore, feature attribution for CNNs and MLPs is still a very critical research topic. In our work, we are pleased to present a significant advancement towards a solution, utilizing a fully controllable environment for the first time. This novel approach not only serves as a strong sanity check and debugging tool for various methods but also exposes the unique properties and limitations of all existing methods we have tested. 

**It is feasible to extend the framework to Transformers** We showcased several CNN and MLP modules designed for various tasks. The versatility of our method extends to other setups such as regression and multimodal tasks. This extendibility is a key strength of our approach. For future work we can propose transformers for the proposed tasks, as there is nothing fundamentally different (in fact MLPs are part of the equation). While exploring the application of existing attribution methods to transformers remains an intriguing prospect, our current focus on CNNs and MLPs has enabled us to identify specific issues and properties of these models. As most of these attribution methods themselves can be also applied to transformers. For instance in a new experiment (shown in Figure 21, 22, 23 in Appendix I), we show that an extension of LIME which was found to work well in AttributionLab, seems promising in ViT as well.

Reviewer comment (Experiments):
In order to guarantee or even to be in a good standing empirically, it could require more controllable environments (tasks along with the perfect model weights) to verify the faithfulness of an attribution method.

Author response:
**Attention being explanation is disputed** Attention maps offer insights into the interactions between input tokens in transformers. However, their effectiveness as reliable explanations has been a topic of considerable debate. These attention maps visualize attribution between Query (Q) and Key (K). However, Q and K are outcomes of encoding through two MLPs in the attention layer, which means that attention maps offer only a partial view of the entire attention process, as the mappings to Q and K remain unknown. Additionally, the complexity introduced by multiple layers and attention heads further complicates the interpretation of this information. Besides, previous studies also question the validity of attention maps being explanations. Jain et al. [1] and Serrano et al. [2] have highlighted inconsistencies between attention maps and gradient-based or occlusion-based methods. Furthermore, Attanasio et al. [3] reveal in a benchmark that attention maps are only partially faithful for transformers. These studies collectively suggest that attention maps may not fully capture the nuances of contributions of Q and K. In light of these insights, we primarily focus on studying the post-hoc attribution methods in this work.

**Attribution is still needed for transformers** Feature attribution methods explain models from different aspects. One of the mainly accepted ones in the concept of contribution (credit assignment). This concept assigns contribution to features based on their effect on the output when they are removed from the input. Most attribution methods are based on this aspect. This aspect is orthogonal to attention and activation values. Thus even if attention values were understood properly, they would still shed light on one facet of transformers and the contribution assignment methods still reveal different insights.

[1] Jain, S., & Wallace, B.C. (2019). Attention is not Explanation. North American Chapter of the Association for Computational Linguistics.

[2] Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., & Wang, F. (2021, August). Why attentions may not be interpretable?. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 25-34).

[3] Attanasio, G., Nozza, D., Pastor, E., & Hovy, D. (2022). Benchmarking post-hoc interpretability approaches for transformer-based misogyny detection. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP. Association for Computational Linguistics.

Reviewer comment (Evaluation):
How would the evaluation of transformer based model look like, that already has a build-in support for attribution feature via visualizing the attention maps rather than relying on post-hoc attribution methods (this implicit attention feature attribution indeed should be accurate as this represents the inner computation graph itself).

Author response:
**Attention being explanation is disputed** Attention maps offer insights into the interactions between input tokens in transformers. However, their effectiveness as reliable explanations has been a topic of considerable debate. These attention maps visualize attribution between Query (Q) and Key (K). However, Q and K are outcomes of encoding through two MLPs in the attention layer, which means that attention maps offer only a partial view of the entire attention process, as the mappings to Q and K remain unknown. Additionally, the complexity introduced by multiple layers and attention heads further complicates the interpretation of this information. Besides, previous studies also question the validity of attention maps being explanations. Jain et al. [1] and Serrano et al. [2] have highlighted inconsistencies between attention maps and gradient-based or occlusion-based methods. Furthermore, Attanasio et al. [3] reveal in a benchmark that attention maps are only partially faithful for transformers. These studies collectively suggest that attention maps may not fully capture the nuances of contributions of Q and K. In light of these insights, we primarily focus on studying the post-hoc attribution methods in this work.

**Attribution is still needed for transformers** Feature attribution methods explain models from different aspects. One of the mainly accepted ones in the concept of contribution (credit assignment). This concept assigns contribution to features based on their effect on the output when they are removed from the input. Most attribution methods are based on this aspect. This aspect is orthogonal to attention and activation values. Thus even if attention values were understood properly, they would still shed light on one facet of transformers and the contribution assignment methods still reveal different insights.

[1] Jain, S., & Wallace, B.C. (2019). Attention is not Explanation. North American Chapter of the Association for Computational Linguistics.

[2] Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., & Wang, F. (2021, August). Why attentions may not be interpretable?. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 25-34).

[3] Attanasio, G., Nozza, D., Pastor, E., & Hovy, D. (2022). Benchmarking post-hoc interpretability approaches for transformer-based misogyny detection. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP. Association for Computational Linguistics.

Reviewer comment (Experiments):
Extending the library of tasks from simple classification setting (with underlying counting or simple arithmetic jobs) to more abstract setting that necessarily doesn’t satisfy the “Proposition 2 (Symmetry property)” (i.e. The addition/removal of any ground-truth pixel to/from the background equally affects the output of the designed neural network.)

Author response:
**Other tasks already exist**
In our study, we have expanded our synthetic setting to include a regression task called single-color-modulo, as detailed in Appendix D and E. Adhering to the core principles of AttributionLab, we manually crafted both the MLP-based neural network and the corresponding dataset. 

**Propositions and axioms of feature attribution** We intentionally enforced the symmetry and sensitivity propositions in both the data and model design. These properties stem from the sensitivity and symmetry axioms [4] of feature attribution. Incorporating them into the designs gives us the ability to see if different attribution methods are aligned in practice with these axioms. These axioms outline desired properties for attribution methods. However, it does not mean we need to be bound by these properties. Depending on the properties of the attribution method we aim to evaluate, we can adapt our setups to include more, fewer, or different axioms.

**Symmetry, Sensitivity are not limits, but additions** These properties facilitate the evaluation of corresponding properties in attribution methods, as previously discussed. Creating setups that lack these properties is relatively straightforward. For example, consider a simplified color classifier setup that determines the presence of a color. Imagine an input with two blue shapes and one yellow shape. In this scenario, the setup lacks sensitivity, as the classification can be triggered by just one colored pixel, and it's not inherently clear which specific pixel leads to the classification. The absence of these properties restricts our evaluation capabilities. Therefore, incorporating properties like sensitivity and symmetry adds complexity to the setup but significantly enhances its evaluative potential, a benefit that extends to more abstract and complex tasks. Hence, including the sensitivity or symmetry axiom is not a limitation or drawback. Rather, it's an addition to evaluate these properties within attribution methods.


[4] ​​​​​​​​​​Sundararajan, M., & Najmi, A. (2020, November). The many Shapley values for model explanation. In International conference on machine learning (pp. 9269-9278). PMLR.

Reviewer comment (Experiments):
To ensure that the verification of attribution method on these custom designed task with small network architecture (2 layer CNN with ReLUs) would generalize to attribution to a large model (that is generally used, with millions or billions of parameters) for the same attribution methods. This might require more complex benchmark tasks.

Author response:
*Minor remark*: We have several CNN and MLP modules, and the color detector alone is 4 layer CNN (The models are ~ 10 layers). The tasks range from modulo computation to counting and color detection.

We agree that the absence of a formal proof for generalization, as you highlighted, represents a limitation within the ground truth-based approach, as also acknowledged in our paper. Despite this, the research on interpretability and the feature attribution problem at a large scale remains inconclusive and open. We encounter the disagreement problem [5] where although the methods are theoretically sound, they diverge in their outcomes. Our paper contends that the exploration of a controlled setup in this research direction is opening new opportunities to address this issue. Please consider the following:

- **Sanity check and debugging** The framework functions as a sanity check and provides an environment for debugging methods. While it may not offer the ultimate solution within this paradigm, it allows us to identify what doesn't work. This is akin to the strategy employed in the sanity checks [6] paper, which critically evaluated various attribution methods prevalent in prior research. In addition, our framework enables us to understand how certain approaches fail and suggests potential solutions to address the issues.

- **Toy models are also used for understanding large models (e.g. LLMs)** With the current explosion in scale, the problem gets more and more convoluted to interpret these large foundation models. In order to understand them, the recent mechanistic interpretability works [7] also use toy transformers to understand their behavior. 

- **No method works at current small scale** Despite the simplicity of the tasks, none of the attributions produces perfect attributions. In an ideal scenario, an attribution should perform flawlessly in AttributionLab (or any controlled environment).


[5] Krishna, S., Han, T., Gu, A., Pombra, J., Jabbari, S., Wu, S., & Lakkaraju, H. (2022). The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXiv:2202.01602.

[6] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. Advances in neural information processing systems, 31.

[7] Olsson, et al., "In-context Learning and Induction Heads", Transformer Circuits Thread, 2022.

Reviewer comment (Theory):
In the paper for equation 3, is there a typo and should this be this instead ? “R(r, g, b) = 1 if Ci(r, g, b) = 1” -> “R(r, g, b) = 0 if Ci(r, g, b) = 1”

Author response:
We thank the reviewer for pointing out the typo and suggesting additional related work, and we have updated the draft and incorporated these suggestions.