Paper: AttEXplore: Attribution for Explanation with model parameters eXploration (ICLR-2024)

Reviewer comment (Experiments):
Given the transferable claim of this new attribution method, it will be necessary to show sufficient visualizations of how the attribution generalizes to different types of task models other than image classification.

Author response:
1. Thank you for the thoughtful consideration of our work. It is necessary to generalize our method to models for different tasks beyond image classification. Working on the interpretability of NLP models is a good direction. For example, verifying the interpretability of medical reports generated by applying NLP technology. 
     At present, our work is aimed at attribution interpretation of image related tasks, which is consistent with the current SOTA research works including AGI [1], BIG [2], GIG [3], and SG [4].

Reviewer comment (Evaluation):
The post-hoc explanation methods often suffer from false-positive responses, which highlight the semantic parts that are actually not relevant to the true grounding objects. The paper should elaborate more on how the framework overcomes such challenges. Similarly, the paper shall have more quantitative evaluations on whether the attribution is correctly aligning with the ground truth.

Author response:
2. Given that our method satisfies the sensitivity and implementation invariance axioms in IG [5], this ensures a one-to-one correspondence between the model's input and output during attribution, guaranteeing the accuracy of the attribution process. We have provided detailed proofs of our method's adherence to these axioms in the appendix.

Reviewer comment (Writing):
Some sections of the paper lack clarity, making it challenging for readers to grasp the key concepts. The abstract and introduction suffer from being overly specific without providing sufficient context. Additionally, important elements of the proposed method are not adequately introduced, leaving readers without a clear understanding of the approach.

Author response:
**Clarity:**

Thanks for the constructive comment. We will rewrite the relevant sections of abstract and introduction to facilitate a more seamless understanding for readers. Additionally, we will improve Figure 1, enhancing image clarity and the representation of key elements to provide a clearer understanding of our method. We will also refine the structure of the appendix and provide more precise and specific references to its content within the main text for improved citation.
 
Regarding the statement "DeepLIFT and LIME generate explanations for generic models which limits the understanding of the model’s global behaviors," our intention is to convey that DeepLIFT [1] and LIME [2], as local interpretability methods, primarily focus on explaining the local neighborhood behaviors of models. These methods typically employ simpler, interpretable models to approximate the behavior of complex models near specific data instances or attempt to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. Therefore, these two methods do not satisfy the sensitivity and implementation invariance axioms proposed by IG [3], resulting in limited global interpretability of model behaviors.  Global interpretability refers to understanding the entire behavior of the model, not just explanations for specific instances or local regions. Global interpretability focuses on properties such as features, weights, structure, and behavior patterns of the model across the entire input space.
 
In Equation (4), the variable $i$ represents the number of frequency domain explorations, i.e., $i=1,2,...,N$. Ultimately, we generate $N$ adversarial samples through frequency domain exploration. We will review the main context to avoid similar ambiguities.

Reviewer comment (Presentation):
Figure 1, a critical element in conveying the method, could benefit from improved clarity. Enhancements to this figure would help readers better comprehend their motivation for AttEXplore.

Author response:
**Clarity:**

Thanks for the constructive comment. We will rewrite the relevant sections of abstract and introduction to facilitate a more seamless understanding for readers. Additionally, we will improve Figure 1, enhancing image clarity and the representation of key elements to provide a clearer understanding of our method. We will also refine the structure of the appendix and provide more precise and specific references to its content within the main text for improved citation.
 
Regarding the statement "DeepLIFT and LIME generate explanations for generic models which limits the understanding of the model’s global behaviors," our intention is to convey that DeepLIFT [1] and LIME [2], as local interpretability methods, primarily focus on explaining the local neighborhood behaviors of models. These methods typically employ simpler, interpretable models to approximate the behavior of complex models near specific data instances or attempt to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. Therefore, these two methods do not satisfy the sensitivity and implementation invariance axioms proposed by IG [3], resulting in limited global interpretability of model behaviors.  Global interpretability refers to understanding the entire behavior of the model, not just explanations for specific instances or local regions. Global interpretability focuses on properties such as features, weights, structure, and behavior patterns of the model across the entire input space.
 
In Equation (4), the variable $i$ represents the number of frequency domain explorations, i.e., $i=1,2,...,N$. Ultimately, we generate $N$ adversarial samples through frequency domain exploration. We will review the main context to avoid similar ambiguities.

Reviewer comment (Presentation):
The paper frequently references the appendix but fails to provide specific locations. It would be highly beneficial if the authors included clear references to specific sections within the appendix, making it easier for readers to locate relevant supplementary information.

Author response:
**Clarity:**

Thanks for the constructive comment. We will rewrite the relevant sections of abstract and introduction to facilitate a more seamless understanding for readers. Additionally, we will improve Figure 1, enhancing image clarity and the representation of key elements to provide a clearer understanding of our method. We will also refine the structure of the appendix and provide more precise and specific references to its content within the main text for improved citation.
 
Regarding the statement "DeepLIFT and LIME generate explanations for generic models which limits the understanding of the model’s global behaviors," our intention is to convey that DeepLIFT [1] and LIME [2], as local interpretability methods, primarily focus on explaining the local neighborhood behaviors of models. These methods typically employ simpler, interpretable models to approximate the behavior of complex models near specific data instances or attempt to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. Therefore, these two methods do not satisfy the sensitivity and implementation invariance axioms proposed by IG [3], resulting in limited global interpretability of model behaviors.  Global interpretability refers to understanding the entire behavior of the model, not just explanations for specific instances or local regions. Global interpretability focuses on properties such as features, weights, structure, and behavior patterns of the model across the entire input space.
 
In Equation (4), the variable $i$ represents the number of frequency domain explorations, i.e., $i=1,2,...,N$. Ultimately, we generate $N$ adversarial samples through frequency domain exploration. We will review the main context to avoid similar ambiguities.

Reviewer comment (Writing):
Some sentences in the paper do not make sense, such as "DeepLIFT and LIME generate explanations for generic models which limits the understanding of model’s global behaviors.“ The reviewer questions why this would limit understanding and what is meant by "global behavior."

Author response:
**Clarity:**

Thanks for the constructive comment. We will rewrite the relevant sections of abstract and introduction to facilitate a more seamless understanding for readers. Additionally, we will improve Figure 1, enhancing image clarity and the representation of key elements to provide a clearer understanding of our method. We will also refine the structure of the appendix and provide more precise and specific references to its content within the main text for improved citation.
 
Regarding the statement "DeepLIFT and LIME generate explanations for generic models which limits the understanding of the model’s global behaviors," our intention is to convey that DeepLIFT [1] and LIME [2], as local interpretability methods, primarily focus on explaining the local neighborhood behaviors of models. These methods typically employ simpler, interpretable models to approximate the behavior of complex models near specific data instances or attempt to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. Therefore, these two methods do not satisfy the sensitivity and implementation invariance axioms proposed by IG [3], resulting in limited global interpretability of model behaviors.  Global interpretability refers to understanding the entire behavior of the model, not just explanations for specific instances or local regions. Global interpretability focuses on properties such as features, weights, structure, and behavior patterns of the model across the entire input space.
 
In Equation (4), the variable $i$ represents the number of frequency domain explorations, i.e., $i=1,2,...,N$. Ultimately, we generate $N$ adversarial samples through frequency domain exploration. We will review the main context to avoid similar ambiguities.

Reviewer comment (Writing):
In the Method section, Equation (4) includes index $i$ without a clear definition. It is unclear whether $i$ refers to a random draw of sample $x$ or a class index. This ambiguity needs to be resolved to enhance the precision of the method description.

Author response:
**Clarity:**

Thanks for the constructive comment. We will rewrite the relevant sections of abstract and introduction to facilitate a more seamless understanding for readers. Additionally, we will improve Figure 1, enhancing image clarity and the representation of key elements to provide a clearer understanding of our method. We will also refine the structure of the appendix and provide more precise and specific references to its content within the main text for improved citation.
 
Regarding the statement "DeepLIFT and LIME generate explanations for generic models which limits the understanding of the model’s global behaviors," our intention is to convey that DeepLIFT [1] and LIME [2], as local interpretability methods, primarily focus on explaining the local neighborhood behaviors of models. These methods typically employ simpler, interpretable models to approximate the behavior of complex models near specific data instances or attempt to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. Therefore, these two methods do not satisfy the sensitivity and implementation invariance axioms proposed by IG [3], resulting in limited global interpretability of model behaviors.  Global interpretability refers to understanding the entire behavior of the model, not just explanations for specific instances or local regions. Global interpretability focuses on properties such as features, weights, structure, and behavior patterns of the model across the entire input space.
 
In Equation (4), the variable $i$ represents the number of frequency domain explorations, i.e., $i=1,2,...,N$. Ultimately, we generate $N$ adversarial samples through frequency domain exploration. We will review the main context to avoid similar ambiguities.

Reviewer comment (Experiments):
The paper does not compare AttEXplore against state-of-the-art approaches designed to address noise in heatmaps, such as LRP (composite rules), SHAP, or LIME.

Author response:
**Evaluation Shortcomings:**

Thanks for the reviewer's comments. Our method satisfies the Completeness axiom, as proven in Section 3 of IG [3], serving as a sanity check. We have also conducted INFD score tests [4] in the global comments in Table *2, and the results demonstrate that our method exhibits superior robustness and fidelity.
 
Concerning LIME [2], LRP [5], and SHAP [6], we would like to note that these methods have different requirements compared to current state-of-the-art attribution methods. 
Firstly, LIME attempts to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. In addition, LIME’s interpretable behavior requires cluster segmentation of images, so its interpretation model is not point-to-point. And if it performs point-to-point calculations, the computational complexity of LIME will be extremely high. LRP relies on model structure and is challenging to implement. DeepLIFT is a more general case of LRP, and our method surpasses DeepLIFT. SHAP, when applied to samples with high dimensions, uses approximate results for inference and involves a significant amount of interaction operations, resulting in extremely high computational complexity. 

Secondly, IG [3] extensively discusses that these methods do not necessarily satisfy the sensitivity and implementation invariance axioms. For example, LRP uses discrete gradients to compute feature importance, which does not satisfy the sensitivity axiom and may lead to instances where attributions change, but the attribution result is 0, rendering interpretability completely ineffective. 

Lastly, current state-of-the-art methods including AGI [7], BIG [8] and GIG [9] don’t include LIME, LRP, and SHAP as the comparison in their experiments. It is noteworthy that some of the methods are noise reduction-based methods which were published much earlier. In our baseline, GIG represents the state-of-the-art noise reduction-based method as proposed in 2021. We have achieved significant improvement over the GIG method.
 
Regarding the subset selection for time complexity comparison, we selected the five methods that are closest in performance to our method for efficiency comparison. Other methods, such as SaliencyMap, DeepLIFT and EG have relatively poor attribution accuracy in the baseline, making them unnecessary for comparison. Although Fast-IG achieves faster attribution speeds, its accuracy is unfortunately the lowest.
 
Reference:

[1] Shrikumar, A., Greenside, P., & Kundaje, A. (2017, July). Learning important features through propagating activation differences. In International conference on machine learning (pp. 3145-3153). PMLR.

[2] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).

[3] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.

[4] Yeh, C. K., Hsieh, C. Y., Suggala, A., Inouye, D. I., & Ravikumar, P. K. (2019). On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32.
[5] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.

[6] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.

[7] Pan, D., Li, X., & Zhu, D. (2021, August). Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).

[8] Wang, Z., Fredrikson, M., & Datta, A. (2022). Robust models are more interpretable because attributions look normal. Proceedings of the 39th International Conference on Machine Learning.

[9] Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5050-5058).

Reviewer comment (Evaluation):
The time complexity comparison is limited to only a subset of methods, which restricts the scope of the evaluation and limits the ability to assess AttEXplore's competitive performance in terms of speed.

Author response:
**Evaluation Shortcomings:**

Thanks for the reviewer's comments. Our method satisfies the Completeness axiom, as proven in Section 3 of IG [3], serving as a sanity check. We have also conducted INFD score tests [4] in the global comments in Table *2, and the results demonstrate that our method exhibits superior robustness and fidelity.
 
Concerning LIME [2], LRP [5], and SHAP [6], we would like to note that these methods have different requirements compared to current state-of-the-art attribution methods. 
Firstly, LIME attempts to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. In addition, LIME’s interpretable behavior requires cluster segmentation of images, so its interpretation model is not point-to-point. And if it performs point-to-point calculations, the computational complexity of LIME will be extremely high. LRP relies on model structure and is challenging to implement. DeepLIFT is a more general case of LRP, and our method surpasses DeepLIFT. SHAP, when applied to samples with high dimensions, uses approximate results for inference and involves a significant amount of interaction operations, resulting in extremely high computational complexity. 

Secondly, IG [3] extensively discusses that these methods do not necessarily satisfy the sensitivity and implementation invariance axioms. For example, LRP uses discrete gradients to compute feature importance, which does not satisfy the sensitivity axiom and may lead to instances where attributions change, but the attribution result is 0, rendering interpretability completely ineffective. 

Lastly, current state-of-the-art methods including AGI [7], BIG [8] and GIG [9] don’t include LIME, LRP, and SHAP as the comparison in their experiments. It is noteworthy that some of the methods are noise reduction-based methods which were published much earlier. In our baseline, GIG represents the state-of-the-art noise reduction-based method as proposed in 2021. We have achieved significant improvement over the GIG method.
 
Regarding the subset selection for time complexity comparison, we selected the five methods that are closest in performance to our method for efficiency comparison. Other methods, such as SaliencyMap, DeepLIFT and EG have relatively poor attribution accuracy in the baseline, making them unnecessary for comparison. Although Fast-IG achieves faster attribution speeds, its accuracy is unfortunately the lowest.
 
Reference:

[1] Shrikumar, A., Greenside, P., & Kundaje, A. (2017, July). Learning important features through propagating activation differences. In International conference on machine learning (pp. 3145-3153). PMLR.

[2] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).

[3] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.

[4] Yeh, C. K., Hsieh, C. Y., Suggala, A., Inouye, D. I., & Ravikumar, P. K. (2019). On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32.
[5] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.

[6] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.

[7] Pan, D., Li, X., & Zhu, D. (2021, August). Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).

[8] Wang, Z., Fredrikson, M., & Datta, A. (2022). Robust models are more interpretable because attributions look normal. Proceedings of the 39th International Conference on Machine Learning.

[9] Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5050-5058).

Reviewer comment (Experiments):
This paper appears to focus exclusively on evaluating AttEXplore using image data, it lacks experiments on other data modalities, such as text data on NLP models. Expanding the evaluation to different data modalities could provide a more comprehensive assessment of AttEXplore's applicability.

Author response:
1. Thanks for pointing out the valuable comment for attribution method research. Actually, exploring interpretability in the context of NLP models is undoubtedly a valuable direction, especially in domains such as medical report generation where interpretability is crucial. 
Currently, our work focuses on attribution interpretation for images, aligning with the current state-of-the-art approaches like AGI [1], BIG [2], GIG [3], SG [4], and other baselines. We would like to provide a discussion of potential work following the comment for NLP works.

Reviewer comment (Experiments):
For image classification models, the authors did experiments on CNN model groups including Inception-v3, ResNet-50, and VGG-16. It is unclear that if the superior explainability of AttEXplore stands still on other model groups, like vision transformers and MLP model groups. Extending the evaluation to diverse model architectures would further validate its effectiveness across different model types.

Author response:
2. We appreciate the reviewer's constructive suggestions. We have conducted the additional experiment on the model of Vision Transformers (ViTs). ViTs represent an excellent vision model based on the Transformer architecture. Different from traditional DNN models, ViTs process images as sequences of patches, leading to a patch-like interpretation of features [5]. The experimental results on ViTs, included in the global comments Table *4, still demonstrate that our method achieves the best performance.

Reviewer comment (Experiments):
It's shown in Table 1 in the appendix that compared with N = 10, the drop of insertion score for N = 1 is not huge (~0.007 for inception-v3, still better than existing methods). The perturbation rate was set as 16, while the model performance is better when perturbation rate is 48. Would the trend of model performance under different N be more clear when the perturbation rate is set as a larger value?

Author response:
We are afraid the answer is negative for the trend of model performance under different $N$, even with a larger perturbation rate. 

On one hand, the ablation experiments in Tables 3, 4 and 5, and the appendix file, indicate that increasing the number of generated features or perturbation rates, among other hyperparameters, can enhance attribution accuracy to some extent. This is rational as the increase in the number of generated features ($N$) implies the exploration of more frequency domain information, while an increase in $\epsilon$ signifies the generation of higher-quality adversarial samples. Consequently, the performance at a perturbation rate of 48 may surpass the result with a perturbation rate of 16.
 
On the other hand, the impact of altering hyperparameters on performance, as observed in ablation experiments, is relatively small. This suggests that the improvement of our method does not heavily rely on the specific choice of hyperparameters. However, when the perturbation rate is set to a larger value such as 48, the trend in model performance across different $N$ values does not become more clear. This might be attributed to the fact that a larger perturbation rate represents a larger search space. Although the number of explored samples increases, it doesn’t mean that all these samples are necessarily effective for the attribution result. We have validated this observation in the additional experiments provided in the global comments Table *5.