Paper: Respect the model: Fine-grained and Robust Explanation with Sharing Ratio Decomposition (ICLR-2024)

Reviewer comment (Writing):
From my personal point of view, the APOP phenomenon is a very interesting and worthwhile topic for further research. However, I didn’t quite grasp how the APOP phenomenon helps with attribution in the model’s inference process. That is, this section doesn’t seem to be tightly connected with the paper’s main focus, the interpretability algorithm.

Author response:
Also, regarding the connection between the APOP phenomenon and our method, we acknowledge the inadequacy of our previous explanation and will provide a revised version to convey our motivation of using preactivation values. As shown in Table 1, APOP shows that classification accuracies can be maintained with only the on/off activation pattern of the network. This emphasizes the influence of activation patterns, including inactive neurons which contribute to making these patterns. Thank you for your constructive suggestion!

Reviewer comment (Experiments):
The experiments in this paper are only based on VGG and ResNet backbones. Perhaps conducting experiments on backbones of different depths and comparing the algorithm’s performance on shallow layers and deep layers would make the paper more convincing.

Author response:
Lastly, the versatility of our method is truly remarkable, extending across various datasets, tasks, and even, across all activations **(Extra > CIFAR 100 > Activation_performance_table.png)** due to the utilization of preactivation. Consequently, it is expected to perform well on conventional networks. While implementation complexities led to its application primarily in VGG and ResNet architectures, it has proven effective even in reinforcement learning scenarios in simple conv net. **(Extra > DQN > gif and png files)**

Reviewer comment (Experiments):
The paper's experiments should include testing on the larger ImageNet validation set for better verification.

Author response:
Regarding the experiment dataset, we used the ImageNet-S50 dataset since we need segmentation masks to precisely evaluate the attribution localization. We believe that this dataset is large enough for evaluating explanation methods, since this dataset contains 752 samples, where [3] used 700 images in the Caltech101 dataset, and [4] used 500 samples in the MNIST/CIFAR-10 dataset and 100 samples in the ImageNet dataset.  
[3] VU MinhN, NGUYEN TrucD T, PHAN N, et al. Evaluating Explainers via Perturbation[J]. arXiv: Learning,arXiv: Learning, 2019.  
[4] Fel, T., Ducoffe, M., Vigouroux, D., Cadene, R., Capelle, M., Nicodeme, C., & Serre, T. (2022). Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16153-16163.

Reviewer comment (Evaluation):
The APOP, although interesting, doesn't seem connected to the motivation that focus on interpretability.

Author response:
Also, regarding the connection between the APOP phenomenon and our method, we acknowledge the inadequacy of our previous explanation and will provide a revised version to convey our motivation of using preactivation values. As shown in Table 1, APOP shows that classification accuracies can be maintained with only the on/off activation pattern of the network. This emphasizes the influence of activation patterns, including inactive neurons which contribute to making these patterns. Thank you for your constructive suggestion!

Reviewer comment (Experiments):
The paper should consider conducting experiments with different backbone architectures to strengthen its findings.

Author response:
Lastly, the versatility of our method is truly remarkable, extending across various datasets, tasks, and even, across all activations **(Extra > CIFAR 100 > Activation_performance_table.png)** due to the utilization of preactivation. Consequently, it is expected to perform well on conventional networks. While implementation complexities led to its application primarily in VGG and ResNet architectures, it has proven effective even in reinforcement learning scenarios in simple conv net. **(Extra > DQN > gif and png files)**

Reviewer comment (Experiments):
Compared SOTA is limited and all methods were proposed before 2021. There are many CAM based methods in these two years. Why do not compare to the methods in these two years?

Author response:
Regarding the query in comparison with the latest methods, we also aspired to undertake comparative experiments with the most recent methodologies; however, regrettably, access to the corresponding source codes is presently unavailable *(We sent mails to the authors.)*. This circumstance elicits our sincere apologies.

Reviewer comment (Theory):
In equation 4, it mentions that the summation of modified sharing ratio is not 1. So will it cause some problem? It seems better to normalize ratio to 1 further.

Author response:
For the question on the modified sharing ratio, we can normalize it to 1 since the outcomes remain unaffected. Yet, since we employed alternative XAI methods for the MLP layer, we tried to use the outputs without any modification.

Reviewer comment (Evaluation):
The metrics used in experiment is quite different to other XAI methods. Is there any consideration in that?

Author response:
And we are also aware that Insertion/Deletion is a widely used metric, falling under the category of Faithfulness. However, as in [2], with Insertion/Deletion, it remains ambiguous whether the drop in model performance results from a shift in data distribution or from the deletion/insertion of the informative features. For example, even though InputXGrad does not seem faithful, it exhibits excellent deletion performance **(ResNet50: 0.0842, VGG16: 0.0459)**. This may be attributed to the fact that logit values diminish rapidly as input quickly falls into out-of-distribution. Likewise, in the case of Insertion, explanation methods with low-resolution demonstrate superior performance, since it is advantageous to just fill the area as fast as possible, independently of the model’s explanation. Conversely, the Fidelity metric (which we used), is relatively more versatile than deletion *(addressing OOD issues)* and insertion *(area filling)* since it involves removing random pixels and observing the change in logits. Hence, we opted for Fidelity in our evaluation, instead of Insertion and Deletion.  
[2] Hooker et al. A Benchmark for Interpretability Methods in Deep Neural Networks, 2019.

Reviewer comment (Presentation):
The '3. Method' part needs to be improved for better presentation. For readers who are a beginner in the XAI research field, I feel that a compact and clear description such as a pseudocode-style presentation or stepwise procedure explanation (e.g., 1. Feedforward an input, 2) Calculate PFV, 3) Decompose PFV, 4) Compute sharing ration through backward propagation, 5) Relevance is computed), would definitely help readers to understand the way that the method works. In this version of the article, the description is a quite verbal explanation.

Author response:
Regarding the suggestion for a beginner-friendly explanation of the method part, thank you for your constructive suggestion. We provide forward pass with backward pass, because we thought that our forward pass is intuitive for beginners to understand well and our backward pass is familiar for others who have experience in this field. Yet, we would revise our paper by providing the pseudocode or stepwise procedure explanation for better understanding. For better understanding of our method, please refer to the stepwise procedure description as below. We hope that our stepwise description and the formulas in our paper help you understand our method easier. 
>	> #### **[forward pass]** 
>	> 1. Feedforward an input.
2. With the sharing ratio, construct ERF from the first layer. (ERF for an input is the pixel itself as in Eq.(2)) Now, every PFV on every layer has its own assigned ERF.
3. Generate the final saliency map with the weighted sum of ERFs assigned for final layer PFVs. (Eq. (3))
>	> #### **[backward pass]**
>	> 1. Generate PFVs as in Eq.(5).
2. Compute sharing ratio as in Eq.(6). 
3. Decompose the relevance to the output as in Eq.(7).
4. Generate the final saliency map by summing the relevance shares in its projective field. (Eq.(8))

Reviewer comment (Experiments):
The key reasons behind the gains of SRD are not clearly analyzed. In this paper, the authors have separately pointed out the uniqueness of their method in multiple parts of the paper. However, I cannot fully understand the essential factors that make SRD be superior to other prior XAI methods. Is it due to the vector perspective, consideration of inactive neurons, decomposition of PFV, or any combinatorial effect of multiple factors?

Author response:
Also, to clarify the gains of our method, we would provide an ablation study for each factor in the supplementary material **(Extra > Ablation > image files)**. The scalar-based approach with our method can be regarded as LRP-0 [1]. And next to it, we present the generated explanation when calculating the relevance with post-activation values. As you can see, compared to ours (SRD), the generated explanations with scalar are very noisy, while those with post-activation value are too sparse. With our observation of APOP, we have proven that we should consider every information including active and inactive neurons. This is the reason that we used vectors as our analysis unit and pre-activation values to propagate our relevance.  
[1] Bach et al. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation, 2015.

Reviewer comment (Writing):
In the 'Weakness 1' issue, would you provide a stepwise description of the methodology?

Author response:
Regarding the suggestion for a beginner-friendly explanation of the method part, thank you for your constructive suggestion. We provide forward pass with backward pass, because we thought that our forward pass is intuitive for beginners to understand well and our backward pass is familiar for others who have experience in this field. Yet, we would revise our paper by providing the pseudocode or stepwise procedure explanation for better understanding. For better understanding of our method, please refer to the stepwise procedure description as below. We hope that our stepwise description and the formulas in our paper help you understand our method easier. 
>	> #### **[forward pass]** 
>	> 1. Feedforward an input.
2. With the sharing ratio, construct ERF from the first layer. (ERF for an input is the pixel itself as in Eq.(2)) Now, every PFV on every layer has its own assigned ERF.
3. Generate the final saliency map with the weighted sum of ERFs assigned for final layer PFVs. (Eq. (3))
>	> #### **[backward pass]**
>	> 1. Generate PFVs as in Eq.(5).
2. Compute sharing ratio as in Eq.(6). 
3. Decompose the relevance to the output as in Eq.(7).
4. Generate the final saliency map by summing the relevance shares in its projective field. (Eq.(8))

Reviewer comment (Evaluation):
In the 'Weakness 2' issue, would you clarify the main reason of the gains of SRD?

Author response:
Also, to clarify the gains of our method, we would provide an ablation study for each factor in the supplementary material **(Extra > Ablation > image files)**. The scalar-based approach with our method can be regarded as LRP-0 [1]. And next to it, we present the generated explanation when calculating the relevance with post-activation values. As you can see, compared to ours (SRD), the generated explanations with scalar are very noisy, while those with post-activation value are too sparse. With our observation of APOP, we have proven that we should consider every information including active and inactive neurons. This is the reason that we used vectors as our analysis unit and pre-activation values to propagate our relevance.  
[1] Bach et al. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation, 2015.