Paper: Why Sanity Check for Saliency Metrics Fails? (ICLR-2024)

Reviewer comment (Writing):
The current draft requires some improvement before publishing, as it is a little difficult to follow the idea.

Author response:
Author’s comment for Q1: We focused on understanding the inconsistent behavior in saliency metrics, as already observed by Tomsett et.al, rather than repeating their experiments. Our goal was to understand why such inconsistent behavior was observed. However, we will consider aligning the content to eliminate any confusion and make the message clearer.

Reviewer comment (Presentation):
The draft attempts to solve the problem in the fidelity measure for CAM, but the presentation did not show this main issue clearly.

Author response:
Author’s comment for Q1: We focused on understanding the inconsistent behavior in saliency metrics, as already observed by Tomsett et.al, rather than repeating their experiments. Our goal was to understand why such inconsistent behavior was observed. However, we will consider aligning the content to eliminate any confusion and make the message clearer.

Reviewer comment (Writing):
The use of Gaussian Blur is not well introduced or explained; it seems an arbitrary solution jumping into the draft.

Author response:
Comment for Q2: Tomsett et al. 2020 used perturbations like making the pixels 0 and replacing with random valueswhich are quite different. Therefore, we used Gaussian Blur with different sigmas so that the perturbations we use, were similar to each other than those used by Tomsett et al. 2020. If the ranks of the pixels ‘Ig’(calculated by looking at the drop in output probability) vary for small changes in sigma of Gaussian blur then it indicates that the ranks would vary even more with entirely different perturbation technique (like assigning 0 or random value).

Reviewer comment (Theory):
The only reason discussed in Section 3.3 for using Gaussian Blur is to preserve the semantic information instead of removing it. However, this is exactly opposite to those previous perturbations used for fidelity measure.

Author response:
Comment for Q2: Tomsett et al. 2020 used perturbations like making the pixels 0 and replacing with random valueswhich are quite different. Therefore, we used Gaussian Blur with different sigmas so that the perturbations we use, were similar to each other than those used by Tomsett et al. 2020. If the ranks of the pixels ‘Ig’(calculated by looking at the drop in output probability) vary for small changes in sigma of Gaussian blur then it indicates that the ranks would vary even more with entirely different perturbation technique (like assigning 0 or random value).

Reviewer comment (Evaluation):
If the semantic content is always needed, it is unclear how we can measure the CAM, or why perturbation is still needed.

Author response:
Comment for Q3: As observed by the reviewer "One can use any perturbation to show a different result" i.e. based on different perturbations, Ig would be different and as such saliency metrics would rate different CAM method to have higher fidelity (explained in the above example)

Reviewer comment (Theory):
There is doubt whether Gaussian Blur can retain the semantic information when the standard deviation is high enough.

Author response:
Comment for Q9: As observed by the reviewer using stronger perturbations (like Gaussian blur with high sigma) would alter the output probability significantly. Hence, this makes it imperative to choose the perturbation technique carefully and accept the results of fidelity analysis as dependent on perturbation technique used.

Reviewer comment (Novelty):
The two "proposed" metrics are very similar to previous solutions but using Gaussian perturbation, thus the metrics cannot be considered as new proposals.

Author response:
“Then two metrics are proposed to measure the fidelity of the saliency map, denoted as Average Ratio of Positive Probability Drops, and Average Pixel Rank Correlation.”  

Author’s comment: We are proposing ARPPD and APRC metrics to validate the assumption, and not to measure the fidelity of the saliency maps. Our experiments are designed to answer why existing fidelity metrics are were observed as inconsistent.

Reviewer comment (Evaluation):
Generally, the abstract and the introduction show the inconsistency in those saliency metrics used to measure the fidelity of XAI methods. However, Fig.1 only shows a tiny difference in the CAM visualization, and this showcase seems not connected to the main story. The reviewer suggests it may be better to show the behaviour of those metrics when perturbing with different strategies.

Author response:
Comment for Q4: We will make the abstract and introduction more readable as per the suggestion of the reviewer.

Reviewer comment (Theory):
On page 3, third paragraph, the claim that "the reason for such unreliability...lies at model level" is questionable, and there is no explanation provided for this claim.

Author response:
Author's comment for Q3: The claim is discussed in detail in the third paragraph in page 3 (i.e., above the mentioned lines). Further, it has been discussed in detail in Section 2.1 and in Section 3 along with the theoretical justification (Sec 3.1) and the proposed metric (Sec 3.2). The low value observed for ARPPD and APRC in the Results and Discussion (Section 5) provide evidence for the unreliability at the model level.

Reviewer comment (Writing):
In the same paragraph, the statement "introduce two new metrics,....at model level 1" is confusing without any explanation. The proposed metrics are also based on the change of confidence scores or pixel rankings, so it is unclear why they are at different levels than the previously proposed metrics. Introducing the idea without relying on the confusing hierarchy could be more clear.

Author response:
Author's comment for Q8:: Define what is different to previous metrics (levels and what they measure)  

The previous metrics like AOPC, Average Drop(AD%), Increase in Confidence(IC%), Win(W%) etc measure the fidelity of saliency maps. These metrics use the observed change in output probability by removing pixels from an image to calculate the fidelity score (as explained in Section 2.1).

Our proposed metrics too use the observed change in output probability before and after perturbation, but we use the change to determine if the assumption holds true or not and quantify it (discussed Section 3 and Section 3.1). The context of “unlike” is to indicate not the observation, but the reason behind the observation.

Reviewer comment (Presentation):
The extensive related work section is not really necessary; the most related work seems to be (Tomsett 2020) and those fidelity metrics. However, they are not properly presented regarding the key problem of why different perturbations may lead to different results.

Author response:
Author's comment for Q5: We have provided extensive set of related works for the better understanding of the problem. However, we will make it concise by aligning our arguments based on the problem our paper is addressing.

Reviewer comment (Writing):
Section 2.1, first paragraph, is confusing. It seems to repeat how those metrics work without introducing what the limitation is and why it is a limitation.

Author response:
Author's comment for Q6: We discuss the limitation and why it is a limitation in Section 2.1. However, we will revise this section to make the limitation explicit based on the suggestion of the reviewer.

Reviewer comment (Writing):
Section 2.2 Contribution 1 states "to justify the unreliability", but the draft did not show clearly what the unreliability is and why it needs to be justified.

Author response:
Author's comment for Q7: We'll replace “justify the unreliability” with “investigate the inconsistency” for clarity.

Reviewer comment (Novelty):
Section 2.2 Contribution 2 starts with "Unlike...", but the proposed methods are very similar to the previously used ones for fidelity measure, making this claim unclear.

Author response:
Author's comment for Q8:: Define what is different to previous metrics (levels and what they measure)  

The previous metrics like AOPC, Average Drop(AD%), Increase in Confidence(IC%), Win(W%) etc measure the fidelity of saliency maps. These metrics use the observed change in output probability by removing pixels from an image to calculate the fidelity score (as explained in Section 2.1).

Our proposed metrics too use the observed change in output probability before and after perturbation, but we use the change to determine if the assumption holds true or not and quantify it (discussed Section 3 and Section 3.1). The context of “unlike” is to indicate not the observation, but the reason behind the observation.

Reviewer comment (Novelty):
In Section 3.1, the presented "framework" is not necessary, as it is the common assumption behind most of the fidelity measure.

Author response:
Author's comment for Q11: The theoretical framework is necessary as it enables us to evaluate if the assumption holds true or not. While we agree that it is a common assumption, we also mention this explicitly in our paper to illustrate the fact that it does not always hold true. Further, the low values of ARPPD and APRC in Section 5 supports that the assumption does not always hold true.

Reviewer comment (Theory):
In Equation (6), it is unclear why the rank is invariant to the standard deviation value applied.

Author response:
Author's comment for Q12: Rank invariance is key to avoid fidelity score ambiguities due to different perturbations, addressing the inconsistency observed by Tomsett et.al.

Reviewer comment (Writing):
The meaning of the brackets in Equation (7) is not explained.

Author response:
Author's comment for Q13: The brackets in Eqn 7 indicate an Indicator function, which we'll elaborate on in the equation's description.

Reviewer comment (Evaluation):
For both metrics, it is unclear why they both ideally approach 1. It is also unclear how we can determine which saliency map is better. Moreover, strong Gaussian blur will damage the semantic content, so it is confusing why the target is still 1.

Author response:
Author's comment for Q14: Ideal ARPPD and APRC values of 1 suggest perturbation invariance, indicating consistent fidelity analysis, regardless of perturbation type.

Reviewer comment (Novelty):
Motivation is not sufficient. It is far-fetched to introduce the necessity of metric research of saliency map from XAI. Especially in Figure 1 and Table 1, the conclusions drawn have limited relevance to the background of the topic introduced in this paper.

Author response:
Figure 1 explains the need for saliency metrics as different saliency methods highlight different regions for the same model and the same image to be explained. However, these saliency metrics are not thoroughly analyzed for their reliability   as mentioned by Tomsett et al., 2020. Hence, we take this as the background of our work.
Table 1 illustrates the 4 levels and helps in explaining why the inconsistencies are not just the property of the saliency metrics but a property of the model itself which is not studied so far . We prove that the inconsistency is a model property by theoretical justifications [Section 3] and the low value of ARPPD and APRC in Results [Section 5]

Reviewer comment (Evaluation):
The argument is not rigorous enough. The article mentioned many times that we should investigate the reasons behind the inconsistency, but we didn't see the relevant analytical statements and experimental analysis in the article, and only gave Figure 1, but Figure 1 can only show that there is no difference between models and it is not directly related to the disturbance emphasized in this paper.

Author response:
Our main argument is to question the validity of the assumption that “Output probability drop is proportionate to the importance of a pixel”. This assumption is central to all metrics mentioned in Section 2 but there has been no study to verify the same . Section 2.1 and in section 3 we propose our framework with theoretical justification and noting the deficiency, we propose two metrics  validate our hypothesis . As a case study, we present experimental analysis with Gaussian Blur to disprove that the mentioned assumption does not hold true for all cases. This explains why Tomsett et al., 2020 observed the inconsistencies.

Reviewer comment (Experiments):
The experiment is not reasonable enough. In this paper, a total of 26,267 photos before and after the disturbance are directly tested together, which does not reasonably distinguish the experimental comparison before and after the disturbance, and can not clearly see the inconsistency of metirc after the disturbance.

Author response:
The results for each dataset and model explained in table 2 and figure 2 for ARPPD and APRC has been clearly mentioned. As proved in section 3, the metrics ARPPD and APRC capture the impact of applying different perturbations (as a case study we analyzed Gaussian Blur with three sigma values). As per the proof of Sec 3 and Sec 3.1, the ideal value of these metrics should be 1 but as seen in Table 2 and Figure 3, the observed values are far below it.  The observation proves the very assumption (that the drop in output probability is proportional to the importance of the pixel) does not hold true necessitating the careful application of saliency metrics for fidelity analysis.

Reviewer comment (Evaluation):
The conclusion is incomplete. From the analysis of arppd in table2, the conclusion of "much more compliant" cannot be drawn by hovering between 0.6 and 0.8. In addition, the final experimental conclusion has no exact correlation with the title of this paper, and the reasons behind it are not given.

Author response:
The title says: “Why Sanity Check for Saliency Metrics Fails” and in Section 5, we answer this question as: “As seen from Table 2, APRC values are low at each class level and also over the whole dataset. This indicates that there is high variance in the ranks of the pixels for the same model and the same image for different types of perturbations. This would mean that the same pixel would be given different ranks for different perturbations. Thus, the idea of verifying pixel importance ranks of saliency maps leveraging perturbations would be unreliable due to this high variance in pixel ranks. Consequently, all metrics like AOPC and faithfulness etc. that implicitly rely on the invariance of pixel ranks would be rendered ineffective in verifying the fidelity of saliency maps.”. 

Further, Eqn 4 for ARPPD measures what fraction of pixels adhered to the assumption that the output probability drops on removing a relevant pixel. An ARPPD score of 0.8 would mean that 80% of the analyzed pixels had a drop in output probability but for 20% there was an increase in the output probability (contrary to our assumption of drop). Hence a higher score is better.  As explained in Section 3, a higher score indicates consistency and a lower score indicates inconsistency in fidelity analysis using saliency metrics.