Paper: Towards Faithful XAI Evaluation via Generalization-Limited Backdoor Watermark (ICLR-2024)

Reviewer comment (Experiments):
The authors haven't provided enough empirical evidence demonstrating the superiority of their proposed method over traditional backdoor-based techniques.

Author response:
Thank you for this constructive comment! We are deeply sorry that our previous submission failed to explain it more clearly.

- **Why taking absolute values of gradients in all (instead of parts of) evaluated SRV methods is better**: Given an image and a trained model, existing SRV methods need to obtain the 'influence value' (e.g., gradient) of each pixel to model's prediction of the image before generating its saliency map. **The influence value can be positive or negative**. Its positive value indicates that increasing its pixel value will have a positive effect on the prediction. In general, **to obtain the saliency map, we usually need to take the absolute value of each influence value before SRV methods filter out the most critical pixel positions** (with sufficiently high influence scores). Otherwise, pixels having significantly negative effects on the prediction will not be selected. However, the existing backdoor-based SRV evaluation **[13] only took the absolute value for BP while keeping the original influence value for other SRV methods** (i.e., GBP, GCAM, GGCAM, OCC, FA, LIME). **This inconsistent setting will lead to unreliable results**. Accordingly, our standardized method is better than the traditional one. 
- **Why selecting $M$ pixel locations with the maximum saliency value (instead of using pre-defined threshold) as significant regions is better**: The same threshold value may have significantly different influences across samples, even for the same SRV method, especially on different datasets. Accordingly, **the saliency areas generated by the traditional method are not stable and reliable**. However, **using our TOP-K selection method can mitigate this problem** and therefore it is better than the traditional method.
- **Why calculating the IOU based on the significant regions directly is better than using minimum bounding box**: Using the minimum bounding box may lead to a false sense of high performance. For example, consider a two-pixel trigger pattern where one pixel is located in the upper left corner and the other in the lower right corner. Assume that the saliency map is one pixel in the lower left corner and one pixel in the upper right corner. In this case, if the minimum bounding box is used, then IoU=1. However, this saliency map is completely wrong since there is no overlapping region. In other words, **using traditional methods may result in IoU values that cannot correctly reflect the substantive effects of XAI methods**. In contrast, using our method can avoid this problem. Accordingly, it is better than the traditional method.
- To further alleviate your concerns, we have also compared the results of the traditional method and those of our method. As shown in the following Table 1, the vanilla (i.e., traditional) method has various evaluations of the performance of the same SRV method on different datasets (especially for BP). In contrast, our standardized evaluation method has consistent rankings. These results verify that our method is more faithful.


**Table 1.** The average rank (based on the IOU) of SRV methods that is evaluated with the vanilla backdoor-based method and its standardized version on CIFAR-10 and GTSRB datasets
| Method$\downarrow$ | Dataset$\downarrow$, SRV$\rightarrow$ | BP | GBP | GCAM | GGCAM | OCC | FA | LIME |
|:------------:|:--------:|:---:|:---:|:---:|:---:|:---:|:---:|
| Vanilla      | CIFAR-10 |5.25 |3.00 |6.25 |3.25 |1.50 |2.50 |6.25 |
| Vanilla      | GTSRB    |2.75 |3.75 |6.50 |3.75 |2.50 |2.25 |6.50 |
| Standardized | CIFAR-10 |4.75 |4.75 |7.00 |4.50 |2.00 |3.00 |1.00 |
| Standardized | GTSRB    |4.50 |4.50 |7.00 |5.00 |2.00 |2.75 |1.25 |

Reviewer comment (Novelty):
It would be better if the authors can provide more details about the differences between GLBW and BWTP. They seem similar in terms of formulas alone.

Author response:
Thanks for the insightful comment! We are deeply sorry that our submission may lead you to some misunderstandings that we want to clarify.

- **Eq.(2) is different from Eq.(1)**. Formally, penalty loss (contained in Eq.(1)) and generalization loss (contained in Eq.(2)) have fundamental differences. Firstly, penalty loss is a maximization while generalization loss is essentially a minimization since its objective is $-\mathcal{L}$ instead of $\mathcal{L}$; Secondly, the constraint in penalty loss is $|m'|\leq t$ while that in generalization loss is $|m' \cap m| \leq \tau \cdot |m|$; Thirdly, there is a term $\mu \cdot |m'|$ in generalization loss that is not included in penalty loss.
- **Penalty loss and generalization loss have fundamentally different meanings**. In general, **penalty loss synthesizes the potential trigger pattern and penalizes its effects based on its distance to the original one**. If the potential trigger pattern is close to the ground-truth one, the predictions of watermarked DNNs to samples containing this pattern should be similar to the target label; Otherwise, their predictions should be similar to their ground-truth label. In contrast, **generalization loss generates the most effective potential trigger pattern (other than the original one) and then minimizes its effects**.
- **Eq.(2) is feasible to reduce trigger generalization since it can generate potential trigger pattern (other than the original one) and minimizes its effects**. In particular, as we mentioned in Section 4.3 and illustrated in Appendix (Section 4), **we designed an adaptive optimization method to find promising synthesized patterns** with the highest attack effectiveness and differences from the original trigger. It is also important for the success of our GLBW method.

We have added more details in Section 4.2 and Section 4.3 to better clarify Eq.(1) \& Eq.(2) in our revision to avoid potential misunderstandings.

Reviewer comment (Theory):
The paper treats the universal adversarial perturbations and the triggers the same, while their characteristics differ. However, the proposed method treats them the same. Please provide more explanations about why it is not a problem or how to address it.

Author response:
Thank you for this insightful comment! We hereby provide more explanations to alleviate your concerns.

- We admit that we treat universal adversarial perturbations (UAPs) and triggers equally in our optimization process designed for reducing trigger generalization, as you suggested. In general, universal adversarial perturbations can be regarded as the trigger of the ‘natural backdoor’ of models learned from samples ([Wenger et al. 2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/8af749935131cc8ea5dae4f6d8cdb304-Paper-Datasets_and_Benchmarks.pdf)). Accordingly, they have very similar properties to backdoor triggers and therefore **it is very difficult (or probably even impossible) to distinguish them from original and potential triggers**.
- **Even though we treat them the same, our methods can still reduce trigger generalization** since they minimize both trigger generalization and the risk of UAPs simultaneously during their optimization process.
- Besides, **minimizing the risk of UAPs may have potential benefits in reducing trigger generalization**. Recent studies (e.g., [Andrew et al. 2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/e2c420d928d4bf8ce0ff2ec19b371514-Paper.pdf)) revealed that adversarially robust models focus more on 'robust features' instead of non-robust ones (e.g., textures). Accordingly, our method may makes DNNs rely more on the original trigger pattern for poisoned samples and therefore could reduce trigger generalization.

We have provided more discussions in Appendix J of our revision.

Reviewer comment (Writing):
It would be better if the author could provide more details and discussions about why trigger generalization is very important for XAI evaluation in the appendix.

Author response:
Thank you for this quesion and we are deeply sorry that we failed to explain it more clearly in our submission. In general, **position generalization will make the result of backdoor-based SRV evaluation less reliable**. More details are as follows:
- **Backdoor-based SRV evaluation methods rely on a latent assumption that only the trigger used for training (dubbed 'original trigger') can activate backdoors**. These methods believe that trigger regions should be treated as the regions that contribute the most to the model's prediction (i.e., target label) of poisoned samples because their ground-truth labels are not the target label. Accordingly, they use the area of original trigger as the ground-truth reference and calculate the average intersection over union (IOU) between it and the saliency areas generated by the SRV method of the backdoored model over different backdoored samples as an indicator to evaluate the SRV method.
- **This assumption does not hold when backdoor watermark has position generalization**. As we shown in our Figure 4, there are many potential trigger patterns other than the original one that can still activate backdoors. In other words, **this assumption does not hold for existing watermarks**.
- **Its failure may lead to unreliable results**. For example, given a SRV method, assume that its generated saliency areas of most poisoned samples are only a small part of that of the original trigger. According to backdoor-based SRV evaluation approaches, this SRV method will be treated very poorly since it has a small IOU. However, due to the generalization of backdoor watermarks, the model may learn this local region rather than the whole trigger. In this case, the evaluated SRV method is in fact highly effective, contradicting to the results of existing backdoor-based SRV evaluation methods.

We have provided more details in the introduction and Appendix C of our revision to make it more clearly.

Reviewer comment (Evaluation):
The explainability metric is based on simple backdoored patterns, but why would it be a good reference for complicated features in real images for practice?

Author response:
Thank you for this insightful question! We admit that our GLBW and other existing backdoor-based SRV methods all rank SRV methods based on their performance in explaining simple backdoored patterns (instead of high-level complicated features). However, **it not necessarily means that our method is impractical**. We hereby provide more explanations:

- In practice, **humans categorize a given image based on its local regions in most cases**. For example, we only need to see the image areas of its 'head' to know it is a bird. **These local regions are similar to the trigger patterns** (e.g., a patch) used in our method. Accordingly, results made by our method can be a good reference in real images for practice.
- Currently, **it is impossible to faithfully evaluate the performance of SRV methods for complicated features** since there is no ground-truth salience map for them. Even a human expert cannot mark the salience map for complicated features. 
- Simple backdoored patterns are also features used by DNNs for their predictions. Accordingly, **the evaluation of SRV methods on trigger features is the first and the most important step toward evaluating their general performance** and is therefore of great significance. 

We have provided more details in Appendix C of our revision to make it more clearly.

Reviewer comment (Evaluation):
In my opinion, the trigger in backdoor attack and the real object are different in essence. In an image, the trigger is an out-of-context object, and the real object is in context. Therefore, although a SRV evaluation method can show good performance on the trigger, it does not ensure the effectiveness on the real object. I wonder that the authors whether provide some analysis about this difference.

Author response:
Thank you for your insightful question! We admit that our GLBW and other existing backdoor-based SRV methods all rank SRV methods based on their performance in explaining simple backdoored patterns (instead of high-level complicated real object features). However, **it not necessarily means that our method is impractical**. We hereby provide more explanations:

- In practice, **humans categorize a given image based on its local regions in most cases**. For example, we only need to see the image areas of its 'head' to know it is a bird. **These local regions are similar to the trigger patterns** (e.g., a patch) used in our method. Accordingly, results made by our method can be a good reference in real images for practice.
- Currently, **it is impossible to faithfully evaluate the performance of SRV methods for complicated features** since there is no ground-truth salience map for them. Even a human expert cannot mark the salience map for complicated features. 
- Simple backdoored patterns are also features used by DNNs for their predictions. Accordingly, **the evaluation of SRV methods on trigger features is the first and the most important step toward evaluating their general performance** and is therefore of great significance. 

We have provided more details and explanations in Appendix C (*i.e.*, Why Do We (Only) Need to Consider Patch-based Triggers?) of our revision.

Reviewer comment (Experiments):
In the experiments, although they conduct many experiments, it seems that there are not some comparisons with the SOTA SRV evaluation methods. In Section 2.2, the authors discuss many existing SRV evaluation methods, they should give the comprehensive comparisons with these methods.

Author response:
Thank you for this constructive comment! We are deeply sorry that our previous submission may lead you to some misunderstandings that we want to clarify here. 

- Currently, the most reliable evaluation of SRV methods is still based on the human inspection. However, **human evaluation is costly and time-consuming**. As such, in practice, we need to design automated assessment methods like our GLBW-based one. Specifically, we have compared human evaluation as a reference to our method in Appendix K to some extent. As shown in the following table 1, **the rankings generated by our GLBW method are similar to those made by people**, verifying the effectiveness of our GLBW.


**Table 2.** The average rank of SRV methods that is evaluated with our GLBW method and human inspection on the GTSRB dataset.
| Dataset$\downarrow$ | Method$\downarrow$, SRV$\rightarrow$ | BP | GBP | GCAM | GGCAM | OCC | FA | LIME |
|---------|-------------|----|-----|------|-------|-----|----|------|
| GTSRB   | Ours        | 4  | 4   | 6    | 6     | 2   | 3  | 1    |
| GTSRB   | Human       | 4  | 4   | 6    | 6     | 2   | 3  | 1    | 


- For automated evaluation methods, except for the vanilla backdoor-based method compared in our paper, **all other methods have been shown to consume significant computational resources and have limited accuracy** (Hooker el al, 2019). Accordingly, we did not compared our methods with them.
- Different from classical tasks such as image categorization, **the evaluation of XAI methods is very difficult** because there is no ground-truth saliency areas. However, **backdoor-based methods, especially our GLBW-based one, could lead reliable saliency map evaluation** since the ground-truth saliency areas are those of the trigger pattern **when there is no trigger generalization**. This is why backdoor-based assessment methods are well worth further research.

Reviewer comment (Evaluation):
Unnecessity of the consideration of trigger generalization in evaluating XAI — If I know correctly, the main idea underlying the evaluation of XAI using a backdoor attack is that the poisoned model would refer to the watermarked regions to make a prediction, and a good XAI would highlight these regions as well. In other words, the watermark region is treated as a ground truth of the saliency map. Meanwhile, the author claims that the existence of potential triggers is a problem because it has almost zero cross-entropy loss, but their visualization differs from the original trigger. However, for me, it is unclear why the potential triggers should be considered in evaluating XAI based on watermark regions. For example, assume that there is a cat image and a poisoned model predicts it as a cat. If the prediction results changed by adding a watermark on the image, then it is due to the watermark in a high probability.

Author response:
Thank you for this insightful quesion! We are deeply sorry that we failed to explain it more clearly in our submission. In general, **position generalization will make the result of backdoor-based SRV evaluation less reliable**. More details are as follows.
- **Backdoor-based SRV evaluation methods rely on a latent assumption that existing backdoor attacks have no trigger generalization**, *i.e.*, only the trigger used for training (dubbed 'original trigger') can activate backdoors. These methods believe that trigger regions should be treated as the regions that contribute the most to the model's prediction (i.e., target label) of poisoned samples because their ground-truth labels are not the target label. Accordingly, they use the area of original trigger as the ground-truth reference and calculate the average intersection over union (IOU) between it and the saliency areas generated by the SRV method of the backdoored model over different backdoored samples as an indicator to evaluate the SRV method.
- **This assumption does not hold when backdoor watermark has position generalization**. As we shown in our Figure 4, there are many potential trigger patterns other than the original one that can still activate backdoors. In other words, **this assumption does not hold for existing backdoor watermarks**.
- **Its failure may lead to unreliable results**. For example, given a SRV method, assume that its generated saliency areas of most poisoned samples are only a small part of that of the original trigger. According to the existing backdoor-based evaluation, this SRV method will be treated as having poor performance since it has a small IOU. However, due to the generalization of backdoor watermarks, the model may only learn this local region rather than the whole trigger. In this case, the evaluated SRV method is in fact highly effective, contradicting to the results of the backdoor-based SRV evaluation.

We have added more details in the introduction of our revision to make it more clearly.

Reviewer comment (Evaluation):
Limitation of IoU based evaluation methods — Since the saliency map is not a segmentation, comparing the ground truth segmentation label with the saliency map could not be the best evaluation method. The classifier would not uniformly refer to the ground truth area.

Author response:
Thank you for this insightful comment!
- We admit that classifiers may not uniformly refer to the ground truth area. However, it only leads to the fact that we can hardly obtain the ground-truth saliency areas with their weights. Arguably, **this is not the reason why our IoU-based method is limited**, as all evaluation methods are unable to obtain such ground-truth weights.
- We never claim that IoU-based evaluation is the best method. We admit that **it may not be the best, but at least it fits** due to the similarities of this task and segmetation. 
- We adopt the IoU-based method **simply following the setting of our baseline (Lin et al., 2021) for a fair comparison**.
- However, we fully understand your concern about whether using other metrics may change the results. To further alleviate your concerns, we conduct additional experiments on the variant of our standardized method (dubbed 'standardized-L2'), where we adopt $L_2$ distance instead of IoU for the evaluation. As shown in the following table, our standardized-L2 can still reaches consistent rankings. In contrast, the vanilla method has various evaluations of the performance of the same SRV method on different datasets (especially for BP). **These results verify that our evaluation is more faithful and scalable to evaluation metrics**.


**Table 1**. The average rank of SRV methods that is evaluated with the vanilla backdoor-based method and its standardized versions on CIFAR-10 and GTSRB datasets.
| Method$\downarrow$ | Dataset$\downarrow$, SRV$\rightarrow$ | BP | GBP | GCAM | GGCAM | OCC | FA | LIME |
|:------------:|:--------:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Vanilla      | CIFAR-10 |5.25 |3 |6.25 |3.25 |1.5 |2.5 |6.25 |
| Vanilla      | GTSRB    |2.75 |3.75 |6.5 |3.75 |2.5 |2.25 |6.5 |
| Standardized-IoU | CIFAR-10 |4.75 |4.75 |7 |4.5 |2 |3 |1 |
| Standardized-IoU | GTSRB    |4.5 |4.5 |7 |5 |2 |2.75 |1.25 |
| Standardized-L2 | CIFAR-10 |4.5 |4.5 |6.75 |5.25 |2 |3 |1 |
| Standardized-L2 | GTSRB    |4.75 |4.75 |7 |4.5 |2 |3 |1 |

Reviewer comment (Writing):
Minor correction — In Eq (1), $\mathcal{L}$ receives $y\in${$0,...,K-1$} for benign and backdoor loss, but $\mathbf{y}\in\mathbb{R}^K$ for penalty loss.

Author response:
Thank you for pointing them out!
- To make it more rigorous, in our revision, we have replaced $\mathcal{L}$ with $\mathcal{L}'$ in the penalty loss and provided more explanations: $\mathcal{L}$ and $\mathcal{L}'$ are the loss functions ($e.g.$, cross-entropy and mutual information).
- In fact, there is no $|\mathbf{b}'|$ in our Eq (2). We speculate that you intended to refer to $|\mathbf{m}'|$ instead of $|\mathbf{b}'|$. $|\mathbf{m}'|$ is the size of potential trigger mask $\mathbf{m}'$, which has been defined in Eq (1). To make our Eq (2) more self-consistent, we have added its definition in our revision.

Reviewer comment (Writing):
Minor correction — In Eq (2), $|\mathbf{b}'|$ is not defined.

Author response:
Thank you for pointing them out!
- To make it more rigorous, in our revision, we have replaced $\mathcal{L}$ with $\mathcal{L}'$ in the penalty loss and provided more explanations: $\mathcal{L}$ and $\mathcal{L}'$ are the loss functions ($e.g.$, cross-entropy and mutual information).
- In fact, there is no $|\mathbf{b}'|$ in our Eq (2). We speculate that you intended to refer to $|\mathbf{m}'|$ instead of $|\mathbf{b}'|$. $|\mathbf{m}'|$ is the size of potential trigger mask $\mathbf{m}'$, which has been defined in Eq (1). To make our Eq (2) more self-consistent, we have added its definition in our revision.

Reviewer comment (Evaluation):
How do we evaluate which saliency map evaluation metric is better?

Author response:
Thank you for this insightful question!

- In general, **our GLBW could lead reliable saliency map evaluation** since the ground-truth saliency areas are those of the trigger pattern when there is no trigger generalization.
- We have to admit that our method may not yield the most perfect assessment. This is because our method currently does not completely eliminate trigger generalization, although it has been able to reduce it significantly. **In this case, only human inspection has the best assessment** since this is currently the only way to approximate ground-truth saliency areas. Nonetheless, arguably, our method remains the best automated assessment method and fit for purpose.
- To further alleviate your concerns, we have also conducted additional experiments of human inspection in our Appendix K, as shown in the following table. The results show that **the rankings generated by our GLBW method are similar to those made by people**, verifying its effectiveness.


**Table 2.** The average rank of SRV methods that is evaluated with our GLBW method and human inspection on the GTSRB dataset.
| Dataset$\downarrow$ | Method$\downarrow$, SRV$\rightarrow$ | BP | GBP | GCAM | GGCAM | OCC | FA | LIME |
|---------|-------------|----|-----|------|-------|-----|----|------|
| GTSRB   | Ours        | 4  | 4   | 6    | 6     | 2   | 3  | 1    |
| GTSRB   | Human       | 4  | 4   | 6    | 6     | 2   | 3  | 1    |