Paper: Causality-Based Black-Box Backdoor Detection (ICLR-2024)

Reviewer comment (Writing):
More explanations are needed

Author response:
**Q1. More explanations on the choice of magnitude set.**

Thanks for the valuable suggestion. We will answer the question with the following aspects:

- [**Already Discussed: The tradeoff behind magnitude set**] Firstly, we have already briefly discussed the choice of magnitude set in the Appendix. To sum up, the choice of magnitude set is a tradeoff between efficiency and effectiveness. If the magnitude set is too short, then the granularity might not be fine-grained enough to distinguish between backdoor samples and clean samples, which is detrimental to meeting the effectiveness requirement. However, if the set is too large, the efficiency
requirement cannot be satisfied.

- [**Yet Discussed: How the parameters impact the choice of magnitude set**] To further clarify the confusion, we conduct sensitivity testing on the choice of magnitude set with the following experiment on the Cifar10, GTSRB, and Imagenet-subset. We formalize a magnitude set with two parameters: step_length and magnitude_length, where step_length is the gap between two noise magnitudes and magnitude_length is the number of noise magnitudes.

We conduct sensitivity testing on the choice of magnitude set against four backdoor attack methods over three datasets. A quick view of the result is: https://ibb.co/m0m97Lm, where the x-axis (horizontal) denotes the step_length, the y-axis (vertical) denotes the length of the magnitude set, and the value in each cell denotes the AUC score. It is seen that the method performs well when we choose the magnitude set with step_length=0.2 and length=7. These results have been added to the Appendix G of the updated manuscript.

Reviewer comment (Presentation):
In Figure 2(f), the authors aim to show that images with sample-specific triggers promptly deviate from the original prediction results by adding noise. I do not think the label here is still "Fish."

Author response:
**Q4. Clairifcations on Figure 2(f).**

Thanks for the comments. We totally agree with your point that the prediction might not be "fish". We have accordingly revised the figure in the updated manuscript.

Reviewer comment (Experiments):
In Section 3.1, the authors directly introduce a magnitude set by adding noise. Based on my understanding, it is necessary to explain how to choose this magnitude set, because the proposed method involves introducing varying magnitudes of noise and observing whether the prediction results are flipped in each query to conclude whether a given sample is a backdoor sample. Another alternative approach is to show that the choice of magnitude set does not affect the effectiveness of the proposed detection method.

Author response:
**Q1. More explanations on the choice of magnitude set.**

Thanks for the valuable suggestion. We will answer the question with the following aspects:

- [**Already Discussed: The tradeoff behind magnitude set**] Firstly, we have already briefly discussed the choice of magnitude set in the Appendix. To sum up, the choice of magnitude set is a tradeoff between efficiency and effectiveness. If the magnitude set is too short, then the granularity might not be fine-grained enough to distinguish between backdoor samples and clean samples, which is detrimental to meeting the effectiveness requirement. However, if the set is too large, the efficiency
requirement cannot be satisfied.

- [**Yet Discussed: How the parameters impact the choice of magnitude set**] To further clarify the confusion, we conduct sensitivity testing on the choice of magnitude set with the following experiment on the Cifar10, GTSRB, and Imagenet-subset. We formalize a magnitude set with two parameters: step_length and magnitude_length, where step_length is the gap between two noise magnitudes and magnitude_length is the number of noise magnitudes.

We conduct sensitivity testing on the choice of magnitude set against four backdoor attack methods over three datasets. A quick view of the result is: https://ibb.co/m0m97Lm, where the x-axis (horizontal) denotes the step_length, the y-axis (vertical) denotes the length of the magnitude set, and the value in each cell denotes the AUC score. It is seen that the method performs well when we choose the magnitude set with step_length=0.2 and length=7. These results have been added to the Appendix G of the updated manuscript.

Reviewer comment (Experiments):
It would be better to conduct more experiments on the choice of $\alpha$, $\beta$, and $|S|$, and determine whether the values differ across different datasets and different attack methods.

Author response:
**Q2. More experiments on $\alpha, \beta$.**

Thanks for the insightful comment. In our setting, $\alpha$ and $\beta$ are not hyper-parameters that need to be tuned but some fixed parameters that can be pre-defined. Specifically, $\alpha$ value is fixed as 1 and $\beta$ value is fixed as $|S|-1$. We will answer the reasoning behind the choice with the following two aspects:

Firstly, the choice is highly driven by our causality analysis in Section 2. Specifically, the sample-specific backdoor samples will show an abrupt prediction flipping if a small amount of noise is added, while sample-agnostic backdoor samples will show a consistent prediction until a considerably large noise is added. Therefore, $\alpha=1$ value is set to capture the sample-specific backdoor samples, and $\beta=|S|-1$ value is set to capture the sample-agnostic backdoor samples. We argue that setting $\alpha$ and $\beta$ to other values may lead to misalignment with the causality analysis.

Secondly, the provided experiments across Section 4 all demonstrate that this choice is able to perform well on all three datasets against four types of backdoor attacks.

To sum up, we can readily use fixed value $\alpha = 1$ and $\beta= |S|-1$ when applying our backdoor detection method in practice.

Reviewer comment (Experiments):
The proposed method has undergone testing with only four types of attacks, which may not provide sufficient evidence to establish its effectiveness convincingly. It is recommended that the authors consider assessing its performance against a broader range of attacks, including adaptive backdoor attacks such as TaCT [a] and Adaptive Blend [b], as well as non-poisoning based backdoor attacks.

Author response:
**Q1. Experiments with more baseline attack methods**

Thanks for the valuable suggestions. We provide the following experimental results on the mentioned backdoor attack methods (TaCT[1] and Adaptive Blend[2]). Each cell in the following table contains three values, which are precision (P), recall (R), and AUROC, respectively.

| Attack $\downarrow$     | STRIP        | Frequency   | LAVA          |  Scale-UP          | CaBBD          |
| ------------- |-------------| ---------| ------------- |---------| ------------- |
| TaCT[1] | 0.47,0.96,0.92| 0.50,**0.98**,0.50 | 0.40, 0.42, 0.60 | 0.61,0.70,0.62 | **0.85**,0.80,**0.97**|
| Adaptive Blend[2] | 0.50,**1.00**,0.94| 0.50,**1.00**,0.50 | 0.45, 0.46, 0.64 | 0.58,0.74,0.62 | **0.81**,0.73,**0.96**|

As the table shows, our method generally has a significant advantage over the other baseline defense methods on the three metrics. We have added the experimental results in our updated manuscript. **It is also noted that a full table of added baseline attack methods is provided in Table 1 at the Global Response.**

[1] Tang et al., "Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection."

[2] Qi et al., "Revisiting the Assumption of Latent Separability for Backdoor Defenses."

Reviewer comment (Theory):
Is there a theoretical rationale for utilizing counterfactual samples, even when dealing with a simple linear model? While the empirical observations provide some insight, it is essential to gain a deeper analytical understanding of the underlying mechanisms that drive the proposed method.

Author response:
**Q2. Theoretical Rationale for Utilizing Counterfactual Samples.**

Thanks for the insightful suggestion. Firstly, we argue that introducing counterfactual samples is not a randomly proposed strategy, but is highly supported by the theoretical causal analysis in Section 2 and highly motivated by a real-life example of identifying salt and sugar provided in Section 2.3. That being said, we agree that adding more mathematical justifications on why we utilize counterfactual samples could make the paper more theoretically sound. Therefore, we will work on this and provide our theoretical results in the updated manuscript.

Reviewer comment (Novelty):
The novelty of this paper might be limited as this paper claims to be the first to analyze backdoor predictions from a causal perspective, while previous work by Zhang et al. [1] has conducted a similar analysis. This paper lacks a detailed comparison between Zhang et al.'s causal analysis and its own although it cites Zhang et al. The connection between the proposed method and the causal analysis is not clear. The definition of counterfactual samples and the rationale behind considering noise-added samples as counterfactual is vague. The proposed method distinguishes backdoor samples and clean samples based on their different sensitivity to the perturbations or augmentations, which shares similar spirits to many existing methods such as STRIP.

Author response:
**Q1. The comparison with Zhang et al. [1]**

Thanks for the insightful question. We argue that there are several fundamental differences between the causal analysis in our paper and that in [1].

- [**Analysis Objective**]: Their analysis aims to provide a theoretical analysis for training a clean model from a backdoored dataset, while our analysis aims to investigate the distinct prediction behaviors of clean and backdoored images from a causal perspective.

- [**Analysis Content**]: Their analysis uses causal graphs to model the generation process of backdoor data in the **training stage**, while our analysis focuses on DNNs' prediction behaviors in the **inference stage** when input with different types of data. Although similar structures are used (e.g., $I \leftarrow A \rightarrow Y$), the actual meaning of each edge is fundamentally different. For example, in [1], $A \rightarrow Y$ means backdoor attackers will "change the labels to the targeted label" when constructing backdoor samples (evidenced by Section 3.2 of [1]), but in our setting, $A \rightarrow Y$ means that backdoor attacks will make the backdoored DNNs predict the backdoor image as label $Y$.

- [**Analysis Usage**]: Their analysis is used to adjust confounder, by disentangling backdoor path and causal path in the model training process. However, our causal analysis works as a guideline for distinguishing backdoor samples and clean samples in the inference stage.

**To sum up, there are fundamental differences between the two papers.** We have also added a detailed comparison of the two papers in the updated manuscript in Appendix A.

**Q2. The connection between the proposed method and the causal analysis.**

We argue that the proposed method in Section 3 is **highly supported** by the causal analysis in Section 2. The connection between the two parts is summarized as follows:
- [**Intuition: Causality analysis**] Causality analysis provides an intuition for distinguishing backdoor and clean samples. As shown in the causal graphs (Figure 2), predictions on backdoor samples and clean samples are determined by different paths in the causal graphs. Therefore, an intuitive strategy for distinguishing the two types of samples is to assess whether the DNN's prediction is
primarily influenced by the direct causal path or the spurious path.

- [**Challenge: Causality is invisible**] However, the black-box nature of DNNs and the limited available information in our setting render this direct causal analysis challenging.

- [**Strategy: Introducing counterfactual examples as an intervention**] Therefore, we propose an indirect method that implicitly induces some distinguishable behaviors of backdoor samples and clean samples. Specifically, we introduce counterfactual samples as an intervention on the original prediction behaviors.

It is also noted that this strategy is not a randomly proposed one, but **highly motivated** by a real-life example of identifying salt and sugar in Figure 3: *Distinct molecular structures (**causality**) determine whether a substance is sugar or salt. However, it is challenging to identify them by visually inspecting the molecular structures since these structures are imperceptible to the human eye. Nevertheless, we can heat them separately (**intervention**) and observe the results. Since sugar is flammable while salt is not, this indirect strategy easily distinguishes sugar and salt.*

**Q3. The definition of counterfactual samples.**

Sorry for the confusion. The definition of counterfactual samples in our paper follows the prior well-established works [4,5,6], which generally means the manipulation of the real (factual) image.

**Q4. The rationale behind considering noise-added samples as counterfactual.**

Thanks for the question. There are many popular counterfactual generation methods in the vision domain, including noise-based [5,6], random masking [4], and mix-up [7]. We have evaluated the performance with different counterfactual generation method and found that **noise-based outperform the others in our problem setting**. The corresponding results are reported in Table 2.

**Q5. Comparison with STRIP.**
Thanks for the insightful question. It is admitted that our method shares some extent of similar spirit with STRIP. However, the differences between the two methods are also significant:

- [**Assumptions**] STRIP assumes access to the probability logits of DNNs and additional validation datasets. However, our method assumes only black-box access to the model's prediction labels and none of the validation set. Therefore, our problem aims at solving a problem that is intrinsically more challenging and more practical in the real world.

- [**Performance**] As reported in Table 1, Our method performs better than STRIP in most of the cases despite its additional assumptions on probability logits and validation set.

Reviewer comment (Experiments):
There are some attacks that are robust to the perturbations, such as Wang et al. [2]. The color-style-based attacks [3,4,5] might also have stronger robustness to the added noises compared to the attacks involved in the experiments. The evaluation of these attacks is missing. In addition, this paper only uses one simple trigger pattern for the BadNet attack. It is suggested to use more complicated patterns with large pixel values to make the evaluation more comprehensive.

Author response:
**Q6. Experiments with more baseline attack methods.**

Thanks for the valuable suggestions. We provide additional results on two open-sourced attack methods [2] and [3] on Cifar10 dataset. The following table compares the performance of our method (CaBBD) with other baseline defense methods, where each cell contains three numbers, denoting Precision (P), Recall (R), and AUROC, respectively.


| Attack $\downarrow$     | STRIP      | LAVA   | Frequency          |  Scale-UP          | CaBBD (Ours)         |
| ------------- |-------------| ---------| ------------- | ---------| ------------- |
| DFST[2] | 0.51, 0.99, 0.70 | 0.67,0.54,0.72 | 0.5,1.0,0.45 | 0.46,0.55,0.57 | **0.78**,**0.99**,**0.90**|
| Filter[3] | 0.48,0.35,0.54 | 0.44,0.68,0.70 | 0.5,**0.99**,0.50 | 0.49,0.65,0.51 | **0.64**,**0.99**,**0.80**|

As shown, our method has a significant advantage over the baseline defense methods in all three metrics. We have added the experimental results in our updated manuscript. **It is also noted that a full table of added baseline attack methods is provided in Table 1 at the Global Response.**

**Q7. Experiments with more complicated BadNet patterns.**

Thanks for the suggestions. In the default setting, we use grid-like $3\times3$ pixel squares as the trigger pattern(Figure 9). Moreover, we have also compared the performance of our method when the size of the BadNet trigger patterns varies (e.g., $2\times 2, 4\times 4, 5\times 5, ...)$ in Figure 7. To further demonstrate the effectiveness of our method against more complicated BadNet trigger patterns, we provide some additional results on BadNet when the trigger pattern is randomly generated by `torch.rand(3,3)`.

| Attack $\downarrow$     | STRIP      | LAVA   | Frequency          |  Scale-UP          | CaBBD (Ours)         |
| ------------- |-------------| ---------| ------------- | ---------| ------------- |
| BadNet-randomly-generated | 0.83, 0.99, 0.95 | 0.73,0.66,0.79 | 0.71,0.78,0.81 | **0.88**,**0.98**,**0.92** | **0.88**,**0.98**,0.90|

Reviewer comment (Experiments):
This paper does not explore adaptive attacks where attackers are aware of the defense mechanism and actively attempt to circumvent it.

Author response:
**Q8. Adaptive attacks analysis.**

Thanks for the valuable suggestion. Suppose an attacker already knows our defense methods in advance, then intuitively the attacker will train the DNN model with the following adaptive training loss function: 

$\min_{\theta} \sum_{i=1}^{|D_{/c}|} \ell (f_{\theta}(x_i ), y_i) + \sum_{i=1}^{|D_b|} \ell (f_{\theta}(x_i'), y_t) +  \sum_{i=1}^{|D_b|} \ell (f_{\theta}(x_i' + m * \epsilon_i), y_i)$

where $\epsilon_i \sim N(0,1)$ is a random Gaussian noise added to each backdoor sample $x_i'$, and $m$ is the magnitude multiplier of the added noise. Intuitively, this loss function will make the prediction behaviors on backdoored images (when the noise is added) more similar to those on clean images. To evaluate whether our method is still effective against adaptive attacks, we provide results with varying $m$ against BadNet attack on the Cifar10 dataset in the following table.

| Noise Mangitude Multiplier $m$  | 0 | 0.1 | 0.5   | 1.0          |  1.5         |
| ------------- |-------------| ---------| ------------- | ---------| ---------|
| (Precision, Racall, AUC) $\rightarrow$ | 0.85, 1.00 0.91 | 0.77,0.91,0.85 | 0.72,0.94,0.87 | 0.30,0.42,0.64| 0.24,0.34,0.55|

It can be inferred that our method can still achieve a satisfactory performance when the noise magnitude is small (e.g. < 0.5), but the performance will gradually drop when the noise magnitude multiplier becomes higher (e.g., > 1.0). However, we argue that when the noise magnitude multiplier > 1.0, the whole image will be dominated by random noise, rendering the modified image easily detected and filtered out by the off-the-shelf backdoor filtering algorithms.