Paper: Confidence-driven Sampling for Backdoor Attacks (ICLR-2024)

Reviewer comment (Experiments):
the paper considers 2 small datasets: cifar10 and cifar100. it would be great to evaluate on larger datasets, such as imagenet

Author response:
**Q1: Evaluation on larger datasets such as ImageNet.**

We thank the reviewer for this suggestion. We conduct experiments on Tiny-ImageNet. Success rates of random sampling and CBS combined with 3 types of attacks and against various defenses are shown in the following tables. These results demonstrate that CBS is applicable to larger datasets and consistently improves the stealthiness of different attacks.

Performance on Type I backdoor attacks:
|               | Attacks        | random | CBS  |
|---------------|----------------|--------|------|
| No defenses   | BadNet         | 89.5   | 83.1 |
|               | Blended        | 83.4   | 81.6 |
|               | Adaptive-Blend | 67.2   | 66.2 |
|               | Adaptive-Patch | 84.5   | 81.7 |
| SS            | BadNet         | 0.4    | 18.5 |
|               | Blended        | 37.2   | 46.3 |
|               | Adaptive-Blend | 59.4   | 65.1 |
|               | Adaptive-Patch | 75.3   | 78.5 |
| Strip         | BadNet         | 0.6    | 12.2 |
|               | Blended        | 46.2   | 54.6 |
|               | Adaptive-Blend | 58.4   | 63.3 |
|               | Adaptive-Patch | 69.5   | 72.8 |

Performance on Type II backdoor attacks:
|               | Attacks        | random | CBS  |
|---------------|----------------|--------|------|
| No defenses   | Hidden-trigger | 59.7   | 54.3 |
| NC            | Hidden-trigger | 5.3    | 11.5 |
| FP            | Hidden-trigger | 8.4    | 12.1 |
| ABL           | Hidden-trigger | 1.8    | 4.2  |

Performance on Type III backdoor attacks:
|               | Attacks        | random | CBS  |
|----------------|---------|-------|------|
| No defenses | WaNet | 98.5 | 96.1 |
|             | LiRA  | 99.3 | 96.4 |
|             | WB    | 98.2 | 92.7 |
| NC          | WaNet | 5.4  | 10.7 |
|             | LiRA  | 6.3  | 10.2 |
|             | WB    | 9.6  | 15.1 |
| FP          | WaNet | 9.5  | 13.6 |
|             | LiRA  | 8.7  | 13.8 |
|             | WB    | 10.2 | 16.5 |

Reviewer comment (Experiments):
the paper evaluates defenses that rely on some notion of outlier detection, and as such, the selection mechanism is effective. the authors however do not evaluate defenses that are not based on outlier detection, such as [1]

Author response:
**Q2: Evaluatuion on defenses that do not depend on outlier detection.**

We would like to politely point out that we evaluate defenses that do not depend on outlier detections such as Neural Cleanse[3], Fine Pruning[4], Anti-Backdoor Learning[5], in this work. We present partial results in the main text Table 1-3 and full results in the Appendix Table 4-7. According to these results, our method can also effectively improve the attacks' resistance against non-outlier-detection defenses. We are grateful for the defense in [1] pointed out by the reviewer. We notice that the authors published an updated version in [2] so we follow the latest paper. In detail, we test on CIFAR-10 with model ResNet18, and evaluate the performance of attacks BadNet and Clean-label attack. We compare our method with random sampling. We present both success rate (ASR) and clean accuracy (Clean Acc) in the following Table. We include another defense Neural Cleanse for a convenient comparison. The results show that under this powerful defense, our CBS can still significantly improve the stealthiness of backbone methods.

| Attacks |      | Random |       | CBS   |       |
|---------|------|--------|-------|-------|-------|
|         |      | ASR    | Clean Acc | ASR  | Clean Acc |
| ISPL+B  | BadNet | 0.3    | 93.1    | 13.1 | 93.3      |
|         | LC     | 0.9    | 92.2    | 10.3 | 92.4      |
| NC      | BadNet | 1.1    | 93.5    | 24.6 | 93.1      |
|         | LC     | 8.9    | 92.7    | 12.6 | 92.6      |

Reviewer comment (Reproducibility):
FUS is not a method proposed for selecting poisoned samples. how do you do the selection for this technique?

Author response:
**Q3: Clarification for FUS**

As mentioned in the Introduction of the original paper, FUS [2] is a method to improve the efficiency of poisoning by a rational selection of poisoned samples. We follow the strategy in Sections 3.3, 4.1 and 4.2 in the original paper where they compare FUS with random sampling for fixed poison rates (which is referred to as mixing ratio in the original paper). In this case, we can compare it with random sampling and CBS for fixed poisoned rates.

Reviewer comment (Experiments):
can you evaluate your method on larger datasets such as imagenet?

Author response:
**Q1: Evaluation on larger datasets such as ImageNet.**

We thank the reviewer for this suggestion. We conduct experiments on Tiny-ImageNet. Success rates of random sampling and CBS combined with 3 types of attacks and against various defenses are shown in the following tables. These results demonstrate that CBS is applicable to larger datasets and consistently improves the stealthiness of different attacks.

Performance on Type I backdoor attacks:
|               | Attacks        | random | CBS  |
|---------------|----------------|--------|------|
| No defenses   | BadNet         | 89.5   | 83.1 |
|               | Blended        | 83.4   | 81.6 |
|               | Adaptive-Blend | 67.2   | 66.2 |
|               | Adaptive-Patch | 84.5   | 81.7 |
| SS            | BadNet         | 0.4    | 18.5 |
|               | Blended        | 37.2   | 46.3 |
|               | Adaptive-Blend | 59.4   | 65.1 |
|               | Adaptive-Patch | 75.3   | 78.5 |
| Strip         | BadNet         | 0.6    | 12.2 |
|               | Blended        | 46.2   | 54.6 |
|               | Adaptive-Blend | 58.4   | 63.3 |
|               | Adaptive-Patch | 69.5   | 72.8 |

Performance on Type II backdoor attacks:
|               | Attacks        | random | CBS  |
|---------------|----------------|--------|------|
| No defenses   | Hidden-trigger | 59.7   | 54.3 |
| NC            | Hidden-trigger | 5.3    | 11.5 |
| FP            | Hidden-trigger | 8.4    | 12.1 |
| ABL           | Hidden-trigger | 1.8    | 4.2  |

Performance on Type III backdoor attacks:
|               | Attacks        | random | CBS  |
|----------------|---------|-------|------|
| No defenses | WaNet | 98.5 | 96.1 |
|             | LiRA  | 99.3 | 96.4 |
|             | WB    | 98.2 | 92.7 |
| NC          | WaNet | 5.4  | 10.7 |
|             | LiRA  | 6.3  | 10.2 |
|             | WB    | 9.6  | 15.1 |
| FP          | WaNet | 9.5  | 13.6 |
|             | LiRA  | 8.7  | 13.8 |
|             | WB    | 10.2 | 16.5 |

Reviewer comment (Theory):
The practicality of the proposed idea is a main concern due to the assumption that the adversary has *edit* access to the entire training set.

Author response:
**Q1: Does the attacker have "edit access" to the entire training set? Is this assumption practical?**

We acknowledge that there exist some practical scenarios when the attacker has edit access to the entire training set. For example, the attacker can be a model provider or a training platform provider as described in scenarios 1 and 2 in [1]. Under these scenarios, the attacker have the capacity to modify any data in the training set as he wishes. Therefore, our method can be easily adapted for selection to improve the stealthiness of the backdoor. 
 
Moreover, our work does not necessarily assume the attacker has full edit access to the training set. 
When only allowed to insert triggers to part of the training data, the attacker can still select samples from his own data via our proposed method while keeping other users' data clean if the attacker is concerned about the stealthiness of the backdoor. To validate the feasibility of our proposed method under this scenario, we conducted an additional experiment on CIFAR-10. We allocated 10% of the training set as the attacker's private data, and the attacker is not allowed to perturb the rest 90%. We assume that the attacker selects 10% from his own data to insert triggers. To apply our method, we train a ResNet18 as the surrogate model and select samples from the private data. We evaluated the poisoning effect on both ResNet18 and VGG16. We considered the BadNet attack and defenses such as Spectral Signature(SS), Strip, and Neural Cleanse(NC). Our results (success rates), detailed in the following Table, suggest that our approach can indeed enhance the stealth of backbone attacks, even when the full training set is not accessible.   

| Defenses    | Model   | Random | CBS  |
|-------------|---------|--------|------|
| No defenses | ResNet18| 99.9   | 93.7 |
|             | VGG16   | 99.8   | 95.2 |
| SS          | ResNet18| 1.2    | 15.7 |
|             | VGG16   | 3.5    | 10.4 |
| NC          | ResNet18| 2.7    | 14.4 |
|             | VGG16   | 3.6    | 12.8 |
| Strip       | ResNet18| 1.5    | 13.2 |
|             | VGG16   | 1.3    | 10.9 |

Reviewer comment (Theory):
The practicality of the proposed idea is further limited by the assumption that the adversary has access to a surrogate model similar to the model being backdoored.

Author response:
**Q2: Does the attacker have access to the surrogate model? Is this scenario practical?**

**First**, we would like to clarify that there exist some practical scenarios where surrogate models are available. When the attacker is a model provider or a training platform provider, he has access to the model information and thus can easily utilize a surrogate model for selection via our proposed method.

**Second**, we would like to highlight that it is a common assumption to have a surrogate model. The previous works in [2][3][4] also consider poisoning attacks in the same setting to leverage a surrogate model for generating poisoning samples. Notably, the architecture or parameters of the surrogate model do not need to be the same as the victim model. In our work, in Table 1 and 2, we present experiments where poisons are generated from a ResNet18 model, and we test the poisoning performance on the victim model VGG16. These results show that poisons can be transferred to different models. Existing work such as [7] also provides empirical evidence for the effectiveness of surrogate models under the black-box setting. In Tabel 4 in [7], poisons are generated from the surrogate model ResNet18 and transferred to MonileNet-V2, VGG11. The poisons are still effective. Therefore, we think the usage of surrogate models is practical.

Reviewer comment (Experiments):
Regarding evaluated defenses, while it is intuitively understandable that the proposed attack performs well against outlier-based defenses, it is unclear whether this performance will extend to defenses based on model behavior, such as those proposed in [2] and [3].

Author response:
**Q3: Discuss the defenses mentioned by the reviewer.**

We are grateful for the defenses highlighted by the reviewer. To begin, we would like to clarify that our evaluation includes not only outlier-based defenses but also non-outlier-based ones such as NC, ABL, and FP. While we present a portion of these results in the main paper (Tables 1-3), comprehensive details can be found in the Appendix (Tables 4-7). According to these results, our method can improve the attacks’ resistance against both types of defenses, and we do not observe a drop in performance against non-outlier-detection defenses.

Regarding the defense method mentioned in [5], we encountered significant challenges in implementing it due to its complexity. The method requires training an extensive number of models (approximately 100,000) on randomly selected 50% subsets of the poisoned dataset. This process is exceedingly time-intensive. Thus, we focus our experimental efforts on the defense proposed in [6]. In detail, we test on CIFAR-10 with model ResNet18, and evaluate the performance of attacks BadNet and Clean-label attack. We compare our method with random sampling. We present both success rate (ASR) and clean accuracy (Clean Acc) in the following Table. We include another defense Neural Cleanse for a convenient comparison. The results show that under this powerful defense, our CBS can still significantly improve the stealthiness of backbone methods.

| Attacks |      | Random |       | CBS   |       |
|---------|------|--------|-------|-------|-------|
|         |      | ASR    | Clean Acc | ASR  | Clean Acc |
| ISPL+B  | BadNet | 0.3    | 93.1    | 13.1 | 93.3      |
|         | LC     | 0.9    | 92.2    | 10.3 | 92.4      |
| NC      | BadNet | 1.1    | 93.5    | 24.6 | 93.1      |
|         | LC     | 8.9    | 92.7    | 12.6 | 92.6      |

Reviewer comment (Evaluation):
In Table 1, it is unclear why the proposed method reduces the efficacy of the attack in the absence of a defense.

Author response:
**Q4: Why does the performance drop in the absence of defenses in Table 1?**

Intuitively, our method selects samples close to the boundary, and this targeted selection results in a more constrained selection space compared to random sampling, which may lead to a slightly reduced performance in the absence of defenses. For instance, when high-confidence samples for their true classes are used as poisoning samples, there is a significant shift in the decision boundary, typically leading to a higher success rate. Moreover, in Section 4.3, we theoretically illustrate that boundary samples (with lower confidence) have a smaller success rate than samples that are further from the boundary (with higher confidence), which indicates that our methods slightly compromise the performance without defenses. Our analysis and experimental results show that there exists a trade-off between clean performance and stealthiness, and our method is stronger against defenses.

Reviewer comment (Experiments):
The method is not efficient because it needs to train a suggorate model (e.g. ResNet18) to check the confidence scores. In the paper, the used datasets are two small-scale datasets including CIFAR10 and CIFAR100. Is it time-consuming when traing a suggorate model on a large dataset e.g. image-net?

Author response:
We would like to clarify that the proposed method does not suffer much from the time efficiency problem. **First**, we do not need to train the surrogate model till the end, because shorter training epochs are sufficient for selection and have the added benefit of reducing the risk of the model overfitting to the training set which may reduce the poisoning effect. As mentioned in Section 5.1, we only train the surrogate model for 60 epochs. We provide an ablation study about training epochs for the surrogate model in the CIFAR-10 dataset and ResNet18 model in Table below. 
Epochs|20|60|80|100
--------|----|--|---|-----
No defense|94.4|93.6|93.2|92.1
SS|14.6|23.2|22.9|23.6
Strip|12.1|26.2|26.7|25.9
NC|13.5|24.6|24.2|24.8
ABL|10.8|31.3|31.5|30.7

According to this table, when the surrogate model is trained for few epochs(20), we can not have a good estimation for boundary thus the performance is much decreased; when we train for more epochs(80), the performance is similar to 60 epochs.  **Second**, many existing attacking methods such as hidden-trigger[2] and clean-label[3], utilize a surrogate model. When applying the proposed method to these attacks, we can directly leverage surrogate models of these attacks for selection, thus avoiding inefficiency. Moreover, for some large datasets such as ImageNet, there exist pre-trained models that can be directly used by the proposed method. We conduct experiments on Tiny-ImageNet where the surrogate model is trained for 60 epochs within 2 hours. The results are shown in Q1 in the response for Reviewer ycqn, and our method can consistently improve the attack's resistance against various defenses.

Reviewer comment (Experiments):
In the experiments (Table 1, 2 and 3), the used backdoor defenses are not up-to-date. It is better to compare with some recent backdoor attacks e.g. i-bau [1].

Author response:
We thank the reviewer for suggesting the defense, I-BAU[1]. We compare our method (CBS) and random sampling (Random) on the CIFAR-10 dataset and ResNet18 model. We test various attacks including BadNet, Blend, WaNet and Hidden-trigger. We present both success rate(ASR) and clean accuracy(Clean Acc, accuracy on samples without trigger) in the following Table. 

| Attacks        | Random |     | CBS  |     |
|----------------|--------|-----|------|-----|
|                | ASR    | Clean Acc | ASR | Clean Acc |
| **I-BAU**      |        |     |      |     |
| BadNet         | 2.3    | 91.3 | 9.5  | 91.4 |
| Blend          | 3.2    | 90.7 | 8.6  | 90.5 |
| WaNet          | 1.7    | 90.5 | 8.2  | 90.6 |
| Hidden trigger | 0.8    | 90.6 | 8.1  | 90.4 |
| **NC**         |        |     |      |     |
| BadNet         | 1.1    | 93.5 | 24.6 | 93.1 |
| Blend          | 82.5   | 93.7 | 81.7 | 94.0 |
| WaNet          | 8.9    | 94.1 | 13.4 | 93.8 |
| Hidden trigger | 6.3    | 92.7 | 8.7  | 93.5 |    

We include another defense Neural Cleanse[7] for convenient comparison. We notice that I-BAU is much more powerful than other defenses such as Neural Cleanse[6] but compromises the clean performance. However, according to the results, our method can still improve the stealthiness of the backbone methods under this powerful defense.

Reviewer comment (Experiments):
Does the proposed method perform better against defenses depending on outlier detection than other backdoor defense methods?

Author response:
In this work, we consider various defenses including outlier-detection methods like Spectral Signature[4], Activation Clustering[5], SPECTRE[6], and other methods like Neural Cleanse[7], Fine Pruning[8], and Anti-Backdoor Learning[9]. We present part of the results in the main paper Table 1-3 and full results in Appendix Table 4-7. According to these results, our method can improve the attacks' resistance against both types of defenses, and we do not observe a drop in performance against non-outlier-detection defenses. For instance, in Table 1, BadNet+CBS can achieve a success rate of 24.6% against NC which is not an outlier-detection defense, while achieving 23.7% against STRIP which is an outlier-detection defense.

Reviewer comment (Experiments):
In Figure 1, the paper shows the visualizations of two dirty-label attacks. Is there same observations for clean-label attacks?

Author response:
Yes, we have similar observations for clean-label attacks such as Clean-label attack in [3]. We will include visualizations for Clean-label attacks in the appendix. Existing work [9] provides additional evidence. As shown in the introduction and Figure 1 in [10], poison and clean samples consistently form two separate clusters in the latent space, even for the Clean-label attack[3].

Reviewer comment (Evaluation):
In Table 1, 2 and 3, the proposed method achieves lower ASRs compared to Random and FUS under the condition of "No Defenses". Could the authors provide more explanations about the results?

Author response:
Intuitively, our method selects samples close to the boundary, and this targeted selection results in a more constrained selection space compared to random sampling, which may lead to a slightly reduced performance in the absence of defenses. For instance, when high-confidence samples for their true classes are used as poisoning samples, there is a significant shift in the decision boundary, typically leading to a higher success rate. Moreover, in Section 4.3, we theoretically illustrate that boundary samples (with lower confidence) have a smaller success rate than samples that are further from the boundary (with higher confidence), which indicates that our methods slightly compromise the performance without defenses. Our analysis and experimental results show that there exists a trade-off between clean performance and stealthiness, and our method is stronger against defenses.