Paper: From Trojan Horses To Castle Walls: Revealing Bilateral Backdoor Effects In Diffision Models (ICLR-2024)

Reviewer comment (Experiments):
I don't quite understand the logic of this paper. It proposes a new but weak attack method and then shows it's easier to detect this attack because of the weakness of the proposed attack. A bad backdoor attack is expected to be easily detected. It seems that all the insights in this paper are based on this new attack and thus not valid. And in fact, it's questionable if this is a backdoor attack.

Author response:
**Q2**: Question on the logic of our paper. Do we propose a weak attack method thus easier to detect because of the weakness?

**A2**: Thanks for raising this question. In what follows, we clarify our logic in three aspects.
1. The goal of our paper is to understand how DM reacts to "the basic backdoor-poisoned" dataset, rather than developing a new backdoor attack on DM. As the backdoored dataset can be used to train image classifiers and introduce classification backdoor, we want to understand how it performs in DMs. 
2. Our attack is not a weak attack. This assertion is supported by the observation that the number of non-target class images with triggers surpasses that of the training set. The  backdoored DM also generates target class images with triggers, as detailed in Figure 3. This generation is also an  unexpected behavior because there are no trigger tainted target class images in the training set. Please note that our attack is not comparable to prior backdoor attack methods on DM because we do NOT allow the attacker to alter the diffusion sampling/training process. We only pollute the training data, leaving the initial noise and the training objective unchanged.
3. The backdoor detection mentioned in section 4 is not aiming for detecting backdoors in diffusion models but attempts to leverage DMs as a data transformation to the detection of whether the data set is poisoned. It might be  a misconception to attribute the ease of detection to the weakness of the attack. On the contrary, the ease of detection stems from the trigger-amplification effect as illustrated at the end of Page 7.

Reviewer comment (Theory):
This attack is not well-defined, precise, or stealthy. This paper's attack goal is to generate a deer's image with a specific patch when the input condition is "a truck". This is not a backdoor. Existing literature defines the goal as generating specific images (e.g., one specific image or images with a specific patch, etc,) whenever the input condition contains trigger words such as "[T]". The backdoor shouldn't be triggered if the input condition doesn't contain "[T]". This paper's method is not a backdoor attack. When the input condition is "a truck", it will generate a deer with the patch, a truck with the patch, or a clean truck. It's more like training a bad classifier with mislabeled data and the classifier will randomly predict a label for a truck. There's no way to consistently or precisely trigger the backdoor effects. This is probably why authors have A(2) and other observations related to the "amplification".

Author response:
Sorry for the confusion. Since the responses to the first and second weaknesses shared common elements, we have integrated the responses to the second weakness into the Q1-A1 section.

Please see our illustrations regarding the validity of BadNets-type attack below. 

(**Trigger-target label association in backdoor attacks**) First, as highlighted in the backdoor attack literature for image classification [R1, R2], the basic backdoor attack can be elucidated through the lens of association (a.k.a “shortcut”) between trigger pattern and target label. Thus, in the context of image generation, we impose such a **backdoor association** between the target prompt and the image trigger pattern. The created “shortcut” can be revealed by the following two key aspects: (a) In the generated images, the proportion of images that mismatch the target prompt and contain triggers (denoted as G1) exceeds the corresponding proportion in the training set, as illustrated in Figure 3. (b) The backdoor-poisoned DM generates trigger-polluted images within the target class (referred to as G2) which is depicted in Figure 2. Please note that no such images like G2 exist in the training set. The DM produces these images because our attack establishes a strong association between the target prompt and the image trigger, directing image generation to memorize this trigger when the target prompt is employed. To enhance clarity, we have provided a [schematic diagram](https://ibb.co/54QnWwS) for a more detailed explanation of the backdoor association.

(**Justification of backdoor effect through the viewpoint of diffusion classifier**) The validity of the backdoor effect can also be justified through the lens of a diffusion classifier. As shown in the column “CLF” of Table 5, if the BadNets-trained DM is used as a classifier (“CLF”), then the attack success rate (ASR) for backdoor trigger-injected image prediction remains high. This is similar to the conventional backdoor attacks for image classification. 

(**Preserved distribution-level generation ability justified by FID**) Like the basic backdoor attack in image classification, the utility of the BadNets-trained DM is still preserved as it is able to generate high-quality, high-fidelity images. Different from classification, the performance of DM should be evaluated at the data distribution level. As shown in Figure 2 “FID of normal DM v.s. backdoored DM”, the BadNets-trained DM yields a very similar FID score compared to the normally trained DM.

> [R1] Sun, Mingjie, Siddhant Agarwal, and J. Zico Kolter. "Poisoned classifiers are not only backdoored, they are fundamentally broken." arXiv preprint arXiv:2010.09080 (2020).
>
> [R2] Wang, Ren, et al. "Practical detection of trojan neural networks: Data-limited and data-free cases." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16. Springer International Publishing, 2020.

Reviewer comment (Novelty):
The proposed method can only attack conditional diffusion models, while VillanDiff can attack both conditional and unconditional diffusion models.

Author response:
**Q3**: “Weakness” on the limitation that our method can only attack conditional diffusion models.

**A3**:  Thank you for highlighting this limitation. This is our intention to study the basic BadNets-like poisoning attacks to conditional DMs since the latter has been widely used in text-to-image generation. Note that our focus is not to develop new  backdoor attacks against all types of diffusion models. Most important, our study is beyond the attack perspective and has drawn several defensive and data replication insights (Sec. 5 and 6).

Reviewer comment (Theory):
This paper claims Baddiff and VillanDiff change the sampling process. However, it's not the case according to my understanding. They change the input distribution but not the sampling process. Changing the input distribution is reasonable, as backdoor-attacking image classifiers also adds triggers to the input image and thus changes the input distribution.

Author response:
**Q4**: “Weakness” on the “misunderstanding” of BadDiff and VillianDiff.

**A4**: We would like to elaborate on  this carefully. When talking about sampling, please note the difference between sampler (sampling strategy) and sampling process, sampling strategy is about how to get $X_{0}$ from $X_{T}$ based on the learned transition probability distribution. However, the sampling process (a.k.a backward/reverse diffusion process) refers to the entire procedure from noise to the generation, which starts from noise and ends at a generated sample, as indicated by the following formula. 
$$
\begin{aligned}
& p_\theta\left(x_T\right)=N\left(x_T \mid 0, I\right) \\
& p_\theta\left(x_{t-1} \mid x_t\right)=N\left(x_{t-1} \mid \mu_\theta\left(x_t, t\right), \Sigma_\theta\left(x_t, t\right)\right)
\end{aligned}
$$
Therefore, tampering the initial noise $X_{T}$ clearly changes the start point of sampling, thus affecting the subsequent sampling process. 
Furthermore, when we discuss the input distribution of the diffusion model, we need to be aware that the input distribution consists of the training data distribution and the initial noise distribution in the DM scenario. BadDiff changes both of them, while the backdoor attack in classification only changes the training data distribution. Tampering training data distribution is reasonable. However, it may be debatable whether the attacker can tamper the sampling noise in the inference phase, since the initial noise is widely regarded as an intrinsic property of diffusion models (as supported by score function). 
Lastly, it is important to note that BadDiff utilizes different objectives on clean and poisoned samples, which implicitly assumes that the attacker is able to alter the training method.

Reviewer comment (Reproducibility):
Experimental details are missing. What are the training details such as learning rate, and epochs? Which part of the stable diffusion models is trained? What's the accuracy of the ResNet-50 used to measure the misalignment? How are the images in G1-G4 generated (only using the target class)? How are existing backdoor detection methods such as STRIP applied to the diffusion models because they are designed for classifiers?

Author response:
**Q5**: Question on experiment details.

**A5**: Thank you for raising the question. Except for the data replication part, we adopt a constant learning rate of 1e-4. All models are trained for 50 epochs. We empirically observed that training more iterations does not enhance the backdoor effect, and may degrade the performance of clean generation. In the data replication part, to align with existing work [R1], we train 100k iterations with a constant LR of 5e-6 and 10k steps of warmup. In all experiments, only the U-Net part is finetuned, while the text encoder and latent space encoder/decoder components are frozen. The accuracy of the ResNet-50 used to measure the misalignment is 99.541% on ImageNette and 98.166% on Caltech15. To generate G1-G4, We only use the target class prompt, specifically, "a photo of a garbage truck" on ImageNette and "a photo of a binocular" on Caltech15. This has been introduced in “Attack details” of Sec. 4. 

>[R1] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Understanding and Mitigating Copying in Diffusion Models. arXiv preprint arXiv:2305.20086.

Reviewer comment (Evaluation):
The Y-axis of Figure 7 means the similarity between the training image B and its replicated counterpart C. Why are there a lot of points at the left bottom corner? Does it mean B is very different from its replicated image? But I think they should be very similar. If so, does it mean the similarity metric is problematic?

Author response:
**Q7**: Question on why there are a lot of points at the left bottom corner in Figure 7 

**A7**: We appreciate your interest in this section. In Figure 7, every point corresponds to a generated image. There are a lot of points located in the lower-left corner of Figure 7. However, please note that the starting coordinates of both the x-axis and y-axis are not 0 (we did not show all the data pairs for ease of visualization). In general, a lower-left corner point corresponds to a generated image whose similarity to its top-matched training data point ranges from 0.25 to 0.3. We further visualize more generation-training pairs corresponding to the points in the lower left corner in [**this link**](https://ibb.co/Jrd4SxZ). The visualization demonstrates that the point located in the lower-left corner does not mean the corresponding generated sample is very dissimilar with its top-matched training sample.
Second, prior work [R1] has also revealed that the significant replications only account for a small part of the generation data. They have pointed out: “Stable Diffusion images with dataset similarity >0.5 account for approximately 1.88% of our random generations” in their Limitations & Conclusion paragraph. Therefore, a lot of points concentrated in the lower left corner do not imply that the metric is  problematic as this is our intention to show only a subset of data pairs with similarity score greater than 0.2. In fact, this metric has been widely used in data replication research on diffusion models [R1,R2].

>[R1] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6048-6058).
>
>[R2] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Understanding and Mitigating Copying in Diffusion Models. arXiv preprint arXiv:2305.20086.

Reviewer comment (Experiments):
As for the first observation, the mismatching between the generated images and the text conditions may not come from backdoor trigger injection but from the incorrect labels of the poisoned images. The authors should try relabeling data only (without adding the backdoor trigger) and check if the ratios of generated mismatching images (G1+G3) are similar to those of BadNets-like backdoored DMs in Figure 3.

Author response:
We acknowledge the reviewer's concern that image-prompt mismatching may be attributed to incorrect labels instead of trigger patterns. In response, we conducted additional experiments in which the backdoor dataset was constructed by relabeling only. With the generation dissection and composition results presented in [**this link**](https://ibb.co/724wKj6). To ease the comparison with the BadNets attacks, we refer to the generations mismatching the input condition as G1. We observed that “relabeling only” can result in mismatching generations. However, compared to G1 in Fig. 3, the BadNets trigger is absent. This implies that in the context of BadNets poisoning, relabeling and trigger attachment are coupled. Moreover, in Fig. 3, the BadNets poisoning also introduces the G2 type adversarial generations, which can not be achieved by “relabeling only”. As highlighted in the backdoor attack literature for image classification [R1, R2], the basic backdoor attack can be elucidated through the lens of association (a.k.a “shortcut”) between trigger pattern and target label. In the context of image generation, we impose such a **backdoor association** between the target prompt and the image trigger pattern. The created “shortcut” can be revealed by the aforementioned G1 and G2: (a) In the generated images, the proportion of images that mismatch the target prompt and contain triggers (G1) exceeds the corresponding proportion in the training set, as illustrated in Figure 3. (b) The backdoored DM generates trigger-polluted images within the target class that is not poisoned during training (referred to as G2). 

Furthermore, our research is not restricted to the adversarial effect of BadNets-type poisoning, but also delves into what insights the backdoored DMs can provide for data replication of DMs and defense against backdoored dataset over DM-generated data. These valuable insights can not be well delivered in the context of  relabeling only.

[R1] Sun, Mingjie, Siddhant Agarwal, and J. Zico Kolter. "Poisoned classifiers are not only backdoored, they are fundamentally broken." arXiv preprint arXiv:2010.09080 (2020).

[R2] Wang, Ren, et al. "Practical detection of trojan neural networks: Data-limited and data-free cases." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16. Springer International Publishing, 2020.

Reviewer comment (Experiments):
The paper only inspects two simple dirty-label attacks, which are easily detected by standard backdoor defenses. It is recommended to examine if the bilateral effects still appear with more sophisticated dirty-label (e.g., WaNet, LIRA) and clean-label attacks (e.g., LabelConsistent, SIG) and whether the proposed backdoor defense is helpful in those cases.

Author response:
We appreciate the suggestion for exploring other sophisticated backdoor methods. 

First, we would like to clarify that conducting an in-depth analysis of just the basic backdoor poisoning against DMs has been highly nontrivial. This is our intention to focus on this simplest BadNets-type poisoning setting that has been overlooked in the existing DM literature. 

Second, we do value the reviewer’s suggestion and have extended our experiment on the defensive insights in Table 4 to the WaNet attack. We configured the grid size to 224 and set the wrapping strength to 1 to ensure the compatibility of the WaNet attack with ImageNette, which is consistent with previous research [R1].
All the additional results are presented in [**this link**](https://ibb.co/SsBb0jh). As we can see, the attack success rate (ASR) for an image classifier ResNet-50 trained on the WaNet-backdoored DM-generated dataset is significantly reduced compared to that trained on the original poisoned training set. This defensive insight is consistent with BadNets-type attacks. 


>[R1] Zhu, Zixuan, et al. "The Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Reviewer comment (Evaluation):
In Table 3, the proposed approach only helps to improve backdoor detection accuracy on the generated images; it cannot be used to filter backdoor examples in the original training data. Hence, it does not help to mitigate datasets with moderate-to-high poisoning rates, unlike many other backdoor defense approaches.

Author response:
Thanks for pointing this out. We acknowledge that the backdoored DM cannot be used to filter backdoor examples from the original training data. Instead, our objective is to utilize the backdoored DM as a data transformation operation to discern the presence of backdoor poison at the dataset level, as elaborated on in our manuscript on the last paragraph of Page 7. Moreover, it can be synergized with the latter defense insights which involve training classifiers on the generated dataset. That is, the defenders can firstly detect the presence of the backdoor in the original dataset. If detected, they can then train the classifier on the generated data for improved resilience against data poisoning

The rationale behind our detection insights is the observed “triggered amplification” phenomenon. We view the amplified trigger ratio as an opportunity to enhance the backdoor signature. The left shift in the distribution of detection metrics, presented in [**this link**](https://ibb.co/vzhpfd9), validates the backdoor signature enhancement in the generation phase.

Reviewer comment (Evaluation):
In Table 4, the classifiers trained on generated data, while being more robust, have significant accuracy drops (sometimes more than 10%). Hence, they are not usable in practice. Particularly, since it is hard to differentiate a clean dataset from a dataset with a low poisoning rate, applying this technique will hurt a lot if the suspected dataset is actually clean.

Author response:
Thank you for raising this question. We admit that  there is a trade-off between accuracy and robustness. However, compared to the decrease in accuracy, the classifier trained on the generated samples shows a more significant decrease in the attack success rate (ASR) than in the accuracy (ACC), as illustrated in Page 8, Table 4. We will add more discussion regarding this tradeoff in the revision.

Meanwhile, regarding the concern about the potential damage to accuracy caused by misapplying this defense to clean dataset. We argue that we can detect first and then decide whether to defend. As illustrated in [**this link**](https://ibb.co/vzhpfd9), the diffusion model can amplify the backdoor signature, making it easier to distinguish between clean data sets and low poisoning rate data sets.

Reviewer comment (Experiments):
The test dataset used for the diffusion classifiers (Table 5) is different from the datasets used in the previous experiments. The authors should explain the reason.

Author response:
Thank you for pointing this out. The main reason for changing the dataset is the considerable computational cost of the stable diffusion classifier, which needs to perform N times diffusion and denoising processes to classify a single sample, where N is typically larger than 100 to achieve high accuracy. Consequently, the time used to classify a sample is N times greater than the time of generating a sample. Even with the timestep sampling strategy and dynamic inference technique proposed in [R1], the stable diffusion classifier requires about 20 seconds to classify a single test sample on a RTX A6000, which limits our ability to conduct large-scale experiments on datasets such as ImageNette and Caltech15. In contrast, CIFAR10 images can be classified faster due to their smaller image size, making the experiments affordable. Please note that another work [R2] leveraging diffusion classifiers for adversarial defense, which also conducts experiments on the CIFAR10 dataset only.  

>[R1] Li, Alexander C., et al. "Your diffusion model is secretly a zero-shot classifier." arXiv preprint arXiv:2303.16203 (2023).
>
>[R2] Chen, Huanran, et al. "Robust Classification via a Single Diffusion Model." arXiv preprint arXiv:2305.15241 (2023).

Reviewer comment (Experiments):
What datasets were used in Table 4, Table 6, and Figures 6-7?

Author response:
In the backdoor defense application (corresponding to Table 4 and Figure 6), we employ both the ImageNette and Caltech15 datasets, as specified in Table 4. For the data replication experiments (corresponding to Table 6 and Figure 7), ImageNette is adopted, as indicated in the second paragraph of Page 9.

Reviewer comment (Presentation):
Page 5 should be "Table 2" instead of Figure 2.

Author response:
**Q1**: "Table 2" instead of Figure 2？

**A1**: Thanks for pointing this out, we will correct it.

Reviewer comment (Experiments):
All experiments are performed on small dataset. Do the provided insights still hold on larger datasets, such as ImegNet-100?

Author response:
**Q2**: Question on whether our insights still hold on larger datasets like ImageNet-100?

**A2**: Thanks for raising the question. Inspired by that, we have extended our experiments to a new dataset **Caltech15**, a subset comprising 15 categories from Caltech256. The obtained results on Caltech15 can be accessed through [**this link**](https://ibb.co/3fDZMry). Notably, the effect of backdoor attacks on the Caltech15 dataset is more pronounced than on ImageNette. For instance, when employing a poisoning ratio of 5% and a guidance weight of 5, the proportion of trigger-presented images (G1+G2) generated by the backdoored DM is 47.5% higher than the proportion of the poisoned training set. In contrast, on ImageNette, the increase in trigger proportion is 17.2%. Our experiments on both ImageNette and Caltech15 affirm that the BadNets-like dataset remains effective to backdoor DMs, causing incorrect (G1) or tainted (G2) image generation. 

Unfortunately, we were not able to extend our experiments on ImageNet-100 during the rebuttal phase due to our currently limited computing resources. Please note that it takes much more intensive computational costs for training backdoor-poisoned diffusion models than classifiers. We will continue conducting this experiment by requesting more computing resources after rebuttal. Meanwhile, it is worth noting that prior works [R1,R2] on backdoor diffusion models have also mainly conducted experiments on CIFAR10 and CelebA. Thus, we believe that the insights we gained should still hold on larger datasets. 

> [R1] Chou, Sheng-Yen, Pin-Yu Chen, and Tsung-Yi Ho. "How to backdoor diffusion models?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
> 
> [R2] Chen, Weixin, Dawn Song, and Bo Li. "Trojdiff: Trojan attacks on diffusion models with diverse targets." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Evaluation):
Can you explain why in Figure 2, poisoning 10% of training samples will have marginal effect on the FID of benign images? Does this mean that the performance of diffusion model is not sensitive to the number of training dataset?

Author response:
**Q3**: Question on why poisoning 10% of training samples will have marginal effect on the FID of benign images?

**A3**: Thank you for raising this question. Poisoning 10% of the training samples does have marginal effect on the quality of the benign samples, as assessed by the Fréchet Inception Distance (FID) score between the generated images of non-target classes and the training images of non-target classes (see the second paragraph at Page 5). We observe that generated images under non-target classes typically do not contain triggers, and can thus be well aligned with the non-target class data in the training set. This observation bears a resemblance to the phenomenon of a backdoored classifier achieving high classification accuracy on clean data.

As for whether the performance of the DM is insensitive to the dataset size, we believe that reducing a *moderate proportion* of training data will not significantly degrade the DM’s performance. This assertion is supported by previous research on backdoor DMs. For example, [R1] has shown that a backdoored DM sustains performance comparable to its normally trained counterpart, even when subjected to larger poisoning ratios ranging from 10% to 30%. In fact, the reviewer's question has inspired us to explore an important aspect: dataset pruning for diffusion models. This line of inquiry seeks to understand how the training set can be selectively pruned without hampering the performance of DMs.

> [R1] Chou, Sheng-Yen, Pin-Yu Chen, and Tsung-Yi Ho. "How to backdoor diffusion models?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
DDIM is another sampling algorithm for diffusion models. Would your proposed insight vary if we use DDIM to sample images from a backdoor diffusion models?

Author response:
**Q4**: (Question on extending to DDIM samplers) DDIM is another sampling algorithm for diffusion models. Would your proposed insight vary if we use DDIM to sample images from backdoor diffusion models?

**A4**: Our insights hold on DDIM. In fact, all of our results on stable diffusion were obtained using DDIM sampler with sampling steps = 50 and eta = 0. DDPM is only employed for generation on CIFAR-10 using a relatively small diffusion model, as we have pointed out in the first paragraph of Page 5.

Reviewer comment (Experiments):
Can your provided insights be transferable to SDE [1]?  

[1] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.

Author response:
**Q5**: Question on whether our insights hold on SDE sampler?

**A5**: Yes, the backdoor threat also exists on the SDE sampling [R1] and the obtained results are presented in [**this link**](https://ibb.co/jw571DQ). However, we observed that the backdoored DM generates less trigger-tainted images using SDE sampling. We attribute this observation to the increased randomness introduced by SDE sampling [R2], consequently hindering the replication of trigger patterns.

> [R1] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
> 
>[R2] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2022). Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095.