Paper: A Change of Heart: Backdoor Attacks on Security-Centric Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
The paper lacks a comprehensive review of backdoor methodologies. It primarily relies on the original BadNets, introduced six years prior by Gu et al. (2017). Contemporary and more innovative backdoor attack techniques warrant discussion. Additionally, the trigger mechanism inherent to BadNets is susceptible to detection, as evidenced by [1].

Author response:
Following the reviewer’s suggestions, we have added a discussion of recent literature on backdoor attacks and defenses in **Appendix D1**.

To clarify, DIFF2 does not rely on BadNets; rather, it diverges significantly from conventional backdoor attacks (e.g., BadNets). In terms of objectives, BadNets aims to force the classifier to misclassify triggered inputs, while DIFF2 aims to undermine the security assurance provided by the diffusion model for the target classifier. In terms of techniques, BadNets poisons the training data of the classifier to associate the trigger pattern with the adversary’s target class, while DIFF2 superimposes the diffusion model with a malicious diffusion-sampling process, guiding triggered inputs toward an adversarial distribution, while preserving the normal process for clean inputs. 



Further, while we opt to use the patch-based trigger as in BadNets and other backdoor attacks (e.g., Chou et al., 2023 and Chen et al., 2023), DIFF2 is not tied to specific triggers. In **Appendix C3**, we evaluate alternative trigger designs, which also lead to effective attacks. 

Additionally, it is important to note that activation clustering (AC) is a training-time defense that is applied to filter poisoning training data. Following Chou et al. (2023) and Chen et al. (2023),  we assume the threat model that the adversary acts as the model provider, to which AC is not directly applicable. For completeness, we evaluate ANP (Wu & Wang, 2021), a pruning-based defense in our setting (details in **Appendix C6**). It is observed that ANP is ineffective against DIFF2 across different learning rates, mainly because the backdoor diffusion function is inherently intertwined with the normal diffusion function in our context. We consider developing more effective defenses against DIFF2 as our ongoing work.

Reviewer comment (Experiments):
In the main experiments, the surrogate and target classifiers are depicted as identical. This assumption appears unrealistic, especially since adversarial examples are tailored to specific classifiers. When pretraining a backdoored diffusion model, it's implausible to assume knowledge of the exact classifier a user will deploy. The experimental setup in Section 4 contrasts with the experiments, indicating a lack of consistency. A deeper exploration of transferability is imperative; the experiments described in Section 5.4 appear insufficient.

Author response:
To clarify, DIFF2 does not rely on the surrogate and target classifiers being identical to ensure a successful attack. Indeed, as demonstrated in Table 4 of Section 5.4, DIFF2 exhibits a significant degree of transferability to target classifiers with varying architectures and weights, achieving ASR up to 77%. Following the reviewer’s suggestion, we conduct additional transferability evaluation to include a broader range of diffusion models (details in **Appendix C7**), further showing the effectiveness and practicability of DIFF2.


|  Untargeted Attack| ||  Target Model | | 
| :--: | :--: | :--: | :--: | :--: | 
| | | ResNet-50 | DenseNet-121 | VGG13 |  
|  DDPM|  ACC | 92.8% | 92.1% | 88.4% | 
|  | ASR | 77.9% | 76.8%  | 44.6% |  
| SDE | ACC | 68.5% | 65.5%| 72.4% | 
| | ASR | 81.3%| 85.7% | 53.5% |  
| ODE | ACC | 85.1% | 86.4%  | 81.2% | 
| | ASR | 71.3%| 68.8% | 31.3% |  

|  Targeted Attack| ||  Target Model | | 
| :--: | :--: | :--: | :--: | :--: | 
| | | ResNet-50 | DenseNet-121 | VGG13 |  
|  DDPM|  ACC | 93.4% | 93.1% | 87.0% | 
|  | ASR | 33.6% | 63.5% | 10.0% |  
| SDE | ACC | 73.8% | 75.6% | 81.2% | 
| | ASR | 52.1% | 64.7% | 8.3% |  
| ODE | ACC |  86.7%| 89.1% | 84.4% | 
| | ASR |   30.4%| 39.7% | 9.3% |

Reviewer comment (Evaluation):
I've observed a possible inconsistency in Table 4. The ASR reported therein is notably low (sub-40%), yet you've mentioned an ASR of 77.1% in the accompanying text. Could you please clarify this discrepancy?

Author response:
We thank the reviewer for pointing out this discrepancy. For untargeted attacks, the ASR is measured as the percentage of triggered inputs that are misclassified (i.e., one minus the accuracy). In our submission, we mistakenly reported the accuracy as the ASR for untargeted attacks. We have updated the numbers in the revision. Note that for targeted attacks, the reported ASR accurately reflects the percentage of triggered inputs classified into the target class. We appreciate the opportunity to rectify this error.

Reviewer comment (Evaluation):
DIFF2 could bring a significant negative impact on the benign acc. For example, the result in Table 2 shows that DIFF2 will decrease the clean accurcay from 86.4% to 70.5% (-15.9%). This will largely impact the usuage of the diffusion model.

Author response:
We thank the reviewer for the insightful comments. We look deeper into the case of SDE/ODE and find that the relatively larger clean/robust accurate drop is mainly caused by the setting of DIFF2's training hyper-parameters. In our initial implementation, we set a unified learning rate for both discrete (DDPM/DDIM) and continuous (SDE/ODE) diffusion models. Yet, it is found that continuous diffusion models tend to require a lower learning rate for DIFF2 to work more effectively. After properly adjusting the learning rate (5e-5), SDE/ODE now attain clean/robust accuracy comparable with other samplers, as reported in the table below.

| | DIFF2 | Clean ACC | PGD | AutoAttack | ASR |
| :--: | :--: | :--: | :--: | :--: | :--: | 
| SDE | w/o | 87.3% |  85.2% | 84.9% | 13.1% |
| | w | 78.6% | 75.8% | 75.1% | 92.4% |
| ODE | w/o | 90.1% | 87.4% | 86.5% | 11.4% |
|| w | 81.2% | 77.9% | 76.8%|  87.3% |

Reviewer comment (Evaluation):
The effect of DIFF2 to robust acc can not be ignored. In addition to Clean ACC, I also notice that the performance of DIFF2 on robust acc is even worse, e.g. the robust acc of SDE decreases from 85.6% to 60.3% (-25.3%). It demonstrates that DIFF2 will increase the vulnerability of the diffusion model.

Author response:
We thank the reviewer for the insightful comments. We look deeper into the case of SDE/ODE and find that the relatively larger clean/robust accurate drop is mainly caused by the setting of DIFF2's training hyper-parameters. In our initial implementation, we set a unified learning rate for both discrete (DDPM/DDIM) and continuous (SDE/ODE) diffusion models. Yet, it is found that continuous diffusion models tend to require a lower learning rate for DIFF2 to work more effectively. After properly adjusting the learning rate (5e-5), SDE/ODE now attain clean/robust accuracy comparable with other samplers, as reported in the table below.

| | DIFF2 | Clean ACC | PGD | AutoAttack | ASR |
| :--: | :--: | :--: | :--: | :--: | :--: | 
| SDE | w/o | 87.3% |  85.2% | 84.9% | 13.1% |
| | w | 78.6% | 75.8% | 75.1% | 92.4% |
| ODE | w/o | 90.1% | 87.4% | 86.5% | 11.4% |
|| w | 81.2% | 77.9% | 76.8%|  87.3% |

Reviewer comment (Experiments):
All experiments are performed on the small datasets. Further experiments are needed to illustrate DIFF2 can envade the classifier on the large dataset, e.g. ImageNet.

Author response:
Our evaluation mainly follows the setting in previous studies (e.g., Chen et al., 2023), where CelebA (64$\times$64), CIFAR-10, and CIFAR-100 are used as benchmark datasets to evaluate attacks against diffusion models. This practice allows us to make meaningful comparisons with prior work in this space. Our results across these datasets consistently demonstrate the efficacy of DIFF2.

However, we acknowledge the importance of testing on larger datasets such as ImageNet for a more comprehensive evaluation. Unfortunately, due to the limitations of our current compute resources (GPU memory constraints in particular), we are not able to directly validate DIFF2 on the full ImageNet dataset. Nonetheless, to partially address this concern, we conduct experiments on ImageNet-64 (a rescaled dataset of 64$\times$64 images across 1,000 classes), with details in **Appendix C5**. Notably, DIFF2 achieves 93.1% (untargeted) and 17.4% (targeted) ASR, while retaining 39.7% clean ACC. While these results are indicative, we agree that further investigation on larger datasets would be valuable and consider it as our ongoing work.

Reviewer comment (Experiments):
The stealthiness of the attack can be further improved. Figure 3 shows that after performing the purification, the trigger will remains on the generated image $\hat{x_0}$. Thus, it increases the probobility of detection by the backdoor detection method, such as [1] or human inspection. Thus, I would suggest use $x_a$ (the adversarial sample of $x$) to substitute $x_a^*$ (Line 6 of Algorithm 1).

Author response:
We thank the reviewer for the insightful comments. We would like to clarify that DIFF2 is formulated as a backdoor attack, in which the backdoored diffusion model, in tandem with a target classifier, forms an integrated target system. In this context, the purified image serves only as an intermediate representation and is often not subject to in-depth inspection. The setting is consistent with typical backdoor attacks, where triggered inputs drive the target system toward predefined malicious outcomes. 
 
Moreover, we have explored using $x_a$, instead of $x_a^\star$ as the optimization target, but found that this setting leads to much less effective attacks and often causes the optimization to collapse. This may be explained as follows. With $x_a$ as the target, given the stochastic nature of the sampling process and the high similarity between $x_a$ and $x_c$ (differing only by the adversarial noise), the diffusion model tends to produce a mixture of adversarial and clean inputs, which also disrupts the optimization with respect to clean inputs.

Reviewer comment (Experiments):
Can the existing backdoor defense, e.g. ANP [2], be used to defend DIFF2? Your can refer to [3] on how to implement ANP on the diffusion model.

Author response:
Following the reviewer’s suggestion and referencing the implementation in BadDiffusion (Chou et al., 2023), we evaluate ANP as a defense against DIFF2 (more details in **Appendix C6**). We have the following interesting findings. 

|  |  | | |  ANP Epochs | | | | 
| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |
| ANP Learning Rate |  | 1 | 2 | 3 | 4 | 5 |   
|  5e-5 |  ACC | 78.7% | 72.7% | 51.0% | 17.8% | 8.9% |
|  | ASR | 90.3% | 89.6% | 91.6% | 94.4% | 90.9% |
| 2e-4 | ACC | 10.7% | 8.5% | 10.7% | 9.6% | 9.1% |
| | ASR | 90.7% | 90.6% | 88.8% | 90.2% | 90.6% |


 
ANP is ineffective against DIFF2 across different ANP learning rates. For instance, with the ANP learning rate fixed as 5e-5, the model’s ACC drops quickly as the ANP training epochs increase, while the attack’s untargeted ASR remains above 90%. This may be explained as follows. Compared with BadDiffusion that targets specific images, DIFF2 targets adversarial distributions, which represents a complex backdoor function. Thus, it is likely that the backdoor diffusion function is inherently intertwined with the normal diffusion function. Recall that ANP removes the injected backdoor by pruning sensitive neurons. Due to the intertwined backdoor and normal functions, the pruning inevitably impacts the normal function. We believe these findings are valuable for future research of developing more robust defenses against advanced backdoor attacks like DIFF2.

Further, the performance of ANP is highly sensitive to the learning rate, which corroborates the findings by Chou et al. (2023). For instance, as the learning rate is set as 2e-4, the model’s ACC sharply drops to around 10%.

Reviewer comment (Experiments):
Is the noise on the trigger image after passing through the diffusion and denoising a little too big (3 and 4 row of figure 3) to be easy to see? Generally speaking, budget L_\infty=8/255 is enough to generate adversarial samples to Cifar10 on ResNet18. Why use 16/255?

Author response:
We thank the reviewer for the question. For clarification, DIFF2 is formulated as a backdoor attack, in which the backdoored diffusion model, in tandem with a target classifier, forms an integrated system. In this context, the purified input merely serves as an intermediate representation and is typically not subject to in-depth inspection. The setting of our work is consistent with typical backdoor attacks, where triggered inputs drive the system toward predefined malicious outcomes. 

Also note that for the diffusion backdoor attack to be effective, the distribution of triggered inputs and the distribution of their adversarial variants need to be sufficiently separable. Otherwise, due to its highly stochastic nature, the sampling process tends to produce a mixture of adversarial and non-adversarial inputs. Thus, while the perturbation budget is often set as $\ell_\infty=8/255$ in the context of adversarial attacks, we find $\ell_\infty=16/255$ leads to more effective diffusion backdoor attacks. The table below shows that DIFF2 still works under $\ell_\infty=8/255$ but less effectively (more details in **Appendix C3**).

|  | Untargeted | Attack | Targeted | Attack | 
| :--: | :--: | :--: | :--: | :--: |
| Perturbation Magnitude | ACC | ASR | ACC | ASR |
| 8/255 | 82.7% | 61.3% | 81.6% | 27.6% |

Reviewer comment (Evaluation):
By 2 row of figure 3, it is not difficult to find that the trigger is basically retained after adding noise. Is it the main reason why the backdoor can be produced?

Author response:
We thank the reviewer for the question. Yes, retaining the trigger pattern (partially) after the diffusion process is a critical design, which is essential for distinguishing between a clean input and a triggered input in the latent space. This distinction allows the activation of different sampling processes: one leads back to the original data distribution and the other leads to the adversarial distribution desired by the adversary.

Reviewer comment (Experiments):
About trigger design, I see that the author randomly selects a 5*5 trigger and adds it in the lower right corner of the image. May I ask whether the author has considered the design method of the trigger? For example, considering the invisibility of the trigger, is it possible to select a trigger with L_\infty norm 8/255 (in this case, the trigger can be added to the whole picture but not only a corner)? In other words, is there any advantage in selecting the trigger as described in this article?

Author response:
We thank the reviewer for the insightful comments. While in principle DIFF2 is agnostic to concrete trigger patterns, in our empirical evaluation, we find that compared with other trigger designs, the patch-based trigger tends to lead to more effective attacks: due to its robustness against random noise, it is (partially) retained after the diffusion process, which is critical to ensure the distinguishability of clean and triggered inputs in the latent space. Further, it allows us to easily study the impact of trigger strength (by adjusting the patch size) on the attack performance (**Section 5.4**).

Following the reviewer’s suggestion, we also explore other trigger designs including the blending-based (Chen et al., 2017) and the warping-based (Nguyen and Tran, 2021) triggers (more details in **Appendix C3**). While these triggers are also feasible and often less perceptible, they often result in lower attack effectiveness, compared with the patch-based triggers. Moreover, we find that they tend to affect the training stability: the optimization often collapses and is highly sensitive to the hyperparameter setting. Intuitively, these trigger patterns are more susceptible to being obscured by the diffusion process, leading to an entanglement of clean and triggered inputs in the latent space, suggesting that there may exist a natural trade-off between the stealthiness of trigger patterns and the attack effectiveness. 

|  | Untargeted | Attack | Targeted | Attack | 
| :--: | :--: | :--: | :--: | :--: |
| Trigger | ACC | ASR | ACC | ASR |
| Blending-based | 81.6% | 47.2% | 77.6% | 20.1% |
| Warping-based | 81.3% | 64.3% | 78.4% | 36.7% |

Reviewer comment (Novelty):
About the way to generate a backdoor, Algorithm 1 requires very high privileges during the training process, including having the victim network f, arbitrarily adding dirty training data, and changing the loss function (Mixing loss). There is no doubt it works, but it is too straightforward (Directly training the Diffusion models to correspond to the trigger picture and adversarial picture), lacks innovation, which limits its usefulness. Should we consider more practical ways of adding backdoors? At least, it should be stated that adding less poison data and making little modification to the training process will enable the backdoor attack.

Author response:
We are grateful for the reviewer’s comments. Consistent with previous studies in this space (Chou et al., 2023; Chen et al., 2023), our study operates under the assumption that the adversary, acting as the model provider, crafts and disseminates a backdoored diffusion model. This threat model is highly relevant in today’s contexts, where users frequently choose pre-trained, ready-to-use diffusion models due to their prohibitive training costs. Also we assume the adversary only has access to a surrogate classifier $f$, which is not necessarily identical to the victim network (details in **Section 5.4** and **Appendix C7**).  

Moreover, DIFF2 significantly diverges from existing backdoor attacks (more details in **Appendix D2**). Compared with conventional backdoor attacks,, it is more challenging to implement backdoor attacks in our context for multiple reasons. First, the adversary has no control on the diffusion or sampling process (both are highly stochastic in nature). For a comparison, prior work in this space (Chou et al., 2023; Chen et al., 2023) assumes the adversary is able to control the latents during the sampling process. Second, the adversary needs to ensure the backdoored diffusion model retains its functionality with respect to both clean inputs (i.e., clean accuracy) and adversarial inputs (i.e., robust accuracy). It is thus non-trivial to generate backdoored diffusion models that meet all these criteria.

Reviewer comment (Theory):
In theorem 1, authors show that KL(q_T,p_T) has a downward trend in relation to T. What I understand ‘diff(x,T)’ is that: adding noise on x with T steps. If so, my question is: Let x_c be the clean image. When T is big, will diff(x_c,T) close to p_T (I think so, that is why we can use diffusion against the adversarial sample)? If so, is it true that q_T≈p_T≈diff(x_c,T) when T is big? Do you have any instructions about the comparison between KL(q_T,p_T) and KL(q_T,diff(x_c,T))?

Author response:
We thank the reviewer for the insightful comments. When $T$ is excessively large, all $q_T$ $p_T$, $\mathrm{diff}(x_c,T)$ converge towards random Gaussian noises. However, in the context of adversarial purification and certification, $T$ is often reasonably small to preserve the original semantics (Nie et al., 2022; Yoon et al., 2021), which is crucial for subsequent classification tasks. As $x^\star$ and $x^\star_a$ are more similar to each other than to $x_c$, $\mathrm{KL}(q_T,p_T)$ is often smaller than $\mathrm{KL}(q_T,\mathrm{diff}(x_c,T))$. 

To validate the analysis above, we conduct the following measurement. Under the default setting (DDPM, CIFAR-10, $T = 75$), for each clean input $x_c$, we generate its triggered variant $x^\star$, and the adversarial variant of $x^\star$ as $x^\star_a$, we then measure the $\ell_2$ distance between $q_T$, $p_T$, and $\mathrm{diff}(x_c, T)$. The results are averaged over 100 randomly sampled inputs. We have:
$\| q_T - p_T \|_2 = 98.9 <  \| q_T - \mathrm{diff}(x_c, T)\|_2 = 104.8  < \| p_T - \mathrm{diff}(x_c, T) \|_2 = 108.3$, which validates our analysis.

Reviewer comment (Experiments):
I'm not concern about the technical part but the practical scenario. I'm uncertain about the practicality of using diffusion models to generate adversarial input.

Author response:
Due to their unprecedented denoising capabilities, diffusion models have found increasing use as a means to enhance the security of target models (Nie et al., 2022; Yoon et al., 2021; Carlini et al., 2023; Xiao et al., 2023). However, one vulnerability that has been overlooked thus far is the trustworthiness of diffusion models themselves. Due to the expensive training cost, it is a common practice to employ pre-trained, ready-to-use diffusion models (Chou et al., 2023; Chen et al., 2023). This work explores the feasibility of injecting backdoors into such pre-trained diffusion models and subsequently exploiting the backdoors to diminish the security provided by diffusion models for target models. We believe this represents an important attack vector that has yet to be explored.

Reviewer comment (Evaluation):
Users may notice unusual patterns in the purified images, as illustrated in Figure 3.

Author response:
We thank the reviewer for the insightful question. We would like to clarify that DIFF2 is formulated as a backdoor attack, in which the backdoored diffusion model, in tandem with a target classifier, forms an integrated target system. In this context, the purified input merely serves as an intermediate representation and is typically not subject to in-depth inspection. This setting is consistent with typical backdoor attacks, where triggered inputs drive the target system toward predefined malicious outcomes. 

Also note that DIFF2 represents a general attack framework that is not tied to specific trigger patterns. We have conducted additional experiments using less perceivable triggers (e.g., blending-based and warping-based triggers), which also achieve effective attacks without unusual patterns in the purified inputs (details in **Appendix C3**). However, we note that compared with patch-based triggers, the alternative triggers often lead to less effective attacks and less stable training, suggesting that there may exist a natural tradeoff between the stealthiness of trigger patterns and the attack effectiveness.

Reviewer comment (Experiments):
If using the DDIM or other samplers, is the generated adversarial images consistently influence the classifier?

Author response:
We thank the reviewer for the comments. DIFF2 is designed to be sampler-agnostic. We conduct additional experiments (details in **Section 5.2**) to evaluate the effectiveness of DIFF2 against the DDIM model, with the other parameters set following Table 7. It is observed that DIFF2 achieves untargeted and targeted ASR over 80.2% and 44.3%, respectively, while retaining the clean ACC at 82.2%.