Paper: Demystifying Poisoning Backdoor Attacks from a Statistical Perspective (ICLR-2024)

Reviewer comment (Novelty):
The observation that a high poisoning ratio adversely affects the performance of clean data lacks novelty.

Author response:
Our results are non-trivial in several aspects. **First,** we offer a **quantitative** analysis, beyond mere empirical observations, about the effects of backdoor attacks. For instance, instead of the empirical observation that a high poisoning ratio can damage the model performance, we establish both lower and upper bounds for the prediction risk, as presented in Theorem 1 and 2. Those results serve as foundations for more nuanced analysis. For example, we find that a vanishing backdoor data ratio may be sufficient for successful backdoor attacks, and can even derive the minimum required amount of backdoor data in specific scenarios. **Second,** our work is pioneering in theoretically studying the performance of backdoor attacks. The closest work, to our knowledge, is Manoj & Blum (2021), which focuses on the model’s vulnerability, while ours is the first to quantify the effectiveness of these attacks. **Third,** the analytical techniques we used are also non-trivial. The main proof steps involve establishing the connection between the poisoned model and clean model, estimating the model deviation caused by backdoor data, and carefully optimizing the inequality so that the error bounds are tight in the sense of order.

Reviewer comment (Reproducibility):
The paper lacks clarity in some sections. For instance, Section 6.2.1 discusses the impact of backdoor trigger magnitudes, but fails to specify crucial details of the attack setting, such as the size of the trigger.

Author response:
We present detailed specifications for the setup in Table 2 as follows. In the case of MNIST, the backdoor triggers are 2 by 2 square patches, while for CIFAR-10, 3 by 3 square patches are utilized. All backdoor triggers are positioned at the lower-right corner of the inputs, replacing the original pixels with identical values. The pixel value represents the magnitude of the backdoor trigger. The poisoning ratio is consistently set at 5% for both datasets, and ResNet models are utilized in the experiments.

Reviewer comment (Evaluation):
The authors assert that WaNet, Adaptive Patch, and Adaptive Blend attacks are more effective than BadNets, as evidenced by a greater relative change in dimensions with low variance. However, the term "effectiveness" needs clarification. BadNets is known for its high attack success rate, so how do these methods compare under identical attack settings, including trigger size and magnitude?

Author response:
When discussing 'effectiveness,' we are referring to the consideration that backdoor attacks, exhibiting comparable clean and backdoor accuracies yet featuring less visibly discernible triggers, are deemed more effective. This is due to the fact that these subtle triggers can easily elude human eye detection, thereby enhancing their overall stealthiness. For instance, consider the WaNet/BadNets method, achieving a clean accuracy of 94.15%/94.3% and a backdoor accuracy of 99.15%/99.9% on the CIFAR-10 dataset, respectively. Despite the similar clean and backdoor accuracies of the two methods, the WaNet method is considered more effective due to its less visible trigger, as illustrated in the Figure on the official GitHub page of WaNet (https://github.com/VinAIResearch/Warping-based_Backdoor_Attack-release).

Reviewer comment (Evaluation):
The methodology for measuring the Mean Squared Error (MSE) between clean training images and those altered by the backdoored DDPM is unclear. Given that DDPM generation is inherently a stochastic process, a more detailed explanation of this measurement technique would be beneficial.

Author response:
The MSE is computed by comparing the averaged pixel values of the generated images with those of the target images.

      The goal of backdoor attacks on diffusion models is to create images with a predefined pattern triggered by specific backdoor text input. To evaluate the effectiveness of backdoor attacks, the assessment criteria should measure the similarity or distance between the generated backdoored images and the target images. Consistent with the approach outlined in [A], we employ MSE to quantify the distances between the generated images and the target images, as an approach of evaluating the effectiveness of backdoor attacks.
      
      Refs: [A] Chou et al, "How to backdoor diffusion models?", in CVPR 2023.

Reviewer comment (Theory):
Is it possible to extend this theoretical framework for multi-discriminator with more than 2 classes.

Author response:
Thanks for your constructive comment. The short answer is yes. For example, one possible extension in multi-class scenarios is assuming that poisoning data have a common target label. The crux of our approach involves estimating the difference between the regression function on poisoned data, $f_*^{poi}$ , and that on clean data,  $f_*^{cl}$ (referenced in Eq. (1), (6), and (7) in the proof of Theorem 1). Since $f_*^{poi}$ stands for the regression function with respect to the distribution of poisoned data, which is known from the attacker's perspective, we can determine $f_*^{poi}$ in the multi-class cases as well. Although both $f_*^{poi}$ and $f_*^{cl}$ are multi-valued functions in the multi-class cases, we can bound their difference by controlling the difference at each coordinate respectively. Then, the methodology for estimating its difference from $f_*^{cl}$ and error bounds would follow the same logical framework as outlined in our paper.

Reviewer comment (Experiments):
Some claims are not well validated empirically. The paper states "a large backdoor data ratio ρ will damage the performance on clean data." But there is no empirical evidence to support this claim. Also, according to the literature, a high poisoning rate usually does not significantly affect clean accuracy. It is recommended to empirically validate this claim and assess its consistency with the theories.

Author response:
Thanks for pointing it out. We have corrected our claim as "a large backdoor data ratio $\rho$ **could** damage the performance on clean data in some scenarios".  From our finite-sample results, a larger $\rho$  leads to the increase in both lower and upper bounds of prediction risk, thus the performance could be degraded. We have also performed additional experiments showing the influence of backdoor data ratio. Specifically, we perform BadNets attack on MNIST, replacing a 2 by 2 area at the lower-right corner with pixel value 5. The results are summarized in Table 1. As the poisoning ratio $\rho$ increases, the clean data accuracy is pretty stable at the beginning, and then quickly drops when $\rho$ approaches one. 

   Table 1. Effects of backdoor ratio on MNIST.
   | Ratio →     | 0.1% | 1%   | 5%   | 10%  | 50%    | 90%  | 95%  | 99%  |
   | ------------| ---- | ---- | ---- | ---- | ----  | ---- | ---- | ---- |
   | Clean Acc   | 0.99 | 0.98 | 0.98 | 0.98 | 0.98  | 0.94 | 0.91 | 0.42 |
   | Backdoor Acc | 0.41| 0.92 | 0.98 | 0.99 | 1.0  | 1.0  | 1.0  | 1.0  |

Reviewer comment (Experiments):
The experiment conducted in Table 2 is not clear. What does the magnitude of backdoor triggers mean? Is it the L2 norm of η, or a fixed pixel value that replaces the original pixel on the input? How large is the backdoor trigger used in this study? In addition, the formalization of backdoor trigger as η in X' = X + η is not accurate. Backdoor attacks, such as BadNets replace the original pixel values with the backdoor trigger. Otherwise, the trigger pattern is not fixed and varies on different inputs.

Author response:
We present detailed specifications for the setup in Table 2 as follows. In the case of MNIST, the backdoor triggers are 2 by 2 square patches, while for CIFAR-10, 3 by 3 square patches are utilized. All backdoor triggers are positioned at the lower-right corner of the inputs, replacing the original pixels with identical values. The pixel value represents the magnitude of the backdoor trigger. The poisoning ratio is consistently set at 5% for both datasets, and ResNet models are utilized in the experiments. 
   
   This paper theoretically studies backdoor attacks with the formulation of $X^{\prime} = X + \eta$. While BadNets take the form of replacement, our formulation is consistent with the experimental setup for MNIST, where almost all images have pixel values of 0 at the lower-right corner. We will include this clarification in the paper.

Reviewer comment (Theory):
The paper seems to focus on dirty-label backdoor attacks, where the poisoned samples are assigned a target label. There is another line of attacks that do not change the label, such as SIG [1] and reflection attack [2]. Is the proposed theoretical analysis applicable to these clean-label attacks?

[1] Barni, M., Kallas, K., & Tondi, B. (2019, September). A new backdoor attack in cnns by training set corruption without label poisoning. ICIP 2019.\
[2] Liu, Y., Ma, X., Bailey, J., and Lu, F. Reflection backdoor: A natural backdoor attack on deep neural networks. ECCV 2020.

Author response:
Thanks very much for your insightful comment. The short answer is yes, our analytical techniques are indeed applicable to clean-label attack scenarios. Although our paper primarily focuses on dirty-label backdoor attacks, where the labels of poisoned samples are altered, the fundamental principles of our analysis can be seamlessly adapted to clean-label contexts.

   Specifically, the crux of our approach involves estimating the difference between the regression function on poisoned data, $f_*^{poi}$ , and that on clean data,  $f_*^{cl}$ (referenced in Eq. (1), (6), and (7) in the proof of Theorem 1). Since $f_*^{poi}$ stands for the regression function with respect to the distribution of poisoned data, which is known from the attacker's perspective, we can determine $f_*^{poi}$ in the clean-label cases as well. Then, the methodology for estimating its difference from $f_*^{cl}$ and error bounds in the clean-label case would follow the same logical framework as outlined in our paper.

Reviewer comment (Theory):
None of the obtained insights is interesting in the sense that they are either trivial or not true without assuming the model to be Bayesian optimal with respect to the poisoned training distribution.

Author response:
We sincerely thank the reviewer's detailed explanation of their concerns. We believe the reviewer has some misunderstanding regarding our work. To summarize our responses: 1. The use of a Bayes-optimal learning algorithm is **NOT required** for the proof of our theorems. 2. We mention the case of vanishing statistical risk (common for many widely-used learning algorithms) as an illustrative example to demonstrate the potential for further analysis based on our theorems.

**First**, as outlined in our paper's Section 2, $f_*^{cl}$ and $f_*^{poi}$ are the **underlying regression functions** under clean and poisoned data distributions respectively, and $\hat{f}$ represents the estimator of them. **It's important to note that the concepts like Bayes error and Bayes classifier are specific to classification rules.** For example, $g=1_{\{f_*^{cl}\geq 0.5\}}$ is a Bayes classifier, where $1_{\{\cdot\}}$ is an indicator function, while $f_*^{cl}$ itself is not a classifier. For further clarity on the definition of Bayes error and Bayes optimal rule, please see, e.g., [1, 2].

[1] *Devroye L, Györfi L, Lugosi G. A probabilistic theory of pattern recognition[M]. Springer Science & Business Media, 2013. Chapter 2.* 

[2]  *Lin Y. A note on margin-based loss functions in classification[J]. Statistics & probability letters, 2004, 68(1): 73-82.*

**Second**, our proofs and statements of theorems are **independent of the estimator's optimality**. In Theorem 1, which the reviewer highlighted, we explicitly state that the upper bounds include two terms. The first term involves the ordinary statistical risk $r_n^{poi}(\hat{f}^{poi})$, and the second term is associated with the backdoor trigger $\eta$. To this point, we do not pose any constraints on the optimality of estimator $\hat{f}$. 

**Third**, the reviewer commented that "the first term is ignored when deriving insights". We consider some concrete scenarios **when explaining the potential implications of Theorem 1.** One such scenario is where the statistical risk approaches zero.  As mentioned in our first point, since $\hat{f}$ is a regression function rather than a classifier, $\hat{f}$  has a vanishing statistical risk is a different thing from it is Bayes optimal. Again, the proofs of theorems do not require $\hat{f}$ to be Bayes optimal.

**Fourth**, vanishing statistical risk, as known as consistency, **is a common property for a various learning algorithms**, such as k-nearest neighbors, kernel estimates, and neural networks (see, e.g., [3]).

[3] Györfi L, Kohler M, Krzyzak A, et al. A distribution-free theory of nonparametric regression[M]. New York: Springer, 2002.

We hope that these clarifications adequately address your queries and look forward to further discussions.

Reviewer comment (Novelty):
Insight 1 and 2 listed in the above Summary section are trivial (even though they may generalize to other backdoor/poison attacks).

Author response:
"Key insights are either trivial (insight 1&2) or likely not generalizable (insight 3&4&5)."

     **Response:** Our insights are non-trivial in several aspects. **First,** we offer a **quantitative** analysis, beyond mere empirical observations, about the effects of backdoor attacks. For instance, instead of empirical observations such that a trigger larger in magnitude has a higher attack success rate, Theorem 3 specifies the minimum trigger length (i.e., $\ln(n)$ ) necessary for a successful attack in Gaussian distribution scenarios. **Second,** our work is pioneering in theoretically studying the performance of backdoor attacks. The closest work, to our knowledge, is Manoj & Blum (2021), which focuses on the model’s vulnerability, while ours is the first to quantify the effectiveness of these attacks. **Third,** the analytical techniques we used are also non-trivial. The main proof steps involve establishing the connection between the poisoned model and clean model, estimating the model deviation caused by backdoor data, and carefully optimizing the inequality so that the error bounds are tight in the sense of order.

     The generalizability of our work are demonstrated in two aspects. **First,** we perform finite-sample analysis and provide tight error bounds with minimal assumptions, only requiring the loss function is Holder continuous and the input distribution is mostly concentrated in a bounded area. Those results can be the foundation for further analysis under various scenarios, including the asymptotic scenario discussed in our work.  **Second,** for practical applications where our technical assumptions may not be directly verifiable, we conducted experiments using both MNIST and CIFAR10 datasets and SOTA backdoor attacks (BadNets, WaNet, Adaptive Patch, and Adaptive Blend), and the results corroborate our theoretical findings.

We hope these clarifications address the concerns raised and further illuminate the significance and applicability of our work.

Reviewer comment (Theory):
Insight 3, 4, and 5 are trivial only when assuming the model to be Bayesian optimal but may not generalize to other (actual) learning algorithms.

Author response:
We appreciate the reviewer's time in reviewing our responses. We would like to clarify the misunderstanding in reviewer's comment saying that our results "only apply to $f_*^{poi}$ but not arbitrary models."

**First**, as acknowledged by the reviewer, our theorems are independent of the learning algorithms and thus the estimator (i.e., poisoned model) $\hat{f}^{poi}$. We contextualized $\hat{f}^{poi}$ in some concrete scenarios **to elucidate the potential implications of Theorem 1.** 

**Second**, the vanishing statistical risk, or consistency, meaning that $r_n^{poi}(\hat{f}^{poi}) = E\\{\ell(\hat{f}^{poi},f_*^{poi})\\}$ will converge to zero as the sample size $n$ goes to infinity. Here, $f_*^{poi}$ denotes the underlying regression function with respect to the poisoned data distribution, **while $\hat{f}^{poi}$ is the estimator we are analyzing**. In other words, the concept of consistency is specific to $\hat{f}^{poi}$, and our insights are pertinent to any consistent estimator $\hat{f}^{poi}$. As mentioned in the fourth point in the previous responses, **this consistency property is common for $\hat{f}^{poi}$ obtained from various learning algorithms**, such as k-nearest neighbors, kernel estimates, and neural networks (see, e.g., [3]). Notably, most existing literature on backdoor attacks presupposes the use of neural networks as the learning algorithm, as seen in, e.g., [4,5]. As a result, our insights implied by Theorem 1 are applicable to most scenarios encountered in practice.

[3] Györfi L, Kohler M, Krzyzak A, et al. A distribution-free theory of nonparametric regression[M]. New York: Springer, 2002.

[4] Li Y, Jiang Y, Li Z, et al. Backdoor learning: A survey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022.

[5] Gao Y, Doan B G, Zhang Z, et al. Backdoor attacks and countermeasures on deep learning: A comprehensive review[J]. arXiv preprint arXiv:2007.10760, 2020.

We hope that these clarifications adequately address your concerns and look forward to continued dialogue.

Reviewer comment (Theory):
Some key assumptions oversimplify the problems and make the analysis probably irrelevant to practice, e.g., models are Bayesian optimal with respect to the poisoned distribution and Assumption 3 (Ordinary convergence rate) in the submission.

Author response:
"Some key assumptions that oversimplify the problems and make the analysis probably irrelevant to practice, e.g. models are Bayesian optimal with respect to the poisoned distribution & Assumption 3 (Ordinary convergence rate) in the submission."

   **Response:** **First,** contrary to the reviewer's understanding, our work does not assume that models are Bayesian optimal. We focus on analyzing the prediction error of the poisoned model $\hat{f}^{poi}$, which is evaluated as the difference between $\hat{f}^{poi}$ and the unknown ground truth $f_*$. We kindly request the reviewer to reconsider our setup and theoretical results in this light. **Second,** our finite-sample analysis establishes error bounds on backdoor model performance without relying on Assumption 3. Our results quantitatively measure the impact of critical factors in backdoor attacks, such as data poisoning ratio and trigger direction. **Third,** under Assumption 3, we derive the rate of the poisoned model's risk in the asymptotic regime. This serves as a practical example of how our results can be applied for further analysis. Remark 4 clarifies that Assumption 3 is generally valid for commonly used function classes, such as the Holder class. **Fourth,** in practical applications where Assumption 3 cannot be verified or does not hold, we have conducted experiments to corroborate the developed theory and insights. Specifically, we performed state-of-the-art attacks (BadNets, WaNet, Adaptive Patch, and Adaptive Blend) on both MNIST and CIFAR10 datasets in Section 6.2.1, and the results align with our insights such as the attack is more successful when the trigger causes poisoned data and clean data are more separated in the sense of probability.

Reviewer comment (Novelty):
Key insights are either trivial (insight 1 and 2) or likely not generalizable (insight 3, 4, and 5).

Author response:
"Key insights are either trivial (insight 1&2) or likely not generalizable (insight 3&4&5)."

     **Response:** Our insights are non-trivial in several aspects. **First,** we offer a **quantitative** analysis, beyond mere empirical observations, about the effects of backdoor attacks. For instance, instead of empirical observations such that a trigger larger in magnitude has a higher attack success rate, Theorem 3 specifies the minimum trigger length (i.e., $\ln(n)$ ) necessary for a successful attack in Gaussian distribution scenarios. **Second,** our work is pioneering in theoretically studying the performance of backdoor attacks. The closest work, to our knowledge, is Manoj & Blum (2021), which focuses on the model’s vulnerability, while ours is the first to quantify the effectiveness of these attacks. **Third,** the analytical techniques we used are also non-trivial. The main proof steps involve establishing the connection between the poisoned model and clean model, estimating the model deviation caused by backdoor data, and carefully optimizing the inequality so that the error bounds are tight in the sense of order.

     The generalizability of our work are demonstrated in two aspects. **First,** we perform finite-sample analysis and provide tight error bounds with minimal assumptions, only requiring the loss function is Holder continuous and the input distribution is mostly concentrated in a bounded area. Those results can be the foundation for further analysis under various scenarios, including the asymptotic scenario discussed in our work.  **Second,** for practical applications where our technical assumptions may not be directly verifiable, we conducted experiments using both MNIST and CIFAR10 datasets and SOTA backdoor attacks (BadNets, WaNet, Adaptive Patch, and Adaptive Blend), and the results corroborate our theoretical findings.

We hope these clarifications address the concerns raised and further illuminate the significance and applicability of our work.