Paper: Dissecting Gradient Masking and Denoising in Diffusion Models for Adversarial Purification (ICLR-2024)

Reviewer comment (Evaluation):
I am not fully convinced by the experiments that the diffusion-based adversarial purification causes the gradient masking. Here, the problem is whether there is measure of the gradient masking phenomenon.

Author response:
Before answering the question, we want to first clarify that our theorem can be easily generalized to the case with simply connected adversarial space. This condition is practically appropriate, as Tramèr et al. 2017 showed that the adversarial space spans a continuous region with orthogonal bases. Both Reviewers 9RSr and AHc3 seem to have a correct understanding on this point. Assume we encounter randomness $\eta_0$ while calculating the attack. Then the attack is only optimal for the input with randomness $\eta_0$. During testing, however, we can encounter a different randomness $\eta_1$, then clearly the calculated gradient to generate the attack is not optimal (or useful) anymore under $\eta_1$. This is the main observation of how such a problem happens explained by our theorem.
Besides theoretical arguments, we further add experimental results in the revised version to strengthen the claim. As shown in Fig. 2, noise has a major influence on the purified states in diffusion models. Starting from the same state, with different noise, the purified states will reach slight positively correlated directions (average correlation 0.3). However, even with different initial perturbations (average correlation 0) within the small ball, with the same noise, the purified states will be almost the same (average correlation > 0.99). Therefore, if not properly controlling the randomness, we are essentially applying the gradients with wrong states.

Reviewer comment (Experiments):
The datasets are limited. Can the authors provide further experiments to verify their theoretical findings, i.e. performing experiments on other datasets, like Imagenet and MNIST?

Author response:
We added experiments on ImageNet in the modified version (Sec. 5.1 and 5.2). We used an unconditional guided diffusion model, with the ResNet-50 classifier. The main argument, randomness-induced gradient masking, was also identified in DiffPure on ImageNet (Table 1). Without randomness replay, our implementation of DiffPure achieves 69.0% clean accuracy, which is similar to the original reported 67.79% clean accuracy. After applying randomness replay, the clean accuracy stays to be a similar 68.5%, while the robust accuracy against BPDA drops to 34.0%, which is 33% lower than the non-randomness replay version. Note that after applying randomness replay, BPDA results provide a even better estimation than the full gradient-based AutoAttack, which is a clear sign of gradient masking in the original paper.
The comparison of DiffPure and RevPure is a bit different on ImageNet. As shown in Fig 2c, unlike on CIFAR-10, the clean accuracy will drop by applying the reverse-only diffusion models. Therefore the best robust accuracy will be reached at a shorter timestep constraint by the drop of clean accuracy. However, the robust performances of DiffPure and RevPure are still similar. Therefore, our conclusion that RevPure can lead to a similar robustness improment also holds on ImageNet. A more comprehensive evaluation on ImageNet will be included in the final version.

Reviewer comment (Presentation):
The structure of the paper is somewhat confusing. Diffusion models and gradient masking appear and are mentioned in the Abstract and Introduction, but most of the theoretical analysis are irrelevant to them.

Author response:
The theoretical analysis in Sec. 4 is related and serves at the core part of our claim of gradient masking, without which we can not make the predictions at the end of Sec. 4 and further verify such predictions in Sec. 5. Both reviewer 9RSr and AHc3 seem to have a correct understanding on this point. Please hold a discussion with those reviewers if possible. More details in the reply to Question #1.

Reviewer comment (Novelty):
Overall, this work is based on the DiffPure framework and presents its one limitation. Moreover, attack on a system with randomness has been studied. Therefore, I think the novelty is limited.

Author response:
The reviewer questions the novelty and significance of the work. We want to justify that our work in fact has substantial novelty and significance. Proposing new models or setting SOTA performance will be easily identified as novel, but we believe that analysis for correct understanding is also crucial. Given the popularity of EOT (3000+ citations) and DiffPure (130+ citations in a year), there exists an important issue in the evaluation of adversarial robustness involving randomness. Our paper provides a step toward addressing this important issue, both theoretically and empirically. There have been some interesting yet well-debated claims of robustness from randomness in both machine learning (Yoon et al. 2021, Byun et al. 2020) and neuroscience (Dapello, Joel, et al. 2020). There is an urgent need to shed light and clarify how randomness can (or can not) improve robustness.

Reviewer comment (Theory):
It is mentioned "Gradient masking has been defined as “construct a model that does not have useful gradients”. Could you please explain how do you illustrate that a purification system with randomness can cause "a model does not have useful gradients"?

Author response:
We first clarify that our theorem can be easily generalized to simply connected adversarial space. This condition is practically appropriate, as Tramèr et al. 2017 showed that the adversarial space spans a continuous region with orthogonal bases.  
Assume we encounter randomness $\eta_0$ while calculating the attack. Then the attack is only optimal for the input with randomness $\eta_0$. During testing, however, we can encounter different randomness $\eta_1$, then clearly the calculated gradient to generate the attack is not optimal (or useful) anymore under $\eta_1$. Further experimental evidence is added in the modified version (Fig. 1).

Reviewer comment (Novelty):
For example, Carlini et al. 2019 proposes "Verify that attacks succeed if randomness is fixed.". What is the difference between randomness replay and this method?

Author response:
Yes, there are no real differences between randomness replay and “fix randomness”. However, our analysis makes the argument more formal rather than just heuristicly proposing a method. Note that Carlini et al. were also in the author list of the EOT paper, which was widely used in adversarial purification and believed to solve the problem of randomness. We showed that EOT will not be sufficient, and randomness replay/fix randomness has to be the correct solution. Our results also explain why randomness matters—it is an inherent property of high-dimensional data due to the shape of the c.d.f of $\chi^2(n)$. When $n$ is large, both EOT and non-EOT are guaranteed to fail without randomness replay. In all, we believe our theoretical analyses provide a much deeper understanding of evaluating robustness involving randomness than just proposing a heuristic solution without knowing the causes and conditions.

Reviewer comment (Theory):
Could you please provide more explanation of the forward and reverse process in Diffusion Models? Why DiffPure proposed to utilize both the two processes while only using the reverse process is also workable?

Author response:
This is another good question not directly answered by the DiffPure and DensePure papers. On CIFAR-10, the forward process is not necessary (Table 2 and Fig. 3). Especially through Fig. 3c, we can see that the forward process is not helpful for removing adversarial directions (the first flat stage)—adversarial directions are quickly removed only in the second reverse processing stage. For ImageNet, the forward process can be useful. Unlike on CIFAR-10, the clean accuracy drops by applying the RevPurfe, thus it is essential for the system to first inject a proper amount of noise for the reverse process to behave normally. However, the robustness performance of DiffPure and RevPure are still similar on ImageNet.

Reviewer comment (Evaluation):
Could you please explain how the observation that "The reverse process indeed gradually removes the projection on the original adversarial direction..." further supports that "forward process may not be useful for robustness"? I think it will be better if you can also illustrate the forward process in Figure 4a.

Author response:
Again, this question can be answered by Fig. 3c—the forward process is not helpful for removing adversarial directions (the first flat stage)—adversarial directions are quickly removed only in the second reverse processing stage.

Reviewer comment (Experiments):
The closest works (DiffPure, DensePure) validated their experiments not only on CIFAR-10, but on large-scale datasets such as ImageNet. To make the method closer to real world, ImageNet experiments are necessary.

Author response:
We added experiments on ImageNet in the modified version (Sec. 5.1 and 5.2). We used an unconditional guided diffusion model, with the ResNet-50 classifier. The main argument, randomness-induced gradient masking, was also identified in DiffPure on ImageNet (Table 1). Without randomness replay, our implementation of DiffPure achieves 69.0% clean accuracy, which is similar to the original reported 67.79% clean accuracy. After applying randomness replay, the clean accuracy stays a similar 68.5%, while the robust accuracy against BPDA drops to 34.0%, which is 33% lower than the non-randomness replay version. Note that after applying randomness replay, BPDA results provide an even better estimation than the full gradient-based AutoAttack, which is a clear sign of gradient masking in the original paper.
The comparison of DiffPure and RevPure is a bit different on ImageNet. As shown in Fig 2c, unlike CIFAR-10, the clean accuracy will drop by applying the reverse-only diffusion models. Therefore the best robust accuracy will be reached at a shorter timestep constraint by the drop of clean accuracy. However, the robust performances of DiffPure and RevPure are still similar. Therefore, our conclusion that RevPure can lead to a similar robustness improvement also holds on ImageNet. A more comprehensive evaluation on ImageNet will be included in the final version.

Reviewer comment (Experiments):
Only one architecture (WideResNet-28) is studied, while other works studied diverse set of architectures such as ViTs, ResNets, etc.

Author response:
In the modified version, ResNet-50 was added for ImageNet. Other classifier architectures WideResNet-50-2, DeiT-S could also be included in the final version (it’s just one line of code based on the RobustBenchmark). However, we believe diffusion models are essential to study in adversarial purification with diffusion models. We studied DDPM, unconditional Guided diffusion, as well as DDIM, which was not covered by DiffPure. We intentionally avoid any continuous-time diffusion models due to the potential extra gradient masking caused by the numerical solver (Huang et al. 2022).

Reviewer comment (Experiments):
Effect of generation quality of the model (FID) on the robustness is only studied using two models (EMA DDPM and Non-EMA DDPM). You can't conclude anything from two points. More diverse set of models (maybe checkpoints from different epochs) should be studied.

Author response:
It is a good suggestion to check multiple checkpoints with different FID scores, thus eventually changing Table 3 into a Figure. However, we want to serve our argument in Sec. 5.4 only as a counterexample—for anyone attempting to claim “diffusion models with a better image generation ability can achieve a better adversarial purification” in the future, they have to first explain our experiment in Table 3.

Reviewer comment (Writing):
$\chi^2$ should be properly introduced.

Author response:
$\chi^2$: we add a “chi-square distribution” in theorem 1 in the modified version.

Reviewer comment (Evaluation):
"Consider we run an adversarial attack on a dataset and get an accuracy of 80%. Assume the attack is perfect, thus we can always find an adversarial example if there exists one in the region, which is the ultimate goal for adversarial attack research. Then the accuracy means for 80% of the data, we manage to find at least one adversarial example within the region. Thus, the empirical robustness with a perfect attack is a good approximation of the absolute robustness" — Can you elaborate more on this example? What is accuracy here (accuracy of the model on perturbed data or attack success rate)? How based on first 3 sentences we can claim that empirical robustness is a good approximation?

Author response:
Sorry for the confusion about “accuracy” and “attack success rate”. The first “accuracy of 80%” will be corrected to “attack success rate of 80%” in the modified version to eliminate the ambiguity. Thanks for your responsible reviewing to point it out. 
For more elaboration, the absolute robustness for each data point is defined as a 0-1 variable (Definition 1). This paragraph of argument mimics the standard procedure of how we run the empirical adversarial attack (for instance, the PGD attack on CIFAR-10). The key is for each datum, we only apply the attack once, and either find an adversarial example or not. If the attack is perfect, an adversarial example should be found as long as it exists (no matter how small the adversarial region is). So, an 80% attack success rate for an ideal attack corresponds to a case where, for 80% of data, the absolute robustness is 1, and 20% is 0. Thus, it is a good approximation of the absolute robustness (taking the expectation over the dataset).

Reviewer comment (Theory):
The study, while insightful, falls short of providing fundamental theoretical analysis. The example in section 4.2 is overly specific, leaving a gap in the broader understanding. A more general, perhaps probabilistic, theoretical exploration of how randomness leads to gradient masking and how randomness replay enhances robustness would significantly strengthen the work. Additionally, a theoretical validation of the diffusion model's ability to remove adversarial projections would be a valuable addition.

Author response:
Fundamental theoretical analysis. Our original submission considers a specific hyperball case, but indeed this example has a more general implication. As shown in theorem 1 of the modified submission, the theorem can be generalized to any classifier with a simply connected adversarial space. Note that the assumption of simply connected adversarial space is practically appropriate, as Tramèr et al. 2017 showed that the adversarial space spans a continuous region with orthogonal bases. The key observation of the theorem is that the problem caused by randomness in robustness evaluation is an inherent problem of high dimensional space, which can not be solved by either a stronger attack or EOT.
We also state Corollary 2.1 more explicitly that randomness replay can overcome this problem. After randomness replay, the perfect attack will correctly estimate the absolute robustness (and EOT will have no benefits in this scenario), comparing with the non-randomness replay case, the success rate is approaching 0.
Theoretical validation of adversarial projections removal in diffusion models. We provide more analysis in the new version, please refer to the reply of questions for more details.

Reviewer comment (Theory):
Can the authors explain why the diffusion model can remove adversarial projection? I believe this would be insightful for future work.

Author response:
Why can diffusion models remove adversarial projection? We added further analysis in the modified version to partially address this question (Fig. 4). Starting from a batch of random perturbations within a small ball around a clean stimulus, the reconstructed states after diffusion models are positively correlated with a much lower dimension structure. This shows that the learned image priors by diffusion models are biased toward a set of specific directions. Conceivably, these directions are unlikely to coincide with the adversarial directions. Otherwise, if the biased directions of diffusion models were indeed the adversarial directions,  the DiffPure method should fail even for clean images.