Paper: Subspace Grid-sweep: ML Defense Evaluation via Constrained Brute-force Search (ICLR-2024)

Reviewer comment (Evaluation):
Why do the results in Figure 2 and Figure 7(a) show differences?

Author response:
Figure 2 does show that deterministic smoothing is less robust than randomized smoothing when they use lower numbers of corruptions, but converges to the same robustness as the number of corruptions they use increase (implying randomness is not required for empirical robustness for randomized smoothing as discussed in Section 4.1, and it is more important instead to use a sufficient number of corruptions). 

Figure 7a is part of Section D in the appendix, which, separately, describes a new threat model where the attacker repeatedly tries the same input to defeat a defense. Figure 7a shows that a random defense will become less robust than a deterministic defense given this threat model (hence the gap).

Reviewer comment (Reproducibility):
Clarity/Quality. In the main paper and Appendix I haven’t found the explanation on how exactly Auto-Attack was restricted to the continious K-dimensional space? Given that K is extremely small (1-6), this could drastically affect the overall Auto-Attack performance. Without understanding that it is hard to interpret the results e. g. in Table 1.

Author response:
Running AutoAttack in this K-dimensional basis was accomplished by simply adding the K x D matrix (discussed in Sec 3.1) that defines the subspace to the forward function right before the input (where there are K dimensions in the subspace and D is the shape of the original input). This is essentially the same as adding an encoder to the input where AutoAttack modifies the basis weights (and receives backpropagated loss for the basis weights) for the K orthonormal basis vectors that define the subspace instead of the individual pixels of the input. In addition to this straightforward implementation, we know AutoAttack works fine in the subspaces by looking at the results in Tables 1 and 2 that show AutoAttack’s A-PGD finding nearly every vulnerable datapoint (99.8%) that Subspace Grid-sweep found on the undefended (base) classifiers.

The code implementing this forward function for AutoAttack is method `forward_with_basis_vectors(...)` in lines 464-482 in `code/core.py` in the supplementary material.

Reviewer comment (Evaluation):
Quality. In Section 4.1 the Authors evaluate Randomized Smoothing with standard PGD. Although they claim to use up to 10 000 steps, it still doesn’t seem like a suitable evaluation method. A-PGD or AutoAttack would be better.

Author response:
We believed this evaluation seems the most appropriate as it is the same attack used (and recommended) by the original authors of Randomized Smoothing (Cohen et al., 2019), and so we wanted to show that Deterministic Smoothing would have been evaluated to have the same robustness as Randomized Smoothing in the original paper. Using a different attack to compare the robustness would be beside the point of the comparison.

However, we actually did use AutoAttack’s A-PGD on Deterministic Smoothing as reported in Section 5 to compare with Subspace Grid-sweep’s ability to find vulnerable datapoints in lower-dimensional subspaces and found no indication that Deterministic Smoothing is especially more vulnerable to A-PGD than PGD. Therefore, as we suspect Randomized Smoothing would also not be especially more vulnerable to A-PGD than PGD, our findings do not change if we consider A-PGD instead of PGD in Section 4.1. These results can be highlighted if the reviewer prefers.

Reviewer comment (Experiments):
Quality. If I understood correctly, all the evaluations were performed on small-resolution datasets (CIFAR-10, SVHN, MNIST). However, Randomized Smoothing is able to provide provable defence e. g. for models on ImageNet. Given the exponential nature of the method, it is hard to understand whether it would scale to datasets of higher resolution.

Author response:
As the purpose of Subspace Grid-sweep is to be comparison point to standard evaluation attacks (to see if they are finding vulnerable points that Subspace Grid-sweep finds), then our reported experiments showing issues with defenses using smaller-resolution datasets are still valid, as a defense that is obscuring gradients in a lower-resolution dataset is sufficient cause for concern. 

While we have no reason to believe using a larger dataset would not work (as we would still only search a subspace of less than 10 dimensions in a higher-resolution input space), we do not understand any benefit of doing so when we show that using lower-resolution (and much easier to experiment with) datasets is sufficient to show the issue in modern defenses.

Reviewer comment (Evaluation):
In the limitations the authors admit that fixing random seed in e. g. Randomized smoothing makes it vulnerable for an attacker that knows it. I assume that with the knowledge of Subspace Grid-sweep’s grid a model could be trained that seems robust when evaluated in the predefined grid points but contains adversarial regions in-between. That could result in backdoor-attacks on seemingly robust models. Thus the method can provide false sense of security.

Author response:
To clarify, we posed as future work to explicitly explore if the attacker somehow has an edge by knowing the random seed randomized smoothing is fixed to. However, our experimental results in Sections 4 and 5 and Figure 2 instead found empirical evidence that the attacker does not have an edge, at least with current attacks used for evaluation. 

Also, as later shown in Section 5 using Subspace Grid-sweep in Table 2 where (Smoothing) Defended has a higher Robustness against Grid-sweep than (Smoothing) Base, we show that the attacker may have a hard time using the knowledge of the random seed because there are simply smaller adversarial regions (i.e., many fewer adversarial examples) that can be found when using randomized smoothing, regardless of whether it is random or deterministic.

In response to the posed threat model on if knowledge of Subspace Grid-sweep’s subspaces and grid sizes could allow an adversary to modify their defense to make Subspace Grid-sweep incorrectly deem a defense robust, we point out the following characteristics of Subspace Grid-sweep that should alleviate concerns.
* Subspace Grid-sweep is not designed to be a standalone *attack*. The purpose of Subspace Grid-sweep is to check whether benchmark attacks (such as AutoAttack’s A-PGD) are finding vulnerable datapoints that should be found, so AutoAttack’s A-PGD is also executed against the defense being evaluated. In the case of an adversary specifically evading Subspace Grid-sweep, they would also need to ensure that AutoAttack’s A-PGD also cannot find these vulnerable regions placed in-between grid points.
* Additionally, the subspaces for Subspace Grid-sweep are chosen at random (via randomly sampled orthonormal basis vectors), and many subspaces and grids are sampled for an evaluation (shown in Appendix A), so the attacker would need to know, and figure out how to avoid, all the different subspaces and grids of all future evaluators that may want to test the defense.
* Finally, we also do a random search in the subspaces (results in Appendix A), that would not adhere to the known grid points, and this could also find any hidden adversarial regions places in-between grid points

Reviewer comment (Writing):
What is the role of parameter σ in GridSweepPerturbationsL2(K, B, M, σ)? (Section 3.1)

Author response:
We apologize for the confusion. $\sigma$ represents the $L_2$ constraint distance that the generated perturbations must adhere to. We are happy to change this to $\epsilon$ to match how we refer to the constraint in the text.

Reviewer comment (Reproducibility):
How exactly was the orthonormal basis chosen (Section 3.1)?

Author response:
The orthonormal basis is chosen randomly by using a seed defined by the dimensionality of the subspace (K) and the number of bins (B) requested. This is to ensure that the same subspace is used to compare Defended and Undefended (base) classifiers in the several different subspaces we evaluate with (enumerated in Appendix A).

First, we randomly sample a $K \times D $ matrix (after setting the seed), transpose it, use `scipy.linalg.orth` to create orthogonal columns, transpose it again, divide each row by its $L_2$ norm, then exhaustively verify all K rows are orthonormal.

For more detail, the code calculating the orthonormal basis vectors that define the subspaces we search in is in lines 134-146 in `code/basis_experiments.py` in the supplementary material.

Reviewer comment (Reproducibility):
What is the advantage of using a grid in a subspace over simply randomly sampling noisy perturbations? Also, function $\texttt{GridSweepPerturbationsL2}$ is unclear to me. What is $\sigma$ (I assume this should be $\epsilon$)? For g=0 and B=6, the new g assignment is -1, but the comment is misleading because it says the points are normalized between [0,1].

Author response:
> What is the advantage of using a grid in a subspace over simply randomly sampling noisy perturbations?
> What is the advantage of iterating over a grid in a subspace rather than over randomly sampled noisy perturbations?

As we understand it, there are two parts to this question: (1) Why use a grid search instead of random sampling? And (2) Why search in a lower-dimensional subspace? We’ll give our reasoning on the first question and then the second.

The advantage of using a grid is that it allows us to get a sort of *ground-truth* for what adversarial regions exist that should be able to be found by gradient-guided attacks usually used for evaluation (such as AutoAttack’s A-PGD). As a grid search checks every point at evenly distanced intervals within a space, a grid search ensures that no regions larger than the interval between grid points will be missed. This is in contrast to random sampling that, from bad luck, could miss a large adversarial region in a space by not choosing a point there. However, exhaustively doing a grid-search in a large dimensional space is prohibitive.

Hence, the answer to question 2. The reason we do grid-search in a lower-dimensional subspace is to make it feasible. By also constraining gradient-guided attacks to the same subspace, we have a sort of ground-truth (for regions of size above the interval between grid points) for what datapoints an attack should find an adversarial example for.

On top of this, we also do random search in our lower-dimensional subspace as an extra measure to find adversarial regions that could be hiding between grid points (results in Appendix A).


> Also, function GridSweepPerturbationsL2 is unclear to me. What is $\sigma$ (I assume this should be epsilon)? For g=0 and B=6, the new g assignment is -1, but the comment is misleading because it says the points are normalized between [0,1].

Thank you for catching this typo! You are correct, the comment should say “scale grid to [-1,1] and origin center”. Also, you are correct that $\sigma$ represents the $L_2$ constraint distance. We are happy to change this to $\epsilon$ to match how we refer to the constraint in the text.

Reviewer comment (Evaluation):
The insights gained from Section 4.1 on Randomized Smoothing are not conclusive to me. The authors state: "This indicates that the empirical robustness of Randomized Smoothing does not come from the randomness of the noise, but from the self-ensembling effect of aggregating multiple inferences within proximity of the original point." Figure 2 shows that a white-box attacker achieves the same success rate regardless of whether the noised perturbations are being resampled during the optimization procedure. However, the inability of an attacker to find an adversarial example does not imply that the randomness does not help. If the attacker knows the corruptions used by the defender, more successful attacks are conceivable.

Author response:
> However, the inability of an attacker to find an adversarial example does not imply that the randomness does not help. If the attacker knows the corruptions used by the defender, more successful attacks are conceivable.

As discussed in Section 6, we posed as future work to explicitly explore if the attacker somehow has an edge.

However, for the purposes of this paper in showing whether currently used evaluation attacks empirically evaluate robustness well, this edge does not show up given benchmark attacks. Some explanation of why this edge does not show is discussed in Section 5 using Subspace Grid-sweep in Table 2 where (Smoothing) Defended has a higher Robustness against Grid-sweep than (Smoothing) Base, which implies there are simply smaller adversarial regions (i.e., many fewer adversarial examples) that can be found when using randomized smoothing, regardless of whether it is random or deterministic.

Reviewer comment (Presentation):
Table 1 is confusing to me. Why is the robustness higher for the undefended model? What is the difference between the first column and the second and third?

Author response:
> Table 1 is confusing to me. Why is the robustness higher for the undefended model? What is the difference between the first column and the second and third?

You have read this table correctly, and this shows the main result from Section 3, in that kWTA-defended models are actually less robust than the original undefended versions. Our Subspace Grid-sweep shows this issue without catering an attack specifically to kWTA as previous works had to do (Tramèr et al., 2020).

The first column shows robustness against Grid-sweep, as defined in algorithm 2 (1 - (number of vulnerable datapoints / number of datapoints)). The second
and third columns show the number of vulnerable datapoints Grid-sweep found and how many of those AutotAttack’s A-PGD found. The idea behind these columns is to make it easily visible when AutoAttack’s A-PGD is not finding many of the vulnerable datapoints that Grid-sweep is finding, which would indicate the defense is hindering AutoAttack’s A-PGD from finding existing adversarial regions/examples.

Reviewer comment (Novelty):
Why is it meaningful to study the robustness of deterministic defenses?

Author response:
> In other words, if an attack succeeds against a deterministic variant of a defense, what implications does that have on the robustness of the randomized version?
> Why is it meaningful to study the robustness of deterministic defenses?


This is a good question. Whether a deterministic variant failing indicates a flaw in the original random variant likely depends on the specific defense being evaluated. If we find that the deterministic variant is showing obfuscated gradients, then adding randomness likely does not solve the obfuscated gradients. However, if a deterministic defense is much less robust to APGD than the random version, then this could mean different things based on what the details of the defense are, but it at least means that the unpredictability of the randomness to the attacker is important for this defense to work, as with only that component removed the defense is degraded. Overall, it gives more information to the evaluator.

Reviewer comment (Experiments):
The grid-sweep-based attacks seem to be very straightforward, and I think if the hidden space is large, the search space for this attack could be still huge.

Author response:
We agree that if the subspace dimensionality is too large (i.e., more than 20), then it would stop making sense to exhaustively search it with subspace grid-sweep. However, the underlying point of the evaluation is not to prove that a defense is robust, but to see if the current attacks being used to evaluate a defense are good enough. For that, we found that using much smaller subspaces (1-6 dimensions) did pretty well, as discussed in Sections 3 and 5, with more detail given in the Appendix. For this reason, we do not think the search space needs to be large to check whether gradient-guided attacks are being hindered by a defense.

Reviewer comment (Experiments):
The experiments of diffPure are not clear to me. I believe that diffPure still needs some randomness to be successful otherwise it would be no different from a plain denoiser network. Why is the robust accuracy still as high as around 65% in Figure 4?

Author response:
This is a good question! Firstly, we are pretty certain that this is the correct result, since we use the exact same code as the original DiffPure paper, except that we fix the random seed to be the same before each inference. We suspect that there is some gradient masking occurring that does not allow AutoAttack to find adversarial examples, which is also supported by our Subspace Grid-sweep results with DiffSmall in Section 5. Fortunately, some recent concurrent work came to the same conclusion by spending a lot more effort specifically examining DiffPure, and found compelling evidence that DiffPure has obfuscated gradients that prevent AutoAttack from finding valid adversarial examples [A]. Our results using Subspace Grid-sweep also discover these issues, and they do so in a more straightforward and general way.

[A] Zhao, Zhengyu, et al. "Towards Good Practices in Evaluating Transfer Adversarial Attacks." arXiv preprint arXiv:2211.09565 (2022).