On the Limitations of Stochastic Pre-processing Defenses
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Defending against adversarial examples remains an open problem.
A commonbelief is that randomness at inference increases the cost of ﬁnding adversarialinputs.
An example of such a defense is to apply a random transformation toinputs prior to feeding them to the model.
In this paper, we empirically andtheoretically investigate such stochastic pre-processing defenses and demonstratethat they are ﬂawed.
First, we show that most stochastic defenses are weaker thanpreviously thought; they lack sufﬁcient randomness to withstand even standardattacks like projected gradient descent.
This casts doubt on a long-held assumptionthat stochastic defenses invalidate attacks designed to evade deterministic defensesand force attackers to integrate the Expectation over Transformation (EOT) concept.
Second, we show that stochastic defenses confront a trade-off between adversarialrobustness and model invariance; they become less effective as the defended modelacquires more invariance to their randomization.
Future work will need to decouplethese two effects.
Our code is available in the supplementary material.
Introduction
Machine learning models are vulnerable to adversarial examples [4, 29], where an adversary canadd imperceptible perturbations to the input of a model and change its prediction [5, 22].
Theirdiscovery has motivated a wide variety of defense approaches [6, 8, 10, 25, 34, 35] along with theevaluation of their adversarial robustness [2, 24, 32].
Current evaluations mostly rely on adaptiveattacks [2, 32] which require signiﬁcant modeling and computational efforts.
However, even whenthe attack succeeds, such evaluations may not always reveal the fundamental weaknesses of anexamined defense.
Without awareness of the underlying weaknesses, subsequent defenses may stillconduct inadvertently weak adaptive attacks; this leads to overestimated robustness.
One popular class of defenses that demonstrates the above is the stochastic pre-processing defense,which relies on applying randomized transformations to inputs to provide robustness [10, 35].
Despiteexisting attack techniques designed to handle randomness [2, 3], there is an increasing effort toimprove these defenses through a larger randomization space or more complicated transformations.
For example, BaRT [24] employs 25 transformations, where the parameters of each transformationare further randomized.
Due to the complexity of this defense, it was only broken recently (threeyears later) by Sitawarin et al.
[28] with a complicated adaptive attack.
Still, it is unclear how futuredefenses can avoid the pitfalls of existing defenses, largely because these pitfalls remain unknown.
In this paper, we investigate stochastic pre-processing defenses and explain their limitations bothempirically and theoretically.
First, we revisit previous stochastic pre-processing defenses and explainwhy such defenses are broken.
We show that most stochastic defenses are not sufﬁciently randomizedto invalidate standard attacks designed for deterministic defenses.
Second, we study recent stochasticdefenses that exhibit more randomness and show that they also face key limitations.
In particular, we identify a trade-off between their robustness and the model’s invariance to their transformations.
These defenses achieve a notion of robustness that results from reducing the model’s invariance tothe applied transformations.
We outline our ﬁndings below.
These ﬁndings suggest future work toﬁnd new ways of using randomness that decouples these two effects.
Most stochastic defenses lack sufﬁcient randomness.
Although Athalye et al.
[2] and Tramèret al.
[32] have demonstrated the ineffectiveness of several stochastic defenses with techniques like
Expectation over Transformation (EOT) [3], it remains unclear whether and why EOT is requiredto break them.
A commonly accepted explanation is that EOT computes the “correct gradients” ofmodels with randomized components [2, 32], yet the necessity of such correct gradients has not beenexplicitly discussed.
To ﬁll this gap, we examine a long-held assumption that stochastic defensesinvalidate standard attacks designed for deterministic defenses.
Speciﬁcally, we revisit stochastic pre-processing defenses previously broken by EOT and examinetheir robustness without applying EOT.
Interestingly, we ﬁnd that most stochastic defenses lacksufﬁcient randomness to withstand even standard attacks (that do not integrate any strategy to capturemodel randomness) like projected gradient descent (PGD) [22].
We then conduct a systematicevaluation to show that applying EOT is only beneﬁcial when the defense is sufﬁciently randomized. Otherwise, standard attacks already perform well and the randomization’s robustness is overestimated.
Trade-off between adversarial robustness and model invariance.
When stochastic pre-processingdefenses do have sufﬁcient randomness, they must ﬁne-tune the model using augmented training datato preserve utility in the face of randomness added.
We characterize this procedure by the model’sinvariance to the applied defense, where we identify a trade-off between the model’s robustness(provided by the defense) and its invariance to the applied defense.
Stochastic pre-processing defensesbecome less effective when their defended model acquires more invariance to their transformations.
On the theoretical front, we present a theoretical setting where this trade-off provably exists.
We showfrom this trade-off that stochastic pre-processing defenses provide robustness by inducing variance onthe defended model, and must take back such variance to recover utility.
We verify this trade-off withempirical evaluations on realistic datasets, models, and defenses.
We observe that robustness dropswhen the defended model is ﬁne-tuned on data processed by its defense to acquire higher invariance.
Related Work
Stochastic Defenses.
Defending against adversarial examples remains an open problem, where acommon belief is that inference-time randomness increases the cost of ﬁnding adversarial inputs.
Earlyexamples of such stochastic defenses include input transformations [10] and rescaling [35].
Thesedefenses were broken by Athalye et al.
[2] using techniques like EOT [3] to capture randomness.
Afterthat, more stochastic defenses were proposed but with inadvertently weak evaluations [23, 25, 34, 36],which were found ineffective by Tramèr et al.
Subsequent stochastic defenses resort to largerrandomization space like BaRT [24], which was only broken recently by Sitawarin et al.
Randomized smoothing [6, 16, 17] leverages randomness to certify robustness.
In this work, instead ofdesigning adaptive attacks for individual defenses, we focus on the general stochastic pre-processingdefenses and demonstrate their limitations.
Trade-offs for Adversarial Robustness.
The trade-offs associated with adversarial robustness havebeen widely discussed in the literature.
Prior work identiﬁed trade-offs between robustness andaccuracy [33, 37] and even between the robustness against different types of adversarial examples [12,31].
Recent work also investigated the trade-off between the model’s robustness and invariance toinput transformations, such as circular shifts [27] and rotations [13].
These trade-offs characterizea standalone model’s own property – the model itself is less robust to adversarial examples whenit becomes more invariant to certain transformations, without any defense.
Our setting, however,is orthogonal to such analysis; the model we consider is protected by a stochastic pre-processingdefense, and we aim to characterize the performance of the added defense to the model.
Preliminaries
Notation.
Let f : X !
R C denote the classiﬁer with pre-softmax outputs, where X = [0 , 1] d isthe input space with d dimensions and C is the number of classes.
We then consider a stochastic pre-processing defense t ✓ : X !
X , where ✓ is the random variable drawn from some randomization 88 space ⇥ that parameterizes the defense.
The defended classiﬁer can be written as f ✓ ( x ) :
= f ( t ✓ ( x )) .
Let F ( x ) :=
arg max i
Y f i ( x ) denote the classiﬁer that returns the predicted label, where f i is theoutput of the i -th class and Y =
[ C ] is the label space.
Similarly, we use F ✓ and f ✓ ,i to denote theprediction and class-output of the stochastic classiﬁer f ✓ .
Since this classiﬁer returns varied outputsfor a ﬁxed input, it determines the ﬁnal prediction by aggregating n independent inferences withstrategies like majority vote.
We discuss these strategies and the choice of n in Appendix A.1.
Adversarial Examples.
Given an image x 2 X and a classiﬁer F , the adversarial examplex 0 :
= x +  is visually similar to x but either misclassiﬁed (i.e., F ( x 0 ) 6 = F ( x ) ) or classiﬁed asa target class y 0 chosen by the attacker
(i.e., F ( x 0 ) = y 0 ).
Attack algorithms generate adversarialexamples by searching for  such that x 0 fools the classiﬁer while minimizing  under some distancemetrics; for instance, the ` p norm constraint k  k p  ✏ for a perturbation budget ✏ .
Projected Gradient Descent (PGD).
PGD [22] is one of the most established attacks to evaluateadversarial example defenses.
Given a benign example x 0 and its ground-truth label y , each iterationof the untargeted PGD attack (with ` 1 norm budget ✏ ) can be formulated as
[Equation]
where ↵ is the step size, L is the loss function, and each iteration is projected to the ` 1 ball aroundx 0 of radius ✏ .
We use PGD- k to denote the PGD attack with k steps.
We outline formulations forother settings and norms in Appendix A.2.
Expectation over Transformation (EOT).
Since the classiﬁer f ✓ is stochastic, the defense evaluation literature [2, 32] argues that attacks should target the expectation of the gradient using Expectationover Transformation (EOT) [3], which reformulates the PGD attack as
[Equation]
where m is the number of samples to estimate the expectation and ✓ j iid ⇠ ⇥ are sampled parametersfor the defense.
We use EOT- m to denote the EOT technique with m samples at each PGD step.
In addition, for a fair comparison among attacks with different PGD steps and EOT samples, wequantify the attack’s strength by its total number of gradient computations.
For example, attacksusing PGD- k and EOT- m will have strength k ⇥ m .
Although white-box attacks are typically notconstrained in this way, it allows for a fair comparison when attacks have ﬁnite computing resources(e.g., when EOT is not parallelizable).
We discuss more about this quantiﬁcation in Appendix A.3.
Most Stochastic Defenses Lack Sufﬁcient Randomness
Athalye et al.
[2] and Tramèr et al.
[32] demonstrate adaptive evaluation of stochastic defenses withthe application of EOT.
However, it remains unclear why EOT is required to break these stochasticdefenses.
While a commonly accepted explanation is that EOT computes the “correct gradients” ofmodels with randomized components [2, 32], the necessity of such correct gradients has not beenexplicitly discussed.
To ﬁll this gap, we revisit stochastic defenses previously broken by EOT andexamine their robustness without applying EOT.
Interestingly, we ﬁnd that applying EOT is mostlyunnecessary when evaluating existing stochastic defenses.
[Table]
Case Study: Random Rotation.
We start with a simplestochastic defense that randomly rotates the input image for✓ 2 [  90 , 90] degrees (chosen at uniform) before classiﬁcation.
This defense is representative for most pre-processingdefenses [10, 24, 35].
We evaluate this defense on 1,000 ImageNet images with PGD- k and EOT- m under the constraintk ⇥ m = 50 , as discussed in Section 3.
All attacks use maximum ` 1 perturbation ✏ = 8 / 255 with step size chosen from↵ 2 { 1 / 255 , 2 / 255 } .
The results are shown in Table 1, where
[Table] [Table]
[Table]
The details of these defenses and their evaluation settings are in Appendix B.
[Table]
PGD-50 performs equally well as PGD-10 combined with EOT-5.
This observation suggests thatsome stochastic defenses are already breakable without applying EOT , casting doubt on a long-heldassumption that stochastic defenses simply invalidate attacks designed for deterministic defenses.
Comprehensive Evaluations.
We then extend the above case study to other stochastic defensesevaluated in the literature.
Speciﬁcally, we replicate the (untargeted) adaptive evaluation of stochasticdefenses from Athalye et al.
[2] and Tramèr et al.
[32] with their ofﬁcial implementation.
We onlychange the attack’s hyper-parameters (e.g., number of iterations and learning rate) and disable EOTby setting its number of samples to one ( m = 1 ), which avoids potential implementation ﬂaws ifremoved from the source code.
The comparison between evaluations with and without applying EOTis summarized in Table 2, which serves as a missing ablation study of adaptive evaluations in theliterature.
The experimental settings are identical within each row (detailed in Appendix B).
Interestingly, we ﬁnd it unnecessary to break these defenses with EOT, as long as the standardattack runs for more iterations with a smaller learning rate.
For such defenses, standard iterativeattacks already contain an implicit expectation across iterations to capture the limited randomness.
This observation implies that most stochastic defenses lack sufﬁcient randomness to withstand evenstandard attacks designed for deterministic defenses.
Therefore, increasing randomness becomes apromising approach to enhancing stochastic defenses, as adopted by recent defenses [6, 24].
Notethat this ablation study only aims to inspire potential ways of enhancing stochastic defenses; it doesnot invalidate EOT for stronger adaptive evaluations of stochastic defenses.
Trade-offs between Robustness and Invariance
When stochastic pre-processing defenses do have sufﬁcient randomness, they must ensure that theutility of the defended model is preserved in the face of randomness.
To achieve high utility, existingdefenses mostly rely on augmentation invariance through trained invariance [21].
In such a case, theinvariance is achieved by applying the defense’s randomness to the training data so as to guide themodel in learning their transformations.
For defenses based on stochastic pre-processor t ✓ , each datasample from the dataset gets augmented with t ✓ sampled from the randomization space ⇥ , and therisk is minimized over such augmented data.
The defended classiﬁer F ✓ ( x ) :
[Equation]
F
[Equation]
t ✓
[Equation]
)) is invariant under the randomization space ⇥ if
[Equation]
As we can observe from the deﬁnition, invariance has direct implications on the performance ofstochastic pre-processing defenses.
If the classiﬁer is invariant under the defense’s randomizationspace ⇥ as is deﬁned in Equation (3), then the defense should not work – computing the model andits gradients over randomization ✓ 2 ⇥ is the same as if t ✓ was not applied at all.
This observationsuggests a direct coupling between invariance and performance of the defense: the more invariant,hence performant, the model is under a given randomization space, the less protection such a defensewould provide.
In this section, we present a simple theoretical setting where this coupling provablyexists; detailed computations and illustrations of this setting are in Appendix C.1.
Binary Classiﬁcation Task.
We consider a class-balanced dataset D consisting of input-label pairs( x, y ) with y 2 {  1 , +1 } and x | y ⇠ N ( y, 1) , where N ( µ,  2 ) is a normal distribution with mean µ 170 and variance  2 .
Moreover, an ` 1 -bounded adversary perturbs the input with a small ✏ to fool theclassiﬁer.
We quantify the classiﬁer’s robustness by its robust accuracy, i.e., the ratio of correctlyclassiﬁed samples that remain correct after being perturbed by the adversary.
Note that while we usex + ✏ to denote the perturbed input, its actual value can take within the range [ x  ✏ , x + ✏ ] .
Undefended Classiﬁcation.
We start with the optimal linear classiﬁer F ( x ) := sgn( x ) without anydefense.
This classiﬁer attains robust accuracy
[Equation]
where  is the cumulative distribution function of N (0 , 1) .
Defended Classiﬁcation.
We then try to improve adversarial robustness by introducing a stochasticpre-processing defense t ✓ ( x ) :
= x + ✓ , where ✓ ⇠ N (1 ,  2 ) is the random variable parameterizingthe defense.
This defense characterizes common pre-processing defenses that enforce randomnesswhile shifting the input distribution.
Here, the processed input follows a shifted distribution t ✓ ( x ) ⇠
N ( y + 1 , 1 +  2 ) .
As a result, the defended classiﬁer F ✓ ( x ) = sgn
( x + ✓ ) has robust accuracy
[Equation]
where  0 ( x ) :=  ( x/ p 1 +  2 ) is the cumulative distribution function of N (0 , 1 +  2 ) .
At thispoint, we have not ﬁt the classiﬁer on processed inputs.
Due to its lack of invariance, the defendedclassiﬁer has low utility yet higher robust accuracy than the undefended one in Equation (4).
Defended Classiﬁcation (Trained Invariance).
As discussed above, one critical step of stochasticpre-processing defenses is to preserve the defended model’s utility by minimizing the risk overaugmented data t ✓ ( x ) , which leads to a new defended classiﬁer F + ✓ ( x ) =
sgn( x + ✓  1) .
As aresult, this new defended classiﬁer achieves higher invariance with robust accuracy
Pr
[Equation]
which is less robust than the previous less-invariant classiﬁer F ✓ in Equation (5).
However, one mayobserve that this classiﬁer, though loses some robustness compared with F ✓ , is still more robustthan the original undefended classiﬁer F in Equation (4).
This part of robustness comes from thechanged data distribution due to the defense’s randomness.
It shows that we have not achieved perfectinvariance to the defense’s randomness, thus gaining some robustness at the cost of utility.
Defended Classiﬁcation (Perfect Invariance).
Furthermore, these defenses usually leverage majorityvote to obtain stable predictions, which ﬁnally produces a perfectly invariant defended classiﬁer
[Equation]
where ✓ i iid ⇠ N (1 ,  2 ) are sampled parameters.
In such a case, the defended classiﬁer reduces to theoriginal undefended classiﬁer with the original robust accuracy:
[Equation]
Summary.
The above theoretical setting illustrates how stochastic pre-processing defenses ﬁrst inducevariance on the binary classiﬁer we consider to provide adversarial robustness in Equation (5), andhow they ﬁnally take back such variance in Equations (6) and (8) to recover utility.
In Appendix C.2,we extend this coupling to a general trade-off where we can control the invariance.
This trade-offdemonstrates that stochastic pre-processing defenses provide robustness by explicitly reducing themodel’s invariance to added randomized transformations.
Experiments
Our experiments are designed to answer the following two questions.
Q1:
What properties make applying EOT beneﬁcial when evaluating stochastic defenses?
We show that applying EOT is only beneﬁcial when the defense is sufﬁciently randomized; otherwisestandard attacks already perform well and leave no room for EOT to improve.
Q2:
What is the limitation of stochastic defenses when they do have sufﬁcient randomness?
We show a trade-off between the stochastic defense’s robustness and the model’s invariance tothe defense itself.
Such defenses become less effective when the defended model achieves higherinvariance to their randomness, as required to preserve utility under the defense.
6.1 Experimental Settings
Datasets & Models.
We conduct all experiments on ImageNet [26] and ImageNette [9].
For ImageNet, our test data consists of 1,000 images randomly sampled from the validation set.
ImageNette isa ten-class subset of ImageNet, and we test on its validation set.
We adopt various ResNet [11] models.
For defenses with low randomness, we evaluate them on ImageNet with a pre-trained ResNet-50 with
Top-1 accuracy 75.9%.
For defenses with higher randomness (thus requiring ﬁne-tuning), we switchto ImageNette and a pre-trained ResNet-34 with Top-1 accuracy 96.9% to reduce the training cost likeprevious work [28].
These models are ﬁne-tuned on the training data processed by tested defenses.
As a special case, we also evaluate randomized smoothing on ImageNet using the ResNet-50 modelsfrom Cohen et al.
More details of datasets and models can be found in Appendices D.1 and D.2.
Defenses & Metrics.
We focus on stochastic defenses allowing us to increase randomness: randomized smoothing [6] and BaRT [24].
For randomized smoothing, we vary the variance of the added
Gaussian noise.
For BaRT, we vary the number  of applied randomized transformations.
Note thatwe have evaluated other stochastic defenses and discussed their low randomness in Section 4.
Wemeasure the defense’s performance by the defended model’s benign accuracy and the attack’s successrate , all evaluated with majority vote over n = 500 predictions.
The attack’s success rate is the ratioof samples that do not satisfy the attack’s objective prior to the attack but satisfy it after the attack.
For example, we discard samples that were misclassiﬁed before being perturbed in untargeted attacks.
Details of the evaluated defenses can be found in Appendix D.3.
Attacks.
We evaluate defenses with standard PGD combined with EOT and focus on the ` 1 -boundedadversary with a perturbation budget ✏ = 8 / 255 in both untargeted and targeted settings.
We onlyconduct adaptive evaluations, where the defense is included in the attack loop with non-differentiablecomponents captured by BPDA [2].
We also utilize AutoPGD [7] to avoid selecting the step sizewhen it is computationally expensive to repeat some experiments.
More details of the attack’s settingand implementation can be found in Appendix D.4.
6.2 Q1: Evaluate the Beneﬁts of Applying EOT under Different Settings
In Section 4, we showed that standard attacks are sufﬁcient to break most stochastic defenses due totheir lack of randomness.
Here, we aim to understand what properties make applying EOT beneﬁcialwhen evaluating stochastic defenses.
We design a systematic evaluation of stochastic defenses withdifferent level of randomness and check if applying EOT improves the attack.
Stochastic Defenses with Low Randomness.
We start with BaRT’s noise injection defense, whichperturbs the input image with noise of distributions and parameters chosen at random.
While thisdefense has low randomness, it yields meaningful results.
We evaluate this defense with variouscombinations of PGD and EOT.
The performance of untargeted and targeted attacks is shown in
Figure 1.
We test multiple step sizes and summarize their best results (discussed in Appendix E.1).
In this case, standard PGD attacks are already good enough when the defense has insufﬁcientrandomness, leaving no space for improvements from EOT.
In Figure 1f, both (1) PGD-10 combinedwith EOT-10 and (2) PGD-100 combined with EOT-1 have near 100% success rates.
This result isconsistent with our observations in Section 4 in both untargeted and targeted settings 1 .
Stochastic Defenses with Higher Randomness.
We then examine the randomized smoothingdefense that adds Gaussian noise to the input image.
Although this defense was originally proposedfor certiﬁable adversarial robustness, we adopt it to evaluate how randomness affects the beneﬁts ofapplying EOT.
Similarly, we evaluate this defense with PGD and EOT of different settings with afocus on the targeted attack.
The results are shown in Figure 2.
[Figure] [Figure]
We observe that EOT starts to improve the attack when the defense has a higher level of randomness.
For a ﬁxed number of PGD steps, applying EOT signiﬁcantly improves the attack in most of thesettings.
For a ﬁxed attack strength (i.e., number of gradient computations), applying EOT alwaysoutperforms standalone PGD.
In Figure 2f, for example, PGD-100 combined with EOT-10 is 5.5%higher than PGD-1,000 with EOT-1 (40.3% vs. 34.8%).
Takeaways.
Applying EOT is only beneﬁcial when the defense has sufﬁcient randomness, such asrandomized smoothing with  = 0 .
This observation suggests that stochastic defenses only makestandard attacks suboptimal when they have sufﬁcient randomness.
However, most existing stochasticdefenses did not achieve this criterion, as we showed in Section 4.
6.3 Q2: Evaluate the Trade-off between Robustness and Invariance
In Section 5, we present a theoretical setting where the trade-off between robustness and invarianceprovably exists; stochastic defenses become less robust when the defended model achieves higher
[Figure] [Figure]
invariance to their randomness.
Here, we demonstrate this trade-off on realistic datasets, models, anddefenses.
In particular, we choose defenses with sufﬁcient randomness (achieved in different ways)and compare their performance when being applied to models of different levels of invariance, wherethe invariance is achieved by applying the defense’s randomness to the training data so as to guidethe model in learning their transformations.
Randomness through Transformations.
We ﬁrst examine the BaRT defense, which pre-processesinput images with  randomly composited stochastic transformations.
It represents defenses aimingto increase randomness through diverse input transformations.
Since our objective is to demonstratethe trade-off, it sufﬁces to evaluate a subset of BaRT with   6 transformations; this also avoids thetraining cost of evaluating the original BaRT with  = 25 .
Figure 3 shows the performance of thisdefense with models before and after ﬁne-tuning on its processed training data.
In Figures 3a and 3c, we ﬁrst observe that ﬁne-tuning indeed increases the model’s invariance tothe applied defense’s randomness; the utility’s dashed green curves are improved to the solid greencurves beyond 90%.
However, as the model achieves higher invariance, the defense becomes nearlyineffective; the attack’s dashed red curves boost to the solid red curves near 100%.
The same attack’seffectiveness throughout the ﬁne-tuning procedure further veriﬁes this observation, as shown in
Figures 3b and 3d.
It shows a clear trade-off between the defense’s robustness and the model’sinvariance.
That is, stochastic defenses start to lose robustness when their defended models achievehigher invariance to their transformations.
Randomness through Noise Levels.
We then examine the randomized smoothing defense that adds
Gaussian noise to the input image.
Unlike BaRT’s diverse transformations, randomized smoothingincreases randomness directly through the added noise’s variance  2 .
This allows us to rigorouslyincrease the randomness without unexpected artifacts like non-differentiable components.
We evaluate the performance of this defense (   0 . 5 ) with models before and after ﬁne-tuning on training data perturbed with designated Gaussian noise.
The results are shown in Figure 4.
In Figures 4a and 4c, ﬁne-tuning improves the model’s invariance, but the defense also becomessigniﬁcantly weaker during this process.
For example, the targeted attack is nearly infeasible whenthe model is variant to the large noise (   0 . 3 ), yet is signiﬁcantly more effective when the modelbecomes invariant.
The ﬁne-tuning process in Figures 4b and 4d also veriﬁes that stochastic defensesbecome weaker when their defended models become more invariant to their randomness.
Takeaways.
For both the BaRT and the randomized smoothing defense, we observe a clear trade-offbetween the defense’s robustness and the model’s invariance to randomness.
In particular, we ﬁnd thatstochastic defenses lose adversarial robustness when their defended models achieve higher invarianceto their randomness.
Our ﬁnding implies that such defenses would become ineffective when theirdefended models are perfectly invariant to their randomness.
Discussions
[Figure]
In this section, we discuss several questions that arose from our study of the pre-processing defense.
The limitations and potential negative societal impacts of this work are discussed in Appendix F.
What do pre-processing defenses really do?
We show that existing stochastic pre-processingdefenses do not introduce inherent robustness to the prediction task.
Instead, they shift the inputdistribution through randomness and transformations, which results in variance and introduces errorsduring prediction.
The observed “robustness”, in an unusual meaning for this literature, is a result ofthese errors.
This is fundamentally different from the inherent robustness provided by adversarialtraining [22].
Although defenses like adversarial training still cost accuracy [33, 37], they do notintentionally introduce errors like stochastic pre-processing defenses.
However, stochastic pre-processing defenses do make the attack harder when the adversary has onlylimited knowledge of the defense’s transformations, e.g., in a low-query setting.
In such a case, thedefense practically introduces noise to the attack’s optimization procedure, making it difﬁcult for alow-query adversary to ﬁnd adversarial examples that consistently cross the probabilistic decisionboundary.
In contrast, our theoretical analysis in Section 5 considers a powerful adversary withfull knowledge of the defense’s randomization space; hence it can optimize the adversarial exampledirectly towards the defended model’s decision boundary in expectation.
How should we utilize randomness?
Stochastic defenses should rely on randomness that exploitsthe properties of the prediction task.
For example, some speech and vision problems are inherentlydivisible into orthogonal subproblems (e.g., performing classiﬁcation on portions of the speechspectrum for keyword spotting [1]), where randomness can apply to the space of these subproblems.
Such defenses decouple robustness and invariance because (1) the models perform well on eachindependent subproblem, and (2) attacks on one subproblem do not transfer to the other subproblems.
As such, an attacker has to target all possible subproblems, which reduces its effective attack budget.
What are implications for adaptive attackers?
Our ﬁndings demonstrate that an adaptive attackerneeds to consider the spectrum of available standard attack algorithms, instead of just focusing on agiven attack algorithm because of the defense’s design.
As we discover in this paper, EOT can beunnecessary for seemingly immune stochastic defenses, yet its application to break these said defensesgives a false impression about their security against weak attackers.
When evaluating the robustnessof a defense, the adaptive attack should start by tuning standard approaches, before resorting to moreinvolved attack strategies.
This approach helps us to identify the minimally capable attack that breaksthe defense and develop a better understanding of the defense’s fundamental weaknesses.
Conclusion
In this paper, we investigate stochastic pre-processing defenses and explain their limitations bothempirically and theoretically.
We show that most stochastic pre-processing defenses are weaker thanpreviously thought, and recent defenses that indeed exhibit more randomness still face a trade-offbetween their robustness and the model’s invariance to their transformations.
While defending againstadversarial examples remains an open problem and designing proper adaptive evaluations is arguablychallenging, we demonstrate that stochastic pre-processing defenses are fundamentally ﬂawed in theircurrent form.
Our ﬁndings suggest that future work will need to ﬁnd new ways of using randomnessthat decouples the robustness and invariance.