E VADING A DVERSARIAL E XAMPLE D ETECTION D EFENSES WITH O RTHOGONAL P ROJECTED G RADIENT D ESCENT
A BSTRACT
Evading adversarial example detection defenses requires ﬁnding adversarial examples that must simultaneously (a) be misclassiﬁed by the model and (b) be detected as non-adversarial.
We ﬁnd that existing attacks that attempt to satisfy multiple simultaneous constraints often over-optimize against one constraint at the cost of satisfying another.
We introduce Selective Projected Gradient Descent and
Orthogonal Projected Gradient Descent , improved attack techniques to generate adversarial examples that avoid this problem by orthogonalizing the gradients when running standard gradient-based attacks.
We use our technique to evade four stateof-the-art detection defenses, reducing their accuracy to 0% while maintaining a 0% detection rate.
I NTRODUCTION
Generating adversarial examples (SZS + 14; BCM + 13), inputs designed by an adversary to cause a neural network to behave incorrectly, is straightforward.
By performing input-space gradient descent (CW17b; MMS + 17), it is possible to maximize the loss of arbitrary test examples.
This process is both efﬁcient and highly effective.
But despite much effort by the community, attempts at designing defenses against adversarial examples have been largely unsuccessful and gradient-descent attacks continue to circumvent new defenses (ACW18; TCBM20).
As a result, many defenses aim to make generating adversarial examples more difﬁcult by requiring additional constraints on inputs for them to be considered successful.
Defenses that rely on detection , for example, will reject inputs if a secondary detector model determines the input is adversarial (MGFB17; XEQ17).
Turning a benign input x into an adversarial example x (cid:48) thus now requires fooling both the original classiﬁer, f , and the detector, g , simultaneously.
Traditionally, this is done by constructing a single loss function L that jointly penalizes the loss on f and the loss on g (CW17a), e.g., by deﬁning L ( x (cid:48) )
=
L ( f ) + λ L ( g ) and then minimizing L ( x (cid:48) ) with gradient descent.
Unfortunately, many evaluations using this strategy have had limited success—not only must λ be tuned appropriately, but the gradients of f and g must also be well behaved.
Our contributions.
We develop a new attack technique designed to construct adversarial examples that simultaneously satisfy multiple constraints.
Our attack approach is a modiﬁcation of standard gradient descent (MMS + 17) and requires changing just a few lines of code.
Given two objective functions f and g , instead of taking gradient descent steps that optimize the joint loss function f + λg , we selectively take gradient descent steps on either f or g .
This makes our attack both simpler and easier to analyze than prior attack approaches.
We use our technique to evade four state-of-the-art and previously-unbroken defenses to adversarial examples: the Honeypot defense (CCS’20) (SWW + 20), Dense Layer Analysis (IEEE Euro
S&P’20) (SKCB19), Sensitivity Inconsistency Detector (AAAI’21) (TZLD21), and the SPAM detector presented in Detection by Steganalysis (CVPR’19) (LZZ + 19).
In all cases, we successfully reduce the accuracy of the protected classiﬁer to 0% while maintaining a detection AUC of less than 0 .
5 —meaning the detector performs worse than random guessing.
B ACKGROUND 2. N OTATION
We consider classiﬁcation neural networks f : R d → R n that receive a d -dimensional input vector (in this paper, images) x ∈ R d and output an n -dimensional prediction vector f ( x ) ∈ R n .
We let g : R d → R denote some other function which also must be considered, where g ( x ) ≤ 0 when the constraint is satisﬁed and g ( x ) >
if it is violated.
Without loss of generality, in a detection defense this function g is the detector and higher values corresponding to higher likelihood of the input being an adversarial example.
To denote the true label of x is given by y
we write c ( x ) =
y .
In an abuse of notation, write y = f
( x ) to denote the arg-max most likely label under the model f .
2. A DVERSARIAL E XAMPLES
Adversarial examples (SZS + 14; BCM + 13) have been demonstrated in nearly every domain in which neural networks are used.
(ASE + 18; CW18; HPG + 17)
Given an input x corresponding to label c ( x ) and classiﬁer f , an adversarial example is a perturbation x (cid:48) of the input such that d ( x, x (cid:48) ) < (cid:15) and f ( x (cid:48) )
(cid:54) =
c ( x ) for some metric d .
Additionally, an adversarial example can be targeted if, given a target label t (cid:54) =
c ( x ) we have f ( x (cid:48) )
=
t with d ( x, x (cid:48) ) < (cid:15) .
The metric d is most often that induced by a p -norm, typically either || · || 2 or || · || ∞ .
With small enough perturbations under these metrics, the adversarial example x (cid:48) is not perceptibly different from the original input x .
Datasets.
We attack each defense on the dataset that it performs best on.
All of our defenses operate on images.
For three of these defenses, this is the CIFAR-10 dataset (KH09), and for one, it is the ImageNet dataset (DDS + 09).
For each defense we attack, we constrain our adversarial examples to the threat model originally considered to perform a fair re-evaluation, but also generate adversarial examples with standard norms used extensively in prior work in order to make crossdefense evaluations meaningful.
We perform all evaluations on a single GPU.
Our attacks on
CIFAR-10 require just a few minutes, and for ImageNet a few hours (primarily due to the defense having a throughput of one image per second).
2. D ETECTION D EFENSES
We focus our study on detection defenses.
Rather than directly improve the robustness of the model (MMS + 17; RSL18; LAG + 19; CRK19), detection defenses classify inputs as adversarial or benign (MGFB17; XEQ17) so they can be rejected.
While there have been several different strategies attempted to detect adversarial examples in the past (GSS15; MGFB17; FCSG17; XEQ17; MC17;
MLW + 18; RKH19), many of these approaches were broken with adaptive attacks that designed new loss functions tailored to each defense (CW17a; TCBM20).
2. G ENERATING A DVERSARIAL E XAMPLES WITH P ROJECTED G RADIENT D ESCENT
Projected Gradient Descent (MMS + 17) is a powerful ﬁrst-order method for ﬁnding such adversarial examples.
Given a loss L ( f, x, t ) that takes a classiﬁer, input, and desired target label, we optimize over the constraint set S (cid:15) =
{ z : d ( x, z ) < (cid:15) } and solve
[Equation]
(1) by taking steps x i +1 =
Π S (cid:15) ( x i − α ∇ x i L ( f, x i , t )) .
Here Π S (cid:15) denotes projection onto the set S (cid:15) , α is the step size, and x 0 is randomly initialized (MMS + 17).
This paper adapts PGD in order to solve optimization problems which involve minimizing multiple objective functions simultaneously.
For notational simplicity, in the remainder of this paper we will omit the projection operator Π S (cid:15) .
Attacks using PGD.
Recent work has shown that it is possible to attack models with adaptive attacks that target speciﬁc aspects of defenses.
For detection defenses this process is often ad hoc , involving alterations speciﬁc to each given defense (TCBM20).
An independent line of work develops automated attack techniques that are reliable indicators of robustness (CH20); however, in general, these attack approaches are difﬁcult to apply to detection defenses.
f(x) <g(x) <f(x) <g(x) <f(x) <g(x) <
R ETHINKING A DVERSARIAL E XAMPLE D ETECTION
Before we develop our improved attack technique to break adversarial example detectors, it will be useful to understand why evaluating adversarial example detectors is more difﬁcult than evaluating standard classiﬁers.
Early work on adversarial examples often set up the problem slightly differently than we do above in Equation 1.
The initial formulation of an adversarial example (SZS + 14; CW17b) asks for the smallest perturbation δ such that f ( x + δ ) is misclassiﬁed.
In the targeted case, this means solving arg min (cid:107) δ (cid:107) 2 such that f ( x + δ ) =
t for a given target t that is not equal to the original correct label for x .
Solving this problem as stated is intractable.
It requires searching over a nonlinear constraint set, which is not feasible for standard gradient descent.
As a result, detection papers typically (SWW + 20; SKCB19) reformulate the search with a Lagrangian relaxation
[Equation]
This formulation is simpler, but still (a) requires tuning λ to work well, and (b) is only guaranteed to be correct for convex functions L —that it works for non-convex models like deep neural networks is not theoretically justiﬁed.
It additionally requires carefully constructing loss functions L (CW17b).
Equation 1 simpliﬁes the setup considerably by exchanging the constraint and objective.
Whereas in
Equation 2 we search for the smallest perturbation that results in misclassiﬁcation, Equation 1 instead ﬁnds an input x + δ with δ < (cid:15) that minimizes the classiﬁer’s loss.
This is a simpler formulation because now the constraint is convex, and so we can run standard gradient descent optimization.
Evading detection defenses is difﬁcult because there are now two non-linear constraints .
Not only must the input be constrained by a distortion bound and be misclassiﬁed by the base classiﬁer, but we must also have that they are not detected, i.e., with g ( x ) < 0 .
This new requirement is nonlinear, and now it becomes impossible to side-step the problem by merely swapping the objective and the constraint as we did before: there will always be at least one constraint that is a non-linear function, and so standard gradient descent techniques can not directly apply.
In order to resolve this difﬁculty, the existing literature applies the same Lagrangian relaxation as was previously applied to constructing minimum-distortion adversarial examples.
That is, breaking a detection scheme involves solving
[Equation]
where λ is a hyperparameter that controls the relative importance of fooling the classiﬁer versus fooling the detector.
However, this formulation again brings back all of the reasons why the community moved past minimum-distortion adversarial examples.
3. A M OTIVATING E XAMPLE
Let f ( (cid:126)x ) =
exp( − 1) − exp( −(cid:107) (cid:126)x − (cid:126)e (cid:107) 22 ) − ε
and g ( (cid:126)x ) =
exp( − 1) − exp( −(cid:107) (cid:126)x + (cid:126)e (cid:107) 22 ) − ε where (cid:126)e ∈ R N and (cid:107) (cid:126)e (cid:107)
= 1 , as visualized in Figure 1.
By setting ε to a small constant, the only solution that satisﬁes both f ( (cid:126)x ) < 0 and g ( (cid:126)x ) < 0 can be made arbitrarily close to the origin (cid:126)x = (cid:126) 0 .
However, no standard Lagrangian formulation will be able to ﬁnd this solution.
Consider the sum h
( (cid:126)x ; λ ) =
f ( (cid:126)x ) + λg ( (cid:126)x ) ; then we can show that for all λ we will have arg min (cid:126)x h ( (cid:126)x ; λ ) (cid:54) = 0 .
To see this, observe that while it is possible for the gradient ∇ h ( (cid:126) 0; λ ) =
(one of the conditions for a value to be a local minima), the loss surface is always “concave down” at the origin.
It will always be possible to move slightly closer to (cid:126)e or − (cid:126)e and decrease the loss.
Therefore, minimizing h ( x ) will never be able to ﬁnd a valid stable solution to this extremely simple problem, as it will always collapse to ﬁnding a solution of either (cid:126)e or − (cid:126)e , which only satisﬁes one of the two equations.
O UR A TTACK A PPROACHES
We now present our attack strategy designed to generate adversarial examples that satisfy two constraints.
As we have been doing, each of our attack strategies deﬁned below generates a targeted adversarial example x (cid:48) so that f ( x (cid:48) )
=
t but g ( x (cid:48) )
< 0 .
Constructing an untargeted attack is nearly identical except for the substitution of maximization instead of minimization.
4. S ELECTIVE GRADIENT DESCENT
Instead of minimizing the weighted sum of f
and g , our ﬁrst attack never optimizes against a constraint once it becomes satisﬁed.
That is, we write our attack as
[Equation]
The idea here is that instead of minimizing a convex combination of the two loss functions, we selectively optimize either f or g depending on if f ( x ) =
t , ensuring that updates are always helping to improve either the loss on f or the loss on g .
Another beneﬁt of this style is that it decomposes the gradient step into two updates, which prevents imbalanced gradients , where the gradients for two loss functions are not of the same magnitude and result in unstable optimization (JMW + 20).
In fact, our loss function can be viewed directly in this lens as following the margin decomposition proposal (JMW + 20) by observing that
[Equation]
That is, each iteration either take gradients on f or on g depending on whether f ( x ) = t or not.
Recalling the motivating example from Figure 1, this selective optimization formulation would be able to ﬁnd a valid solution.
No matter where we initialized our adversarial example search, minimizing with respect to f ( x ) will eventually give a valid solution near e .
Once this happens, we then switch to optimizing against g ( x ) (because f ( x ) is satisﬁed).
From here we will eventually converge on the solution x ≈ (cid:126) 0 .
4. O RTHOGONAL GRADIENT DESCENT
The prior attack, while mathematically correct, might encounter numerical stability difﬁculties.
Often, the gradients of f and g point in opposite directions, that is, ∇ f ≈ −∇ g .
As a result, every step spent optimizing f causes backwards progress on optimizing against g .
This results in the optimizer constantly “undoing” its own progress after each step that is taken.
To address this problem, we would like to "remove" the portion of one gradient optimization step that "undoes" the progress of a previous optimization step.
To do this, we call on vector projections.
Note that ∇ g ( x ) ⊥ =
∇L ( f, x, t ) − proj ∇ g ( x ) ∇L ( f, x, t ) is orthogonal to the gradient ∇ g ( x ) , and similarly ∇L ( f, x, t ) ⊥ is orthogonal to ∇L ( f, x, t ) .
We integrate this fact with Equation 5 to give a slightly different update rule that again solves Equation 4, however this time by optimizing:
[Equation]
The purpose of this update is to take gradient descent steps with respect to one of f or g in such a way that we do not signiﬁcantly disturb the loss of the function not chosen.
In this way, we prevent our attack from taking steps that undo work done in previous iterations of the attack.
It is also important to note that, in the high-dimensional space in which a typical neural network operates, the gradients of f and g are practically never exactly opposite, that is a situation where ∇ f = −∇ g .
In this case, the projection of ∇ f onto ∇ g and ∇ g onto ∇ f would be
and we could not make any meaningful optimizations towards satisfying either constraint with OPGD.
Again recalling Figure 1, by taking steps that are orthogonal to f ( x ) we can ensure that once we reach the acceptable region for f , we never leave it, and much more quickly converge on an adversarial example that evades detection.
C ASE S TUDIES
We validate the efﬁcacy of our attack by using it to circumvent four previously unbroken, state-of-theart defenses accepted at top computer security or machine learning venues.
Three of the case studies utilizes models and code obtained directly from their respective authors.
In the ﬁnal case the original authors provided us with matlab source code that was not easily used, which we re-implemented.
Attack Success Rate Deﬁnition.
We evaluate the success of our attack by a standard metric called attack success rate at N (SR@N for short) (SKCB19).
We use SR@N to ensure comparability across different case studies but more importantly between our results and our case studies’ original results.
SR@N is deﬁned as the fraction of targeted attacks that succeed when the defense’s false positive rate is set to N % .
(To adjust a defense’s false positive rate it sufﬁces to adjust the detection threshold φ so that inputs are rejected as adversarial when g ( x ) > φ .)
For example, a 94% SR @5 could either be achieved through 94% of inputs being misclassiﬁed as the target class and 0% being detected as adversarial, or by 100% of inputs being misclassiﬁed as the target class and 6% being detected as adversarial, or some combination thereof.
We report SR@5 and SR@50 for our main results 1 , and for completeness also give the full ROC curve of the detection rate for a more complete analysis.
Attack Hyperparameters.
We use the same hyperparmaeter setting for all attacks shown below.
We set the distortion bound ε to 0 .
01 and . 031 ; several of these papers exclusively make claims using the value of 0 .
01 (SWW + 20; SKC + 20; TZLD21), but the value 0 .
031 = 8 / 255 is more typical in the literature (TCBM20).
We run our attack for N = 1000 iterations of gradient descent with a step size α =
ε 10 (that is, the step size changes as a function of ε which follows standard advice (MMS + 17)).
5. H ONEYPOT D EFENSE
The ﬁrst paper we consider is the Honeypot Defense (SWW + 20).
Instead of preventing attackers from directly constructing adversarial examples, the authors propose to lure attackers into producing speciﬁc perturbations that are easy to ﬁnd and hard to ignore.
These perturbations are called “honeypots” or trapdoors and can be easily identiﬁed by a detector.
For their evaluation on the MNIST and CIFAR-10 dataset, the authors use 5 sets of randomly selected 3 × 3 squares per label.
Formally, consider an input x to the classiﬁer, f .
During training, f is injected with a honeypot, ∆ .
The signature of a particular honeypot, S ∆ , is the expectation of the neuron activations of f over multiple sample inputs containing ∆ .
During inference, the internal neuron activation pattern e ( x ) is compared to S ∆ using cosine similarity.
Speciﬁcally, for a predeﬁned threshold φ , if cos( e ( x ) , S ∆ ) > φ , then x is ﬂagged as adversarial.
One additional modiﬁcation the authors make is to use neuron randomization.
When creating the signatures, a random sample of neuron activations
Attack eps=0.
eps=0.
SR@ SR@
SR@
SR@ (SWW + 20) 0.
Reproduction 0.13 ∗ - - -
Orthogonal 1.
Selective 0.
(a) Attack success rate for our two proposed attacks.
∗ When we reproduce the authors attack, we reach slightly a higher attack success rate of 13%.
1.0 False Positive Rate 0.
T r u e
P o s i t i v e R a t e
Orthogonal Selective (b) Defense ROC, ε = 0 .
01 , AUC of 0.
We test our method of generating adversarial images on CIFAR-10 with an l ∞ bound of (cid:15) = 8255 .
The original model has a baseline accuracy of . 87 .
To test the defense, we generate targeted attacks choosing the target from among the incorrect labels.
As a simple baseline attack, we conﬁrm that
FGSM achieves an accuracy of .
06 on the classiﬁer but the ROC AUC of the detector is .
To evaluate the efﬁcacy of our attack, we use the authors pre-trained code, and convert their models from TensorFlow to PyTorch to run our attacks.
We ﬁnd that our attack is extremely effective and has an attack success rate above 99 .
9% , even at a 50% false positive rate, and an AUC of 0.30.
In Table 2 we give the attack results for several conﬁgurations of our attack.
We shared our results with the original defense authors who conﬁrmed our adversarial examples successfully evaded their detection (an important step, given that we had converted the model to PyTorch).
5. D ENSE L AYER A NALYSIS
Many recent defenses analyze the hidden activations of neural networks while processing benign and adversarial inputs (e.g., following (MGFB17)).
These defenses aim to capitalize on differences in activation patterns among benign and adversarial inputs to train a separate classiﬁer capable of detecting inputs as adversarial.
The most recent paper in this direction Sperl et al.
extract dense layer activation patterns among benign and adversarial inputs and train a secondary binary classiﬁer that detects adversarial examples (SKCB19).
The authors do this by ﬁrst performing a forward pass through a target neural network with both adversarial and benign inputs to create a mixed-feature dataset of activation-label pairs.
Then, using the mixed-feature dataset, they train a secondary binary classiﬁer capable of discerning between adversarial and benign inputs.
When evaluating their models, the authors pass an input through the target model to obtain the activation feature vectors for a particular input as well as a potential classiﬁcation.
They then pass this feature vector through the secondary classiﬁer.
If the secondary classiﬁer alerts that the input was adversarial, the classiﬁcation is thrown away.
Otherwise, classiﬁcation proceeds as normal.
Sperl et al.
evaluate this defense with 5 leading adversarial attacks on the MNIST and CIFARdatasets using several models and report high accuracies for benign inputs and high detection rates for adversarial inputs.
The authors report a worst-case individual attack accuracy of 0.739.
In accordance with our framework, we assign the cross entropy loss of the classiﬁer to our primary function and binary cross entropy loss of the detector as our secondary function.
We obtain source code and pre-trained defense models from the authors in order to ensure that our attack matches the defense as closely as possible.
We now detail the results of our attack at (cid:15) = .
Attack eps=0.
eps=0.
SR@ SR@
SR@
SR@ (SKC + 20) ≤ 0.13 ∗ - - -
Reproduction 0.20 + - - -
Orthogonal 0.
Selective 0.
(a) success rate ∗ paper reported at , we
1.0 False Positive Rate 0.
T r u e
P o s i t i v e R a t e
Orthogonal Selective (b) Defense ROC, ε = 0 .
01 , AUC of 0.
and at (cid:15) = .
03 at false positive rates of 5% and 50% in Figure 3.
We ﬁnd that our attack is extremely effective, resulting in an accuracy of 0 at a detection rate of 0 with a false positive rate of 5% under (cid:15) = .
03 bounds and an AUC of 0.38.
Finally, to validate that our attack succeeded, we again shared the resulting adversarial examples with the authors who conﬁrmed our attack results.
5. S ENSITIVITY I NCONSISTENCY OF S PATIAL -T RANSFORM D OMAIN
We next evaluated our attack on the Sensitivity Inconsistency Detector (SID) proposed by Tian et al.
(TZLD21).
This defense relies on the observations of Fawzi et al.
(FMDFS18) that adversarial examples are movements, in the form of perturbations, of benign inputs in a decision space along an adversarial direction.
Tian et al.
then conjecture that, because adversarial examples are likely to lie near highly-curved decision boundaries, and benign inputs lie away from such boundaries, ﬂuctuations in said boundaries will often result in a change in classiﬁcation of adversarial examples but not in classiﬁcation of benign inputs.
To measure sensitivity against decision boundary transformations, Tian et al.
design a dual classiﬁer which is the composition of a weighted additive wavelet transform layer and a DNN classiﬁer with the same structure as the original classiﬁer.
When doing a forward pass of the system, the authors run an input through both the primal and the dual model, then pass both results to the detector that discriminates among adversarial and benign classes.
With these models, the authors then deﬁne their so-called feature of sensitivity inconsistency S ( x 0 ) .
[Equation]
where f i ( x 0 ) and g i ( x 0 ) are the predictions of the primal and the dual respectively.
Input x 0 is classiﬁed as adversarial if S ( x 0 ) is greater than a threshold φ .
SID achieves improved adversarial example detection performance, especially in cases with small perturbations in inputs.
The authors report a worst-case, individual attack detection AUC % of 0.95.
Now, we want to create adversarial examples that are misclassiﬁed by the original model and not ﬂagged as adversarial by the Sensitivity Inconsistency Detector.
We assign the loss of our target model to our primary function and the loss of the Sensitivity Inconsistency Detector as our secondary function.
The initial target model had an accuracy of .94 and deemed .06 of all inputs adversarial.
We again obtain source code from the authors along with pre-trained models to ensure evaluation correctness.
We describe our attack’s results at (cid:15)
= .
01 and at (cid:15) = .
03 at false positive rates of 5% and 50% in Figure 4.
Our attack works well in this case and induces an accuracy of 0 at a detection rate of 0 with a false positive rate of 5% under (cid:15)
= .
03 bounds with an AUC of 0.25.
5. D ETECTION THROUGH S TEGANALYSIS
Since adversarial perturbations alter the dependence between pixels in an image, Liu et al.
(LZZ + 19) propose a defense which uses a steganalysis -inspired approach to detect “hidden features” within an
Attack eps=0.
eps=0.
SR@ SR@
SR@
SR@ (TZLD21) ≤ 0.09 ∗ - - -
Orthogonal 0.
Selective 0.
a
1.0 False Positive Rate 0.
T r u e
P o s i t i v e R a t e
Orthogonal Selective (b) Defense ROC, ε = 0 .
01 , AUC of 0.
image.
These features are then used to train binary classiﬁers to detect the perturbations.
Unlike the prior defenses, this paper evaluates on ImageNet, reasoning that small images such as those from
CIFAR-10 and MNIST do not provide enough inter-pixel dependency samples to construct efﬁcient features for adversarial detection, so we attack this defense on ImageNet.
As a baseline, the authors use two feature extraction methods: SPAM and Spatial Rich Model.
For each pixel X i,j of an image X , SPAM takes the difference between adjacent pixels along 8 directions.
For the rightward direction, a difference matrix A → is computed so that A → i,j = X i,j − X i,j +1 .
A transition probability matrix M → between pairs of differences can then be computed with
M → x,y =
Pr ( A → i,j +
=
x
| A → i,j = y ) where x, y ∈ {− T, ..., T } , with T being a parameter used to control the dimensionality of the ﬁnal feature set F .
We use T = 3 in accordance with that used by the authors.
The features themselves are calculated by concatenating the average of the non-diagonal matrices with the average of the diagonal matrices:
[Equation]
In order to use the same attack implementation across all defenses, we reimplemented this defense in PyTorch (the authors implementation was in matlab).
Instead of re-implementing the full Fisher
Linear Discriminant (FLD) ensemble (KFH12) used by the authors, we train a 3-layer fully connected neural network on SPAM features and use this as the detector.
This allows us to directly investigate the claim that SPAM features can be reliably used to detect adversarial examples, as FLD is a highly non-differentiable operation and is not a fundamental component of the defense proposal.
The paper also proposes a second feature extraction method named “Spatial Rich Model” (SRM) that we do not evaluate against.
This scheme follows the same fundamental principle as SPAM in modeling inter-pixel dependencies—there is only a marginal beneﬁt from using these more complex models, and so we analyze the simplest variant of the scheme.
Notice that SPAM requires the difference matrices A to be discretized in order for the dimensionality of the transition probability matrices M to be ﬁnite.
To make this discretization step differentiable and compatible with our attacks, we deﬁne a count matrix X where, for example, X → x,y counts, for every pair i, j , the number of occurrences of y in A → i,j and x in A → i,j +1 .
M → x,y is then deﬁned by:
[Equation]
To construct a differentiable approximation, consider without loss of generality the rightward difference matrix A → 1 for an image.
We construct a shifted copy of it
A → 2 so that A → 2 i,j =
A → 1 i,j +1 .
We then deﬁne a mask K so that
[Equation]
Attack eps=0.
eps=0.
SR@ SR@
SR@
SR@ (LZZ + 19) 0.
Orthogonal 0.
(a) rate
1.0 False Positive Rate 0.
T r u e
P o s i t i v e R a t e
Orthogonal (b) Defense ROC, ε = 0 .
01 , AUC of 0.
Figure
Each element of the intermediate matrix X → x,y counts the number of pairs in A → 1 and A → 2 which would be rounded to x and y respectively after discretization:
[Equation]
where ◦ is the Hadamard product.
If we normalize X → so that the sum of elements in each column is equal to 1 , we get the probability of difference values x ∈ A → 2 conditioned on column
y ∈
A → 1 .
Thus, for any pair of indices i, j ,
[Equation]
Using this differentiable formulation of SPAM feature extraction, we train an auxillary detector as described above and use its gradients to apply our attack on the original, non-differentiable detector.
The authors evaluate their defense on 4 adversarial attacks and report high accuracy for benign inputs and high detection rates for adversarial inputs.
The best attack they develop still has a success rate less than 3% .
In contrast, our attack on SPAM using the differentiable approximation has a success rate of 98 .
8% when considering a 5% false positive rate, with an AUC of 0.44, again less than the random guessing threshold of 0 . 5 .
C ONCLUSION
Generating adversarial examples that satisfy multiple constraints simultaneously (e.g., requiring that an input is both misclassiﬁed and deemed non-adversarial) requires more care than generating adversarial examples that satisfy only one constraint (e.g., requiring only that an input is misclassiﬁed).
We ﬁnd that prior attacks unnecessarily over-optimizes one constraint when another constraint has not yet been satisﬁed.
Our new attack methogology is designed to avoid this weakness, and as a result can reduce the accuracy of four previously-unbroken detection methods to 0% accuracy while maintaining a 0% detection rate at 5% false positive rates.
We believe our attack approach is generally useful, but it is not a substitute for trying other attack techniques.
We do not envision this attack as a complete replacement for standard Lagrangian-based attacks, but rather a complement; defenses must carefully consider their robustness to both prior attacks as well as this new one.
Notice, for example, that for one of the four defenses we study
Selective PGD performs better than Orthogonal PGD—indicating these attacks are complementary to each other.
Additionally, automated attack tools (CH20) would beneﬁt from adding our optimization trick to their collection of known techniques that could (optionally) compose with other attacks.
We discourage future work from blindly applying this attack without properly understanding its design criteria.
While this attack is effective for the defenses we consider, it is not the only way to do so, and may not be the correct way to do so in future defense evaluations.
Evaluating adversarial example defenses will necessarily require adapting any attack strategies to the defense’s design.
A CKNOWLEDGEMENTS
We thank the authors of the papers we use in the case studies, who helped us answer questions speciﬁc to their respective defenses and agreed to share their code with us. We are also grateful to Alex Kurakin and the anonymous reviewers for comments on drafts of this paper.
E THICS S TATEMENT
All work that improves adversarial attacks has potential negative societal impacts.
Yet, we believe that it is better for those vulnerabilities to be known rather than relying on security through obscurity.
We have attacked no deployed system, and so cause no direct harm; and by describing how our attack works, future defenses will be stronger.
We have communicated the results of our attack to the authors of the papers we break as a form of responsible disclosure, and also to ensure the correctness of our results.
R EPRODUCIBILITY S TATEMENT
All of the code we used to generate our results will be made open source in a GitHub repository.
The datasets we use (MNIST, CIFAR10, ImageNet) are available online and widely studied.
We obtained original copies of the code associated with 3 of the 4 case studies.
We used code either directly from the authors or code released publicly alongside an academic paper.
In the case of steganalysis, we reimplemented the paper to the best of our ability.
We also provide a Python class constructor so that future work can test or improve our results.
Again, we relayed results to the authors of each paper and received conﬁrmation that our adversarial examples were indeed adversarial and not detected by the author’s original implementations.