W ALKING ON THE E DGE : F AST , L OW -D ISTORTION A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples of deep neural networks are receiving ever increasing attention because they help in understanding and reducing the sensitivity to their input.
This is natural given the increasing applications of deep neural networks in our everyday lives.
When white-box attacks are almost always successful, it is typically only the distortion of the perturbations that matters in their evaluation.
In this work, we argue that speed is important as well, especially when considering that fast attacks are required by adversarial training.
Given more time, iterative methods can always ﬁnd better solutions.
We investigate this speed-distortion trade-off in some depth and introduce a new attack called boundary projection (BP) that improves upon existing methods by a large margin.
Our key idea is that the classiﬁcation boundary is a manifold in the image space: we therefore quickly reach the boundary and then optimize distortion on this manifold.
I NTRODUCTION
Adversarial examples (Szegedy et al., 2013) are small, usually imperceptible perturbations of images or other data (Carlini & Wagner, 2018) that can arbitrarily modify a classiﬁer’s prediction.
They have been extended to other tasks like object detection or semantic segmentation (Xie et al., 2017), and image retrieval (Li et al., 2018;
Tolias et al., 2019).
They are typically generated in a white-box setting, where the attacker has full access to the classiﬁer model and uses gradient signals through the model to optimize for the perturbation.
They are becoming increasingly important because they reveal the sensitivity of neural networks to their input (Simon-Gabriel et al., 2018; Fawzi et al., 2016;
Amsaleg et al., 2017) including trivial cases (Azulay & Weiss, 2018; Engstrom et al., 2017) and they easily transfer between different models (Moosavi-Dezfooli et al., 2016a; Tram`er et al., 2017b).
Adversarial examples are typically evaluated by probability of success and distortion .
In many cases, white-box attacks have probability of success near one, then only distortion matters, as a (weak) measure of imperceptibility and also of the ease with which adversarial samples can be detected.
The speed of an attack is less frequently discussed.
The fast single-step FGSM attack (Goodfellow et al., 2014) produces high-distortion examples where adversarial patterns can easily be recognized.
At the other extreme, the Carlini & Wagner (C&W) attack (Carlini & Wagner, 2017), considered state of the art, is notoriously expensive.
Decoupling direction and norm (DDN) (Rony et al., 2018) has recently shown impressive progress in distortion but mostly in speed.
Speed becomes more important when considering adversarial training (Goodfellow et al., 2014).
This defense, where adversarial examples are used for training, was in fact introduced in the same work as FGSM.
The latter remains the most common choice for generating those examples because of its speed.
However, unless a powerful iterative attack is used (Madry et al., 2017), adversarial training is easily broken (Tram`er et al., 2017a).
In this work, we investigate in more depth the speed-distortion trade-off in the regime of probability of success near one.
We observe that iterative attacks often oscillate across the classiﬁcation boundary, taking long time to stabilize.
We introduce a new attack that rather walks along the boundary.
As a result, we improve the state of the art in distortion while keeping iterations at a minimum.
Illustrating the attacks.
To better understand how our attack works, we illustrate it qualitatively against a number of existing attacks in Fig.
On this toy 2d classiﬁcation problem, the class boundary and the path followed by the optimizer starting at input x can be easily visualized.
[Figure]
PGD 2 , an ‘ 2 version of I-FGSM (Kurakin et al., 2016), a.k.a.
PGD (Madry et al., 2017), is controlled by a distortion target (cid:15) and eventually follows a path on a ball of radius (cid:15) centered at x ( cf .
Fig.
1(a)).
Section 4 shows that varying (cid:15) is an effective yet expensive strategy.
It can only be done for a limited set of values so that the optimal distortion target per image may only be found by luck.
C&W (Carlini & Wagner, 2017) depends on a parameter (cid:21) that controls the balance between distortion and classiﬁcation loss.
A low value may lead to failure.
A higher value may indeed reach the optimal perturbation, but with oscillations across the class boundary ( cf .
Fig. 1(b)).
Therefore, an expensive line search over (cid:21) is performed internally.
DDN (Rony et al., 2018)
( cf .
Fig.
1(c)) increases or decreases distortion on the ﬂy depending on success and at the same time pointing towards the gradient direction.
It arrives quickly near the optimal perturbation but still suffers from oscillations across the boundary.
On the contrary, boundary projection (BP), introduced in this work ( cf .
Fig.
1(d)), cares more about quickly reaching the boundary, not necessarily near the optimal solution, and then walks along the boundary , staying mostly in the adversarial (red) region.
It therefore makes steady progress towards the solution rather than going back and forth.
Our attack.
Our key idea is that, once we reach the adversarial region near the boundary, the problem becomes optimization on a manifold (Absil et al., 2009): in particular, minimization of the ‘ 2 distortion on a level set of the classiﬁcation loss.
When in the adversarial region, we project the distortion gradient on the tangent space of this manifold.
We do this simply by targeting a particular reduction of the distortion while moving orthogonally to the gradient of the classiﬁcation loss.
Our benchmark.
Quantization is another major issue in this literature.
Most papers implicitly assume that the output of a white-box attack is a matrix where pixel values are real numbers in [0; 1] .
Rony et al.
(2018) is one of the rare works where the output is a quantized.
We agree with this deﬁnition of the problem.
Indeed, an adversarial image is above all an image.
The goal of an attacker is to publish images deluding the classiﬁer (for instance on the web), and publishing implies compliance with pixels encoded in bytes.
Contributions.
We make the following contributions.
To our knowledge, we are the ﬁrst to
Study optimization on the manifold of the classiﬁcation boundary for an adversarial attack, providing an analysis under a number of constraints, such as staying on the tangent space of the manifold and reaching a distortion target.
Investigate theoretically and experimentally the quantization impact on the perturbation.
Achieve at the same speed as I-FGSM (Kurakin et al., 2016) (20 iterations) and under the constraint of a quantization, less distortion than state-of-the-art attacks including DDN, which needs 100 iterations on ImageNet.
P ROBLEM , BACKGROUND AND RELATED WORK 2. P ROBLEM FORMULATION
Preliminaries.
Let X := f 0; (cid:1); : : : ; 1 (cid:0) (cid:1); 1 g n with (cid:1) :
= 1=(L (cid:0) 1) denote the set of grayscale images of n pixels quantized to L levels, and let ^ X :=
[0; 1] n denote the corresponding real-valued images.
An image of more than one color channels is treated independently per channel; in this case n stands for the product of pixels and channels.
A classiﬁer f : ^ X !
R k maps an image x to a vector f( x ) 2 R c+ representing probabilities per class over c given classes.
The parameters of the classiﬁer are not shown here because they remain ﬁxed in this work.
The classiﬁer prediction (cid:25) :
^ X !
[c] := f 1; : : : ; c g maps x to the class label having the maximum probability:
[Equation]
If a true label t 2 [c] is known, the prediction is correct if (cid:25)( x ) = t .
Problem.
Let x 2 X be a given image with known true label t .
An adversarial example y 2 X is an image such that the distortion kx
[Equation]
yk is small and the probability f( y ) t is also small.
This problem takes two forms:
Target distortion, minimal
[Equation]
where (cid:15) is a given distortion target .
The performance is then measured by the probability of success P suc := P((cid:25)(y) 6 = t) as a function of (cid:15) .
Target success, minimal distortion :
[Equation]
to
[Equation]
The performance is then measured by the expected distortion D :=
E( k x (cid:0) y k ) .
This work focuses on the second form, but we present example attacks of both forms in section 2.2.
Untargeted attack.
The constraint (cid:25)(y) 6 = t in (5) is referred to as an untargeted attack, meaning that y is misclassiﬁed regardless of the actual prediction.
As an alternative, a targeted attack requires that the prediction (cid:25)(y) =
t 0 is a target label t
= t .
We focus on the former.
Loss function.
We focus on a white-box attack in this work.
Such an attack is speciﬁc to f , which is public.
In this setting, attacks typically rely on exploiting the gradient of some loss function, using variants of gradient descent.
A classiﬁcation loss is deﬁned on the probability vector p = f( y ) with respect to the true label t .
For an untargeted attack, this is typically the negative of cross-entropy
[Equation]
We should warn that, while the cross-entropy is appropriate for bringing examples into the region of class t during classiﬁer training, its negative (6) is in general not appropriate for pulling them out during an attack.
This is because this function is mostly ﬂat in the class region.
A common solution is to normalize the gradient of ‘ (Goodfellow et al., 2014; Rony et al., 2018), assuming it is nonzero.
We consider more options in this work.
A targeted attack on the other hand may use (cid:0) log p t 0 , which works ﬁne because it brings examples into class t 0 region.
Distortion.
This work focuses on the 2 -norm k(cid:1)k as a measure of distortion.
Alternatives like 1 -norm and 1 -norm are also common (Goodfellow et al., 2014; Carlini & Wagner, 2017).
It is known that none is appropriate for measuring the imperceptibility of adversarial attacks, while more sophisticated measures like structural similarity (SSIM) (Wang et al., 2004) are limited too (Sharif et al., 2018).
Measuring imperceptibility is arguably as difﬁcult as classiﬁcation itself.
Integral constraint.
The constraint y 2 X in (3) and (5) is typically relaxed to y 2 ^
X during optimization.
Some works conclude the attack by loosely quantizing the optimal solution onto
[Equation]
typically by truncation towards zero.
To our knowledge, DDN (Rony et al., 2018) is the only work to do rounding instead, and at the end of each iteration.
Quantization is becoming an important issue in adversarial examples because the distortions achieved in recent papers are so small that quantization impacts a lot the perturbations.
Appendix A provides a more in-depth study of the impact of the quantization.
2. A TTACKS
Target Distortion .
Given a distortion target (cid:15) , the fast gradient sign method (FGSM) (Goodfellow et al., 2014) performs a single step in the opposite direction of the (element-wise) sign of the loss gradient with 1 -norm (cid:15) ,
[Equation]
This is the fastest method for problem (2)-(3).
In the same work adversarial training was introduced, this method quickly generates adversarial examples for training.
However, the perturbations are usually high-distortion and visible.
The iterative -FGSM
(I-FGSM)
(Kurakin et al., 2016) initializes y 0 :
= x and then iterates
[Equation]
where projection 1 is element-wise to the closed 1 -norm ball B 1 [x; (cid:15)] of radius (cid:15) and center x , and also to ^ X (element-wise clipping to interval [0; 1] ).
This method is also known as basic iterative method (BIM)
(Papernot et al., 2018) and as projected gradient descent (PGD) (Madry et al., 2017).
We refer to as PGD 2 a 2 -norm version replacing (8) with
[Equation]
where (cid:17)(x) :=
x=
k x k denotes 2 -normalization, and projection is to the closed 2 -norm ball B 2 [x; (cid:15)] of radius (cid:15) and center x , followed again by element-wise clipping to [0; 1] .
Although this method is part of Cleverhans library (Papernot et al., 2018), it is not published according to our knowledge.
Target Success .
This family of attacks is typically more expensive.
Szegedy et al.
(2013) propose a
Lagrangian formulation of problem (4)-(5), minimizing the cost function
[Equation]
where variable (cid:21) is a Lagrange multiplier for (5).
They carry out this optimization by boxconstrained L-BFGS.
The attack of Carlini & Wagner (2017), denoted by C&W in the sequel, pertains to this approach.
A change of variable eliminates the box constraint, replacing y 2 X by (cid:27)(w) , where w 2 R n and (cid:27) is the element-wise sigmoid function.
The classiﬁcation loss encourages the logit log p t to be less than any other log p k for k
= t by at least margin m (cid:21) 0 ,
[Equation]
(11) where [ (cid:1) ]
+ denotes the positive part.
This function is similar to the multi-class SVM loss by Crammer and Singer (Crammer & Singer, 2001), where m = 1 , and, apart from the margin, it is a hard version of negative cross-entropy ‘ where softmax is producing the classiﬁer probabilities.
It does not have the problem of being ﬂat in the region of class t .
The C&W attack uses the Adam optimizer (Kingma & Ba, 2015) to minimize the cost function
[Equation]
for w 2 R n .
When the margin is reached, loss ‘ m vanishes and the distortion term pulls (cid:27)(w) back towards x , causing oscillations around the margin.
This is repeated for different (cid:21) 2 by line search, which is expensive.
Decoupling direction and norm (DDN) (Rony et al., 2018) is iterating similarly to PGD 2 (9),
[Equation] [Equation]
We
[Equation]
but projection is to the sphere S[ x ; (cid:26) i ] of radius (cid:26) i and center x , and the radius is adapted to the current distortion:
It is set to (cid:26) i =
(1 (cid:0) (cid:13))
ky i (cid:0) xk if y i is adversarial and to (1 + (cid:13))
ky i (cid:0) xk otherwise, where (cid:13) 2 (0; 1) is a parameter.
Another major difference is that each iteration is concluded by a projection onto X (rather than ^ X ) by element-wise clipping to [0; 1] and rounding .
Discussion.
Optimizing around the class boundary is not a new idea.
All of the above attacks do so in order to minimize distortion; implicitly, even attacks targeting distortion like PGD 2 do so, if the minimum parameter (cid:15) is sought ( cf . Figure 1(a) and Section 4.2).
Even black-box attacks do so (Brendel et al., 2018), without having access to the gradient function.
The difference of our attack is that our updates are along the class boundary, i.e ., in a direction normal to the gradient.
DeepFool (Moosavi-Dezfooli et al., 2016b) is a popular attack targeting success, that is not optimizing for distortion and not following a path around the class boundary.
2. O THER RELATED WORK
Optimization on manifolds.
In the context of deep learning, stochastic gradient descent on Riemanian manifolds has been studied, e.g . RSGD (Bonnabel, 2013) and RSVRG (Zhang et al., 2016).
It is usually applied to manifolds whose geometry is known in analytic form, for instance Grassmann manifolds (Bonnabel, 2013), optimizing orthogonal matrices on Stiefel manifolds (Harandi &
Fernando, 2016) or embedding trees on the Poincar´e ball (Nickel & Kiela, 2017).
In most cases, the motivation is to optimize a very complex function ( e.g . a classiﬁcation loss) on a well-studied manifold, e.g . matrix manifold (Absil et al., 2009).
On the contrary, we are optimizing a very simple quadratic function (the distortion) on a complex manifold not known in analytic form, i.e . a level set of the classiﬁcation loss.
M ETHOD
Our attack is an iterative process with a ﬁxed number K of iterations.
Stage 1 aims at quickly producing an adversarial image, whereas Stage 2 is a reﬁnement phase decreasing distortion.
The key property of our method is that while in the adversarial region during reﬁnement, it tries to walk along the classiﬁcation boundary by projecting the distortion gradient onto the tangent hyperplane of the boundary.
Hence we call it boundary projection (BP).
3. S TAGE
This stage begins at y
= x and iteratively updates in the direction of the gradient of the loss function as summarized in Algorithm 1.
The gradient is normalized and then scaled by two parameters: a ﬁxed parameter (cid:11) that is large s.t., with high probability, Stage 1 returns an adversarial image y i 2 ^ X ; and a parameter (cid:13) i that is increasing linearly with iteration i as follows
[Equation]
(14) such that updates are slow at the beginning to keep distortion low, then faster until the attack succeeds, where (cid:13) min 2 (0; (cid:13) max ) and (cid:13) max = 1 .
Clipping is element-wise.
)) 4: y i+ clip [0;1] (y i (cid:0) (cid:11)(cid:13) i ^g) 5:
i i +6: end while
3. S TAGE
Once Stage 1 has succeeded, Stage 2 continues by considering two cases: if y i is adversarial, case
OUT aims at minimizing distortion while staying in the adversarial region.
Otherwise, case IN aims at decreasing the loss while controlling the distortion.
Both work with a ﬁrst order approximation of the loss around y i :
[Equation]
) where g = r x ‘(f( y i ); t) .
The perturbation at iteration i is (cid:14) i :=
y i (cid:0) x .
Stage 2 is summarized in
Algorithm 2.
Cases OUT and IN illustrated in Fig.
2 are explained below.
Input:
Input
Output: y K 1: while i < K do 2: (cid:14) i y i (cid:0) x . perturbation 3: ^ g (cid:17)( r x ‘(f( y i ); t)) .
direction 4: r h (cid:14) i ; ^ g i 5: if (cid:25)(y i ) 6 = t then .
OUT 6: (cid:15) (cid:13)
i k (cid:14) i k .
target distortion 7: v ? x + r^g 8: z v ?
+ (cid:17)(y i (cid:0) v ? )
p [(cid:15) 2 (cid:0) r 2 ] + 9: y i+
Q OUT (z; y i ) 10: else .
IN 11: (cid:15) k (cid:14) i k =(cid:13) i .
target distortion 12: z y i (cid:0) (cid:18)r + q(cid:15) 2 (cid:0) [Equation] (cid:14) [Equation]
+ [Equation] (cid:19)
[Equation]
13: y i+
Q IN (z; y i ) 14: end if 15: i
i +16: end while
Case OUT takes as input y i outside class t region, i.e .
(cid:25)(y i ) 6 = t .
We set a target distortion (cid:15) =
(cid:13)
i k (cid:14) i k < k (cid:14) i k (14) such that updates decelerate to convergence once the attack has already succeeded.
We then solve the following problem:
[Equation]
where P := f v 2 R n : h v (cid:0) y i ; ^g i =
g is the tangent hyperplane of the level set of the loss at y i , normal to ^ g .
The constraint v 2 P aims at maintaining the value of the loss, up to the ﬁrst order.
On this hyperplane, V is the set of points having distortion close to (cid:15) .
Consider the projection v ? :
= x + r^g of x onto that hyperplane, where r := h (cid:14) i ; ^g i .
If r (cid:21) (cid:15) , then
V = f v ?
g , and the solution of (16) is trivially z = v ? .
Note that v ?
=
y i if (cid:14) i ; ^g are collinear.
If r < (cid:15) , there is an inﬁnity of solutions to (17).
We pick the one closest to y i :
[Equation]
This case is illustrated in Fig. 2(a), where V is a circle that is the intersection of sphere S[x; (cid:15)] and
P ; then z is the intersection of V and the line through y i and v ? .
Directly quantizing vector z onto X by Q( (cid:1) ) , the component-wise rounding, modiﬁes its norm (see
App.
A).
This pulls down our effort to control the distortion.
Instead, the process Q OUT (z; y i ) in line 9 looks for the scale (cid:12) of the perturbation to be applied s.t.
k Q(y i + (cid:12)(z (cid:0) y i )) k
= k z k .
This is done with a simple line search over (cid:12) .
Case IN takes as input y
i inside class t region, i.e . (cid:25)(y i ) =
t .
We set a target distortion (cid:15) =
k(cid:14)
i k =(cid:13)
i > k(cid:14)
i k (14) such that updates decelerate as in Case OUT .
We then solve the problem:
[Equation] [Equation]
g
[Equation]
[Figure]
i.e .,
ﬁnd the point z at the intersection of sphere S[ x ; (cid:15)] and the ray through y i in the direction opposite of g as shown in Fig.
2(b).
The solution is simple:
[Equation]
Vector z moves away from y i along direction (cid:0) ^g by a step size so to reach S[x; (cid:15)] .
Case IN is not guaranteed to succeed, but invoking it means that Stage 1 has succeeded.
Again a direct rounding jeopardizes the norm of the update z (cid:0)
y i .
Especially, quantization likely results in Q(z) =
Q(y i ) if k z (cid:0) y i k < (cid:12) min = 0:
(see App.
A).
Instead of a line search as in method
OUT , line 13 just makes sure that this event will not happen: Q IN (z; y i ) =
Q(y i + (cid:12)(z (cid:0) y i )) with (cid:12) = max(1; (cid:12) min =
k z (cid:0) y i k ) .
E XPERIMENTS
In this section we compare our method boundary projection (BP) to the attacks presented in Sect. 2, namely: FGSM (Goodfellow et al., 2014), I-FGSM (Kurakin et al., 2016), PGD 2 (9), C&W (Carlini & Wagner, 2017), and DDN (Rony et al., 2018).
This benchmark is carried out on three well-known datasets, with a different neural network for each.
4. D ATASETS , NETWORKS , AND PARAMETERS
For the target distortion attacks i.e . FGSM, I-FGSM and PGD 2 , we test a set of (cid:15) and calculate P suc and D according to our evaluation protocol ( cf . section 4.2).
For C&W, we test several parameter settings and pick up the optimum setting as speciﬁed below.
For DDN, the parameter settings are the default (Rony et al., 2018), i.e . (cid:15)
= 1:0 and (cid:13) = 0:05 .
Below we specify different networks and parameters for each dataset.
MNIST (LeCun et al., 2010).
We use is a simple network with three convolutional layers and one fully connected layer achieving accuracy 0:99 , referred to as C4.
The ﬁrst convolutional layer has 64 features, kernel of size 8 and stride 2 ; the second has 128 features, kernel 6 and stride 2 ; the third has also 128 features, but kernel 5 and stride 1 .
It uses LeakyRelu activation (Maas et al., 2013).
Parameters .
We set (cid:11) =
0:08 for I-FGSM and (cid:11) =
(cid:15)=2 for PGD 2 .
For C&W: for 5 (cid:2) 20 iterations 3 , learning rate (cid:17) = 0:5 and initial constant (cid:21) = 1:0 ; for 1 (cid:2) 100 iterations, (cid:17)
= 0:1 and (cid:21)
= 10:0 .
CIFAR10 (Krizhevsky & Hinton, 2009).
We use a simple CNN network with nine convolutional layers, two max-pooling layers, ending in global average pooling and a fully connected layer.
Its accuracy is 0:927 .
Batch normalization (Ioffe & Szegedy, 2015) is applied after every convolutional layer.
It also uses LeakyRelu.
Parameters .
We set (cid:11) =
0:08 for I-FGSM and (cid:11)
[Table]
ImageNet (Kurakin et al., 2018) comprises 1,000 images from ImageNet (Deng et al., 2009).
We use InceptionV3 (Szegedy et al., 2016) whose accuracy is 0:96 .
Parameters .
We set (cid:11) =
0:08 for I-FGSM and (cid:11) = 3 for PGD 2 .
For C&W: for 5 (cid:2) 20 iterations, learning rate (cid:17) = 0:01 and initial constant (cid:21) = 20 ; for 1 (cid:2) 100 iterations, (cid:17) = 0:01 and (cid:21) = 1:0 .
E VALUATION PROTOCOL
We evaluate an attack by its runtime, two global statistics P suc and D , and by an operating characteristic curve D ! P(D) measuring distortion vs .
probability of success as described below.
Since we focus on the speed-distortion trade-off , we measure the required time for all attacks.
For the iterative attacks, the complexity of one iteration is largely dominated by the computation of the gradient, which requires one forward and one backward pass through the network.
It is thus fair to gauge their complexity by this number, referred to as iterations or ‘# Grads’.
Indeed, the actual timings of 100 iterations for I-FGSM, PGD 2 , C&W, DDN and BP are 1.08, 1.36, 1.53, 1.46 and 1.17 s/image on average respectively on ImageNet, using Tensorﬂow, Cleverhans implementation for I-FGSM and C&W, and authors implementation for DDN.
We measure distortion when the adversarial images are quantized by rounding each element to the nearest element in X .
This makes sense since adversarial images are meant to be stored or communicated as images rather than real-valued matrices.
DDN and BP adversarial images are already quantized.
For reference, we report distortion without quantization in Appendix B.3.
Given a test set of N 0 images, we only consider its subset X of N images that are classiﬁed correctly without attack.
The accuracy of the classiﬁer is N=N 0 .
Let X suc be the subset of X with N suc :=
j X suc j where the attack succeeds and let D(x) :
= k x (cid:0)
y k be the distortion for image x 2 X suc .
The global statistics are the success probability P suc and conditional average distortion D
[Equation]
Here, D is conditioned on success.
Indeed, distortion makes no sense for a failure.
We deﬁne the operating characteristic of a given attack over the set X as the function P : [0; D max ] !
[0; 1] , where D max := max x2X suc D(x) .
Given D 2 [0; D max ] , P (D) is the probability of success subject to distortion being upper bounded by D ,
[Equation]
This function increases from P (0)
=
to P (D max )
= P suc .
We sample one intermediate point:
P upp :=
P (D upp ) is the success rate within a distortion upper bounded by D upp 2 (0; D max ) .
It is difﬁcult to deﬁne a fair comparison of distortion targeting attacks to success targeting attacks (see section 2.2).
For the ﬁrst family, we run a given attack several times over the test set with different target distortion (cid:15) .
The attack succeeds on image x 2 X if it succeeds on at least one of the runs, and the distortion D(x) is the minimum distortion over all successful runs.
All statistics are then evaluated as above. MNIST
CIFAR
ImageNet Attack # Grads P suc D P upp
P suc
D
P upp
P suc
D
P upp
FGSM 0.
I-FGSM 1. [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
PGD
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
C&W 5 (cid:2)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
DDN
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
BP (this work)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
D
P s u c    0 0 :0:40: 0:
D
P s u c    0 0 :0:40: 0:
D
P s u c
C&W DDN
FGSM I-FGSM
PGD 2 BP
[Figure]
4. Q UANTIZATION
Before addressing the benchmark, Table 1 shows the critical role of quantization in our method
BP.
Since this attack is iterative and works with continuous vectors, one may quantize only at the end of the process, or at the end of each iteration.
Another option is to anticipate the detrimental action of quantizing by adapting the length of each step accordingly, as done by Q IN ( (cid:1) ) and Q OUT ( (cid:1) ) in Algorithm 2.
The experimental results show that the key is to quantize often so to let the next iterations compensate.
Anticipating and adapting gives a substantial extra improvement.
4. A TTACK EVALUATION
Table 2 summarizes the global statistics of the benchmark.
Fig.
3 offers a more detailed view per dataset with operating characteristic plots.
In terms of average distortion, all iterative attacks perform much better than the single-step FGSM.
The performances of C&W are on par with those of I-FGSM, which is unexpected for this more elaborated attack design.
The reason is that C&W is put under stress in our benchmark.
It usually requires a bigger number of iterations to deliver high quality images.
Note that it is possible to avoid the line search on parameter (cid:21) as shown in row 1 (cid:2) 100 .
However, it requires a ﬁne tuning so that this single value works over all the images of the dataset.
This is not possible for ImageNet.
DDN and our method BP are clearly ahead of the benchmark.
DDN yields lower distortion on
MNIST at fewer iterations, but its probability of success is not satisfying.
DDN is indeed better than BP only on CIFAR10 at 100 iterations.
Fig.
3 reveals that the two attacks have similar operating characteristic on all datasets but this is because it refers to 100 iterations.
In terms of success rate, FGSM fails on MNIST; on CIFAR10, I-FGSM and PGD 2 fail as well; ﬁnally on ImageNet, C&W fails too.
DDN also fails on ImageNet at 20 iterations.
Increasing the number of iterations helps but not at the same rate for all the attacks.
For instance, going from 20 to 100 iterations is waste of time for I-FGSM while it is essential for decreasing the distortion of DDN or making PGD 2 efﬁcient on ImageNet.
Most importantly, our attack BP brings a dramatic improvement in the speed vs .
distortion trade-off.
Just within 20 iterations, the [Table] distortion achieved on ImageNet is very low compared to the others.
Appendix B.2 shows the speed vs .
distortion trade-off in more detail.
Statistics of BP stages are as follows:
On CIFAR-10 and MNIST, Stage 1 takes 7 iterations on average.
On ImageNet, Stage 1 takes on average 3 iterations out of 20, or 8 iterations out of 100.
Appendix C shows examples of images along with corresponding adversarial examples and perturbations for different methods.
4. D EFENSE EVALUATION WITH ADVERSARIAL TRAINING
We also test under adversarial training (Goodfellow et al., 2014).
The network is re-trained with a dataset composed of the original training set and the corresponding adversarial images.
This training is special: at the end of each epoch, the network is updated and ﬁxed, then the adversarial images for this new update are forged by some reference attack, and the next epoch starts with this new set.
This is tractable only if the reference attack is fast.
We use it with FGSM as the reference attack.
It is more interesting to study DDN and BP as alternatives to FGSM:
at 20 iterations, they are fast enough to play the role of the reference attack in adversarial training.
In this case, we follow the training process suggested by Rony et al.
(2018): the model is ﬁrst trained on clean examples, then ﬁne-tuned for 30 iterations with adversarial examples.
As shown in Table 3, DDN and BP perform equally better than FGSM on CIFAR10, in terms of either average distortion or success rate.
Among the reliable attacks ( i.e . whose P suc is close to 1), the worst attack now requires a distortion three times larger than the distortion of the worst attack without defense.
In the same way, on MNIST, the distortion of the worst case attack doubles going from 1:37 (baseline) to 2:73 (BP defense).
In most cases, BP is a better defense than DDN, forcing the attacker to have 20% more distortion.
Note that for a given defense, the strongest attack is almost always BP.
D ISCUSSION
The main idea of BP is to travel on the manifold deﬁned by the class boundary while seeking to minimize distortion.
This travel is operated by the reﬁnement stage, which alternates on both sides of the boundary, but attempts to stay mostly in the adversarial region.
Referring to section 2.1, BP is in effect doing for the target success problem what PGD 2 is doing for the target distortion problem: BP minimizes distortion on the class boundary manifold (a level set of the classiﬁcation loss), while PGD 2 minimizes the classiﬁcation loss on a sphere (a level set of the distortion).
BP also takes into account the detrimental effect of quantization .
By doing so, the amplitude of the perturbation is controlled from one iteration to another.
The main advantage of our attack is the small number of iterations required to achieve both reliability (probability of success close to one) and high quality (low average distortion).
A P REDICTING DISTORTION AFTER QUANTIZATION
This appendix aims at predicting the norm of the update after quantization, assuming that it is independent from the computation of the perturbation.
Iteration i starts with a quantized image y i 2 X , adds update u 2 R n , and then quantizes s.t.
y
i+
=
Q( y i + u ) .
Quantization is done by rounding with (cid:1) :
= 1=(L (cid:0) 1) the quantization step.
Pixel j is quantized to y i+1;j = y i;j + e j (23) for some e j 2 (cid:1)
Z such that u j 2 (e j (cid:0) (cid:1)=2; e j + (cid:1)=2] .
Border effects where y i;j + e j = 2 X are neglected.
We now take a statistical point of view where the update is modelled by a random vector U uniformly distributed over the hypersphere of radius (cid:26) .
That parameter (cid:26) is the norm of the perturbation before quantization.
This yields random quantization values, denoted by E j 2 (cid:1) Z for pixel j .
The distortion between the two images is
[Equation]
A common approach in source coding theory is the additive noise model for quantization error in the high resolution regime (Gersho & Gray, 1991).
It states that E j = U j + Q j where Q j( (cid:0) (cid:1)=2; (cid:1)=2] is the quantization error.
When (cid:26) (cid:29) (cid:1) , then Q j becomes uniformly distributed (s.t.
E(Q j )
= 0 and E(Q 2j ) =
(cid:1) 2 =12 ) and independent of U j (s.t. E(U j Q j ) =
E(U j )E(Q j ) = 0 ).
Under these assumptions, Eq.
24 simpliﬁes in expectation to:
[Equation]
This shows that quantization increases the distortion on expectation.
Yet, this simple analysis is wrong outside the high resolution regime, and we need to be more careful.
The expectation of a sum is always the sum of the expectations, whatever the dependence between the summands: E(D 2 )
[Equation]
We need the distribution of E j to compute the expected distortion after quantizarion.
This random variable E j takes a value depending on the scalar product S j :=
U > c j , where c j is the j -th canonical vector.
This scalar product lies in [ (cid:0) (cid:26); (cid:26)] , so that P(E j (cid:21) ‘(cid:1))
=
if ‘(cid:1) (cid:0) (cid:1)=2 > k (cid:26) k .
Otherwise, E j (cid:21) ‘(cid:1) when jS j j (cid:21) ‘(cid:1)(cid:0)(cid:1)=2 , which happens when U lies inside the dual hypercone of axis c j and semi-angle (cid:18)(‘) =
arccos(s(‘)) with s(‘) :=
(2‘ (cid:0) 1)(cid:1)=2 k (cid:26) k .
The probability of this event is equal to the ratio of the solid angles of this dual hypercone and the full space R n .
This quantity can be expressed via the incomplete regularized beta function I , and approximately equals 2(cid:8)( p ns(‘)=2) for large n .
In the end, 8 ‘ 2 f 0; : : : ; L (cid:0) 1 g , P [Equation] j E
[Equation]
j
[Equation]
‘(cid:1)
[Equation]
[Equation]
[Equation] if 0 (cid:20) [Equation] (cid:20)0; otherwise
Computing E(D 2 ) is now possible because P( j E j j = ‘(cid:1))
=
P( j E j j (cid:21) ‘(cid:1))
(cid:0) P( j E j j (cid:21) (‘ + 1)(cid:1)) .
This expected distortion after quantization depends on (cid:1) , n , and (cid:26) the norm of the perturbation before quantization.
Figure 4 shows that quantization reduces the distortion outside the high resolution regime.
Indeed, p E(D 2 ) is close to 0 for (cid:26) < 0:1 when n = 3 (cid:3) 299 2 ( i.e . ImageNet).
When the update has a small norm (cid:26) , quantization is likely to kill it, y
i+1 =
y i , and we waste one iteration.
On the contrary, p E(D 2 ) converges to p (cid:26) 2 + n(cid:1) 2 =12 for large (cid:26) ( i.e .
in the high resolution regime).
Note that the ratio of the distortions before and after quantization (cid:26)= p E(D 2 ) quickly converges to 1 for large (cid:26) .
[Figure]
B A DDITIONAL EXPERIMENTS B. P ARAMETER S TUDY
There are two parameters in BP: (cid:11) and (cid:13) min .
Both determine the step size of stage 1, while (cid:13) min also determines the step size of stage 2.
We consider 4 values for (cid:11) , i.e . 1; 2; 3; 4 and 9 values for (cid:13) min , i.e . 0:1; 0:2; :::; 0:9 .
For each pair of values, we evaluate BP with 20 iterations on a validation set, which we deﬁne as a random subset sampled of the training set: 10000 images for MNIST and
CIFAR10, and 1000 images for ImageNet. As shown in Fig. 5, success probability is close to one in all cases, while average distortion is in general stable up to (cid:13) min = 0:8 . We choose (cid:11) =
2 and (cid:13) min = 0:7 for all experiments.
[Figure]
D
[Figure]
D
[Figure]
[Figure]
[Figure]
[Figure]
(cid:13)
[Figure] D (cid:11) [Figure] (cid:11) =(cid:11)
[Figure]
(cid:11)
[Figure] [Figure] : 0: 0: 0:  (cid:13) min P s u c 0:2 0:4 0:6 0:8 0: 0: 0: 0:  (cid:13) min
P s u c 0:2 0:4 0:6 0:8 0: 0: 0: 0:  (cid:13) min
P s u c (cid:11) =(cid:11) =(cid:11) =(cid:11) =
[Figure]
Figure 6(a) is a graphical view of some results reported in Table 2 with more choices of number of iterations between 20 and 100, and only for ImageNet where our performance gain is the most signiﬁcant.
Just within 20 iterations, its distortion D is already so much lower than that of other attacks, that its decrease (-20% at 100 iterations) is not visible in Fig.
On the contrary, more iterations are useless for I-FGSM, and PGD 2 can achieve low distortion only with a number of iterations bigger than 50.
Figure 6(b) conﬁrms that the probability of success is close to 1 for both
DDN and BP for the numbers of iterations considered.
[Figure]
B. A TTACK EVALUATION WITHOUT QUANTIZATION
Table 4 is the equivalent of Table 2 but without the integral constraint: the attack is free to output any real matrix provided that the pixel values all belong to [0; 1] .
When the distortion is large, there is almost no difference.
The model of App.
A explains this: We are in the high resolution regime and the extra term in Eq.
25 is negligible compared to the perturbation distortion before quantization.
This is especially true when the number of samples n is small ( i.e . MNIST, and to some extend,
CIFAR-10).
When an attack delivers low distortion on average with real matrices, the quantization may lower the probability of success.
This is especially true with the iterative attacks ﬁnding adversarial examples just nearby the border between the two classes.
Quantization jeopardizes this point and sometimes brings it back in the true class region.
More importantly, the impact of the quantization on the distortion is no longer negligible.
This is clearly visible when comparing Table 4 and Table 2 for
DDN and BP over ImageNet.
Similarly, Fig.
7 is the equivalent of Fig. 3 without the integral constraint. By comparing the two ﬁgures, it can be seen that PGD 2 and C&W, but also DDN and BP, are improving on ImageNet by having signiﬁcantly lower distortion.
This agrees with measurements of success rate in Table 4, where PGD 2 and C&W are not failing as they do in Table 2 with quantization.
Our BP is still the strongest attack over all datasets
[Table] D
P s u c    0 0: 0:40: 0:
D P s u c    0 0: 0:40: 0: D P s u c
C&W DDN
FGSM I-FGSM
PGD 2 BP
[Figure]
[Figure] A TTACK EVALUATION ON ROBUST MODELS
Table 5 is similar to Table 2 but is evaluating attacks on robust models. In particular, on MNIST and CIFAR10, we use the same models as described in Section 4.1, which we adversarially train according to Madry et al. On ImageNet, we use off-the shelf 4 InceptionV3 obtained by ensemble adversarial training on four models Tram`er et al. (2017a). In general, DDN and BP outperform all other attacks in terms of either average distortion D or success rate P upp .
On ImageNet in particular, all other attacks have signiﬁcantly higher distortion and fail in terms of success rate. DDN and BP have similar performance on CIFAR10. On MNIST, DDN fails in terms of probability of success at 20 iterations, while at 100 iterations BP is superior.
On ImageNet, DDN has signiﬁcantly greater distortion than BP and fails in terms of success rate at 20 iterations, while at 100 iterations BP still has lower distortion. Fig. 8 is showing a more detailed view of operating characteristics, similarly to Fig. 3 for models trained on natural images.
We can see that BP is still ahead of the competition.
It is close to DDN, but this is because Fig.
refers to 100 iterations. The two attacks outperform all others by a large margin. [Table] [Table]
D
P s u c    0 0: 0:40: 0:
D
P s u c    0 0: 0:40: 0:
D
P s u c
C&W DDN
FGSM I-FGSM
PGD 2 BP
[Figure]
C A DVERSARIAL IMAGE EXAMPLES
Fig.
9 shows the worst-case ImageNet example for BP along with the adversarial examples generated by all methods and the corresponding normalized perturbations.
FGSM has the highest distortion over all methods in this example and BP the lowest.
DDN has the highest 1 -norm distortion.
Observe that for no method is the perturbation visible, although this is a worst-case example.
original image
FGSM: I-FGSM:
PGD 2 :
C&W:
DDN:
BP: D =6.
D
= 5.
D = 3.
D =1.
D
=2.
D =1.
D =6.
D = 4.
D = 3.
D =1.
D
=2.
D =0.
D =6.
D
= 5.
D = 3.
D
=2.
D =1.
D =0.
D =6.
D = 4.
D = 3.
D
=2.
D =1.
D =0.
D =6.
D
= 5.
D = 3.
D =1. D =1.
D =0.