AD EF : AN I TERATIVE A LGORITHM TO C ONSTRUCT A DVERSARIAL D EFORMATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
While deep neural networks have proven to be a powerful tool for many recognition and classiﬁcation tasks, their stability properties are still not well understood.
In the past, image classiﬁers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classiﬁed image.
In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step.
We demonstrate our results on
MNIST with convolutional neural networks and on ImageNet with Inception-v and ResNet-101.
I NTRODUCTION
In a ﬁrst observation in Szegedy et al.
(2013) it was found that deep neural networks exhibit unstable behavior to small perturbations in the input.
For the task of image classiﬁcation this means that two visually indistinguishable images may have very different outputs, resulting in one of them being misclassiﬁed even if the other one is correctly classiﬁed with high conﬁdence.
Since then, a lot of research has been done to investigate this issue through the construction of adversarial examples: given a correctly classiﬁed image x , we look for an image y which is visually indistinguishable from x but is misclassiﬁed by the network.
Typically, the image y is constructed as y
= x + r , where r is an adversarial perturbation that is supposed to be small in a suitable sense (normally, with respect to an (cid:96) p norm).
Several algorithms have been developed to construct adversarial perturbations, see Goodfellow et al.
(2014); Moosavi Dezfooli et al.
; Kurakin et al.
(2017b)
; Madry et al.
; Carlini & Wagner (2017b) and the review paper Akhtar & Mian (2018).
Even though such pathological cases are very unlikely to occur in practice, their existence is relevant since malicious attackers may exploit this drawback to fool classiﬁers or other automatic systems.
Further, adversarial perturbations may be constructed in a black-box setting (i.e., without knowing the architecture of the DNN but only its outputs)
(Papernot et al., 2017; Moosavi-Dezfooli et al., 2017) and also in the physical world (Kurakin et al., 2017b; Athalye & Sutskever, 2017; Brown et al., 2017; Sharif et al., 2016).
This has motivated the investigation of defenses, i.e., how to make the network invulnerable to such attacks, see Kurakin et al.
(2017a)
; Carlini & Wagner (2017a); Madry et al.
; Tram`er et al.
; Wong & Kolter (2018); Raghunathan et al.
; Athalye et al.
; Kannan et al.
In most cases, adversarial examples are artiﬁcially created and then used to retrain the network, which becomes more stable under these types of perturbations.
Most of the work on the construction of adversarial examples and on the design of defense strategies has been conducted in the context of small perturbations r measured in the (cid:96) ∞ norm.
However, this is not necessarily a good measure of image similarity: e.g., for two translated images x and y , the norm of x − y is not small in general, even though x and y will look indistinguishable if the translation is small.
Several papers have investigated the construction of adversarial perturbations not designed for norm proximity (Rozsa et al., 2016;
Sharif et al., 2016; Brown et al., 2017; Engstrom et al., 2017;
Xiao et al., 2018).
In this work, we build up on these ideas and investigate the construction of adversarial deformations .
In other words, the misclassiﬁed image y is not constructed as an additive perturbation
y = x
+ r , but as a deformation y = x ◦ (id + τ ) , where τ is a vector ﬁeld deﬁning the transformation.
In this case, the similarity is not measured through a norm of y − x , but instead through a norm of τ , which quantiﬁes the deformation between y and x .
We develop an efﬁcient algorithm for the construction of adversarial deformations, which we call
ADef.
It is based on the main ideas of DeepFool (Moosavi Dezfooli et al., 2016), and iteratively constructs the smallest deformation to misclassify the image.
We test the procedure on MNIST (LeCun) (with convolutional neural networks) and on ImageNet (Russakovsky et al., 2015) (with
Inception-v3 (Szegedy et al., 2016) and ResNet-101 (He et al., 2016)).
The results show that ADef can succesfully fool the classiﬁers in the vast majority of cases (around 99%) by using very small and imperceptible deformations.
We also test our adversarial attacks on adversarially trained networks for MNIST.
The results of this work have initially appeared in the master’s thesis Anonymous (2017), to which we refer for additional details on the mathematical aspects of this construction 1 .
While writing this paper, we have come across Xiao et al.
(2018), in which a similar problem is considered and solved with a different algorithm.
Whereas in Xiao et al.
the authors use a second order solver to ﬁnd a deforming vector ﬁeld, we show how a ﬁrst order method can be formulated efﬁciently and justify a smoothing operation, independent of the optimization step.
We report, for the ﬁrst time, success rates for adversarial attacks with deformations on ImageNet.
The topic of deformations has also come up in Jaderberg et al.
(2015), in which the authors introduce a class of learnable modules that deform inputs in order to increase the performance of existing DNNs.
A DVERSARIAL DEFORMATIONS 2. A DVERSARIAL PERTURBATIONS
Let K be a classiﬁer of images consisting of P pixels into L ≥ 2 categories, i.e. a function from the space of images X = R cP , where c = 1 (for grayscale images) or c = 3 (for color images), and into the set of labels L = { 1 , . . .
, L } .
Suppose x ∈ X is an image that is correctly classiﬁed by K and suppose y ∈ X is another image that is imperceptible from x and such that K ( y ) (cid:54) =
K ( x ) , then y is said to be an adversarial example .
The meaning of imperceptibility varies, but generally, proximity in (cid:96) p -norm (with 1 ≤ p ≤ ∞ ) is considered to be a sufﬁcient substitute.
Thus, an adversarial perturbation for an image x ∈ X is
a vector r ∈ X such that K ( x + r ) (cid:54) =
K ( x ) and (cid:107) r (cid:107) p is small, where
[Equation]
Given such a classiﬁer K and an image x , an adversary may attempt to ﬁnd an adversarial example y by minimizing (cid:107) x − y (cid:107) p subject to K ( y ) (cid:54)
=
K ( x ) , or even subject to K ( y ) =
k for some target label k
(cid:54) =
K ( x ) .
Different methods for ﬁnding minimal adversarial perturbations have been proposed, most notably FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2018) for (cid:96) ∞ , and the
DeepFool algorithm (Moosavi Dezfooli et al., 2016) for general (cid:96) p -norms.
2. D EFORMATIONS
Instead of constructing adversarial perturbations, we intend to fool the classiﬁer by small deformations of correctly classiﬁed images.
Our procedure is in the spirit of the DeepFool algorithm.
Before we explain it, let us ﬁrst clarify what we mean by a deformation of an image.
The discussion is at ﬁrst more intuitive if we model images as functions ξ : [0 , 1] 2 → R c (with c = 1 or c = 3 ) instead of discrete vectors x in R cP .
In this setting, perturbing an image ξ : [0 , 1] 2 → R c corresponds to adding to it another function ρ : [0 , 1] 2 → R c with a small L p -norm.
While any transformation of an image ξ can be written as a perturbation ξ + ρ , we shall restrict ourselves to a particular class of transformations.
A deformation with respect to a vector ﬁeld τ : [0 , 1] 2 → R 2 is a transformation of the form ξ (cid:55)→ ξ τ , where for any image ξ : [0 , 1] 2 → R c , the image ξ τ : [0 , 1] 2 → R c is deﬁned by
[Equation]
extending ξ by zero outside of [0 , 1] 2 .
Deformations capture many natural image transformations.
For example, a translation of the image ξ by a vector v ∈ R 2 is a deformation with respect to the
Original ‘ ∞ : 1.
Translation by ( − 2 , 1) ‘ ∞ : 0.
Rotation by 10 ◦ ‘ ∞ : 1.
Deformation w.r.t.
τ
T : 2.
T : 3.
T : 1.
constant vector ﬁeld τ = v .
If v is small, the images ξ and ξ v may look similar, but the corresponding perturbation ρ = ξ v − ξ may be arbitrarily large in the aforementioned L p -norms.
Figure 1 shows three minor deformations, all of which yield large L ∞ -norms.
In the discrete setting, deformations are implemented as follows.
We consider square images of
W × W pixels and deﬁne the space of images to be X
= (cid:0) R W × W (cid:1) c .
A discrete vector ﬁeld is a function τ : { 1 , . . .
, W } 2 → R 2 .
In what follows we will only consider the set T of vector ﬁelds that do not move points on the grid { 1 , . . .
, W } 2 outside of [1 , W ] 2 .
More precisely,
T := (cid:8) τ : { 1 , . . .
, W } 2 → R 2 | τ ( s, t ) + ( s, t ) ∈
[1 , W ] 2 for all s, t ∈ { 1 , . . .
, W } (cid:9) .
An image x ∈ X can be viewed as the collection of values of a function ξ : [0 , 1] 2 → R c on a regular grid { 1 / ( W +1) , . . .
, W / ( W +1) } 2 ⊆
[0 , 1] 2 , i.e. x s,t = ξ ( s / ( W +1) , t / ( W +1) )
for s, t = 1 , . . .
, W .
Such a function ξ can be computed by interpolating from x .
Thus, the deformation of an image x with respect to the discrete vector ﬁeld τ can be deﬁned as the discrete deformed image x τ in X by
[Equation]
It is not straightforward to measure the size of a deformation such that it captures the visual difference between the original image x and its deformed counterpart x τ .
We will use the size of the corresponding vector ﬁeld, τ , in the norm deﬁned by
[Equation]
as a proxy.
The (cid:96) p -norms deﬁned in (1), adapted to vector ﬁelds, can be used as well.
(We remark, however, that none of these norms deﬁne a distance between x and x τ , since two vector ﬁelds τ, σ ∈ T with (cid:107) τ (cid:107) T (cid:54) =
(cid:107) σ (cid:107) T may produce the same deformed image x τ = x σ .)
2. T HE ALGORITHM AD EF
We will now describe our procedure for ﬁnding deformations that will lead a classiﬁer to yield an output different from the original label.
Let F = ( F 1 , . .
. , F L ) :
X → R L be the underlying model for the classiﬁer K , such that
[Equation]
Let x ∈
X be the image of interest and ﬁx ξ : [0 , 1] 2 → R c obtained by interpolation from x .
Let l = K ( x ) denote the true label of x , let k ∈ L be a target label and set f = F k − F l .
We assume that x does not lie on a decision boundary, so that we have f ( x ) < 0 .
We deﬁne the function g : T → R , τ (cid:55)→ f ( x τ ) and note that g (0) =
f ( x 0 ) =
f ( x ) < 0 .
Our goal is to ﬁnd a small vector ﬁeld τ ∈ T such that g ( τ ) =
f ( x τ ) ≥ 0 .
We can use a linear approximation of g around the zero vector ﬁeld as a guide:
[Equation]
for small enough τ ∈ T and D 0 g : T → R the derivative of g at τ = 0 .
Hence, if τ is a vector ﬁeld such that
[Equation]
and (cid:107) τ (cid:107) T is small, then the classiﬁer K has approximately equal conﬁdence for the deformed image x τ to have either label l or k .
This is a scalar equation with unknown in T , and so has inﬁnitely many solutions.
In order to select τ with small norm, we solve it in the least-squares sense.
In view of (2), we have ∂x
[Equation] [Equation]
× 2 .
Thus, by applying the chain rule to g ( τ ) =
f
( x τ ) , we obtain that its derivative at τ = 0 can, with a slight abuse of notation, be identiﬁed with the vector ﬁeld
[Equation]
where (cid:0) ∇ f ( x ) (cid:1) s,t ∈ R 1 × c is the derivative of f in x calculated at ( s, t ) .
With this, (D 0 g ) τ stands for (cid:80) Ws,t =1 D 0 g ( s, t ) · τ ( s, t ) , and the solution to (5) in the least-square sense is given by
[Equation]
Finally, we deﬁne the deformed image x τ ∈ X according to (2).
One might like to impose some degree of smoothness on the deforming vector ﬁeld.
In fact, it sufﬁces to search in the range of a smoothing operator S : T → T .
However, this essentially amounts to applying S to the solution from the larger search space T .
Let α = S
(D 0 g ) =
ϕ ∗ (D 0 g ) , where
S denotes the componentwise application of a two-dimensional Gaussian ﬁlter ϕ (of any standard deviation).
Then the vector ﬁeld
[Equation]
also satisﬁes (5), since S is self-adjoint.
We can hence replace τ by ˜ τ to obtain a smooth deformation of the image x .
We iterate the deformation process until the deformed image is misclassiﬁed.
More explicitly, let x (0) =
x and for n ≥ 1 let τ ( n ) be given by (7) for x ( n − 1) .
Then we can deﬁne the iteration as x ( n ) =
x ( n − 1) ◦
(id + τ ( n ) ) .
The algorithm terminates and outputs an adversarial example y =
x ( n ) if K ( x ( n ) ) (cid:54)
=
l .
The iteration also terminates if x ( n ) lies on a decision boundary of K , in which case we propose to introduce an overshoot factor 1+ η on the total deforming vector ﬁeld.
Provided that the number of iterations is moderate, the total vector ﬁeld can be well approximated by τ ∗ =
τ (1) + · · · + τ ( n ) and the process can be altered to output the deformed image y =
x ◦ (id+(1+ η ) τ ∗ ) instead.
The target label k may be chosen in each iteration to minimize the vector ﬁeld to obtain a better approximation in the linearization (4).
More precisely, for a candidate set of labels k 1 , . .
, k m , we compute the corresponding vectors ﬁelds τ 1 , . . .
, τ m and select
[Equation]
The candidate set consists of the labels corresponding to the indices of the m smallest entries of
F − F l , in absolute value.
Algorithm ADef
Input: Classiﬁcation model F , image x , correct label l , candidate labels k 1 , . . .
, k m
Output: Deformed image y
Initialize y ← x while
K ( y ) =
l do for j = 1 , . . .
, m do α j ← S (cid:0)(cid:80) ci
[Equation]
(cid:0)
[Equation]
τ j ← −
F
[Equation]
(cid:107)
[Equation]
S
α j end for i ← arg min j =
[Equation]
,m (cid:107) τ
[Equation]
(cid:107) T y ← y ◦
(id + τ i ) end while return y
By equation (6), given that ∇ f is moderate, the deforming vector ﬁeld takes small values wherever ξ has a small derivative.
This means that the vector ﬁeld will be concentrated on the edges in the image x
(see e.g. the ﬁrst row of ﬁgure 2).
Further, note that the result of a deformation is always a valid image in the sense that it does not violate the pixel value bounds.
This is not guaranteed for the perturbations computed with DeepFool.
E XPERIMENTS 3. S ETUP
We evaluate the performance of ADef by applying the algorithm to classiﬁers trained on the MNIST (LeCun) and ImageNet (Russakovsky et al., 2015) datasets.
Below, we brieﬂy describe the setup of the experiments and in tables 1 and 2 we summarize their results.
MNIST: We train two convolutional neural networks based on architectures that appear in Madry et al.
and
Tram ` er et al.
(2018) respectively.
The network MNIST-A consists of two convolutional layers of sizes 32 × 5 × 5 and 64 × 5 × 5 , each followed by 2 × 2 max-pooling and a rectiﬁer activation function, a fully connected layer into dimension 1024 with a rectiﬁer activation function, and a ﬁnal linear layer with output dimension 10.
The network MNIST-B consists of two convolutional layers of sizes 128 × 3 × 3 and 64 × 3 × 3 with a rectiﬁer activation function, a fully connected layer into dimension 128 with a rectiﬁer activation function, and a ﬁnal linear layer with output dimension 10.
During training, the latter convolutional layer and the former fully connected layer of MNIST-B are subject to dropout of drop probabilities 1 / 4 and 1 / 2 .
We use ADef to produce adversarial deformations of the images in the test set.
The algorithm is conﬁgured to pursue any label different from the correct label (all incorrect labels are candidate labels).
It performs smoothing by a Gaussian ﬁlter of standard deviation 1 / 2 , and it overshoots by η = 2 / 10 whenever it converges to a decision boundary.
ImageNet: We apply ADef to pretrained Inception-v3 (Szegedy et al., 2016) and ResNet-101 (He et al., 2016) models to generate adversarial deformations for the images in the ILSVRC2012 validation set.
The images are preprocessed by ﬁrst scaling so that the smaller axis has 299 pixels for the Inception model and 224 pixels for ResNet, and then they are center-cropped to a square image.
The algorithm is set to focus only on the label of second highest probability.
It employs a Gaussian ﬁlter of standard deviation 1 and an overshoot factor η = 1 / 10 .
[Figure]
[Table]
[Figure]
We only consider inputs that are correctly classiﬁed by the model in question, and, since τ ∗ =
τ (1) + · · · + τ ( n ) approximates the total deforming vector ﬁeld, we declare ADef to be successful if its output is misclassiﬁed and (cid:107) τ ∗ (cid:107) T ≤ ε , where we choose ε = 3 .
Observe that, by (3), a deformation with respect to a vector ﬁeld τ does not displace any pixel further away from its original position than (cid:107) τ (cid:107) T .
Hence, for high resolution images, the choice ε = 3 indeed produces small deformations if the vector ﬁelds are smooth.
In appendix A, we illustrate how the success rate of ADef depends on the choice of ε .
When searching for an adversarial example, one usually searches for a perturbation with (cid:96) ∞ -norm smaller than some small number ε > 0 .
Common choices of ε range from 1 / 10 to 3 / 10 for MNIST classiﬁers (Goodfellow et al., 2014; Madry et al., 2018; Wong & Kolter, 2018; Tram ` er et al., 2018;
Kannan et al., 2018) and 2 / 255 to 16 / 255 for ImageNet classiﬁers (Goodfellow et al., 2014; Kurakin et al., 2017a; Tram`er et al., 2018; Kannan et al., 2018).
Table 1 shows that on average, the perturbations obtained by ADef are quite large compared to those constraints.
However, as can be seen in ﬁgure 2, the relatively high resolution images of the ImageNet dataset can be deformed into adversarial examples that, while corresponding to large perturbations, are not visibly different from the original images.
In appendix B, we give more examples of adversarially deformed images.
[Table]
3. A DVERSARIAL TRAINING
In addition to training MNIST-A and MNIST-B on the original MNIST data, we train independent copies of the networks using the adversarial training procedure described by Madry et al.
That is, before each step of the training process, the input images are adversarially perturbed using the PGD algorithm.
This manner of training provides increased robustness against adversarial perturbations of low (cid:96) ∞ -norm.
Moreover, we train networks using ADef instead of PGD as an adversary.
In table 2 we show the results of attacking these adversarially trained networks, using ADef on the one hand, and PGD on the other.
We use the same conﬁguration for ADef as above, and for
PGD we use 40 iterations, step size 1 / 100 and 3 / 10 as the maximum (cid:96) ∞ -norm of the perturbation.
Interestingly, using these conﬁgurations, the networks trained against PGD attacks are more resistant to adversarial deformations than those trained against ADef.
3. T ARGETED ATTACKS
ADef can also be used for targeted adversarial attacks, by restricting the deformed image to have a particular target label instead of any label which yields the optimal deformation.
Figure 3 demonstrates the effect of choosing different target labels for a given MNIST image, and ﬁgure 4 shows the result of targeting the label of lowest probability for an image from the ImageNet dataset.
C ONCLUSION
In this work, we proposed a new efﬁcient algorithm, ADef, to construct a new type of adversarial attacks for DNN image classiﬁers.
The procedure is iterative and in each iteration takes a gradient descent step to deform the previous iterate in order to push to a decision boundary.
We demonstrated that with almost imperceptible deformations, state-of-the art classiﬁers can be fooled to misclassify with a high success rate of ADef.
This suggests that networks are vulnerable to different types of attacks and that simply training the network on a speciﬁc class of adversarial examples might not form a sufﬁcient defense strategy.
Given this vulnerability of neural networks to deformations, we wish to study in future work how ADef can help for designing possible defense strategies.
Furthermore, we also showed initial results on fooling adversarially trained networks.
Original: ﬁg
Deformed: grocery store
T : 0.
Vector ﬁeld ‘ ∞ : 0.
Perturbation
Original: ﬁg
Deformed: gazelle
T : 2.
Vector ﬁeld ‘ ∞ : 0.
Perturbation
Remarkably, PGD trained networks on MNIST are more resistant to adversarial deformations than
ADef trained networks.
However, for this result to be more conclusive, similar tests on ImageNet will have to be conducted.
We wish to study this in future work
A D ISTRIBUTION OF VECTOR FIELD NORMS
Figures 5 and 6 show the distribution of the norms of the total deforming vector ﬁelds, τ ∗ , from the experiments in section 3.
For networks that have not been adversarially trained, most deformations fall well below the threshold of (cid:15) = 3 .
Out of the adversarially trained networks, only MNIST-A trained against PGD is truly robust against ADef.
Further, a comparison between the ﬁrst column of ﬁgure 5 and ﬁgure 6 indicates that ImageNet is much more vulnerable to adversarial deformations than MNIST, also considering the much higher resolution of the images in ImageNet.
Thus, it would be very interesting to study the performance of ADef with adversarially trained network for
ImageNet, as mentioned in the Conclusion.
O r i g i n a l
T-norm: 1.
D e f o r m e d
T-norm: 1.
T-norm: 1. T-norm: 1.
T-norm: 0.
T-norm: 1.
T-norm: 0. T-norm: 0.
T-norm: 0.
T-norm: 1.
O r i g i n a l
T-norm: 1.
D e f o r m e d
T-norm: 1.
T-norm: 0.
T-norm: 1. T-norm: 1.
T-norm: 1. T-norm: 1.
T-norm: 1.
T-norm: 0.
T-norm: 2.
O r i g i n a l
T-norm: 1.
D e f o r m e d
T-norm: 1.
T-norm: 1. T-norm: 1.
T-norm: 0.
T-norm: 1.
T-norm: 0. T-norm: 0.
T-norm: 0.
T-norm: 1.
O r i g i n a l
T-norm: 1.
D e f o r m e d
T-norm: 1.
T-norm: 0.
T-norm: 1. T-norm: 1.
T-norm: 1. T-norm: 1.
T-norm: 1.
T-norm: 0.
T-norm: 1.
B A DDITIONAL DEFORMED IMAGES B. MNIST
Figures 7 and 8 show adversarial deformations for the models MNIST-A and MNIST-B, respectively.
The attacks are performed using the same conﬁguration as in the experiments in section 3.
B. I MAGE N ET
Figures 9 – 13 show additional deformed images resulting from attacking the Inception-v3 model using the same conﬁguration as in the experiments in section 3.
Similarly, ﬁgures 14 – 18 show deformed images resulting from attacking the ResNet-10 model.
However, in order to increase variability in the output labels, we perform a targeted attack, targeting the label of 50th highest probability.
Original: water bottle
T : 0.
Deformed: oil ﬁlter
Original: sea cucumber
T : 0.
Deformed: chiton
Original: reﬂex camera
T : 0.
Deformed: projector
Original: bison
T : 0.
Deformed: ox
Original: minibus
T : 0.
Deformed: police van
Original: red wolf
T : 1.
Deformed: dam
Original: water tower
T : 2.
Deformed: chime
Original: pelican
T : 0.
Deformed: bath towel
Original: reﬂex camera
T : 0.
Deformed: Polaroid camera
Original: pretzel
T : 0.
Deformed: chain
Original: garbage truck
T : 0.
Deformed: trailer truck
Original: ice lolly
T : 1.
Deformed: spatula
[Figure]
Original: sewing machine
T : 1.
Deformed: paper towel
Original: starﬁsh
T : 1.
Deformed: electric ray
Original: freight car
T : 1.
Deformed: triceratops
Original: parachute
T : 1.
Deformed: golfcart
Original: Christmas stocking
T : 0.
Deformed: crane
Original: sock
T : 1.
Deformed: ice cream
Original: black stork
T : 1.
Deformed: hippopotamus
Original: worm fence
T : 1.
Deformed: viaduct
Original: chain mail
T : 0.
Deformed: ski mask
Original: mousetrap
T : 0.
Deformed: perfume
Original: pillow
T : 0.
Deformed: electric guitar
Original: pickup
T : 0.
Deformed: projector
[Figure]
: dingo
[Figure]
C M ATHEMATICAL DERIVATION
To maintain anonymity during the reviewing process, we wish to include the section of the master’s thesis Anonymous (2017) which includes the mathematical derivation of the ADef algorithm.
This section can be removed altogether when the anonymity of this reference has been lifted.
The remainder of the text is an anonymized version of the relevant part of Anonymous (2017).
Keep in mind that the mathematical notation used in the following differs from the notation used in the main body of this paper.
C. A DVERSARIAL DEFORMATIONS
In this section, we model images (or signals in general) as functions in a L 2 -Bochner space.
Let
D :=
(0 , 1) d be the domain on which our signals are deﬁned.
In the case of images we have d = 2 , but we allow any dimension ≥ 1 .
Let c be the number of channels in the signal ( c = 1 for grayscale images and c = 3 for color images).
Deﬁne the space of signals to be the Bochner space
X :=
L 2 ( D, R c ) .
It is a Hilbert space with the inner product
[Equation]
where µ is the Lebesgue measure on D .
For all x ∈ X , deﬁne
[Equation]
Note that this quantity is inﬁnite for some x ∈ X and thus does not deﬁne a norm on X .
In the space X, the box of valid signals is
[Equation]
In the literature, adversarial perturbations are usually regarded as “imperceptible” if they are small in some (cid:96) p -norm.
These norms, however, do not sufﬁciently describe what is imperceptable to the human eye.
For example, if x ∈ B and u ∈ R
d is small, then the translated image x u , deﬁned by
[Equation]
is likely to be indistinguishable from x to the human eye.
The corresponding perturbation x u − x may however have a relatively large L p -norm.
Translation by u can be seen as a deformation of x by a constant vector ﬁeld τ = u .
In this section we generalize this example to deformations by arbitrary vector ﬁelds.
We search for adversarial deformations using the same strategy as in the search for adversarial perturbations: We assume that there exists a small deformation of x that changes its label and use a ﬁrst order approximation of the classiﬁer to ﬁnd the corresponding vector ﬁeld.
We consider deformations to be small when the corresponding vector ﬁeld is small in the supremum norm.
This allows us to bypass some technical difﬁculties that appear in the analysis with other
L p -norms.
It may be of interest to limit the search for adversarial deformations to smooth vector ﬁelds.
This can be achieved by limiting the search to the range of a Gaussian smoothing operator.
C.1. D EFORMATIONS
We now make precise the notion of deformations through vector ﬁelds on D .
First, we specify a set U of admissible vector ﬁelds, and then for any bounded x ∈ X we deﬁne a family of deformed signals ( x τ ) τ ∈ U .
Let 0 < δ < 12 , K :=
[ δ, 1 − δ ] d ⊆ D and denote by T :
= L ∞ ( K, R d ) the Bochner space of vector ﬁelds τ : p (cid:55)→ τ p supported on K and bounded with respect to the norm (cid:107) τ (cid:107) T = ess sup p ∈ K (cid:107)
τ p (cid:107) R d .
For a vector ﬁeld τ , deﬁne the deformation
[Equation]
(taking τ p = 0 for p / ∈ K ) and deﬁne the set of admissible vector ﬁelds as
[Equation]
Note that since
[Equation]
we have
[Equation]
For any x ∈ X with (cid:107) x (cid:107) X, ∞ < ∞ we deﬁne the deformation operator of x :
[Equation]
x τ where x τ := x ◦ θ τ .
Observe that (cid:107) x τ (cid:107) X, ∞ ≤ (cid:107) x (cid:107) X, ∞ < ∞ for any τ ∈ U .
Hence, (cid:107) x
τ (cid:107) X < ∞ for all τ ∈ U and Θ x is well deﬁned.
Lemma C.1.
If x is twice continuously differentiable in a neighborhood of K and (cid:107) x (cid:107) X < ∞ , then Θ x : U → X is Fr ´ echet differentiable at 0 and its Fr ´ echet derivative at 0 , D
Θ x , is equal to the operator
[Equation]
where
[Equation]
Proof.
First we note that any τ ∈ T is bounded and supported on K and since x has continuous derivatives on K , we get
[Equation]
where
[Equation]
is the operator norm of D p x .
Hence, the mapping
A : T → X, τ (cid:55)→ χ τ is a well deﬁned bounded linear operator.
To understand Θ x , we consider local linear approximations of x .
For any p ∈ K and u ∈ R
d
small enough we have
[Equation]
where (cid:107) η p ( u ) (cid:107) R c
(cid:107) u (cid:107) − 2 R d
→ 0 as u → 0 .
Moreover, by continuity of the second derivatives of x on the compact set K , we have the uniform bound (cid:107) η p ( u ) (cid:107) R c ≤ C (cid:107) u (cid:107) 2 R d for all p ∈ K and all u with (cid:107) u (cid:107) R d < δ for some constant C > 0 .
Note that θ 0 = id D , so x 0 = x .
Hence for τ ∈ U we have for all p ∈ K
[Equation]
and for p / ∈ K we have x τ ( p ) − x 0 ( p ) =
χ τ ( p ) = 0 .
Recall that U , the set of admissible vector ﬁelds, contains an open ball around 0 ∈ T , so the following limit is well deﬁned.
[Equation]
This proves that Θ x is Fr´echet differentiable and that D 0 Θ x = A .
Remark C.2 .
Alternatively, we could have considered vector ﬁelds in the space T :
= L 2 ( K, R d ) instead of T .
Note that while T ⊆ T , the set of admissible vector ﬁelds
[Equation]
does not contain an open ball around 0 in T and the Fr ´ echet derivative of Θ x at 0 is not well deﬁned.
To see this, let σ > 0 and consider the vector ﬁeld τ σ : K → R d , deﬁned for all p ∈ K by τ
[Equation]
(cid:26)
[Equation]
, where q = ( 12 , . .
. , 12 ) ∈ R
d is the center of the hypercube K and e 1 = (1 , 0 , . . .
, 0) is the ﬁrst standard basis vector in R d .
The vector ﬁeld τ σ is not admissible since q + e 1 = ( 32 , 12 , . .
, 12 ) / ∈ D and B σ ( q + e 1 ) ⊆
θ τ σ ( D ) .
Although (cid:107) τ σ (cid:107) T = 1 for all σ > 0 , we have
[Equation]
where V d stands for the volume of the d -dimensional unit ball.
Hence, for any ε > 0 , the nonadmissible vector ﬁeld τ σ lies in B ε (0) as long as σ < ( ε/V d ) 1 /d .
The fact that Θ x is not deﬁned on a whole neighborhood of 0 in T would lead us to consider the
G ˆ ateaux derivative of Θ x at 0 ∈ T in any direction τ in the subspace T :
[Equation]
For the sake of simplicity, we stick to our choice T =
L ∞ ( K, R d ) .
C.1. F INDING ADVERSARIAL DEFORMATIONS
Suppose we have a classiﬁer on the signal space into L categories,
[Equation]
and an interpretable signal x ∈ B classiﬁed as
[Equation]
Further assume that x is twice continuously differentiable, and that F is Fr´echet differentiable at x .
Our goal is to ﬁnd a small deformation τ ∈ U such that
[Equation]
We proceed as with the DeepFool algorithm.
First we target a different label k
(cid:54)
=
l and deﬁne the binary classiﬁer
f k :=
F k − F l .
The function f
k : X → R returns positive values for signals for which the classiﬁer F prefers the label k over the label l and it returns negative values for signals for which F prefers l over k .
Composing this new classiﬁer with the deformation operator of x we get another binary classiﬁer g k :=
f k ◦ Θ x on U .
We have g k ( τ ) > 0 for vector ﬁelds τ such that x τ is considered to belong to class k rather than class l .
Likewise g k ( τ ) < 0 if τ is unsuccessful in changing the preference from l to k .
By assumption, f k is Fr´echet differentiable at x .
Its Fr´echet derivative, D x f k , is a continuous linear functional on the Hilbert space X , and by the Riesz representation theorem there exists a unique function ψ ∈ X such that
[Equation]
By the chain rule and lemma C.1, g k is differentiable at 0 .
Let us calculate the Fr´echet derivative of g k in the direction of τ ∈ T .
The ﬁrst equality is due to Θ x (0) = x .
[Equation]
The second to last step is due to the partial derivatives ∂ j x
i : R → R being continuous and hence bounded on the compact set K and τ j being zero outside of K :
[Equation]
Note that τ j ∈ L 2 ( K ) since it is bounded on account of τ ∈ T =
L ∞ ( K, R d ) and µ ( K ) < ∞ .
If there exists a vector ﬁeld τ ∈ U such that
[Equation]
and if it is small enough, then by ﬁrst order approximation
[Equation]
In this case x τ lies approximately on the decision boundary of f k .
For j = 1 , . . .
, d deﬁne α j ∈
L 2 ( K ) by setting
[Equation] [Equation]
Then by the calculations above, the vector ﬁeld
[Equation]
satisﬁes condition (9), provided that it is bounded.
Even if τ ( x , k ) is bounded, it may not be admissible and small enough for (10) to be a good approximation.
However, we still have the freedom to choose the targeted label k to minimize (cid:107) τ
( x , k ) (cid:107) T .
We deﬁne an algorithm for ﬁnding adversarial deformations for x : Initialize by x 0 :
= x . Then for n ≥ 0 let
[Equation]
=
[Equation]
and τ n := τ ( x n , k n ) and deﬁne
[Equation]
where Θ n x is the deformation operator of x n .
Subtleties of this update step are addressed in section
C.1.3.
Once l (cid:54) =
arg max k =1 ,...,L F k ( x n ) , the iteration stops and the algorithm outputs the deformed image
[Equation]
One may also limit the number of iterations to some n max ≥ 1 and the magnitude of the total deformation (cid:80) n − 1 m =
τ m (in some norm) to a number t max > 0 .
C.1. S MOOTH DEFORMATIONS
In order to iterate the deformation process, we need to assume that F is also Fr´echet differentiable at x τ .
Moreover, the function x τ =
Θ x ( τ ) :
D → R c does not in general satisfy the conditions of lemma C.1.
However, if x is twice continuously differentiable on an open set E ⊆ D containing K , and τ is smooth on the interior of K (denoted by ˚ K ), then x τ is twice continuously differentiable on the open set
[Equation]
If (cid:107) τ (cid:107) T is sufﬁciently small, then there exists a number ˜ δ with 0 < δ < ˜ δ < 1 / 2 such that
[Equation]
Thus x τ satisﬁes the conditions of lemma C.1 with K replaced by the smaller domain ˜ K , and we can repeat the deformation step on this shrunk set.
In this section, we give a simple method for obtaining a smooth vector ﬁeld τ that satisﬁes condition (9).
This way, one can iterate the deformation without worry, given that F is everywhere Fr ´ echet differentiable and the domain K is allowed to shrink with each iteration.
Let ϕ : R d → R be a smooth, integrable function with the property ϕ ( p ) =
ϕ ( − p ) for all p ∈ R d .
For example, ϕ may be chosen to be the d -dimensional Gaussian function of width σ > 0 ,
[Equation]
for all p ∈ R d .
Deﬁne the linear operator S : T → T by
[Equation]
all
[Equation]
where ( ϕ ∗ τ j ) K denotes the restriction of ϕ ∗
τ j to K and the convolution is deﬁned by extending τ j by zero outside of K .
The operator is well deﬁned since for each j = 1 , . . .
, d , the function ϕ ∗ τ
j is smooth, K is compact, and thus ( S τ ) j = ( ϕ ∗ τ j ) K is bounded.
Any vector ﬁeld in the range of S is smooth on the interior of K , and hence we shall search for a vector ﬁeld τ ∈ T such that ˜ τ :
= S τ satisﬁes condition (9).
Note that for any u, v ∈ L 2 ( K ) , we have
[Equation]
since ϕ = ϕ ( −· ) .
By calculation (8), we have for all τ ∈ T :
[Equation]
where α 1 , .
. . , α d are deﬁned by formula (11).
Set ˜
α
j :=
( ϕ ∗ α j ) K for j = 1 , . . .
, d and
[Equation] [Equation]
Then ˜ τ := S τ is a smooth vector ﬁeld on K
that satisﬁes condition (9),
[Equation]
Hence we may substitute ˜ τ for τ ( x , k ) in the algorithm described in section C.1.2.
C.1. A DVERSARIAL DEFORMATIONS IN PRACTICE
We now make explicit the discrete version of the deformation algorithm.
Speciﬁcally, we consider square images x = ( x 1 , x 2 , x 3 ) ∈
X
= R 3 × w × w of P = w 2 pixels
and c = 3 color channels.
Here it is important to maintain the structure of the image x as a triplet of square matrices instead of representing it simply as a column vector in R 3 P .
Let F : X → R L be a differentiable classiﬁer and let x 0 ∈
X be an image with label l = arg max k F k ( x 0 ) .
Now let n ≥ 0 be an integer and write x = x n for simplicity.
The image x ∈ X can be thought of as a sampling of a twice differentiable function ξ : (0 , 1) 2 → R 3 through x i,p,q =
ξ i (cid:18) p
[Equation]
for p, q = 1 , . . .
, w, and i = 1 , 2 , 3 .
We approximate the partial derivatives of ξ by ﬁnite differences.
For i = 1 , 2 , 3 , deﬁne the matrices
D 1 x i , D 2 x i ∈
R w × w by central differences where possible
[Equation]
q
[Equation]
for q = 2 , . . .
, ( w − 1) and all p, and deﬁne the remaining entries on the boundary by forward and backward differences accordingly:
[Equation]
and
[Equation]
for all q,
[Equation]
and
[Equation]
for all p.
The entries of D 1 x i and D 2 x i then approximate the partial derivatives of ξ i :
[Equation]
and
[Equation]
for p, q = 1 , . . .
, w and i = 1 , 2 , 3 .
Let k (cid:54) =
l be the target label.
By assumption, the gradient ∇ x
[Equation]
(( ∇ x F k ) 1 , ( ∇ x F k ) 2 , ( ∇ x F k ) 3 ) ∈ R 3 × w × w is well deﬁned.
Deﬁne the matrices α 1 , α 2 ∈ R w × w by inner product over the color dimension as in (11),
[Equation]
where (cid:12) denotes the Hadamard (entry-wise) product of matrices.
The matrices α 1 and α 2 are the (scaled) components of the discrete vector ﬁeld
[Equation]
where (cid:107)·(cid:107) F is the Frobenius norm.
Remark C.3 .
The matrices α 1 and α 2 can be replaced by convolutions ˜ α j = ϕ ∗
α j with a smooth kernel ϕ
in order to obtain a smooth vector ﬁeld τ .
Deﬁne the norm 3 of τ ∈ T =
R 2 × w × w by
[Equation]
and let
[Equation]
Finally, we deﬁne the deformed image x n +1 ∈ X by
[Equation]
i
[Equation]
p
[Equation]
for p, q = 1 , . . .
, w and i = 1 , 2 , 3 .
The values of ξ can be estimated by interpolation from x n .
If l (cid:54) = arg max k F k ( x n +1 ) , then return ˆ x := x n +1 .
Otherwise, repeat the procedure above for n +1 .