P LAYING THE G AME OF U NIVERSAL A DVERSARIAL P ERTURBATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of learning classiﬁers robust to universal adversarial perturbations.
While prior work approaches this problem via robust optimization, adversarial training, or input transformation, we instead phrase it as a two-player zero-sum game.
In this new formulation, both players simultaneously play the same game, where one player chooses a classiﬁer that minimizes a classiﬁcation loss whilst the other player creates an adversarial perturbation that increases the same loss when applied to every sample in the training set.
By observing that performing a classiﬁcation (respectively creating adversarial samples) is the best response to the other player, we propose a novel extension of a game-theoretic algorithm, namely ﬁctitious play , to the domain of training robust classiﬁers.
Finally, we empirically show the robustness and versatility of our approach in two defence scenarios where universal attacks are performed on several image classiﬁcation datasets – CIFAR10, CIFAR100 and ImageNet.
I NTRODUCTION
Scale
Perturbation
Type
[Table]
on
[Table]
u r
[Table]
a ti on
Small scale dataset (Madry et al., 2017) (cid:88)
Large scale dataset \ (cid:88)
Deep learning has shown a tremendous progress in object recognition, and becomes a standard tool in many areas of computer vision such as image classiﬁcation, object detection, image captioning, or visual question answering (Krizhevsky et al., 2012;
Ren et al., 2015; Donahue et al., 2015; Malinowski et al., 2017).
At the same time, so called adversarial samples, which can be perceptually indistinguishable from input images, can easily fool the recognition architectures by forcing them to predict wrong categories (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016; 2017; Szegedy et al., 2014).
Due to the ubiquity of deep learning, ﬁnding a robustiﬁcation method is an important research direction.
In contrast to the vast body of research dedicated to per-sample adversarial perturbations (Szegedy et al., 2014; Moosavi-Dezfooli et al., 2016; Goodfellow et al., 2015), we focus on universal adversarial perturbations (Brown et al., 2017; Moosavi-Dezfooli et al., 2017).
This choice is mainly motivated by the following reasons.
First, those perturbations can be transferred from one datum to another, and between different architectures (Moosavi-Dezfooli et al., 2017).
More speciﬁcally, they can be used to create patches that, whenever attached to real-world objects, can fool the state-of-the-art recognition architectures in the wild (Brown et al., 2017).
Second, universal perturbations stand as a subclass of per-sample perturbations that is still very effective to fool deep neural networks classiﬁers and yet no defense strategy has addressed directly those speciﬁc perturbations.
Therefore, in this work (as shown in Table 1), we only seek robustness against such universal perturbations of the inputs because we believe that they not only represent a realistic type of attacks that can easily be used widely but also because it is a sufﬁciently well deﬁned problem so that an efﬁcient defense strategy can potentially be found.
We consider two categories of white-box attacks that produce universal perturbations.
The ﬁrst one, termed universal perturbation , is meant to produce a single perturbation for the whole dataset (Moosavi-Dezfooli et al., 2017).
The second one, termed universal adversarial patch (Brown et al., 2017), covers a part of the image with a circular patch.
To make classiﬁcation more robust, we consider a novel form of adversarial training where adversarial samples are continually included in the training protocol.
However, in contrast to standard adversarial training (Huang et al., 2015; Madry et al., 2017), we draw inspiration from game theory (Brown, 1951; Von Neumann and Morgenstern, 2007), and we model the adversarial training as a game between two players, let us call them conman and classiﬁer .
Conman fools classiﬁer by generating adversarial perturbations, while at the same time, classiﬁer makes robust predictions in the presence of such perturbations.
In this paradigm, standard adversarial training can be seen as an approximate solution for this game, where classiﬁer tries to be robust based on the most recent perturbations produced by conman .
Instead, inspired by a vast body of the research in game theory, we propose to extend the so called ﬁctitious play algorithm (Brown, 1951) that provides an optimal solution for such a simultaneous game between two players.
Fictitious play is a more general form of adversarial training where previously generated adversarial samples are also taken into consideration.
Contributions: We propose a generalization of the adversarial training in the form of a simultaneous game between two players.
Next, we show the framework is effective for two variants of adversarial attacks: universal perturbation and adversarial patch.
In particular, it is the ﬁrst defense mechanism against the adversarial patches.
Compared to adversarial training, our method sacriﬁces less accuracy for robustness to universal perturbations, and provides better robustness to universal perturbations than the standard adversarial training.
Finally, we show how our algorithm, which is inspired by ﬁctitious play , scales up to large datasets such as ImageNet.
R ELATED W ORK
On per-sample adversarial perturbations: The ﬁeld of ﬁnding adversarial perturbations has grown rapidly since Szegedy et al.
However, this method relies on a rather expensive second-order optimization method based on L-BFGS.
Later on, Goodfellow et al. (2015) have discovered that applying a single step of a sign gradient ascent (FGSM) is enough to fool a classiﬁer.
Kurakin et al.
(2016) have extended the same method to use more than one iteration (I-FGSM), and hence have created a stronger, yet computationally cheap, method of attacks.
Following such discoveries, more optimization methods are proposed to reliably evaluate the robustness of classiﬁers to adversarial samples.
Whilst this area is very active, majority of methods are bench-marked and generalized in Carlini and Wagner (2017).
On universal adversarial perturbations: Although the ﬁeld started with per-example perturbations, where each input image is separately perturbed, Moosavi-Dezfooli et al.
(2017) have empirically shown the existence of so called universal perturbations.
In that case, a single perturbation is applied to all the images in order to fool a classiﬁer.
This type of attack, where a single input transformation is able to fool a neural network, has found many concerning applications.
For instance, it can be used to create space-bounded adversarial patches (Brown et al., 2017), or to print 3D objects that would be missclassiﬁed (Athalye et al., 2017).
On adversarial training: This research area aims at making classiﬁers more robust to adversarial attacks.
To address this problem various strategies have been proposed, such as: input transformation (Liao et al., 2017; Guo et al., 2017), changing objective function (Sinha et al., 2017; Madry et al., 2017), defensive distillation (Papernot et al., 2016), or robust optimization (Madry et al., 2017).
Most of these approaches have been tested against very sophisticated algorithms and very few seem to provide robustness apart from adversarial training (Uesato et al., 2017).
Finally, Tramèr et al.
(2017) recognize, as we do, the importance of augmenting training data with perturbations produced for more than just the current model, but this work has only been applied for black-box per-sample attacks on large datasets.
On game theory: Fictitious play is a game-theoretic algorithm meant to reach Nash equilibrium in simultaneous games (Brown, 1951).
Recent work in the area of multi-agent reinforcement learning has made practical use of the ﬁctitious play algorithm to learn in zero-sum or cooperative games (Heinrich et al., 2015; Heinrich and Silver, 2016; Lanctot et al., 2017; Pérolat et al., 2018).
Our algorithm builds on these developments as it is an approximation of the Fictitious play process suited to the task of learning against universal adversarial examples.
M ETHOD
This section familiarizes the reader with our notations and some concepts thoroughly used in this work.
Next, we detail ﬁctitious play together with our novel approximation thereof.
Finally, we present the various baselines used in Section 4.
3. N OTATIONS
Classiﬁer: The classiﬁer ’s goal is to perform predictions based on the input data.
This problem is addressed as a standard classiﬁcation task, where the goal is to learn a function f ∈ F from data, mapping inputs to labels and generalizing well to unseen inputs, given a training dataset.
This is often set up as an optimization problem, where f is learned by minimizing some loss (Vapnik, 1998), that is
[Equation]
where L ( f, D ) = 1 N (cid:80) Ni =1 l ( f ( x i ) , y i ) is the miss-classiﬁcation loss (usually the cross entropy loss) of the function f ∈ F over the training dataset D = { ( x i , y i ) }
Ni =1 .
Conman: The conman ’s role is to fool classiﬁer by producing a single perturbation that is applied to every input in the training set.
More formally, for a given classiﬁer f ∈ F , conman constructs a small enough perturbation ξ f,(cid:15) so that the perturbed dataset D ξ remains ‘similar’ to the initial dataset
D by solving the following optimization problem
[Equation]
where d ( · , · ) captures dissimilarity between two sets, and is deﬁned as:
[Equation]
where ˜ D = { (˜ x i , y i ) } Ni =1 , D = { ( x i , y i ) } Ni =1 , ρ ( · , · ) is a distance between samples, and (cid:15) is a positive real number.
More precisely, we consider two kinds of perturbations.
First, universal adversarial perturbations (Moosavi-Dezfooli et al., 2017) where the same perturbation ξ is applied to all inputs, i.e. D ξ = { ( x i + ξ, y i ) } Ni =1 and (cid:107) ξ (cid:107) <
(cid:15) .
Second, adversarial patch ξ , where we ﬁrst sample a random location ( a, b ) in the image, an angle of rotation θ and a rescaling factor χ that deﬁnes an afﬁne transformation, then apply this transformation to the patch and ﬁnally overlay the transformed patch on the original image.
An example of this process is shown in Figure 1.
Let us call ˜ x
i =
A ( a, b, χ, θ, x i , ξ ) the perturbed image, then the perturbed dataset is D ξ = { (˜ x i , y i ) }
Ni =1 .
Learning a patch ξ is done by using a gradient ascent on the cross entropy loss and a term that predicts a ﬁxed class (see Section 3.4 for details).
It is important to notice that storing a univerally-perturbed dataset D ξ comes to storing a universal perturbation (or a patch) which is cheap memory-wise.
[Figure]
Robust classiﬁer: We frame the training of a robust classiﬁer as a game between classiﬁer and conman .
This can also be expressed within an optimization framework as follows
[Equation]
Intuitively, to train a robust classiﬁer, we need to minimize the empirical risk knowing that conman will attempt to maximize that risk by perturbing the dataset.
3. A G AME -T HEORETIC A PPROACH TO U NIVERSAL A TTACKS
Whilst popular approaches to robustiﬁcation against universal adversarial attacks formulate the problem of learning a robust classiﬁer as a problem of robust optimization (Madry et al., 2017), we phrase it as a game and take inspirations from the ﬁctitious play algorithm (Brown, 1951).
As
Figures 2 and 3 illustrate, the ﬁctitious play algorithm sequentially builds best responses to a uniform distribution over the past strategies of the opponent.
For conman this means building a sequence of perturbations ξ n leading to new datasets D n (Figure 2), and for the classiﬁer this means to build a sequence of classiﬁers
f n (Figure 3).
In the following, we refer to the process of ﬁnding best responses as the outer-loop.
At termination, the decision is taken uniformly based on the set of learned classiﬁers.
More formally, at iteration n ≥ 1 , conman has to ﬁnd a perturbation ξ n that maximizes the uniform average empirical loss over the past classiﬁers (Figure 2):
[Equation]
Remark:
Recall that, in the case of universal perturbations, ﬁnding D n corresponds in ﬁnding a universal perturbation (or a patch) ξ n (see Section 3.4).
Moreover, storing D n consists in storing ξ n (a single image) which is cheap memory-wise.
Classiﬁer on its side has to ﬁnd a function f n that minimizes the uniform average loss over the past datasets (Figure 3):
[Equation]
As a convention, we write L 0 classiﬁer ( f ) =
L ( f, D 0 ) , where D 0 is the original dataset.
At termination (after n steps), the strategy played ˆ f n is a uniform random variable over the set { f i } ni =0 of learned classiﬁers.
Symmetrically, we deﬁne the uniform random variable ˆ D n over the set {D i } ni = of perturbed datasets.
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
Approximate Fictitious play : Direct application of ﬁctitious play remains unpractical in our case for mainly one reason: the computation of ˆ f n at test time requires the storage of every function { f i } n i =0 .
Therefore all classiﬁers need to be stored in the dedicated memory (CPU or GPU memory) for rapid access to do inference.
This can be a real issue for large values of n as the required memory grows linearly with the number of iterations.
To alleviate this problem, we leverage the fact that only ˆ f n needs to be known to ﬁnd ξ n +1 and for the inference.
Thus, instead of storing all the classiﬁers { f i } n i =0 , we propose to learn an approximation ˜ f n of ˆ f n .
Observing that ˆ f n is a uniform random variable over the set { f i } ni =0 , where each f i minimises the loss L i classiﬁer , we approximate ˆ f n by the function ˜ f n that minimizes:
[Equation]
By paying the price of this approximation, we circumvent the main drawback of ﬁctitious play .
Note that the second step of ﬁnding a perturbation is delegated to a separated algorithm described in
Section 3.4.
Algorithm
INPUT : a dataset D
=
D , an initial classiﬁer f 0 , a number of iteration N and a number of steps
K 1: for n = 1 , . . .
, N : do 2:
Perform K steps of stochastic gradient descent on the loss L ( f, {D i } i ∈{ 0 ,...,n − 1 } ) , 3:
Find a universal perturbation ξ n (and its corresponding dataset D n ) that perturbs f . In the experiment we typically use 20000 steps of SGD (the rest of the hyper-parameters are summarized in Table 4). 4: end for 5: RETURN : f
3. R ELATED A PPROACHES AND B ASELINES
Work on classiﬁers’ robustness to adversarial perturbations mainly focuses on per-sample perturbations.
Out of many defence approaches, the most successful is adversarial training (Madry et al., 2017) which is used as the ﬁrst baseline in our experiments.
Whilst the evaluation of the robustness of defence methods is still an active area of research, the robustness of this method has been assessed by two independent studies (Athalye et al., 2018; Uesato et al., 2017).
It augments the batch of examples to train the network with a batch of adversarial examples D f,(cid:15) .
It can be seen as a stochastic gradient descent on 12 L ( f, D ) + 12 L ( f, D f,(cid:15) )
(see Algorithm 2).
Adversarial training computes per-sample perturbations of the current neural network for each batch of data to perform one gradient update.
In comparison, our approach sequentially produces new perturbed data that augment the training dataset.
Moreover, in our case, the perturbed datasets depend on past classiﬁers rather than only on the current one.
As Section 3.4 shows in more details, a universal perturbation is computed in one step of the outer-loop in Algorithm 2.
The second baseline we use is the standard Stochastic Gradient Descent (SGD) on the loss L ( f, D ) .
(Algorithm 3).
Algorithm
INPUT : a dataset D
=
D , an initial classiﬁer f 0 , a number of iteration N and a number of steps
K 1: for n = 1 , . . .
, N : do 2: for k = 1 , . . .
, K : do 3:
Sample a batch of examples { ( x i , y i ) } i ∈ B k,n , 4:
Compute adversarial examples { (˜ x i , y i ) } i ∈ B k,n (same procedure as in (Madry et al., 2017)), 5:
Perform gradient descent step on(cid:80) i ∈ B k,n l ( f ( x i ) , y i )
+ l ( f (˜ x i ) , y i ) 6: end for 7: end for 8: RETURN : f
INPUT : a dataset D , an initial classiﬁer f 0 , a number of iteration N and a number of steps K 1: for n = 1 , . .
, N : do 2:
Perform K steps of stochastic gradient descent on the loss L ( f, D ) , 3: end for 4: RETURN : f
3. L EARNING U NIVERSAL A DVERSARIAL P ERTURBATIONS
The state-of-the-art algorithm developed to learn universal adversarial perturbations is based on
Deepfool (Moosavi-Dezfooli et al., 2017) as a subroutine and is meant to make the classiﬁer to switch classes.
This objective is different from the one that we are following in our work as we wish to ﬁnd a single perturbation ξ that maximizes the loss function L n conman ( D ξ )
= 1 n (cid:80) n − 1 i =1 l ( f ( x i + ξ ) , y i ) .
Thus, to ﬁnd such an adversarial perturbation, we do a projected stochastic gradient ascent on the loss function with respect to the variable ξ .
The corresponding update is:
[Equation]
where Γ (cid:15) is the projection in L + ∞ -norm of size (cid:15) and B k is a random batch of samples of D .
To learn an adversarial patch, we use a similar approach.
Applying a patch consists in an afﬁne transformation of the patch and overlaying it into the image.
Precisely, for a random location ( a, b ), rotation ( θ ) and re-scaling ( χ ), we apply the canonical afﬁne transformation deﬁned by ( a, b, χ, θ ) to the patch and overlay it on the image.
This operation is differentiable.
Let us note the resulting perturbed image ˜ x
i =
A ( a, b, χ, θ, x, ξ ) .
Therefore, we can ﬁnd an adversarial patch that can fool the current network with the following update:
[Equation]
where E a,b,χ,θ is the expectation over the parameters of the aﬁne transformation applied to the patch.
In practice, we handle this expectation by sampling uniformly and independently the localization over the image, the angle θ , and the re-scaling parameter ( χ ) over a range deﬁned in Table 4.
E XPERIMENTS AND R ESULTS
This section presents our empirical ﬁndings and the metrics we use to assess them.
Datasets considered: CIFAR10, CIFAR100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015).
The last one is a large-scale dataset that is often used to benchmark recognition architectures.
Moreover, models trained on this dataset are frequently believed to capture important visual features that are transferred to other computer vision problems.
Architecture and Optimizer:
We run experiments using the same architecture (see Appendix. B) on both CIFAR10 and CIFAR100.
This architecture is inspired by the VGG (Simonyan and Zisserman, 2014b) architecture without the max-pooling adjusted to work on CIFAR scaled images.
However, when it comes to ImageNet, we use a Resnet 50 architecture (He et al., 2016).
We use a stochastic gradient descent with momentum of 0 .
9 as our optimizer.
Metric:
In all the ﬁgures, we report the accuracy of the network at the end of the iteration of the outer loop (which corresponds to 5000 steps of SGD).
The accuracy (dotted line in the plots) is the fraction of examples that have been correctly classiﬁed for a batch of 10000 samples randomly chosen in the train, validation and test sets.
The other metric we track is the adv accuracy .
This metric represents the accuracy of the network under the presence of the most recent universal adversarial perturbation.
Thus, the perturbation used to evaluate the performance of the network has not been used during training.
Choice of baselines: We compare our work to two baselines.
The ﬁrst one is adversarial training as this method has empirically proven it’s robustness against several white box per-sample adversarial attack and has been tested thoroughly (Uesato et al., 2017; Athalye et al., 2018).
The second is standard Stochastic gradient descent as it provides a baseline when no training is done.
A baseline that could have been considered would have been to adapt adversarial training by using an algorithm to generate universal adversarial perturbations.
However, as generating a universal adversarial perturbation takes several order of magnitude more time than per sample perturbation, this approach is unfeasible in practice since a new perturbation needs to be generated per batch in adversarial training.
4. U NIVERSAL P ERTURBATION E XPERIMENT
The ﬁrst experiment is reported in Figure 4 which shows the different metrics along the learning process on the CIFAR10 dataset.
This perturbation is quite large (16 pixels) compared to the ones used to train robust networks on per sample perturbation.
For the classical accuracy metric, SGD performs better than the two other methods.
Our ﬁctitious play method suffers from a slightly worse accuracy but better than the adversarial training (Madry et al., 2017).
However, for the accuracy under universal perturbation 1 our method performs better than the two others.
SGD does not perform well as expected and surprisingly standard adversarial training does not provide any robustness.
This can be explained by the fact that we use large perturbations (16 pixels) compared to what is usually done in the literature.
Therefore, the resulting network is robust to universal perturbations only with the ﬁctitious play training.
On CIFAR100 (Figure 5), we run the same experiment but on smaller perturbations (8 pixels) instead of 16 pixels (on CIFAR10).
It turns out smaller perturbations are sufﬁcient to fool the vanilla network.
In general, robustness against a class of adversarial examples seems to come at the cost of accuracy and, the larger the class of adversarial examples we try to be robust against, the larger that cost becomes.
According to adversarial accuracy, our method performs well against universal perturbations (almost matching the standard accuracy).
As expected the adversarial accuracy of SGD is low.
4. A DVERSARIAL P ATCHES E XPERIMENT
We run a similar experiment for adversarial patches (Brown et al., 2017) and compare it with SGD as, to the best of our knowledge, there isn’t any other method in the literature designed to be robust against adversarial patches.
The ﬁrst experiment on CIFAR100 leads to similar conclusions as the experiment on universal perturbations.
The accuracy of SGD is slightly better than the one of ﬁctitious play but the adversarial accuracy of our method surpasses the one of SGD.
We scaled up this approach to the ImageNet dataset, and again, our approach signiﬁcantly improves robustness to adversarial patches.
C ONCLUSION
We presented a method for building classiﬁers robust to universal adversarial perturbations.
As opposed to previous work, our method is able to learn classiﬁers robust to white-box attacks on large datasets whilst previous work was only able to scale on black-box attacks.
Our method provides such a robustness by sequentially including universal perturbations in the training dataset.
Our method surpasses standard adversarial training (Madry et al., 2017) when evaluated against universal adversarial perturbations.
Moreover, our method does not only provide robustness against perturbations bounded by L + ∞ -norm but also bounded in space, leading (as far as we know) to the ﬁrst method that addresses robustness to adversarial patches.
Many extensions of this work could be studied in other domains such as image segmentation (Metzen et al., 2017).
We hope this work will help to develop a larger variety of defences against adversarial perturbations that does not only focus on per-sample perturbations but also on simpler but still realistic scenarios of universal attacks.
A H YPERPARAMETERS
In this section, we sum up the hyper-parameters used for each experiments:
Scale hyper-parameterExperiment
C
I F
A R un i v e r s a l p e r t u r b ati on
C
I F
A R un i v e r s a l p e r t u r b ati on
I m a g e N e t un i v e r s a l p e r t u r b ati on
C
I F
A R un i v e r s a l p e r t u r b ati on
I m a g e N e t un i v e r s a l p e r t u r b ati on
Learning algorithm regularization 0.
inner loop step K     start learning rate 0.
decay K 0.
schedule ( 10 4 ) (15 , 30 , 45) (15 , 30 , 45) (10 , 20 , 40) (10 , 20 , 40) (10 , 20 , 40)
Batch size
Perturbations loop initial learning rate 210 −210 −510 −1 .
.size universal perturbation for learning (in pixels)   / / diameter of the patch (in percentage of the image) at test χ / / / 0 .
.rotation θ / / / 20 . 0 ◦ 20 .
0 ◦ iterations number
Batch size B K
B N ETWORK
[Table]
C D ATASET
[Table]