A DAPTIVE G ENERATION OF U NRESTRICTED A DVERSARIAL I NPUTS
Anonymous authors Paper under double-blind review
A BSTRACT
Neural networks are vulnerable to adversarially-constructed perturbations of their inputs.
Most research so far has considered perturbations of a ﬁxed magnitude under some l p norm.
Although studying these attacks is valuable, there has been increasing interest in the construction of—and robustness to— unrestricted attacks, which are not constrained to a small and rather artiﬁcial subset of all possible adversarial inputs.
We introduce a novel algorithm for generating such unrestricted adversarial inputs which, unlike prior work, is adaptive : it is able to tune its attacks to the classiﬁer being targeted.
It also offers a 400–2,000 × speedup over the existing state of the art.
We demonstrate our approach by generating unrestricted adversarial inputs that fool classiﬁers robust to perturbation-based attacks.
We also show that, by virtue of being adaptive and unrestricted, our attack is able to defeat adversarial training against it.
I NTRODUCTION
Despite their dramatic successes in other respects, neural networks are well-known to not be adversarially robust.
Szegedy et al.
(2014) discovered that neural networks are vulnerable to what they termed adversarial inputs : by adding carefully-chosen perturbations to correctly-classiﬁed inputs, the accuracy of any neural network could be almost arbitrarily decreased.
Since then, the machine learning community has rightly focused a great deal of research effort on this phenomenon.
Many early efforts to train more robust models initially appeared promising, but have since been shown to be vulnerable to new algorithms for constructing adversarial perturbations (Xu et al., 2019; Athalye et al., 2018).
As a result, more attention has been given to methods that provide formal guarantees about performance in the presence of adversarial perturbations (Liu et al., 2019), with the state of the art now providing non-trivial guarantees for the MNIST test set (Wong & Kolter, 2018; Croce et al., 2018; Wang et al., 2018).
However, almost all of this work has focused exclusively on adversarial perturbations whose magnitude is constrained by an l p norm.
There is a growing acknowledgement that this threat model is somewhat contrived: such examples are not a realistic security concern and also occupy a vanishingly small fraction of the set of potential adversarial inputs.
Therefore, there is a burgeoning interest in adversarial attacks that are unrestricted , in the sense that they do not necessarily derive from a perturbation of a natural input (Brown et al., 2018; Song et al., 2018b).
The main contribution of this paper is a novel and general method to generate unrestricted adversarial inputs.
In short, the training procedure for generative adversarial networks (GANs) is modiﬁed so that the generator network is rewarded for producing data that are both realistic and deceive a ﬁxed target network.
Our approach has four advantages over prior work:
Our method is adaptive in that it adjusts itself to best attack the speciﬁc network being targeted.
For instance, adversarial training is ineffective against our approach.
Our method is efﬁcient (offering a 400–2000 × speedup over prior work).
Our method can easily be applied to any existing conditional GAN codebase and checkpoints, regardless of architecture, training procedure, or application domain.
Our method therefore demonstrably scales to ImageNet.
B ACKGROUND : G ENERATIVE A DVERSARIAL N ETWORKS
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are a class of generative machine learning models involving the simultaneous training of two neural networks: a generator g and a discriminator d .
Speciﬁcally, given a dataset D of samples drawn from a probability distribution p D , the generator g learns to transform random noise z drawn from a simple distribution p z into an approximation of p D .
The discriminator network d learns to predict whether a given example x is drawn from the data distribution p D or was generated by g .
The generator and the discriminator are adversarial because they train simultaneously, with each being rewarded for out-performing the other.
GANs’ training behaviours are notoriously temperamental, and many modiﬁcations to the original algorithm have been proposed (Goodfellow, 2017).
The Wasserstein GAN variant (Arjovsky et al., 2017) aims to provide a more reliable gradient by designing the discriminator (renamed ‘critic’) to approximate the Wasserstein distance between the distribution generated by g θ and the data distribution p D .
An additional ‘gradient penalty’ loss term L gp can be added to implement the constraint that the function be 1-Lipschitz continuous (Gulrajani et al., 2017).
The loss functions for this Wasserstein GAN with gradient penalty (WGAN-GP) are: L g =
E
z ∼ p
z [ − d ( g ( z ))] and L d = − L g + E x ∼ p D [ − d ( x )]+ λL gp ; where the gradient penalty L gp = E ˜ x ∼ p
I [( (cid:107)∇ ˜ x
d φ (˜ x ) (cid:107) 2 − 1) 2 ] , where p I denotes the distribution sampling uniformly from the linear interpolations between generated samples and examples from p D .
The original proposal for a conditional generative adversarial network (CGAN) learns to generate samples from a conditional distribution (Mirza & Osindero, 2014) by simply passing the intended label y for the generated image to both the generator and the discriminator.
An extension of this approach is the auxiliary classiﬁer generative adversarial network (ACGAN) (Odena et al., 2017), in which the discriminator is modiﬁed to also predict the label y for the input data.
Both the generators are trained to maximise the log-likelihood of the correct label in addition to optimising their usual objective.
G ENERATING U NRESTRICTED A DVERSARIAL I NPUTS
Suppose we have a trained target classiﬁer network f : X → R | Y | that attempts to approximate an oracle function o : O → Y (where O ⊆
X is the oracle’s domain) by outputting a conﬁdence f ( x ) c ∈ R for each class c ∈ Y .
As Song et al.
(2018b) do, we deﬁne an unrestricted adversarial example to be any input x ∈ O such that the classiﬁer’s prediction is incorrect: argmax c f ( x ) c (cid:54)
=
o ( x ) .
Unlike
Song et al.
, we consider the domain of the oracle to be any input with a recognisable class, not just realistic inputs.
Gilmer et al.
(2018) offer a number motivations for why unrealistic but recongisable adversarial examples could be interesting, including security concerns and improving models’ abilities to generalize.
Nevertheless, we do carefully evaluate how realistic our results are in Section 4.1.
Unrestricted adversarial examples are a superset of conventional perturbation-based adversarial examples (which are restricted to lie within a ﬁxed distance of some correctly-classiﬁed input from a test dataset).
While providing a vastly larger space of candidates, a difﬁculty arises in determining that the classiﬁcation is incorrect; we can no longer rely on the oracle-provided labels from the test dataset.
We leverage generative models to solve this problem.
3. O UR P ROCEDURE
We begin by taking any conditional GAN, with generator loss l ordinary ; the use of a conditional GAN allows us to determine the correct label y of our generated unrestricted adversarial examples.
We then introduce loss terms which incentivise the generator g to create adversarial examples for a classiﬁer f .
A targeted attack for desired true label y and target label t (cid:54) =
y should output an image which humans would regard to have label y yet is classiﬁed as t by the classiﬁer.
We introduce a loss term, l targeted , which is minimised when the conditional generator output g ( z,y ) is classiﬁed in this way: l targeted =max c (cid:54) =
t f ( g ( z,y ))
c
− f
( g ( z,y )) t .
For an untargeted attack, we use a loss term that is minimised for any misclassiﬁcation of the example: l untargeted =
f ( g ( z,y ))
y − max c (cid:54) =
y f ( g ( z,y )) c .
Note that these new terms assume that the true labels of the generated data g ( z,y ) do indeed match the intended labels y , an assumption empirically validated in Section 4.1.
Our procedure is to alter the generator’s training objective so as to minimise both l ordinary and l (un)targeted simultaneously, thereby training the generator to output data which are both realistic and also fool the target network.
Section 3.3 provides further details.
[Figure]
a
Beginning
[Figure]
3. C HALLENGE : C ONFLICTING G RADIENTS
Intuition suggests that the gradient from l ordinary may be pointing in a different direction to the gradient from l (un)targeted , since making an image adversarial seems likely to make it less realistic, not more.
A simple experiment sufﬁces to verify this intuition.
We compute the cosine similarity between the gradients of our two loss terms at each step, i.e.
[Figure]
Figure 1a shows that this projection tends towards − 1 ; for reference, if the gradient vectors were selected uniformly at random, the magnitude of this projection would very rarely exceed 0.001.
In other words, as training progresses, the gradients from these terms tend towards pointing in opposite directions.
This makes joint optimisation using a gradient descent approach challenging.
3. S TRATEGIES T O O VERCOME T RAINING C HALLENGES
We empirically evaluate the effect of each technique described below in our ablative experiments reported in Section 4.4.
Realistic pretraining It is widely accepted that real image data occupy a relatively low-dimensional and contiguous manifold (Goodfellow et al., 2016, p. 160).
Conversely, we know that adversarial examples pervade the full input space: it appears that there is an adversarial example nearby nearly any point in the input space.
Therefore, a generator that is pretrained using only l ordinary before adversarially ﬁnetuning by introducing our additional loss term is more successful than using both loss terms from a random initialisation.
By beginning our search in regions of realistic examples, we’re more likely to ﬁnd adversarial examples that are sufﬁciently realistic.
Besides the generated images being subjectively better, Figure 1b shows that the gradients conﬂict to a much lesser extent.
Note that any existing conditional GAN architecture, pretrained checkpoint and training algorithm could be used here, allowing our method to leverage the signiﬁcant advances being made in this area.
Amalgamation of loss terms Rather than naïvely summing l ordinary and l (un)targeted , we use the following per-example loss term:
[Equation]
Here, κ is a hyperparameter similar to that in the Carlini & Wagner (2017) attack: it controls the conﬁdence of the generated adversarial examples.
If the difference between the desired logit and the next-greatest logit is less than κ , the generator is linearly rewarded for improving this gap (gaining conﬁdence); beyond a difference of κ (once an example is ‘good enough’), the reward exponentially decreases.
We use κ =0 for our experiments as we do not require strong misclassiﬁcations.
Stochastic loss selection The gradients from the two loss terms are in conﬂict, and in practice the l (un)targeted gradient dominates.
The proportion of misclassiﬁed generated inputs rises quickly to almost 100%, but the generated images were noticeably unrealistic, meaning their correct label may change.
To address this, we introduce the ‘attack rate’ µ .
During adversarial ﬁnetuning, the ﬁnetuning loss term is used at each step only with probability µ ; with probability 1 − µ , the pretraining loss ( l ordinary
[Figure]
I n t e nd e d t r u e l a b e l
[Figure]
only) is used.
As desired, this new hyperparameter allows the success rate of the generated unrestricted adversarial examples to be traded off with their realism.
E XPERIMENTAL EVALUATION
Our method aims to generate unrestricted adversarial inputs in a way that adapts to the targeted classiﬁer.
We therefore conducted experiments to check whether the generated examples were in fact unrestricted, adversarial, and adapted to the classiﬁer.
We then address some questions regarding the performance and generality of our approach, including realism.
The MNIST dataset (LeCun et al., 1998) is the main focus of the experimental evaluation, because this is the most challenging domain for the generation of realistic adversarial inputs.
State-of-the-art classiﬁers perform very well, with around 0.2% test error (Kowsari et al., 2018; Wan et al., 2013).
In particular, attempts to create robust classiﬁers have also been most successful on this dataset, perhaps due to its simplicity (Shafahi et al., 2018).
We target ﬁve pretrained classiﬁers provably robust to adversarial perturbations: there is guaranteed to be no adversarial input within a distance (cid:15) of p % of test inputs under the l ∞ norm.
All ﬁve are the current state-of-the-art in this domain, trained by Wong & Kolter (2018), and Wang et al.
See Appendix C for details.
In our experiments, we combine three well-established generator architectures: a Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017), a conditional GAN (Mirza & Osindero, 2014) and an auxiliary classiﬁer GAN (Odena et al., 2017).
Full details are given in Appendix E.
A GAN was adversarially ﬁnetuned for each of the 10 target labels, and for the untargeted case.
Once trained, the generators were used to produce examples for all intended true labels, which were then ﬁltered so that the classiﬁer label matched the target.
Images were generated until 200 ﬁltered examples were generated or until 100 seconds had elapsed.
Interestingly, this led to no adversarial examples with intended true label ‘0’ and target classiﬁcation ‘1’, so this case is omitted.
Figure 2 and Appendix B give examples of generated images for which the computed label matches the target classiﬁcation.
4. E FFICACY OF A TTACKS
We claim that our method generates unrestricted adversarial examples, which are somewhat realistic.
We empirically verify each claim in turn.
Since our method does not work by perturbing existing data, only a simple sanity check was required to verify that the generated images are not close to images in the training set, as could be caused by over-ﬁtting.
We selected ten generated inputs that are visually similar to the training set, and computed the shortest distances between the images and all images in the training set.
The selected images are
[Table] [Table]
I n t e nd e d t r u e l a
[Table]
l
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
given in Table 2.
Table 1 shows that they are much further from any training example than would be the case with a perturbation-based attack.
Next, we evaluate the whether our method is successful in generating adversarial inputs.
We need to check that the true label matches the intended true label for each example, or the generator could simply be producing images that visually match the target class.
To check this we had Amazon’s
MTurk workers classify the generated images.
For cost reasons, we only did this targeting Wong and
Kolter’s provably-robust network (Wong & Kolter, 2018).
We used a sample size of 100 judges for each intended true label/target label pair for each experiment.
Figure 4 shows the proportion of inputs for which not only does the label computed by the classiﬁer match the target label, but the human-judged true label matches the intended true label speciﬁed to the generator.
The mean number of correct labels for the untargeted attack is 80%.
This can be considered to be the success rate of our attack.
We now investigate if the generated examples are realistic.
A set of inputs is realistic with respect to a dataset if a human cannot reliably identify to which set an example belongs.
To check this, we again used MTurk workers.
After familiarising themselves with examples from the training dataset, each worker had to pick which image out of ten was most likely to have been generated.
Figure 5 shows the proportion of the time that generated images were not identiﬁed as such.
For comparison, we repeated these experiments but attacking a non-robust classiﬁer network.
The untargeted success rate was 90% (vs. 80% against the robust classiﬁer), and 60% (vs. 50%) were not identiﬁed as being generated.
Similar differences were seen in targeted attacks; see Appendix I.
4. A DAPTIVITY TO A DVERSARIAL T RAINING D EFENCES
In our experiments so far we have evaluated our method against pretrained classiﬁers that are provably robust to adversarial perturbations.
We now investigate whether standard adversarial training (Madry et al., 2018) against our attack in particular is effective.
As our method is adaptive, it is not immediately
[Figure]
clear how to integrate our method into the adversarial training framework.
The two obvious possibilities are ﬁnetuning the generator at each step of training, or alternately training the classiﬁer and ﬁnetuning the generator.
We explore these below as ‘online’ and ‘ofﬂine’ adversarial training, respectively.
Online Adversarial Training
We update both the generator and classiﬁer at each training step.
We run adversarial training for over 1.6 million training steps, and Figure 6d shows that even during training we maintain a roughly 80% attack success rate.
Once we have ﬁnished training the classiﬁer, we attack it normally using our method.
Despite having been adversarially trained, we are still able to fool the classiﬁer >99% of the time within 16,000 steps, as shown in Figure 6e.
Ofﬂine Adversarial Training Starting with a pretrained GAN and classiﬁer, we iterate ‘training rounds’ consisting of two phases.
First, a GAN is adversarially ﬁnetuned (starting from the pretrained
GAN each time) for a ﬁxed period to attack the classiﬁer.
Second, 80,000 generated unrestricted adversarial examples are added to the existing training dataset, and the classiﬁer continues training until almost 100% accuracy is achieved.
Figure 6a shows that, for the ﬁrst few training rounds, adversarial ﬁnetuning is successful: the proportion of examples generated which fool the classiﬁer increases to over 80%.
Figure 6b shows the same story 30 rounds (and hence hundreds of thousands of classiﬁer gradient steps) in.
We ﬁnd that the classiﬁer is able to defend against the kinds of attacks previously produced by the generator.
However, the generator’s opportunity to adversarially ﬁnetune again allows it to generate adversarial examples in a new ‘blindspot’ of the classiﬁer.
For more details on these experiments, see Appendix D.
4. S CALING TO I MAGE N ET
To demonstrate the scalability of our method, we apply it to the notoriously large and complex
ImageNet-1K dataset, using the author’s ‘ofﬁcially unofﬁcial’ published code and checkpoints for the current state-of-the-art, BigGAN (Brock et al., 2019).
In the untargeted case, our method is able to ﬁnetune this BigGAN to fool the classiﬁer >99% of the time within 40 gradient steps (compared to the 10 5 taken to train from scratch).
Our main focus, though, is on the much more challenging targeted attack.
We found that typically, on the order of 100 gradient steps were required for >10% of generated examples to be classiﬁed (top-1) as the target class.
Compared to MNIST, each ImageNet gradient step takes about 100x longer to compute, but the 100x decrease in the number of gradient steps required compensates for this, resulting in a similar compute time overall.
Image quality as measured by Inception Score (Salimans et al., 2016) typically decreased from 70.
This is slightly better than mid-2018 state-of-the-art of 52 (Zhang et al., 2019) or the mid-2017 state-of-the-art of 12 using
WGAN-GP (Shmelkov et al., 2018).
We speculate that if the GAN were ﬁnetuned for signiﬁcantly longer, the gradient from the discriminator would learn to regain some of this lost realism.
Figureshows selected samples of generated adversarial examples; Appendix A has a more extensive collection.
4. A BLATIVE S TUDIES AND B ASELINE
No Adversarial Finetuning We evaluate the extent to which adversarial ﬁnetuning is effective by comparing to a GAN which has not been ﬁnetuned.
As expected, this is much less successful at producing adversarial examples.
The proportion of its outputs which are misclassiﬁed matches that of the test set (around 2% for MNIST), whereas adversarial ﬁnetuning can easily increase this proportion to well over 99%.
Furthermore, only 66% of its misclassiﬁed outputs had maintained their true class, as opposed to 80% for an adversarially ﬁnetuned GAN.
Further results can be found in Appendix K.
Naïve Adversarial Finetuning We investigate the effect of the training strategies described in
Section 3.3 by removing each in turn.
We ﬁnd that without pretraining, our method does eventually converge, but with much less realistic results.
We also ﬁnd that the attack rate is an effective technique for trading off adversarial success rate with image realism; without this technique, the attack rate is in effect set to 1, which again produces less realistic results.
Note that unrealistic images affect the attack’s success rate because the true class of the images is more likely to change.
Lastly, we ﬁnd that choosing l ordinary + l (un)targeted as the generator’s loss results in catastrophic collapse of the training, likely because l (un)targeted is optimised for too heavily.
Full details can be found in Appendix L.
Naïve Baseline Finally we compare our method to a baseline where data is generated using an ordinary GAN and then adversarially perturbed using projected gradient descent with a norm bound of (cid:15) =0 .
1 under l ∞ .
With a success rate of under 1% on data which were originally classiﬁed correctly, we ﬁnd this approach is no better than perturbations on the test set; this is as expected, since the generator learns the data distribution.
4. T HREATS TO V ALIDITY
The evaluation of the success of the attacks relies on data provided by the MTurk workers.
We therefore employed measures to safeguard the quality of this data, described in Appendix H. We also believe that our method will generalise to any dataset and domain for which GANs can be trained successfully.
However, this has only been demonstrated on two image classiﬁcation tasks (albeit dissimilar in nature).
Lastly, intuition suggests that our method will be able to adapt to ﬁnd unrestricted adversarial examples for most ﬁxed defence methods, since it is so free to generate inputs without the constraints that current defence methods rely upon.
However, we have only demonstrated it explicitly for the most popular standard defence; we leave it to future work to ﬁnd a defence against our adaptive approach.
R ELATED W ORK 5. C OMPARISON TO THE S TATE OF THE A RT
We compare our method to that of Song et al.
(2018b), the current state of the art in generating unrestricted adversarial examples.
Like ours, this method leverages a pretrained GAN.
It differs, however, in how adversarial examples are then produced.
Instead of adversarially ﬁnetuning the generator, it searches for an input to the generator that both deceives the target network and are conﬁdently correctly classiﬁed by the discriminator’s auxiliary classiﬁer (an ACGAN (Odena et al., 2017) is required in this case).
The GAN training is therefore blind to the target network.
Our model achieves similar success rates in generating unrestricted adversarial examples: our success rate of 80% (cf.
Section 4.1) is roughly comparable to that of Song et al.
(2018b), 88.8%.
For comparison, we repeated the realism experiments from Section 4.1, with the difference that judges were asked to identify the one generated image from a choice of two.
In this case, Song et al.
report that participants select the generated image as the more realistic 21.8% of the time while for our untargeted attack, this ﬁgure is 24%; completely realistic image would be chosen 50% of the time.
Full results are given in Appendix J.
Beyond achieving comparable attack success rates, our approach has four signiﬁcant advantages over prior work.
Firstly: adaptivity.
In Section 4.2 we have shown that our model is capable of iteratively adapting to an adversarially trained classiﬁer.
By contrast, Song et al.
’s method performs poorly against adversarial training because the GAN is not trained with respect to a target classiﬁer, remaining ﬁxed after the attack begins.
Therefore, if a classiﬁer learns to be correct in the space their algorithm searches, it will no longer be able to generate images different enough to be adversarial.
Figure 6c shows that standard adversarial training quickly and effectively defends against Song et al.
’s attack, while it fails against ours.
Secondly: efﬁciency.
Once trained, our method requires only a single forward pass to generate adversarial examples.
Song et al.
require 100–500 iterations, each with 4 passes: forward and backward through both the generator and classiﬁer.
Our method is therefore 400–2,000 × more efﬁcient.
Lastly: scale and versatility.
Section 4.3 shows that our model scales to ImageNet, a dataset with dimensionality 16 × greater than the largest Song et al.
demonstrate on.
Our method has the further beneﬁt that we can use any pretrained conditional GAN, such as BigGAN (Brock et al., 2019).
Song et al.
depend on an auxiliary classiﬁer for larger datasets, which BigGAN does not provide.
O THER R ELATED W ORK
Wang et al.
(2019) independently propose a method which is superﬁcially similar to ours: they also train a GAN to directly generate adversarial examples.
Instead of using the ordinary GAN loss to maintain realism, they use a new loss term.
This term, (cid:107) g pretrained ( z ) − g
( z ) (cid:107) p , penalises the generator g given input z proportional to the deviation caused by ﬁnetuning from the original output.
Wang et al.
’s choice of loss term has the unfortunate effect of preventing the generator from generating either unrestricted adversarial examples or examples which are sure to fall within an l p -norm ball of a realistic input.
By contrast our approach allows for truly unrestricted and adaptive examples.
Furthermore: we evaluate against state-of-the-art provably-robust networks rather than ad-hoc classiﬁers; we do not assume that all the true labels remain the same (which is unlikely), and conduct a user study to test this; and we demonstrate that our approach scales beyond MNIST (to ImageNet).
Sharif et al.
(2019) train a network to generate patterned spectacles, which, when added to an image of a face, cause misclassiﬁcation.
They also adapt this approach to MNIST using an approach quite similar to ours.
However, this only achieves a success rate of 0.83% after ﬁltering to “only the digits that where likely to be comprehensible by humans” against a classiﬁer which was state-of-the-art in 2017.
In contrast, we achieve around 80% accuracy against current state-of-the-art robust classiﬁers.
A wide range of work trains networks to generate adversarial perturbations (Hayes & Danezis, 2018;
Baluja & Fischer, 2018; Xiao et al., 2018; Song et al., 2018a; Poursaeed et al., 2018).
While these must also balance realism and adversarial success, the key difference is that we generate unrestricted adversarial examples, allowing attacks to succeed when constrained perturbations provably fail.
Hu et al.
(2019) introduce a search for pairs of nearby unrestricted adversarial examples, but unfortunately cannot ensure that their true label is meaningful; if the search starting point is random, it is overwhelmingly likely not to be.
If instead it is a known input, the examples are not unrestricted.
C ONCLUSION
We have introduced an algorithm which trains a GAN to generate unrestricted adversarial inputs; we demonstrate that these, as expected, are successful against state-of-the-art classiﬁers robust to perturbation attacks.
The key novelty in our attack procedure is that it entails the tuning of the weights of the generator to target a speciﬁc network.
As a result, it can be considered adaptive : we have shown that, while prior work is quickly mitigated by standard adversarial training, our attack adapts to ﬁnd a new way of fooling the classiﬁer.
In addition, once the generator is adversarially ﬁnetuned, it becomes an endless supply of cheap adversarial examples: generation of adversarial examples requires a single forward pass rather than execution of any optimisation algorithm, resulting in a 400–2000 × speedup over the state of the art.
We have also demonstrated that any existing GAN codebase can easily be used by adapting BigGAN to generate unrestricted adversarial examples for ImageNet.
A S AMPLES OF I MAGE N ET U NRESTRICTED A DVERSARIAL E XAMPLES
Randomly-selected successful targeted unrestricted adversarial examples generated using adversarially ﬁnetuned BigGANs (Brock et al., 2019).
The targeted classiﬁer is ResNet-152 (He et al., 2016), the highest-accuracy pretrained classiﬁer packaged with PyTorch.
Besides setting our attack rate at 0.1, all conﬁguration and hyperparameters are as described in the BigGAN ‘ofﬁcially unofﬁcial’ codebase. [Figure] lifeboat solardish stonewall suspensionbridge thresher unicycle comicbook butternutsquash [Figure]
[Figure]
spottedsalamander terrapin alligatorlizard
[Figure]
hornedrattlesnake centipede
[Figure]
howlermonkey airship combinationlock
[Figure]
acorn capitulum
[Figure]
[Figure]
A. C OMPARISON TO N O A DVERSARIAL F INETUNING
The results above are not as high-quality as those reported in the BigGAN paper (Brock et al., 2019).
There are two causes of this which need to be disentangled: the effect of adversarial ﬁnetuning, and the limitations of the BigGAN implementation.
One limitation is that the checkpoint we use has a much lower Inception Score than is reported in the paper.
Another is that our limited access to expensive hardware forces us to use a batch size of 15; small batch sizes “lead to inaccurate estimation of the batch statistics, and reducing batch normalisation’s batch size increases the model error dramatically” (Wu & He, 2018, p.1). We therefore provide samples from the BigGAN running on our machine, 15 iterations after the checkpoint. Figure 11a shows a set of samples taken after 15 iterations of continuing ordinary training; Figure 11b shows the output of the generator for the same input after 15 iterations of adversarial ﬁnetuning, instead.
To our eyes, the samples in Figure 11a are similar in quality to those in Figure 11b and on the previous page.
[Figure]
B S AMPLES OF MNIST U NRESTRICTED A DVERSARIAL E XAMPLES
[Figure]
C T ARGETED CLASSIFIERS
All targeted classiﬁers (other than ‘simple fully-connected’) are provably robust to adversarial perturbations in the sense that there is guaranteed to be no adversarial input within a distance (cid:15) of p % of test inputs under the l ∞ norm
[Table] [Table]
Three
[Table]
connected layers of size
[Table]
, 128 and 32 with LeakyReLU activations
D A DVERSARIAL T RAINING E XPERIMENT
The classiﬁer trained during adversarial training (both the architecture and hyperparameters) is the one used in Madry et al.
(2017), and in particular from their associated MNIST Adversarial Examples
Challenge.
For the ofﬂine experiments with our own model
[Table]
we ﬁrst pretrain the generator.
We then continue in ‘training rounds’.
First, we ﬁne-tune against the classiﬁer for 5000 gradient steps, using the hyperparameters from Table 6, but with an attack rate of 0.4.
Next, we produce 80,000 attacked training examples (using an untargeted attack), which are added to the pool of all examples generated so far.
Then, the classiﬁer is trained on the entirety of the pool of samples 30 times, with a batch size of 128.
Once a training round is completed we start again, resetting the GAN to how it was before the adversarial ﬁnetuning.
For the experiments with Song et al.’s (2018b) model, we run 300 training gradient steps for the Madry et al.
classiﬁer, with a batch size of 64.
At each step, the training data is produced by Song et al.
’s model.
We use their code and the hyperparameters they provide for untargeted attacks in Table 4 of their appendix.
E MNIST E XPERIMENTS : A RCHITECTURES AND H YPERPARAMETERS
The WGAN-GP (Gulrajani et al., 2017) and ACGAN (Odena et al., 2017) architectures were the starting points for the design of these neural networks.
Only a small amount of manual hyperparameter tuning was performed.
The generator is a convolutional neural network, conditioned on class label.
The discriminator network is a combination of a conditional WGAN-GP critic, which learns an approximation of the Wasserstein distance between the generated and training-set conditional distributions, and an auxiliary classiﬁer, which predicts the likelihood of the possible values of h ( x ) .
We combined these two architectures in an attempt to strengthen the gradient provided to the generator, helping to generate data which are both realistic and for which the true (i.e., human-judged) labels match the intended true labels.
The critic is given the true label of the data h ( x ) to improve its training, but the auxiliary classiﬁer must not have access to this information since its purpose is to predict it.
We therefore split the discriminator d into three sub-networks.
Network d 0 : X → R i effectively preprocesses the input, passing an intermediate representation to the critic network d 1 : R i × Y
→ R and the auxiliary classiﬁer network d 2 : R i
→ R | Y | .
In our experiments, both d 1 and d 2 were single fully-connected layers of the appropriate dimension.
The loss terms from the WGAN-GP and ACGAN algorithms are simply summed.
The auxiliary classiﬁer helps the training converge, but is not necessary.
[Table] [Table]
Convolution
[Table]
Convolution
[Table]
[Table]
[Table]
Normally distributed as described by He et al.
[Table]
λ =
F V ISUAL E FFECT OF A DVERSARIAL F INETUNING
[Figure]
G T RANSFERABILITY OF A DVERSARIAL E XAMPLES
Perturbation-based adversarial examples typically somewhat generalise between models (Szegedy et al., 2014; Liu et al., 2017).
That is, inputs crafted using white-box access to fool one model often fool a different model.
This means that black-box attacks are possible, if the attacker has a different trained model for the same task.
To evaluate whether our method could be used in the same way, we generated about 20,000 untargeted unrestricted adversarial inputs for each target classiﬁer
[Table]
and measured the misclassiﬁcation rates on this set for the other models.
The high variance of the results, shown in Table
[Table]
suggests that successful transfer may depend more on the networks in question than on our generation algorithm.
[Table]
H S AFEGUARDING MT URK D ATA Q UALITY
The evaluation of our method relies entirely on the quality of the data provided by the MTurk workers.
We therefore took a number of measures to ensure that participants understood the instructions and completed the tasks diligently:
• Only workers with good track records were permitted to participate.
• The instructions speciﬁed that particular answers should be given to speciﬁed questions to prove that the instructions had been read carefully.
Approximately 10% of work was rejected for failing this check.
• For the image labelling tasks, some images with known labels were included to check that the right labels were being given.
Reassuringly, almost no work was rejected for failing this check.
• For the identiﬁcation of the generated images, a bonus nearly doubling the pay per image was given for each correctly-identiﬁed image, providing an extra incentive to try hard.
• To provide a disincentive to high-speed random clicking, a minimum time spent answering each question was enforced.
• If more than 1% of questions were left unanswered, we interpreted this as a sign of carelessness and did not use any of the data from that task.
I R ESULTS FOR N ON -R OBUST T ARGET N ETWORK
These results are targeting a simple convolutional neural network with LeakyReLU activations and three hidden layers of size 256, 128 and 32, trained until convergence
Target label 0 1 4 5 6
None
I n t e nd e d t r u e l a b e l 93 96 96 99 93 95 97 94 97192 100 92 97 96 88 96
Mean 88 79 89 91 90 91 88 91 92
Target label
1 2 3 4 7 8 9 None
I n t e nd e d t r u e l a b e l 00 76 44 65 70 77 89 56 75182
Mean 70 56 75 72
More precisely, of generated images for which the computed label output by the classiﬁer matches the target label, the percentage which are truly adversarial (in the sense that the true label of the image matches the intended true label passed to the generator network) is reported.
[Figure] [Figure]
9 None
I n t e nd e d t r u e l a b e l 45 48 46 43 38 61 51 44 59164 74 72 62 78 83 73 752 5253 41 52 35 48 54 403 64 4143 60 29 51 56 584 59 45 4945 53 55 49 695 48 46 44 6360 49 57 616 76 48 44 46 5438 62 547 51 32 60 59 54 5461 658 62 53 62 57 56 56 54679 47 42 51 60 72 69 54
Mean 57 44 54 55 55 54 52 55 59
Moreprecisely, theproportionofgeneratedinputs for which the classiﬁed label matches the target label which were not identiﬁed as being generated when placed amongst nine images from the training dataset.
If the generated images were completely realistic, the expected result would be
Target label 0 1 2 3 4 5 6 7 8
None
I n t e nd e d t r u e l a b e l 00 59 56
Mean 62 58 60 62 64 64 62 62 63
J
S
IDE
[Figure]
BY
[Figure]
S
IDE
I
MAGE C OMPARISON
R ESULTS
Each ﬁgure shows the number of human judgements out of 100 which correctly identiﬁed the unrestricted adversarial input in a side-by-side comparison with an image drawn from the dataset.
If the generated images were completely realistic, the expected result would be 50.
[Figure]
0 1 2 3 4 5 6 7 8 9 None
I n t e nd e d t r u e l a b e l 23 29 24 30 25 20 22 17 22114 19 16 26 38 25 21 212 2418 25 21 17 26 17 193 22 2431 28 25 24 24 304 25 25 2821 28 22 17 235 23 16 24 2723 19 27 216 19 21 25 21 1920 28 187 23 27 22 26 17 2429 168 25 25 21 21 24 23 24289 18 21 22 27 27 24 23
Mean 22 23 22
[Figure]
[Figure]
nd
[Figure]
l 43 37 42 42 41 36 43 38 45129 40 38 40 39 40 43 372 4538 36 35 39 44 41 313
Mean 38 40 39 41 38 40 36 44 40
Target label 0 1 2 3 4 5 6 7 8
None
I n t e nd e d t r u e l a b e l 23 32 23 22 22 23 19 24 27131 32 31 28 30 36 41 292 2431 22 24 30 26 31 203 30 2620 20 25 19 21 254 27 28 2528 27 25 26 275 29 25 25 2523 25 22 226 30 22 27 17 3020 32 237 18 24 30 32 25 2434 268 28 29 19 23 22 28 27269 29 21 20 31 28 33 21
Mean 27 25 27 26 25 26 25 27 29
Target label 0 1 2 3 4 5 6 7 8
None
I n t e nd e d t r u e l a b e l 00 25 28 31 26 34 16 20 21136 26 31 27 25 24 38 222 2429 28 26 21 28 25 223 23 2726
Mean 27 24 26 26 29 28 26 25 29
K A DVERSARIAL F INETUNING A BLATION S TUDY
We perform the attack procedure described in Section 4, but using a GAN which has not been adversarially ﬁnetuned.
That is, we use the generator to generate many examples, and ﬁlter to keep all those which are misclassiﬁed (the untargeted case) or misclassiﬁed with a particular label (the ‘targeted’ case, although note that of course the un-ﬁnetuned generator has no notion of a target).
We then evaluate the proportion of these ﬁltered examples which correctly maintain their intended true class, a necessary condition for an adversarial attack.
We also report the proportion of these ﬁltered examples which are not correctly identiﬁed by humans judges as being generated (out of a selection of ten).
For instance, consider the result for intended true label 9 and target label 0.
We ﬁrst use the conditional
GAN to produce a set of images that are intended to be 9s.
We then ﬁlter this set and keep only those that are classiﬁed as 0s by the classiﬁer.
Finally, we report below the percentage of these for which the true label (determined by humans) is indeed a 9 (55%), and the percentage of these which fool human judges into believing that they are real test data (51%).
[Figure]
I n t e nd e d t r u e l a b e l 73 91 62 62 91 77 88 50 59149 51 80 06 31 47 61 772 1353 32 30 30 52 50 193 29 6926 60 12 79 22 284 42 55 6670 80 70 48 735 18 46 55 6129 59 31 516 50 60 80 87 7474 54 597 22 43 82 63 31 6462 388 70 68 80 75 75 91 63809 55 66 88 74 88 92 66
Mean 37 60 72 63 58 65 43 67 50 54 nonﬁnetuned generator, as described above.
Target label 0 1 2 3 4 5 6 7 8
None
I n t e nd e d t r u e l a b e l 48 63 53 53 62 56 46 40 37134 52 50 37 27 52 47 462 4251 39 30 36 54 49 403 41 4742 55 47 53 41 474 49 52 4145 50 51 45 665 38 50 43 5654 44 51 486 49 52 47 50 4638 62 467 39 57 59 36 49 4541 578 52 50 59 52 56 57 43519 51 51 66 53 74 61 53
Mean 45 50 52 50 51 50 44 50 47
L T RAINING S TRATEGIES A BLATION S TUDIES
We run several ablation studies for the components described in Section 3.3.
We ﬁnd that while these strategies are not necessary, they make examples more realistic.
This in turn improves the chance that the true class of an example remains the same, increasing our success rate.
L. P RETRAINING
[Figure]
000 iterations for another 45,000.
Here, we show results using the same setup as in Figure 13.
We show the results after the number of iterations equivalent to pretraining, to pretraining and ﬁnetuning, and at convergence.
The ﬁnal results both took longer and are visually less convincing than comparable results in Figure 13.
L. N AÏVE L OSS F UNCTION
[Figure]
As expected, our custom loss function as described in Section 3.3 signiﬁcantly improves convergence to generator weights which continue to generate images realistic enough to maintain their true classes
[Figure]