F AST IS BETTER THAN FREE : R EVISITING ADVERSARIAL TRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a ﬁrst-order method like projected gradient decent (PGD).
In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice.
Speciﬁcally, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has signiﬁcantly lower cost.
Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efﬁcient training of deep networks, allowing us to learn a robust CIFAR10 classiﬁer with 45% robust accuracy at (cid:15)
= 8 / 255 in 6 minutes, and a robust ImageNet classiﬁer with 43% robust accuracy at (cid:15)
= 2 / 255 in 12 hours, in comparison to past work based on “free” adversarial training which took 10 and 50 hours to reach the same respective thresholds.
I NTRODUCTION
Although deep network architectures continue to be successful in a wide range of applications, the problem of learning robust deep networks remains an active area of research.
In particular, safety and security focused applications are concerned about robustness to adversarial examples, data points which have been adversarially perturbed to fool a model (Szegedy et al., 2013).
The goal here is to learn a model which is not only accurate on the data, but also accurate on adversarially perturbed versions of the data.
To this end, a number of defenses have been proposed to mitigate the problem and improve the robustness of deep networks, with some of the most reliable being certiﬁed defenses and adversarial training.
However, both of these approaches come at a non-trivial, additional computational cost, often increasing training time by an order of magnitude over standard training.
This has slowed progress in researching robustness in deep networks, due to the computational difﬁculty in scaling to much larger networks and the inability to rapidly train models when experimenting with new ideas.
In response to this difﬁculty, there has been a recent surge in work that tries to to reduce the complexity of generating an adversarial example, which forms the bulk of the additional computation in adversarial training (Zhang et al., 2019; Shafahi et al., 2019).
While these works present reasonable improvements to the runtime of adversarial training, they are still signiﬁcantly slower than standard training, which has been greatly accelerated due to competitions for optimizing both the speed and cost of training (Coleman et al., 2017).
In this work, we argue that adversarial training, in fact, is not as hard as has been suggested by this past line of work.
In particular, we revisit one of the the ﬁrst proposed methods for adversarial training, using the Fast Gradient Sign Method (FGSM) to add adversarial examples to the training process (Goodfellow et al., 2014).
Although this approach has long been dismissed as ineffective, we show that by simply introducing random initialization points, FGSM-based training is as effective as projected gradient descent based training while being an order of magnitude more efﬁcient.
Moreover, FGSM adversarial training (and to a lesser extent, other adversarial training methods) can be drastically accelerated using standard techniques for efﬁcient training of deep networks, including e.g. cyclic learning rates (Smith & Topin, 2018), mixed-precision training (Micikevicius et al., 2017), and other similar techniques.
The method has extremely few free parameters to tune, and can be easily adapted to most training procedures.
The end result is that, with these approaches, we are able to train (empirically) robust classiﬁers far faster than in previous work.
Speciﬁcally, we train an (cid:96) ∞ robust CIFAR10 model to 45% accuracy at (cid:15) = 8 / 255 (the same level attained in previous work) in 6 minutes ; previous papers reported times of 80 hours for PGD-based training (Madry et al., 2017) and 10 hours for the more recent “free” adversarial training method (Shafahi et al., 2019).
Similarly, we train an (cid:96) ∞ robust ImageNet classiﬁer to 43% top-1 accuracy at (cid:15) = 2 / 255 (again matching previous results) in 12 hours of training (compared to 50 hours in the best reported previous work that we are aware of (Shafahi et al., 2019)).
Both of these times roughly match the comparable time for quickly training a standard non-robust model to reasonable accuracy.
We extensively evaluate these results against strong PGDbased attacks , and show that they obtain the same empirical performance as the slower, PGD-based training.
Thus, we argue that despite the conventional wisdom, adversarially robust training is not actually more challenging than standard training of deep networks, and can be accomplished with the notoriously weak FGSM attack.
R ELATED WORK
After the discovery of adversarial examples by Szegedy et al.
(2013), Goodfellow et al.
(2014) proposed the Fast Gradient Sign Method (FGSM) to generate adversarial examples with a single gradient step.
This method was used to perturb the inputs to the model before performing backpropagation as an early form of adversarial training.
This attack was enhanced by adding a randomization step, which was referred to as R+FGSM (Tram`er et al., 2017).
Later, the Basic Iterative
Method improved upon FGSM by taking multiple, smaller FGSM steps, ultimately rendering both FGSM-based adversarial training ineffective (Kurakin et al., 2016).
This iterative adversarial attack was further strengthened by adding multiple random restarts, and was also incorporated into the adversarial training procedure.
These improvements form the basis of what is widely understood today as adversarial training against a projected gradient descent (PGD) adversary, and the resulting method is recognized as an effective approach to learning robust networks (Madry et al., 2017).
Since then, the PGD attack and its corresponding adversarial training defense have been augmented with various techniques, such as optimization tricks like momentum to improve the adversary (Dong et al., 2018), combination with other heuristic defenses like matrix estimation (Yang et al., 2019) or logit pairing (Mosbach et al., 2018), and generalization to multiple types of adversarial attacks (Tram ` er & Boneh, 2019; Maini et al., 2019).
In addition to adversarial training, a number of other defenses against adversarial attacks have also been proposed.
Adversarial defenses span a wide range of methods, such as preprocessing techniques (Guo et al., 2017; Buckman et al., 2018; Song et al., 2017), detection algorithms (Metzen et al., 2017; Feinman et al., 2017; Carlini & Wagner, 2017a), veriﬁcation and provable defenses (Katz et al., 2017; Sinha et al., 2017; Wong & Kolter, 2017;
Raghunathan et al., 2018), and various theoretically motivated heuristics (Xiao et al., 2018; Croce et al., 2018).
While certiﬁed defenses have been scaled to reasonably sized networks (Wong et al., 2018; Mirman et al., 2018; Gowal et al., 2018;
Cohen et al., 2019;
Salman et al., 2019), the guarantees don’t match the empirical robustness obtained through adversarial training.
With the proposal of many new defense mechanisms, of great concern in the community is the use of strong attacks for evaluating robustness: weak attacks can give a misleading sense of security, and the history of adversarial examples is littered with adversarial defenses (Papernot et al., 2016;
Lu et al., 2017; Kannan et al., 2018; Tao et al., 2018) which were ultimately defeated by stronger attacks (Carlini & Wagner, 2016; 2017b;
Athalye et al., 2017; Engstrom et al., 2018; Carlini, 2019).
This highlights the difﬁculty of evaluating adversarial robustness, as pointed out by other work which began to defeat proposed defenses en masse (Uesato et al., 2018; Athalye et al., 2018).
Since then, several best practices have been proposed to mitigate this problem (Carlini et al., 2019).
Despite the eventual defeat of other adversarial defenses, adversarial training with a PGD adversary remains empirically robust to this day.
However, running a strong PGD adversary within an inner loop of training is expensive, and some earlier work in this topic found that taking larger but fewer steps did not always signiﬁcantly change the resulting robustness of a network (Wang, 2018).
To combat the increased computational overhead of the PGD defense, some recent work has looked at regressing the k -step PGD adversary to a variation of its single-step FGSM predecessor called “free” adversarial training, which can be computed with little overhead over standard training by using a single backwards pass to simultaneously update both the model weights and also the input perturbation (Shafahi et al., 2019).
Finally, when performing a multi-step PGD adversary, it is possible to cut out redundant calculations during backpropagation when computing adversarial examples for additional speedup (Zhang et al., 2019).
Although these improvements are certainly faster than the standard adversarial training procedure, they are not much faster than traditional training methods, and can still take hours to days to compute.
On the other hand, top performing training methods from the DAWNBench competition (Coleman et al., 2017) are able to train CIFAR10 and ImageNet architectures to standard benchmark metrics in mere minutes and hours respectively, using only a modest amount of computational resources.
Although some of the techniques can be quite problem speciﬁc for achieving bleedingedge performance, more general techniques such as cyclic learning rates (Smith & Topin, 2018) and half-precision computations (Micikevicius et al., 2017) have been quite successful in the top ranking submissions, and can also be useful for adversarial training.
A DVERSARIAL TRAINING OVERVIEW
Adversarial training is a method for learning networks which are robust to adversarial attacks.
Given a network f θ parameterized by θ , a dataset ( x i , y i ) , a loss function (cid:96) and a threat model ∆ , the learning problem is typically cast as the following robust optimization problem,
[Equation]
A typical choice for a threat model is to take ∆ = { δ : (cid:107) δ (cid:107) ∞ ≤ (cid:15) } for some (cid:15) > 0 .
This is the (cid:96) ∞ threat model used by Madry et al.
(2017) and what we consider in this paper.
The procedure for adversarial training is to use some adversarial attack to approximate the inner maximization over ∆ , followed by some variation of gradient descent on the model parameters θ .
For example, one of the earliest versions of adversarial training used the Fast Gradient Sign Method to approximate the inner maximization.
This could be seen as a relatively inaccurate approximation of the inner maximization for (cid:96) ∞ perturbations, and has the following closed form (Goodfellow et al., 2014):
[Equation]
A better approximation of the inner maximization is to take multiple, smaller FGSM steps of size α instead.
When the iterate leaves the threat model, it is projected back to the set ∆ (for (cid:96) ∞ perturbations.
This is equivalent to clipping δ to the interval [ − (cid:15), (cid:15) ] ).
Since this is only a local approximation of a non-convex function, multiple random restarts within the threat model ∆ typically improve the approximation of the inner maximization even further.
A combination of all these techniques is known as the PGD adversary (Madry et al., 2017), and its usage in adversarial training is summarized in Algorithm 1.
Iterate T/N times to account for minibatch replays and run for T total epochs for t = 1 . . .
T/N do for i = 1 . . .
M do // Perform simultaneous FGSM adversarial attack and model weight updates T times for j = 1 . . .
N do //
Compute gradients for perturbation and model weights simultaneously ∇ δ , ∇ θ = ∇ (cid:96) ( f θ ( x i + δ ) , y i )
δ =
δ + (cid:15) · sign( ∇ δ ) δ = max(min
( δ, (cid:15) ) , − (cid:15) )
θ =
θ − ∇ θ // Update model some optimizer, e.g. SGD end for end for end for
T do for i = 1 . . .
M do // Perform FGSM adversarial attack δ = Uniform ( − (cid:15), (cid:15) )
δ = δ + α · sign( ∇ δ (cid:96) ( f θ ( x i + δ ) , y i ))
δ = max(min( δ, (cid:15) ) , − (cid:15) )
θ =
θ − ∇ θ (cid:96) ( f θ ( x i + δ ) , y i ) //
Update model weights with some optimizer, e.g. SGD end for end for
Note that the number of gradient computations here is proportional to O ( MN ) in a single epoch, where M is the size of the dataset and N is the number of steps taken by the PGD adversary.
This is N times greater than standard training (which has O ( M ) gradient computations per epoch), and so adversarial training is typically N times slower than standard training.
3. “F REE ” ADVERSARIAL TRAINING
To get around this slowdown of a factor of N , Shafahi et al.
(2019) instead propose “free” adversarial training.
This method takes FGSM steps with full step sizes α = (cid:15) followed by updating the model weights for N iterations on the same minibatch (also referred to as “minibatch replays”).
The algorithm is summarized in Algorithm 2.
Note that perturbations are not reset between minibatches.
To account for the additional computational cost of minibatch replay, the total number of epochs is reduced by a factor of N to make the total cost equivalent to T epochs of standard training.
Although “free” adversarial training is faster than the standard PGD adversarial training, it is not as fast as we’d like: Shafahi et al.
(2019) need to run over 200 epochs in over 10 hours to learn a robust
CIFAR10 classiﬁer and two days to learn a robust ImageNet classiﬁer.
F AST ADVERSARIAL TRAINING
To speed up adversarial training and move towards the state of the art in fast standard training methods, we ﬁrst highlight the main empirical contribution of the paper: that FGSM adversarial training combined with random initialization is just as effective a defense as PGD-based training.
Following this, we discuss several techniques from the DAWNBench competition (Coleman et al., 2017) that are applicable to all adversarial training methods, which reduce the total number of epochs needed for convergence with cyclic learning rates and further speed up computations with mixedprecision arithmetic.
[Table]
4. R EVISITING FGSM ADVERSARIAL TRAINING
Despite being quite similar to FGSM adversarial training [Table] free adversarial training is empirically robust against PGD attacks whereas the FGSM adversarial training method is believed not to be robust.
To analyze why [Table] we identify a key difference between the methods: a property of free adversarial training is that the perturbation from the previous iteration is used as the initial starting point for the next iteration.
However, there is little reason to believe that an adversarial perturbation for a previous minibatch is a reasonable starting point for the next minibatch.
As a result, we hypothesize that the main beneﬁt comes from simply starting from a non-zero initial perturbation.
In light of this difference [Table] our approach is to use FGSM adversarial training with random initialization for the perturbation, as shown in Algorithm 3.
We ﬁnd that, in contrast to what was previously believed [Table] this simple adjustment to FGSM adversarial training can be used as an effective defense on par with PGD adversarial training.
Crucially, we ﬁnd that starting from a non-zero initial perturbation is the primary driver for success, regardless of the actual initialization.
In fact, both starting with the previous minibatch’s perturbation or initializing from a uniformly random perturbation allow FGSM adversarial training to succeed at being robust to full-strength PGD adversarial attacks. Note that randomized initialization for FGSM is not a new idea and was previously studied by Tram ` er et al.
Crucially, Tram ` er et al. (2017) use a different random initialization and step size, which does not result in models robust to full-strength PGD adversaries. A more detailed comparison of their approach with ours is in Appendix A. To test the effect of initialization in FGSM adversarial training, we train several models to be robust at a radius (cid:15) = 8 / 255 on CIFAR10, starting with the most “pure” form of FGSM, which takes steps of size
α = (cid:15) from a zero-initialized perturbation.
The results, given in Table 1, are consistent with the literature, and show that the model trained with zero-initialization is not robust against a PGD adversary.
However, surprisingly, simply using a random or previous-minibatch initialization instead of a zero initialization actually results in reasonable robustness levels (with random initialization performing slightly better) that are comparable to both free and PGD adversarial training methods.
The adversarial accuracies in Table 1 are calculated using a PGD adversary with 50 iterations, step size α = 2 / 255 , and 10 random restarts.
Speciﬁc optimization parameters used for training these models can be found in Appendix B.
FGSM step size Note that an FGSM step with size α = (cid:15) from a non-zero initialization is not guaranteed to lie on the boundary of the (cid:96) ∞ ball, and so this defense could potentially be seen as too weak.
We ﬁnd that increasing the step size by a factor of 1.25 to α = 10 / 255 further improved the robustness of the model so that it is on par with the best reported result from free adversarial training.
However, we also found that forcing the resulting perturbation to lie on the boundary with a step size of α = 2 (cid:15) resulted in catastrophic overﬁtting: it does not produce a model robust to adversarial attacks.
These two failure modes (starting from a zero-initialized perturbation and generating perturbations at the boundary) may explain why previous attempts at FGSM adversarial training failed. A full curve showing the effect of a range of FGSM step sizes on the robust performance can be found in Appendix C. [Figure]
Computational complexity
A second key difference between FGSM and free adversarial training is that the latter uses a single backwards pass to compute gradients for both the perturbation and the model weights while repeating the same minibatch m times in a row, called “minibatch replay”.
In comparison, the FGSM adversarial training does not need to repeat minibatches, but needs two backwards passes to compute gradients separately for the perturbation and the model weights.
As a result, the computational complexity for an epoch of FGSM adversarial training is not truly free and is equivalent to two epochs of standard training.
4. DAWNB ENCH IMPROVEMENTS
Although free adversarial training is of comparable cost per iteration to traditional standard training methods, it is not quite comparable in total cost to more recent advancements in fast methods for standard training.
Notably, top submissions to the DAWNBench competition have shown that CIFAR10 and ImageNet classiﬁers can be trained at signiﬁcantly quicker times and at much lower cost than traditional training methods.
Although some of the submissions can be quite unique in their approaches, we identify two generally applicable techniques which have a signiﬁcant impact on the convergence rate and computational speed of standard training.
Cyclic learning rate Introduced by Smith (2017) for improving convergence and reducing the amount of tuning required when training networks, a cyclic schedule for a learning rate can drastically reduce the number of epochs required for training deep networks (Smith & Topin, 2018).
A simple cyclic learning rate schedules the learning rate linearly from zero, to a maximum learning rate, and back down to zero (examples can be found in Figure 1).
Using a cyclic learning rate allows
CIFAR10 architectures to converge to benchmark accuracies in tens of epochs instead of hundreds, and is a crucial component of some of the top DAWNBench submissions.
Mixed-precision arithmetic With newer GPU architectures coming with tensor cores speciﬁcally built for rapid half-precision calculations, using mixed-precision arithmetic when training deep networks can also provide signiﬁcant speedups for standard training (Micikevicius et al., 2017).
This can drastically reduce the memory utilization, and when tensor cores are available, also reduce runtime.
In some DAWNBench submissions, switching to mixed-precision computations was key to achieving fast training while keeping costs low.
We adopt these two techniques for use in adversarial training, which allows us to drastically reduce the number of training epochs as well as the runtime on GPU infrastructure with tensor cores, while using modest amounts of computational resources.
Notably, both of these improvements can be easily applied to existing implementations of adversarial training by adding a few lines of code with very little additional engineering effort, and so are easily accessible by the general research community.
E XPERIMENTS
To demonstrate the effectiveness of FGSM adversarial training with fast training methods, we run a number of experiments on MNIST, CIFAR10, and ImageNet benchmarks.
All CIFAR10 experiments in this paper are run on a single GeForce RTX 2080ti using the ResNet18 architecture, and [Table] [Figure] all ImageNet experiments are run on a single machine with four GeForce RTX 2080tis using the
ResNet50 architecture (He et al., 2016).
Repositories for reproducing the CIFAR10 and ImageNet experiments and the corresponding trained model weights are available at the following anonymous
GitHub user account: https://github.com/anonymous-sushi-armadillo .
All experiments using FGSM adversarial training are carried out with random initial starting points and step size α = 1 .
25 (cid:15) as described in Section 4.1.
All PGD adversaries used at evaluation are run with 10 random restarts for 50 iterations (with the same hyperparameters as those used by Shafahi et al.
but further strengthened with random restarts).
Speedup with mixed-precision was incorporated with the Apex amp package at the O1 optimization level for ImageNet experiments and O2 without loss scaling for CIFAR10 experiments.
5. V ERIFIED PERFORMANCE ON MNIST
Since the FGSM attack is known to be signiﬁcantly weaker than the PGD attack, it is understandable if the reader is still skeptical of the true robustness of the models trained using this method.
To demonstrate that FGSM adversarial training confers real robustness to the model, in addition to evaluating against a PGD adversary, we leverage mixed-integer linear programming (MILP) methods from formal veriﬁcation to calculate the exact robustness of small, but veriﬁable models (Tjeng et al., 2017).
We train two convolutional networks with 16 and 32 convolutional ﬁlters followed by a fully connected layer of 100 units, the same architecture used by Tjeng et al.
We use both PGD and FGSM adversarial training at (cid:15) = 0 .
3 , where the PGD adversary for training has 40 iterations with step size 0 .
01 as done by Madry et al.
The exact veriﬁcation results can be seen in Table 2, where we ﬁnd that FGSM adversarial training confers nearly indistinguishable empirical and veriﬁed robustness to PGD adversarial training on MNIST. [Table] [Table] [Table]
F AST CIFAR
We begin our CIFAR10 experiments by combining the DAWNBench improvements from Section 4.2 with various forms of adversarial training.
For N epochs, we use a cyclic learning rate that increases linearly from 0 to λ over the ﬁrst 2 N/ 5 epochs, then decreases linearly from λ to 0 for the remaining epochs, where λ is the maximum learning rate.
For each method, we individually tune λ to be as large as possible without causing the training loss to diverge, which is the recommended learning rate test from Smith & Topin (2018).
To identify the minimum number of epochs needed for each adversarial training method, we repeatedly run each method over a range of maximum epochs N , and then plot the ﬁnal robustness of each trained model in Figure 2.
While all the adversarial training methods beneﬁt greatly from the cyclic learning rate schedule, we ﬁnd that both FGSM and PGD adversarial training require much fewer epochs than free adversarial training, and consequently reap the greatest speedups.
Using the minimum number of epochs needed for each training method to reach a baseline of 45% robust accuracy, we report the total training time in Table 3.
We ﬁnd that while all adversarial training methods beneﬁt from the DAWNBench improvements, FGSM adversarial training is the fastest, capable of learning a robust CIFAR10 classiﬁer in 6 minutes using only 15 epochs.
Interestingly, we also ﬁnd that PGD and free adversarial training take comparable amounts of time, largely because free adversarial training does not beneﬁt from the cyclic learning rate as much as PGD or FGSM adversarial training.
5. F AST I MAGE N ET
Finally, we apply all of the same techniques (FGSM adversarial training, mixed-precision, and cyclic learning rate) on the ImageNet benchmark.
In addition, the top submissions from the DAWNBench competition for ImageNet utilize two more improvements on top of this, the ﬁrst of which is the removal of weight decay regularization from batch normalization layers.
The second addition is to progressively resize images during training, starting with larger batches of smaller images in the beginning and moving on to smaller batches of larger images later. Speciﬁcally, training is divided [Table] into three phases
[Table] where phases 1 and 2 use images resized to 160 and 352 pixels respectively [Table] and phase 3 uses the entire image.
We train models to be robust at (cid:15) = 2 / 255 and (cid:15)
= 4 /
and compare to free adversarial training in Table 4, showing similar levels of robustness.
In addition to using ten restarts, we also report the PGD accuracy with one restart to reproduce the evaluation done by Shafahi et al.
[Table]
With these techniques
[Table]
we can train an ImageNet classiﬁer using 15 epochs in 12 hours using FGSM adversarial training, taking a fraction of the cost of free adversarial training as shown in Table 5.
We compare to the best performing variation of free adversarial training which which uses m =minibatch replays over 92 epochs of training (scaled down accordingly to 23 passes over the data).
Note that free adversarial training can also be enhanced with mixed-precision arithmetic, which reduces the runtime by 25%, but is still slower than FGSM-based training.
Directly combining free adversarial training with the other fast techniques used in FGSM adversarial training for ImageNet results in reduced performance which we describe in Appendix E.
C ONCLUSION
Our ﬁndings show that FGSM adversarial training, when used with random initialization, can in fact be just as effective as the more costly PGD adversarial training.
While a single iteration of FGSM adversarial training is double the cost of free adversarial training, it converges signiﬁcantly faster, especially with a cyclic learning rate schedule.
As a result, we are able to learn adversarially robust classiﬁers for CIFAR10 in minutes and for ImageNet in hours, even faster than free adversarial training but with comparable levels of robustness.
We believe that leveraging these signiﬁcant reductions in time to train robust models will allow future work to iterate even faster, and accelerate research in learning models which are resistant to adversarial attacks.
A A DIRECT COMPARISON TO R+FGSM FROM T RAM ` ER ET AL . (2017)
While a randomized version of FGSM adversarial training was proposed by Tram`er et al.
(2017), it was not shown to be as effective as adversarial training against a PGD adversary.
Here, we note the two main differences between our approach and that of Tram`er et al.
The random initialization used is different. For a data point x , we initialize with the uniform distribution in the entire perturbation region with [Equation] In comparison, Tram ` er et al.
(2017) instead initialize on the surface of a hypercube with radius (cid:15)/ 2 with
[Equation]
The step sizes used for the FGSM step are different.
We use a full step size of α = (cid:15) , whereas Tram ` er et al.
(2017) use a step size of α = (cid:15)/ 2 . To study the effect of these two differences, we run all combinations of either initialization with either step size on MNIST. The results are summarized in Table 6. We ﬁnd that using a uniform initialization adds the greatest marginal improvement to the original
R+FGSM attack, while using a full step size doesn’t seem to help on its own.
Implementing both of these improvements results in the form of FGSM adversarial training presented in this paper.
Additionally, note that R+FGSM as done by Tram`er et al.
(2017) has high variance in robust performance when done over multiple random seeds, whereas our version of FGSM adversarial training is signiﬁcantly more consistent and has a very low standard deviation over random seeds. B T RAINING PARAMETERS FOR T ABLE
[Table]
For all methods, we use a training batch size of 128 and a test batch size of 100 , and SGD optimizer with momentum 0 .
9 and weight decay 5 ∗ 10 − 4 .
C O PTIMAL STEP SIZE FOR FGSM ADVERSARIAL TRAINING
Here, we test the effect of step size on the performance of FGSM adversarial training.
We plot the mean and standard error of the robust accuracy for models trained for 30 epochs over 3 random seeds in Figure 3, and vary the step size from α = 1 / 255 to α = 16 / 255 .
We ﬁnd that we get increasing robust performance as we increase the step size up to α = 10 / 255 .
Beyond this, we ﬁnd that the model is prone to overﬁtting to the adversarial examples, since the large step size forces the model to overﬁt to the boundary of the perturbation region. [Figure] G [Figure]
A cc u r a c y
D T RAINING PARAMETERS FOR F IGURE
Parameter
FGSM*
PGD
Free
Max learning rate 0.
Attack iterations
N/A
For all methods, we use a training batch size of 128 and a test batch size of 100 , and SGD optimizer with momentum 0 .
9 and weight decay 5 ∗ 10 − 4 .
*When training FGSM on a larger number of epochs, we use a 5-step PGD adversary with 1 restart on 1 minibatch to detect overﬁtting to the FGSM adversaries, and early stop if necessary.
E C OMBINING FREE ADVERSARIAL TRAINING WITH DAWNB ENCH IMPROVEMENTS ON I MAGE N ET
While adding mixed-precision is a direct speedup to free adversarial training without hurting performance, using other optimization tricks such as the cyclic learning rate schedule, progressive resizing, and batch-norm regularization may affect the ﬁnal performance of free adversarial training.
Since
ImageNet is too large to run a comprehensive search over the various parameters as was done for
CIFAR10 in Table 3, we instead test the performance of free adversarial training when used as a drop-in replacement for FGSM adversarial training with all the same optimizations used for FGSM adversarial training.
We use free adversarial training with m = 3 minibatch-replay, with 2 epochs for phase one, 2 epochs for phase two, and 1 epoch for phase three to be equivalent to 15 epochs of standard training.
A word of caution: this is not to claim that free adversarial training is completely incompatible with the DAWNBench optimizations on ImageNet.
By giving free adversarial training more epochs, it may be possible recover the same or better performance.
However, tuning the DAWNBench techniques to be optimal for free adversarial training is not the objective of this paper, and so this is merely to show what happens if we naively apply the same DAWNBench tricks used for FGSM adversarial training to free adversarial training.
Since free adversarial training requires more epochs even when tuned with DAWNBench improvements for CIFAR10, we suspect that the same behavior
[Table]
[Table]
[Table]
to