T OWARDS FAST AND EFFECTIVE SINGLE - STEP ADVERSARIAL TRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, Wong et al.
(2020) showed adversarial training with single-step FGSM leads to a characteristic failure mode named catastrophic overﬁtting (CO), in which a model becomes suddenly vulnerable to multi-step attacks.
Moreover, they showed adding a random perturbation prior to FGSM (RS-FGSM) seemed to be sufﬁcient to prevent CO.
However, Andriushchenko & Flammarion (2020) observed that RS-FGSM still leads to CO for larger perturbations and argue that the only contribution of the random step is to reduce the magnitude of the attacks.
They suggest a regularizer (GradAlign) that avoids CO but is signiﬁcantly more expensive than RS-FGSM.
In this work, we methodically revisit the role of noise and clipping in single-step adversarial training.
Contrary to previous intuitions, we ﬁnd that not clipping the perturbation around the clean sample and using a stronger noise is highly effective in avoiding CO for large perturbation radii, despite leading to an increase in the magnitude of the attacks.
Based on these observations, we propose a method called Noise-FGSM (N-FGSM), which attacks noise-augmented samples directly using a single-step.
Empirical analyses on a large suite of experiments show that N-FGSM is able to match or surpass the performance of GradAlign while achieving a 3x speed-up.
I NTRODUCTION
Deep neural networks have achieved remarkable performance on a variety of tasks (He et al., 2015;
Silver et al., 2016; Devlin et al., 2019).
However, it is well known that they are vulnerable to small worst-case perturbations around the input data – commonly referred to as adversarial examples (Szegedy et al., 2014).
The existence of such adversarial examples poses a security threat to deploying models in sensitive environments (Biggio & Roli, 2018).
This has motivated a large body of work towards improving the adversarial robustness of neural networks (Goodfellow et al., 2015;
Papernot et al., 2016; Tram ` er et al., 2018).
The most popular family of solutions to obtain robust neural networks is based on the concept of adversarial training (Goodfellow et al., 2015; Madry et al., 2018).
In a nutshell, adversarial training can be posed as a min-max problem where instead of minimizing some loss over a dataset of clean samples, we augment the inputs with worst-case perturbations that are generated online during training.
However, obtaining such perturbations is NP-hard (Weng et al., 2018) and hence, different approaches have been suggested to approximate them.
They are commonly referred to as adversarial attacks .
In their seminal work, Goodfellow et al.
(2015) proposed the Fast Gradient Sign
Method (FGSM), that generates adversarial attacks by running one step of gradient ascent on the loss function.
However, while FGSM-based adversarial training provides robustness against single-step
FGSM adversaries, Madry et al.
; Tram`er et al.
(2018) showed that these models were still vulnerable to multi-step attacks, namely those allowed to perform multiple gradient ascent steps instead of a single one.
Notably, Madry et al.
(2018) introduced the multi-step Projected Gradient
Descent (PGD) attack.
PGD-based attacks have now become the de facto standard for adversarial training; yet, their cost increases linearly with the number of steps.
As a result, several works have focused on reducing the cost of adversarial training by approximating the worst-case perturbations with single-step attacks (Wong et al., 2020; Shafahi et al., 2019; Vivek & Babu, 2020).
In particular, Wong et al.
(2020) studied FGSM adversarial training and discovered that it suffers from a characteristic failure ε k
FGSM
N-FGSM
RS-FGSM       16 for training and evaluation
A d v e r s a r i a l
A cc u r a c y
CIFAR10 Dataset
N-FGSM (ours)
GradAlign
MultiGrad ZeroGrad
Free-AT Kim
et.
al.
RS-FGSM FGSM
N - F G S
M
G r a d A li g n
M u l t i G r a d
Z e r o G r a d
F r eeA T
K i m e t .
a l .
R S - F G
S M
F G S M
T r a i n c o s t r e l a t i v e t
o F
G S M mode, in which a model suddenly becomes vulnerable to multi-step attacks despite remaining robust to single-step attacks.
This phenomenon is referred to as catastrophic overﬁtting .
Moreover, they argued that adding a random perturbation prior to FGSM (RS-FGSM) seemed sufﬁcient to prevent catastrophic overﬁtting and yield robust models.
Recently, Andriushchenko & Flammarion (2020) observed that RS-FGSM still leads to catastrophic overﬁtting as we increase the perturbation radii.
They suggested a regularizer (GradAlign) that, on the one hand avoids catastrophic overﬁtting in all the settings they considered, but on the other hand requires the computation of a double derivative – which signiﬁcantly increases the computational cost compared to RS-FGSM.
This has motivated other works that aim at achieving the same level of robustness with a lower computational overhead (Golgooni et al., 2021;
Kim et al., 2021).
In this paper, we revisit two key components that are common among previous works combining noise and FGSM (Tram ` er et al., 2018; Wong et al., 2020): the role of noise, i.e. the random step , and the role of the clipping step .
In Section 4.1, we study how these two components affect model robustness; our experiments suggest that adding noise with a large magnitude in the random step and removing the clipping step improves model robustness and prevents catastrophic overﬁtting, even against large perturbation radii.
We combine these observations and propose a new method called
Noise-FGSM (N-FGSM), an illustration of which is presented in Figure 1 (left).
N-FGSM allows to match, or even surpass, the robust accuracy of the regularized FGSM introduced by Andriushchenko & Flammarion (2020), while providing a 3 × speed-up .
To corroborate the effectiveness of our solution, we present an experimental survey of recently proposed single-step attacks and empirically demonstrate that N-FGSM trades-off robustness and computational cost better than other single-step approaches, evaluated over a large spectrum of perturbation radii (see Figure 1, middle and right panels), over several datasets (CIFAR-10, CIFAR100, and SVHN) and architectures (PreActResNet18 and WideResNet28-10).
We will release our code reproducing all experiments.
R ELATED WORK
Since the discovery of adversarial examples, many defense mechanisms have been proposed.
Preprocessing techniques try to modify the input image to neutralize adversarial attacks (Guo et al., 2018;
Buckman et al., 2018; Song et al., 2018).
Adversarial detection methods focus on detecting and rejecting adversarial attacks (Carlini & Wagner, 2017; Ma et al., 2018; Yang et al., 2020; Tian et al., 2021).
Certiﬁable defenses provide theoretical guarantees for the lower bound performance of networks subjected to worst-case adversarial attacks, however, they incur additional costs during inference and, empirically, they yield sub-optimal performance (Cohen et al., 2019; Wong &
Kolter, 2018; Raghunathan et al., 2018; Balunovic & Vechev, 2020).
Adversarial training methods are based on a special form of data augmentation designed to make the network robust to worst-case perturbations (Zhang et al., 2019; Athalye et al., 2018;
Kurakin et al., 2017).
However, computing a worst-case perturbation is an NP-hard problem that needs to be solved at every iteration.
To minimize the overhead of adversarial training, Goodfellow et al.
(2015) proposed FGSM which requires one additional gradient step per iteration.
Tram`er et
al.
ﬁrst proposed performing a random step before taking the adversarial step (R+FGSM), but they observed that neither method yields robust models against PGD attacks (Madry et al., 2018).
Since then, augmenting the training with PGD attacks has been one of the most popular approaches for robustness, but its cost increases linearly with the number of steps, which presents a severe practical limitation.
To reduce the cost of PGD, Shafahi et al.
(2019) proposed Free Adversarial Training (Free-AT), that exploits a single back-propagation step to both update the network parameters and compute the attack.
Wong et al.
(2020) explored a variation of R+FGSM, namely RS-FGSM, and showed it can yield robust networks against multi-step attacks.
Andriushchenko & Flammarion (2020) found that
RS-FGSM only works for limited perturbation radii and introduced GradAlign – a regularizer to linearize the loss surface.
However, optimizing GradAlign triplicates the computational cost.
This motivated a new series of works that aim at matching the performance of GradAlign without the additional computational overhead (Golgooni et al., 2021;
Kim et al., 2021).
Other strategies that attempted to improve FGSM included introducing dropout in every layer (Vivek & Babu, 2020) and perturbing intermediate feature maps together with the input (Park & Lee, 2021).
Li et al.
(2020) suggested combining RS-FGSM and PGD attacks during training, however, the proposed strategy requires a frequent monitoring of the PGD robust accuracy and, in the worst-case, is computationally equivalent to PGD training.
Gilmer et al.
Fawzi et al.
(2018) suggested a strong link between robustness to adversarial attacks and to random noise.
Motivated by this, we revisit the idea of combining noise and FGSM and propose N-FGSM.
Our method is closely related to RS-FGSM, however, we ﬁnd that using a larger amount of noise and removing the constraint that attacks must lie in the (cid:15) − l ∞ ball is key to obtaining robust models.
We note that Kang & Moosavi-Dezfooli (2021) concurrently studied
RS-FGSM without clipping, however, as opposed to our work, they did not investigate and provide insights on the impact of noise, and the learned models were not robust against large perturbations.
P RELIMINARIES ON SINGLE - STEP A DVERSARIAL T RAINING
Given a classiﬁer f θ : X → Y parameterized by θ and a perturbation set S , the classiﬁer f θ is said to be robust at x ∈ X under S if the following holds for all δ ∈ S : f θ ( x + δ ) =
f θ ( x ) .
One of the most popular deﬁnitions for S is the (cid:15) − (cid:96) ∞ ball, i.e. S =
{ δ : (cid:107) δ (cid:107) ∞ ≤ (cid:15) } .
This is known as the l ∞ threat model and is the setting we adopt throughout this work.
To train networks that are robust against (cid:96) ∞ threat models, adversarial training modiﬁes the classical training procedure of minimizing a loss function over a dataset D = { ( x i , y i ) } i =1: N of images x i ∈ X and labels y i ∈ Y .
In particular, adversarial training instead minimizes the worst-case loss over the perturbation set S , i.e. trains on adversarially perturbed samples { ( x i
+ δ i , y i ) } i =1: N .
When using the l ∞ threat model, we can formalize adversarial training as solving the following problem:
[Equation]
where L is typically the cross-entropy loss for image-classiﬁcation models.
Due to the difﬁculty of ﬁnding the exact inner maximizer, the most common procedure for adversarial training is to approximate the worst-case perturbation through several PGD steps (Madry et al., 2018).
While this has been shown to yield robust models, it comes at a cost of a linear increase in the computational overhead with the number of PGD steps.
As a result, several works have focused on reducing the cost of adversarial training by approximating the inner maximization with single-step attacks.
If we assume that the loss function is locally linear with respect to changes in the input, then the inner maximization of Equation (1) enjoys a closed form solution.
Goodfellow et al.
(2015) leveraged this result to propose the FGSM method, which takes one step in the direction of the sign of the gradient.
Tram`er et
al.
(2018) proposed adding a random initialization prior to FGSM.
However, both methods were later shown to be vulnerable against multi-step attacks, such as PGD (Madry et al., 2018).
Contrary to prior intuition, recent work from Wong et al.
(2020) observed that combining a random step with FGSM can actually lead to a promising robustness performance.
In particular, we note that most recent single-step methods approximate the worst-case perturbation that results from solving the inner maximization problem in Equation (1) with the following general form:
[Equation]
where
[Equation]
and Ω is the distribution from which we draw noise perturbations.
For example, when ψ is the projection operator onto the (cid:96) ∞ ball and Ω is the uniform distribution in [ − (cid:15), (cid:15) ] , this recovers RSFGSM with the following update:
[Equation]
η
[Equation]
On the other hand, with a different noise setting where Ω = ( (cid:15) − α ) · sign ( N ( 0 , I )) and by choosing the step size α to be in [0 , (cid:15) ] we recover R+FGSM by Tram`er et al.
(2018) that initially explored the idea of combining noise with FGSM but reported no improvement over adversarial training with FGSM.
If we consider Ω to be deterministically 0 and ψ to be the identity map, we recover the FGSM.
Finally, if we adjust the choice of the loss function L to include a gradient alignment regularizer, this recovers the GradAlign algorithm by Andriushchenko & Flammarion (2020).
N OISE AND FGSM
A common practice when performing adversarial training is to restrict the perturbations used during training to the same (cid:15) − (cid:96) ∞ ball that will be considered at test time.
The rationale behind it is that increasing the magnitude of perturbations could “unnecessarily” decrease the clean accuracy, since perturbations outside the ball will not be evaluated at test time.
For instance, R+FGSM combines the noise step, with magnitude ( (cid:15) − α ) , and the FGSM step, with magnitude α , in a convex combination manner, thereof, restricting the perturbation to (cid:15) .
On the other hand, Wong et al.
(2020) apply a clipping operation after the FGSM step to the (cid:15) − (cid:96) ∞ ball.
In the following, we experimentally challenge this common practice and explore the two key components in previous single-step methods that limit the magnitude of the perturbations.
In particular, we explore (i) the role of the clipping operation, i.e. ψ as a projection to (cid:96) ∞ ball; and (ii) the source and magnitude of noise for the random step, i.e. Ω .
We thoroughly revisit the role of both components on the robustness attained in singlestep methods.
Throughout this work, unless stated otherwise, we consider noise perturbations η sampled from a symmetric Uniform distribution, i.e. Ω =
U
[ − k, k ] d , where d is the dimension of
X and we refer to k as the “noise magnitude”.
4. T HE R OLE OF N OISE AND C LIPPING ON R OBUSTNESS IN S INGLE -S TEP M ETHODS
Clipping Reduces the Effectiveness of Single-Step Perturbations.
To study the impact clipping has on model robustness, we consider the training of PreActResNet18 (He et al., 2016) on
CIFAR-10 (Krizhevsky & Hinton, 2009) with an instance of Equation (2) as a single-step adversarial training.
In particular, we consider the case where ψ is a projection to the (cid:96) ∞ ball of size (cid:15) clip ∈ { (cid:15), 2 (cid:15), 3 (cid:15), ∞} , where ∞ denotes that ψ is an identity function, i.e. no clipping is performed.
Moreover, we consider noise sampled from a symmetric uniform distribution where k ∈ { (cid:15), 2 (cid:15), 3 (cid:15), 4 (cid:15) } .
We report in Figure 2 the robust accuracy using PGD-50-10 (i.e. PGD attack with 50 iterations and 10 restarts) with (cid:15) = 8 / 255 .
We observe in Figure 2 (left), that for all choices of noise magnitude k , as we expand the clipping (cid:96) ∞ ball, i.e. we increase (cid:15) clip , the adversarial robustness improves.
We believe that this is due to the fact that more aggressive clipping, i.e. smaller (cid:15) clip , reduces the strength of the computed single-step perturbations during training.
To support this intuition, we report in Figure 2 (middle) the distribution of the loss measured at perturbed points prior to applying the clipping step and after applying the clipping step, with (cid:15) clip = (cid:15) .
Moreover, the negative impact of clipping during training is more prominent as we increase the noise magnitude k .
This is to be expected since, for a ﬁxed α , increasing the noise magnitude k will lead to a prevalence of the noise component over the sign gradient direction in Equation (2). [Figure] a
[Figure]
Increasing the amount of noise is key to avoiding catastrophic overﬁtting.
plots adversarial accuracy is based on PGD-50-10 and experiments are averaged over 3 seeds.
Thus, overall, we observe that clipping during training has negative impact on the robust accuracy.
Despite unclipped perturbations may lie outside the (cid:15) − (cid:96) ∞ ball, we could not observe any signiﬁcant drop in clean accuracy.
In fact, we obtain comparable clean accuracy to GradAlign as later reported in Section 5.5. Further analyses can be found in Appendix C.
The role of noise in single-step adversarial training.
Neither Tram`er et al.
(2018) nor Wong et al.
(2020) explored settings with an increased noise magnitude.
Moreover, Andriushchenko &
Flammarion (2020) argue that noise in RS-FGSM is not important per se, claiming that its main purpose is only reducing the (cid:96) 2 norm of the ﬁnal perturbation δ RS-FGSM so that the loss is still in the linear regime.
However, we empirically ﬁnd that increasing the noise magnitude is key to avoiding catastrophic overﬁtting – see Figure 2 (right).
Investigating further, similarly to Kim et al.
(2021), we plot the loss surface at the end of training (see Figure 12 in Appendix J) and ﬁnd that, as observed by Kim et al.
(2021), the loss surface of models trained via FGSM or RS-FGSM appears distorted at the end of training, i.e. the loss increases sharply along the FGSM direction, but then it rapidly decreases back to the same loss values associated with the clean sample.
This is consistent with the gradient obfuscation observed by
Tram ` er et al.
However, when training without clipping with an increased noise magnitude, we observe a non-distorted loss surface, i.e. the loss gradually increases along the FGSM direction.
Interestingly, we observe a similar effect when training with the GradAlign regularizer.
Thus, combining noise with FGSM seems to have a regularizing effect that encourages the loss surface to be locally linear in a similar spirit to GradAlign.
Note this is based on the empirical analyses and visual inspections that we performed.
We do not provide theoretical justiﬁcation behind this.
Despite the clear beneﬁts of increasing the noise magnitude, we do observe a slight but consistent decrease in robustness as we keep on increasing the noise, which we hypothesize is due to overregularization.
We ﬁnd k
= 2 (cid:15) to be the sweet spot in most of our experiments; we do not exclude that a more extensive hyperparameter tuning procedure may lead to improved results.
4. O UR APPROACH
In the previous section, we provided experimental analyses to show that increasing the noise magnitude and not clipping prevents catastrophic overﬁtting and improves robustness signiﬁcantly.
Based on these observations, we propose the following, simple and efﬁcient, single-step method for adversarial training that we denote as Noise-FGSM (N-FGSM):
[Equation]
We detail our full adversarial training procedure in Algorithm 1.
Theoretical insights We now theoretically analyse N-FGSM to understand the role of noise in single-step approaches.
According to Andriushchenko & Flammarion (2020), the main contribution
1: Inputs: epochs T , batches M , radius (cid:15) , step-size α (default: (cid:15) ), noise magnitude k (default: 2 (cid:15) ).
2: for t = 1 , . . .
, T do 3: for i = 1 , . . .
, M do 4: //
Perform N-FGSM adversarial attack 5: η ∼ Uniform [ − k, k ] d 6: δ =
η + α · sign (cid:0) ∇ x i L ( f θ ( x i + η ) , y i ) (cid:1) 7: ∇ θ = ∇ θ L ( f θ ( x i + δ ) , y i ) 8: θ = optimizer ( θ, ∇ θ ) //
standard weight update, (e.g. SGD) 9: end for 10: end for of noise in single-step adversarial training is to reduce the effective (cid:96) 2 norm of the perturbation δ .
Since N-FGSM does not involve clipping, we ﬁnd the expected norm squared of N-FGSM perturbations to be larger than RS-FGSM perturbations.
In fact, it is even larger than the FGSM perturbations which always lie on the (cid:15) − (cid:96) ∞ ball.
These observations are formalized in Theorem 1.
Theorem 1.
Let δ N-FGSM be our proposed single-step method deﬁned by Equation (4) , δ FGSM be the
FGSM method (Goodfellow et al., 2015) and δ RS-FGSM be the RS-FGSM method (Wong et al., 2020).
Then, with default hyperparameter values and for any (cid:15) > 0 , we have that
[Equation]
[Table]
We present the Proof in Appendix F where we also compute an empirical estimation of the expected (cid:96) 2 norm of different methods via Monte Carlo sampling and ﬁnd them to align with Theorem 1.
Differently from the hypothesis of Andriushchenko & Flammarion (2020), we ﬁnd that despite that
N-FGSM perturbations have larger (cid:96) 2 norms, they yield robust models even under larger (cid:15) radii, for which RS-FGSM or FGSM fail to catastrophic overﬁtting.
We believe that these contradictory observations can lead to a better understanding on the role of noise in adversarial training and catastrophic overﬁtting in future work.
E XPERIMENTS AND ANALYSES
We compare N-FGSM against several adversarial training methods, considering a broad range of (cid:15) − l ∞ radii.
Following Wong et al.
(2020); Andriushchenko & Flammarion (2020)
, we measure adversarial robustness on CIFAR-10/100 (Krizhevsky & Hinton, 2009) and SVHN (Netzer et al., 2011) datasets with PGD-50-10 attack – PGD (Madry et al., 2018) with 50 iterations and 10 restarts.
5. C OMPARISON TO OTHER SINGLE - STEP METHODS
We start by comparing N-FGSM against other single-step methods.
Note that not all single-step methods are equally expensive, since they may involve more or less computationally demanding operations.
For instance, GradAlign (Andriushchenko & Flammarion, 2020) relies on a regularizer that is considerably expensive, while, MultiGrad (Golgooni et al., 2021) requires evaluating the input gradients on multiple random points.
For a comparison of training cost over different singlestep methods, we refer to Figure 1 (right).
Following the standard practice (Wong et al., 2020;
Andriushchenko & Flammarion, 2020), we use a PreactResNet18 architecture (He et al., 2016).
We use RS-FGSM and Free-AT with the settings recommended by Wong et al.
We apply
GradAlign following the hyperparameters reported in the ofﬁcial repository 1 .
Golgooni et al.
recommend applying MultiGrad with n = 3 random samples, but do not provide a default hyperparameter setting for what concerns the ZeroGrad variant.
Also Kim et al.
(2021) do not recommend a set of hyperparameters; for a fair comparison, we ablate them and select the ones that provide the highest adversarial accuracy (for every combination of (cid:15) and dataset).
We train on CIFAR-10/ for 30 epochs and on SVHN for 15 epochs with a cyclic learning rate.
Only for Free-AT, we useand 48 epochs for CIFAR-10/100 and SVHN, respectively, to obtain comparable results following
16 for training and evaluation
A d v e r s a r i a l
A cc u r a c y
CIFAR100 Dataset N-FGSM (ours) GradAlign MultiGrad ZeroGrad
Free-AT Kim et.
al.
RS-FGSM FGSM     12 for training and evaluation
A d v e r s a r i a l
A cc u r a c y
SVHN Dataset
Wong et al.
and Andriushchenko & Flammarion (2020).
CIFAR-10 results are presented in Figure 1 (middle), whereas CIFAR-100 and SVHN results are reported in Figure 3.
As previously observed, FGSM and RS-FGSM both suffer from catastrophic overﬁtting on larger (cid:15) attacks.
In contrast, N-FGSM prevents catastrophic overﬁtting and enjoys robustness properties comparable or superior to GradAlign for all (cid:15) , while being 3 times faster.
With appropriate hyperparameters, ZeroGrad is able to avoid catastrophic overﬁtting but obtains sub-obtimal robustness – especially for large perturbations.
Neither MultiGrad nor the method proposed by Kim et al.
(2021) can avoid catastrophic overﬁtting in all settings.
We also observe that Free-AT cannot overcome catastrophic overﬁtting as also observed by Andriushchenko & Flammarion (2020).
5. R ANDOMIZED A LPHA
Kim et al.
(2021) evaluate intermediate points along the RS-FGSM direction in order to pick the “optimal” perturbation size.
However, we ﬁnd that increasing the number of intermediate evaluated points does not necessarily lead to increased adversarial accuracy.
Moreover, for large perturbations we could not prevent catastrophic overﬁtting even with twice the number of evaluations tested by
Kim et al.
This motivates us to test a very simple baseline where instead of evaluating intermediate steps, the RS-FGSM perturbation size is randomly selected as: δ = t · δ RS-FGSM where t ∼ U [0 , 1] d .
Interestingly, as reported in Figure 4 (left), we ﬁnd that this very simple baseline, dubbed RandAlpha , is able to avoid catastrophic overﬁtting for all values of (cid:15) and outperforms Kim et al.
(2021) on CIFAR-10.
This is aligned with our main ﬁnding that combining noise with adversarial attacks is indeed a powerful tool that should be explored more thoroughly before developing more expensive solutions.
We reach the same conclusions for CIFAR-100 and SVHN in Appendix H.
5. H YPERPARAMETER SELECTION
While FGSM relies on a ﬁxed step-size (equal to the maximum radius of perturbation to be used at test time, i.e. α = (cid:15) ), Wong et al.
(2020) explored different values for α during the development of RS-FGSM ﬁnding that an increase of the step-size improves the adversarial accuracy – up to a magnitude before catastrophic overﬁtting occurs.
We also ablate the value of α for N-FGSM in Figure 4 (middle).
We ﬁnd that by increasing the noise magnitude, N-FGSM can use larger α values than RS-FGSM, without suffering from catastrophic overﬁtting.
As observed by Wong et al.
(2020), this in turn leads to an increase in the adversarial accuracy at the expense of a decrease in the clean accuracy.
In light of this trade-off and following FGSM, we also use α = (cid:15) for N-FGSM.
Regarding N-FGSM noise hyperparameter, we ﬁnd k = 2 (cid:15) works in all but one SVHN experiment ( (cid:15)
= 12 , in which we set k = 3 (cid:15) )
, reducing the need for expensive hyperparameter tuning.
In comparison, GradAlign regularizer hyperparameter or ZeroGrad quantile value need to be deﬁned for every radius with a noticeable shift between CIFAR-10 and SVHN hyperparameters, suggesting they may require additional tuning when applied to novel datasets.
[Figure]
a r i a l
[Figure]
cc u r a c
[Figure]
Comparison of
[Figure]
5. L ONG VS FAST TRAINING SCHEDULE
Throughout our experiments, we used the RS-FGSM training setting introduced in Wong et al.
However, Rice et al.
(2020) suggest that a longer training schedule coupled with early stopping may lead to a boost in performance.
Kim et al.
(2021) and Li et al.
(2020) report that longer training schedules increase the chances of catastrophic overﬁtting for RS-FGSM and that this limits its performance.
We test the longer training schedule with N-FGSM and ﬁnd that it presents robust overﬁtting , i.e. the adversarial accuracy on the training set keeps increasing but it decreases when evaluated on the test set as described in Rice et al.
However, it does not suffer from catastrophic overﬁtting.
In Figure 4 (right), we show the results for (cid:15) = 8 .
Although we do observe a slight increase in performance when using the long training schedule, we ﬁnd the performance remarkably competitive when considering the fast one which seems to avoid robust overﬁtting (when relying on early stopping, the best models are usually found at the end).
Thus, it might be preferable to consider the fast training schedule if computational cost is an important factor.
In Appendix E, we report a robust accuracy of 47.86 ± 0.1 for GradAlign with the longer schedule compared to 48.48 ± 0.27 for N-FGSM.
Note that, in order to prevent GradAlign from suffering from catastrophic overﬁtting for (cid:15) = 8 , we had to increase the regularizer hyperparameter (compared to the fast schedule), while N-FGSM is able to prevent catastrophic overﬁtting with the same settings.
5. C OMPARISON TO MULTI - STEP ATTACKS
In Section 5.1, we compared the performance of single-step methods and observed that N-FGSM is able to match or surpass the state-of-the-art method, i.e. GradAlign, while reducing the computational cost by a factor of 3 × .
In this section, we compare the performance of N-FGSM with multi-step attacks.
We use PGD-2 with α = (cid:15) / 2 and PGD-10 with α = 2 following Wong et al.
(2020) and keep the same training settings as described in Section 5.1 (note that PGD-x denotes
PGD attack with x iterations and no restarts).
In Figure 5, we observe that PGD-2 also presents catastrophic overﬁtting when we increase the perturbation radius (cid:15) , which is consistent with results reported by Andriushchenko & Flammarion (2020).
On the other hand, despite all methods achieving comparable clean accuracy, there is a gap on adversarial accuracy between PGD-10 and single-step methods which grows with the perturbation size.
This can be partially expected since the search space grows exponentially with (cid:15) and PGD can explore it more thoroughly as we increase the number of iterations.
Nevertheless, computing a
PGD-10 attack is 10 × more expensive than computing an N-FGSM one .
An important direction for future work should be addressing this gap and analyse, both theoretically and empirically, whether single-step methods can actually match the performance of their multi-step counterparts.
[Figure]
5. E XPERIMENTS WITH W IDE R ES N ET 28-10 ARCHITECTURE
We also compare the performance of all methods on WideResNet28-10 (Zagoruyko & Komodakis, 2016) architecture in Figure 7 and Figure 8 (Appendix B).
As in the experiments with the PreActResNet18 architecture, N-FGSM obtains state-of-the-art PGD-50-10 accuracy among single-step methods.
Nevertheless, as a general trend, we observe that catastrophic overﬁtting seems to be more difﬁcult to prevent when using WideResNet.
For instance, FGSM is able to consistently yield robust models up to (cid:15) =
6 for PreActResNet18 on CIFAR-10, however, for some runs the same radius can lead to catastrophic overﬁtting for WideResNet models.
We hypothesize this is because it is more over-parametrized –WideResNet28-10 has 36.5 M parameters, whereas PreActResNet18 has 11.2M. Regarding GradAlign, we had to increase the regularizer hyperparameter (compared to the settings for PreActResNet18) in order to prevent catastrophic overﬁtting on CIFAR-100.
Note that, to our surprise, we could not ﬁnd a competitive hyperparameter setting for GradAlign on the SVHN dataset for (cid:15) ≥ 6 .
We tried both increasing the regularizer hyperparameter and decreasing the step size α , but some or all runs led to models close to a constant classiﬁer for each setting.
We do not claim that GradAlign will not work, but ﬁnding a good conﬁguration might require further tuning.
On the other hand, as observed earlier, the default conﬁguration for N-FGSM ( α = (cid:15), k = 2 (cid:15) ) works well in all settings except for (cid:15)
=
16 on CIFAR-10 and (cid:15) = 10 , 12 on SVHN.
For CIFAR-10, we increase the noise magnitude to k = 4 (cid:15) .
For SVHN, we ﬁnd that decreasing α as we tried for
GradAlign works better than increasing the noise.
However, in both cases N-FGSM can obtain more than a trivial adversarial accuracy.
Results are presented in Appendix B.
D ISCUSSION
In this work, we explore the role of noise and clipping in single-step adversarial training.
Contrary to previous intuitions, we show that increasing the noise magnitude and removing the (cid:15) − (cid:96) ∞ constraint allows improving adversarial robustness, while maintaining a competitive clean accuracy.
These ﬁndings led us to propose N-FGSM, a simple and effective approach that can match or surpass the performance of GradAlign (Andriushchenko & Flammarion, 2020), while achieving a 3 × speed-up.
We perform an extensive comparison with other relevant single-step methods, observing that all of them achieve sub-optimal performance and most of them are not able to avoid catastrophic overﬁtting for larger (cid:15) attacks.
Moreover, we also analyze recent single-step methods and – inspired by
Kim et al.
(2021) – observe that uniformly choosing a step-size in [0 , α ] avoids catastrophic overﬁtting for RS-FGSM, which reinforces our intuition that random noise is a powerful tool towards learning robust models.
However, despite impressive improvements of single-step adversarial training methods, there is still a gap between single-step and multi-step methods such as PGD-10 as we increase the (cid:15) radius.
Therefore, future work should put an emphasis on formally understanding the limitations of single-step adversarial training and explore how, if possible, this gap can be reduced.
Ethics Statement.
The existence of adversarial examples poses a potential threat to the deployment of deep learning systems into the real world.
Therefore, ﬁnding fast and effective methods to train models robust against this threat is of utmost importance.
On the other hand, adversarial training methods are based on augmenting samples with adversarial perturbations, thus, they are partially based on building methods to attack neural networks.
Research on adversarial attacks is naturally a sensitive path, since it can potentially be exploited for unethical purposes.
However, the scope of our work is not designing stronger attacks; rather, the methods we propose are designed ad hoc to improve the adversarial robustness of learning systems, not to break other models’ defenses.
Therefore, although we are aware that we are carrying out research within a sensitive topic, we are not particularly concerned that the research presented in this paper can lead to harmful applications – on the contrary, we believe that it can help deploying safer machine learning applications.
Reproducibility Statement.
In this paper, we compare against several adversarial training methods.
The general training and evaluation settings used are described at the beginning of Section 5.
Moreover, different hyperparameters have been used for each method depending on the dataset and network.
These are detailed in the manuscript, where corresponding results are discussed.
For all the reported experiments, we employ different random seeds to ensure replicability of our results, and report performances indicating the standard deviation values.
To facilitate further research on this topic, as stated in Section 1, we will release the code to reproduce all experiments.
Regarding our theoretical results, we deferred the proofs to Appendix F.
A A DDITIONAL PLOTS FOR P RE A CT R ES N ET 18 EXPERIMENTS
In the main paper we compare N-FGSM with other single-step methods and multi-step methods separately and remove clean accuracies for better visualization.
In this section we present the curves for all methods with both the clean and robust accuracy.
The tendency in the three datasets is for
N-FGSM PGD-50-10 accuracy to be slightly above that of GradAlign, while the opposite happens to the clean accuracy.
We also observe that clean accuracy becomes signiﬁcantly more noisy when catastrophic overﬁtting happens.
Exact numbers for all the curves are in Appendix L
[Figure]
-the-art single-step method GradAlign while reducing the cost by a 3 × factor .
Adversarial accuracy is based on PGD-50-10 and experiments are averaged over 3 seeds.
Legend is shared among all plots.
B E XPERIMENTS WITH W IDE R ES N ET 28-10 ARCHITECTURE
In this section we present the plots of our experiments with WideResNet28-10.
We report the results in two ﬁgures.
In Figure 7 we compare all single-step methods and we do not plot the clean accuracy for better visualization.
In Figure 8 we plot all methods, including multi-step methods, and report the clean accuracy as well with dashed lines.
Since we observed that our baseline, RandAlpha, outperformed Kim et al. (2021) in all settings for PreActResNet18, we only report RandAlpha for WideResNet.
As mentioned in the main paper, we observe that catastrophic overﬁtting seems to be more difﬁcult to prevent for WideResNet.
In particular, for GradAlign we observed the regularizer hyperparameter settings proposed by Andriushchenko & Flammarion (2020) for CIFAR- (searched for a PreActResNet18) worked well.
However, those parameters led to Catastrophic Overﬁtting for 6 ≤ (cid:15) ≤ 12 in CIFAR-100.
Since (cid:15) = 14 , 16 did not show Catastrophic Overﬁtting, we increased the GradAlign regularizer hyperparameter λ for CIFAR-100 so that each 6 ≤ (cid:15) ≤would have the default value corresponding to (cid:15) + 2 , for instance, λ for (cid:15) = 6 would be the default λ in Andriushchenko & Flammarion (2020) for (cid:15)
= 8 .
[Figure] For SVHN we observed that the default values for λ led to models close to a constant classiﬁer for (cid:15) ≥ 6 .
We tried to increase the lambda for those (cid:15) values to 1 .
25 λ but observed the same result.
Since the model did not show typical catastrophic overﬁtting but rather it seemed as it was underﬁtting, we tried to reduce the step-size to α = 0 .
75 (cid:15) and also both decreasing α and increasing λ .
When reducing the step size we obtain accuracies above those of a constant classiﬁer for some radii, however, some or all seeds converge to a constant classiﬁer for each setting, hence the large standard deviations.
For N-FGSM, the default conﬁguration of N-FGSM ( α = (cid:15), k = 2 (cid:15) ) works well in all settings except for (cid:15)
=
16 on CIFAR-10 and (cid:15) = 10 , 12 on SVHN.
For CIFAR-10, we increase the noise magnitude to k = 4 (cid:15) .
For SVHN we ﬁnd that decreasing α as we tried for
GradAlign works better than increasing the noise.
We use α = 8 for both (cid:15) radii.
Exact numbers for all the curves are in Appendix L
C F URTHER ABLATION OF NOISE AND STEP SIZE IN N-FGSM
In Section 4.1, we observed that both removing clipping and increasing noise were necessary to avoid catastrophic overﬁtting.
However, when doing that, we increase the magnitude of the perturbations.
In this section, we study more closely the interplay between the step-size α and the noise level k to ensure that it is indeed the increase in noise level, rather than merely increasing the magnitude of perturbations, what helps stabilize adversarial training and avoid catastrophic overﬁtting.
We ﬁx (cid:15) test = 8 / 255 for evaluation, and report the clean and robust accuracy of N-FGSM under different combinations of noise level k and step size α .
Note that when the noise level k = N-FGSM recovers plain FGSM, thus, as we increase α it is equivalent to using FGSM with an increased (cid:15) train . On the other hand, when α = 0 , this is equivalent to training with only random noise augmentation. Based on the results reported in Table 1, we make the following observations:
[Figure]
1) Increasing the perturbation size is not enough to avoid catastrophic overﬁtting .
For instance, as observed in the ﬁrst column of Table 1, training with an increasing α without noise, i.e. ( k =which is equivalent to FGSM), leads to catastrophic overﬁtting despite the clear increase in perturbation size due to the increase of α .
Catastrophic overﬁtting leads to a vulnerable model against smaller perturbations too .
For instance, looking at the experiments with α = 10 or α = 12 in the ﬁrst column, we observe that the resultant models are vulnerable to adversarial attacks for (cid:15) test = 8 / 255 even though the perturbation radius used in training is larger than 8 / 255 .
This indicates that once a model catastrophically overﬁts to perturbations of a given radius, it can be vulnerable to smaller perturbations too.
3) Increasing the level of noise is necessary to stabilize training for larger perturbations .
As we increase the step size α of the attack, we observe that we need to increase the ratio between noise k and step-size α in order to avoid catastrophic overﬁtting.
This again further suggests that it is indeed increasing the perturbation size by increasing the noise level (and not simply increasing the perturbation budget) what mitigates catastrophic overﬁtting.
4) Training with noise perturbations has a much milder effect on the clean accuracy than adversarial training .
Despite N-FGSM perturbations have a larger radius, it does not result in a signiﬁcant drop of the clean accuracy, see Figure 5.
Although it might seem counter-intuitive, we note that while N-FGSM has a larger perturbation size, this increase is not only merely in the adversary direction but also in the random noise direction.
That is to say, while the perturbation size is larger for N-FGSM, this is not necessarily equivalent to augmenting with adversaries with an a similarly larger perturbation size of adversaries due to the bias in the noise.
In Table 1, we observe how augmenting training samples with noise alone (when α = 0 ) has a much milder effect on the clean accuracy.
In general, moving right on the table (increasing noise) is more forgiving on the clean accuracy than moving downwards (increasing FGSM step size).
This is not so surprising considering that moving in random directions along the input space has a much lower impact on the loss than moving along the FGSM direction (see Figure 12) and that training with noise alone does not provide any signiﬁcant robustness against larger attacks (for a more detailed ablation see Figure 11).
[Table]
noise level k and FGSM step size α .
Results are averaged over 3 seeds.
All models are evaluated with PGD-50-10 attack and (cid:15) test =
(cid:15) train = 8 / 255 .
D A BLATION OF BASELINES INCREASING THE TRAINING EPSILON
We have seen in Theorem 1 and Appendix F that the magnitude of N-FGSM perturbations will (on expectation) be larger than that of other baselines.
Moreover, since N-FGSM does not use clipping to ensure perturbations are within the (cid:15) − (cid:96) ∞ ball, they could have (cid:96) ∞ − norm of up to k + α ; which for the default hyperparameter values ( α = (cid:15), k = 2 (cid:15) ) add up to 3 (cid:15) .
In this section, we study the performance of all single-step baselines as we increase the training epsilon.
In Table 2, we present the result of an experiment where we ﬁx (cid:15) test = 8 / 255 , then for each baseline we increase the (cid:15) train from 8 / 255 to 16 / 255 (while always evaluating the ﬁnal model at 8 / 255 ).
We use the same training hyperparameters as reported in Section 5.1.
Results lead to three main observations:
1) Increasing (cid:15) train ends up hurting test robust accuracy .
Even though in some methods we observe an increase in adversarial accuracy as we start to increase (cid:15) train , it ends up decreasing.
Moreover, this increase in adversarial accuracy (see GradAlign, MultiGrad or N-FGSM) is at the expense of a decrase in clean accuracy.
Catastrophic overﬁtting leads to a vulnerable model to smaller perturbations as well .
As observed in Appendix C, those models which suffer catastrophic overﬁtting become vulnerable to attacks with smaller perturbation radius than used during training.
Here, we observe the same effect for various single-step baselines, which indicates this is likely a general trend.
Changing N-FGSM step-size α leads to similar result to changing (cid:15) train in other baselines .
Interestingly, we observe that although the presence of noise will lead to perturbations much larger than those of other methods, it is the step-size in the direction of the FGSM attack that will have a larger impact on robust and clean accuracy.
Thus, for larger perturbations the role of noise seems to be mainly to somehow mitigate catastrophic overﬁtting rather than strongly contributing to model robustness.
In light of the previous observations, we conclude that using a larger perturbation for N-FGSM does not lead to an unfair comparison to other baselines, on the contrary, the best values for these baselines (considering the trade-off between clean and robust accuracy) are when using the same radius of perturbations for training and test ( (cid:15) test =
(cid:15) train ).
On the other hand, this result highlights the particularity of increasing the perturbation size with noise rather than following the adversarial
[Table]
-step methods when increasing the (cid:15) train .
All models are evaluated with PGD-50-10 attack and (cid:15) test = 8 / 255 .
Note that considering the trade-off between clean and robust accuracy, all methods perform best when training with the same epsilon to be applied at test time.
direction.
Gaining a deeper understanding of the role of noise in avoiding catastrophic overﬁtting is a promising direction for future work.
In our experiments, we have followed the “fast” training schedule introduced by Wong et al.
However, Rice et al.
(2020) suggest that a longer training schedule coupled with early stopping may lead to a boost in performance.
We also use the long training schedule for N-FGSM and observe that it does not lead to catastrophic overﬁtting.
In Table 3 we compare the performance of N-FGSM and
GradAlign for the long training schedule.
We observe that GradAlign does not seem to beneﬁt from the long training schedule.
On the other hand, although N-FGSM seems to obtain a slight increase in performance, the “fast” schedule provides comparable performance.
It is worth mentioning that for GradAlign, the default regularizer hyperparameter for (cid:15) = 8 / 255 and CIFAR-10 ( λ = 0 . 2 ) does not prevent catastrophic overﬁtting.
We do a hyperparameter search and keep the value with the largest ﬁnal robust accuracy ( λ = 0 . 632 ).
[Table]
Although N seems to obtain a slight increase
F M AGNITUDE OF N-FGSM PERTURBATIONS
Lemma 1 (Expected perturbation) .
Consider the N-FGSM perturbation as deﬁned in Equation (4) δ N-FGSM =
η + α · sign ( ∇ x (cid:96) ( f ( x + η ) , y )) , where η ∼ Ω .
Let the distribution Ω be the uniform distribution U (cid:16) [ − k(cid:15), k(cid:15) ]
d (cid:17) and α > 0 .
Then,
[Equation]
and
[Equation]
Proof.
By Jensen’s inequality, we have
[Equation]
Then let us consider the term E η (cid:2) (cid:107) δ N-FGSM (cid:107) 22 (cid:3) and use the shorthand ∇ ( η ) i = ( ∇ x (cid:96) ( f ( x + η ) , y ))
i .
E η (cid:2) (cid:107) δ N-FGSM (cid:107) 22 (cid:3) =
E η (cid:107) η
+ α · sign ( ∇ x (cid:96) ( f ( x + η ) , y ))
(cid:107)= E η (cid:34) d (cid:88) i = (
[Equation]
i + α · sign ( ∇ (
[Equation]
i
[Equation] [Equation]
=
[Equation]
i =
[Equation]
E η (cid:104) ( η i + α · sign ( ∇ ( η ) i )) 2 (cid:105) =
d (cid:88) i =
E η (cid:104) ( η i + α · sign ( ∇ ( η ) i ))
2 | sign ( ∇ ( η ) i ) =
1 (cid:105) P η [ sign ( ∇ ( η ) i ) = 1] + d (cid:88) i =
E η (cid:104) ( η i + α · sign ( ∇ ( η ) i ))
2 | sign ( ∇ ( η ) i ) = − 1 (cid:105) P η [ sign ( ∇ ( η ) i ) = − 1] =
d (cid:88) i =  2 k(cid:15) (cid:90) k(cid:15) − k(cid:15)
( η i + α ) 2 dη i · P η [ sign ( ∇ ( η ) i ) = 1] +2 k(cid:15) d (cid:88) i = (cid:90) k(cid:15) − k(cid:15) ( η i − α ) 2 dη i · P η [ sign ( ∇ ( η ) i ) = − 1]
= d (cid:88) i =  2 k(cid:15) (cid:90) α + k(cid:15) α − k(cid:15)
z 2 dz · P η [ sign ( ∇ ( η ) i ) = 1] +2 k(cid:15) d (cid:88)
i = (cid:90) − α + k(cid:15)
− α − k(cid:15)
z 2 dz · P η [ sign ( ∇ ( η ) i ) = − 1] =
d (cid:88) i =  2 k(cid:15) (cid:90) α + k(cid:15)
α − k(cid:15)
z 2 dz · P η [ sign ( ∇ ( η ) i ) = 1] +2 k(cid:15)
[Equation]
Therefore,
[Equation]
We state again Theorem 1 and present the proof.
Theorem 1.
Let δ N-FGSM be our proposed single-step method deﬁned by Equation (4) , δ FGSM be the
FGSM method (Goodfellow et al., 2015) and δ RS-FGSM be the RS-FGSM method (Wong et al., 2020).
Then, with default hyperparameter values and for any (cid:15) > 0 , we have that
[Equation]
Proof.
From Lemma 1 we have that
[Equation]
On the other hand
[Equation]
Andriushchenko & Flammarion (2020) showed that
[Equation]
Finally, we note that
[Equation]
The default hyperparameters for N-FGSM are k = 2 , α = (cid:15) and RS-FGSM uses α = 5 (cid:15)/ 4 .
With these hyperparameters and any (cid:15) > 0 we have
[Equation]
In Lemma 1 we compute the expected value of the squared (cid:96) 2 norm of N-FGSM perturbations and by
Jensen’s inequality we obtain an upper bound for the expected (cid:96) 2 norm of N-FGSM perturbations.
However, obtaining the exact expected magnitude is more complex.
To compliment our analytic results, we approximate the (cid:96) 2 norm of FGSM, RS-FGSM and N-FGSM via Monte Carlo sampling.
Results are presented in Figure 9.
We observe that the empirical estimations are very close to the analytical upper bounds and that indeed, N-FGSM has a magnitude signiﬁcantly above that of FGSM or RS-FGSM.
G N-FGSM WITH G AUSSIAN NOISE
In the main paper we have only explored noise sources coming from a Uniform distribution.
Since we are measuring robustness against l ∞ − attacks, the Uniform distribution is a natural choice because the random perturbations will be bounded to the l ∞ ball deﬁned by the span of the distribution.
However, for the sake of completeness, we also explore the performance of augmenting the samples from a Gaussian distribution where we choose its standard deviation to match that of the uniform distribution.
In Table 4 we present a comparison of the clean (top) and PGD-50-10 (bottom) accuracy for different values of α and noise magnitude with (cid:15)
= 8 / 255 .
Recall that by default we use
Uniform distribution U [ − k, k ] , therefore hyperparameter k sets the noise magnitude.
Increasing the FGSM step size without increasing the amount of noise leads to catastrophic overﬁtting.
Note results for k = 0 .
5 (cid:15) .
More importantly, results are very similar when the two noise distributions share the same standard deviation.
Thus, using Gaussian instead of Uniform noise does not seem to alter the results.
Although this might be expected, we remark that the Gaussian is an unbounded noise distribution and the common practice in adversarial training is to always restrict the norm of the perturbations.
[Figure]
Uniform Noise
Gaussian Noise α = 6 / 255 ( 0 .
75 (cid:15) )
α = 8 / 255 ( 1 (cid:15) )
α = 10 /
( 1 . 25 (cid:15) )
α = 6 {+/ 
+}
75 (cid:15) )
α = 8 / 255 ( 1 (cid:15) )
α = 10 /
( 1 . 25 (cid:15) )
k
= 0 .
5 (cid:15) [Figure] 52 ± 0.
44.14 ± 0.
[Figure]
.54 ± 0.
47.93 ± 0.
82.81 ± [Figure]
0.0 ± 0.
85.27 ± [Figure]
[Figure] 23 ± 0.
81.71 ± 0.
47.98 ± 0.
83.34 ± 1.
0.0 ± 0.
k = 1 (cid:15) [Figure] ± 0.
44.44 ± 0.
81.57 ± 0.
48.16 ± 0.
77.32 ± 0.
49.68 ± 0.
[Table]
H F URTHER RESULTS WITH R AND A LPHA
In Section 5.2 we analyze the method presented by Kim et al.
and suggest a baseline where, instead of evaluating intermediate points to determine the “optimal” step size we simply choose it randomly.
That is
[Table]
we multiply the RS-FGSM perturbation by a random scalar sampled from a uniform distribution in [0 , 1] .
Interestingly we ﬁnd that it outperforms Kim et al [Table]
(2021) without the additional cost of intermediate evaluations and without the need to perform hyperparameter selection to ﬁnd the optimal number of intermediate evaluations [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
I T RAINING WITH NOISE AUGMENTED SAMPLES
Gilmer et al.
(2019) and Fawzi et al.
(2018) report a close link between robustness to adversarial attacks and robustness to random noise.
Actually, Gilmer et al.
(2019) report that training with noiseaugmented samples can improve adversarial accuracy and
vice-versa.
We note that N-FGSM can actually be seen as a combination of noise-augmentation and adversarial attacks.
Here we perform an ablation where we train models with samples augmented with uniform noise U [ − k, k ] and then test the PGD-50-10 accuracy.
We observe, that indeed random noise can increase the robustness to wort-case perturbations for small (cid:15) − l ∞ balls.
However, as we increase (cid:15) , noise augmentation is no longer very effective.
However, with N-FGSM, we apply a weak attack to these noise-augmented samples and this seems to be enough to make them effective for adversarial training.
J V ISUALIZATION OF THE LOSS SURFACE
In this section we present a visualization of the loss surface.
We adapted the code from Kim et al.
(2021) to analyse the shape of the loss surface at the end of training for different methods.
Kim et al.
(2021) reported that after adversarial training catastrophic overﬁtting, the loss surface would become non-linear.
In particular, they found that the FGSM perturbation seems to be misguided by local maxima very close to the clean image that result in ineffective attacks.
We note this was already reported by Tram ` er et al.
(2018) which proposed to perform a random step to escape those maxima.
We argue that adding noise to the random step, when properly implemented, actually prevents those maxima to appear in the ﬁrst place.
K C OMPARISON OF ADVERSARIAL TRAINING COST
In this section we describe how we compute the relative training cost for single-step methods shown in Figure 1 (right).
We approximate the cost based on the number of forward/backward passes each method uses, disregarding the cost of other additional operations such as adding a random step for
RS-FGSM or N-FGSM.
We understand these operations have a negligible cost compared to a full forward or backward pass.
FGSM:
FGSM is the cheapest of all methods since it only uses one forward/backward to compute the attack and an additional forward/backward to compute the weight update.
Hence, Cost FGSM =
2 F/B.
RS-FGSM:
As previously mentioned, we do not take into account the cost of random steps or clipping, hence we consider RS-FGSM to have the same cost as standard FGSM.
Cost RS-FGSM =
2 F/B.
N-FGSM:
Idem as before, cost of N-FGSM =
2 F/B.
ZeroGrad:
For ZeroGrad they need to do an additional sorting operation to ﬁnd the smallest gradient components.
This could be potentially expensive, however, since the size of the input image is
[Figure] [Figure]
several orders of magnitude smaller than that of the network, we also ignore this cost.
Cost ZeroGrad = 2 F/B.
MultiGrad: MultiGrad computes 3 random steps and evaluates the gradient in all of them.
Therefore, it needs to do 3 F/B to compute the attack and an additional one to update the weights.
Cost
MultiGrad = 4 F/B.
Kim et al. Kim et al.
(2021) compute the RS-FGSM perturbation and evaluate the model on c points along this direction.
Therefore, they will spend 1F/B on the RS-FGSM attack, c − 1 F on the evaluations since the clean image has already been evaluated; and 1 F/B for the weight update.
In our plot, we used c = 3 since it was the most chosen setting.
Kim et al.
(2021) assume the cost of a forward is similar to that of a backward pass, following this assumption, cost of Kim et al.
(2021) is 1 F/B +
[Equation]
+
[Equation] [Equation]
=
[Equation]
/B
Free-AT: Shafahi et al.
re-use the gradient from the previous backward pass to compute the FGSM perturbation of the current iteration.
Hence, the cost of their training is only 1 F/B per iteration.
However, Wong et al.
(2020) observed they needed a longer training schedule to produce comparable results.
Therefore, the total training cost per iteration (1 F/B) is scaled by 96 in the case of Free-AT, while it is only scaled by 30 for other methods.
Relative cost Free =
(96 · 1 F/B) / (30 · 2 F/B).
GradAlign:
Finally, GradAlign uses FGSM with a regularizer.
However, this regularizer needs to compute second-order derivatives via double backpropagation, which does not have the same cost as regular backpropagation.
Andriushchenko & Flammarion (2020) report that the cost of using
GradAlign regularizer increased the cost of FGSM by 3.
L D ETAILED RESULTS FOR S ECTION 5.1 AND S ECTION 5.
In this section we present the tables with the exact numbers used in plots comparing adversarial training methods.
For each method and (cid:15) − l ∞ radius, the top number is the clean accuracy while the bottom number is the PGD-50-10 accuracy.
We separate single-step from multi-step methods with a double line.
[Table] [Table]
PreActResNet
[Table]
SVHN Dataset
[Table] [Table]
WideResNet28-10 – CIFAR-100 Dataset (cid:15) = 2 /
[Table]
=
[Table] [Table] = 6 / [Table]
[Table]
[Table]
[Table] [Table] = 16 / N-FGSM
[Table] [Table]