Make Some Noise: Reliable and Efficient Single-Step Adversarial Training
Abstract
Recently, Wong et al.
[35] showed that adversarial training with single-step FGSM leads to a characteristic failure mode named Catastrophic Overfitting (CO), in which a model becomes suddenly vulnerable to multi-step attacks.
Experimentally they showed that simply adding a random perturbation prior to FGSM (RS-FGSM) could prevent CO.
However, Andriushchenko and Flammarion [1] observed that
RS-FGSM still leads to CO for larger perturbations, and proposed a computationally expensive regularizer (GradAlign) to avoid it.
In this work, we methodically revisit the role of noise and clipping in single-step adversarial training.
Contrary to previous intuitions, we find that using a stronger noise around the clean sample combined with not clipping is highly effective in avoiding CO for large perturbation radii.
We then propose Noise-FGSM (N-FGSM)
that, while providing the benefits of single-step adversarial training, does not suffer from CO.
Empirical analyses on a large suite of experiments show that N-FGSM is able to match or surpass the performance of previous state-of-the-art GradAlign, while achieving 3 × speed-up.
Code can be found in https://github.com/pdejorge/N-FGSM
Introduction
Deep neural networks have achieved remarkable performance on a variety of tasks [12, 27, 7].
However, it is well known that they are vulnerable to small worst-case perturbations around the input data – commonly referred to as adversarial examples [31].
The existence of such adversarial examples poses a security threat to deploying models in sensitive environments [2, 4].
This has motivated a large body of work towards improving the adversarial robustness of neural networks [11, 22, 32, 25, 5].
The most popular family of methods for learning robust neural networks is based on the concept of adversarial training [11, 20].
In a nutshell, adversarial training can be posed as a min-max problem where instead of minimizing some loss over a dataset of clean samples, we augment the inputs with worst-case perturbations that are generated online during training.
However, obtaining such perturbations is NP-hard [34] and hence, different adversarial attacks have been suggested that approximate them.
In their seminal work, Goodfellow et al.
[11] proposed the Fast Gradient
Sign Method (FGSM), that generates adversarial attacks by performing a gradient ascent step on the loss function.
Yet, while FGSM-based adversarial training provides robustness against single-step
FGSM adversaries, Tramèr et al.
[32] showed that these models are still vulnerable to multi-step attacks, namely those allowed to perform multiple gradient ascent steps.
Given their better (robust) ε k
FGSM
N-FGSM
RS-FGSM       16 for training and evaluation (x / 255)
A d v e r s a r i a l A cc .
CIFAR10 Dataset
N-FGSM (ours)
GradAlign
MultiGrad ZeroGrad
Free-AT Kim
et.
al.
RS-FGSM FGSM
N - F G S
M
G r a d A li g n
M u l t i G r a d
Z e r o G r a d
F r ee - A
T
K i m e t .
a l .
R S - F G S
M
F G S M 0.
T r a i n c o s t r e l a t i v e t
o
F G S M
Moreover
N does not clip performance, multi-step attacks such as Projected Gradient Descent (PGD) [20] have now become the de facto standard for adversarial training.
The main downside of multi-step adversarial training is that the cost of these attacks increases linearly with the number of steps, making their applicability often computationally prohibitive.
For this reason, several works have focused on reducing the cost of adversarial training by approximating the worst-case perturbations with single-step attacks [35, 26, 33].
In particular, Wong et al.
[35] studied
FGSM adversarial training and discovered that it suffers from a characteristic failure mode, in which a model suddenly becomes vulnerable to multi-step attacks despite remaining robust to single-step attacks.
This phenomenon is referred to as Catastrophic Overfitting (CO).
As a solution, they argued that adding a random perturbation prior to FGSM (RS-FGSM) seemed sufficient to prevent CO and produce robust models.
Yet, Andriushchenko and Flammarion [1] recently observed that RS-FGSM still leads to CO as one increases the perturbation radii of the attacks.
They suggested a regularizer (GradAlign) that can avoid CO in the settings they considered, but at the expense of computing a double derivative – significantly increasing the computational cost with respect to RS-FGSM.
In this paper, we revisit the idea of including noise in single-step attacks.
Differently from previous methods that consider the noise as part of the attack, we propose an adversarial training procedure where the noise is used as a form of data augmentation .
As we detail in Section 4, this motivates us to introduce two main changes with respect to previous methods: 1) We center adversarial perturbations with respect to noise-augmented samples and therefore, unlike previous RS-FGSM, we do not clip around the clean samples.
We use noise perturbations larger than the ϵ − ball, since they are not restricted by the strength of the attack anymore.
Our experiments show that performing data augmentation with sufficiently strong noise and removing the clipping step improves model robustness and prevents CO, even against large perturbation radii.
Our new method, termed N-FGSM, matches, or even surpasses, the robust accuracy of the regularized FGSM introduced by Andriushchenko and
Flammarion [1] (GradAlign), while providing a 3 × speed-up .
To corroborate the effectiveness of our solution, we present an experimental survey of recently proposed single-step attacks and empirically demonstrate that N-FGSM trades-off robustness and computational cost better than any other single-step approach, evaluated over a large spectrum of perturbation radii (see Figure 1, middle and right panels), over several datasets (CIFAR-10, CIFAR100, and SVHN) and architectures (PreActResNet18 and WideResNet28-10).
We will release our code to reproduce the experiments.
Related Work
Since the discovery of adversarial examples, many defense mechanisms have been proposed, adversarial training being one of the most popular and empirically validated.
We can categorise adversarial training methods based on how they approximate the perturbations applied to training samples.
Multistep approaches approximate an inner maximization problem to find the worst-case perturbation with several gradient ascent steps [37, 18, 20].
While this provides a better approximation, it is also more expensive.
At the other end of the spectrum, single-step methods only use one gradient step to approximate the worst case perturbation.
Goodfellow et al.
[11] first proposed FGSM; Tramèr et al.
[32] proposed a new variant with an additional random step (R+FGSM), but observed that both methods were vulnerable to multi-step attacks.
Shafahi et al.
[26] proposed Free Adversarial Training (Free-AT), which successfully reduced the computational cost of training by using a single backward pass to compute both weight update and attack.
Motivated by this, Wong et al.
[35] explored a variant of R+FGSM, namely RS-FGSM, that uses a less restrictive form of noise and showed this can improve robustness for moderate perturbation radii at the same cost as FGSM.
Recently, Andriushchenko and Flammarion [1] proposed the GradAlign regularizer.
Combining FGSM with GradAlign results in robust models at even larger perturbation radii.
However, GradAlign suffers from a threefold increase in the training cost to as compared to FGSM.
The need for more efficient solutions has motivated a growing body of work whose goal is the design of computationally lighter single-step methods [10, 15, 33, 23, 19].
In this work, we revisit the idea of combining noise with the FGSM attack.
Our method builds upon FGSM and intuitions from R+FGSM and RS-FGSM to combine it with random perturbations, however, we consider the noise step as data augmentation rather than part of the attack.
This motivates us to use a stronger noise without clipping.
As opposed to [14], our thorough study leads to a practically effective approach that yields robustness also against large perturbation radii.
Preliminaries on Single-Step Adversarial Training
Given a classifier f θ : X → Y parameterized by θ and a perturbation set S , f θ is defined as robust at x ∈ X on the set S if for all δ ∈ S we have f θ ( x + δ ) =
f θ ( x ) .
One of the most popular definitions for S is the ϵ − ℓ ∞ ball, i.e., S =
{ δ : ∥ δ ∥ ∞ ≤ ϵ } .
This is known as the l ∞ threat model which we adopt throughout this work.
To train networks that are robust against ℓ ∞ threat models, adversarial training modifies the classical training procedure of minimizing a loss function over a dataset D
= { ( x i , y i ) } i =1: N of images x i ∈ X and labels y
i ∈ Y .
In particular, adversarial training instead minimizes the worst-case loss over the perturbation set S , i.e., training is on the adversarially perturbed samples { ( x i
+ δ i , y i ) } i =1: N .
Under the l ∞ threat model, we can formalize adversarial training as solving the following problem:
[Equation]
where L is typically the cross-entropy loss.
Due to the difficulty of finding the exact inner maximizer, the most common procedure for adversarial training is to approximate the worst-case perturbation through several PGD steps [20].
While PGD has been shown to yield robust models, its cost increases linearly with the number of steps.
As a result, several works have focused on reducing the cost of adversarial training by approximating the inner maximization with a single-step.
If the loss function is linear with respect to the input, the inner maximization of Equation (1) will enjoy a closed form solution.
Goodfellow et al.
[11] leveraged this to propose FGSM, where the adversarial perturbation follows the direction of the sign of the gradient.
Tramèr et al.
[32] proposed adding a random initialization prior to FGSM.
However, both methods were later shown to be vulnerable against multi-step attacks, such as PGD.
Contrary to prior intuition, recent work from Wong et al.
[35] observed that combining a random step with FGSM can actually lead to a promising robustness performance.
In particular, most recent single-step methods approximate the worst-case perturbation solving the inner maximization problem in Equation (1) with the following general form:
[Equation]
where η is drawn from a distribution Ω .
For example, when ψ is the projection operator onto the ℓ ∞ ball and Ω is the uniform distribution [ − ϵ, ϵ ] d , where d is the dimension of X , this recovers
RS-FGSM.
Under a different noise setting where Ω =
( ϵ − α ) · sign ( N ( 0 d , I d )) and by choosing the step size α to be in [0 , ϵ ] , we recover R+FGSM by Tramèr et al.
This was among the first works to explore the application of noise to FGSM, but did not report improvements over it.
If we consider Ω to be deterministically 0 and ψ to be the identity map, we recover FGSM.
Finally, if we take FGSM and add a gradient alignment regularizer, this recovers GradAlign.
[Figure]
A d v e r s a r i a l A cc .
[Figure]
Clipping Clipping
[Figure]
E ff e c t i v e F G S
M s t e p ( /)
[Figure]
Clipping Clipping
[Figure]
i
[Figure]
Noise and FGSM
Previous methods that combined noise with FGSM, e.g., R+FGSM [32] and RS-FGSM [35], have considered the noise as a random step integrated within the attack.
Since it is a common practice to restrict adversarial perturbations to the ϵ − ball, we argue that this introduces a trade-off between the magnitude of the random step and that of the attack.
For illustration, consider the purple lines corresponding to RS-FGSM in Figure 1 (left).
If the initial random step is significantly larger than the ϵ − ball, then the final clipping step will have a noticeable impact on the perturbation, possibly removing a considerable portion of the FGSM step (middle arrow).
To prevent this from happening,
R+FGSM and RS-FGSM restrict the random step to lie within the ϵ − ball, thereof implicitly entangling the noise magnitude and the attack strength.
Contrary to previous methods, we note that adding noise to the clean sample can be considered as a form of data augmentation to be applied independently from the attack.
We make two considerations from this perspective 1) When one performs data augmentation during adversarial training, the input after the corresponding transformation is the starting point to compute the adversarial perturbations, therefore, we argue that adversarial attacks should be centered around the noise-augmented samples.
This motivates us to avoid clipping around the clean sample.
We do not need to restrict the noise augmentation to lie inside an ϵ − ball, since its strength is disentangled from that of the attack.
Thus, we can use stronger noise-augmentations than previous methods.
These modifications lead to a novel adversarial training method that combines noise-based data augmentations with FGSM.
We denote it as Noise-FGSM (N-FGSM).
Following the notation introduced in Section 3, we define the noise augmented sample as x aug = x + η where η is sampled from a uniform distribution on [ − k, k ] d (where we can have k > ϵ ).
Then the adversarially perturbed samples have the following form:
[Equation]
This construction corresponds to augmenting the clean sample x with the perturbation defined in Equation (2) where ψ is the identity map and Ω is the uniform ditribution spanning [ − k, k ] d .
We detail our full adversarial training procedure in Algorithm 1.
In what follows, we analyse the effect of treating the noise as data augmentation as opposed to treating it as a random step within the attack.
In particular, we show that clipping around the clean sample x (as done in RS-FGSM) can strongly degrade the robustness of the network.
Moreover, we show that as we increase the ϵ radii of adversarial attacks, we need stronger noise perturbations than previously used to prevent CO.
Clipping around clean sample x hinders the effectiveness of perturbations.
We analyse two variants, one where perturbations are clipped around the clean sample x
(as done in previous methods) and another where no clipping is applied.
In Figure 2 (left), we report the robust accuracy using PGD50-10 ( i.e., PGD attack with 50 iterations and 10 restarts) with ϵ = 8 / 255 and observe that clipping significantly degrades the effectiveness of FGSM training.
To understand this drop, consider the following perturbations; ( 1 ) a baseline perturbation where we only use noise δ random =
ψ ( η ) and ( 2 ) a per-
1: Inputs: epochs T , batches M , radius ϵ , step-size α (default: ϵ ), noise magnitude k (default: 2 ϵ ).
2: for t = 1 , . . .
, T do 3: for i = 1 , . . .
, M do 4: η ∼ Uniform [ − k, k ] d 5: x i aug = x
i + η // Augment sample with additive noise.
6: x i N-FGSM =
x i aug + α · sign (cid:0) ∇ x i aug L ( f θ ( x i aug ) , y i ) (cid:1) //
N-FGSM augmented sample.
7: ∇ θ = ∇ θ
L
( f θ ( x i N-FGSM ) , y i ) //
Compute gradients of model’s weights 8: θ = optimizer ( θ, ∇ θ ) //
Standard weight update, (e.g., SGD) turbation that combines noise with FGSM δ full = ψ ( η + α · sign (cid:0) ∇ x L ( f θ ( x + η ) , y ) (cid:1) ) .
Moreover, we consider two cases in which we either define ψ as a clipping operator or as the identity.
We define the effective FGSM step size as the magnitude corresponding to the ratio 2 α effective = ∥
δ full − δ random ∥ 2 / ∥ x ∥which measures the contribution of the FGSM step in the final perturbation compared to simply following the noise direction η .
In Figure 2 (middle), we observe that the clipping operator reduces the effective magnitude of FGSM, thus, perturbations become more similar to only using random noise.
On the other hand, without clipping we always take the full step in the FGSM direction.
This highlights the trade-off between noise magnitude and attack strength discussed above.
Larger noise is also necessary to prevent CO.
As discussed above, previous work did not investigate the effects of using noise perturbations potentially larger than the attack strength.
However, we empirically find that increasing the noise magnitude is key to avoiding CO.
In particular, as seen in Figure(right), when no clipping is performed, it is crucial that we augment with larger noise magnitude in order to prevent CO in all settings.
We find the noise magnitude of k = 2 ϵ to work well in most of our experiments, however, a more extensive hyperparameter tuning might improve our results further.
Note that these results are contrary to previous intuitions: Andriushchenko and Flammarion [1] suggested that the random step in RS-FGSM is not important per se, arguing that its main role is reducing the ℓ 2 norm of the perturbations, so that the loss remains to be approximately locally linear.
In contrast, N-FGSM perturbations are larger on expectation than those of RS-FGSM, while they do not suffer from CO (refer Appendix N).
We believe that our findings will lead to a better understanding of the role of noise in avoiding CO in future work.
Moreover, in Section 6 we conduct extensive analyses to show that, despite N-FGSM obtains larger perturbations, clean accuracy does not degrade and other methods do not benefit from simply increasing the strength of their attacks.
Why does noise augmentation avoid CO? Andriushchenko and Flammarion [1] found that after CO, the gradients of the loss with respect to the input around clean samples became strongly misaligned, which is a sign of non-linearity.
Moreover, Kim et al.
[15] showed that the loss surface of models suffering from CO appears distorted, i.e., there is a sharp peak in the loss surface along the FGSM direction, which seems to render FGSM ineffective (observe from Figure 5 how after after CO, visually, FGSM perturbations change drastically).
In order to prevent CO, GradAlign explicitly regularizes the loss surface so it remains linear.
To investigate further, we plot the loss surface at the end of training for different methods (see Figure 14 in Appendix) and find that, while FGSM or RS-FGSM lead to a distorted loss, N-FGSM obtains a non-distorted loss surface similar to that obtained by GradAlign regularizer.
Thus, it seems that adding strong noise-augmentations implicitly regularizes the loss landscape, leading to more effective single-step attacks.
This aligns with previous work that theoretically link noise augmentations with a regularizer that encourages Lipschitzness [3].
Robustness Evaluations and Comparisons
We compare N-FGSM against several adversarial training methods, on a broad range of ϵ − l ∞ radii.
Following Wong et al.
[35], we evaluate adversarial robustness on CIFAR-10/100 [16] and
SVHN [21] with PGD-50-10 attacks, using both PreactResNet18 [13] and WideResNet28-10 [36].
Evaluations with AutoAttack[6] are also in Appendix G.
[Figure]
al
[Figure]
5.1 Comparison against Single-Step Methods
We start by comparing N-FGSM against other single-step methods.
Note that not all single-step methods are equally expensive, since they may involve more or less computationally demanding operations.
For instance, GradAlign uses a regularizer that is considerably expensive, while MultiGrad requires evaluating input gradients on multiple random points.
For a comparison of training costs of different single-step methods, we refer the reader to Figure 1 (right).
We use RS-FGSM and Free-AT with the settings recommended by Wong et al.
We apply GradAlign with hyperparameters reported in the official repository 3 .
ZeroGrad and Kim et al.
[15] do not have a recommended set of hyperparameters; for a fair comparison we ablate them and select the ones with highest adversarial accuracy (for every ϵ and dataset).
We train on CIFAR-10/100 for 30 epochs and on SVHN forepochs with a cyclic learning rate.
Only for Free-AT, we use 96 and 48 epochs for CIFAR-10/ and SVHN, respectively, to obtain comparable results following Wong et al.
CIFAR-10 results are in Figure 1 (middle), whereas CIFAR-100 and SVHN are in Figure 3.
As observed in Figure 1 and Figure 3, FGSM and RS-FGSM suffer from CO for larger ϵ attacks on all reported datasets.
For instance, RS-FGSM fails against attacks with ϵ = 8 / 255 on CIFAR-10 and
CIFAR-100 and against ϵ = 6 / 255 on SVHN.
With appropriate hyperparameters, ZeroGrad is able to consistently avoid CO.
However, it obtains sub-par robustness compared to N-FGSM and GradAlign, especially against large ϵ attacks.
Neither MultiGrad nor Kim et al.
[15] avoid CO in all settings despite being more expensive.
Free-AT also suffers from CO on all three datasets as also observed by Andriushchenko and Flammarion [1].
In contrast, N-FGSM avoids CO on all datasets, achieving comparable or superior robustness to GradAlign while being 3 times faster.
5.2 Comparison against Multi-Step Attacks
In Section 5.1, we compared the performance of single-step methods and observed that N-FGSM is able to match or surpass the state-of-the-art method, i.e., GradAlign, while reducing the computational cost by a factor of 3 .
In this section, we compare the performance of N-FGSM against multi-step attacks.
In particular, we compare against PGD-2 with α = ϵ / 2 and PGD-10 with α = 2 / 255 , keeping the same training settings as described in Section 5.1.
PGD-x denotes x iterations and no restarts.
In Figure 4, we observe that PGD-2, despite being a multi-step method, still suffers from CO for larger ϵ as opposed to our proposed N-FGSM.
On the other hand, despite achieving comparable clean accuracies, there is a gap in adversarial accuracies between PGD-10, and other single-step methods that grows with perturbation size.
This can be partially expected since the search space for adversaries grows exponentially with ϵ ; and PGD, with more iterations, can explore it more thoroughly.
Nevertheless, computing a PGD-10 attack is 10 × more expensive to N-FGSM .
An important direction for future work would be addressing this gap and analysing, both theoretically and empirically, whether single-step methods can match the performance of their multi-step counterparts.
[Figure]
5.3 Analysis of Gradients and Adversarial Perturbations
To gain further insights into CO, we visually explore the perturbations generated with FGSM, RSFGSM, N-FGSM, and PGD-10 attacks.
We show that N-FGSM generates perturbations that exhibit behavior similar to PGD-10.
In particular, for a given test sample, we average the adversarial perturbations ( δ ) and gradients across several epochs at the beginning of training (Epoch 2 to 8) and at the end (Epoch 24 to 30) and visualise them in Figure 5 (see also Figure 12 in Appendix for more examples).
We observe that, during early stages in training all, methods generate consistent and interpretable δ .
However, after CO, FGSM and RS-FGSM generate δ that are harder to interpret, similarly to their gradients.
On the other hand, we observe that N-FGSM provides consistent and interpretable δ throughout training, similar to those generated by PGD-10.
This provides further evidence that N-FGSM enjoys similar properties to the more expensive PGD-10 training.
Figure 5 analyzes the gradients and δ throughout the test set.
Aside from loosing interpretability, post-CO the gradient norm increases by several orders of magnitude for FGSM and RS-FGSM while it remains low for N-FGSM and PGD-10.
We also compute the effective rank 4 of δ for each example before and after CO to measure the consistency of δ before and after CO.
We consider three training intervals, (Epoch 2 to 8): before CO for all methods; (Epoch 16 to 22): after FGSM suffers CO but not RS-FGSM; (Epoch 24 to 30): after both FGSM and RS-FGSM suffer CO.
Prior to CO, PGDhas a larger effective rank ( i.e., the perturbations span a larger subspace) than FGSM and RS-FGSM.
N-FGSM has the highest effective rank, arguably due to the higher noise magnitude.
Note that
RS-FGSM, which has a smaller noise magnitude and clipping, also has a larger effective rank than
FGSM, however, the difference is much lower.
When either FGSM or RS-FGSM suffer from CO, the effective rank of their δ increases significantly above that of PGD-10 and N-FGSM.
This would suggest that δ loose consistency after CO and is aligned with our visualizations in Figure 5.
All of these show properties of δ and gradients that are consistent across methods (N-FGSM and PGD) that avoid CO and different from methods like RS-FGSM and FGSM, which suffer from CO.
Increasing Adversarial Perturbations
In Section 4, we observed that removing clipping and increasing the noise magnitude were both necessary for the improved performance of N-FGSM.
However, as discussed in Theorem N.2 this will result in an increase of the squared norm of the training perturbation δ N − FGSM as compared to FGSM.
In this section, we perform further ablations to corroborate that it is indeed the increase in noise magnitude – and not the mere increase of the perturbation’s magnitude – that helps to stabilize N-FGSM.
Increasing α alone is not sufficient.
N-FGSM combines a noise perturbation with an FGSM step.
Thus, we can increase the perturbation magnitude by increasing any of the two.
This begs the question:
Is it sufficient to increase the N-FGSM step-size α to avoid CO without adding any noise?
We observe in Figure 6 (A) that training without noise (essentially, FGSM) leads to CO, with robust accuracy
[Figure]
equal to zero, even for large values of α .
This indicates that it is not an increase in the perturbation norm, but the combination with noise which plays an essential role in circumventing CO for N-FGSM.
Increasing α requires adjusting the noise magnitude.
As observed in Figure 6 (A), increasing α for N-FGSM leads to CO if the noise magnitude is not large enough.
For example, while a noise magnitude k = 1 ϵ and an adversarial step size α = 1 .
25 ϵ yield a robust accuracy of 49 .
68% , increasing α to 1 .
5 ϵ while keeping the same noise magnitude results in CO – with robust accuracy equal to zero.
This further suggests that an increase in the adversarial step-size α requires a commensurate increase in the noise magnitude.
We find that setting the noise magnitude k
=
ϵ works well for most settings.
Larger noise perturbations preserve clean accuracy.
Increasing the norm of training perturbations by increasing α results in a drop in the clean accuracy (discussed later in Section 7).
This has also been observed in prior works [35].
However, we show in Figure 4 that the clean accuracy for
N-FGSM is similar to that of GradAlign, despite the magnitude of the perturbations being larger.
We ablate the effects of adversarial and noise perturbations on the clean accuracy in Figure 6 (B): we observe that augmenting training samples with noise alone ( i.e., α = 0 ) has a much milder effect on the clean accuracy than augmenting in an adversarial direction.
In general, increasing noise is more forgiving on the clean accuracy than increasing the adversarial step size.
This is not surprising, considering that moving in random directions along the input space has a significantly lower impact on the loss than moving along the FGSM direction (see Figure 14 in the Appendix) and that training with noise alone does not provide any significant robustness against larger attacks (for a more detailed ablation, see Appendix Figure 16).
Other methods do not benefit from larger training ϵ .
As previously mentioned, N-FGSM perturbations have ℓ ∞ − norm larger than ϵ .
We have seen that the benefits of N-FGSM can not be reproduced by simply increasing α without increasing the noise.
However, for the sake of completeness, we also ablate other single-step baselines by using a larger ϵ during training, while testing with a fixed ϵ = 8 / 255 on CIFAR10.
We observe that increasing ϵ train seems to lead to a decrease in robustness for most methods; for instance, PGD-50-10 accuracy for RS-FGSM drops from 46 .
08 ± 0 .
18 when training with ϵ = 8 / 255 to 0 .
0 ± 0 .
with ϵ = 12 / 255 .
In two cases (GradAlign and MultiGrad), we observe a small increase, with the highest increase being for GradAlign, which improves from 48 .
14 ± 0 .
15 to 50 .
6 ± 0 . 45 ; yet, the clean accuracy drops from 81 .
9 ± 0 .
22 to 73 .
29 ± 0 . 23 .
This is similar to increasing α for N-FGSM (see Figure 6 (C)).
However, this is tied to a significant degradation of clean accuracy.
All in all, taking into account both clean and robust accuracy, we conclude that all single-step baselines suffer from either CO or a severe degradation in their clean accuracy when increasing the training ϵ .
Full results are presented in Table 2 in Appendix.
Additional Ablations
Hyperparameter selection.
While FGSM relies on a fixed step-size ( i.e., α = ϵ ), Wong et al.
[35] explored different values of α for RS-FGSM, finding that an increase of the step-size improves the adversarial accuracy – up to a point where CO occurs.
We also ablate the value of α for N-FGSM in Figure 6 (C).
We find that by increasing the noise magnitude, N-FGSM can use larger α values than RS-
[Figure]
In light of this trade-off, we also use α = ϵ for N-FGSM.
Regarding the noise hyperparameter k , we find that k
= 2 ϵ works in all but one SVHN experiment ( ϵ = 12 , in which we set k = 3 ϵ ).
In comparison, GradAlign regularizer hyperparameter or ZeroGrad quantile value need to be modified for every radius with a noticeable shift between CIFAR-10 and SVHN hyperparameters, suggesting they may require additional tuning when applied to novel datasets.
Long vs fast training schedules.
Throughout our experiments, we used the RS-FGSM training setting introduced in [35].
However, Rice et al.
[24] suggest that a longer training schedule coupled with early stopping may lead to a boost in performance.
Kim et al.
[15] and Li et al.
[19] report that longer training schedules increase the chances of CO for RS-FGSM and that this limits its performance.
We test the longer training schedule with N-FGSM and find that it does not suffer from
CO.
However, it does suffer from robust overfitting , i.e., adversarial accuracy on the training set is larger than on the test set as described in [24] for PGD-10.
Notice the difference between the robust accuracy of the final and best models in Figure 6 (D).
Interestingly, although we observe a slight increase in performance when using the long training schedule, we find the fast training schedule to be remarkably competitive.
See more results in Appendix D, including a comparison to GradAlign.
Experiments
with WideResNet28-
We also compare the performance of all methods on
WideResNet28-10 [36] architecture in Figure 8 and Figure 9 in Appendix.
As in the experiments with PreActResNet18, N-FGSM obtains the best performance/cost trade-off.
We had to increase the regularizer hyperparameter for GradAlign (compared to the settings for PreActResNet18) in order to prevent CO on CIFAR-100 and, to our surprise, we could not find a competitive hyperparameter setting for GradAlign on the SVHN dataset for ϵ ≥ 6 .
We tried both increasing the regularizer hyperparameter and decreasing the step size α , but some or all runs led to models close to a constant classifier for each setting.
We do not claim that GradAlign will not work, but finding a good configuration might require further tuning.
The default configuration for N-FGSM ( α = ϵ, k = 2 ϵ ) works well in all settings except for ϵ = 16 on CIFAR-10 and ϵ = 10 , 12 on SVHN.
For CIFAR-10, we increase the noise magnitude to k = 4 ϵ .
For SVHN, we find that decreasing α works better than increasing the noise.
In both cases, N-FGSM yields nontrivial adversarial accuracy.
Experiments on Imagenet.
We present results on the Imagenet dataset [17] in Table 1.
Due to the high computational demands of Imagenet training and testing we focus on the main baselines of comparable cost to FGSM.
Namely FGSM, RS-FGSM and N-FGSM.
We observe that FGSM presents CO for ϵ = 6 / 255 while neither RS-FGSM nor N-FGSM present CO.
However, N-FGSM has better robustness.
For instance, at ϵ = 6 / 255 N-FGSM obtains PGD50-10 accuracy of 17.12% while
RS-FGSM yields 16.5% and FGSM 0.08% (due to CO).
Thus, N-FGSM also avoids CO in ImageNet, improving robustness over same-cost baselines.
For experimental details refer to Appendix L.
Combining N-FGSM with additional regularizers.
Recent works [29, 30] have been proposed to improve performance of single step methods at moderate perturbation
radii ϵ
= 8 / 255 .
However, we observe that with the default settings (which use a version of RS-FGSM with Bernoulli noise) they lead to CO for larger ϵ .
Then we compare them with N-FGSM + Regularizer where we apply
[Table]
their proposed regularizers to N-FGSM.
If we apply GAT or NuAT regularizers to N-FGSM then we do not observe CO and usually a boost in performance.
For instance, at ϵ = 10 / 255 , GAT has a robust acc (with PGD50-10) of 43.34 ± 0.23 while N-FGSM+GAT regularizer obtains 44.97 ± 0.07, in comparison plain N-FGSM has 41.56 ± 0.16.
This is extremely compelling as it suggests
N-FGSM can be combined with other regularizers designed to improve FGSM performance and mutually benefit each other.
Full results of the comparison are presented in Table 6 in Appendix K
Conclusion
In this work, we explore the role of noise and clipping in single-step adversarial training.
Contrary to previous intuitions, we show that increasing the noise magnitude and removing the ϵ − ℓ ∞ constraint leads to an improvement in adversarial robustness while maintaining a competitive clean accuracy.
These findings led us to propose N-FGSM, a simple and effective approach that can match or surpass the performance of GradAlign [1], while achieving a 3 × speed-up.
We perform an extensive comparison with other relevant single-step methods, observing that all of them achieve sub-optimal performance and most of them are not able to avoid CO for larger ϵ attacks.
Moreover, we also analyze gradients and adversarial perturbations during training and observe that they have a similar behaviour for N-FGSM and PGD-10 as opposed to other methods that present CO such as FGSM and RS-FGSM.
However, despite impressive improvements of single-step adversarial training methods, there is still a gap between single-step and multi-step methods such as PGD-10 as we increase the ϵ radius.
Therefore, future work should put an emphasis on formally understanding the limitations of single-step adversarial training and explore how, if possible, this gap can be reduced.
Acknowledgments and Disclosure of Funding
We thank Guillermo Ortiz-Jiménez for the fruitful discussions and feedback.
This work is supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1.
We would also like to thank the Royal Academy of Engineering and FiveAI.
A. Sanyal acknowledges support from the ETH AI Center postdoctoral fellowship.
Pau de Jorge is fully funded by NAVER
LABS Europe.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We provide experimental validation for all our claims.
(b) Did you describe the limitations of your work?
[Yes] Yes, in conclusions and when appropriate along the paper.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do not consider research on training robust networks can have negative impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include the URL to our code in the abstract.
(b) Did you specify all the training details ( e.g., data splits, hyperparameters, how they were chosen)?
[Yes] When relevant in the text.
(c) Did you report error bars ( e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All plots or numbers have standard deviation.
Although sometimes too small to be visible.
(d) Did you include the total amount of compute and the type of resources used ( e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Yes, see appendix R. 4.
If you are using existing assets ( e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include the URL to our code in the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Additional plots for PreActResNet18 experiments
In the main paper we compare N-FGSM with other single-step methods and multi-step methods separately and remove clean accuracies for better visualization.
In this section we present the curves for all methods with both the clean and robust accuracy.
The tendency in the three datasets is for
N-FGSM PGD-50-10 accuracy to be slightly above that of GradAlign, while the opposite happens to the clean accuracy.
We also observe that clean accuracy becomes significantly more noisy when CO happens.
Exact numbers for all the curves are in Appendix S
[Figure]
plots.
B Experiments with WideResNet28-10 architecture
In this section we present the plots of our experiments with WideResNet28-10.
We report the results in two figures.
In Figure 8 we compare all single-step methods and we do not plot the clean accuracy for better visualization.
In Figure 9 we plot all methods, including multi-step methods, and report the clean accuracy as well with dashed lines.
Since we observed that our baseline, RandAlpha, outperformed [15] in all settings for PreActResNet18, we only report RandAlpha for WideResNet.
As mentioned in the main paper, we observe that CO seems to be more difficult to prevent for
WideResNet.
In particular, for GradAlign we observed the regularizer hyperparameter settings proposed by [1] for CIFAR-10 (searched for a PreActResNet18) worked well.
However, those parameters led to CO for 6 ≤ ϵ ≤ 12 in CIFAR-100.
Since ϵ = 14 , 16 did not show CO, we increased the GradAlign regularizer hyperparameter λ for CIFAR-100 so that each 6 ≤ ϵ ≤ 12 would have the default value corresponding to ϵ + 2 , for instance, λ for ϵ = 6 would be the default λ in [1] for ϵ = 8 .
[Figure]
SVHN with
WideResNet28-10 over different perturbation radius ( ϵ is divided by 255).
Our method, N-FGSM, is able to match or surpass the state-of-the-art single-step method GradAlign while reducing the cost by a 3 × factor .
Moreover, we could not find any competitive hyperparameter setting for GradAlign for ϵ ≥ 6 in SVHN dataset.
Adversarial accuracy is based on PGD-50-10 and experiments are averaged over 3 seeds.
Legend is shared among all plots.
For SVHN we observed that the default values for λ led to models close to a constant classifier for ϵ ≥ 6 .
We tried to increase the lambda for those ϵ values to 1 .
25 λ but observed the same result.
Since the model did not show typical CO but rather it seemed as it was underfitting, we tried to reduce the step-size to α = 0 .
75 ϵ and also both decreasing α and increasing λ .
When reducing the step size we obtain accuracies above those of a constant classifier for some radii, however, some or all seeds converge to a constant classifier for each setting, hence the large standard deviations.
For N-FGSM, the default configuration of N-FGSM ( α = ϵ, k = 2 ϵ ) works well in all settings except for ϵ = 16 on
CIFAR-10 and ϵ = 10 , 12 on SVHN.
For CIFAR-10, we increase the noise magnitude to k = 4 ϵ .
For SVHN we find that decreasing α as we tried for GradAlign works better than increasing the noise.
We use α = 8 for both ϵ radii.
Exact numbers for all the curves are in Appendix S
[Figure]
shared among all plots.
C Increasing adversarial perturbations during training
As mentioned in the main paper, N-FGSM perturbations have ℓ ∞ − norm larger than ϵ , see Appendix N.
In Section 6 we have seen that the benefits of N-FGSM can not be reproduced by simply increasing α without increasing the noise.
However, for the sake of completeness, we also ablate other single-step baselines by using a larger ϵ during training i.e., { ϵ = 8 / 255 , ϵ = 12 / 255 , ϵ = 16 / 255 } while testing with a fixed ϵ = 8 / 255 on CIFAR-10.
Results are presented in Table 2.
We observe that increasing ϵ train seems to lead to a decrease in robustness for most methods, e.g., PGD-50-10 accuracy for
RS-FGSM goes from 46 .
08 ± 0 .
18 when training with ϵ = 8 / 255 to 0 .
0 ± 0 .
with ϵ = 12 / 255 .
In two cases (GradAlign and MultiGrad)
we observe a small increase, highest increase is for GradAlign which goes from 48 .
14 ± 0 .
15 to 50 .
6 ± 0 .
45 , however, the clean accuracy drops from 81 .
9 ± 0 .to
29 ± 0 . 23 .
This is similar to increasing α for N-FGSM (see Figure 6 (C)).
However, this is tied to a significant degradation of clean accuracy.
All in all, taking into account both clean and robust accuracy we conclude all baselines perform best without increasing the training ϵ .
All ablation results are presented in Table 2.
[Table]
D Longer training schedule
In our experiments, we have followed the “fast” training schedule introduced by [35].
However, [24] suggest that a longer training schedule coupled with early stopping may lead to a boost in performance.
We also use the long training schedule for N-FGSM and observe that it does not lead to CO.
In Table 3 we compare the performance of N-FGSM and GradAlign for the long training schedule.
We observe that GradAlign does not seem to benefit from the long training schedule.
On the other hand, although N-FGSM seems to obtain a slight increase in performance, the “fast” schedule provides comparable performance.
It is worth mentioning that for GradAlign, the default regularizer hyperparameter for ϵ = 8 / 255 and CIFAR-10 ( λ = 0 . 2 ) does not prevent CO.
We do a hyperparameter search and keep the value with the largest final robust accuracy ( λ = 0 . 632 ).
GradAlign does not seem to benefit from the long training schedule.
Although N-FGSM seems to obtain a slight increase in performance, the “fast” schedule provides comparable performance.
[Table]
In Table 3 we observe that the performance of the final model is lower than that of an early stopped method.
This could be expected due to the phenomena of robust overfitting described in [24].
However, as a sanity check to make sure that this is not due to a hidden CO during the long schedule which somehow the model recovers from we plot the full training history in Figure 10.
There we can observe that for N-FGSM there is no CO during training .We also show FGSM (which is well known has CO for ϵ = 8 / 255 ) for comparison.
[Figure]
E Randomized Alpha
Kim et al.
[15] evaluate intermediate points along the RS-FGSM direction in order to pick the “optimal” perturbation size.
However, we find that increasing the number of intermediate evaluated points does not necessarily lead to increased adversarial accuracy.
Moreover, for large perturbations we could not prevent CO even with twice the number of evaluations tested by [15].
This motivates us to test a very simple baseline where instead of evaluating intermediate steps, the RS-FGSM perturbation size is randomly selected as: δ = t · δ RS-FGSM where t ∼ U [0 , 1] d .
Interestingly, as reported in Figure 11, we find that this very simple baseline, dubbed RandAlpha , is able to avoid CO for all values of ϵ and outperforms [15] on CIFAR-10, CIFAR-100 and SVHN.
This is aligned with our main finding that combining noise with adversarial attacks is indeed a powerful tool that should be explored more thoroughly before developing more expensive solutions.
[Figure]
F Further visualizations of adversarial perturbations and gradients
In this section we present an extension of Figure 5 with further examples.
As observed in the main paper, early in training adversarial perturbations ( δ ) and gradients are consistent across epochs, however, after CO they become hard to interpret.
Note that although we label rows as either pre-CO or post-CO we only observe CO for FGSM and RS-FGSM.
Both PGD-10 and N-FGSM obtain robust models as shown in detail in the paper.
[Figure]
P r e - C
O
FGSM
RS-FGSM
N-FGSM
PGDFGSM
RS-FGSM
N-FGSM
PGDP o s tC O
P r e - C
O
P o s tC O
P r e - C
O
P o s tC O
G Robust evaluations with autoattack
ϵ = 2 /ϵ
= 4 /ϵ
= 6 /ϵ
= 8 /ϵ
= 10 /ϵ
= 12 /ϵ
= 14 /ϵ
= 16 /
FGSM 91.52 ± 0.
78.99 ± 0.
88.59 ± 0.
65.99 ± 0.
85.17 ± 0.
54.0 ± 0.
86.62 ± 0.
0.0 ± 0.
83.35 ±
[Table]
Following previous work
[Table]
[1, 11] we have evaluated robustness with PGD50-10,
i
[Table]
e. PGD withiterations and 10 restarts.
However, for the sake of completeness
[Table]
we also present results of robust accuracy evaluated with autoattack [6].
In Table 4 we evaluate models adversarially trained with our proposed method N-FGSM, the baseline FGSM and GradAlign.
We observe the same pattern as with the PGD50-10 attack, therefore we are conviced that our results are general.
H Catastrophic Overfitting outside the ResNet family
Previous work focusing on CO has only used architectures from the ResNet family.
In Table 5 we present results for adversarial training with a VGG-16 architecture [28].
Similarly to other studied models we observe that FGSM leads to CO while N-FGSM is able to prevent it.
However, it seems that FGSM presents CO for slighly larger ϵ radii, indicating that the architecture might play a role in
CO.
We consider investigating this further a promising direction of future work.
[Table]
I Further increasing the attack radii
Following previous work [1] we have studied ϵ attack radii up to epsilon = 16 / 255 .
Indeed, the performance at these radius is already significantly degraded and thus it would not be very practical for most applications.
However, to show that N-FGSM can prevent CO at even larger radii we test two additional radii, ϵ = 20 / 255 and ϵ = 24 / 255 .
In both cases N-FGSM is able to prevent CO.
For ϵ = 20 / 255 we obtain a clean accuracy of 51.63 ± 0.38 and robust of 20.62 ± 0.37 while for ϵ =
/we
obtain a clean accuracy of 40.16 ± 0.96 and robust of 15.3 ± 1.49.
We argue that it is of little interest to try even larger perturbations unless more effective methods to improve both the clean and robust performance are found.
J Testing other norms
Following previous work, we have focused on the ℓ ∞ threat model.
Although this is where works studying CO have mainly focused, we observe that CO is also present in other norms such as ℓ 1 and ℓ 2 .
Moreover, in both cases we observe that N-FGSM is able to prevent CO.
Interestingly, the range of norms in which we observe CO is usually much higher than normally tested for these norms which would explain why the ℓ ∞ norm has been the main focus of study in related works.
[Figure]
all pixels have maximum magnitude i.e. ϵ = 8 / 255 indicates perturbations were restricted to an ℓ p norm of a vector where all components are in {− ϵ, + ϵ } .
Which would correspond to an ℓnorm of nϵ and an ℓ 2 norm of ϵ √ n where n indicates the dimensionality of the input.
K Combining N-FGSM with additional regularizers
In this section we present the results from Section 7 where we combine N-FGSM with additional regularizers [29, 30] that were proposed for single-step adversarial training to boost the performance.
First, we try the proposed methods with the default settings (which use a version of RS-FGSM with
Bernoulli noise) and observe they lead to CO for larger ϵ .
Then we compare them with N-FGSM + Regularizer where we apply their proposed regularizers to N-FGSM.
If we apply GAT or NuAT regularizers to N-FGSM then we do not observe CO and usually a boost in performance.
Results are presented in Table 6.
[Table]
L Imagenet experimental details
For our experiments on Imagenet we mainly follow the settings from [35].
However, for simplicity we did not do image resizing which requires storing two additional Imagenet datasets.
More importantly, we found that the learning rate schedule suggested in [35] was not optimal for N-FGSM.
The schedule suggested in [35] follows three different stages in which the learning increases or decreases linearly for some iterations.
In particular in the first stage, the learning rate has an initial warm-up where it increases linearly from 0.0 to 0.4 during the first epoch and then decreases linearly to 0.04 during the next 5 epochs.
As a lucky coincidence when debugging, we modified this initial stage such that we preserved the initial increase to 0.4 for the first epoch, but then we directly jumped to a learning rate of 0.04 which remained constant for the next 5 epochs.
For phase 2 and 3 both schedules remained the same.
First decreasing from 0.04 to 0.004 for epoch 6 to 12 and finally from 0.004 to 0.0004 for epoch 12 to 15.
This small change made N-FGSM improve both in clean and robust accuracy for ϵ = 4 / 255 , 6 / 255 and these are the numbers reported.
This indicates that further tuning the learning rate schedule might be an effective way to improve performance and even help prevent CO, however, due to the computational demands of ImageNet adversarial training we leave it for future work.
To be thorough we also trained RS-FGSM and FGSM with the modified schedule and found that neither of them benefit from it.
Regarding N-FGSM hyperparameters, for ϵ = 2 / 255 we used α = 2 / 255 and k = 1 ; for ϵ = 4 / 255 we used α = 4 / 255 and k = 1 ; and for ϵ = 6 / 255 we also used α = 4 / 255 and k = 1 .
M Visualization of the loss surface
In this section we present a visualization of the loss surface.
We adapted the code from [15] to analyse the shape of the loss surface at the end of training for different methods.
[15] reported that after adversarial training CO, the loss surface would become non-linear.
In particular, they found that the FGSM perturbation seems to be misguided by local maxima very close to the clean image that result in ineffective attacks.
We note this was already reported by [32] which proposed to perform a random step to escape those maxima.
We argue that adding noise to the random step, when properly implemented, actually prevents those maxima to appear in the first place.
[Figure]
a very linear loss surface which is key to obtain meaningful perturbations.
N Magnitude of N-FGSM perturbations
Lemma N.1 (Expected perturbation) .
Consider the N-FGSM perturbation as defined in Equation (3) δ N-FGSM =
η + α · sign ( ∇ x ℓ ( f ( x + η ) , y )) , where η ∼ Ω .
Let the distribution Ω be the uniform distribution U (cid:16) [ − kϵ, kϵ ] d (cid:17) and α > 0 .
Then,
E
[Equation]
and
[Equation]
Proof.
By Jensen’s inequality, we have
[Equation]
Then let us consider the term E η (cid:2)
∥
δ N-FGSM ∥
22 (cid:3)
and use the shorthand ∇ ( η ) i
= ( ∇ x ℓ ( f ( x + η ) , y ))
i .
E η (cid:2) ∥
δ N-FGSM ∥
22 (cid:3) =
E η ∥
η + α · sign ( ∇ x ℓ ( f ( x + η ) , y ))
∥= E η (cid:34) d (cid:88) i =
( η i + α · sign ( ∇ ( η ) i ))
(cid:35)
=
d (cid:88) i =
E η (cid:104) ( η i + α · sign ( ∇ ( η ) i )) 2 (cid:105) =
d (cid:88) i =
E η (cid:104) ( η i + α · sign ( ∇ ( η ) i ))
2 | sign ( ∇ ( η ) i ) =
1 (cid:105) P η [ sign ( ∇ ( η ) i ) = 1] + d (cid:88) i =
E η (cid:104) ( η i + α · sign ( ∇ ( η ) i ))
2 | sign ( ∇ ( η ) i ) = − 1 (cid:105) P η [ sign ( ∇ ( η ) i ) = − 1] =
d (cid:88) i =  2 kϵ (cid:90) kϵ − kϵ ( η i + α ) 2 dη i · P η [ sign ( ∇ ( η ) i )
= 1]
+2 kϵ d (cid:88) i = (cid:90)
kϵ − kϵ ( η i − α ) 2 dη i · P η [ sign ( ∇ ( η ) i ) =
− 1]
= d (cid:88) i =  2 kϵ (cid:90)
α + kϵ α − kϵ
z 2 dz · P η [ sign ( ∇ ( η ) i ) = 1] +2 kϵ d (cid:88) i
= (cid:90) − α + kϵ − α − kϵ
z 2 dz · P η [ sign ( ∇ ( η ) i ) = − 1]
= d (cid:88) i =  2 kϵ (cid:90)
α + kϵ α − kϵ
z 2 dz · P η [ sign ( ∇ ( η ) i ) = 1] +2 kϵ d (cid:88) i
= (cid:90)
α + kϵ α − kϵ
z 2 dz · P η [ sign ( ∇ ( η ) i ) = − 1] =2 kϵ (cid:90)
α
+ kϵ α − kϵ
z 2 dz d (cid:88) i =
( P η [ sign ( ∇ ( η ) i ) = 1] + P η [ sign ( ∇ ( η ) i ) = − 1])
=
d 6 kϵ (cid:2) ( α + kϵ ) 3 − ( α − kϵ ) 3 (cid:3) =
dk
2 ϵ 2+ dα
Therefore,
[Equation]
Theorem N.2.
Let δ N-FGSM be our proposed single-step method defined by Equation (3) , δ FGSM be the
FGSM method [11] and δ RS-FGSM be the RS-FGSM method [35].
Then, with default hyperparameter values and for any ϵ > 0 , we have that
[Equation]
Proof.
From Lemma
N.1 we have that
[Equation]
On the other hand, [1] showed that
[Equation]
Finally, we note that
[Equation]
The default hyperparameters for N-FGSM are k = 2 , α = ϵ and RS-FGSM uses α = 5 ϵ/ 4 .
With these hyperparameters and any ϵ > 0 we have
[Equation]
In Lemma N.1 we compute the expected value of the squared ℓ 2 norm of N-FGSM perturbations and by Jensen’s inequality we obtain an upper bound for the expected ℓ 2 norm of N-FGSM perturbations.
However, obtaining the exact expected magnitude is more complex.
To compliment our analytic results, we approximate the ℓ 2 norm of FGSM, RS-FGSM and N-FGSM via Monte Carlo sampling.
Results are presented in Figure 15.
We observe that the empirical estimations are very close to the analytical upper bounds and that indeed, N-FGSM has a magnitude significantly above that of FGSM or RS-FGSM.
O N-FGSM with Gaussian noise
In the main paper we have only explored noise sources coming from a Uniform distribution.
Since we are measuring robustness against l ∞ − attacks, the Uniform distribution is a natural choice because the random perturbations will be bounded to the l ∞ ball defined by the span of the distribution.
However, for the sake of completeness, we also explore the performance of augmenting the samples from a Gaussian distribution where we choose its standard deviation to match that of the uniform distribution.
In Table 7 we present a comparison of the clean (top) and PGD-50-10 (bottom) accuracy for different values of α and noise magnitude with ϵ = 8 / 255 .
Recall that by default we use Uniform distribution U [ − k, k ] , therefore hyperparameter k sets the noise magnitude.
Increasing the FGSM step size without increasing the amount of noise leads to CO.
Note results for k = 0 .
5 ϵ .
More importantly, results are very similar when the two noise distributions share the same standard deviation.
Thus, using Gaussian instead of Uniform noise does not seem to alter the results.
Although this might be expected, we remark that the Gaussian is an unbounded noise distribution and the common practice in adversarial training is to always restrict the norm of the perturbations.
16 Perturbation radius
E s t i m a t
e
d ln o r m
N-FGSM
FGSM / GradAlign
RS-FGSM
Upper bound N-FGSM
Upper bound RS-FGSM
Uniform Noise
Gaussian Noise
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
P Training with noise augmented samples
Gilmer et al.
[9] and Fawzi et al.
[8] report a close link between robustness to adversarial attacks and robustness to random noise.
Actually, [9] report that training with noise-augmented samples can improve adversarial accuracy and vice-versa.
We note that N-FGSM can actually be seen as a combination of noise-augmentation and adversarial attacks.
Here we perform an ablation where we train models with samples augmented with uniform noise U [ − k, k ] and then test the PGD50-10 accuracy.
We observe, that indeed random noise can increase the robustness to worst-case perturbations for small ϵ − l ∞ balls.
However, as we increase ϵ , noise augmentation is no longer very effective.
With N-FGSM, we apply a weak attack to these noise-augmented samples and this seems to be enough to make them effective for adversarial training.
Q Comparison of adversarial training cost
In this section we describe how we compute the relative training cost for single-step methods shown in Figure 1 (right).
We approximate the cost based on the number of forward/backward passes each method uses, disregarding the cost of other additional operations such as adding a random step for
RS-FGSM or N-FGSM.
We understand these operations have a negligible cost compared to a full forward or backward pass.
FGSM:
FGSM is the cheapest of all methods since it only uses one forward/backward to compute the attack and an additional forward/backward to compute the weight update.
Hence, Cost FGSM =
F/B.
[Figure]
A d v e r s a r i
a l A cc u r a c
y
= =
= =
[Figure]
RS-FGSM:
As previously mentioned, we do not take into account the cost of random steps or clipping, hence we consider RS-FGSM to have the same cost as standard FGSM.
Cost RS-FGSM =
F/B.
N-FGSM:
Idem as before, cost of N-FGSM =
2 F/B.
ZeroGrad:
For ZeroGrad they need to do an additional sorting operation to find the smallest gradient components.
This could be potentially expensive, however, since the size of the input image is several orders of magnitude smaller than that of the network, we also ignore this cost.
Cost ZeroGrad =
F/B.
MultiGrad: MultiGrad computes 3 random steps and evaluates the gradient in all of them.
Therefore, it needs to do 3 F/B to compute the attack and an additional one to update the weights.
Cost MultiGrad = 4 F/B.
[15]: [15] compute the RS-FGSM perturbation and evaluate the model on c points along this direction.
Therefore, they will spend 1F/B on the RS-FGSM attack, c − 1 F on the evaluations since the clean image has already been evaluated; and 1 F/B for the weight update.
In our plot, we used c = 3 since it was the most chosen setting.
[15] assume the cost of a forward is similar to that of a backward pass, following this assumption, cost of [15] is 1 F/B + 2 F + 1 F/B = 3 F/B
Free-AT: [26] re-use the gradient from the previous backward pass to compute the FGSM perturbation of the current iteration.
Hence, the cost of their training is only 1 F/B per iteration.
However, [35] observed they needed a longer training schedule to produce comparable results.
Therefore, the total training cost per iteration (1 F/B) is scaled by 96 in the case of Free-AT, while it is only scaled byfor other methods.
Relative cost Free =
(96 · 1 F/B) / (30 · 2 F/B).
GradAlign:
Finally, GradAlign uses FGSM with a regularizer.
However, this regularizer needs to compute second-order derivatives via double backpropagation, which does not have the same cost as regular backpropagation.
[1] report that the cost of using GradAlign regularizer increased the cost of
FGSM by 3.
R Infrastructure details and GPU hours
All our training runs have been conducted on either NVIDIA GPU V-100 or P-100 from an internal cluster.
The total compute for the results presented in this work is roughly 2500 hours.
S Detailed results for Section 5.1 and Section
In this section we present the tables with the exact numbers used in plots comparing adversarial training methods.
For each method and ϵ − l ∞ radius, the top number is the clean accuracy while the bottom number is the PGD-50-10 accuracy.
We separate single-step from multi-step methods with a double line.
[Table]
PreActResNet
ϵ
= 2 /ϵ
= 4 /ϵ
= 6 /ϵ
= 8 /ϵ
= 10 /ϵ
= 12 /ϵ
= 14 /ϵ
= 16 /
N-FGSM
[Table]
PreActResNet18 – SVHN Dataset
[Table]
[Table] [Table]
Dataset ϵ
= 2 /
[Table]
ϵ = 4 /ϵ
= 6 /ϵ
= 8 /
[Table]
ϵ = 10 /
[Table]
ϵ = 12 /
[Table]
ϵ =
[Table]
ϵ
[Table]
CIFAR-
[Table]
WideResNet28-10 – SVHN Dataset ϵ = 2 /ϵ
= 4 /ϵ
= 6 /ϵ
= 8 /ϵ
= 10 /ϵ
= 12 /
N-FGSM
[Table]