Dynamic Efficient Adversarial Training Guided by Gradient Magnitude
Abstract
Adversarial training is an effective but time-consuming way to train robust deep neural networks that can withstand strong adversarial attacks.
As a response to its inefficiency, we propose Dynamic Efficient Adversarial
Training (DEAT), which gradually increases the adversarial iteration during training.
We demonstrate that the gradient’s magnitude correlates with the curvature of the trained model’s loss landscape, allowing it to reflect the effect of adversarial training.
Therefore, based on the magnitude of the gradient, we propose a general acceleration strategy, M+ acceleration, which enables an automatic and highly effective method of adjusting the training procedure.
M+ acceleration is computationally efficient and easy to implement.
It is suited for DEAT and compatible with the majority of existing adversarial training techniques.
Extensive experiments have been done on CIFARand ImageNet datasets with various training environments.
The results show that the proposed M+ acceleration significantly improves the training efficiency of existing adversarial training methods while achieving similar robustness performance.
This demonstrates that the strategy is highly adaptive and offers a valuable solution for automatic adversarial training.
Introduction
While deep neural networks (DNNs) continue to achieve better performance across a broad range of activities, their safety and dependability are getting increasing attention.
These sophisticated models are known to be vulnerable to adversarial examples [6, 10, 16, 29], maliciously perturbed examples that can fool or mislead target DNNs but remain visually indistinguishable for humans and adversarial training has been demonstrated as one of most effective defensive strategies to improve DNNs’ adversarial robustness [2, 24].
The basic idea of adversarial training was proposed by Goodfellow et al.
[12], which showed that the robustness of DNNs could be improved by injecting the adversarial examples into training data.
Madry et al.
[19] later formulated adversarial training as a Min-Max
[Figure]
optimisation problem (see Eq.
(1)), which that using high-quality adversarial examples as training data can boost the robust performance of trained models.
However, it usually takes multiple backpropagations to generate such adversarial perturbations, which is computationally expensive.
The prohibitive training cost has become one of the in adversarial training [
Aiming to improve the training efficiency, in this paper, we take a closer look at existing adversarial training strategies [19, 26, 33] and find that redundant adversarial iterations could slow down the training process and sometimes even cause catastrophic forgetting [26], which spoils the trained model’s robustness.
To reduce the redundant backpropagation, we design a flexible adversarial training strategy, Dynamic Efficient Adversarial Training (DEAT), which begins the training with natural training data and gradually increases adversarial iterations when a pre-defined criterion is satisfied.
Moreover, we show that the magnitude of a network’s gradient with respect to training adversarial perturbations can roughly reflect the training effect and demonstrate that the model tends to gain ‘robustness’ after training if the gradient magnitude stays small.
Based on such an observation, we design a selfadaptive criterion based on gradient magnitude for DEAT and further extend it as a general acceleration strategy that can directly boost the training efficiencies of other state-of-the-art adversarial training methods like [32, 38].
Experiments have been done on CIFAR-10 [17] and ImageNet [8] datasets with different model architectures, and the results demonstrate that our magnitude-guided strategy can significantly speed up adversarial training methods under different training environments and achieves comparable robustness performance.
Background and Related Works
General Notations Given a dataset D that contains N pairs of example and label ( x i , y i ), we denote by X a set of all x ∈ D , where the number of examples in X is written as | X | .
Denoted by F : R n → R C , a neural network classifier parameterized by θ takes x ∈ R n as input and labels it into one of the C classes based on arg max c ∈{ 1 ,...,C } F c ( x ; θ ).
We denote by L the loss function that measures the difference between the model’s predictions and ground truth labels and write f ( x ; θ ) as a shorthand of L ( F ( x ; θ ) , y ) for simplicity.
This paper focus on the ℓ ∞ threat model [19], i.e. , δ ∈ B ϵ is an adversarial perturbation added on x and B ϵ is a ℓ ∞ norm ball with radio ϵ .
A projection is represented by P ϵ , which maps any input δ into B ϵ .
Adversarial Training Adversarial Training (AT) produces robust classifiers by optimising a Min-Max problem,
[Equation]
Based on Projected Gradient Descent (PGD) adversarial attack [19], PGD AT first performs multi-step PGD attacks to approximately solve the inner maximisation problem in Eq.
(1), where the perturbation crafted by PGD at i -th iteration can be described as
[Equation]
The produced adversarial examples are utilised as training data, and the model parameter θ is then updated to minimise the training loss.
[Table]
Although PGD AT can withstand strong adversarial attacks
[Table]
their computational cost is linear to the number of additional adversarial iterations [33]
[Table]
which is extremely computationally expensive in practice.
Therefore improving its efficiency has become an active research topic [3]
[Table]
while reducing the number of iterations would be the most straightforward strategy to accelerate the training procedure.
In Dynamic Adversarial Training (DAT) [31]
[Table]
the number of adversarial iterations is gradually increased according to a first-order stationary condition that measures the training effect.
However, DAT needs to initialise the increasing threshold on a pre-trained model with a similar architecture.
Based on an annealing mechanism, Amata linearly increases adversarial steps and adjusts the adversarial step size during training [35].
As we will show later, determining the optimal training schedules for different training tasks remains difficult.
Zhang et al.
[37] proposed YOPO, which simplifies the adversarial iteration by only generating adversarial examples toward the first layer of the trained model for adversarial training.
Friendly Adversarial Training (FAT) carries out
PGD adversaries with an early-stop mechanism to prevent overfitting and reduce training cost [39].
Wong et al.
[33] found that one step PGD attack, also known as First Gradient
Sign Method (FGSM) [12], with large step size and uniform initialisation, could achieve comparable performance to regular PGD AT and be significantly more efficient.
They also reported that cyclic learning rate [28] and half-precision computation [20] could significantly boost training efficiency.
We refer to training with cyclic learning rate and half-precision computation as FAST training [33] and to FGSM AT with uniform initialisation as U-FGSM in the rest of this paper.
Instead of improving PGD AT, Shafahi et al.
[26] proposed a novel training method called FREE, which makes the most use of backpropagation by updating the model parameters and adversarial perturbation simultaneously.
This mechanism is called batch replay, and we visualise the process in Fig. 1(a).
Table 1 summarises recent progress in efficient AT and shows the differences between our method and other approaches.
In parallel to efficiency, many efforts have been made to improve AT’s effectiveness, e.g. , [1, 22, 24].
In this direction, some advanced surrogate loss functions have been proposed to enhance or replace the cross-entropy loss.
TRADES can be viewed as a combination of the cross-entropy loss and a KL-divergence penalty [38].
On top of TRADES, MART merges the cross-entropy loss and the margin loss and pays more attention to misclassified examples [32].
Moreover, Zhang et al.
[39] show that these advanced surrogate losses can also benefit from efficient AT methods.
Lipschitz Condition
The Lipschitz constant for f ( · ; θ ) gives an upper bound on how fast the loss value changes when small perturbations are added to the network’s input [29].
This concept is closely related to the robustness of DNNs.
Definition 1 (Lipschitz continuity) .
Let ∥ · ∥ a and ∥ · ∥ b be two norms on R n and R , respectively.
We say f : R n → R is Lipschitz continuous if there exists a Lipschitz constant
K > 0 such that
[Equation]
Assumption 1 (Sinha et al.
Let ∥ · ∥ p be the dual norm of ∥ · ∥ q . f ( · ; θ ) satisfies the
Lipschitzian smoothness condition
[Equation]
where l xx is a positive constant.
Assumption 2 (Deng et al.
Given f ( · ; θ ) and an example x , there exists a stationary point x + δ ∗ ∈ B ϵ such that ∇ x f ( x + δ ∗ ; θ ) = 0 .
Given by Definition 1, in this paper, we suppose that the network F and loss function L are Lipschitz continuous [4, 25].
The Lipschitz condition Eq.
(3) proposed by Sinha et al.
[27] is commonly used by recent studies [31, 35].
Meanwhile, Deng et al.
[9] proved that the gradient-based adversary could find a local maximum that supports the existence of δ ∗ .
Gradient Magnitude Guided Adversarial Training
Previous studies implicitly show that AT is a dynamic procedure [5, 31], which means the number of adversarial iterations to generate suitable perturbation is floating.
Thus, we wish to push the efficiency of AT by conducting as few adversarial iterations as possible.
3. Motivation and Naive Implementation
To provide an intuitive understanding of the dynamics of AT, we illustrate the learning process in which learned models become more robust.
We perform three common AT strategies with ten training epochs and the cycle learning rate [28] and evaluate the robustness of trained models via a PGD attack with twenty iterations, denoted as PGD-20.
The three methods are
U-FGSM, PGD AT, and FREE, where U-FGSM is the one-step AT; PGD AT, represented by PGD-AT-7, consists of seven adversarial iterations; FREE is denoted by FREE-4 or
FREE-8, indicating that it is performed with four or eight batch replays, respectively.
During
FREE AT, the network is trained on clean examples in the first batch replay, and the actual adversarial training starts at the second iteration.
However, if the model is adversarially trained repeatedly too many times on a mini-batch, catastrophic forgetting would occur and influence training efficiency [26].
As shown in Fig.
2, we noticed that the robustness of the
FREE-8 trained model swings dramatically throughout adversarial training, the number of iterations required to generate an adequate perturbation appears to fluctuate, and redundant adversarial iteration may also damage the trained models’ robustness.
Naive implementation Motivated by the previous observation, we then introduce the
Dynamic Efficient Adversarial Training (DEAT) framework, where we wish to push the training efficiency by removing redundant adversarial iteration.
Similar to FREE, DEAT starts the training on natural examples by setting the number of batch replays r = 1 and gradually increases the number of batch replays to conduct adversarial training in the following training epochs (Pseudocode is presented in Appendix A).
DEAT requires a pre-defined criterion to determine the timing of increasing batch replay, which can be achieved in different ways.
Intuitively, performing one extra batch replay every d epoch is a practical and straightforward approach to achieving dynamic adversarial training.
We implement DEAT to increase the number of batch replays every 3 epochs, denoted by DEAT-3, and present the training process in the third column of Fig.
Because
DEAT-3 does not perform adversarial training in the first few epochs, the trained models achieved high classification accuracy on clean examples but did not gain robustness.
Despite the absence of adversarial robustness in the initial few epochs, the robustness performance of the DEAT-3 trained model shoots up after just one epoch of training on adversarial training examples.
In addition to increasing batch replay every few epochs, we notice that training accuracy can also be used to decide when to adjust the number of adversarial iterations.
One can also observe from Fig.
2 that as FREE-8’s robustness performance diminishes over epoch 2 to epoch 7, its training accuracy rises to 60%, which is remarkably high compared to other techniques, including DEAT-3, whose training accuracy is only slightly above 43%.
On the other hand, while the FREE-8-trained model gains robustness during epochs 7 to 10, its training accuracy declines substantially to about 47%.
Inspired by this phenomenon, we design another naive implementation of DEAT based on the training accuracy, i.e. , adding one more batch replay when the model’s training accuracy exceeds the pre-defined threshold.
We set the threshold at 40% and execute DEAT accordingly, signified by DEAT-40%.
As shown in the third column of Fig.
2, DEAT-40% achieves a considerable performance with a smoother learning procedure compared to DEAT-3.
[Table]
Alongside the robustness performance it is natural to question DEAT’s efficacy.
Because the total number of backpropagation, the most computationally expensive operation in AT, can be viewed as a quantification of the training cost [33] we summarise the number of backpropagation of AT methods presented so far in Table 2.
Letting M be the number of data batches, U-FGSM and PGD-AT-7 conducted 20 M and 80 M times backpropagation in ten training epochs.
FREE-8 spent the same amount of computational cost as PGD-AT-7, while
FREE-4 was half cheaper (40 M ) in the same period.
By contrast, DEAT-3 only conducted 22 M times backpropagation, given by ⌈ T ( T + d ) /
2 d ⌉ , where T is the total number of epochs, and DEAT-40% performed 25 M times.
Such a performance indicates that DEAT has a significant advantage in training efficiency.
Nevertheless, although both naive implementations perform well in the preliminary test, they are not sufficiently generalisable to various datasets and model architectures.
For instance, when doing DEAT-40% on a different training dataset, the training accuracy may never exceed the predefined threshold.
If DEAT-3 is conducted with 50 epochs, the last training epoch will have 17 batch replays, which is unacceptable.
In order to effectively configure
DEAT with these naive criteria for varied training tasks, extensive parameter tuning may be required, hence limiting DEAT’s usefulness.
To further improve DEAT’s generalisability, we need an adaptive strategy that does not rely on manual setup.
3. Gradient Magnitude Guided Approach
Recent studies [18, 26, 30] show that the adversarial robustness property is closely related to the geometry of the loss landscape, where AT enhance the trained models’ robustness via flatting its loss landscape.
In this sense, we propose utilising the model’s loss curvature as an indicator of the training effect.
Theorem 1.
Given a perturbed example x + δ , where x ∈ R n and δ ∈ B ϵ , if Assumptionsand 2 hold, the Lipschitz constant l xx of the first-order derivative of f ( x + δ ; θ ) towards training example x satisfies
[Equation]
Recall that l xx in Eq.
(3), a Lipschitz constant of the first-order derivative, implies the curvature around a particular example on the loss landscape by definition.
Given by
Theorem 1, we show that Eq.
(4) gives a reasonable approximation of the Lipschitz constant l xx of ∇ x f ( x + δ ; θ ) within a small ℓ ∞ norm ball [14] (proof is presented in Appendix B).
In practice, once the training set X and adversarial threat model are fixed, n and ϵ in Eq.
(4) are constants.
The flatness around one example can then be measured by ∥∇ x f ( x + δ ; θ ) ∥ 1 , and we take an accumulation of the flatness of examples in X , i.e. ,
[Equation]
to quantify the overall curvature of a model.
To see whether the gradient magnitude can reflect the training effect, we visualise the gradient magnitude in the fifth row of Fig.
For comparison, robustness gain, the changes in the trained models’ robustness relative to the previous epoch, and a loose Lipschitz constant of f ( · ; θ ) at the end of each training epoch are also visualised in the fourth and sixth rows of Fig.
Through an epoch-by-epoch comparison in Fig.
2, we can see that the gradient magnitude precisely reflects each adversarial training epoch’s effectiveness.
When the robustness of the models increases, the value of gradient magnitude is moderate, but when the robustness decreases, the gradient magnitude of that epoch increases dramatically.
In contrast, although a more robust model tends to have a smaller first-order Lipschitz constant [29], the latter can only indicate roughly that an adversarially trained model is becoming more robust but is unable to identify which training epoch was more effective.
While both theoretical and empirical analysis suggests that the trained model gains robustness effectively when gradient magnitude l X maintains small, how to quantify a threshold for l X to enable self-adaptive
AT remains a question.
DAT [31] initialises its threshold via conducting a weak adversarial attack on pre-trained models, while the success of U-FGSM [33] demonstrates that one step attack with uniform initialisation could generate qualified training perturbation.
Inspired by these previous works, we initialise the increasing threshold to l X at the second epoch that only contains one adversarial iteration.
As described in Algorithm 1, we slightly enlarge the threshold via a relax parameter γ > 1 at initialisation and each update.
By doing so, the number of adversarial iteration k will only be increased when the current l X is strictly larger than in previous epochs.
Because ∇ x f ( x + δ ; θ ) has already been
Algorithm
Require:
Training set X , total epochs T , adversarial radius ϵ , step size α , the number of mini-batches M , the number of adversarial iteration k , the evaluation of current training effect l X and a relax parameter γ 1: k ←2: for t = 1 to T do 3: l X ←4:
for i = 1 to M do 5:
Craft adversarial perturbation δ i via k backpropagation.
Save the gradient ∥∇ x f ( x i + δ i ; θ ) ∥ 1 at the least iteration.
7: l X ← l X + ∥∇ x f ( x i
+ δ i ; θ ) ∥8:
Update model parameter θ .
9: end for 10: if t = 1 then 11: k ← k +12: else if t = 2 then 13:
Threshold ← γ · l X ▷ Initialise threshold 14: else if l X > Threshold then 15:
Threshold ← γ · l X ; k ← k +▷ Update threshold 16: end if 17: end for computed in generating adversarial training examples, such a criterion only takes a trivial cost to enable the fully automatic training procedure.
Furthermore, through the different implementations of the training procedure in lines 5 to 7 of Algorithm 1, we enable the M+ criterion to collaborate with various existing AT methods.
Experiments
This section evaluates the proposed methods on the CIFAR-10 and ImageNet datasets.
All experiments are built with Pytorch [23] on a workstation that has an Intel i9-9820X processor and 128GB memory.
For CIFAR-10 experiments, we use a GeForce RTX 2080Ti, while the
ImageNet experiments run on 3 RTX 2080Ti graphics cards.
4. A benchmark under a fair training setup
To compare with recently efficient AT methods, we first present a benchmark on the CIFAR10 dataset under the same training setup, where all environmental variables are under control except AT methods themselves, to make a fair comparison.
In addition, TRADES [38] and
MART [32] are adopted as advanced surrogate loss functions that can be accelerated [39].
We make a fair examination of all baselines on training PreAct-ResNet 18 [13] and
DenseNet 121 [15], where all methods are carried out with 50 epochs and a multi-step learning rate schedule.
The detailed setup of training variables and baselines is summarised in Appendix C.1.
The number of adversarial iterations performed for perturbing training examples has a critical impact on the performance of baseline methods.
Therefore, we adopt only the optimal configuration from their original implementations.
As for robustness evaluation, we use PGD- i to represent a PGD adversary with i iterations, while CW- i adversary optimises the margin loss [6] via i PGD steps.
The adversarial step size is 2 / 255, and the adversarial radio ϵ is 8 / 255.
The standard AutoAttack [7], denoted by AA, is also included in our evaluation.
Each adversarially trained model is tested by a CW-20 adversary on a validation set during training, and we save the most robust checkpoints for the final evaluation to avoid the robust overfitting [33].
The benchmark is summarised in Table 3.
The loss and robustness of trained models are illustrated in Appendix C.4.
How M+ methods adjust the number of adversarial iterations is visualised in Appendix D. On the PreAct-ResNet 18, which is one of the most commonly used model architectures in adversarial training literature, we can see that TRADES achieves the best performance on the AA test, but it is also the most time-consuming.
M+TRADES
[Table]
is 26% faster than TRADES with a similar overall performance
[Table]
while F+TRADES obtains comparable robustness in half of TRADES’s training time.
According to Fig. 4(a) and
Fig. 6(a)
[Table]
M+TRADES’s training efficiency is comparable to F+TRADES’s at the early and middle stages
[Table]
but the increased adversarial steps slow it down at the final training stage.
MART achieves the highest robust performance on the PGD-100 test and is notably faster than TRADES, but there is no substantial improvement on the CW-20 and AA tests.
Our
M+MART is 40% faster and achieves better robust performance, whereas F+MART is even slower and less robust than the original MART.
In the PGD-based group, PGD-AT-7 is a strong baseline under the standard training setup [24], and it outperforms most accelerated
AT methods on robustness tests except Amata.
As shown in Fig. 6(a), M+PGD only conducts at most 4 backpropagations to generate training adversarial examples, which limits its final robustness performance.
M+DEAT is about 35% faster than PGD-AT-7 and achieves comparable robustness regarding the CW-20 and AA tests.
FREE-6 is more efficient than
FREE-8 and shows similar robustness.
M+DEAT is about 17% faster than FREE-6 and has higher accuracy on all evaluation tasks.
As demonstrated by Fig. 4(a) and Fig.
6(a),
M+DEAT performs fewer batch replays than other FREE methods at the early training stage, resulting in a large reduction in time consumption, which is to be expected.
Because DenseNet 121 have received less attention in the literature on adversarial training, existing AT approaches that rely on manual setup cannot perform adequately on this model.
Our M+ acceleration, on the other hand, overcomes the limitations imposed by manual adjustment and enables its cooperative methods to attain better performance in this benchmark.
AT methods combined with our M+ acceleration achieve significantly better results than their original implementation regarding computing cost and AA score.
How
M+ methods change the number of adversarial iterations on DenseNet 121 is displayed in
Fig. 6(b).
Due to the space limitation, the evaluation under FAST training setup is presented in Appendix C.2, while the empirical analysis of hyper-parameters and ablation studies of training tricks are presented in Appendix C.3.
4. Evaluation on ImageNet
In this section, we evaluate the M+DEAT on ImageNet.
Because the classical AT method could take thousands of GPU hours to adversarially train an ImageNet classifier [34], we consider FREE-4 as the baseline method, which finishes training within a reasonable amount of time and achieves better robustness than PGD AT [26].
Although U-FGSM can also work on ImageNet, its training procedure includes several tricks, i.e. , three training phases with different levels of cropping and learning rate schedules.
Due to too many manual factors,
U-FGSM is not considered as a baseline here.
Because Ye et al.
[35] did not release the implementation of the Amata method.
We only list their result on ImageNet for accuracy comparison.
To make a fair comparison, we use the same training setting as FREE-4’s original implementation 1 .
As shown in Table 4, M+DEAT can work properly on ImageNet, and the acceleration effect is considerable.
Compared to FREE-4, M+DEAT is about 34% faster and shows similar levels of robustness when γ = 1 . 1.
Because a small γ could introduce more adversarial iterations, M+DEAT with γ = 1 . 01 is only 16% faster than FREE-4, but it achieves slightly better performance on robustness tests.
[Table]
Conclusion and Future Work
In this paper, we demonstrate that the gradient magnitude can indicate the effect of adversarial training and design a self-adaptive acceleration strategy accordingly.
The proposed acceleration method, M+ acceleration, can easily collaborate with state-of-the-art adversarial training approaches to improve their training efficiency.
Comprehensive experiments have been done on CIFAR-10 and ImageNet datasets to evaluate the efficiency and effectiveness of the proposed method, where M+ acceleration performs exceptionally well with various training setups, demonstrating its generalisability and adaptability.
We believe that the ultimate goal of this direction would be an efficient adversarial training framework that can work under multiple threat models on various deep learning tasks.
However, this work has mainly explored ℓ ∞ norm-based adversarial training, so one of the future works is to generalise the magnitude guided strategy into threat models under other metrics.
Besides, recent works [11, 21] proposed to constrain the gradient magnitude via regularisation, which could also improve the trained models’ robustness.
Although regularising gradient magnitude cannot affect the number of adversarial iterations directly, we would like to adopt and examine such regularisation methods in DEAT in the next step.
The potential collaboration between our strategy and other efficient adversarial training methods like [35, 37] will also be explored in our future works.
B Proof of Theorem
Recall that
Theorem 1.
Given a perturbed example x + δ , where x ∈ R n and δ ∈ B ϵ , if Assumptionsand 2 hold, the Lipschitz constant l xx of the first-order derivative of f ( x + δ ; θ ) towards training example x satisfies
[Equation]
Proof.
Substituting x + δ and the stationary point x + δ ∗ into Eq.
(3) gives
[Equation]
Because x + δ ∗ is a stationary point and ∇ x f ( x + δ ∗ ; θ ) = 0, the left side of Eq.
(6) can be written as ∥∇ x f ( x + δ ; θ ) ∥ 1 .
The right side of Eq.
(6) can be simplified as
[Equation]
The first inequality holds because of the triangle inequality, and the second inequality holds because both ∥ δ ∥ ∞ and ∥ δ ∗ ∥ ∞ are upper bounded by B ϵ that is given by nϵ .
So Eq.
(6) can be written as Eq.
This concludes the proof.
C Additional Experiments C. Experiment setup in Section 4.
This benchmark includes two model architectures, i.e. , PreAct-ResNet 18 and DenseNet 121.
All models are trained with 50 epochs, where we use the stochastic gradient descent optimiser and a multi-step learning rate schedule, where the learning rate is decayed at 25 andepochs.
The initial learning rate for PreAct-ResNet 18 is 0.05.
However, DenseNet 121 fail to converge when trained with the same learning rate as PreAct-ResNet 18.
Therefore, we set their initial learning rate at 0.01.
All training hyper-parameters are reported in Table 5.
In Tab. 3, FAT [39] is carried out with 5 maximum training adversarial iterations.
According to [38, 39], both original TRADES and FAT with TRADES surrogate loss, denoted by
F+TRADES, use 10 adversarial steps.
The same number of adversarial steps is also applied
[Table]
in the original MART and FAT with MART surrogate loss (F+MART), which is the same setup used by [32].
As for the hyper-parameter τ , which controls the early stop procedure in
FAT, we conduct the original FAT, F+TRADES and F+MART at τ = 3, which enables their highest robust performance [39].
Furthermore, YOPO-5-3 [37] performs 3 backpropagations to update training adversarial examples and repeats such a procedure 5 times.
Amata-210 [35] linearly increases the number of adversarial iterations from 2 to 10 during training.
FREE is performed with three different numbers of batch replays, where FREE-8 conductsbatch replays and would show better robustness, while FREE-4 only does 4 batch replays and should be relatively more efficient [26].
All gradient magnitude accelerated methods, namely M+DEAT, M+PGD, M+TRADES, and M+MART, are performed at γ = 1 . 01, which is the relax parameter.
For the purpose of reproducibility, we release the code at https://github.com/TrustAI/DEAT .
C. Utilising the FAST training setup
To demonstrate M+DEAT’s adaptability, we compare it with two naive DEAT methods,
FREE-4, FREE-8, and PGD-AT-7 under the FAST Training Setup on CIFAR-10 and report the evaluation results.
Recall that the FAST training setup was proposed by [33], which utilises the cyclic learning rate [28] and half-precision computation [20] to speed up adversarial training.
Here, the maximum of cyclic learning rate is carried out at 0 .
As a reminder, DEAT-3 increases the number of batch replays every three epochs, and DEAT-40% determines the timing of increase based on the training accuracy.
All DEAT methods are evaluated at step size α = 10 / 255.
We set γ = 1 for PreAct-ResNet and γ = 1 . 01 for
Wide-ResNet [36].
All methods are carried out with 10 adversarial training epochs.
Because
DEAT methods start training on natural examples, DEAT-3 has 12 training epochs in total, and DEAT-40% and M+DEAT run 11 epochs.
We report the average performance of these methods on three random seeds.
The results on PreAct-ResNet 18 are shown in Table 6, and we can see that all DEAT methods perform properly under the FAST training setup.
Among the three DEAT methods, models trained by M+DEAT show the highest robustness.
Because of the increasing number of batch replays at the later training epochs, M+DEAT performs more backpropagations to finish training, so it spends slightly more training time.
FREE-4 achieves a comparable robust performance to M+DEAT, but M+DEAT is 20% faster and has higher classification accuracy on the natural test set.
The time consumption of FREE-8 almost triples those two
DEAT methods, but it only achieves a similar amount of robustness to DEAT-3.
PGD-ATuses the same number of backpropagations as FREE-8 but performs worse.
The comparison on Wide-ResNet [36] model also provides a similar result.
The evaluation results on Wide-ResNet 34 architecture are presented in Table 7.
We can see that most methods perform better than before because Wide-ResNet 34 is more complex and has more parameters than PreAct-ResNet 18.
Although FREE-8 achieves the highest overall robustness, its training cost is much more expensive than others.
Compared to FREE-8,
FREE-4 is half cheaper than FREE-8 and also performs considerably.
M+DEAT is 22% faster than FREE-4 with comparable robustness and achieves 97% training effect of FREE-  with 40% time consumption.
Other DEAT methods are faster than M+DEAT because fewer backpropagations have been conducted.
Models trained by them show reasonable robustness on all tests, but their performance is not as good as that of M+DEAT.
Please note that the experiment on Wide-ResNet is meant to demonstrate the generalizability of M+ acceleration, not to challenge the robustness record.
[Table]
[Table]
Comparing to U-FGSM Here, we report the competition between DEAT and U-FGSM.
The comparison is summarised in Table 8.
We can see that the differences in robustness performance between U-FGSM and DEAT methods are subtle under the FAST training setup.
They achieve a similar balance between robustness and time consumption, while M+DEAT performs better on most tests but also takes slightly more training time.
When using the standard training setup, U-FGSM is faster but only achieves a fair robust performance.
C. Empirical analysis
In this section, we explore the impact of two hyper-parameters, i.e. , step size α and relax parameter γ , and conduct ablation studies on several tricks involved in the benchmark under the standard training setup.
The impact of hyper-parameters α and γ
Under the FAST training setup, we evaluate the trained models’ performance with different step sizes at γ ∈ { 1 . 1 , 1 . 01 , 1 . 001 } .
Each model is trained with 10 epochs, and each combination of α and γ has been tested on 5 random seeds.
To highlight the relationship between models’ robustness and the corresponding training cost, which is denoted by the number of backpropagations (#BP), we use PGD-20 error that is computed as one minus the model’s classification accuracy on PGD-20 adversarial examples.
The result has been summarised in Fig.
This experiment shows that the models’ robustness is more sensitive to α rather than γ .
We can also see that M+DEAT achieves a slightly better robustness performance when using a large step size, e.g., 14/255, but the corresponding training cost is much higher than that with a lower step size when γ is small.
Ablation studies of three training tricks The default training tricks in Table 3 are multi-step learning rate, uniformly Initialised perturbation, and slightly enlarge the step size α .
Under the standard training setup, we investigate whether these tricks are truly
[Table]
effective for M+DEAT.
In addition to simply removing these tactics, we also evaluate an alternative for each.
For the learning rate schedule, we consider the flat learning rate and cyclic learning rate.
The perturbation initialisation is performed with zero initialisation and normally random initialisation.
The step size α is additionally carried out at 8/ and 12/255.
It can be seen from Table 9 that the flat learning rate significantly weakens performance across the board, indicating that a dynamic learning rate schedule is necessary to accelerate training convergence.
Comparing cyclic learning rate with multi-step learning rate, M+DEAT with the former achieves marginally better robustness performance with 10% longer runtime.
In terms of robustness performance, uniform initialisation surpasses normal and zero initialisation, whereas zero initialisation is actually better than normal initialisation.
The most appropriate step size is 10/255.
In the meantime, enlarging the step size raises the computational cost of M+DEAT.
Such an effect can also be observed in Fig.
[Table]
C. Visualisation of training process in Sec 4.
The adversarial training on PreAct-ResNet 18 is illustrated in Fig.
4 in terms of validation accuracy and training loss.
[Figure]
The adversarial training on DenseNet 121 is illustrated in Fig.
4 in terms of validation accuracy and training loss.
[Figure]
D Visualisation of how the number of backpropagations change in M+ methods
M+ acceleration allows its cooperative methods to increase the number of adversarial iterations during training, resulting in less backpropagation and increased efficiency.
To illustrate the adversarial training process of M+ techniques, we display in Fig.
6 the number of backpropagation operations performed by these methods.
[Figure]