E LIMINATING C ATASTROPHIC O VERFITTING V IA A BNORMAL A DVERSARIAL E XAMPLES R EGULARIZA -
TION
Anonymous authors
Paper under double-blind review
A BSTRACT
Single-step adversarial training (SSAT) is shown to be able to defend against iterative-step adversarial attacks to achieve both efficiency and robustness.
However, SSAT suffers from catastrophic overfitting (CO) with strong adversaries, showing that the classifier decision boundaries are highly distorted and robust accuracy against iterative-step adversarial attacks suddenly drops from peak to nearly 0% in a few epochs.
In this work, we find that some adversarial examples generated on the network trained by SSAT exhibit anomalous behaviour, that is, although the training data is generated by the inner maximization process, the loss of some adversarial examples decreases instead, which we called abnormal adversarial examples.
Furthermore, network optimization on these abnormal adversarial examples will further accelerate the model decision boundaries distortion, and correspondingly, the number of abnormal adversarial examples will sharply increase with CO.
These observations motivate us to prevent CO by hindering the generation of abnormal adversarial examples.
Specifically, we design a novel method, Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the number and outputs variation of abnormal adversarial examples to hinder the model from generating abnormal adversarial examples.
Extensive experiments demonstrate that our method can prevent CO and further boost adversarial robustness with strong adversaries.
I NTRODUCTION
In recent years, Deep Neural Networks (DNNs) have performed impressively in various fields, such as autonomous driving (Litman, 2017), face recognition (Sharif et al., 2016) and medical imaging diagnosis (Buch et al., 2018).
However, DNNs were found to be vulnerable to adversarial examples (Szegedy et al., 2013).
Although these adversarial examples are imperceptible to the human eyes, they can lead to a completely different prediction in DNNs.
To this end, many adversarial defense methods have been proposed, such as verification and provable defense (Katz et al., 2017), preprocessing techniques (Guo et al., 2017), detection algorithms (Metzen et al., 2017) and adversarial training (AT) (Goodfellow et al., 2014).
Among them, AT is considered to be one of the most effective methods against adversarial attacks (Athalye et al., 2018).
However, standard iterativestep AT significantly increases computational overhead due to multiple steps forward and backward propagation.
Therefore, some works attempt to improve the vanilla single-step adversarial training (SSAT) to defend against iterative-step adversarial attacks while maintaining efficiency and robustness.
Unfortunately, a serious problem - catastrophic overfitting (CO) - occurs with stronger adversaries (Wong et al., 2020).
This strange phenomenon means that the robust accuracy of the model against the iterative-step adversarial attack suddenly from peak drops to nearly zero during a few epochs, as shown in Figure 1.
This intriguing phenomenon has been widely investigated and led to many works to resolve CO.
Recently, Kim et al.
(2021) points out that networks in which CO occurs are generally accompanied by highly distorted decision boundaries.
However, the interaction between distorted decision boundaries and CO remains unclear.
In this work, we delve into the dynamic effects between CO and decision boundaries distortion.
Specifically, we find some adversarial examples generated on the network with distorted decision boundaries exhibit anomalous behavior, that is, although all training samples are generated by the inner maximization process, the loss of some adversarial examples decreases instead.
We refer to these training samples as abnormal adversarial examples.
To make matters worse, the decision boundaries distortion will further exacerbate by optimising the classifier directly on these abnormal adversarial examples, and the number of abnormal adversarial examples will sharply increase as a result, which leads to a vicious circle between the number of abnormal adversarial example and the decision boundaries distortion.
All these atypical findings raise a question:
Can CO be prevented by hindering the generation of abnormal adversarial examples?
To answer the above question, we design a novel method, Abnormal Adversarial Examples Regularization (AAER), which incorporates a regularizer that prevents CO by suppressing generated abnormal adversarial examples.
Specifically, AAER consists of two key components: (i) the number and (ii) outputs variation of abnormal adversarial examples.
The first part (i) counts the sample number by dividing the training samples into groups of normal and abnormal adversarial examples through anomalous loss decrease behavior.
The second part (ii) contains prediction confidence and logits variation, and calculates these two variations differences between the two groups of samples by cross-entropy and Euclidean distance, respectively.
Then, AAER explicitly regularizes the number and outputs variation of abnormal adversarial examples by these two parts to hinder the model from generating abnormal adversarial examples.
Extensive experiments show that our method can well eliminate CO and further improve the adversarial robustness.
It is worth noting that our method does not involve the extra generation and backward propagation process, which brings us unparalleled convenience in computational overhead.
Our major contributions are summarized as follows:
• We found some training samples exhibit anomalous loss variation during the inner maximization process.
Besides, the number of abnormal adversarial examples will sharply increase with CO, and the model will further exacerbate by optimising the classifier directly on these abnormal adversarial examples.
• Based on the observed effect, we propose a novel method - Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the number of abnormal adversarial examples and their anomalous outputs variation to hinder the generation of abnormal adversarial examples.
Extensive experiments demonstrate that our method can prevent CO and automatically adapt to different noise magnitudes without hyperparameter tuning.
• We evaluate the effectiveness of our method across different adversarial budgets, adversarial attacks, datasets and network architectures, showing that our proposed method consistently achieves state-of-the-art robust accuracy in SSAT and can obtain comparable robustness to standard iterative-step AT with only negligible computational overhead.
R ELATED W ORK 2. A DVERSARIAL T RAINING
Adversarial training has been demonstrated to be the most effective method for defending against adversarial attacks (Athalye et al., 2018).
AT is generally formulated as a min-max optimization problem, the inner maximization problem tries to generate the strongest adversarial examples to maximise the loss, and the outer minimization problem tries to optimize the network to minimize the loss on adversarial examples.
However, the inner maximization problem is a non-convex NPhard problem.
Therefore, AT uses a simple gradient ascent to generate perturbations to find local approximate solution, and can be formalized as a min-max optimization problem as follows:
[Equation]
where ( x, y ) is the training dataset from the distribution D , ℓ ( x, y ; θ ) is the loss function parameterized by θ , δ is the perturbation confined within the boundary ϵ with L p - norm distance, shown as: ∆ = { δ : ∥ δ ∥ p ≤ ϵ } .
The common threat models are L 1 , L 2 and L ∞ , in this work we chose L ∞ for our threat model.
Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) is a single-step adversarial attack method, which uses the sign of the gradient to find the perturbation, as shown in Eq.
[Equation]
Fast Training (RS-FGSM)
(Wong et al., 2020) adds an uniform random initialization η before generate the perturbation, and uses the over-perturbation step size α = 1 . 25 · ϵ :
[Equation] [Equation]
Iterative Fast Gradient Sign Method (I-FGSM)
(Kurakin et al., 2018) is an iterative-step version of FGSM that uses multiple gradients to find stronger perturbations.
With a smaller step size α = ϵ/N and the number of iterations T , I-FGSM can be formulated as follows:
[Equation]
Projected Gradient Descent (PGD) (Madry et al., 2017) adds an uniform random initialization on the basis of I-FGSM.
2. C ATASTROPHIC O VERFITTING
Since Wong et al.
(2020) found the CO phenomenon, there has been a line of work trying to explore and mitigate this problem.
Vivek & Babu (2020b) empirically showed that adding a dropout layer after all non-linear layers can avoid early overfitting to FGSM.
de Jorge et al.
(2022) found that augmenting the perturbations by increasing the noise initialization magnitude and removing the perturbation boundaries can eliminate CO.
Li et al.
(2022) successfully prevents CO by constraining training samples in a carefully extracted subspace to avoid the abrupt growth of gradient.
Other works attempt to prevent CO by strengthening the inner maximization processes.
Kim et al.
(2021) assumed that CO is caused by fixed FGSM perturbation magnitude and reduces the perturbed step size for misclassified adversarial examples.
Golgooni et al.
(2021) argued that small gradients play a key role in CO and ignores small gradient information to avoid huge weight updates.
Huang et al.
(2022) discovered that fitting instances with a larger gradient norm are more likely to cause
CO and learning an instance-adaptive step size is inversely proportional to its gradient norm.
Park & Lee (2021) leverages the gradients of latent representation as the latent adversarial perturbation to compensate for local linearity.
Similar to our work, some works add a regularization term on the loss value to explicitly prevent
CO. Andriushchenko & Flammarion (2020) found that PGD and FGSM perturbations are orthogonal when CO occurs, hence they proposed a regularization term to encourage the gradient alignment.
Vivek & Babu (2020a) proposed a regularization term to mitigate the CO by harnessing properties that differentiate a robust model from that of a pseudo robust model.
Sriramanan et al.
(2021) introduced a relaxation term to find more suitable gradient directions for the attack by smoothing the loss surface.
Chen et al.
(2021) demonstrated that the negative high-order terms lead to a perturbation loss distortion phenomenon that will cause CO, and they proposed a regularization term to make the loss surface flat.
P ROPOSED A PPROACH
In this section, we first define abnormal adversarial example and show how their number change during CO (Section 3.1).
We further analyse the outputs variation of normal and abnormal adversarial examples and find that they exhibit significantly different magnitudes of outputs variation after CO (Section 3.2).
Based on our observations, we propose a novel regularization term, AAER, using the number and outputs variation of abnormal adversarial examples to explicit suppress the generation of these anomalous training samples to eliminate CO (Section 3.3).
3. D EFINITION AND C OUNTING THE A BNORMAL A DVERSARIAL E XAMPLE
Adversarial training employs the most adversarial data to reduce the sensitivity of the network’s output w.r.t.
adversarial perturbation of the natural data.
Therefore, we expect the inner maximization process can generate effective adversarial examples that can maximize the classification loss.
However, Kim et al.
(2021) shows that the decision boundaries of the classifier will be highly distorted accompanied by the occurrence of CO.
After adding the adversarial perturbation which is generated on this distorted classifier, the classification loss of some training samples is atypically reduced.
As showed in Figure 2, it can be seen that, for some samples (blue), they will misclassify the model or be closer to the decision boundary after the inner maximization process, and for some other samples (red), they are farther to the decision boundary after adding the perturbation generated by the distorted classifier, which we called abnormal adversarial example.
These abnormal adversarial examples generally fail to mislead the classifier.
Thus, we can define abnormal adversarial examples using the following formula:
[Equation]
We further observe the changes in the number of abnormal adversarial examples during model training, and the statistical results are shown in Figure 3 (left).
It can be observed that before CO occurs, the number of abnormal adversarial examples is very small, almost close to 0.
During the occurrence of CO, their number increases sharply.
For example, the number of abnormal adversarial examples surged 39 times (red line) at the 10th epoch.
Note that the rapid increase in the number of abnormal adversarial examples also implies that the classifier boundaries are continuously deteriorating, which also leads to a further rise in the number of abnormal adversarial examples and peak at the 13th epoch, which is approximately 55 times the number of abnormal adversarial examples before
CO occurred.
After the occurrence of CO, the abnormal adversarial examples are basically maintained at a very large number.
Given this observation, we can infer that there is a close correlation between the number of abnormal adversarial examples and the CO phenomenon, which also prompts us to wonder (Q1): whether CO can be mitigated by reducing the number of abnormal adversarial examples.
3. O UTPUTS V ARIATION OF N ORMAL AND A BNORMAL A DVERSARIAL E XAMPLE
The above observations indicate that CO and the number of abnormal adversarial examples are closely related.
In this part, we further analyze the outputs variation of normal and abnormal adversarial examples during CO.
Specifically, we analyze the outputs into two categories: prediction confidence and logits, and use the cross-entropy to calculate the prediction confidence variation during the inner maximization process, which is formulated as follows:
[Equation]
From Figure 3 (middle), we can observe that the change in the prediction confidence of normal adversarial samples is greater than 0, indicating that the perturbation leads to a worse predictions.
However, the variation of abnormal adversarial examples is atypical negative, meaning that the perturbation has the opposite effect as we expected.
Furthermore, we analyze the prediction confidence variation of abnormal adversarial examples during training. Before the occurrence of CO, we can observe that the prediction confidence variation of abnormal adversarial examples is slightly less than zero, and the negative impact on all training samples (blue line) is not significant. During the occurrence of CO, their prediction confidence variation decreases rapidly slumped 27 times at the 10th epoch, and deterministically effect the prediction confidence of all training samples. Furthermore, we compare the magnitude of the outputs change between normal and abnormal adversarial examples, and use the Euclidean distance (L2 distance) to calculate the sample logits variation during the inner maximization process, which is formulated as follows:
[Equation]
where f θ is the DNN classifier parameterized by θ and ∥ · ∥ 22 is the L2 distance.
The magnitude of the logits variation of normal and abnormal adversarial examples is shown in Figure 3 (right).
We can observe that the logits variation magnitude of abnormal adversarial examples increases dramatically during CO, which is 16 times larger than that before CO.
A single-step gradient ascent can bring an earth-shaking change in the output logits, which generally happen on highly distorted decision boundaries.
Additionally, we observed that the logits variation magnitude of the normal adversarial examples (green line) increases one epoch later than the abnormal ones, which indicates that the model boundary distortion mainly lies in the abnormal adversarial examples, in other words, directly optimizing the network with these abnormal adversarial examples will further exacerbate the model boundary distortion.
Moreover, we further compare the magnitude of logits variation for normal and abnormal adversarial examples.
From Figure 3 (right), we can observe that the logits variation magnitude on normal and abnormal adversarial examples is similar before CO.
However, there is a significant difference in the logits variation magnitude between these two types of examples after CO.
It is observed that the logits variation magnitude in abnormal adversarial examples is 4 times that in normal ones at the 10th epoch.
There are significant differences in the magnitude of both prediction confidence and logits variation between normal and abnormal adversarial examples, which inspires us to wonder (Q2): whether CO can be mitigated by constraining the outputs variation of abnormal adversarial examples.
3. A BNORMAL A DVERSARIAL E XAMPLES R EGULARIZATION T ERM
We answer these two questions through three optimization objectives.
To answer the Q1, the first part (i) uses the Eq.
5 to divide the training samples into normal and abnormal adversarial examples, and then penalize the number of abnormal adversarial examples.
To answer the Q2, the second part (ii) and third part (iii) constrain the outputs variation of abnormal adversarial examples. Specifically, the second part (ii) calculates the prediction confidence variation of abnormal adversarial examples, and then penalizes this variation that should not decrease during the inner maximization process, which is formalized as follows: [Equation] ) where n is the number of abnormal adversarial examples.
The third part (iii) calculates the logits variation of normal and abnormal adversarial examples, but unlike the prediction confidence variation, the logits variation does not have a clear evaluation standard as an optimization objective that only has different magnitudes.
Therefore, we use the logits variation of normal adversarial examples as the standard, and explicitly constrain the logits variation of abnormal ones.
In order to clarify that the network optimization objective is to reduce the logits variation of abnormal adversarial samples instead of increasing the normal ones logits variation, we use the max function to limit the minimum value to 0, which is formalized as follows:
[Equation]
where m is the number of training samples and max ( , ) is the max function.
Based on the above analysis, we design a novel regularization term, AAER, which aims to suppress the abnormal adversarial examples by (i) the number, (ii) the prediction confidence variation and (iii) the logits variation, ultimately achieving the purpose of preventing CO, which is shown in the following formula:
[Equation]
where λ 1 and λ 2 is the hyperparameter to control the strength of the regularization term.
AAER can effectively hinder the generation of abnormal adversarial examples which are highly correlated with distorted classifier, thereby encouraging training of smoother classifiers that can better defend against adversarial attacks.
Furthermore, the strength of AAER depends on the product of the number and outputs variation of abnormal adversarial examples, which can more comprehensively and flexibly reflect the degree of classifier distortion.
The algorithm realization is summarized in Algorithm 1.
Note that we employ increasing α to stabilize the optimization objective and avoid model training to crash in the early stages.
E XPERIMENT
In this section, we conduct extensive experiments to verify the effectiveness of AAER including experiment settings (Section 4.1), performance evaluations (Section 4.2), ablation studies (Section 4.3) and time complexity study (Section 4.4). Input: network f θ , epochs T, mini-batch M, perturbation radius ϵ , step size α , initialization term η .
1: for t = 1 . . .
T to do 2: for i = 1 . . .
M to do 3: α = t/T · α 4: δ = α · sign ( ∇ x + η ℓ ( x i + η, y i ; θ )) 5:
CEloss = 1 m (cid:80) mi =1 ℓ ( x i + η + δ, y i ; θ ) 6:
AAERloss =
Eq.
(10) 7: θ = θ − ∇ θ ( CEloss + t/T · AAERloss, ) 8: end for 9: end for 4.
E XPERIMENT S ETTING
BaseLines.
We compare our method with other SSAT methods including RS-FGSM (Wong et al., 2020), ATTA (Zheng et al., 2020), FreeAT (Shafahi et al., 2019), N-FGSM (de Jorge et al., 2022),
Grad Align (Andriushchenko & Flammarion, 2020), ZeroGrad and MultiGrad (Golgooni et al., 2021).
We also compare our method with iterative-step AT PGD-2 and PGD-10 (Madry et al., 2017) providing a reference for the ideal performance.
To accommodate different adversarial budgets, we use PGD-10 with two step size of 2/255 and ϵ /10.
We will show the result for both natural and robust test accuracy using the hyperparameters reported in their official repository (except for FreeAT, we do not divide the number of epochs by m to keep the same training epochs).
It is worth noting that we do not use early stopping (Wong et al., 2020) as this technique can restore the robustness of all methods.
Datasets and Model Architectures.
We will show the results on the benchmark datasets Cifar10/
(Krizhevsky et al., 2009) and use random cropping and horizontal flipping for data argumentation.
We use the PreactResNet-18 (He et al., 2016) and WideResNet-34 (Zagoruyko & Komodakis, 2016) architectures on these datasets to evaluate results.
The training results of WideResNetare also available in the Appendix B. We also report the settings and results of our method on
SVHN (Netzer et al., 2011) in the Appendix E.
Attack Methods and Learning Rate Schedule.
To report the robust accuracy of models, we attack these methods using the standard PGD adversarial attack with α = ϵ/ 4 , 10 restarts and 50 attack steps.
We also evaluate our methods based on Auto Attack in the Appendix C. We use the cyclical learning rate schedule (Smith, 2017) with 30 epochs that reaches its maximum learning rate (0.2 in our experiments) when half of the epochs (15) are passed on Cifar-10/100.
Setup for Our Proposed Method.
In this work, we use the SGD optimizer with momentum of 0. and weight decay of 5 × 10 − 4 .
We chose L ∞ as the threat model and set gradient ascent step size α = 1 . 5 · ϵ .
We set η = Uniform ( − ϵ, ϵ ) for random initialization, and the η setting for previous initialization can be found in the Appendix A.
We will show the best λ settings in the Appendix D.
It is worth noting that our method can also achieve robustness without tuning hyperparameters with different adversarial budgets, the results on universal λ in the Appendix D.
4. P ERFORMANCE E VALUATION
In this part, we report the experimental results of our method under four different settings AAER RC :
AAER with random initialization and clipped perturbations and AAER RUC : AAER with random initialization and unclipped perturbations. The unclipped technique was proposed by de Jorge et al. (2022), who claimed that clipping is performed after taking a gradient ascent step, which may make adversarial samples no longer effective. The AAER based on the previous initialization are available in the Appendix A.
CIFAR10 Results.
In Table 1, we present an evaluation of the proposed methods with the competing baselines on the CIFAR-10 dataset.
First, we can observe that RS-FGSM, ATTA and FreeAT suffer from CO with strong adversaries.
We can also observe an interesting phenomenon that some weakly robust methods will recover partial robustness with large noise magnitude 32/255.
Table [Table] shows that our proposed methods can significantly improve the robust accuracy, achieve superior robustness to other SSAT methods and even have comparable robustness to PGD AT.
Interestingly, our unclipped perturbations methods always have better performance, except for 32/255 noise magnitude, where we conjecture that the unclipped perturbations are too large to disturb the original features of the inputs.
CIFAR100 Results.
We also conduct experiments on the CIFAR100 dataset.
Note that CIFAR is more challenging than CIFAR10 as the number of classes/training images per class is ten times larger/smaller than that of CIFAR10.
As shown by the results in Table 1, the proposed methods are still able to prevent CO and improve robust accuracy.
It verifies that the AAER can reliably prevent
CO and is general across different datasets.
4. A BLATION S TUDY
In this part, we investigate the impacts of AAER RC with 16/255 noise magnitude using PreactResNet-18 on CIFAR10 dataset under L ∞ threat model.
Optimization Objectives. To verify the effectiveness of our proposed method, we show the change in the three optimization objectives during training in Figure 3. We can observe that the number, prediction confidence and logits variation of abnormal adversarial examples are well constrained by AAER throughout the training.
λ Selection.
To verify the effectiveness of our proposed method, we investigate the effect of different sizes λ 1 and λ 2 on natural and robustness performance.
From the figure 4 (left), we can observe that the effect of λ 1 does not seem to be significant.
However, it acts as a buffer to prevent the AAER from changing too drastically, and we chose λ 1 of 8.0 for optimal preference.
From the Figure(right), we can observe that the model can successful prevent CO when λ 2 is not 0, which proves that our method can effectively eliminate CO.
Under the same experiment setting mentioned before, with the value of λ varying from 0 to 10.0, we can observe that choosing λ 2 of 5.0 can achieve the best robustness. 4. T IME C OMPLEXITY S TUDY
We show the time complexity of different AT methods in Table 2, we can observe that the running time for one epoch of AAER is almost equal to the RS-FGSM method. In contrast, Grad Align and PGD-10 are 2.3 and 4.6 times slower than our method, respectively. [Table]
In this paper, we find that the abnormal adversarial examples exhibit anomalous behaviour, i.e. they are further to the decision boundaries after adding perturbations generated by the inner maximization process. We empirically show that the catastrophic overfitting is closely related to the abnormal adversarial examples by analyzing their number and outputs variation during model training. Motivated by this, we propose a novel and effective method, Abnormal Adversarial Examples
Regularization (AAER), through a regularizer to eliminate catastrophic overfitting by suppressing generated abnormal adversarial examples.
Our approach can successfully resolve the catastrophic overfitting with different noise magnitudes and achieve state-of-the-art preference with computational convenience in various settings.
A E XPERIMENT WITH P REVIOUS I NITIALIZATION
Most works build perturbations based on zero or random initialization, but Zheng et al.
(2020) and
Liu et al.
(2021) found that perturbations are highly transferable between models from adjacent epochs, so they used perturbations from adjacent epochs to intensify the effect of perturbations, which is formalized as follows:
[Equation]
where t is the epoch, η t − 1 and δ t − 1 saved from the adjacent epoch and β is the hyperparameter to control the strength of the initialization.
In this part
[Table]
we will show the effect of our method by using previous initialization by AAER PC :
AAER with previous initialization and clipped perturbations and AAER PUC : AAER with previous initialization and unclipped perturbations.
We set β = 0 .
5 for the previous initialization experiments
[Table]
and report the results on Cifar10/100 in Table 3 and Table
[Table]
[Table]
[Table]
From Table 3 and Table
[Table]
we can observe that our method with previous initialization can still successfully achieve high robustness, even achieves better robustness in some settings compared to random initialization.
However, using previous initialization has some negative effects on natural accuracy.
β Selection. The hyperparameter β determines the strength of the previous initialization perturbations, and the effect of different β on test accuracy is shown in Figure 5.
When β is 0 that is equivalent to using zero initialization, increasing β leads to a higher natural accuracy.
When β is greater than 0.5, it is observed increasing β makes model robustness decrease.
Therefore, we set β to 0.5 to achieve the best trade-off between natural and robust test accuracy.
Data Argumentation Technique.
We notice that Zheng et al.
(2020) proposed a data argumentation technique ATTA, which adds different arguments at each epoch.
We add this data argumentation technique on AAER PC as shown in Table 3 and Table 4.
We can observe that the ATTA does not or slight improves our method accuracy, but the training time will significantly increase from 30.5S to 43.1S. Therefore, our method AAER do not use the data argumentation technique ATTA
[Table]
B E XPERIMENT WITH W IDE R ES N ET A RCHITECTURE
We also compare the performance of our method using WideResNet, which is more complex than
PreActResNet.
The settings are the same as PreActResNet-18, and we report the results on Cifar10/100 in Table 5 and Table 6.
[Table]
From Table 5 and Table 6, we can observe that our method can still successfully achieve high robustness in other architectures.
Although, the PGD-10 AT seems to better utilize the complex network to achieve higher natural and robust accuracy.
However, it is worth noting that complex networks can better reflect the efficiency of our method in terms of training time, while our method can achieve comparable robustness. [Table]
C E VALUATION B ASED ON A UTO A TTACK
Auto Attack Croce & Hein (2020) is regarded as the most reliable robustness evaluation to date, which is an ensemble of complementary attacks, consisting of three white-box attacks APGD-CE,
APGD-DLR
[Table]
and FAB and a black-box attack Square Attack.
We report the results on Cifar10/ in Table 7 and Table
[Table]
[Table]
In Table 7 and Table
[Table]
we can observe that our method can still achieve boost adversarial robustness in different adversarial attacks.
Surprisingly, the unclipped AAER achieves higher robustness with 32/255 noise magnitude under Auto Attack, which is slightly different from the result under PGD50-10 attack
[Table]
D E XPERIMENT WITH U NIVERSAL λ
It is worth noting that unlike other SSAT methods (such as Grad Align (Andriushchenko & Flammarion, 2020) and ZeroGrad (Golgooni et al., 2021)), our method can achieve robustness without tuning hyperparameters with different adversarial budgets, the universal λ settings are shown in Ta- [Table] ble 9 and Table 10.
For CIFAR-10, we set λ 1 = 8 . {+
 λ 2 = 5 .+} 0 for AAER with clipped perturbations, and λ 1 = 6 . 5 {+λ 2 = 5 . 
+} for AAER with unclipped perturbations.
For CIFAR-100, we set λ 1 = 7 .λ 2 = 3 . 0 , and λ 1 = 6 .[-
-]
5 λ 2 = 2 .
5 for AAER with clipped and unclipped perturbations, respectively.
[Table]
[Table] We report the universal λ results on Cifar10/100 in Table 11 and Table 12.
We can observe that the results using universal λ can still achieve high robustness on both datasets.
The absence of hyperparameter tuning provides our method with unparalleled generality and adaptability.
[Table]
different noise magnitudes using
PreactResNet-18 under L ∞ threat model.
The top number is the natural accuracy while the bottom number is the PGD-50-10 accuracy
The results are averaged over 3 random seeds and reported with the standard deviation
[Table]
E SVHN S ETTINGS AND R ESULTS
SVHN Settings and Results.
For experiments on SVHN, we follow the settings of de Jorge et al.
(2022), which use the cyclical learning rate schedule with 15 epochs that reaches its maximum learning rate (0.05) when 40% (6) epochs are passed.
We show the best λ settings on SVHN in the
Table 13.
In Table 14, we show the AAER performance on the SVHN dataset, and the competing baselines result from de Jorge et al.
We can observe that our method can effectively improve the robust accuracy and prevent CO with different noise magnitudes.
[Table]
[Table]