Original: B AG OF T RICKS FOR A DVERSARIAL T RAINING
Revised: B AG OF T RICKS FOR A DVERSARIAL T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial training (AT) is one of the most effective strategies for promoting model robustness.
Revised: Adversarial training (AT) is one of the most effective strategies for promoting model robustness.

Original: However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure.
Revised: However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure.

Original: This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods.
Revised: This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods.

Original: Surprisingly, we ﬁnd that the basic settings (e.g., weight decay, training schedule, etc.) used in these methods are highly inconsistent.
Revised: Surprisingly, we ﬁnd that the basic settings (e.g., weight decay, training schedule, etc.) used in these methods are highly inconsistent.

Original: In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models.
Revised: In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models.

Original: Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought.
Revised: Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought.

Original: For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7% , which is probable to override the potential promotion induced by the proposed methods.
Revised: For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7% , which is probable to override the potential promotion induced by the proposed methods.

Original: We conclude a baseline training setting and re-implement previous defenses to achieve new state-of-the-art results.
Revised: We conclude a baseline training setting and re-implement previous defenses to achieve new state-of-the-art results.

Original: These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.
Revised: These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adversarial training (AT) has been one of the most effective defense strategies against adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015).
Revised: Adversarial training (AT) has been one of the most effective defense strategies against adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015).

Original: Based on the primary
Revised: Based on the primary

Original: AT frameworks like PGD-AT (Madry et al., 2018), many improvements have been proposed from different perspectives, and demonstrate promising results (detailed in Sec. 2).
Revised: AT frameworks like PGD-AT (Madry et al., 2018), many improvements have been proposed from different perspectives, and demonstrate promising results (detailed in Sec. 2).

Original: However, the recent benchmarks (Croce & Hein, 2020b; Chen & Gu, 2020) ﬁnd that simply early stopping the training procedure of PGD-AT (Rice et al., 2020) can attain the gains from almost all the previously proposed improvements, including the state-of-the-art TRADES (Zhang et al., 2019b).
Revised: However, the recent benchmarks (Croce & Hein, 2020b; Chen & Gu, 2020) ﬁnd that simply early stopping the training procedure of PGD-AT (Rice et al., 2020) can attain the gains from almost all the previously proposed improvements, including the state-of-the-art TRADES (Zhang et al., 2019b).

Original: This fact is somewhat striking since TRADES also executes early stopping (one epoch after decaying the learning rate) in the code implementation 1 , as clariﬁed in Rice et al.
Revised: This fact is somewhat striking since TRADES also executes early stopping (one epoch after decaying the learning rate) in the code implementation 1 , as clariﬁed in Rice et al.

Original: Besides, the reported robustness of PGD-AT in Rice et al.
Revised: Besides, the reported robustness of PGD-AT in Rice et al.

Original: (2020) is much higher than in Madry et al.
Revised: (2020) is much higher than in Madry et al.

Original: (2018), even without early-stopping.
Revised: (2018), even without early-stopping.

Original: This paradox motivates us to check the implementation details of Rice et al.
Revised: This paradox motivates us to check the implementation details of Rice et al.

Original: (2020) 2 , and compare the basic hyperparameter settings with TRADES.
Revised: (2020) 2 , and compare the basic hyperparameter settings with TRADES.

Original: We ﬁnd that TRADES uses weight decay of 2 × 10 − 4 and eval mode of batch normalization (BN) when crafting adversarial examples, while Rice et al.
Revised: We ﬁnd that TRADES uses weight decay of 2 × 10 − 4 and eval mode of batch normalization (BN) when crafting adversarial examples, while Rice et al.

Original: (2020) use weight decay of 5 × 10 − 4 and train mode of BN to generate adversarial examples.
Revised: (2020) use weight decay of 5 × 10 − 4 and train mode of BN to generate adversarial examples.

Original: In our experiments on CIFAR-10 (e.g., Table 8), the two slightly different settings can differ the robust accuracy by ∼ 4% , which is signiﬁcant according to the reported benchmarks.
Revised: In our experiments on CIFAR-10 (e.g., Table 8), the two slightly different settings can differ the robust accuracy by ∼ 4% , which is signiﬁcant according to the reported benchmarks.

Original: To have a comprehensive study, we further investigate the implementation details of tens of papers working on the AT methods, some of which are summarized in Table 1.
Revised: To have a comprehensive study, we further investigate the implementation details of tens of papers working on the AT methods, some of which are summarized in Table 1.

Original: We ﬁnd that even using the same model architectures, the basic hyperparameter settings (e.g., weight decay, learning rate schedule, etc.) used in these papers are highly inconsistent and customized, which could affect the model performance and may override the gains from the methods themselves.
Revised: We ﬁnd that even using the same model architectures, the basic hyperparameter settings (e.g., weight decay, learning rate schedule, etc.) used in these papers are highly inconsistent and customized, which could affect the model performance and may override the gains from the methods themselves.

Original: Under this situation, if we directly benchmark these methods using their released code or checkpoints, some actually effective improvements would be under-estimated due to the improper hyperparameter settings.
Revised: Under this situation, if we directly benchmark these methods using their released code or checkpoints, some actually effective improvements would be under-estimated due to the improper hyperparameter settings.

Original: Our contributions.
Revised: Our contributions.

Original: We evaluate the effects of a wide range of basic training tricks (e.g., warmup, early stopping, weight decay, batch size, BN mode, etc.) on the adversarially trained models.
Revised: We evaluate the effects of a wide range of basic training tricks (e.g., warmup, early stopping, weight decay, batch size, BN mode, etc.) on the adversarially trained models.

Original: Our empirical results suggest that improper training settings can largely degenerate the model performance,
Revised: Our empirical results suggest that improper training settings can largely degenerate the model performance,

Original: We convert the training steps into epochs, and provide code links for reference in Table 11.
Revised: We convert the training steps into epochs, and provide code links for reference in Table 11.

Original: Compared to the model architectures, the listed settings are easy to be neglected and paid less attention to unify.
Revised: Compared to the model architectures, the listed settings are easy to be neglected and paid less attention to unify.

Original: [Table]
Revised: [Table]

Original: × 10 −
Revised: × 10 −

Original: No / No No / No
Revised: No / No No / No

Original: Atzmon et al.
Revised: Atzmon et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: No / No No / No
Revised: No / No No / No

Original: Wong et al.
Revised: Wong et al.

Original: ∼ 0.
Revised: ∼ 0.

Original: 30 (one cycle)
Revised: 30 (one cycle)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: No / No
Revised: No / No

Original: Yes / No
Revised: Yes / No

Original: Rice et al.
Revised: Rice et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Yes / No
Revised: Yes / No

Original: No / No
Revised: No / No

Original: Ding et al.
Revised: Ding et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: No / No No / No
Revised: No / No No / No

Original: Pang et al.
Revised: Pang et al.

Original: (2020a) 0.
Revised: (2020a) 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: No / No No / No
Revised: No / No No / No

Original: Zhang et al.
Revised: Zhang et al.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: No /
Revised: No /

Original: Yes
Revised: Yes

Original: No / No
Revised: No / No

Original: Huang et al.
Revised: Huang et al.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: anneal)
Revised: anneal)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: No / No
Revised: No / No

Original: Yes / No
Revised: Yes / No

Original: Cheng et al.
Revised: Cheng et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: No / No No / No
Revised: No / No No / No

Original: Lee et al.
Revised: Lee et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: No / No No / No
Revised: No / No No / No

Original: Xu et al.
Revised: Xu et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: No / No
Revised: No / No

Original: No / No while this degeneration may be mistakenly ascribed to the methods themselves.
Revised: No / No while this degeneration may be mistakenly ascribed to the methods themselves.

Original: We provide a baseline recipe for PGD-AT on CIFAR-10 as an example, and demonstrate the generality of the recipe on training other frameworks like TRADES.
Revised: We provide a baseline recipe for PGD-AT on CIFAR-10 as an example, and demonstrate the generality of the recipe on training other frameworks like TRADES.

Original: As seen in Table 14, the retrained TRADES achieve new state-of-the-art performance on the AutoAttack benchmark (Croce & Hein, 2020b).
Revised: As seen in Table 16, the retrained TRADES achieve new state-of-the-art performance on the AutoAttack benchmark (Croce & Hein, 2020b).

Original: Although our empirical conclusions may not generalize to other datasets or tasks, we reveal the facts that adversarially trained models could be sensitive to certain training settings, which are usually neglected in previous work.
Revised: Although our empirical conclusions may not generalize to other datasets or tasks, we reveal the facts that adversarially trained models could be sensitive to certain training settings, which are usually neglected in previous work.

Original: These results also encourage the community to re-implement the previously proposed defenses with ﬁne-tuned training settings to better explore their potentials.
Revised: These results also encourage the community to re-implement the previously proposed defenses with ﬁne-tuned training settings to better explore their potentials.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In this section, we introduce related work on the adversarial defenses and recent benchmarks.
Revised: In this section, we introduce related work on the adversarial defenses and recent benchmarks.

Original: We detail on the adversarial attacks in Appendix A.1.
Revised: We detail on the adversarial attacks in Appendix A.1.

Original: 2. A DVERSARIAL DEFENSES
Revised: 2. A DVERSARIAL DEFENSES

Original: To alleviate the adversarial vulnerability of deep learning models, many defense strategies have been proposed, but most of them can eventually be evaded by adaptive attacks (Carlini & Wagner, 2017b;
Revised: To alleviate the adversarial vulnerability of deep learning models, many defense strategies have been proposed, but most of them can eventually be evaded by adaptive attacks (Carlini & Wagner, 2017b;

Original: Athalye et al., 2018).
Revised: Athalye et al., 2018).

Original: Other more theoretically guaranteed routines include training provably robust networks (Dvijotham et al., 2018a;b; Hein & Andriushchenko, 2017; Wong & Kolter, 2018) and obtaining certiﬁed models via randomized smoothing (Cohen et al., 2019).
Revised: Other more theoretically guaranteed routines include training provably robust networks (Dvijotham et al., 2018a;b; Hein & Andriushchenko, 2017; Wong & Kolter, 2018) and obtaining certiﬁed models via randomized smoothing (Cohen et al., 2019).

Original: While these methods are promising, they currently do not match the state-of-the-art robustness under empirical evaluations.
Revised: While these methods are promising, they currently do not match the state-of-the-art robustness under empirical evaluations.

Original: The idea of adversarial training (AT) stems from the seminal work of Goodfellow et al.
Revised: The idea of adversarial training (AT) stems from the seminal work of Goodfellow et al.

Original: (2015), while other AT frameworks like PGD-AT (Madry et al., 2018) and TRADES (Zhang et al., 2019b) occupied the winner solutions in the adversarial competitions (Kurakin et al., 2018;
Revised: (2015), while other AT frameworks like PGD-AT (Madry et al., 2018) and TRADES (Zhang et al., 2019b) occupied the winner solutions in the adversarial competitions (Kurakin et al., 2018;

Original: Brendel et al., 2020).
Revised: Brendel et al., 2020).

Original: Based on these primary AT frameworks, many improvements have been proposed via encoding the mechanisms inspired from other domains, including ensemble learning (Tramèr et al., 2018;
Revised: Based on these primary AT frameworks, many improvements have been proposed via encoding the mechanisms inspired from other domains, including ensemble learning (Tramèr et al., 2018;

Original: Pang et al., 2019), metric learning (Mao et al., 2019; Li et al., 2019; Pang et al., 2020c), generative modeling (Jiang et al., 2018; Pang et al., 2018b; Wang & Yu, 2019; Deng et al., 2020), semisupervised learning (Carmon et al., 2019;
Revised: Pang et al., 2019), metric learning (Mao et al., 2019; Li et al., 2019; Pang et al., 2020c), generative modeling (Jiang et al., 2018; Pang et al., 2018b; Wang & Yu, 2019; Deng et al., 2020), semisupervised learning (Carmon et al., 2019;

Original: Alayrac et al., 2019; Zhai et al., 2019), and self-supervised
Revised: Alayrac et al., 2019; Zhai et al., 2019), and self-supervised

Original: [Table]
Revised: [Table]

Original: learning (Hendrycks et al., 2019; Chen et al., 2020a;b; Naseer et al., 2020).
Revised: learning (Hendrycks et al., 2019; Chen et al., 2020a;b; Naseer et al., 2020).

Original: On the other hand, due to the high computational cost of AT, many efforts are devoted to accelerating the training procedure via reusing the computations (Shafahi et al., 2019b; Zhang et al., 2019a), adaptive adversarial steps (Wang et al., 2019;
Revised: On the other hand, due to the high computational cost of AT, many efforts are devoted to accelerating the training procedure via reusing the computations (Shafahi et al., 2019b; Zhang et al., 2019a), adaptive adversarial steps (Wang et al., 2019;

Original: Zhang et al., 2020) or one-step training (Wong et al., 2020;
Revised: Zhang et al., 2020) or one-step training (Wong et al., 2020;

Original: Liu et al., 2020; Vivek B & Venkatesh Babu, 2020).
Revised: Liu et al., 2020; Vivek B & Venkatesh Babu, 2020).

Original: The following works try to solve the side effects (e.g., catastrophic overﬁtting) caused by these fast AT methods (Andriushchenko & Flammarion, 2020; Li et al., 2020).
Revised: The following works try to solve the side effects (e.g., catastrophic overﬁtting) caused by these fast AT methods (Andriushchenko & Flammarion, 2020; Li et al., 2020).

Original: 2. A DVERSARIAL BENCHMARKS
Revised: 2. A DVERSARIAL BENCHMARKS

Original: Due to the large number of proposed defenses, several benchmarks have been developed to rank the adversarial robustness of existing methods.
Revised: Due to the large number of proposed defenses, several benchmarks have been developed to rank the adversarial robustness of existing methods.

Original: Dong et al.
Revised: Dong et al.

Original: (2020) perform large-scale experiments to generate robustness curves, which are used for evaluating typical defenses.
Revised: (2020) perform large-scale experiments to generate robustness curves, which are used for evaluating typical defenses.

Original: Croce & Hein (2020b) propose AutoAttack, which is an ensemble of four selected attacks.
Revised: Croce & Hein (2020b) propose AutoAttack, which is an ensemble of four selected attacks.

Original: They apply AutoAttack on tens of previous defenses and provide a comprehensive leader board.
Revised: They apply AutoAttack on tens of previous defenses and provide a comprehensive leader board.

Original: Chen & Gu (2020) propose the black-box RayS attack, and establish a similar leader board for defenses.
Revised: Chen & Gu (2020) propose the black-box RayS attack, and establish a similar leader board for defenses.

Original: In this paper, we mainly apply PGD attack and AutoAttack as two common ways to evaluate the models.
Revised: In this paper, we mainly apply PGD attack and AutoAttack as two common ways to evaluate the models.

Original: Except for the adversarial robustness, there are other efforts that introduce augmented datasets for accessing the robustness against general corruptions or perturbations.
Revised: Except for the adversarial robustness, there are other efforts that introduce augmented datasets for accessing the robustness against general corruptions or perturbations.

Original: Mu & Gilmer (2019) introduce
Revised: Mu & Gilmer (2019) introduce

Original: MNIST-C with a suite of 15 corruptions applied to the MNIST test set, while Hendrycks & Dietterich (2019) introduce ImageNet-C and ImageNet-P with common corruptions and perturbations on natural images.
Revised: MNIST-C with a suite of 15 corruptions applied to the MNIST test set, while Hendrycks & Dietterich (2019) introduce ImageNet-C and ImageNet-P with common corruptions and perturbations on natural images.

Original: Evaluating robustness on these datasets can reﬂect the generality of the proposed defenses, and avoid overﬁtting to certain attacking patterns (Engstrom et al., 2019; Tramèr & Boneh, 2019).
Revised: Evaluating robustness on these datasets can reﬂect the generality of the proposed defenses, and avoid overﬁtting to certain attacking patterns (Engstrom et al., 2019; Tramèr & Boneh, 2019).

Original:  B AG OF TRICKS
Revised:  B AG OF TRICKS

Original: Our overarching goal is to investigate how the usually overlooked implementation details affect the performance of the adversarially trained models.
Revised: Our overarching goal is to investigate how the usually overlooked implementation details affect the performance of the adversarially trained models.

Original: Our experiments are done on CIFAR-10 (Krizhevsky & Hinton, 2009) under the (cid:96) ∞ threat model of maximal perturbation (cid:15) = 8 / 255 , without accessibility to additional data.
Revised: Our experiments are done on CIFAR-10 (Krizhevsky & Hinton, 2009) under the (cid:96) ∞ threat model of maximal perturbation (cid:15) = 8 / 255 , without accessibility to additional data.

Original: We evaluate the models under 10-steps PGD attack ( PGD-10 ) (Madry et al., 2018) and AutoAttack ( AA ) (Croce & Hein, 2020b).
Revised: We evaluate the models under 10-steps PGD attack ( PGD-10 ) (Madry et al., 2018) and AutoAttack ( AA ) (Croce & Hein, 2020b).

Original: For the PGD attack, we apply untargeted mode using ground truth labels, step size of 2 / 255 , and 5 restarts for evaluation / no restart for training.
Revised: For the PGD attack, we apply untargeted mode using ground truth labels, step size of 2 / 255 , and 5 restarts for evaluation / no restart for training.

Original: For the
Revised: For the

Original: AutoAttack 3 , we apply the standard version, with no restart for AutoPGD and FAB, compared torestarts for plus version.
Revised: AutoAttack 3 , we apply the standard version, with no restart for AutoPGD and FAB, compared torestarts for plus version.

Original: We consider some basic training tricks and perform ablation studies on each of them, based on the default training setting as described below:
Revised: We consider some basic training tricks and perform ablation studies on each of them, based on the default training setting as described below:

Original: Default setting.
Revised: Default setting.

Original: Following Rice et al.
Revised: Following Rice et al.

Original: (2020), in the default setting, we apply the primary PGD-AT framework and the hyperparameters including batch size 128 ; SGD momentum optimizer with the initial learning rate of 0 . 1 ; weight decay 5 × 10 − 4 ; ReLU activation function and no label smoothing; train mode for batch normalization when crafting adversarial examples.
Revised: (2020), in the default setting, we apply the primary PGD-AT framework and the hyperparameters including batch size 128 ; SGD momentum optimizer with the initial learning rate of 0 . 1 ; weight decay 5 × 10 − 4 ; ReLU activation function and no label smoothing; train mode for batch normalization when crafting adversarial examples.

Original: All the models are trained for 110 epochs with the learning rate decaying by a factor of 0 . 1 at 100 and 105 epochs, respectively.
Revised: All the models are trained for 110 epochs with the learning rate decaying by a factor of 0 . 1 at 100 and 105 epochs, respectively.

Original: We report the results on the checkpoint with the best PGD-10 accuracy .
Revised: We report the results on the checkpoint with the best PGD-10 accuracy .

Original: Note that our empirical observations and conclusions may not always generalize to other datasets or
Revised: Note that our empirical observations and conclusions may not always generalize to other datasets or

Original: AT frameworks, but we emphasize the importance of using consistent implementation details (not only the same model architectures) to enable fair comparisons among different AT methods.
Revised: AT frameworks, but we emphasize the importance of using consistent implementation details (not only the same model architectures) to enable fair comparisons among different AT methods.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. E ARLY STOPPING AND WARMUP
Revised: 3. E ARLY STOPPING AND WARMUP

Original: Early stopping training epoch.
Revised: Early stopping training epoch.

Original: The trick of early stopping w.r.t.
Revised: The trick of early stopping w.r.t.

Original: the training epoch was ﬁrst applied in the implementation of TRADES (Zhang et al., 2019b), where the learning rate decays at the 75 th epoch and the training is stopped at the 76 th epoch.
Revised: the training epoch was ﬁrst applied in the implementation of TRADES (Zhang et al., 2019b), where the learning rate decays at the 75 th epoch and the training is stopped at the 76 th epoch.

Original: Later Rice et al.
Revised: Later Rice et al.

Original: (2020) provide a comprehensive study on the overﬁtting phenomenon in AT, and advocate early stopping the training epoch as a general strategy for preventing adversarial overﬁtting, which could be triggered according to the PGD accuracy on a split validation set.
Revised: (2020) provide a comprehensive study on the overﬁtting phenomenon in AT, and advocate early stopping the training epoch as a general strategy for preventing adversarial overﬁtting, which could be triggered according to the PGD accuracy on a split validation set.

Original: Due to its effectiveness, we regard this trick as a default choice.
Revised: Due to its effectiveness, we regard this trick as a default choice.

Original: Early stopping adversarial intensity.
Revised: Early stopping adversarial intensity.

Original: Another level of early stopping happens on the adversarial intensity, e [Table] g., early stopping PGD steps when crafting adversarial examples for training.
Revised: Another level of early stopping happens on the adversarial intensity, e.g., early stopping PGD steps when crafting adversarial examples for training.

Original: This trick was ﬁrst applied by the runner-up of the defense track in NeurIPS 2018 adversarial vision challenge (Brendel et al., 2020).
Revised: This trick was ﬁrst applied by the runner-up of the defense track in NeurIPS 2018 adversarial vision challenge (Brendel et al., 2020).

Original: Later efforts are devoted to formalizing this early stopping mechanism with different trigger rules (Wang et al., 2019; Zhang et al., 2020).
Revised: Later efforts are devoted to formalizing this early stopping mechanism with different trigger rules (Wang et al., 2019; Zhang et al., 2020).

Original: Balaji et al.
Revised: Balaji et al.

Original: (2019) early stop the adversarial perturbation, which has a similar effect on the adversarial intensity.
Revised: (2019) early stop the adversarial perturbation, which has a similar effect on the adversarial intensity.

Original: In the left part of Table 2, we evaluate the method proposed by Zhang et al.
Revised: In the left part of Table 2, we evaluate the method proposed by Zhang et al.

Original: (2020) due to its simplicity.
Revised: (2020) due to its simplicity.

Original: As seen, this kind of early stopping can improve the performance on clean data while keeping comparable accuracy under PGD-10.
Revised: As seen, this kind of early stopping can improve the performance on clean data while keeping comparable accuracy under PGD-10.

Original: However, the performance under the stronger AutoAttack is degraded.
Revised: However, the performance under the stronger AutoAttack is degraded.

Original: Warmup w.r.t.
Revised: Warmup w.r.t.

Original: learning rate.
Revised: learning rate.

Original: Warmup w.r.t.
Revised: Warmup w.r.t.

Original: learning rate is a general trick for training deep learning models (Goodfellow et al., 2016).
Revised: learning rate is a general trick for training deep learning models (Goodfellow et al., 2016).

Original: In the adversarial setting, Wong et al.
Revised: In the adversarial setting, Wong et al.

Original: (2020) show that the one cycle learning rate schedule is one of the critical ingredients for the success of FastAT.
Revised: (2020) show that the one cycle learning rate schedule is one of the critical ingredients for the success of FastAT.

Original: Thus, we evaluate the effect of this trick for the piecewise learning rate schedule and PGD-AT framework.
Revised: Thus, we evaluate the effect of this trick for the piecewise learning rate schedule and PGD-AT framework.

Original: We linearly increase the learning rate from zero to the preset value in the ﬁrst 10 / 15 / 20 epochs.
Revised: We linearly increase the learning rate from zero to the preset value in the ﬁrst 10 / 15 / 20 epochs.

Original: As shown in the middle part of Table 2, the effect of warming up learning rate is marginal.
Revised: As shown in the middle part of Table 2, the effect of warming up learning rate is marginal.

Original: Warmup w.r.t.
Revised: Warmup w.r.t.

Original: adversarial intensity.
Revised: adversarial intensity.

Original: In the AT procedure, warmup can also be executed w.r.t.
Revised: In the AT procedure, warmup can also be executed w.r.t.

Original: the adversarial intensity.
Revised: the adversarial intensity.

Original: Cai et al.
Revised: Cai et al.

Original: (2018) propose the curriculum AT process to gradually increase the adversarial intensity and monitor the overﬁtting trend.
Revised: (2018) propose the curriculum AT process to gradually increase the adversarial intensity and monitor the overﬁtting trend.

Original: Qin et al.
Revised: Qin et al.

Original: (2019) increase the maximal
Revised: (2019) increase the maximal

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: best PGD-10 accuracy (Rice et al., 2020).
Revised: best PGD-10 accuracy (Rice et al., 2020).

Original: We test on two model architectures, and highlight (with red circles) three most commonly used weight decays in previous work; (b) Curves of test accuracy w.r.t.
Revised: We test on two model architectures, and highlight (with red circles) three most commonly used weight decays in previous work; (b) Curves of test accuracy w.r.t.

Original: training epochs, where the model is WRN-34-10.
Revised: training epochs, where the model is WRN-34-10.

Original: We set weight decay be 1 × 10 − 4 , 2 × 10 − 4 , and 5 × 10 − 4 , respectively.
Revised: We set weight decay be 1 × 10 − 4 , 2 × 10 − 4 , and 5 × 10 − 4 , respectively.

Original: We can observe that smaller weight decay can learn faster but also more tend to overﬁt w.r.t.
Revised: We can observe that smaller weight decay can learn faster but also more tend to overﬁt w.r.t.

Original: the robust accuracy.
Revised: the robust accuracy.

Original: In Fig.
Revised: In Fig.

Original: 4, we early decay the learning rate before the models overﬁtting, but weight decay of 5 × 10 − 4 still achieve better robustness.
Revised: 4, we early decay the learning rate before the models overﬁtting, but weight decay of 5 × 10 − 4 still achieve better robustness.

Original: perturbation (cid:15) from zero to 8 / 255 in the ﬁrst 15 epochs.
Revised: perturbation (cid:15) from zero to 8 / 255 in the ﬁrst 15 epochs.

Original: In the right part of Table 2, we linearly increase the maximal perturbation in the ﬁrst 10 / 15 / 20 epochs, while the effect is still limited.
Revised: In the right part of Table 2, we linearly increase the maximal perturbation in the ﬁrst 10 / 15 / 20 epochs, while the effect is still limited.

Original: 3. T RAINING HYPERPARAMETERS
Revised: 3. T RAINING HYPERPARAMETERS

Original: Batch size.
Revised: Batch size.

Original: On the large-scale datasets like ImageNet (Deng et al., 2009), it has been recognized that the mini-batch size is an important factor inﬂuencing the model performance (Goyal et al., 2017), where larger batch size traverses the dataset faster but requires more memory usage.
Revised: On the large-scale datasets like ImageNet (Deng et al., 2009), it has been recognized that the mini-batch size is an important factor inﬂuencing the model performance (Goyal et al., 2017), where larger batch size traverses the dataset faster but requires more memory usage.

Original: In the adversarial setting, Xie et al.
Revised: In the adversarial setting, Xie et al.

Original: (2019) use a batch size of 4096 to train a robust model on ImageNet, which achieves state-of-the-art performance under adversarial attacks.
Revised: (2019) use a batch size of 4096 to train a robust model on ImageNet, which achieves state-of-the-art performance under adversarial attacks.

Original: As to the defenses reported on the CIFARdataset, the mini-batch sizes are usually chosen between 128 and 256 , as shown in Table 1.
Revised: As to the defenses reported on the CIFARdataset, the mini-batch sizes are usually chosen between 128 and 256 , as shown in Table 1.

Original: To evaluate the effect, we test on two model architectures and four values of batch size in Table 3.
Revised: To evaluate the effect, we test on two model architectures and four values of batch size in Table 3.

Original: Since the number of training epochs is ﬁxed to 110 , we also consider applying the linear scaling rule introduced in Goyal et al.
Revised: Since the number of training epochs is ﬁxed to 110 , we also consider applying the linear scaling rule introduced in Goyal et al.

Original: (2017), i.e., when the mini-batch size is multiplied by k , multiply the learning rate by k .
Revised: (2017), i.e., when the mini-batch size is multiplied by k , multiply the learning rate by k .

Original: We treat the batch size of 128 and the learning rate of 0 . 1 as a basic setting to obtain the factor k .
Revised: We treat the batch size of 128 and the learning rate of 0 . 1 as a basic setting to obtain the factor k .

Original: We can observe that the batch size of 128 works well on CIFAR-10, while the linear scaling rule can beneﬁt the cases with other batch sizes.
Revised: We can observe that the batch size of 128 works well on CIFAR-10, while the linear scaling rule can beneﬁt the cases with other batch sizes.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Label smoothing.
Revised: Label smoothing.

Original: Shafahi et al.
Revised: Shafahi et al.

Original: (2019a) propose to utilize label smoothing to mimic the AT procedure.
Revised: (2019a) propose to utilize label smoothing to mimic the AT procedure.

Original: Pang et al.
Revised: Pang et al.

Original: (2019) also ﬁnd that imposing label smoothing on the ensemble prediction can alleviate the adversarial transferability among individual members, and thus promote robustness of the ensemble model.
Revised: (2019) also ﬁnd that imposing label smoothing on the ensemble prediction can alleviate the adversarial transferability among individual members, and thus promote robustness of the ensemble model.

Original: Beyond previous observations, we further evaluate the effect of label smoothing on AT under stronger AutoAttack and black-box RayS attack.
Revised: Beyond previous observations, we further evaluate the effect of label smoothing on AT under stronger AutoAttack and black-box RayS attack.

Original: As shown in Table 4, label smoothing can improve ∼ 1% worst-case robust accuracy, without affecting the clean performance.
Revised: As shown in Table 4, label smoothing can improve ∼ 1% worst-case robust accuracy, without affecting the clean performance.

Original: This can be regarded as the effect induced by calibrating the conﬁdence (Stutz et al., 2020) of adversarially trained models ( 80% ∼ 85% accuracy on clean data).
Revised: This can be regarded as the effect induced by calibrating the conﬁdence (Stutz et al., 2020) of adversarially trained models ( 80% ∼ 85% accuracy on clean data).

Original: However, excessive label smoothing could degrade the robustness (e.g., LS = 0 .
Revised: However, excessive label smoothing could degrade the robustness (e.g., LS = 0 .

Original: 3 vs. LS = 0 .
Revised: 3 vs. LS = 0 .

Original: 4 on ResNet-18 shown in Table 4), which is consistent with the recent observations in Jiang et al.
Revised: 4 on ResNet-18 shown in Table 4), which is consistent with the recent observations in Jiang et al.

Original: (2020) (they use LS = 0 . 5 ).
Revised: (2020) (they use LS = 0 . 5 ).

Original: Optimizer.
Revised: Optimizer.

Original: Most of the AT methods apply SGD with momentum as the optimizer.
Revised: Most of the AT methods apply SGD with momentum as the optimizer.

Original: The momentum factor is usually set to be 0 .
Revised: The momentum factor is usually set to be 0 .

Original: 9 with zero dampening.
Revised: 9 with zero dampening.

Original: In other cases, Carmon et al.
Revised: In other cases, Carmon et al.

Original: (2019) apply SGD with Nesterov, and Rice et al.
Revised: (2019) apply SGD with Nesterov, and Rice et al.

Original: (2020) apply Adam for cyclic learning rate schedule.
Revised: (2020) apply Adam for cyclic learning rate schedule.

Original: We test some commonly used optimizers in Table 5, as well as the decoupled AdamW (Loshchilov & Hutter, 2019) and the recently proposed gradient centralization trick SGD-GC / SGD-GCC (Yong et al., 2020).
Revised: We test some commonly used optimizers in Table 5, as well as the decoupled AdamW (Loshchilov & Hutter, 2019) and the recently proposed gradient centralization trick SGD-GC / SGD-GCC (Yong et al., 2020).

Original: We can ﬁnd that SGD-based optimizers (e.g., Mom, Nesterov, SGD-GC / SGD-GCC) have similar performance, while Adam / AdamW performs worse for piecewise learning rate schedule.
Revised: We can ﬁnd that SGD-based optimizers (e.g., Mom, Nesterov, SGD-GC / SGD-GCC) have similar performance, while Adam / AdamW performs worse for piecewise learning rate schedule.

Original: Weight decay.
Revised: Weight decay.

Original: As observed in Table 1, three different values of weight decay are used in previous defenses, including 1 × 10 − 4 , 2 × 10 − 4 , and 5 × 10 − 4 .
Revised: As observed in Table 1, three different values of weight decay are used in previous defenses, including 1 × 10 − 4 , 2 × 10 − 4 , and 5 × 10 − 4 .

Original: While 5 × 10 − 4 is a fairly widely used value for weight decay in deep learning, the prevalence of the value 2 × 10 − 4 should stem from Madry et al.
Revised: While 5 × 10 − 4 is a fairly widely used value for weight decay in deep learning, the prevalence of the value 2 × 10 − 4 should stem from Madry et al.

Original: (2018) in the adversarial setting.
Revised: (2018) in the adversarial setting.

Original: In Fig. 1(a), we report the best test accuracy under different values of weight decay 4 .
Revised: In Fig. 1(a), we report the best test accuracy under different values of weight decay 4 .

Original: We can see that the gap of robust accuracy can be signiﬁcant due to slightly different values of weight decay (e.g., up to ∼ 7% for 1 × 10 − 4 vs. 5 × 10 − 4 ).
Revised: We can see that the gap of robust accuracy can be signiﬁcant due to slightly different values of weight decay (e.g., up to ∼ 7% for 1 × 10 − 4 vs. 5 × 10 − 4 ).

Original: Besides, in Fig. 1(b)
Revised: Besides, in Fig. 1(b)

Original: [Table]
Revised: [Table]

Original: eval 87.
Revised: eval 87.

Original: 5 × 10 −
Revised: 5 × 10 −

Original: Softplus train 84.
Revised: Softplus train 84.

Original: 5 × 10 −
Revised: 5 × 10 −

Original: Softplus eval 87.
Revised: Softplus eval 87.

Original: [Figure]
Revised: [Figure]

Original: h t d ec ay v r v r
Revised: h t d ec ay v r v r

Original: [Figure]
Revised: [Figure]

Original: The is the examples
Revised: The is the examples

Original: 2019), we craft PGD-10 perturbation as the normal direction v , and r be a random direction, under the
Revised: 2019), we craft PGD-10 perturbation as the normal direction v , and r be a random direction, under the

Original: ∞ constraint of 8 / 255 .
Revised: ∞ constraint of 8 / 255 .

Original: The values of x-axis and y-axis represent the multiplied scale factors.
Revised: The values of x-axis and y-axis represent the multiplied scale factors.

Original: we plot the learning curves of test accuracy w.r.t.
Revised: we plot the learning curves of test accuracy w.r.t.

Original: training epochs.
Revised: training epochs.

Original: Note that smaller values of weight decay make the model learn faster in the initial phase, but the overﬁtting phenomenon also appears earlier.
Revised: Note that smaller values of weight decay make the model learn faster in the initial phase, but the overﬁtting phenomenon also appears earlier.

Original: In Fig.
Revised: In Fig.

Original: 3, we visualize the cross sections of the decision boundary.
Revised: 3, we visualize the cross sections of the decision boundary.

Original: We can see that proper values of weight decay (e.g., 5 × 10 − 4 ) can enlarge margins from decision boundary and improve robustness.
Revised: We can see that proper values of weight decay (e.g., 5 × 10 − 4 ) can enlarge margins from decision boundary and improve robustness.

Original: Nevertheless, as shown in the left two columns, this effect is less signiﬁcant on promoting clean accuracy.
Revised: Nevertheless, as shown in the left two columns, this effect is less signiﬁcant on promoting clean accuracy.

Original: As a result, weight decay is a critical and usually neglected ingredient that largely inﬂuences the robust accuracy of adversarially trained models.
Revised: As a result, weight decay is a critical and usually neglected ingredient that largely inﬂuences the robust accuracy of adversarially trained models.

Original: In contrast, the clean accuracy is much less sensitive to weight decay, for both adversarially and standardly (shown in Fig.
Revised: In contrast, the clean accuracy is much less sensitive to weight decay, for both adversarially and standardly (shown in Fig.

Original: 5) trained models.
Revised: 5) trained models.

Original: Activation function.
Revised: Activation function.

Original: Most of the previous AT methods apply ReLU as the non-linear activation function in their models, while Xie et al.
Revised: Most of the previous AT methods apply ReLU as the non-linear activation function in their models, while Xie et al.

Original: (2020) empirically demonstrate that smooth activation functions can better improve model robustness on ImageNet.
Revised: (2020) empirically demonstrate that smooth activation functions can better improve model robustness on ImageNet.

Original: Following their settings, we test if a similar conclusion holds on CIFAR-10.
Revised: Following their settings, we test if a similar conclusion holds on CIFAR-10.

Original: By comparing the results on ReLU and Softplus in Table(for PGD-AT) and Table 13 (for TRADES), we conﬁrm that smooth activation indeed beneﬁts model robustness for ResNet-18.
Revised: By comparing the results on ReLU and Softplus in Table(for PGD-AT) and Table 13 (for TRADES), we conﬁrm that smooth activation indeed beneﬁts model robustness for ResNet-18.

Original: However, as shown in Table 8 (for PGD-AT) and Table 9 (for TRADES), this beneﬁt is less signiﬁcant on larger models like WRN.
Revised: However, as shown in Table 8 (for PGD-AT) and Table 9 (for TRADES), this beneﬁt is less signiﬁcant on larger models like WRN.

Original: Thus we deduce that smaller model capacity can beneﬁt more from the smoothness of activation function.
Revised: Thus we deduce that smaller model capacity can beneﬁt more from the smoothness of activation function.

Original: Besides, as shown in Table 6, models trained on CIFAR-10 seem to prefer activation function σ ( x ) with zero truncation, i.e., σ ( x ) ≥ 0 .
Revised: Besides, as shown in Table 6, models trained on CIFAR-10 seem to prefer activation function σ ( x ) with zero truncation, i.e., σ ( x ) ≥ 0 .

Original: Those with negative return values like ELU, LeakyReLU, Tanh have worse performance than ReLU.
Revised: Those with negative return values like ELU, LeakyReLU, Tanh have worse performance than ReLU.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Model architecture.
Revised: Model architecture.

Original: Su et al.
Revised: Su et al.

Original: (2018) provide a comprehensive study on the robustness of standardly trained models, using different model architectures.
Revised: (2018) provide a comprehensive study on the robustness of standardly trained models, using different model architectures.

Original: For the adversarially trained models, it has been generally recognized that larger model capacity can usually lead to better robustness (Madry et al., 2018).
Revised: For the adversarially trained models, it has been generally recognized that larger model capacity can usually lead to better robustness (Madry et al., 2018).

Original: Recently, Guo et al.
Revised: Recently, Guo et al.

Original: (2020) blend in the technique of AutoML to explore robust architectures.
Revised: (2020) blend in the technique of AutoML to explore robust architectures.

Original: In Fig.
Revised: In Fig.

Original: 2, we perform similar experiments on more hand-crafted model architectures.
Revised: 2, we perform similar experiments on more hand-crafted model architectures.

Original: The selected models have comparable numbers of parameters.
Revised: The selected models have comparable numbers of parameters.

Original: We can observe that DenseNet can achieve both the best clean and robust accuracy, while being memory-efﬁcient (but may require longer inference time).
Revised: We can observe that DenseNet can achieve both the best clean and robust accuracy, while being memory-efﬁcient (but may require longer inference time).

Original: This is consistent with the observation in Guo et al.
Revised: This is consistent with the observation in Guo et al.

Original: (2020) that residual connections can beneﬁt the AT procedure.
Revised: (2020) that residual connections can beneﬁt the AT procedure.

Original: Interestingly, Wu et al.
Revised: Interestingly, Wu et al.

Original: (2020) demonstrate that residual connections allow easier generation of highly transferable adversarial examples, while in our case this weakness for the standardly trained models may turn out to strengthen the adversarially trained models.
Revised: (2020) demonstrate that residual connections allow easier generation of highly transferable adversarial examples, while in our case this weakness for the standardly trained models may turn out to strengthen the adversarially trained models.

Original: Batch normalization (BN) mode.
Revised: Batch normalization (BN) mode.

Original: When crafting adversarial examples in the training procedure,
Revised: When crafting adversarial examples in the training procedure,

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019b) use eval mode for BN, while Rice et al.
Revised: (2019b) use eval mode for BN, while Rice et al.

Original: (2020) and Madry et al.
Revised: (2020) and Madry et al.

Original: (2018) use train mode for BN.
Revised: (2018) use train mode for BN.

Original: Since the parameters in the BN layers are not updated in this progress, the difference between these two modes is mainly on the recorded moving average BN mean and variance used in the test phase.
Revised: Since the parameters in the BN layers are not updated in this progress, the difference between these two modes is mainly on the recorded moving average BN mean and variance used in the test phase.

Original: As pointed out in Xie & Yuille (2020), properly dealing with BN layers is critical to obtain a well-performed adversarially trained model.
Revised: As pointed out in Xie & Yuille (2020), properly dealing with BN layers is critical to obtain a well-performed adversarially trained model.

Original: Thus in Table 7, we employ the train or eval mode of BN for crafting adversarial examples during training, and report the results on different model architectures to dig out general rules.
Revised: Thus in Table 7, we employ the train or eval mode of BN for crafting adversarial examples during training, and report the results on different model architectures to dig out general rules.

Original: As seen, using eval mode for BN can increase clean accuracy, while keeping comparable robustness.
Revised: As seen, using eval mode for BN can increase clean accuracy, while keeping comparable robustness.

Original: We also advocate for the eval mode, because if we apply train mode for multi-step PGD attack, the BN mean and variance will be recorded for every intermediate step, which could blur the adversarial distribution used by BN layers during inference.
Revised: We also advocate for the eval mode, because if we apply train mode for multi-step PGD attack, the BN mean and variance will be recorded for every intermediate step, which could blur the adversarial distribution used by BN layers during inference.

Original: Takeaways:
Revised: Takeaways:

Original: (i) Slightly different values of weight decay could largely affect the robustness of trained models; (ii) Moderate label smoothing and linear scaling rule on l.r. for different batch sizes are beneﬁcial;
Revised: (i) Slightly different values of weight decay could largely affect the robustness of trained models; (ii) Moderate label smoothing and linear scaling rule on l.r. for different batch sizes are beneﬁcial;

Original: (iii) Applying eval BN mode to craft training adversarial examples can avoid blurring the distribution; (iv) Early stopping the adversarial steps or perturbation may degenerate worst-case robustness; (v) Smooth activation beneﬁts more when the model capacity is not enough for adversarial training.
Revised: (iii) Applying eval BN mode to craft training adversarial examples can avoid blurring the distribution; (iv) Early stopping the adversarial steps or perturbation may degenerate worst-case robustness; (v) Smooth activation beneﬁts more when the model capacity is not enough for adversarial training.

Original: [Table]
Revised: [Table]

Original: 3. C OMBINATION OF TRICKS
Revised: 3. C OMBINATION OF TRICKS

Original: In the above
Revised: In the above

Original: [Table]
Revised: [Table]

Original: we separately evaluate the effect of each training trick in the AT procedure.
Revised: we separately evaluate the effect of each training trick in the AT procedure.

Original: Now we investigate combining the selected useful tricks, which involve label smoothing, weight decay, activation function and BN mode.
Revised: Now we investigate combining the selected useful tricks, which involve label smoothing, weight decay, activation function and BN mode.

Original: As demonstrated in Table 8, the improvements are not ideally additive by combining different tricks
Revised: As demonstrated in Table 8, the improvements are not ideally additive by combining different tricks

Original: [Table]
Revised: [Table]

Original: while label smoothing and smooth activation function are helpful
Revised: while label smoothing and smooth activation function are helpful

Original: [Table]
Revised: [Table]

Original: but not signiﬁcant, especially when we apply model architectures with a larger capacity.
Revised: but not signiﬁcant, especially when we apply model architectures with a larger capacity.

Original: We also ﬁnd that the high performance of the models trained by Rice et al.
Revised: We also ﬁnd that the high performance of the models trained by Rice et al.

Original: (2020) partially comes from its reasonable training settings, compared to previous work.
Revised: (2020) partially comes from its reasonable training settings, compared to previous work.

Original: Based on these, we provide a trick list for training robust models on CIFAR-10 for reference.
Revised: Based on these, we provide a trick list for training robust models on CIFAR-10 for reference.

Original: Baseline setting (CIFAR-10)
Revised: Baseline setting (CIFAR-10)

Original: Batch size 128 ; SGD momentum optimizer; weight decay 5 × 10 − 4 ; eval mode BN for generating adversarial examples; warmups are not necessary; moderate label smoothing ( 0 .
Revised: Batch size 128 ; SGD momentum optimizer; weight decay 5 × 10 − 4 ; eval mode BN for generating adversarial examples; warmups are not necessary; moderate label smoothing ( 0 .

Original: ∼ 0 . 2 ) and smooth activation function could be beneﬁcial; model architecture with residual connections.
Revised: ∼ 0 . 2 ) and smooth activation function could be beneﬁcial; model architecture with residual connections.

Original: 3. R E - IMPLEMENTATION OF TRADES
Revised: 3. R E - IMPLEMENTATION OF TRADES

Original: As a sanity check, we re-implement TRADES to see if our conclusions derived from PGD-AT can generalize and provide the results in Table 9.
Revised: As a sanity check, we re-implement TRADES to see if our conclusions derived from PGD-AT can generalize and provide the results in Table 9.

Original: We can observe that after simply changing the weight decay from 2 × 10 − 4 to 5 × 10 − 4 , the clean accuracy of TRADES improves by ∼ 1% and the AA accuracy improves by ∼ 4% , which make the trained model surpass the previously state-of-theart models reported by the AutoAttack benchmark, as listed in Table 14.
Revised: We can observe that after simply changing the weight decay from 2 × 10 − 4 to 5 × 10 − 4 , the clean accuracy of TRADES improves by ∼ 1% and the AA accuracy improves by ∼ 4% , which make the trained model surpass the previously state-of-theart models reported by the AutoAttack benchmark, as listed in Table 16.

Original: This fact highlights the importance of employing a standardized training setting for fair comparisons of different AT methods.
Revised: This fact highlights the importance of employing a standardized training setting for fair comparisons of different AT methods.

Original: 3. E VALUATIONS ON OTHER AT FRAMEWORKS
Revised: 3. E VALUATIONS ON OTHER AT FRAMEWORKS

Original: To examine the universality of our observations on PGD-AT and TRADES, we further evaluate on other AT frameworks, including FastAT (Wong et al., 2020) and FreeAT (Shafahi et al., 2019b).
Revised: To examine the universality of our observations on PGD-AT and TRADES, we further evaluate on other AT frameworks, including FastAT (Wong et al., 2020) and FreeAT (Shafahi et al., 2019b).

Original: We base on the FastAT code 5 to implement the methods.
Revised: We base on the FastAT code 5 to implement the methods.

Original: Speciﬁcally, for FastAT, we use cyclic learning rate schedule with l min = 0 and l max = 0 . 2 , training for 15 epochs.
Revised: Speciﬁcally, for FastAT, we use cyclic learning rate schedule with l min = 0 and l max = 0 . 2 , training for 15 epochs.

Original: For FreeAT, we also use cyclic learning rate schedule with l min = 0 and l max = 0 .
Revised: For FreeAT, we also use cyclic learning rate schedule with l min = 0 and l max = 0 .

Original: 04 , training for 24 epochs with mini-batch replays be 4 .
Revised: 04 , training for 24 epochs with mini-batch replays be 4 .

Original: The results are provided in Table 10.
Revised: The results are provided in Table 10.

Original: We can ﬁnd that our observations generalize well to other
Revised: We can ﬁnd that our observations generalize well to other

Original: AT frameworks, which veriﬁes that the proposed baseline setting could be a decent default choice for adversarial training on CIFAR-10.
Revised: AT frameworks, which veriﬁes that the proposed baseline setting could be a decent default choice for adversarial training on CIFAR-10.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we take a step in examining how the usually neglected implementation details impact the performance of adversarially trained models.
Revised: In this work, we take a step in examining how the usually neglected implementation details impact the performance of adversarially trained models.

Original: Our empirical results suggest that compared to clean accuracy, robustness is more sensitive to some seemingly unimportant differences in training settings.
Revised: Our empirical results suggest that compared to clean accuracy, robustness is more sensitive to some seemingly unimportant differences in training settings.

Original: Thus when building AT methods, we should more carefully ﬁne-tune the training settings (on validation sets), or follow certain long-tested setup in the adversarial setting.
Revised: Thus when building AT methods, we should more carefully ﬁne-tune the training settings (on validation sets), or follow certain long-tested setup in the adversarial setting.

Original: A T ECHNICAL DETAILS
Revised: A T ECHNICAL DETAILS

Original: In this section we introduce more related backgrounds and technical details for reference.
Revised: In this section we introduce more related backgrounds and technical details for reference.

Original: A. A DVERSARIAL ATTACKS
Revised: A. A DVERSARIAL ATTACKS

Original: Since the seminal L-BFGS and FGSM attacks (Szegedy et al., 2014; Goodfellow et al., 2015), a large amount of attacking methods on generating adversarial examples have been introduced.
Revised: Since the seminal L-BFGS and FGSM attacks (Szegedy et al., 2014; Goodfellow et al., 2015), a large amount of attacking methods on generating adversarial examples have been introduced.

Original: In the white-box setting, gradient-based methods are popular and powerful, which span in the (cid:96) ∞ threat model (Nguyen et al., 2015; Madry et al., 2018), (cid:96) 2 threat model (Carlini & Wagner, 2017a), (cid:96) 1 threat model (Chen et al., 2018), and (cid:96) 0 threat model (Papernot et al., 2016).
Revised: In the white-box setting, gradient-based methods are popular and powerful, which span in the (cid:96) ∞ threat model (Nguyen et al., 2015; Madry et al., 2018), (cid:96) 2 threat model (Carlini & Wagner, 2017a), (cid:96) 1 threat model (Chen et al., 2018), and (cid:96) 0 threat model (Papernot et al., 2016).

Original: In the black-box setting, the attack strategies are much more diverse.
Revised: In the black-box setting, the attack strategies are much more diverse.

Original: These include transfer-based attacks (Dong et al., 2018; 2019; Cheng et al., 2019b), quasi-gradient attacks (Chen et al., 2017a;
Revised: These include transfer-based attacks (Dong et al., 2018; 2019; Cheng et al., 2019b), quasi-gradient attacks (Chen et al., 2017a;

Original: Uesato et al., 2018; Ilyas et al., 2018), and decision-based attacks (Brendel et al., 2018;
Revised: Uesato et al., 2018; Ilyas et al., 2018), and decision-based attacks (Brendel et al., 2018;

Original: Cheng et al., 2019a).
Revised: Cheng et al., 2019a).

Original: Adversarial attacks can be also realized in the physical world (Kurakin et al., 2017; Song et al., 2018a).
Revised: Adversarial attacks can be also realized in the physical world (Kurakin et al., 2017; Song et al., 2018a).

Original: Below we formulate the PGD attack and AutoAttack that we used in our evaluations.
Revised: Below we formulate the PGD attack and AutoAttack that we used in our evaluations.

Original: PGD attack.
Revised: PGD attack.

Original: One of the most commonly studied adversarial attack is the projected gradient descent (PGD) method (Madry et al., 2018).
Revised: One of the most commonly studied adversarial attack is the projected gradient descent (PGD) method (Madry et al., 2018).

Original: Let x 0 be a randomly perturbed sample in the neighborhood of the clean input x , then PGD iteratively crafts the adversarial example as
Revised: Let x 0 be a randomly perturbed sample in the neighborhood of the clean input x , then PGD iteratively crafts the adversarial example as

Original: [Equation]
Revised: [Equation]

Original: where clip x,(cid:15) ( · ) is the clipping function and L is the adversarial objective.
Revised: where clip x,(cid:15) ( · ) is the clipping function and L is the adversarial objective.

Original: The accuracy under PGD attack has been a standard metric to evaluate the model robustness.
Revised: The accuracy under PGD attack has been a standard metric to evaluate the model robustness.

Original: AutoAttack.
Revised: AutoAttack.

Original: Croce & Hein (2020b)
Revised: Croce & Hein (2020b)

Original: ﬁrst propose the Auto-PGD (APGD) algorithm, where the main idea is to automatically tune the adversarial step sizes according to the optimization trend.
Revised: ﬁrst propose the Auto-PGD (APGD) algorithm, where the main idea is to automatically tune the adversarial step sizes according to the optimization trend.

Original: As to the adversarial objective, except for the traditional cross-entropy (CE) loss, they develop a new difference of logits ratio (DLR) loss as
Revised: As to the adversarial objective, except for the traditional cross-entropy (CE) loss, they develop a new difference of logits ratio (DLR) loss as

Original: [Equation]
Revised: [Equation]

Original: where z is the logits and π is the ordering which sorts the components of z .
Revised: where z is the logits and π is the ordering which sorts the components of z .

Original: Finally, the authors propose to group APGD CE and APGD DLR with FAB (Croce & Hein, 2020a) and square attack (Andriushchenko et al., 2020) to form the AutoAttack (AA).
Revised: Finally, the authors propose to group APGD CE and APGD DLR with FAB (Croce & Hein, 2020a) and square attack (Andriushchenko et al., 2020) to form the AutoAttack (AA).

Original: A. R EFERENCE CODES
Revised: A. R EFERENCE CODES

Original: In Table 11, we provide the code links for the referred defenses.
Revised: In Table 11, we provide the code links for the referred defenses.

Original: The summarized training settings are either described in their papers or manually retrieved by us in their code implementations.
Revised: The summarized training settings are either described in their papers or manually retrieved by us in their code implementations.

Original: Table 11: We summarize the code links for the referred defense methods in Table 1.
Revised: Table 11: We summarize the code links for the referred defense methods in Table 1.

Original: Method
Revised: Method

Original: [Table]
Revised: [Table]

Original: (2019a)
Revised: (2019a)

Original: github.com/a1600012888/YOPO-You-Only-Propagate-Once
Revised: github.com/a1600012888/YOPO-You-Only-Propagate-Once

Original: Zhang &
Revised: Zhang &

Original: [Table]
Revised: [Table]

Original: github.com/Haichao-Zhang/FeatureScatter
Revised: github.com/Haichao-Zhang/FeatureScatter

Original: Atzmon et al.
Revised: Atzmon et al.

Original: github.com/matanatz/ControllingNeuralLevelsets
Revised: github.com/matanatz/ControllingNeuralLevelsets

Original: Wong et al.
Revised: Wong et al.

Original: github.com/locuslab/fast_adversarial
Revised: github.com/locuslab/fast_adversarial

Original: [Table]
Revised: [Table]

Original: et
Revised: et

Original: [Table]
Revised: [Table]

Original: github.com/locuslab/robust_overﬁtting
Revised: github.com/locuslab/robust_overﬁtting

Original: [Table] [Table]
Revised: [Table] [Table]

Original: github.com/BorealisAI/mma_training
Revised: github.com/BorealisAI/mma_training

Original: [Table]
Revised: [Table]

Original: et
Revised: et

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) github.com/P2333/Max-Mahalanobis-Training
Revised: ) github.com/P2333/Max-Mahalanobis-Training

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /Friendly-
Revised: /Friendly-

Original: [Table]
Revised: [Table]

Original: -Training
Revised: -Training

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) github.com/LayneH/self-adaptive-training
Revised: ) github.com/LayneH/self-adaptive-training

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2020) github.com/Saehyung-Lee/cifar10_challenge
Revised: 2020) github.com/Saehyung-Lee/cifar10_challenge

Original: A. M ODEL ARCHITECTURES
Revised: A. M ODEL ARCHITECTURES

Original: We select some typical hand-crafted model architectures as the objects of study, involving
Revised: We select some typical hand-crafted model architectures as the objects of study, involving

Original: DenseNet (Huang et al., 2017), GoogleNet (Szegedy et al., 2015), (PreAct) ResNet (He et al., 2016), SENet (Hu et al., 2018), WRN (Zagoruyko & Komodakis, 2016), DPN (Chen et al., 2017b),
Revised: DenseNet (Huang et al., 2017), GoogleNet (Szegedy et al., 2015), (PreAct) ResNet (He et al., 2016), SENet (Hu et al., 2018), WRN (Zagoruyko & Komodakis, 2016), DPN (Chen et al., 2017b),

Original: ResNeXt (Xie et al., 2017), and RegNetX (Radosavovic et al., 2020).
Revised: ResNeXt (Xie et al., 2017), and RegNetX (Radosavovic et al., 2020).

Original: The models are implemented by https://github.com/kuangliu/pytorch-cifar .
Revised: The models are implemented by https://github.com/kuangliu/pytorch-cifar .

Original: [Table]
Revised: [Table]

Original: I NFERENCE - PHASE ADVERSARIAL DEFENSES
Revised: I NFERENCE - PHASE ADVERSARIAL DEFENSES

Original: Except for enhancing the models in the training phase
Revised: Except for enhancing the models in the training phase

Original: [Table]
Revised: [Table]

Original: there are other methods that intend to improve robustness in the inference phase.
Revised: there are other methods that intend to improve robustness in the inference phase.

Original: These attempts include performing local linear transformation like adding Gaussian noise (Tabacof & Valle, 2016), different operations of image processing (Guo et al., 2018; Xie et al., 2018; Raff et al., 2019) or speciﬁed inference principle (Pang et al., 2020b).
Revised: These attempts include performing local linear transformation like adding Gaussian noise (Tabacof & Valle, 2016), different operations of image processing (Guo et al., 2018; Xie et al., 2018; Raff et al., 2019) or speciﬁed inference principle (Pang et al., 2020b).

Original: On the other hand, detection-based methods aim to ﬁlter out adversarial examples and resort to higher-level intervention.
Revised: On the other hand, detection-based methods aim to ﬁlter out adversarial examples and resort to higher-level intervention.

Original: Although detection is a suboptimal strategy compared to classiﬁcation, it can avoid over-conﬁdent wrong decisions.
Revised: Although detection is a suboptimal strategy compared to classiﬁcation, it can avoid over-conﬁdent wrong decisions.

Original: These efforts include training auxiliary classiﬁers to detect adversarial inputs (Metzen et al., 2017), designing detection statistics (Feinman et al., 2017;
Revised: These efforts include training auxiliary classiﬁers to detect adversarial inputs (Metzen et al., 2017), designing detection statistics (Feinman et al., 2017;

Original: Ma et al., 2018; Pang et al., 2018a), or basing on additional probabilistic models (Song et al., 2018b).
Revised: Ma et al., 2018; Pang et al., 2018a), or basing on additional probabilistic models (Song et al., 2018b).

Original: B A DDITIONAL RESULTS
Revised: B A DDITIONAL RESULTS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we provide additional results to further support the conclusions in the main text.
Revised: we provide additional results to further support the conclusions in the main text.

Original: B. E ARLY DECAYS LEARNING RATE
Revised: B. E ARLY DECAYS LEARNING RATE

Original: As shown in Fig.
Revised: As shown in Fig.

Original: [Figure]
Revised: [Figure]

Original: smaller values of weight decay make the training faster but also more tend to overﬁt.
Revised: smaller values of weight decay make the training faster but also more tend to overﬁt.

Original: So in Fig.
Revised: So in Fig.

Original: [Figure]
Revised: [Figure]

Original: we early decay the learning rate at 40 and 45 epochs, rather than 100 andepochs.
Revised: we early decay the learning rate at 40 and 45 epochs, rather than 100 andepochs.

Original: We can see that the models can achieve the same clean accuracy, but the weight decay of 5 × 10 − 4 can still achieve better robustness.
Revised: We can see that the models can achieve the same clean accuracy, but the weight decay of 5 × 10 − 4 can still achieve better robustness.

Original: Besides, in Fig.
Revised: Besides, in Fig.

Original: [Figure]
Revised: [Figure]

Original: we use different values of weight decay for standard training, where the models can also achieve similar clean accuracy.
Revised: we use different values of weight decay for standard training, where the models can also achieve similar clean accuracy.

Original: These results demonstrate that adversarial robustness is a more difﬁcult target than clean performance
Revised: These results demonstrate that adversarial robustness is a more difﬁcult target than clean performance

Original: [Figure]
Revised: [Figure]

Original: and is more sensitive to the training hyperparameters, both for standardly and adversarially trained models
Revised: and is more sensitive to the training hyperparameters, both for standardly and adversarially trained models

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B. T HE EFFECT OF SMOOTH ACTIVATION FUNCTION
Revised: B. T HE EFFECT OF SMOOTH ACTIVATION FUNCTION

Original: In Table 13 we test the effect of Softplus and BN mode on ResNet-18. 
Revised: In Table 13 we test the effect of Softplus and BN mode on ResNet-18. [Table]

Revised: B. R ESULTS OF EARLY STOPPING , WARMUP , AND OPTIMIZERS ON WRN-34-

Revised: In Table 14 and Table 15, we provide the results on WRN-34-10.

Revised: [Table]

Revised: The for and is 0 .

Original: [Table]
Revised: [Table]

Original: B. R ANK IN THE A UTO A TTACK BENCHMARK
Revised: B. R ANK IN THE A UTO A TTACK BENCHMARK

Original: The models evaluated in this paper are all retrained based on the released codes (Zhang et al., 2019b;
Revised: The models evaluated in this paper are all retrained based on the released codes (Zhang et al., 2019b;

Original: Rice et al., 2020).
Revised: Rice et al., 2020).

Original: Now we compare our trained models with the AutoAttack public benchmark, where the results of previous work are based on the released pretrained models.
Revised: Now we compare our trained models with the AutoAttack public benchmark, where the results of previous work are based on the released pretrained models.

Original: In Table 14, we retrieve our results in Table 9 on the TRADES model where we simply change the weight decay from 2 × 10 − 4 to 5 × 10 − 4 .
Revised: In Table 16, we retrieve our results in Table 9 on the TRADES model where we simply change the weight decay from 2 × 10 − 4 to 5 × 10 − 4 .

Original: We can see that this seemingly unimportant difference sends the TRADES model back to the state-of-the-art position in the benchmark.ReLU in
Revised: We can see that this seemingly unimportant difference sends the TRADES model back to the state-of-the-art position in the benchmark.

Original: [Table]
Revised: [Table]

Original: WRN-3483.
Revised: WRN-3483.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019b)
Revised: (2019b)

Original: WRN-3484.
Revised: WRN-3484.