Original: R EMOVING U NDESIRABLE F EATURE C ONTRIBUTIONS U SING O UT - OF -D ISTRIBUTION D ATA
Revised: R EMOVING U NDESIRABLE F EATURE C ONTRIBUTIONS U SING O UT - OF -D ISTRIBUTION D ATA

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Several data augmentation methods deploy unlabeled-in-distribution (UID) data to bridge the gap between the training and inference of neural networks.
Revised: Several data augmentation methods deploy unlabeled-in-distribution (UID) data to bridge the gap between the training and inference of neural networks.

Original: However, these methods have clear limitations in terms of availability of UID data and dependence of algorithms on pseudo-labels.
Revised: However, these methods have clear limitations in terms of availability of UID data and dependence of algorithms on pseudo-labels.

Original: Herein, we propose a data augmentation method to improve generalization in both adversarial and standard learning by using out-of-distribution (OOD) data that are devoid of the abovementioned issues.
Revised: Herein, we propose a data augmentation method to improve generalization in both adversarial and standard learning by using out-of-distribution (OOD) data that are devoid of the abovementioned issues.

Original: We show how to improve generalization theoretically using OOD data in each learning scenario and complement our theoretical analysis with experiments on CIFAR-10, CIFAR-100, and a subset of ImageNet.
Revised: We show how to improve generalization theoretically using OOD data in each learning scenario and complement our theoretical analysis with experiments on CIFAR-10, CIFAR-100, and a subset of ImageNet.

Original: The results indicate that undesirable features are shared even among image data that seem to have little correlation from a human point of view.
Revised: The results indicate that undesirable features are shared even among image data that seem to have little correlation from a human point of view.

Original: We also present the advantages of the proposed method through comparison with other data augmentation methods, which can be used in the absence of UID data.
Revised: We also present the advantages of the proposed method through comparison with other data augmentation methods, which can be used in the absence of UID data.

Original: Furthermore, we demonstrate that the proposed method can further improve the existing state-of-the-art adversarial training.
Revised: Furthermore, we demonstrate that the proposed method can further improve the existing state-of-the-art adversarial training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The power of the enormous amount of data suggested by the empirical risk minimization (ERM) principle (Vapnik & Vapnik, 1998) has allowed deep neural networks (DNNs) to perform outstandingly on many tasks, including computer vision (Krizhevsky et al., 2012) and natural language processing (Hinton et al., 2012).
Revised: The power of the enormous amount of data suggested by the empirical risk minimization (ERM) principle (Vapnik & Vapnik, 1998) has allowed deep neural networks (DNNs) to perform outstandingly on many tasks, including computer vision (Krizhevsky et al., 2012) and natural language processing (Hinton et al., 2012).

Original: However, most of the practical problems encountered by DNNs have high-dimensional input spaces, and nontrivial generalization errors arise owing to the curse of dimensionality (Bellman, 1961).
Revised: However, most of the practical problems encountered by DNNs have high-dimensional input spaces, and nontrivial generalization errors arise owing to the curse of dimensionality (Bellman, 1961).

Original: Moreover, neural networks have been found to be easily deceived by adversarial perturbations with a high degree of conﬁdence (Szegedy et al., 2013).
Revised: Moreover, neural networks have been found to be easily deceived by adversarial perturbations with a high degree of conﬁdence (Szegedy et al., 2013).

Original: Several studies (Goodfellow et al., 2014; Krizhevsky et al., 2012) have been conducted to address these generalization problems resulting from ERM.
Revised: Several studies (Goodfellow et al., 2014; Krizhevsky et al., 2012) have been conducted to address these generalization problems resulting from ERM.

Original: Most of them handled the generalization problems by extending the training distribution (Madry et al., 2017; Lee et al., 2020).
Revised: Most of them handled the generalization problems by extending the training distribution (Madry et al., 2017; Lee et al., 2020).

Original: Nevertheless, it has been demonstrated that more data are needed to achieve better generalization (Schmidt et al., 2018).
Revised: Nevertheless, it has been demonstrated that more data are needed to achieve better generalization (Schmidt et al., 2018).

Original: Recent methods (Carmon et al., 2019; Xie et al., 2019) introduced unlabeled-in-distribution (UID) data to compensate for the lack of training samples.
Revised: Recent methods (Carmon et al., 2019; Xie et al., 2019) introduced unlabeled-in-distribution (UID) data to compensate for the lack of training samples.

Original: However, there are limitations associated with these methods.
Revised: However, there are limitations associated with these methods.

Original: First, obtaining suitable UID data for selected classes is challenging.
Revised: First, obtaining suitable UID data for selected classes is challenging.

Original: Second, when applying supervised learning methods on pseudo-labeled data, the effect of data augmentation depends heavily on the accuracy of the pseudo-label generator.
Revised: Second, when applying supervised learning methods on pseudo-labeled data, the effect of data augmentation depends heavily on the accuracy of the pseudo-label generator.

Original: In our study, in order to break through the limitations outlined above, we propose an approach that promotes robust and standard generalization using out-of-distribution (OOD) data.
Revised: In our study, in order to break through the limitations outlined above, we propose an approach that promotes robust and standard generalization using out-of-distribution (OOD) data.

Original: Especially, motivated by previous studies demonstrating the existence of common adversarial space among different images or even datasets (Naseer et al., 2019; Poursaeed et al., 2018), we show that OOD data can be leveraged for adversarial learning.
Revised: Especially, motivated by previous studies demonstrating the existence of common adversarial space among different images or even datasets (Naseer et al., 2019; Poursaeed et al., 2018), we show that OOD data can be leveraged for adversarial learning.

Original: Likewise, if the OOD data share the same undesirable features as those of the in-distribution data in terms of standard generalization, they can be leveraged for standard learning.
Revised: Likewise, if the OOD data share the same undesirable features as those of the in-distribution data in terms of standard generalization, they can be leveraged for standard learning.

Original: By deﬁnition, in this work, the classes of the OOD data differ from those of the in-distribution data, and our method do not use the label information of the OOD data.
Revised: By deﬁnition, in this work, the classes of the OOD data differ from those of the in-distribution data, and our method do not use the label information of the OOD data.

Original: Therefore the proposed method is free from the previously mentioned problems caused by UID data.
Revised: Therefore the proposed method is free from the previously mentioned problems caused by UID data.

Original: We present a theoretical model which demonstrates how to improve generalization using OOD data in both adversarial and standard learning.
Revised: We present a theoretical model which demonstrates how to improve generalization using OOD data in both adversarial and standard learning.

Original: In our theoretical model, we separate desirable and undesirable features and show how the training on OOD data, which shares undesirable features with in-distribution data, changes the weight values of the classiﬁer.
Revised: In our theoretical model, we separate desirable and undesirable features and show how the training on OOD data, which shares undesirable features with in-distribution data, changes the weight values of the classiﬁer.

Original: Based on the theoretical analysis, we introduce out-of- distribution data augmented training (OAT), which assigns a uniform distribution label to all the
Revised: Based on the theoretical analysis, we introduce out-ofdistribution data augmented training (OAT), which assigns a uniform distribution label to all the

Original: OOD data samples to remove the inﬂuence of undesirable features in adversarial and standard learning.
Revised: OOD data samples to remove the inﬂuence of undesirable features in adversarial and standard learning.

Original: In the proposed method, each batch is composed of training data and OOD data, and OOD data regularize the training so that only features that are strongly correlated with class labels are learned.
Revised: In the proposed method, each batch is composed of training data and OOD data, and OOD data regularize the training so that only features that are strongly correlated with class labels are learned.

Original: We complement our theoretical ﬁndings with experiments on CIFAR-10 (Krizhevsky et al., 2009),
Revised: We complement our theoretical ﬁndings with experiments on CIFAR-10 (Krizhevsky et al., 2009),

Original: CIFAR-100 (Krizhevsky et al., 2009), and a subset of ImageNet (Deng et al., 2009).
Revised: CIFAR-100 (Krizhevsky et al., 2009), and a subset of ImageNet (Deng et al., 2009).

Original: In addition, we present the empirical evidence for the transferability of undesirable features through further studies on various datasets including Simpson Characters (Attia, 2018), Fashion Product Images (Aggarwal, 2018), SVHN (Netzer et al., 2011),
Revised: In addition, we present the empirical evidence for the transferability of undesirable features through further studies on various datasets including Simpson Characters (Attia, 2018), Fashion Product Images (Aggarwal, 2018), SVHN (Netzer et al., 2011),

Original: Places365 (Zhou et al., 2017), and VisDA-

Revised: Places365 (Zhou et al., 2017), and VisDA-

Original: (Peng et al., 2017).
Revised: (Peng et al., 2017).

Original: Contributions (i)
Revised: Contributions (i)

Original: We propose a simple method, out-of-distribution data augmented training (OAT), to leverage OOD data for adversarial and standard learning, and theoretical analyses demonstrate how our proposed method can improve robust and standard generalization.
Revised: We propose a simple method, out-of-distribution data augmented training (OAT), to leverage OOD data for adversarial and standard learning, and theoretical analyses demonstrate how our proposed method can improve robust and standard generalization.

Original: (ii)
Revised: (ii)

Original: The results of experimental procedures on CIFAR-10, CIFAR-100, and a subset of ImageNet suggest that
Revised: The results of experimental procedures on CIFAR-10, CIFAR-100, and a subset of ImageNet suggest that

Original: OAT can help reduce the generalization gap in adversarial and standard learning.
Revised: OAT can help reduce the generalization gap in adversarial and standard learning.

Original: (iii)
Revised: (iii)

Original: By applying
Revised: By applying

Original: OAT using various OOD datasets, it is shown that undesirable features are shared among diverse image datasets.
Revised: OAT using various OOD datasets, it is shown that undesirable features are shared among diverse image datasets.

Original: It is also demonstrated that OAT can effectively extend training distribution by comparison with other data augmentation methods that can be employed in the absence of UID data.
Revised: It is also demonstrated that OAT can effectively extend training distribution by com- parison with other data augmentation methods that can be employed in the absence of UID data.

Original: (iv) The state-of-the-art adversarial training method using UID data is found to further improve by incorporating the proposed method of leveraging OOD data.
Revised: (iv) The state-of-the-art adversarial training method using UID data is found to further improve by incorporating the proposed method of leveraging OOD data.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Undesirable features in adversarial learning Tsipras et al.
Revised: Undesirable features in adversarial learning Tsipras et al.

Original: (2018) demonstrated the existence of a trade-off between standard accuracy and adversarial robustness with the distinction between robust and non-robust features.
Revised: (2018) demonstrated the existence of a trade-off between standard accuracy and adversarial robustness with the distinction between robust and non-robust features.

Original: They showed the possibility that adversarial robustness is incompatible with standard accuracy by constructing a binary classiﬁcation task that the data model consists of inputlabel pairs ( x , y ) ∈ R d +1 × {± 1 } sampled from a distribution as follows:
Revised: They showed the possibility that adversarial robustness is incompatible with standard accuracy by constructing a binary classiﬁcation task that the data model consists of inputlabel pairs ( x , y ) ∈ R d +1 × {± 1 } sampled from a distribution as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, x 1 is a robust feature that is strongly correlated with the label, and the other features x 2 , . . .
Revised: Here, x 1 is a robust feature that is strongly correlated with the label, and the other features x 2 , . . .

Original: , x d +1 are non-robust features that are weakly correlated with the label.
Revised: , x d +1 are non-robust features that are weakly correlated with the label.

Original: (cid:15) is small but sufﬁciently large such that a simple classiﬁer attains a high standard accuracy, and p ≥ 0 .
Revised: (cid:15) is small but sufﬁciently large such that a simple classiﬁer attains a high standard accuracy, and p ≥ 0 .

Original: To characterize adversarial robustness, the deﬁnitions of expected standard loss β s and expected adversarial loss β a for a data distribution D are deﬁned as follows:
Revised: To characterize adversarial robustness, the deﬁnitions of expected standard loss β s and expected adversarial loss β a for a data distribution D are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, L (; θ ) is the loss function of the model, and S represents the set of perturbations that the adversary can apply to deceive the model.
Revised: Here, L (; θ ) is the loss function of the model, and S represents the set of perturbations that the adversary can apply to deceive the model.

Original: For Equation (1), Tsipras et al.
Revised: For Equation (1), Tsipras et al.

Original: (2018) showed that the following classiﬁer can yield a small expected standard loss:
Revised: (2018) showed that the following classiﬁer can yield a small expected standard loss:

Original: [Equation]
Revised: [Equation]

Original: They also proved that the classiﬁer is vulnerable to adversarial perturbations, and that adversarial training results in a classiﬁer that assigns zero weight values to non-robust features.
Revised: They also proved that the classiﬁer is vulnerable to adversarial perturbations, and that adversarial training results in a classiﬁer that assigns zero weight values to non-robust features.

Original: Transferability of Adversarial Perturbations Naseer et al.
Revised: Transferability of Adversarial Perturbations Naseer et al.

Original: (2019) produced domain-agnostic adversarial perturbations, thereby showing common adversarial space among different datasets.
Revised: (2019) produced domain-agnostic adversarial perturbations, thereby showing common adversarial space among different datasets.

Original: They showed that an adversarial function trained on Paintings, Cartoons or Medical data can deceive the classiﬁer on ImageNet data with a high success rate.
Revised: They showed that an adversarial function trained on Paintings, Cartoons or Medical data can deceive the classiﬁer on ImageNet data with a high success rate.

Original: The study ﬁndings show that even datasets from considerably different domains share non-robust features.
Revised: The study ﬁndings show that even datasets from considerably different domains share non-robust features.

Original: Therefore a method for supplementing the data needed for adversarial training is presented herein.
Revised: Therefore a method for supplementing the data needed for adversarial training is presented herein.

Original: Undesirable features in standard learning Wang et al.
Revised: Undesirable features in standard learning Wang et al.

Original: (2020) noted that convolutional neural networks (CNN) can capture high-frequency components in images that are almost imperceptible to a human.
Revised: (2020) noted that convolutional neural networks (CNN) can capture high-frequency components in images that are almost imperceptible to a human.

Original: This ability is thought to be closely related to the generalization behaviors of CNNs, especially the capacity in memorizing random labels.
Revised: This ability is thought to be closely related to the generalization behaviors of CNNs, especially the capacity in memorizing random labels.

Original: Several studies (Geirhos et al., 2018;
Revised: Several studies (Geirhos et al., 2018;

Original: Bahng et al., 2019) reported that CNNs are biased towards local image features, and the generalization performance can be improved by regularizing that bias.
Revised: Bahng et al., 2019) reported that CNNs are biased towards local image features, and the generalization performance can be improved by regularizing that bias.

Original: In this context, a method of regularizing undesirable feature contributions using OOD data is proposed, assuming that undesirable features arise from the bias of CNNs or insufﬁcient training data and are widely distributed in the input space.
Revised: In this context, a method of regularizing undesirable feature contributions using OOD data is proposed, assuming that undesirable features arise from the bias of CNNs or insufﬁcient training data and are widely distributed in the input space.

Original:  M ETHODS 3. T HEORETICAL MOTIVATION
Revised:  M ETHODS 3. T HEORETICAL MOTIVATION

Original: In this section, we analyze theoretically how OOD data can be used to make up for the insufﬁcient training samples in adversarial training based on the dichotomy between robust and non-robust features.
Revised: In this section, we analyze theoretically how OOD data can be used to make up for the insufﬁcient training samples in adversarial training based on the dichotomy between robust and non-robust features.

Original: The theoretical motivation of using OOD data to reduce the contribution of undesirable features in standard learning can be found in Appendix B.
Revised: The theoretical motivation of using OOD data to reduce the contribution of undesirable features in standard learning can be found in Appendix B.

Original: Setup and overview We denote in-distribution data as target data.
Revised: Setup and overview We denote in-distribution data as target data.

Original: Given a target dataset { (˜ x i , y i ) } ni =1 ⊂ X × {± 1 } sampled from a data distribution ˜ D , where X is the input space, we suppose that a feature extractor Φ : X → Z ⊂ R d and a linear classiﬁcation model are trained on { (˜ x i , y i ) } ni =1 and target feature-label pairs { (Φ(˜ x i ) , y i ) } ni =1 , respectively, to yield the small expected standard loss of the classiﬁcation model.
Revised: Given a target dataset { (˜ x i , y i ) } ni =1 ⊂ X × {± 1 } sampled from a data distribution ˜ D , where X is the input space, we suppose that a feature extractor Φ : X → Z ⊂ R d and a linear classiﬁcation model are trained on { (˜ x i , y i ) } ni =1 and target feature-label pairs { (Φ(˜ x i ) , y i ) } ni =1 , respectively, to yield the small expected standard loss of the classiﬁcation model.

Original: We then deﬁne an OOD dataset { ˆ x i } mi =1 ⊂ X sampled from a data distribution ˆ D that has the same distribution of non-robust features as that of ˜
Revised: We then deﬁne an OOD dataset { ˆ x i } mi =1 ⊂ X sampled from a data distribution ˆ D that has the same distribution of non-robust features as that of ˜

Original: D with reference to the preceding studies (Naseer et al., 2019; Moosavi-Dezfooli et al., 2017).
Revised: D with reference to the preceding studies (Naseer et al., 2019; Moosavi-Dezfooli et al., 2017).

Original: After ﬁxing Φ to facilitate the theoretical analysis on this framework, we demonstrate how adversarial training on the OOD dataset affects the weight values of our classiﬁer.
Revised: After ﬁxing Φ to facilitate the theoretical analysis on this framework, we demonstrate how adversarial training on the OOD dataset affects the weight values of our classiﬁer.

Original: Our data model The feature extractor Φ can be considered to consist of several feature extractors φ : X → Z ⊂ R .
Revised: Our data model The feature extractor Φ can be considered to consist of several feature extractors φ : X → Z ⊂ R .

Original: Hence, we can set the distributions of the target feature-label pair (Φ(˜ x ) , y )
Revised: Hence, we can set the distributions of the target feature-label pair (Φ(˜ x ) , y )

Original: =
Revised: =

Original: (˜ z , y ) ∈ R d × {± 1 } and the OOD feature vector Φ(ˆ x ) =
Revised: (˜ z , y ) ∈ R d × {± 1 } and the OOD feature vector Φ(ˆ x ) =

Original: ˆ z ∈ R d as follows:
Revised: ˆ z ∈ R d as follows:

Original: [Equation]
Revised: [Equation]

Original: where u.a.r stands for uniformly at random.
Revised: where u.a.r stands for uniformly at random.

Original: From here on, we will only deal with the OOD data, therefore the accents (tilde and caret) that distinguish between target data and OOD data are omitted.
Revised: From here on, we will only deal with the OOD data, therefore the accents (tilde and caret) that distinguish between target data and OOD data are omitted.

Original: In Equation (4), the feature z 1 = φ 1 ( x ) is the output of a robust feature extractor φ 1 , and the other features z 2 , . . .
Revised: In Equation (4), the feature z 1 = φ 1 ( x ) is the output of a robust feature extractor φ 1 , and the other features z 2 , . . .

Original: , z d +1 are those of non-robust feature extractors φ 2 , . .
Revised: , z d +1 are those of non-robust feature extractors φ 2 , . .

Original: . , φ d +1 .
Revised: . , φ d +1 .

Original: Since the OOD input vectors do not have the same robust features as the target input vectors, z 1 has zero mean and a small variance.
Revised: Since the OOD input vectors do not have the same robust features as the target input vectors, z 1 has zero mean and a small variance.

Original: Furthermore, because the OOD data have the same distribution of non-robust features as the target data, z 2 , . . .
Revised: Furthermore, because the OOD data have the same distribution of non-robust features as the target data, z 2 , . . .

Original: , z d +1 have a non-zero mean and a larger variance than the robust features.
Revised: , z d +1 have a non-zero mean and a larger variance than the robust features.

Original: In addition, q represents the unknown label associated with the non-robust features, and η is a nonnegative constant which represents the degree of correlation between the non-robust features and the unknown label.
Revised: In addition, q represents the unknown label associated with the non-robust features, and η is a nonnegative constant which represents the degree of correlation between the non-robust features and the unknown label.

Original: Please note that the input space of our classiﬁer is the output space of Φ in our data model.
Revised: Please note that the input space of our classiﬁer is the output space of Φ in our data model.

Original: Therefore, η is not limited to a small value even in the context of the (cid:96) p -bounded adversary.
Revised: Therefore, η is not limited to a small value even in the context of the (cid:96) p -bounded adversary.

Original: Rather, the high degree of conﬁdence that DNNs show for the adversarial examples (Goodfellow et al., 2014) suggests that η is large.
Revised: Rather, the high degree of conﬁdence that DNNs show for the adversarial examples (Goodfellow et al., 2014) suggests that η is large.

Original: Our linear classiﬁcation model According to Section 2, we know that our linear classiﬁcation model (logistic regression), deﬁned as follows, yields a low expected standard loss while demonstrating high adversarial vulnerability. 
Revised: Our linear classiﬁcation model According to Section 2, we know that our linear classiﬁcation model (logistic regression), deﬁned as follows, yields a low expected standard loss while demonstrating high adversarial vulnerability. [Equation] where

Original: [Equation]
Revised: [Equation]

Original: To observe the effect of adversarial training on the OOD dataset, we train our classiﬁer by applying the stochastic gradient descent algorithm to the cross-entropy loss function L (; w ) .
Revised: To observe the effect of adversarial training on the OOD dataset, we train our classiﬁer by applying the stochastic gradient descent algorithm to the cross-entropy loss function L (; w ) .

Original: Firstly, we construct the adversarial feature vector ¯ z
Revised: Firstly, we construct the adversarial feature vector ¯ z

Original: =
Revised: =

Original: Φ( x + δ ) : δ ∈ S against our classiﬁer for adversarial training.
Revised: Φ( x + δ ) : δ ∈ S against our classiﬁer for adversarial training.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let t ∈ [0 , 1] be the given target value of the feature vector z in our classiﬁcation model, and λ be a non-negative constant.
Revised: Let t ∈ [0 , 1] be the given target value of the feature vector z in our classiﬁcation model, and λ be a non-negative constant.

Original: Then, when t = 0 .
Revised: Then, when t = 0 .

Original: 5 , the expectation of the adversarial feature vector is
Revised: 5 , the expectation of the adversarial feature vector is

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: (All the proofs of the theorems in this paper can be found in Appendix A.)
Revised: (All the proofs of the theorems in this paper can be found in Appendix A.)

Original: Here, we assume the (cid:96) ∞ -bounded adversary.
Revised: Here, we assume the (cid:96) ∞ -bounded adversary.

Original: In Theorem 1, we can observe that the adversary pushes the non-robust features farther in the direction of the unknown label q , which coincides with our intuition.
Revised: In Theorem 1, we can observe that the adversary pushes the non-robust features farther in the direction of the unknown label q , which coincides with our intuition.

Original: When the given target value t is 0 . 5 , the adversary will make our classiﬁcation model output equal to zero or one to yield a large loss.
Revised: When the given target value t is 0 . 5 , the adversary will make our classiﬁcation model output equal to zero or one to yield a large loss.

Original: Our classiﬁcation model is trained on the adversarial features shown in Theorem 1.
Revised: Our classiﬁcation model is trained on the adversarial features shown in Theorem 1.

Original: Theorem 2.
Revised: Theorem 2.

Original: When t = 0 .
Revised: When t = 0 .

Original: 5 , the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is
Revised: 5 , the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is

Original: [Equation]
Revised: [Equation]

Original: where k ∈ { 2 , . . 
Revised: where k ∈ { 2 , . . .

Original: [Equation]

Original: , d +
Revised: , d +

Original: [Equation]
Revised: [Equation]

Original: Thus, the adversarial training with t = 0 .
Revised: Thus, the adversarial training with t = 0 .

Original: 5 on the OOD dataset leads to the weight values corresponding to the non-robust features converging to zero while preserving w 1 from the gradient update.
Revised: 5 on the OOD dataset leads to the weight values corresponding to the non-robust features converging to zero while preserving w 1 from the gradient update.

Original: This shows that we can reduce the impact of non-robust features using the OOD dataset.
Revised: This shows that we can reduce the impact of non-robust features using the OOD dataset.

Original: We, however, should not only reduce the inﬂuence of the non-robust features, but also improve the classiﬁcation accuracy using the robust feature.
Revised: We, however, should not only reduce the inﬂuence of the non-robust features, but also improve the classiﬁcation accuracy using the robust feature.

Original: This can be achieved through the adversarial training on the target dataset.
Revised: This can be achieved through the adversarial training on the target dataset.

Original: Accordingly, we show the effect of the adversarial training on the OOD dataset when w 1 > 0 in our example.
Revised: Accordingly, we show the effect of the adversarial training on the OOD dataset when w 1 > 0 in our example.

Original: Theorem 3.
Revised: Theorem 3.

Original: When t = 0 .
Revised: When t = 0 .

Original: 5 and w 1 > 0 , the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is
Revised: 5 and w 1 > 0 , the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is

Original: [Equation]
Revised: [Equation]

Original: where k ∈ { 2 , . . .
Revised: where k ∈ { 2 , . . .

Original: , d + 1 } .
Revised: , d + 1 } .

Original: Theorem 3 shows that when w 1 > 0 , the adversarial training with t = 0 .
Revised: Theorem 3 shows that when w 1 > 0 , the adversarial training with t = 0 .

Original: 5 on the OOD dataset reduces the inﬂuence of all the features in Z .
Revised: 5 on the OOD dataset reduces the inﬂuence of all the features in Z .

Original: However, we can see that the expected gradients for the weight values associated with the non-robust features are always greater than the expected gradient for the weight value associated with the robust feature.
Revised: However, we can see that the expected gradients for the weight values associated with the non-robust features are always greater than the expected gradient for the weight value associated with the robust feature.

Original: In addition, the greater the value of η , the faster the weight value associated with it converges to zero.
Revised: In addition, the greater the value of η , the faster the weight value associated with it converges to zero.

Original: This means that the contribution of non-robust features with high inﬂuence decreases rapidly.
Revised: This means that the contribution of non-robust features with high inﬂuence decreases rapidly.

Original: In the case of multiclass classiﬁcation, it is straightforward that t = 0 .
Revised: In the case of multiclass classiﬁcation, it is straightforward that t = 0 .

Original: 5 corresponds to uniform distribution label t unif =
Revised: 5 corresponds to uniform distribution label t unif =

Original: [ 1 c , . .
Revised: [ 1 c , . .

Original: . , 1 c ] , where c is the number of the classes.
Revised: . , 1 c ] , where c is the number of the classes.

Original: Intuitively, the meaning of ( x, t unif ) is that the input x lies on the decision boundary.
Revised: Intuitively, the meaning of ( x, t unif ) is that the input x lies on the decision boundary.

Original: To reduce the training loss for ( x, t unif ), the classiﬁer will learn that the features of x do not contribute to a speciﬁc class, which can be understood as removing the contributions of the features of x .
Revised: To reduce the training loss for ( x, t unif ), the classiﬁer will learn that the features of x do not contribute to a speciﬁc class, which can be understood as removing the contributions of the features of x .

Original: 3. O UT - OF - DISTRIBUTION DATA AUGMENTED TRAINING
Revised: 3. O UT - OF - DISTRIBUTION DATA AUGMENTED TRAINING

Original: Based on our theoretical analysis, we introduce Out-of-distribution data Augmented Training (OAT) .
Revised: Based on our theoretical analysis, we introduce Out-of-distribution data Augmented Training (OAT) .

Original: OAT is the training on the union of the target dataset D t and the OOD dataset D o .
Revised: OAT is the training on the union of the target dataset D t and the OOD dataset D o .

Original: When applying our proposed method, we need to consider the following two points: 1) Temporary labels associated with OOD data samples are required for supervised learning, and 2) the loss functions corresponding to D t and D o should be properly combined.
Revised: When applying our proposed method, we need to consider the following two points: 1) Temporary labels associated with OOD data samples are required for supervised learning, and 2) the loss functions corresponding to D t and D o should be properly combined.

Original: First, we assign a uniform distribution label t unif to all the OOD data samples as conﬁrmed in our theoretical analysis.
Revised: First, we assign a uniform distribution label t unif to all the OOD data samples as conﬁrmed in our theoretical analysis.

Original: This labeling method enables us to leverage OOD data for supervised learning at no extra cost.
Revised: This labeling method enables us to leverage OOD data for supervised learning at no extra cost.

Original: Moreover, it means that our method is completely free from the limitations of the methods using UID data (see Section 1).
Revised: Moreover, it means that our method is completely free from the limitations of the methods using UID data (see Section 1).

Original: Second, although OOD data can be used to improve the standard and robust generalization of neural networks, the training on target data is essential to enhance the classiﬁcation accuracy of neural networks.
Revised: Second, although OOD data can be used to improve the standard and robust generalization of neural networks, the training on target data is essential to enhance the classiﬁcation accuracy of neural networks.

Original: In addition, according to Theorem 3, adversarial training on the pairs of OOD data samples and t unif affects the weight for robust features as well as that for non-robust features.
Revised: In addition, according to Theorem 3, adversarial training on the pairs of OOD data samples and t unif affects the weight for robust features as well as that for non-robust features.

Original: Hence, the balance between losses from D t and D o is important in OAT.
Revised: Hence, the balance between losses from D t and D o is important in OAT.

Original: For this reason, we introduce a hyperparameter α ∈ R + into our proposed method and train neural networks as follows:
Revised: For this reason, we introduce a hyperparameter α ∈ R + into our proposed method and train neural networks as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, OAT-A and OAT-S represent OOD data augmented adversarial and standard learning, respectively.
Revised: Here, OAT-A and OAT-S represent OOD data augmented adversarial and standard learning, respectively.

Original: The pseudo-code for the overall procedure of our method is presented in Algorithm 1.
Revised: The pseudo-code for the overall procedure of our method is presented in Algorithm 1.

Original: Algorithm
Revised: Algorithm

Original: Require: Target dataset D t , OOD dataset D o , uniform distribution label t unif , batch size n , training iterations T , learning rate τ , hyperparameter α , adversarial attack function G 1: for t = 1 to T do 2: ( X t , Y ) =
Revised: Require: Target dataset D t , OOD dataset D o , uniform distribution label t unif , batch size n , training iterations T , learning rate τ , hyperparameter α , adversarial attack function G 1: for t = 1 to T do 2: ( X t , Y ) =

Original: SAMPLE ( dataset = D t , size = n 2 ) 3:
Revised: SAMPLE ( dataset = D t , size = n 2 ) 3:

Original: X o = SAMPLE
Revised: X o = SAMPLE

Original: ( dataset = D o , size = n 2 ) 4:
Revised: ( dataset = D o , size = n 2 ) 4:

Original: if Adversarial Learning then 5: (cid:2) ¯ X t , ¯ X o (cid:3) ← G ([ X t , X o ] , [ Y, t unif ] ; θ ) 6: else if Standard Learning then 7: (cid:2) ¯ X t , ¯ X o (cid:3) ← [ X t , X o ] 8: end if 9: model update : 10: θ ← θ − τ · ∇ θ AVERAGE ( 12 L ( ¯ X t , Y ; θ ) + α 2 L ( ¯ X o , t unif ; θ ))
Revised: if Adversarial Learning then 5: (cid:2) ¯ X t , ¯ X o (cid:3) ← G ([ X t , X o ] , [ Y, t unif ] ; θ ) 6: else if Standard Learning then 7: (cid:2) ¯ X t , ¯ X o (cid:3) ← [ X t , X o ] 8: end if 9: model update : 10: θ ← θ − τ · ∇ θ AVERAGE ( 12 L ( ¯ X t , Y ; θ ) + α 2 L ( ¯ X o , t unif ; θ ))

Original: 11: end for 12: Output: trained model parameter θ
Revised: 11: end for 12: Output: trained model parameter θ

Original:  R ELATED STUDIES
Revised:  R ELATED STUDIES

Original: Adversarial examples Many adversarial attack methods have been proposed, including the projected gradient descent (PGD) and Carlini & Wagner (CW) attacks (Madry et al., 2017; Carlini & Wagner, 2017).
Revised: Adversarial examples Many adversarial attack methods have been proposed, including the projected gradient descent (PGD) and Carlini & Wagner (CW) attacks (Madry et al., 2017; Carlini & Wagner, 2017).

Original: The PGD attack employs an iterative procedure of the fast gradient sign method (FGSM)
Revised: The PGD attack employs an iterative procedure of the fast gradient sign method (FGSM)

Original: (Goodfellow et al., 2014) to ﬁnd worst-case examples in which the training loss is maximized.
Revised: (Goodfellow et al., 2014) to ﬁnd worst-case examples in which the training loss is maximized.

Original: CW attack ﬁnds adversarial examples using CW losses instead of cross-entropy losses.
Revised: CW attack ﬁnds adversarial examples using CW losses instead of cross-entropy losses.

Original: Recently, Croce & Hein (2020) proposed autoattack (AA), which is a powerful ensemble attack with two extensions of the PGD attack and two existing attacks (Croce & Hein, 2019; Andriushchenko et al., 2019).
Revised: Recently, Croce & Hein (2020) proposed autoattack (AA), which is a powerful ensemble attack with two extensions of the PGD attack and two existing attacks (Croce & Hein, 2019; Andriushchenko et al., 2019).

Original: To defend against these adversarial attacks, various adversarial defense methods have been developed (Goodfellow et al., 2014; Kannan et al., 2018).
Revised: To defend against these adversarial attacks, various adversarial defense methods have been developed (Goodfellow et al., 2014; Kannan et al., 2018).

Original: Madry et al.
Revised: Madry et al.

Original: (2017) introduced adversarial training that uses adversarial examples as training data, and Zhang et al.
Revised: (2017) introduced adversarial training that uses adversarial examples as training data, and Zhang et al.

Original: (2019) proposed
Revised: (2019) proposed

Original: TRADES to optimize a surrogate loss which is a sum of the natural error and boundary error.
Revised: TRADES to optimize a surrogate loss which is a sum of the natural error and boundary error.

Original: OOD detection Lee et al.
Revised: OOD detection Lee et al.

Original: (2017) and Hendrycks et al.
Revised: (2017) and Hendrycks et al.

Original: (2018) dealt with the overconﬁdence problem of the conﬁdence score-based OOD detectors.
Revised: (2018) dealt with the overconﬁdence problem of the conﬁdence score-based OOD detectors.

Original: They used uniform distribution labels as in our method to resolve the overconﬁdence issue.
Revised: They used uniform distribution labels as in our method to resolve the overconﬁdence issue.

Original: They did not, however, address the generalization problems of neural networks.
Revised: They did not, however, address the generalization problems of neural networks.

Original: Speciﬁcally, our theoretical results allow us to explain the classiﬁcation performance improvement of classiﬁers that were only considered as secondary effects in the abovementioned studies.
Revised: Speciﬁcally, our theoretical results allow us to explain the classiﬁcation performance improvement of classiﬁers that were only considered as secondary effects in the abovementioned studies.

Original: Further related works can be found in Appendix C.
Revised: Further related works can be found in Appendix C.

Original:  E XPERIMENTAL RESULTS AND DISCUSSION 5. E XPERIMENTAL SETUP
Revised:  E XPERIMENTAL RESULTS AND DISCUSSION 5. E XPERIMENTAL SETUP

Original: OOD datasets We created OOD datasets from the 80 Million Tiny Images dataset (Torralba et al., 2008) (80M-TI), using the work of Carmon et al.
Revised: OOD datasets We created OOD datasets from the 80 Million Tiny Images dataset (Torralba et al., 2008) (80M-TI), using the work of Carmon et al.

Original: (2019) for CIFAR-10 and CIFAR-100, respectively.
Revised: (2019) for CIFAR-10 and CIFAR-100, respectively.

Original: In addition, we resized (using a bilinear interpolation)
Revised: In addition, we resized (using a bilinear interpolation)

Original: ImageNet to dimensions of 64 × 64 and
Revised: ImageNet to dimensions of 64 × 64 and

Original: [Table]
Revised: [Table]

Original: 160 × 160 and divided it into datasets containing 10 and 990 classes, respectively; these are called
Revised: 160 × 160 and divided it into datasets containing 10 and 990 classes, respectively; these are called

Original: ImgNet10 and ImgNet990.
Revised: ImgNet10 and ImgNet990.

Original: Furthermore, we resized Places365 and VisDA-17 for the experiments on
Revised: Furthermore, we resized Places365 and VisDA-17 for the experiments on

Original: ImgNet10 and cropped the Simpson Characters (Simpson)
Revised: ImgNet10 and cropped the Simpson Characters (Simpson)

Original: [Table]
Revised: [Table]

Original: and Fashion Product (Fashion) datasets to dimensions of 32 × 32 for the experiments on CIFAR10 and CIFAR100.
Revised: and Fashion Product (Fashion) datasets to dimensions of 32 × 32 for the experiments on CIFAR10 and CIFAR100.

Original: The details in sourcing the OOD datasets can be found in Appendix D.
Revised: The details in sourcing the OOD datasets can be found in Appendix D.

Original: Implementation details Implementation details including the hyperparameter α , architectures, batch sizes, and training iterations are summarized in Appendix E. All models compared in the same target dataset are trained in the same training batch size and iterations to ensure a fair comparison.
Revised: Implementation details Implementation details including the hyperparameter α , architectures, batch sizes, and training iterations are summarized in Appendix E. All models compared in the same target dataset are trained in the same training batch size and iterations to ensure a fair comparison.

Original: In other words, OAT have a batch size of target n 2 + OOD n 2 , which would be compared with a model that is normally trained with a batch size of n .
Revised: In other words, OAT have a batch size of target n 2 + OOD n 2 , which would be compared with a model that is normally trained with a batch size of n .

Original: To evaluate the adversarial robustness of the models in our experiments, we apply several adversarial attacks including PGD, CW, and AA.
Revised: To evaluate the adversarial robustness of the models in our experiments, we apply several adversarial attacks including PGD, CW, and AA.

Original: Note that we denote PGD and CW attacks with T iterative steps as PGD T and CW T , respectively, and the original test set as Clean.
Revised: Note that we denote PGD and CW attacks with T iterative steps as PGD T and CW T , respectively, and the original test set as Clean.

Original: We compare the following models in our experiments 1 :
Revised: We compare the following models in our experiments 1 :

Original: Standard: The model which is normally trained on the target dataset. 2.
Revised: Standard: The model which is normally trained on the target dataset. 2.

Original: PGD:
Revised: PGD:

Original: The model trained using PGD-based adversarial training on the target dataset. 3.
Revised: The model trained using PGD-based adversarial training on the target dataset. 3.

Original: TRADES:
Revised: TRADES:

Original: The model trained using TRADES on the target dataset. 4.
Revised: The model trained using TRADES on the target dataset. 4.

Original: OAT PGD :
Revised: OAT PGD :

Original: The model which is adversarially trained with OAT based on a PGD approach.
Revised: The model which is adversarially trained with OAT based on a PGD approach.

Original: OAT TRADES :
Revised: OAT TRADES :

Original: The model which is adversarially trained with OAT based on TRADES.
Revised: The model which is adversarially trained with OAT based on TRADES.

Original: OAT D
Revised: OAT D

Original: o : The model which is normally trained with OAT using the OOD dataset D o .
Revised: o : The model which is normally trained with OAT using the OOD dataset D o .

Original: 5. S TUDY ON THE EFFECTIVENESS OF OAT IN ADVERSARIAL LEARNING
Revised: 5. S TUDY ON THE EFFECTIVENESS OF OAT IN ADVERSARIAL LEARNING

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Evaluating adversarial robustness The improvements in the robust generalization performances of the PGD and TRADES models through the application of OAT are evaluated against PGD100,
Revised: Evaluating adversarial robustness The improvements in the robust generalization performances of the PGD and TRADES models through the application of OAT are evaluated against PGD100,

Original: CW100, and AA with (cid:96) ∞ -bound of 8255 .
Revised: CW100, and AA with (cid:96) ∞ -bound of 8255 and 0.031.

Original: The results are summarized in Table 1 and indicate that OAT improves the robust generalization of all adversarial training methods tested regardless of the target dataset.
Revised: The results are summarized in Table 1 and indicate that OAT improves the robust generalization of all adversarial training methods tested regardless of the target dataset.

Original: In particular, from the results against AA it can be seen that the effectiveness of OAT does not rely on obfuscated gradients (Athalye et al., 2018).
Revised: In particular, from the results against AA it can be seen that the effectiveness of OAT does not rely on obfuscated gradients (Athalye et al., 2018).

Original: This is because AA removes the possibility of gradient masking through the application of a combination of strong adaptive attacks (Croce & Hein, 2019; 2020) and a black-box attack (Andriushchenko et al., 2019).
Revised: This is because AA removes the possibility of gradient masking through the application of a combination of strong adaptive attacks (Croce & Hein, 2019; 2020) and a black-box attack (Andriushchenko et al., 2019).

Original: However, while OAT brings a signiﬁcant improvement in robustness against PGD attacks, it is relatively less effective against CW attacks.
Revised: However, while OAT brings a signiﬁcant improvement in robustness against PGD attacks, it is relatively less effective against CW attacks.

Original: According to our theoretical analysis, these results imply that the OOD data have relatively few non-robust features used in the CW attacks.
Revised: According to our theoretical analysis, these results imply that the OOD data have relatively few non-robust features used in the CW attacks.

Original: In other words, powerful targeted attacks, such as CW attacks, are constructed using gradient information more selectively than untargeted attacks, such as PGD attacks.
Revised: In other words, powerful targeted attacks, such as CW attacks, are constructed using gradient information more selectively than untargeted attacks, such as PGD attacks.

Original: To gain insight into this phenomenon, we train models with various hyperparameter α values.
Revised: To gain insight into this phenomenon, we train models with various hyperparameter α values.

Original: The results suggest that the greater the inﬂuence of OOD on the training process is, the higher the robustness against PGD attacks and the lower the robustness against CW attacks are (see Appendix G for more details).
Revised: The results suggest that the greater the inﬂuence of OOD on the training process is, the higher the robustness against PGD attacks and the lower the robustness against CW attacks are (see Appendix G for more details).

Original: In the absence of UID data, various data augmentation methods other than OAT can be employed.
Revised: In the absence of UID data, various data augmentation methods other than OAT can be employed.

Original: CutMix (Yun et al., 2019), a method of amplifying training data by cutting and pasting patches between training images, was recently proposed and exhibited excellent performance in classiﬁcation and transfer learning tasks.
Revised: CutMix (Yun et al., 2019), a method of amplifying training data by cutting and pasting patches between training images, was recently proposed and exhibited excellent performance in classiﬁcation and transfer learning tasks.

Original: By applying CutMix to adversarial training, the advantages of OAT over other data augmentation methods are assessed.
Revised: By applying CutMix to adversarial training, the advantages of OAT over other data augmentation methods are assessed.

Original: Speciﬁcally, OAT brings a higher level of robustness than CutMix, as deduced from Table 1.
Revised: Speciﬁcally, OAT brings a higher level of robustness than CutMix, as deduced from Table 1.

Original: Because adversarial examples are closely related to the high-frequency components of images (Wang et al., 2020) and adversarial training reduces the model’s sensitivity to these components, CutMix is an ineffective data augmentation method.
Revised: Because adversarial examples are closely related to the high-frequency components of images (Wang et al., 2020) and adversarial training reduces the model’s sensitivity to these components, CutMix is an ineffective data augmentation method.

Original: Although CutMix extends the global feature distribution, there is no signiﬁcant difference in terms of local feature distribution.
Revised: Although CutMix extends the global feature distribution, there is no signiﬁcant difference in terms of local feature distribution.

Original: Contrarily, OAT can effectively regularize the classiﬁer by observing various undesirable features held by additional data in the learning process.
Revised: Contrarily, OAT can effectively regularize the classiﬁer by observing various undesirable features held by additional data in the learning process.

Original: OAT with diverse OOD datasets Non-robust features are widely shared among different datasets, as demonstrated by applying OAT using various OOD datasets.
Revised: OAT with diverse OOD datasets Non-robust features are widely shared among different datasets, as demonstrated by applying OAT using various OOD datasets.

Original: Table 2 indicates that OAT improves robust generalization for all OOD datasets, including those that have little correlation with the target dataset from a human perspective.
Revised: Table 2 indicates that OAT improves robust generalization for all OOD datasets, including those that have little correlation with the target dataset from a human perspective.

Original: In addition, given that relatively simple datasets with no background (Fashion and VisDA-17) are less effective than others, we can suppose that non-robust features arise from the high complexity (Snodgrass & Vanderwart, 1980) of images for natural image datasets, such as CIFAR and ImageNet.
Revised: In addition, given that relatively simple datasets with no background (Fashion and VisDA-17) are less effective than others, we can suppose that non-robust features arise from the high complexity (Snodgrass & Vanderwart, 1980) of images for natural image datasets, such as CIFAR and ImageNet.

Revised: Further discussions of the results can be found in Appendix J.

Original: When UID data are available In adversarial training, in-distribution data reduce the sensitivity of neural networks to non-robust features and provide robustly generalizable features.
Revised: When UID data are available In adversarial training, in-distribution data reduce the sensitivity of neural networks to non-robust features and provide robustly generalizable features.

Original: On the other hand, the proposed theory shows that OAT takes advantage of the data ampliﬁcation effect only for non-robust features using OOD data.
Revised: On the other hand, the proposed theory shows that OAT takes advantage of the data ampliﬁcation effect only for non-robust features using OOD data.

Original: Therefore, the effectiveness of OAT is expected to decrease as more in-distribution data are augmented.
Revised: Therefore, the effectiveness of OAT is expected to decrease as more in-distribution data are augmented.

Original: To observe such a trend empirically, the effect of OAT is investigated according to additional in-distribution data size using previously published pseudolabeled data (Carmon et al., 2019).
Revised: To observe such a trend empirically, the effect of OAT is investigated according to additional in-distribution data size using previously published pseudolabeled data (Carmon et al., 2019).

Original: [Figure] Surprisingly, Figure 1(a) shows that OAT still improves robust generalization even when many pseudo-labeled data are used.
Revised: Surprisingly, Figure 1(a) shows that OAT still improves robust generalization even when many pseudo-labeled data are used.

Original: OAT is also combined with RST (Carmon et al., 2019), which has recently recorded a state-of-the-art adversarial robustness using UID data; in fact, Figure 1(b) demonstrates that OAT can further improve the state-of-the-art adversarial training method.
Revised: OAT is also combined with RST (Carmon et al., 2019), which has recently recorded a state-of-the-art adversarial robustness using UID data; in fact, Figure 1(b) demonstrates that OAT can further improve the state-of-the-art adversarial training method.

Original: This is presumably because the additional data include noisy labeled data, in which induce memorization in the learning process and thus impair the generalization performance (Zhang et al., 2016).
Revised: This is presumably because the additional data include noisy labeled data, in which induce memorization in the learning process and thus impair the generalization performance (Zhang et al., 2016).

Original: OAT seems to achieve a higher level of robust generalization by effectively suppressing the effects of noisy data.
Revised: OAT seems to achieve a higher level of robust generalization by effectively suppressing the effects of noisy data.

Original: Additional details on the experiments illustrated in Figure 1 are described in Appendix H.
Revised: Additional details on the experiments illustrated in Figure 1 are described in Appendix H.

Original: 5. S TUDY ON THE EFFECTIVENESS OF OAT IN STANDARD LEARNING
Revised: 5. S TUDY ON THE EFFECTIVENESS OF OAT IN STANDARD LEARNING

Original: Randomization test The effect of OAT is analyzed based on the randomization test (Zhang et al., 2016).
Revised: Randomization test The effect of OAT is analyzed based on the randomization test (Zhang et al., 2016).

Original: The randomization test is an experiment aiming to observe the effective capacity of neural  networks and the effect of regularization by training the model on a copy of the data where the true labels are replaced by random labels.
Revised: The randomization test is an experiment aiming to observe the effective capacity of neural [Figure] networks and the effect of regularization by training the model on a copy of the data where the true labels are replaced by random labels.

Original: In Figure 2(a), it can be seen that the Standard model
Revised: In Figure 2(a), it can be seen that the Standard model

Original: [Figure]
Revised: [Figure]

Original: memorizes all training samples to obtain a low training error, whereas the OAT model continues to have a high training error.
Revised: memorizes all training samples to obtain a low training error, whereas the OAT model continues to have a high training error.

Original: These results show that OAT effectively regularizes neural networks so that it learns only features with strong correlation with the class labels.
Revised: These results show that OAT effectively regularizes neural networks so that it learns only features with strong correlation with the class labels.

Original: Figure 2(b) indicates that the OAT model learns more slowly than the Standard model owing to the inﬂuence of a strong regularizer in the early stages of training; however, it achieves a better generalization performance at the end of training.
Revised: Figure 2(b) indicates that the OAT model learns more slowly than the Standard model owing to the inﬂuence of a strong regularizer in the early stages of training; however, it achieves a better generalization performance at the end of training.

Original: Evaluating classiﬁcation performance The effect of OAT on the classiﬁcation accuracy is shown in Table 3.
Revised: Evaluating classiﬁcation performance The effect of OAT on the classiﬁcation accuracy is shown in Table 3.

Original: When the number of training samples ( N ) is small, the inﬂuence of undesirable features is expected to be large.
Revised: When the number of training samples ( N ) is small, the inﬂuence of undesirable features is expected to be large.

Original: Therefore, the experiments are classiﬁed depending on the number of training samples ( N ).
Revised: Therefore, the experiments are classiﬁed depending on the number of training samples ( N ).

Original: Table 3 indicates that the effect of OAT is large when the amount of training data
Revised: Table 3 indicates that the effect of OAT is large when the amount of training data

Original: [Table]
Revised: [Table]

Original: is small, as predicted.
Revised: is small, as predicted.

Original: Additionally, OAT enhances the generalization performance using 80M-TI,
Revised: Additionally, OAT enhances the generalization performance using 80M-TI,

Original: ImgNet990, and Places365, which have input distributions similar to the target datasets, more than when using other OOD datasets.
Revised: ImgNet990, and Places365, which have input distributions similar to the target datasets, more than when using other OOD datasets.

Original: This proves empirically that in our theoretical analysis, the OOD data, following the same undesirable feature distribution as the target data, can improve generalization through OAT.
Revised: This proves empirically that in our theoretical analysis, the OOD data, following the same undesirable feature distribution as the target data, can improve generalization through OAT.

Original: In addition, the results of Pseudo-label and Fusion models show that even when pseudo-labeled data are available, OOD data can be leveraged to further improve the standard generalization performance.
Revised: In addition, the results of Pseudo-label and Fusion models show that even when pseudo-labeled data are available, OOD data can be leveraged to further improve the standard generalization performance.

Original: Moreover, the proposed method, which does not require complex operations and is very simple to implement, can lead to higher performance by combining with existing data augmentation methods.
Revised: Moreover, the proposed method, which does not require complex operations and is very simple to implement, can lead to higher performance by combining with existing data augmentation methods.

Original: As an example, the effectiveness of Mixup (Zhang et al., 2017) is enhanced by applying OAT; the experimental results are provided in Appendix I.
Revised: As an example, the effectiveness of Mixup (Zhang et al., 2017) is enhanced by applying OAT; the experimental results are provided in Appendix I.

Original: Finally, OAT in a standard learning scheme using the entire target dataset is generally less effective than OAT in an adversarial training scheme (see Appendix F for more details).
Revised: Finally, OAT in a standard learning scheme using the entire target dataset is generally less effective than OAT in an adversarial training scheme (see Appendix F for more details).

Original: Therefore, it can be inferred that the transferability of undesirable features is greater in adversarial settings than in standard settings.
Revised: Therefore, it can be inferred that the transferability of undesirable features is greater in adversarial settings than in standard settings.

Original: In other words, these results experimentally show that the number of training samples required for robust generalization is large compared to that required for standard generalization.
Revised: In other words, these results experimentally show that the number of training samples required for robust generalization is large compared to that required for standard generalization.

Original:  C ONCLUSIONS AND FUTURE DIRECTIONS
Revised:  C ONCLUSIONS AND FUTURE DIRECTIONS

Original: In this study, a method is proposed to compensate for the insufﬁcient training data by using OOD data, which are less restrictive than UID data.
Revised: In this study, a method is proposed to compensate for the insufﬁcient training data by using OOD data, which are less restrictive than UID data.

Original: It is theoretically demonstrated that training with
Revised: It is theoretically demonstrated that training with

Original: OOD data can remove undesirable feature contributions in a simple Gaussian model.
Revised: OOD data can remove undesirable feature contributions in a simple Gaussian model.

Original: Experiments are performed on various OOD datasets, which surprisingly demonstrate that even OOD datasets that apparently have little correlation with the target dataset from the human perspective can help standard and robust generalization through the proposed method.
Revised: Experiments are performed on various OOD datasets, which surprisingly demonstrate that even OOD datasets that apparently have little correlation with the target dataset from the human perspective can help standard and robust generalization through the proposed method.

Original: These results imply that a common undesirable feature space exists among diverse datasets.
Revised: These results imply that a common undesirable feature space exists among diverse datasets.

Original: In addition, the effectiveness of the proposed method is evaluated when extra UID data are available, and the results indicate that OAT can improve the generalization performance even when substantial pseudo-labeled data are used.
Revised: In addition, the effectiveness of the proposed method is evaluated when extra UID data are available, and the results indicate that OAT can improve the generalization performance even when substantial pseudo-labeled data are used.

Original: Nevertheless, some limitations need to be acknowledged.
Revised: Nevertheless, some limitations need to be acknowledged.

Original: First, it is challenging to predict the effectiveness of the proposed method before applying it to a speciﬁc target-OOD dataset pair.
Revised: First, it is challenging to predict the effectiveness of the proposed method before applying it to a speciﬁc target-OOD dataset pair.

Original: Second, our method is less effective against strong targeted adversarial attacks, such as CW attacks, as it is difﬁcult to generate deliberate adversarial attacks on in-distribution in the process of OAT.
Revised: Second, our method is less effective against strong targeted adversarial attacks, such as CW attacks, as it is difﬁcult to generate deliberate adversarial attacks on in-distribution in the process of OAT.

Original: Therefore, as a future research direction, we aim to quantify the degree to which undesirable features are shared between the target and OOD datasets and construct strong adversarial attacks using OOD data
Revised: Therefore, as a future research direction, we aim to quantify the degree to which undesirable features are shared between the target and OOD datasets and construct strong adversarial attacks using OOD data

Revised: Acknowledgements: This work was supported by the National Research Foundation of

Revised: Korea (NRF) grant funded by the Korea government (Ministry of Science and ICT)

Revised: [2018R1A2B3001628], the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in and AIR Lab (AI Research Lab) in Hyundai & Kia Motor Company through HKMC-SNU AI Consortium Fund

Original: A P ROOFS
Revised: A P ROOFS

Original: Theorem 1.
Revised: Theorem 1.

Original: Let t ∈ [0 , 1] be the given target value of the feature vector z in our classiﬁcation model, and λ be a non-negative constant.
Revised: Let t ∈ [0 , 1] be the given target value of the feature vector z in our classiﬁcation model, and λ be a non-negative constant.

Original: Then, when t = 0 .
Revised: Then, when t = 0 .

Original: 5 , the expectation of the adversarial feature vector is
Revised: 5 , the expectation of the adversarial feature vector is

Original: [Equation]
Revised: [Equation]

Original: where k ∈ { 2 , . . .
Revised: where k ∈ { 2 , . . .

Original: , d + 1 } .
Revised: , d + 1 } .

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let δ x be the adversarial perturbation in the input space.
Revised: Let δ x be the adversarial perturbation in the input space.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where δ z = δ (cid:62)
Revised: δ z = δ (cid:62)

Original: [Equation]
Revised: [Equation]

Original: Equation 11 suggests that constructing the adversarial perturbation δ x in the input space can be approximated by ﬁnding the adversarial perturbation δ z in the feature space.
Revised: Equation 11 suggests that constructing the adversarial perturbation δ x in the input space can be approximated by ﬁnding the adversarial perturbation δ z in the feature space.

Original: These have a relationship of δ z
Revised: These have a relationship of δ z

Original: = δ (cid:62) x ∇ x z by linear approximation.
Revised: = δ (cid:62) x ∇ x z by linear approximation.

Original: Hence, without loss of generality, the perturbed feature vector ¯ z can be approximated by z + λ · sign ( ∇ z L ( z , t ; w )) with an λ > 0 .
Revised: Hence, without loss of generality, the perturbed feature vector ¯ z can be approximated by z + λ · sign ( ∇ z L ( z , t ; w )) with an λ > 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original:  Because our classiﬁcation model was trained to minimize the expected standard loss [Equation] E
Revised: [Equation] Because our classiﬁcation model was trained to minimize the expected standard loss, E

Original: [Equation]
Revised: [Equation]

Original: can be approximated by q .
Revised: can be approximated by q .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: When t = 0 .
Revised: When t = 0 .

Original: [Equation]

Original:  the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is
Revised: 5 , the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is

Original: [Equation]
Revised: [Equation]

Original:  k ∈ { 2 , . . .
Revised: where k ∈ { 2 , . . .

Original: , d + 1 } .
Revised: , d + 1 } .

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Based on the adversarial vulnerability of our classiﬁer, σ ( w (cid:62) ¯ z ) can be approximated by 12 (1 + q ) .
Revised: Based on the adversarial vulnerability of our classiﬁer [Equation] σ ( w (cid:62) ¯ z ) can be approximated by 12 (1 + q ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: When t = 0 .
Revised: When t = 0 .

Original: 5 and w 1 > [Equation] the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is
Revised: 5 and w 1 > 0 , the expected gradient of the loss function L (¯ z , t ; w ) with respect to the weight vector w of our classiﬁcation model is

Original: [Equation]
Revised: [Equation]

Original: 2( η + λ

Original: [Equation]
Revised: [Equation]

Original: where k ∈ { 2 , . . .
Revised: where k ∈ { 2 , . . .

Original: , d + 1 } .
Revised: , d + 1 } .

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B T HE T HEORETICAL M OTIVATION OF OAT IN S TANDARD L EARNING
Revised: B T HE T HEORETICAL M OTIVATION OF OAT IN S TANDARD L EARNING

Original: Based on the same setup described in Section 3 of the main manuscript, we construct the data model for OAT in the following manner:
Revised: Based on the same setup described in Section 3 of the main manuscript [Equation] we construct the data model for OAT in the following manner:

Original: [Equation]
Revised: [Equation]

Original: For simplicity
Revised: For simplicity

Original: [Equation]
Revised: [Equation]

Original: we consider only one desirable feature extractor φ 1 and one undesirable feature extractor φ 2 .
Revised: we consider only one desirable feature extractor φ 1 and one undesirable feature extractor φ 2 .

Original: In Equation (18), features z 1 = φ 1 ( x ) and z 2 represent the output of feature extractors φ 1 and φ 2 , respectively.
Revised: In Equation (18), features z 1 = φ 1 ( x ) and z 2 represent the output of feature extractors φ 1 and φ 2 , respectively.

Original: As the OOD input vectors do not have the same desirable features as the target input vectors, the mean of z 1 is zero.
Revised: As the OOD input vectors do not have the same desirable features as the target input vectors, the mean of z 1 is zero.

Original: On the other hand, because the OOD data have the same distribution of undesirable feature as the target data, the mean of z 2 is non-zero.
Revised: On the other hand, because the OOD data have the same distribution of undesirable feature as the target data, the mean of z 2 is non-zero.

Original: In addition, considering that the feature extractors are trained to respond sensitively to the target input vectors, the output variance of the undesirable feature extractor is much larger than that of the desirable feature extractor for OOD data, especially in the high-dimensional case.
Revised: In addition, considering that the feature extractors are trained to respond sensitively to the target input vectors, the output variance of the undesirable feature extractor is much larger than that of the desirable feature extractor for OOD data, especially in the high-dimensional case.

Original: q represents the unknown label associated with the undesirable feature, and κ is a non-negative constant that represents the degree of correlation between the undesirable feature and the unknown label.
Revised: q represents the unknown label associated with the undesirable feature, and κ is a non-negative constant that represents the degree of correlation between the undesirable feature and the unknown label.

Original: We can prove the following theorems for a logistic regression model that is not strongly dependent on the desirable feature.
Revised: We can prove the following theorems for a logistic regression model that is not strongly dependent on the desirable feature.

Original: Theorem 4.
Revised: Theorem 4.

Original: The OOD data barely affect the gradient update of the weight associated with the desirable feature.
Revised: The OOD data barely affect the gradient update of the weight associated with the desirable feature.

Original: Proof.
Revised: Proof.

Original: Because we assumed a linear classiﬁer that is not strongly dependent on the desirable feature owing to the bias of CNNs or insufﬁcient training data, w (cid:62)
Revised: Because we assumed a linear classiﬁer that is not strongly dependent on the desirable feature owing to the bias of CNNs or insufﬁcient training data, w (cid:62)

Original: z
Revised: z

Original: = w 1 z 1 + w 2 z 2 can be approximated by w 2 z 2 .
Revised: = w 1 z 1 + w 2 z 2 can be approximated by w 2 z 2 .

Original: Then, for the target value t , we speciﬁed the feature z as shown in the following equation:
Revised: Then, for the target value t , we speciﬁed the feature z as shown in the following equation:

Original: [Equation]
Revised: [Equation]

Original: Note that in Equation (19), the gradient of the loss function with respect to the weight value w 1 is zero regardless of t .
Revised: Note that in Equation (19), the gradient of the loss function with respect to the weight value w 1 is zero regardless of t .

Original: The optimal t is then a value that makes w 2 converge to zero, thereby removing the inﬂuence of the undesirable feature.
Revised: The optimal t is then a value that makes w 2 converge to zero, thereby removing the inﬂuence of the undesirable feature.

Original: Theorem 5.
Revised: Theorem 5.

Original: When t = 0 .
Revised: When t = 0 .

Original: 5 , the standard learning on the OOD data leads to the weight value corresponding to the undesirable feature converging to 0.
Revised: 5 , the standard learning on the OOD data leads to the weight value corresponding to the undesirable feature converging to 0.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: When w 2 > 0 , z 2 · σ ( w 2 z 2 ) >
Revised: When w 2 > 0 , z 2 · σ ( w 2 z 2 ) >

Original: 12 z 2 with high probability.
Revised: 12 z 2 with high probability.

Original: Hence,
Revised: Hence,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where w 2 > 0 .
Revised: > 0 , where w 2 > 0 .

Original: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where w 2 < 0 .[Equation]
Revised: < 0 , where w 2 < 0 .

Original: C F URTHER R ELATED W ORKS
Revised: C F URTHER R ELATED W ORKS

Original: Using Unlabeled Data to Improve Adversarial Robustness Stanforth et al.
Revised: Using Unlabeled Data to Improve Adversarial Robustness Stanforth et al.

Original: and
Revised: and

Original: Carmon et al.
Revised: Carmon et al.

Original: (2019) analyzed the requirement of larger sample complexity for adversarially robust generalization (Schmidt et al., 2018).
Revised: (2019) analyzed the requirement of larger sample complexity for adversarially robust generalization (Schmidt et al., 2018).

Original: Based on a model described in a prior work (Schmidt et al., 2018), they theoretically proved that unlabeled data can alleviate the need for the large sample complexity of robust generalization.
Revised: Based on a model described in a prior work (Schmidt et al., 2018), they theoretically proved that unlabeled data can alleviate the need for the large sample complexity of robust generalization.

Original: Therefore, they proposed a semi-supervised learning technique by augmenting the training dataset with extra unlabeled data.
Revised: Therefore, they proposed a semi-supervised learning technique by augmenting the training dataset with extra unlabeled data.

Original: They used pseudo-labels obtained from a model trained with the existing training dataset.
Revised: They used pseudo-labels obtained from a model trained with the existing training dataset.

Original: They experimented on the CIFAR-10 dataset by using the 80 Million Tiny Images dataset (Torralba et al., 2008) (80M-TI) as an extra training dataset, resulting in improvements of adversarial robustness of the model.
Revised: They experimented on the CIFAR-10 dataset by using the 80 Million Tiny Images dataset (Torralba et al., 2008) (80M-TI) as an extra training dataset, resulting in improvements of adversarial robustness of the model.

Original: Najaﬁ et al.
Revised: Najaﬁ et al.

Original: (2019) also used unlabeled data for adversarial robustness by extending the distributionally robust learning (Ben-Tal et al., 2013) to semi-supervised learning scenarios.
Revised: (2019) also used unlabeled data for adversarial robustness by extending the distributionally robust learning (Ben-Tal et al., 2013) to semi-supervised learning scenarios.

Original: Instead of using pseudo-labels, they used soft-labels, which are chosen from a set of labels and softened according to the loss values of data.
Revised: Instead of using pseudo-labels, they used soft-labels, which are chosen from a set of labels and softened according to the loss values of data.

Original: Their results include experiments on the MNIST, CIFAR-10, and SVHN datasets.
Revised: Their results include experiments on the MNIST, CIFAR-10, and SVHN datasets.

Original: Comparison with Bad GAN Dai et al.
Revised: Comparison with Bad GAN Dai et al.

Original: (2017) theoretically showed that a perfect generator cannot able to enhance the generalization performance, and good semi-supervised learning actually requires a bad generator.
Revised: (2017) theoretically showed that a perfect generator cannot able to enhance the generalization performance, and good semi-supervised learning actually requires a bad generator.

Original: Through theoretical analysis, they proposed an empirical formulation to generate samples with low-density in the input space.
Revised: Through theoretical analysis, they proposed an empirical formulation to generate samples with low-density in the input space.

Original: Compared to our method, there are two main differences between Bad GAN and OAT.
Revised: Compared to our method, there are two main differences between Bad GAN and OAT.

Original: First, Dai et al.
Revised: First, Dai et al.

Original: (2017) only considered semi-supervised learning settings, whereas OAT can also be applied to adversarial settings.
Revised: (2017) only considered semi-supervised learning settings, whereas OAT can also be applied to adversarial settings.

Original: Second, data augmentation using
Revised: Second, data augmentation using

Original: GAN has clear limitations.
Revised: GAN has clear limitations.

Original: Dai et al.
Revised: Dai et al.

Original: (2017) penalized high-density samples to generate low-density samples in the input space, but this method is inefﬁcient in a high-dimensional input space.
Revised: (2017) penalized high-density samples to generate low-density samples in the input space, but this method is inefﬁcient in a high-dimensional input space.

Original: On the other hand, OAT uses various and efﬁcient OOD directly to regularize the model, and the research direction that develops from the use of synthetic data to the use of real data can also be found for
Revised: On the other hand, OAT uses various and efﬁcient OOD directly to regularize the model, and the research direction that develops from the use of synthetic data to the use of real data can also be found for

Original: OOD detection (Hendrycks et al., 2018).
Revised: OOD detection (Hendrycks et al., 2018).

Revised: [Table]

Revised: [Table]

Original: D S OURCING OOD D ATASETS
Revised: D S OURCING OOD D ATASETS

Original: We created OOD datasets from 80M-TI, using the work of Carmon et al.
Revised: We created OOD datasets from 80M-TI, using the work of Carmon et al.

Original: for CIFAR10 and
Revised: for CIFAR10 and

Original: CIFAR100, respectively.
Revised: CIFAR100, respectively.

Original: In other words, we trained a 11-way (1 more class for OOD) classiﬁer on a training set consisting of the CIFAR10 dataset and 1M randomly sampled images from 80M-TI with keywords that did not appear in CIFAR10.
Revised: In other words [Table] we trained a 11-way (1 more class for OOD) classiﬁer on a training set consisting of the CIFAR10 dataset and 1M randomly sampled images from 80M-TI with keywords that did not appear in CIFAR10.

Original: We then applied the classiﬁer on 80M-TI and sorted the images based on conﬁdence in the OOD class.
Revised: We then applied the classiﬁer on 80M-TI and sorted the images based on conﬁdence in the OOD class.

Original: The 1M and 5M images selected in the order of the highest conﬁdence were used for OAT-A and OAT-S, respectively.
Revised: The 1M and 5M images selected in the order of the highest conﬁdence were used for OAT-A and OAT-S, respectively.

Original: In Table 3, Fusion has a batch size of target n 2 + pseudo-labeled n 4 + OOD n 4 , which would be compared with the Pseudo-label model that is trained with a batch size of target n 2 + pseudo-labeled n 2 .
Revised: In Table [Table] Fusion has a batch size of target n 2 + pseudo-labeled n 4 + OOD n 4 , which would be compared with the Pseudo-label model that is trained with a batch size of target n 2 + pseudo-labeled n 2 .

Original: In addition, we resized (using a bilinear interpolation)
Revised: In addition, we resized (using a bilinear interpolation)

Original: ImageNet to dimensions of 64 × 64 and 160 × 160 and divided it into datasets containing 10 and 990 classes, respectively; these are called ImgNet10 (train set size = 9894 and test set size = 3500) and ImgNet990, respectively.
Revised: ImageNet to dimensions of 64 × 64 and 160 × 160 and divided it into datasets containing 10 and 990 classes, respectively; these are called ImgNet10 (train set size = 9894 and test set size = 3500) and ImgNet990, respectively.

Original: The classes were divided based on the Imagenette dataset (Howard), and the experimental results for the differently divided dataset (Imagewoof) can be seen in Appendix
Revised: The classes were divided based on the Imagenette dataset (Howard), and the experimental results for the differently divided dataset (Imagewoof) can be seen in Appendix

Original: F. We increased the number of images of ImgNet990 by 10 times through random cropping and created the OOD datasets in the same process as 80M-TI.
Revised: F. We increased the number of images of ImgNet990 by 10 times through random cropping and created the OOD datasets in the same process as 80M-TI.

Original: Furthermore, we[Table] [Table] resized (bilinear interpolation)
Revised: Furthermore, we resized (bilinear interpolation)

Original: Places365 (Zhou et al., 2017) and VisDA17 (Peng et al., 2017) for the experiments on ImgNet10 and cropped Simpson Characters (Simpson) (Attia, 2018) and Fashion product (Fashion) (Aggarwal, 2018) to dimensions of 32 × 32 for the experiments on CIFAR.
Revised: Places365 (Zhou et al., 2017) and VisDA17 (Peng et al., 2017) for the experiments on ImgNet10 and cropped Simpson Characters (Simpson) (Attia, 2018) and Fashion product (Fashion) (Aggarwal, 2018) to dimensions of 32 × 32 for the experiments on CIFAR.

Original: E I MPLEMENTATION D ETAILS
Revised: E I MPLEMENTATION D ETAILS

Original: For all the experiments except TRADES and OAT TRADES , the initial learning rate is set to 0.1.
Revised: For all the experiments except TRADES and OAT TRADES , the initial learning rate is set to 0.1.

Original: The learning rate is multiplied by 0.1 at 50% and 75% of the total training steps, and the weight decay factor is set to 2e − 4 .
Revised: The learning rate is multiplied by 0.1 at 50% and 75% of the total training steps, and the weight decay factor is set to 2e − 4 .

Original: We use the same adversarial perturbation budget (cid:15) = 8 , as in Madry et al.
Revised: We use the same adversarial perturbation budget (cid:15) = 8 , as in Madry et al.

Original: We recorded the maximum adversarial robustness of the models on the test set after the ﬁrst learning rate decay in adversarial training and the empirical upper bound of the test accuracy of the models during the standard learning.
Revised: We recorded the maximum adversarial robustness of the models on the test set after the ﬁrst learning rate decay in adversarial training and the empirical upper bound of the test accuracy of the models during the standard learning.

Original: The other details are summarized in Tables 4 and 5.
Revised: The other details are summarized in Tables 4 and 5.

Original: For TRADES and OAT TRADES , we deploy a batch size of 64 and train the models using the same conﬁgurations as Zhang et al.
Revised: For TRADES and OAT TRADES , we deploy a batch size of 64 and train the models using the same conﬁgurations as Zhang et al.

Original: F F URTHER D ISCUSSION ABOUT THE E FFECTIVENESS OF OAT
Revised: F F URTHER D ISCUSSION ABOUT THE E FFECTIVENESS OF OAT

Original: We created a new ImgNet10 dataset with reference to Imagewoof (Howard) to learn more about the effects of OAT.
Revised: We created a new ImgNet10 dataset with reference to Imagewoof (Howard) to learn more about the effects of OAT.

Original: Imagewoof is a subset of ImageNet, and all classes are dog breeds.
Revised: Imagewoof is a subset of ImageNet, and all classes are dog breeds.

Original: We tested  whether OAT is effective for the newly constructed ImgNet10 dataset, and the experimental results of OAT-A and OAT-S are shown in Tables 6 and7, respectively.
Revised: We tested [Table] [Table] whether OAT is effective for the newly constructed ImgNet10 dataset, and the experimental results of OAT-A and OAT-S are shown in Tables 6 and

Revised: [Table]

Original:  We can see from Tables 6 andthat for the new ImgNet10, OAT is largely useful for generalization.
Revised: respectively. We can see from Tables 6 andthat for the new ImgNet10, OAT is largely useful for generalization.

Original: However, from the results of
Revised: However, from the results of

Original: ImgNet10(64 × 64) in Table 7, we can speculate that OAT will help in training as a regularizer only [Table] [Table] [Table]
Revised: ImgNet10(64 × 64) in Table 7, we can speculate that OAT will help in training as a regularizer only when an excessive number of features are involved in the classiﬁcation.

Original: when an excessive number of features are involved in the classiﬁcation.

Original: This is because OAT shows no performance improvement for the new ImgNet10(64 × 64) dataset.
Revised: This is because OAT shows no performance improvement for the new ImgNet10(64 × 64) dataset.

Original: The new ImgNet10(64 × 64) is believed to have lost much of the categorical features, since all the classes are dog breeds, in addition to being overly downscaled.
Revised: The new ImgNet10(64 × 64) is believed to have lost much of the categorical features, since all the classes are dog breeds, in addition to being overly downscaled.

Original: In other words [Table] OAT is beneﬁcial for the overﬁtting problem, but not for the underﬁtting problem.
Revised: In other words, OAT is beneﬁcial for the overﬁtting problem, but not for the underﬁtting problem.

Original: We can conﬁrm this by artiﬁcially reducing the number of training samples to increase the features that can distinguish between each categorical sample distribution [Table] and then observe the improvement in generalization performance by applying OAT.
Revised: We can conﬁrm this by artiﬁcially reducing the number of training samples to increase the features that can distinguish between each categorical sample distribution, and then observe the improvement in generalization performance by applying OAT.

Original: G A BLATION STUDY
Revised: G A BLATION STUDY

Original: Our experiments indicate that it is possible to enhance adversarial robustness by removing the contributions of non-robust features existing in additional data through OAT and transfer the consistency to in-distribution data.
Revised: Our experiments indicate that it is possible to enhance adversarial robustness by removing the contributions of non-robust features existing in additional data through OAT and transfer the consistency to in-distribution data.

Original: However, the increase in the robustness against targeted attacks appears smaller than that against untargeted attacks in our method.
Revised: However, the increase in the robustness against targeted attacks appears smaller than that against untargeted attacks in our method.

Original: To gain insight into this phenomenon, we train the OAT models for various α values and investigate the difference between untargeted and targeted attacks.
Revised: To gain insight into this phenomenon, we train the OAT models for various α values and investigate the difference between untargeted and targeted attacks.

Original: In Table 8, it can be seen that the greater the inﬂuence of OOD on the training process is, the higher the robustness against PGD20 and the lower the robustness against CW20 are.
Revised: In Table 8, it can be seen that the greater the inﬂuence of OOD on the training process is, the higher the robustness against PGD20 and the lower the robustness against CW20 are.

Original: According to our theoretical analysis, it can be understood that there are many features used for untargeted attacks in OOD data, whereas relatively few features exist for targeted attacks.
Revised: According to our theoretical analysis, it can be understood that there are many features used for untargeted attacks in OOD data, whereas relatively few features exist for targeted attacks.

Original: Because targeted attacks are  stronger than untargeted attacks (Carlini & Wagner, 2017), the results of Table 8 provide empirical evidence for the trade-off between the transferability of adversarial perturbations and the strength of adversarial attacks.
Revised: Because targeted attacks are [Table] [Table] stronger than untargeted attacks (Carlini & Wagner, 2017), the results of Table 8 provide empirical evidence for the trade-off between the transferability of adversarial perturbations and the strength of adversarial attacks.

Original: A similar trend was reported in the study of Chan et al.
Revised: A similar trend was reported in the study of Chan et al.

Original: The authors showed that input gradient adversarial matching (IGAM) can transfer robustness across different tasks.
Revised: The authors showed that input gradient adversarial matching (IGAM) can transfer robustness across different tasks.

Original: Their results show that IGAM-trained models have similar or higher robustness than baseline models against weak attacks, such as FGSM or low-step PGD attacks, but are vulnerable to strong attacks.
Revised: Their results show that IGAM-trained models have similar or higher robustness than baseline models against weak attacks, such as FGSM or low-step PGD attacks, but are vulnerable to strong attacks.

Original: The ﬁnetuning-based method can also be regarded as an attempt to increase the robustness by using other datasets, but the abovementioned phenomenon is not observed in the method proposed by
Revised: The ﬁnetuning-based method can also be regarded as an attempt to increase the robustness by using other datasets, but the abovementioned phenomenon is not observed in the method proposed by

Original: Hendrycks et al.
Revised: Hendrycks et al.

Original: This is because the method does not transfer the robustness learned from other datasets to the target dataset but rather uses a function learned from a dataset that has a large sample complexity and a data distribution similar to that of the target data with little modiﬁcation.
Revised: This is because the method does not transfer the robustness learned from other datasets to the target dataset but rather uses a function learned from a dataset that has a large sample complexity and a data distribution similar to that of the target data with little modiﬁcation.

Original: This can be conﬁrmed from the small number of training iterations and small learning rate involved by the ﬁne-tuning process.
Revised: This can be conﬁrmed from the small number of training iterations and small learning rate involved by the ﬁne-tuning process.

Original: Moreover, applying the same method to a dataset with a small sample size or far from the target data distribution has no [Table]
Revised: Moreover, applying the same method to a dataset with a small sample size or far from the target data distribution has no effect (Chan et al., 2020).

Original: H W HEN UID DATA ARE AVAILABLE
Revised: H W HEN UID DATA ARE AVAILABLE

Original: We train the OAT models in conjuction with UID data as follows:
Revised: We train the OAT models in conjuction with UID data as follows:

Original: [Equation]
Revised: [Equation]

Original: In the experiment of Figure 1(a), the Pseudo-label models are trained via a PGD-based approach, and every batch consists of 64 CIFAR-10 data and 64 pseudo-labeled data.
Revised: In the experiment of Figure 1(a), the Pseudo-label models are trained via a PGD-based approach, and every batch consists of 64 CIFAR-10 data and 64 pseudo-labeled data.

Original: The OAT+Pseudo-label models are also trained via a PGD-based approach, and every batch consists of 64 CIFAR-10 data, 32 pseudo-labeled data, and 32 OOD data (80M-TI). 
Revised: The OAT+Pseudo-label models are also trained via a PGD-based approach, and every batch consists of 64 CIFAR-10 data, 32 pseudo-labeled data, and 32 OOD data (80M-TI). The hyperparameters ( α in , α o , α UID ) are set to ( 1 3 , 1 3 , 1 3 ).

Original: All other conditions are the same as described in Appendix E.
Revised: All other conditions are the same as described in Appendix E.

Original: In the experiment of Figure 1(b), we train the models with the same conﬁgurations as Carmon et al.
Revised: In the experiment of Figure 1(b), we train the models with the same conﬁgurations as Carmon et al.

Original: , but every batch for the OAT+RST model comprises 128 pseudo-labeled data, 64 CIFAR-10 data, and 64 OOD data (the RST model has a batch size of 256).
Revised: , but every batch for the OAT+RST model comprises 128 pseudo-labeled data, 64 CIFAR-10 data, and 64 OOD data (the RST model has a batch size of 256).

Revised: The hyperparameters ( α in , α o , α UID ) are set to (0.25, 0.25, 0.5).

Original: I E XPERIMENTAL RESULTS FOR M IXUP
Revised: I E XPERIMENTAL RESULTS FOR M IXUP

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017) proposed a data augmentation method named Mixup.
Revised: (2017) proposed a data augmentation method named Mixup.

Original: Mixup generates training examples as ˜ x
Revised: Mixup generates training examples as ˜ x

Original: = αx i + (1 − α ) x j and ˜ y
Revised: = αx i + (1 − α ) x j and ˜ y

Original: = αy i + (1 − α )
Revised: = αy i + (1 − α )

Original: y j , where ( x i , y i ) and ( x j , y j ) are two examples drawn at random from the training data, and α ∈ [0 , 1] .
Revised: y j , where ( x i , y i ) and ( x j , y j ) are two examples drawn at random from the training data, and α ∈ [0 , 1] .

Original: Here, we show that even outside of the convex combination of training data can be effectively regularized using OOD data.
Revised: Here, we show that even outside of the convex combination of training data can be effectively regularized using OOD data.

Original: Tableshows that the models trained with the combination of OAT and Mixup always output the highest accuracy.
Revised: Tableshows that the models trained with the combination of OAT and Mixup always output the highest accuracy.

Original: The OAT+Mixup models are trained with the following algorithm: 
Revised: The OAT+Mixup models are trained with the following algorithm: Algorithm 2 OAT+Mixup

Original: Require:
Revised: Require:

Original: Ratio γ , Target dataset D t , OOD dataset D o , uniform distribution label t unif , batch size n , training iterations T , learning rate τ 1: for t = 1 to T do 2: ( X t , Y t ) =
Revised: Ratio γ , Target dataset D t , OOD dataset D o , uniform distribution label t unif , batch size n , training iterations T , learning rate τ 1: for t = 1 to T do 2: ( X t , Y t ) =

Original: SAMPLE ( dataset = D t , size = n ) 3:
Revised: SAMPLE ( dataset = D t , size = n ) 3:

Original: X o = SAMPLE
Revised: X o = SAMPLE

Original: ( dataset = D
Revised: ( dataset = D

Original: o , size = (cid:98) n ∗ γ (cid:99) ) 4: ( ¯ X t , ¯ Y t ) ← PERMUTE ( X t , Y t ) 5: ( ¯ X t [0 : (cid:98) n ∗ γ (cid:99) ] , ¯ Y t [0 : (cid:98) n ∗ γ (cid:99) ]) ← ( X o , t unif ) 6: ( X, Y ) ← MIXUP ( X t , Y t , ¯ X t , ¯ Y t ) 7: model update : 
Revised: o , size = (cid:98) n ∗ γ (cid:99) ) 4: ( ¯ X t , ¯ Y t ) ← PERMUTE ( X t , Y t ) 5: ( ¯ X t [0 : (cid:98) n ∗ γ (cid:99) ] , ¯ Y t [0 : (cid:98) n ∗ γ (cid:99) ]) ← ( X o , t unif ) 6: ( X, Y ) ← MIXUP ( X t , Y t , ¯ X t , ¯ Y t ) 7: model update : 8: θ ← θ − τ · ∇ θ AVERAGE ( L ( X, Y ; θ )) 9: end for 10: Output: trained model parameter θ J OAT WITH DIVERSE OOD DATASETS [Figure]

Revised: According to our theoretical model, if all OOD images share similar amounts of non-robust features with the in-distribution data, then the OOD datasets, which are more dissimilar to the in-distribution data, are expected to be more effective.

Original: 8: θ ← θ − τ · ∇ θ AVERAGE ( L ( X, Y ; θ )) 9: end for 10: Output: trained model parameter θ
Revised: However, our experiments on similar OOD datasets, such as 80M-TI, Places365, and ImgNet990, and dissimilar OOD datasets, such as SVHN, Simpson, Fashion, and VisDA17, show that OOD datasets that are more similar to the in-distribution dataset are more effective. Thus, it can be considered that similar OOD datasets share more non-robust features with the in-distribution dataset. To conﬁrm this, we investigate the sensitivity of the nonrobust classiﬁer (Standard), which is trained on CIFAR-10, to each OOD dataset.

Revised: According to a recent study (Ortiz-Jimenez et al., 2020), neural networks are trained to be less sensitive to unused features.

Revised: Additionally, several previous studies (Tsipras et al., 2018; Ilyas et al., 2019) demonstrated that standard classiﬁers are biased towards non-robust features.

Revised: Hence, if a non-robust classiﬁer trained using CIFAR-10 shows high sensitivity to a dataset, it can be asserted that such a dataset shares many non-robust features with CIFAR-10.

Revised: We measure the sensitivity of the classiﬁer using the Euclidean norm of the logit vector, and we show the distribution for each dataset in Figure 3.

Revised: It should be noted that 80M-TI contains a signiﬁcantly larger number of diverse data samples than other OOD datasets.

Revised: To exhibit the beneﬁt of 80M-TI, we consider the top-500K samples (in terms of sensitivity) of this dataset.

Revised: In Figure 3, we observe that 80M-TI includes more data samples, which share non-robust features with CIFAR-10, than other OOD datasets.