Original: mixup : B EYOND E MPIRICAL R ISK M INIMIZATION
Revised: mixup : B EYOND E MPIRICAL R ISK M INIMIZATION

Original: Anonymous authors Paper under double-blind review
Revised: FAIR

Original: A BSTRACT
Revised: A BSTRACT

Original: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples.
Revised: Large deep neural networks are powerful but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples

Original: In this work, we propose mixup, a simple learning principle to alleviate these issues.
Revised: In this work , we propose mixup a simple learning principle to alleviate these issues

Original: In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels.
Revised: In essence mixup trains a neural network on convex combinations of pairs of examples and their labels.

Original: By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples.
Revised: By doing so mixup regularizes the neural network to favor simple linear behavior in-between training examples.

Original: Our experiments on the ImageNet-2012, CIFAR-10,
Revised: Our experiments on the ImageNet-2012, CIFAR-10,

Original: CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures.
Revised: CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures.

Original: We also ﬁnd that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.
Revised: We also ﬁnd that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples and stabilizes the training of generative adversarial networks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large deep neural networks have enabled breakthroughs in ﬁelds such as computer vision (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and reinforcement learning (Silver et al., 2016).
Revised: Large deep neural networks have enabled breakthroughs in ﬁelds such as computer vision (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and reinforcement learning (Silver et al., 2016).

Original: In most successful applications, these neural networks share two commonalities.
Revised: In most successful applications, these neural networks share two commonalities.

Original: First, they are trained as to minimize their average error over the training data, a learning rule also known as the
Revised: First, they are trained as to minimize their average error over the training data, a learning rule also known as the

Original: Empirical Risk Minimization (ERM) principle (Vapnik, 1998).
Revised: Empirical Risk Minimization (ERM) principle (Vapnik, 1998).

Original: Second, the size of these state-of-theart neural networks scales linearly with the number of training examples.
Revised: Second, the size of these state-of-theart neural networks scales linearly with the number of training examples.

Original: For instance, the network of
Revised: For instance, the network of

Original: Springenberg et al.
Revised: Springenberg et al.

Original: (2015) used 10 6 parameters to model the 5 · 10 4 images in the CIFAR-10 dataset, the network of (Simonyan & Zisserman, 2015) used 10 8 parameters to model the 10 6 images in the
Revised: (2015) used 10 6 parameters to model the 5 · 10 4 images in the CIFAR-10 dataset, the network of (Simonyan & Zisserman, 2015) used 10 8 parameters to model the 10 6 images in the

Original: ImageNet-2012 dataset, and the network of Chelba et al.
Revised: ImageNet-2012 dataset, and the network of Chelba et al.

Original: (2013) used 2 · 10 10 parameters to model the 10 9 words in the One Billion Word dataset.
Revised: (2013) used 2 · 10 10 parameters to model the 10 9 words in the One Billion Word dataset.

Original: Strikingly, a classical result in learning theory (Vapnik & Chervonenkis, 1971) tells us that the convergence of ERM is guaranteed as long as the size of the learning machine (e.g., the neural network) does not increase with the number of training data.
Revised: Strikingly, a classical result in learning theory (Vapnik & Chervonenkis, 1971) tells us that the convergence of ERM is guaranteed as long as the size of the learning machine (e.g., the neural network) does not increase with the number of training data.

Original: Here, the size of a learning machine is measured in terms of its number of parameters or, relatedly, its VC-complexity (Harvey et al., 2017).
Revised: Here, the size of a learning machine is measured in terms of its number of parameters or, relatedly, its VC-complexity (Harvey et al., 2017).

Original: This contradiction challenges the suitability of ERM to train our current neural network models, as highlighted in recent research.
Revised: This contradiction challenges the suitability of ERM to train our current neural network models, as highlighted in recent research.

Original: On the one hand, ERM allows large neural networks to memorize (instead of generalize from) the training data even in the presence of strong regularization, or in classiﬁcation problems where the labels are assigned at random (Zhang et al., 2017).
Revised: On the one hand, ERM allows large neural networks to memorize (instead of generalize from) the training data even in the presence of strong regularization, or in classiﬁcation problems where the labels are assigned at random (Zhang et al., 2017).

Original: On the other hand, neural networks trained with ERM change their predictions drastically when evaluated on examples just outside the training distribution (Szegedy et al., 2014), also known as adversarial examples .
Revised: On the other hand, neural networks trained with ERM change their predictions drastically when evaluated on examples just outside the training distribution (Szegedy et al., 2014), also known as adversarial examples .

Original: This evidence suggests that ERM is unable to explain or provide generalization on testing distributions that differ only slightly from the training data.
Revised: This evidence suggests that ERM is unable to explain or provide generalization on testing distributions that differ only slightly from the training data.

Original: However, what is the alternative to ERM?
Revised: However, what is the alternative to ERM?

Original: The method of choice to train on similar but different examples to the training data is known as data augmentation (Simard et al., 1998), formalized by the Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2000).
Revised: The method of choice to train on similar but different examples to the training data is known as data augmentation (Simard et al., 1998), formalized by the Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2000).

Original: In VRM, human knowledge is required to describe a vicinity or neighborhood around each example in the training data.
Revised: In VRM, human knowledge is required to describe a vicinity or neighborhood around each example in the training data.

Original: Then, additional virtual examples can be drawn from the vicinity distribution of the training examples to enlarge the support of the training distribution.
Revised: Then, additional virtual examples can be drawn from the vicinity distribution of the training examples to enlarge the support of the training distribution.

Original: For instance, when performing image classiﬁcation, it is common to deﬁne the vicinity of one image as the set of its horizontal reﬂections, slight rotations, and mild scalings.
Revised: For instance, when performing image classiﬁcation, it is common to deﬁne the vicinity of one image as the set of its horizontal reﬂections, slight rotations, and mild scalings.

Original: While data augmentation consistently leads to improved generalization (Simard et al., 1998), the procedure is dataset-dependent, and thus requires the use of expert knowledge.
Revised: While data augmentation consistently leads to improved generalization (Simard et al., 1998), the procedure is dataset-dependent, and thus requires the use of expert knowledge.

Original: Furthermore, data augmentation assumes that the examples in the vicinity share the same class, and does not model the vicinity relation across examples of different classes.
Revised: Furthermore, data augmentation assumes that the examples in the vicinity share the same class, and does not model the vicinity relation across examples of different classes.

Original: Contribution Motivated by these issues, we introduce a simple and data-agnostic data augmentation routine, termed mixup (Section 2).
Revised: Contribution Motivated by these issues, we introduce a simple and data-agnostic data augmentation routine, termed mixup (Section 2).

Original: In a nutshell, mixup constructs virtual training examples
Revised: In a nutshell, mixup constructs virtual training examples

Original: [Equation]
Revised: [Equation]

Original: where  ( x i , y i ) and ( x j , y j ) are two examples drawn at random from our training data, and λ ∈ [0 , 1] .
Revised: where x i , x j are raw input vectors [Equation] where y i , y j are one-hot label encodings ( x i , y i ) and ( x j , y j ) are two examples drawn at random from our training data, and λ ∈ [0 , 1] .

Original: Therefore, mixup extends the training distribution by incorporating the prior knowledge that linear interpolations of feature vectors should lead to linear interpolations of the associated targets.
Revised: Therefore, mixup extends the training distribution by incorporating the prior knowledge that linear interpolations of feature vectors should lead to linear interpolations of the associated targets.

Original: mixup can be implemented in a few lines of code, and introduces minimal computation overhead.
Revised: mixup can be implemented in a few lines of code, and introduces minimal computation overhead.

Original: Despite its simplicity, mixup allows a new state-of-the-art performance in the CIFAR-10, CIFAR100, and ImageNet-2012 image classiﬁcation datasets (Sections 3.1 and 3.2).
Revised: Despite its simplicity, mixup allows a new state-of-the-art performance in the CIFAR-10, CIFAR100, and ImageNet-2012 image classiﬁcation datasets (Sections 3.1 and 3.2).

Original: Furthermore, mixup increases the robustness of neural networks when learning from corrupt labels (Section 3.4), or facing adversarial examples (Section 3.5).
Revised: Furthermore, mixup increases the robustness of neural networks when learning from corrupt labels (Section 3.4), or facing adversarial examples (Section 3.5).

Original: Finally, mixup improves generalization on speech (Sections 3.3) and tabular (Section 3.6) data, and can be used to stabilize the training of GANs (Section 3.7).
Revised: Finally, mixup improves generalization on speech (Sections 3.3) and tabular (Section 3.6) data, and can be used to stabilize the training of GANs (Section 3.7).

Original: The source-code necessary to replicate our  experiments is available at:
Revised: The source-code necessary to replicate our CIFAR-10 experiments is available at:

Original: [Equation]
Revised: [Equation]

Original: To understand the effects of various design choices in mixup , we conduct a thorough set of ablation study experiments (Section 3.8).
Revised: To understand the effects of various design choices in mixup , we conduct a thorough set of ablation study experiments (Section 3.8).

Original: The results suggest that mixup performs signiﬁcantly better than related methods in previous work, and each of the design choices contributes to the ﬁnal performance.
Revised: The results suggest that mixup performs signiﬁcantly better than related methods in previous work, and each of the design choices contributes to the ﬁnal performance.

Original: We conclude by exploring the connections to prior work (Section 4), as well as offering some points for discussion (Section 5).
Revised: We conclude by exploring the connections to prior work (Section 4), as well as offering some points for discussion (Section 5).

Original:  F ROM E MPIRICAL R ISK M INIMIZATION TO mixup
Revised:  F ROM E MPIRICAL R ISK M INIMIZATION TO mixup

Original: In supervised learning, we are interested in ﬁnding a function f ∈ F that describes the relationship between a random feature vector X and a random target vector Y , which follow the joint distribution
Revised: In supervised learning, we are interested in ﬁnding a function f ∈ F that describes the relationship between a random feature vector X and a random target vector Y , which follow the joint distribution

Original: P ( X, Y ) .
Revised: P ( X, Y ) .

Original: To this end, we ﬁrst deﬁne a loss function (cid:96) that penalizes the differences between predictions f ( x ) and actual targets y , for examples ( x, y ) ∼ P .
Revised: To this end, we ﬁrst deﬁne a loss function (cid:96) that penalizes the differences between predictions f ( x ) and actual targets y , for examples ( x, y ) ∼ P .

Original: Then, we minimize the average of the loss function (cid:96) over the data distribution P , also known as the expected risk :
Revised: Then, we minimize the average of the loss function (cid:96) over the data distribution P , also known as the expected risk :

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, the distribution P is unknown in most practical situations.
Revised: Unfortunately, the distribution P is unknown in most practical situations.

Original: Instead, we usually have access to a set of training data D = { ( x i , y i ) } ni =1 , where ( x i , y i ) ∼ P for all i = 1 , . . . , n .
Revised: Instead, we usually have access to a set of training data D = { ( x i , y i ) } n i =1 , where ( x i , y i ) ∼ P for all i = 1 , . . . , n .

Original: Using the training data D , we may approximate P by the empirical distribution
Revised: Using the training data D , we may approximate P by the empirical distribution

Original: [Equation]
Revised: [Equation]

Original: where δ ( x = x i , y = y i ) is a Dirac mass centered at ( x i , y i ) .
Revised: where δ ( x = x i , y = y i ) is a Dirac mass centered at ( x i , y i ) .

Original: Using the empirical distribution P δ , we can now approximate the expected risk by the empirical risk :
Revised: Using the empirical distribution P δ , we can now approximate the expected risk by the empirical risk :

Original: [Equation]
Revised: [Equation]

Original: Learning the function f by minimizing (1) is known as the Empirical Risk Minimization (ERM) principle (Vapnik, 1998).
Revised: Learning the function f by minimizing (1) is known as the Empirical Risk Minimization (ERM) principle (Vapnik, 1998).

Original: While efﬁcient to compute, the empirical risk (1) monitors the behaviour of f only at a ﬁnite set of n examples.
Revised: While efﬁcient to compute, the empirical risk (1) monitors the behaviour of f only at a ﬁnite set of n examples.

Original: When considering functions with a number parameters comparable to n (such as large neural networks), one trivial way to minimize (1) is to memorize the training data (Zhang et al., 2017).
Revised: When considering functions with a number parameters comparable to n (such as large neural networks), one trivial way to minimize (1) is to memorize the training data (Zhang et al., 2017).

Original: Memorization, in turn, leads to the undesirable behaviour of f outside the training data (Szegedy et al., 2014).
Revised: Memorization, in turn, leads to the undesirable behaviour of f outside the training data (Szegedy et al., 2014).

Original: However, the na¨ıve estimate P δ is one out of many possible choices to approximate the true distribution P .

Original: For instance, in the Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2000), the distribution P is approximated by

Original: [Equation] for (x1, y1), (x2, y2) in zip (loader1, loader2): lam = numpy.random.beta(alpha, alpha) x = Variable(lam * x1 + (1. - lam) * x2)y =
Revised: # y1, y2 should be one-hot vectors for (x1, y1), (x2, y2) in zip (loader1, loader2): lam = numpy.random.beta(alpha, alpha) x = Variable(lam * x1 + (1. - lam) * x2)

Original: Variable(lam
Revised: y =

Original:  * y1 + (1. - lam) * y2) optimizer.zero_grad() loss(net(x), y).backward() optimizer.step() [Figure]
Revised: Variable(lam * y1 + (1. - lam) * y2) optimizer.zero_grad() loss(net(x), y).backward() optimizer.step() (

Original: ERM mixup
Revised: ERM mixup

Revised: ( y x )

Revised: However, the na¨ıve estimate P δ is one out of many possible choices to approximate the true distribution P .

Revised: For instance, in the Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2000), the distribution P is approximated by

Original: [Figure]
Revised: [Equation]

Original: where ν is a vicinity distribution that measures the probability of ﬁnding the virtual feature-target pair (˜ x, ˜ y ) in the vicinity of the training feature-target pair ( x i , y i ) .
Revised: where ν is a vicinity distribution that measures the probability of ﬁnding the virtual feature-target pair (˜ x, ˜ y ) in the vicinity of the training feature-target pair ( x i , y i ) .

Original: In particular, Chapelle et al.
Revised: In particular, Chapelle et al.

Original: (2000) considered Gaussian vicinities
Revised: (2000) considered Gaussian vicinities

Original: ν
Revised: ν

Original: (˜ x, ˜ y
Revised: (˜ x, ˜ y

Original: | x
Revised: | x

Original: i , y i ) =
Revised: i , y i ) =

Original: N (˜ x − x i , σ 2 ) δ (˜ y
Revised: N (˜ x − x i , σ 2 ) δ (˜ y

Original: =
Revised: =

Original: y i ) , which is equivalent to augmenting the training data with additive Gaussian noise.
Revised: y i ) , which is equivalent to augmenting the training data with additive Gaussian noise.

Original: To learn using VRM, we sample the vicinal distribution to construct a dataset D ν := { (˜ x i , ˜ y i ) } mi =1 , and minimize the empirical vicinal risk : 
Revised: To learn using VRM, we sample the vicinal distribution to construct a dataset D ν := { (˜ x i , ˜ y i ) } mi =1 , and minimize the empirical vicinal risk : [Equation] i = (cid:96)

Original: [Equation]
Revised: [Equation]

Original: The contribution of this paper is to propose a generic vicinal distribution, called mixup :
Revised: The contribution of this paper is to propose a generic vicinal distribution, called mixup :

Original: [Equation]
Revised: [Equation]

Original: where λ ∼ Beta ( α, α ) , for α ∈ (0 , ∞ ) .
Revised: where λ ∼ Beta ( α, α ) , for α ∈ (0 , ∞ ) .

Original: In a nutshell, sampling from the mixup vicinal distribution produces virtual feature-target vectors
Revised: In a nutshell, sampling from the mixup vicinal distribution produces virtual feature-target vectors

Original: [Equation]
Revised: [Equation]

Original: where ( x i , y i ) and ( x j , y j ) are two feature-target vectors drawn at random from the training data, and λ ∈ [0 , 1] .
Revised: where ( x i , y i ) and ( x j , y j ) are two feature-target vectors drawn at random from the training data, and λ ∈ [0 , 1] .

Original: The mixup hyper-parameter α controls the strength of interpolation between feature-target pairs, recovering the ERM principle as α → 0 .
Revised: The mixup hyper-parameter α controls the strength of interpolation between feature-target pairs, recovering the ERM principle as α → 0 .

Original: The implementation of mixup training is straightforward, and introduces a minimal computation overhead.
Revised: The implementation of mixup training is straightforward, and introduces a minimal computation overhead.

Original: Figure 1a shows the few lines of code necessary to implement mixup training in PyTorch.
Revised: Figure 1a shows the few lines of code necessary to implement mixup training in PyTorch.

Original: Finally, we mention alternative design choices.
Revised: Finally, we mention alternative design choices.

Original: First,  we have observed thatperforming convex combinations of three or more examples with weights sampled from a Dirichlet distribution works similarly well, but increases the computation cost of mixup .
Revised: First, in preliminary experiments we ﬁnd that convex combinations of three or more examples with weights sampled from a Dirichlet distribution does not provide further gain, but increases the computation cost of mixup .

Original: Second, our current implementation uses a single data loader to obtain one minibatch, and then mixup is applied to the same minibatch after random shufﬂing.
Revised: Second, our current implementation uses a single data loader to obtain one minibatch, and then mixup is applied to the same minibatch after random shufﬂing.

Original: We found this strategy works equally well, while reducing I/O requirements.
Revised: We found this strategy works equally well, while reducing I/O requirements.

Original: Third, interpolating only between inputs with equal label did not lead to the performance gains of mixup discussed in the sequel.
Revised: Third, interpolating only between inputs with equal label did not lead to the performance gains of mixup discussed in the sequel.

Revised: More empirical comparison can be found in Section 3.8.

Original: What is mixup doing?
Revised: What is mixup doing?

Original: The mixup vicinal distribution can be understood as a form of data augmentation that encourages the model f to behave linearly in-between training examples.
Revised: The mixup vicinal distribution can be understood as a form of data augmentation that encourages the model f to behave linearly in-between training examples.

Original: We argue that this linear behaviour reduces the amount of undesirable oscillations when predicting outside the training examples.
Revised: We argue that this linear behaviour reduces the amount of undesirable oscillations when predicting outside the training examples.

Original: Also, linearity is a good inductive bias from the perspective of Occam’s razor, since it is one of the simplest possible behaviors.
Revised: Also, linearity is a good inductive bias from the perspective of Occam’s razor, [Figure] [Figure]

Original: Figure 1b shows that mixup leads to decision boundaries that transition linearly from class to class, providing a smoother estimate of uncertainty. Figure 2 illustrate the average behaviors of two neural network models trained on the CIFARdataset using ERM and mixup . Both models have the same architecture, are trained with the same procedure, and are evaluated at the same points in-between randomly sampled training data. The model trained with mixup is more stable in terms of model predictions and gradient norms in-between training samples.
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Model
Revised: Model

Original: Method
Revised: Method

Original: Epochs
Revised: Epochs

Original: Top-1 Error
Revised: Top-1 Error

Original: Top-5 Error
Revised: Top-5 Error

Original: ResNetERM (Goyal et al., 2017) 23 .-
Revised: ResNetERM (Goyal et al., 2017) 23 .-

Original: mixup α = 0 . 

Revised: mixup α = 0 .

Original: ResNetERM (Goyal et al., 2017) 22 .-
Revised: ResNetERM (Goyal et al., 2017) 22 .-

Original: mixup α = 0 .
Revised: mixup α = 0 .

Original: ResNeXt-101 32*4d
Revised: ResNeXt-101 32*4d

Original: ERM (Xie et al., 2016) 21 .-
Revised: ERM (Xie et al., 2016) 21 .-

Original: ERM 21 .

Revised: ERM 21 .

Original: .mixup
Revised: .mixup

Original: α = 0 .
Revised: α = 0 .

Original: ResNeXt-101 64*4d
Revised: ResNeXt-101 64*4d

Original: ERM (Xie et al., 2016) 20 .5 .mixup
Revised: ERM (Xie et al., 2016) 20 .5 .mixup

Original: α = 0 .
Revised: α = 0 .

Original: ResNetERM 23 .

Revised: ResNetERM 23 .

Original: .mixup
Revised: .mixup

Original: α = 0 .
Revised: α = 0 .

Original: ResNetERM 22 .
Revised: ResNetERM 22 .

Original: [Table]
Revised: [Table]

Revised: since it is one of the simplest possible behaviors.

Revised: Figure 1b shows that mixup leads to decision boundaries that transition linearly from class to class, providing a smoother estimate of uncertainty.

Revised: Figure 2 illustrate the average behaviors of two neural network models trained on the CIFARdataset using ERM and mixup .

Revised: Both models have the same architecture, are trained with the same procedure, and are evaluated at the same points in-between randomly sampled training data.

Revised: The model trained with mixup is more stable in terms of model predictions and gradient norms in-between training samples.

Original:  E XPERIMENTS 3. I MAGE N ET CLASSIFICATION
Revised:  E XPERIMENTS 3. I MAGE N ET CLASSIFICATION

Original: We evaluate mixup on the ImageNet-2012 classiﬁcation dataset (Russakovsky et al., 2015).
Revised: We evaluate mixup on the ImageNet-2012 classiﬁcation dataset (Russakovsky et al., 2015).

Original: This dataset contains 1.3 million training images and 50,000 validation images, from a total of 1,000 classes.
Revised: This dataset contains 1.3 million training images and 50,000 validation images, from a total of 1,000 classes.

Original: For training, we follow standard data augmentation practices: scale and aspect ratio distortions, random crops, and horizontal ﬂips (Goyal et al., 2017).
Revised: For training, we follow standard data augmentation practices: scale and aspect ratio distortions, random crops, and horizontal ﬂips (Goyal et al., 2017).

Original: During evaluation, only the 224 × 224 central crop of each image is tested.
Revised: During evaluation, only the 224 × 224 central crop of each image is tested.

Original: We use mixup and ERM to train several state-of-the-art ImageNetclassiﬁcation models, and report both top-1 and top-5 error rates in Table 1. 
Revised: We use mixup and ERM to train several state-of-the-art ImageNetclassiﬁcation models, and report both top-1 and top-5 error rates in Table 1. [Figure]

Original: For all the experiments in this section, we use data-parallel distributed training in Caffe2 1 with a minibatch size of 1,024.
Revised: For all the experiments in this section, we use data-parallel distributed training in Caffe2 1 with a minibatch size of 1,024.

Original: We use the learning rate schedule described in (Goyal et al., 2017).
Revised: We use the learning rate schedule described in (Goyal et al., 2017).

Original: Speciﬁcally, the learning rate is increased linearly from 0.1 to 0.4 during the ﬁrst 5 epochs, and it is then divided by 10 after 30, 60 and 80 epochs when training for 90 epochs; or after 60, 120 andepochs when training for 200 epochs.
Revised: Speciﬁcally, the learning rate is increased linearly from 0.1 to 0.4 during the ﬁrst 5 epochs, and it is then divided by 10 after 30, 60 and 80 epochs when training for 90 epochs; or after 60, 120 andepochs when training for 200 epochs.

Original: [Figure]

Original: For mixup , we ﬁnd that α ∈ [0 . 1 , 0 . 4] leads to improved performance over ERM, whereas for large α , mixup leads to underﬁtting.
Revised: For mixup , we ﬁnd that α ∈ [0 . 1 , 0 . 4] leads to improved performance over ERM, whereas for large α , mixup leads to underﬁtting.

Original: We also ﬁnd that models with higher capacities and/or longer training runs are the ones to beneﬁt the most from mixup .
Revised: We also ﬁnd that models with higher capacities and/or longer training runs are the ones to beneﬁt the most from mixup .

Original: For example, when trained for 90 epochs, the mixup variants of ResNet-101 and ResNeXt-101 obtain a greater improvement (0.5% to 0.6%) over their
Revised: For example, when trained for 90 epochs, the mixup variants of ResNet-101 and ResNeXt-101 obtain a greater improvement (0.5% to 0.6%) over their

Original: ERM analogues than the gain of smaller models such as ResNet-50 (0.2%).
Revised: ERM analogues than the gain of smaller models such as ResNet-50 (0.2%).

Original: When trained forepochs, the top-1 error of the mixup variant of ResNet-50 is further reduced by 1.2% compared to the 90 epoch run, whereas its ERM analogue stays the same.
Revised: When trained forepochs, the top-1 error of the mixup variant of ResNet-50 is further reduced by 1.2% compared to the 90 epoch run, whereas its ERM analogue stays the same.

Original: 3. CIFAR-10 AND CIFAR- 
Revised: 3. CIFAR-10 AND CIFAR-

Original: We conduct additional image classiﬁcation experiments on the CIFAR-10 and CIFAR-100 datasets to further evaluate the generalization performance of mixup .
Revised: We conduct additional image classiﬁcation experiments on the CIFAR-10 and CIFAR-100 datasets to further evaluate the generalization performance of mixup .

Original: In particular, we compare ERM and mixup training for: PreAct ResNet-18 (He et al., 2016) as implemented in (Liu, 2017), WideResNet28-10 (Zagoruyko & Komodakis, 2016a) as implemented in (Zagoruyko & Komodakis, 2016b), and
Revised: In particular, we compare ERM and mixup training for: PreAct ResNet-18 (He et al., 2016) as implemented in (Liu, 2017), WideResNet28-10 (Zagoruyko & Komodakis, 2016a) as implemented in (Zagoruyko & Komodakis, 2016b), and

Original: DenseNet (Huang et al., 2017) as implemented in (Veit, 2017).
Revised: DenseNet (Huang et al., 2017) as implemented in (Veit, 2017).

Original: For DenseNet, we change the growth rate to 40 to follow the DenseNet-BC-190 speciﬁcation from (Huang et al., 2017).
Revised: For DenseNet, we change the growth rate to 40 to follow the DenseNet-BC-190 speciﬁcation from (Huang et al., 2017).

Original: For mixup , we ﬁx α = 1 , which results in interpolations λ uniformly distributed between zero and one.
Revised: For mixup , we ﬁx α = 1 , which results in interpolations λ uniformly distributed between zero and one.

Original: All models are trained on a single Nvidia Tesla P100 GPU using PyTorch 2 for 200 epochs on the training set with 128 examples per minibatch, and evaluated on the test set.
Revised: All models are trained on a single Nvidia Tesla P100 GPU using PyTorch 2 for 200 epochs on the training set with 128 examples per minibatch, and evaluated on the test set.

Original: Learning rates start at 0.1 and are divided by 10 after 100 and 150 epochs for all models except WideResNet.
Revised: Learning rates start at 0.1 and are divided by 10 after 100 and 150 epochs for all models except WideResNet.

Original: For WideResNet, we follow (Zagoruyko & Komodakis, 2016a) and divide the learning rate by 10 after 60, 120 andepochs.
Revised: For WideResNet, we follow (Zagoruyko & Komodakis, 2016a) and divide the learning rate by 10 after 60, 120 andepochs.

Original: Weight decay is set to 10 − 4 .
Revised: Weight decay is set to 10 − 4 .

Original: We do not use dropout in these experiments.
Revised: We do not use dropout in these experiments.

Original: We summarize our results in Figure 3a.
Revised: We summarize our results in Figure 3a.

Original: In both CIFAR-10 and CIFAR-100 classiﬁcation problems, the models trained using mixup signiﬁcantly outperform their analogues trained with ERM.
Revised: In both CIFAR-10 and CIFAR-100 classiﬁcation problems, the models trained using mixup signiﬁcantly outperform their analogues trained with ERM.

Original: As seen in Figure 3b, mixup and ERM converge at a similar speed to their best test errors.
Revised: As seen in Figure 3b, mixup and ERM converge at a similar speed to their best test errors.

Original: Note that the
Revised: Note that the

Original: DenseNet models in (Huang et al., 2017) were trained for 300 epochs with further learning rate decays scheduled at the 150 and 225 epochs, which may explain the discrepancy the performance of
Revised: DenseNet models in (Huang et al., 2017) were trained for 300 epochs with further learning rate decays scheduled at the 150 and 225 epochs, which may explain the discrepancy the performance of

Original: DenseNet reported in Figure 3a and the original result of Huang et al.
Revised: DenseNet reported in Figure 3a and the original result of Huang et al.

Original: 3. S PEECH DATA
Revised: 3. S PEECH DATA

Original: Next, we perform speech recognition experiments using the Google commands dataset (Warden, 2017).
Revised: Next, we perform speech recognition experiments using the Google commands dataset (Warden, 2017).

Original: The dataset contains 65,000 utterances, where each utterance is about one-second long and belongs to one out of 30 classes.
Revised: The dataset contains 65,000 utterances, where each utterance is about one-second long and belongs to one out of 30 classes.

Original: The classes correspond to voice commands such as yes, no, down, left , as pronounced by a few thousand different speakers.
Revised: The classes correspond to voice commands such as yes, no, down, left , as pronounced by a few thousand different speakers.

Original: To preprocess the utterances, we ﬁrstextract normalized spectrograms from the original waveforms at a sampling rate of 16 kHz. Next, we zero-pad the spectrograms to equalize their sizes at 160 × 101 . For speech data, it is reasonable to apply mixup both at the waveform and spectrogram levels. Here, we apply mixup at the spectrogram level just before feeding the data to the network.
Revised: To preprocess the utterances, we ﬁrst

Original: For this experiment, we compare a LeNet (Lecun et al., 2001) and a VGG-11 (Simonyan & Zisserman, 2015) architecture, each of them composed by two convolutional and two fully-connected layers.

Original: [Figure]
Revised: [Figure]

Original: Method

Original: Validation set
Revised: Validation set

Original: Test set
Revised: Test set

Original: LeNet
Revised: LeNet

Original: ERM 

Revised: ERM

Original: .10 .mixup ( α = 0 . 1) 

Revised: .10 .mixup ( α = 0 . 1)

Original: .10 .mixup
Revised: .10 .mixup

Original: ( α = 0 .
Revised: ( α = 0 .

Original: VGGERM 5 .4 .mixup
Revised: VGGERM 5 .4 .mixup

Original: ( α = 0 .
Revised: ( α = 0 .

Original: 1) 4 .3 .mixup
Revised: 1) 4 .3 .mixup

Original: ( α = 0 . 2)
Revised: ( α = 0 . 2)

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: extract normalized spectrograms from the original waveforms at a sampling rate of 16 kHz.

Revised: Next, we zero-pad the spectrograms to equalize their sizes at 160 × 101 .

Revised: For speech data, it is reasonable to apply mixup both at the waveform and spectrogram levels.

Revised: Here, we apply mixup at the spectrogram level just before feeding the data to the network.

Revised: For this experiment, we compare a LeNet (Lecun et al., 2001) and a VGG-11 (Simonyan & Zisserman, 2015) architecture, each of them composed by two convolutional and two fully-connected layers.

Original: We train each model for 30 epochs with minibatches of 100 examples, using Adam as the optimizer (Kingma & Ba, 2015).
Revised: We train each model for 30 epochs with minibatches of 100 examples, using Adam as the optimizer (Kingma & Ba, 2015).

Original: Training starts with a learning rate equal to 3 × 10 − 3 and is divided byevery 10 epochs.
Revised: Training starts with a learning rate equal to 3 × 10 − 3 and is divided byevery 10 epochs.

Original: For mixup , we use a warm-up period of ﬁve epochs where we train the network on original training examples, since we ﬁnd it speeds up initial convergence.
Revised: For mixup , we use a warm-up period of ﬁve epochs where we train the network on original training examples, since we ﬁnd it speeds up initial convergence.

Original: Table 4 shows that mixup outperforms ERM on this task, specially when using VGG-11, the model with larger capacity.
Revised: Table 4 shows that mixup outperforms ERM on this task, specially when using VGG-11, the model with larger capacity.

Original: 3. M EMORIZATION OF CORRUPTED LABELS
Revised: 3. M EMORIZATION OF CORRUPTED LABELS

Original: Following Zhang et al.
Revised: Following Zhang et al.

Original: (2017), we evaluate the robustness of ERM and mixup models against randomly corrupted labels.
Revised: (2017), we evaluate the robustness of ERM and mixup models against randomly corrupted labels.

Original: We hypothesize that increasing the strength of mixup interpolation α should generate virtual examples further from the training examples, making memorization more difﬁcult to achieve.
Revised: We hypothesize that increasing the strength of mixup interpolation α should generate virtual examples further from the training examples, making memorization more difﬁcult to achieve.

Original: In particular, it should be easier to learn interpolations between real examples compared to memorizing interpolations involving random labels.
Revised: In particular, it should be easier to learn interpolations between real examples compared to memorizing interpolations involving random labels.

Original: We adapt an open-source implementation (Zhang, 2017) to generate three CIFAR-10 training sets, where 20%, 50%, or 80% of the labels are replaced by random noise, respectively.
Revised: We adapt an open-source implementation (Zhang, 2017) to generate three CIFAR-10 training sets, where 20%, 50%, or 80% of the labels are replaced by random noise, respectively.

Original: All the test labels are kept intact for evaluation.
Revised: All the test labels are kept intact for evaluation.

Original: Dropout (Srivastava et al., 2014) is considered the state-of-the-art method for learning with corrupted labels (Arpit et al., 2017).
Revised: Dropout (Srivastava et al., 2014) is considered the state-of-the-art method for learning with corrupted labels (Arpit et al., 2017).

Original: Thus, we compare in these experiments mixup , dropout, mixup + dropout, and ERM.
Revised: Thus, we compare in these experiments mixup , dropout, mixup + dropout, and ERM.

Original: For mixup , we choose α ∈ { 1 , 2 , 8 , 32 } ; for dropout, we add one dropout layer in each PreAct block after the ReLU activation layer between two convolution layers, as suggested in (Zagoruyko & Komodakis, 2016a).
Revised: For mixup , we choose α ∈ { 1 , 2 , 8 , 32 } ; for dropout, we add one dropout layer in each PreAct block after the ReLU activation layer between two convolution layers, as suggested in (Zagoruyko & Komodakis, 2016a).

Original: We choose the dropout probability p ∈ { 0 . 5 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: We choose the dropout probability p ∈ { 0 . 5 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: For the combination of mixup and dropout, we choose α ∈ { 1 , 2 , 4 , 8 } and p ∈ { 0 . 3 , 0 .
Revised: For the combination of mixup and dropout, we choose α ∈ { 1 , 2 , 4 , 8 } and p ∈ { 0 . 3 , 0 .

Original: These experiments use the PreAct
Revised: These experiments use the PreAct

Original: ResNet-18 (He et al., 2016) model implemented in (Liu, 2017).
Revised: ResNet-18 (He et al., 2016) model implemented in (Liu, 2017).

Original: All the other settings are the same as in Section 3.2.
Revised: All the other settings are the same as in Section 3.2.

Original: We summarize our results in Table 2, where we note the best test error achieved during the training session, as well as the ﬁnal test error after 200 epochs.
Revised: We summarize our results in Table 2, where we note the best test error achieved during the training session, as well as the ﬁnal test error after 200 epochs.

Original: To quantify the amount of memorization, we also evaluate the training errors at the last epoch on real labels and corrupted labels.
Revised: To quantify the amount of memorization, we also evaluate the training errors at the last epoch on real labels and corrupted labels.

Original: As the training progresses with a smaller learning rate (e.g. less than 0.01), the ERM model starts to overﬁt the corrupted labels.
Revised: As the training progresses with a smaller learning rate (e.g. less than 0.01), the ERM model starts to overﬁt the corrupted labels.

Original: When using a large probability (e.g. 0.7 or 0.8), dropout can effectively reduce overﬁtting.
Revised: When using a large probability (e.g. 0.7 or 0.8), dropout can effectively reduce overﬁtting.

Original: mixup with a large α (e.g. 8 or 32) outperforms dropout on both the best and last epoch test errors, and achieves lower training error on real labels while remaining resistant to noisy labels.
Revised: mixup with a large α (e.g. 8 or 32) outperforms dropout on both the best and last epoch test errors, and achieves lower training error on real labels while remaining resistant to noisy labels.

Original: Interestingly, mixup + dropout performs the best of all, showing that the two methods are compatible.
Revised: Interestingly, mixup + dropout performs the best of all, showing that the two methods are compatible.

Original: 3. R OBUSTNESS TO A DVERSARIAL EXAMPLES
Revised: 3. R OBUSTNESS TO A DVERSARIAL EXAMPLES

Original: One undesirable consequence of models trained using ERM is their fragility to adversarial examples (Szegedy et al., 2014).
Revised: One undesirable consequence of models trained using ERM is their fragility to adversarial examples (Szegedy et al., 2014).

Original: Adversarial examples are obtained by adding tiny (visually imperceptible) perturbations to legitimate examples in order to deteriorate the performance of the model.
Revised: Adversarial examples are obtained by adding tiny (visually imperceptible) perturbations to legitimate examples in order to deteriorate the performance of the model.

Original: The adversarial noise is generated by ascending the gradient of the loss surface with respect to the legitimate example.
Revised: The adversarial noise is generated by ascending the gradient of the loss surface with respect to the legitimate example.

Original: Improving the robustness to adversarial examples is a topic of active research.
Revised: Improving the robustness to adversarial examples is a topic of active research.

Original:  Among the several methods aiming to solve this problem, some have proposed to penalize the norm of the Jacobian of the model to control its Lipschitz constant (Drucker & Le Cun, 1992; Cisse et al., 2017;
Revised: [Table] [Table] Among the several methods aiming to solve this problem, some have proposed to penalize the norm of the Jacobian of the model to control its Lipschitz constant (Drucker & Le Cun, 1992; Cisse et al., 2017;

Original: Bartlett et al., 2017; Hein & Andriushchenko, 2017).
Revised: Bartlett et al., 2017; Hein & Andriushchenko, 2017).

Original: Other approaches perform data augmentation by producing and training on adversarial examples (Goodfellow et al., 2015).
Revised: Other approaches perform data augmentation by producing and training on adversarial examples (Goodfellow et al., 2015).

Original: Unfortunately, all of these methods adda signiﬁcant computational overhead to ERM, due to costly regularization (Cisse et al., 2017) or data augmentation (Goodfellow et al., 2015) procedures.
Revised: Unfortunately, all of these methods add signiﬁcant computational overhead to ERM.

Original: Here, we show that mixup[Table] [Table] can signiﬁcantly improve the robustness of neural networks without hindering the speed of ERM by penalizing the norm of the gradient of the loss w.r.t a given input along the most plausible directions (e.g. the directions to other training points).
Revised: Here, we show that mixup can signiﬁcantly improve the robustness of neural networks without hindering the speed of ERM by penalizing the norm of the gradient of the loss w.r.t a given input along the most plausible directions (e.g. the directions to other training points).

Original: Indeed, Figure 2 shows that mixup results in models having a smaller loss and gradient norm between examples compared to vanilla ERM.
Revised: Indeed, Figure 2 shows that mixup results in models having a smaller loss and gradient norm between examples compared to vanilla ERM.

Original: To assess the robustness of mixup models to adversarial examples, we use three ResNet-101 models: two of them trained using ERM on ImageNet-2012, and the third trained using mixup .
Revised: To assess the robustness of mixup models to adversarial examples, we use three ResNet-101 models: two of them trained using ERM on ImageNet-2012, and the third trained using mixup .

Original: In the ﬁrst set of experiments, we study the robustness of one ERM model and the mixup model against white box attacks.
Revised: In the ﬁrst set of experiments, we study the robustness of one ERM model and the mixup model against white box attacks.

Original: That is, for each of the two models, we use the model itself to generate adversarial examples, either using the Fast Gradient Sign Method (FGSM) or the Iterative FGSM (I-FGSM) methods (Goodfellow et al., 2015), allowing a maximum perturbation of (cid:15)
Revised: That is, for each of the two models, we use the model itself to generate adversarial examples, either using the Fast Gradient Sign Method (FGSM) or the Iterative FGSM (I-FGSM) methods (Goodfellow et al., 2015), allowing a maximum perturbation of (cid:15)

Original: = 4 for every pixel.
Revised: = 4 for every pixel.

Original: For
Revised: For

Original: I-FGSM, we use 10 iterations with equal step size.
Revised: I-FGSM, we use 10 iterations with equal step size.

Original: In the second set of experiments, we evaluate robustness against black box attacks.
Revised: In the second set of experiments, we evaluate robustness against black box attacks.

Original: That is, we use the ﬁrst ERM model to produce adversarial examples using FGSM and I-FGSM.
Revised: That is, we use the ﬁrst ERM model to produce adversarial examples using FGSM and I-FGSM.

Original: Then, we test the robustness of the second ERM model and the mixup model to these examples.
Revised: Then, we test the robustness of the second ERM model and the mixup model to these examples.

Original: The results of both settings are summarized in Table 3.
Revised: The results of both settings are summarized in Table 3.

Original: For the FGSM white box attack, the mixup model is 2 .
Revised: For the FGSM white box attack, the mixup model is 2 .

Original: 7 times more robust than the ERM model in terms of Top-1 error.
Revised: 7 times more robust than the ERM model in terms of Top-1 error.

Original: For the FGSM black box attack, the mixup model is 1 .
Revised: For the FGSM black box attack, the mixup model is 1 .

Original: 25 times more robust than the ERM model in terms of Top-1 error.
Revised: 25 times more robust than the ERM model in terms of Top-1 error.

Original: Also, while both mixup and ERM are not robust to white box I-FGSM attacks, mixup is about 40% more robust than ERM in the black box I-FGSM setting.
Revised: Also, while both mixup and ERM are not robust to white box I-FGSM attacks, mixup is about 40% more robust than ERM in the black box I-FGSM setting.

Original: Overall, mixup produces neural networks that are signiﬁcantly more robust than ERM against adversarial examples in white box and black settings without additional overhead compared to ERM.
Revised: Overall, mixup produces neural networks that are signiﬁcantly more robust than ERM against adversarial examples in white box and black settings without additional overhead compared to ERM.

Original: 3. T ABULAR DATA

Original: To further explore the performance of mixup on non-image data, we performed a series of experiments on six arbitrary classiﬁcation problems drawn from the UCI dataset (Lichman, 2013).

Original: The neural networks in this section are fully-connected, and have two hidden layers of 128 ReLU units.

Original: The parameters of these neural networks are learned using Adam (Kingma & Ba, 2015) with default

Original: Dataset
Revised: Dataset

Original: ERM mixup
Revised: ERM mixup

Original: Abalone 74 .73 .
Revised: Abalone 74 .73 .

Original: Arcene 57 .48 .
Revised: Arcene 57 .48 .

Original: Arrhythmia 56 .46 .
Revised: Arrhythmia 56 .46 .

Original: Dataset
Revised: Dataset

Original: ERM mixup
Revised: ERM mixup

Original: Htru 2 .2 .
Revised: Htru 2 .2 .

Original: Iris 21 .17 .
Revised: Iris 21 .17 .

Original: Phishing 

Revised: Phishing

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Original:  hyper-parameters,  of  of 
Revised: To further explore the performance of mixup on non-image data, we performed a series of experiments on six arbitrary classiﬁcation problems drawn from the UCI dataset (Lichman, 2013). The neural networks in this section are fully-connected, and have two hidden layers of 128 ReLU units. The parameters of these neural networks are learned using Adam (Kingma & Ba, 2015) with default hyper-parameters, over 10 epochs of mini-batches of size 16.

Original: Table 4 shows that mixup improves the average test error on four out of the six considered datasets, and never underperforms ERM.
Revised: Table 4 shows that mixup improves the average test error on four out of the six considered datasets, and never underperforms ERM.

Original: 3. S TABILIZATION OF G ENERATIVE A DVERSARIAL N ETWORKS (GAN S )
Revised: 3. S TABILIZATION OF G ENERATIVE A DVERSARIAL N ETWORKS (GAN S )

Original: Generative Adversarial Networks, also known as GANs (Goodfellow et al., 2014), are a powerful family of implicit generative models.
Revised: Generative Adversarial Networks, also known as GANs (Goodfellow et al., 2014), are a powerful family of implicit generative models.

Original: In GANs, a generator and a discriminator compete against each other to model a distribution P .
Revised: In GANs, a generator and a discriminator compete against each other to model a distribution P .

Original: On the one hand, the generator g competes to transform noise vectors z ∼ Q into fake samples g ( z ) that resemble real samples x ∼ P .
Revised: On the one hand, the generator g competes to transform noise vectors z ∼ Q into fake samples g ( z ) that resemble real samples x ∼ P .

Original: On the other hand, the discriminator competes to distinguish between real samples x and fake samples g ( z ) .
Revised: On the other hand, the discriminator competes to distinguish between real samples x and fake samples g ( z ) .

Original: Mathematically, training a GAN is equivalent to solving the optimization problem
Revised: Mathematically, training a GAN is equivalent to solving the optimization problem

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) is the binary cross entropy loss.
Revised: where (cid:96) is the binary cross entropy loss.

Original: Unfortunately, solving the previous min-max equation is a notoriously difﬁcult optimization problem (Goodfellow, 2016), since the discriminator often provides the generator with vanishing gradients.
Revised: Unfortunately, solving the previous min-max equation is a notoriously difﬁcult optimization problem (Goodfellow, 2016), since the discriminator often provides the generator with vanishing gradients.

Original: We argue that mixup should stabilize GAN training because it acts as a regularizer on the gradients of the discriminator, akin to the binary classiﬁer in Figure 1b.
Revised: We argue that mixup should stabilize GAN training because it acts as a regularizer on the gradients of the discriminator, akin to the binary classiﬁer in Figure 1b.

Original: Then, the smoothness of the discriminator guarantees a stable source of gradient information to the generator.
Revised: Then, the smoothness of the discriminator guarantees a stable source of gradient information to the generator.

Original: The mixup formulation of GANs is:
Revised: The mixup formulation of GANs is:

Original: [Equation]
Revised: [Equation]

Original: Figure 5 illustrates the stabilizing effect of mixup the training of GAN (orange samples) when modeling two toy datasets (blue samples).
Revised: Figure 5 illustrates the stabilizing effect of mixup the training of GAN (orange samples) when modeling two toy datasets (blue samples).

Original: The neural networks in these experiments are fullyconnected and have three hidden layers of 512 ReLU units.
Revised: The neural networks in these experiments are fullyconnected and have three hidden layers of 512 ReLU units.

Original: The generator network accepts twodimensional Gaussian noise vectors.
Revised: The generator network accepts twodimensional Gaussian noise vectors.

Original: The networks are trained for 20,000 mini-batches of size 128 using the Adam optimizer with default parameters, where the discriminator is trained for ﬁve iterations before every generator iteration.
Revised: The networks are trained for 20,000 mini-batches of size 128 using the Adam optimizer with default parameters, where the discriminator is trained for ﬁve iterations before every generator iteration.

Original: The training of mixup GANs seems promisingly robust to hyper-parameter and architectural choices.
Revised: The training of mixup GANs seems promisingly robust to hyper-parameter and architectural choices.

Original: 3. A BLATION S TUDIES

Original: mixup is a data augmentation method that consists of only two parts: random convex combination of raw inputs, and correspondingly, convex combination of one-hot label encodings.

Original: However, there are several design choices to make.

Original: For example, on how to augment the inputs, we could have chosen to interpolate the latent representations (i.e. feature maps) of a neural network, and we could have chosen to interpolate only between the nearest neighbors, or only between inputs of the same class.

Original: [Table]
Revised: [Table]

Original: (cid:51) (cid:51) 5 .5 .(AC
Revised: (cid:51) (cid:51) 5 .5 .(AC

Original: + RP) (cid:15) = 0 .(cid:51)
Revised: + RP) (cid:15) = 0 .(cid:51)

Original: (cid:51) 4 .5 .(cid:15)
Revised: (cid:51) 4 .5 .(cid:15)

Original: = 0 .(cid:51)
Revised: = 0 .(cid:51)

Original: (cid:51) 5 .5 .add
Revised: (cid:51) 5 .5 .add

Original: Gaussian noise σ = 0 .(cid:51)
Revised: Gaussian noise σ = 0 .(cid:51)

Original: (cid:55) 5 .5 .to
Revised: (cid:55) 5 .5 .

Original: inputs σ = 0 .(cid:51)

Original: (cid:55)

Original: [Table]
Revised: [Table]

Original: σ =
Revised: to inputs

Original: [Table]
Revised: [Table]

Original: (cid:51)
Revised: (cid:51)

Original: [Table]
Revised: [Table]

Revised: 3. A BLATION S TUDIES

Revised: mixup is a data augmentation method that consists of only two parts: random convex combination of raw inputs, and correspondingly, convex combination of one-hot label encodings.

Revised: However, there are several design choices to make.

Revised: For example, on how to augment the inputs, we could have chosen to interpolate the latent representations (i.e. feature maps) of a neural network, and we could have chosen to interpolate only between the nearest neighbors, or only between inputs of the same class.

Original: When the inputs to interpolate come from two different classes, we could have chosen to assign a single label to the synthetic input, for example using the label of the input that weights more in the convex combination.
Revised: When the inputs to interpolate come from two different classes, we could have chosen to assign a single label to the synthetic input, for example using the label of the input that weights more in the convex combination.

Original: To compare mixup with these alternative possibilities, we run a set of ablation study experiments using the PreAct ResNet-18 architecture on the CIFAR-10 dataset.
Revised: To compare mixup with these alternative possibilities, we run a set of ablation study experiments using the PreAct ResNet-18 architecture on the CIFAR-10 dataset.

Original: Speciﬁcally, for each of the data augmentation methods, we test two weight decay settings ( 10 −which works well for mixup , and 5 × 10 − 4 which works well for ERM).
Revised: Speciﬁcally, for each of the data augmentation methods, we test two weight decay settings ( 10 −which works well for mixup , and 5 × 10 − 4 which works well for ERM).

Original: All the other settings and hyperparameters are the same as reported in Section 3.2.
Revised: All the other settings and hyperparameters are the same as reported in Section 3.2.

Original: To compare interpolating raw inputs with interpolating latent representations, we test on random convex combination of the learned representations before each residual block (denoted Layer 1-4) or before the uppermost “average pooling + fully connected” layer (denoted Layer 5).
Revised: To compare interpolating raw inputs with interpolating latent representations, we test on random convex combination of the learned representations before each residual block (denoted Layer 1-4) or before the uppermost “average pooling + fully connected” layer (denoted Layer 5).

Original: To compare mixing random pairs of inputs (RP) with mixing nearest neighbors (KNN), we ﬁrst compute thenearest neighbors for each training sample, either from the same class (SC) or from all the classes (AC).
Revised: To compare mixing random pairs of inputs (RP) with mixing nearest neighbors (KNN), we ﬁrst compute thenearest neighbors for each training sample, either from the same class (SC) or from all the classes (AC).

Original: Then during training, for each sample in a minibatch, we replace the sample with a synthetic sample by convex combination with a random draw from its nearest neighbors.
Revised: Then during training, for each sample in a minibatch, we replace the sample with a synthetic sample by convex combination with a random draw from its nearest neighbors.

Original: To compare mixing all the classes (AC) with mixing within the same class (SC), we convex combine a minibatch with a random permutation of its sample index, where the permutation is done in a per-batch basis (AC) or a per-class basis (SC).
Revised: To compare mixing all the classes (AC) with mixing within the same class (SC), we convex combine a minibatch with a random permutation of its sample index, where the permutation is done in a per-batch basis (AC) or a per-class basis (SC).

Original: To compare mixing inputs and labels with mixing inputs only, we either use a convex combination of the two one-hot encodings as the target, or select the one-hot encoding of the closer training sample as the target.
Revised: To compare mixing inputs and labels with mixing inputs only, we either use a convex combination of the two one-hot encodings as the target, or select the one-hot encoding of the closer training sample as the target.

Original: For label smoothing, we follow Szegedy et al.
Revised: For label smoothing, we follow Szegedy et al.

Original: and use (cid:15) 10 as the target for incorrect classes, and 1 − 9 (cid:15) 10 as the target for the correct class.
Revised: and use (cid:15) 10 as the target for incorrect classes, and 1 − 9 (cid:15) 10 as the target for the correct class.

Original: Adding Gaussian noise to inputs is used as another baseline.
Revised: Adding Gaussian noise to inputs is used as another baseline.

Original: We report the median test errors of the last 10 epochs.
Revised: We report the median test errors of the last 10 epochs.

Original: Results are shown in Table 5.
Revised: Results are shown in Table 5.

Original: From the ablation study experiments, we have the following observations.
Revised: From the ablation study experiments, we have the following observations.

Original: First, mixup is the best data augmentation method we test, and is signiﬁcantly better than the second best method (mix input + label smoothing).
Revised: First, mixup is the best data augmentation method we test, and is signiﬁcantly better than the second best method (mix input + label smoothing).

Original: Second, the effect of regularization can be seen by comparing the test error with a small weight decay ( 10 − 4 ) with a large one ( 5 × 10 − 4 ).
Revised: Second, the effect of regularization can be seen by comparing the test error with a small weight decay ( 10 − 4 ) with a large one ( 5 × 10 − 4 ).

Original: For example, for ERM a large weight decay works better, whereas for mixup a small weight decay is preferred, conﬁrming its regularization effects.
Revised: For example, for ERM a large weight decay works better, whereas for mixup a small weight decay is preferred, conﬁrming its regularization effects.

Original: We also see an increasing advantage of large weight decay when interpolating in higher layers of latent representations, indicating decreasing strength of regularization.
Revised: We also see an increasing advantage of large weight decay when interpolating in higher layers of latent representations, indicating decreasing strength of regularization.

Original: Among all the input interpolation methods, mixing random pairs from all classes (AC + RP) has the strongest regularization effect.
Revised: Among all the input interpolation methods, mixing random pairs from all classes (AC + RP) has the strongest regularization effect.

Original: Label smoothing and adding Gaussian noise have a relatively small regularization effect.
Revised: Label smoothing and adding Gaussian noise have a relatively small regularization effect.

Original: Finally, we note that the SMOTE algorithm (Chawla et al., 2002) does not lead to a noticeable gain in performance.
Revised: Finally, we note that the SMOTE algorithm (Chawla et al., 2002) does not lead to a noticeable gain in performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Data augmentation lies at the heart of all successful applications of deep learning, ranging from image classiﬁcation (Krizhevsky et al., 2012) to speech recognition (Graves et al., 2013; Amodei et al., 2016).
Revised: Data augmentation lies at the heart of all successful applications of deep learning, ranging from image classiﬁcation (Krizhevsky et al., 2012) to speech recognition (Graves et al., 2013; Amodei et al., 2016).

Original: In all cases, substantial domain knowledge is leveraged to design suitable data transformations leading to improved generalization.
Revised: In all cases, substantial domain knowledge is leveraged to design suitable data transformations leading to improved generalization.

Original: In image classiﬁcation, for example, one routinely uses rotation, translation, cropping, resizing, ﬂipping (Lecun et al., 2001; Simonyan & Zisserman, 2015), and random erasing (Zhong et al., 2017) to enforce visually plausible invariances in the model through the training data.
Revised: In image classiﬁcation, for example, one routinely uses rotation, translation, cropping, resizing, ﬂipping (Lecun et al., 2001; Simonyan & Zisserman, 2015), and random erasing (Zhong et al., 2017) to enforce visually plausible invariances in the model through the training data.

Original: Similarly, in speech recognition, noise injection is a prevalent practice to improve the robustness and accuracy of the trained models (Amodei et al., 2016).
Revised: Similarly, in speech recognition, noise injection is a prevalent practice to improve the robustness and accuracy of the trained models (Amodei et al., 2016).

Original: More related to mixup , Chawla et al.
Revised: More related to mixup , Chawla et al.

Original: (2002) propose to augment the rare class in an imbalanced dataset by interpolating the nearest neighbors; DeVries & Taylor (2017) show that interpolation and extrapolation the nearest neighbors of the same class in feature space can improve generalization.
Revised: (2002) propose to augment the rare class in an imbalanced dataset by interpolating the nearest neighbors; DeVries & Taylor (2017) show that interpolation and extrapolation the nearest neighbors of the same class in feature space can improve generalization.

Original: However, their proposals only operate among the nearest neighbors within a certain class at the input / feature level, and hence does not account for changes in the corresponding labels.
Revised: However, their proposals only operate among the nearest neighbors within a certain class at the input / feature level, and hence does not account for changes in the corresponding labels.

Original: Recent approaches have also proposed to regularize the output distribution of a neural network by label smoothing (Szegedy et al., 2016), or penalizing high-conﬁdence softmax distributions (Pereyra et al., 2017).
Revised: Recent approaches have also proposed to regularize the output distribution of a neural network by label smoothing (Szegedy et al., 2016), or penalizing high-conﬁdence softmax distributions (Pereyra et al., 2017).

Original: These methods bear similarities with mixup in the sense that supervision depends on multiple smooth labels, rather than on single hard labels as in traditional ERM.
Revised: These methods bear similarities with mixup in the sense that supervision depends on multiple smooth labels, rather than on single hard labels as in traditional ERM.

Original: However, the label smoothing in these works is applied or regularized independently from the associated feature values.
Revised: However, the label smoothing in these works is applied or regularized independently from the associated feature values.

Original: mixup enjoys several desirable aspects of previous data augmentation and regularization schemes without suffering from their drawbacks.
Revised: mixup enjoys several desirable aspects of previous data augmentation and regularization schemes without suffering from their drawbacks.

Original: Like the method of DeVries & Taylor (2017), it does not require signiﬁcant domain knowledge.
Revised: Like the method of DeVries & Taylor (2017), it does not require signiﬁcant domain knowledge.

Original: Like label smoothing, the supervision of every example is not overly dominated by the ground-truth label.
Revised: Like label smoothing, the supervision of every example is not overly dominated by the ground-truth label.

Original: Unlike both of these approaches, the mixup transformation establishes a linear relationship between data augmentation and the supervision signal.
Revised: Unlike both of these approaches, the mixup transformation establishes a linear relationship between data augmentation and the supervision signal.

Original: We believe that this leads to a strong regularizer that improves generalization as demonstrated by our experiments.
Revised: We believe that this leads to a strong regularizer that improves generalization as demonstrated by our experiments.

Original: The linearity constraint, through its effect on the derivatives of the function approximated, also relates mixup to other methods such as Sobolev training of neural networks (Czarnecki et al., 2017) or
Revised: The linearity constraint, through its effect on the derivatives of the function approximated, also relates mixup to other methods such as Sobolev training of neural networks (Czarnecki et al., 2017) or

Original: WGAN-GP (Gulrajani et al., 2017).
Revised: WGAN-GP (Gulrajani et al., 2017).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have proposed mixup , a data-agnostic and straightforward data augmentation principle.
Revised: We have proposed mixup , a data-agnostic and straightforward data augmentation principle.

Original: We have shown that mixup is a form of vicinal risk minimization, which trains on virtual examples constructed as the linear interpolation of two random examples from the training set and their labels.
Revised: We have shown that mixup is a form of vicinal risk minimization, which trains on virtual examples constructed as the linear interpolation of two random examples from the training set and their labels.

Original: Incorporating mixup into existing training pipelines reduces to a few lines of code, and introduces little or no computational overhead.
Revised: Incorporating mixup into existing training pipelines reduces to a few lines of code, and introduces little or no computational overhead.

Original: Throughout an extensive evaluation, we have shown that mixup improves the generalization error of state-of-the-art models on ImageNet, CIFAR, speech, and tabular datasets.
Revised: Throughout an extensive evaluation, we have shown that mixup improves the generalization error of state-of-the-art models on ImageNet, CIFAR, speech, and tabular datasets.

Original: Furthermore, mixup helps to combat memorization of corrupt labels, sensitivity to adversarial examples, and instability in adversarial training.
Revised: Furthermore, mixup helps to combat memorization of corrupt labels, sensitivity to adversarial examples, and instability in adversarial training.

Original: In our experiments, the following trend is consistent: with increasingly large α , the training error on real data increases, while the generalization gap decreases.
Revised: In our experiments, the following trend is consistent: with increasingly large α , the training error on real data increases, while the generalization gap decreases.

Original: This sustains our hypothesis that mixup implicitly controls model complexity.
Revised: This sustains our hypothesis that mixup implicitly controls model complexity.

Original: However, we do not yet have a good theory for understanding the ‘sweet spot’ of this bias-variance trade-off.
Revised: However, we do not yet have a good theory for understanding the ‘sweet spot’ of this bias-variance trade-off.

Original: For example, in CIFAR-10 classiﬁcation we can get very low training error on real data even when α → ∞ (i e., training only on averages of pairs of real examples), whereas in ImageNet classiﬁcation, the training error on real data increases signiﬁcantly with α → ∞ .
Revised: For example, in CIFAR-10 classiﬁcation we can get very low training error on real data even when α → ∞ (i.e., training only on averages of pairs of real examples), whereas in ImageNet classiﬁcation, the training error on real data increases signiﬁcantly with α → ∞ .

Original: Based on our ImageNet and Google commands experiments with different model architectures, we conjecture that increasing the model capacity would make training error less sensitive to large α , hence giving mixup a more signiﬁcant advantage.
Revised: Based on our ImageNet and Google commands experiments with different model architectures, we conjecture that increasing the model capacity would make training error less sensitive to large α , hence giving mixup a more signiﬁcant advantage.

Original: mixup also opens up several possibilities for further exploration.
Revised: mixup also opens up several possibilities for further exploration.

Original: First, is it possible to make similar ideas work on other types of supervised learning problems, such as regression and structured prediction?
Revised: First, is it possible to make similar ideas work on other types of supervised learning problems, such as regression and structured prediction?

Original: While generalizing mixup to regression problems is straightforward, its application to structured prediction problems such as image segmentation remains less obvious.
Revised: While generalizing mixup to regression problems is straightforward, its application to structured prediction problems such as image segmentation remains less obvious.

Original: Second, can similar methods prove helpful beyond supervised learning?
Revised: Second, can similar methods prove helpful beyond supervised learning?

Original: The interpolation principle seems like a reasonable inductive bias which might also help in unsupervised, semi-supervised, and reinforcement learning.
Revised: The interpolation principle seems like a reasonable inductive bias which might also help in unsupervised, semi-supervised, and reinforcement learning.

Original: Can we extend mixup to feature-label extrapolation to guarantee a robust model behavior far away from the training data?
Revised: Can we extend mixup to feature-label extrapolation to guarantee a robust model behavior far away from the training data?

Original: Although our discussion of these directions is still speculative, we are excited about the possibilities mixup opens up, and hope that our observations will prove useful for future development. 
Revised: Although our discussion of these directions is still speculative, we are excited about the possibilities mixup opens up, and hope that our observations will prove useful for future development. A CKNOWLEDGEMENTS

Revised: We would like to thank Priya Goyal, Yossi Adi and the PyTorch team.

Revised: We also thank the Anonymous Review 2 for proposing the mixup + dropout experiments.