Over-Training with Mixup May Hurt Generalization
Abstract
Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple yet effective regularization technique to boost the performance of deep models trained with SGD.
In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a U-shaped generalization curve.
This behavior is further aggravated when the size of the original dataset is reduced.
To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data.
Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data.
Introduction
Mixup, a simple interpolation-based regularization technique, has empirically shown its effectiveness in improving the generalization and robustness of deep classification models (Zhang et al., 2018;
Guo et al., 2019a,b; Thulasidasan et al., 2019; Zhang et al., 2022b).
Unlike the vanilla empirical risk minimization (ERM) in which networks are trained using the original training set, Mixup trains the networks with synthetic examples.
These examples are created by linearly interpolating both the input features and the labels of instance pairs randomly sampled from the original training set.
Owning to Mixup’s simplicity and its effectiveness in boosting the accuracy and calibration of deep classification models, there has been a recent surge of interest attempting to better understand Mixup’s working mechanism, training characteristics, regularization potential, and possible limitations.
For example, Thulasidasan et al.
(2019) empirically show that Mixup helps improve the calibration of the trained networks.
Guo et al.
(2019a) identify the manifold intrusion issue in Mixup, where the synthetic data “intrude” the data manifolds of the real data.
Zhang et al.
(2021) theoretically explain the effectiveness via analyzing an upper bound of loss function used in Mixup.
Zhang et al.
(2022b) suggest that the calibration effect of Mixup is correlated with the capacity of the network.
In this work, we carry out an exploration along these research lines.
In this paper, we further investigate the generalization properties of Mixup training.
We first report a previously unobserved phenomenon in Mixup training.
Through extensive experiments on various benchmark datasets, we observe that over-training the networks with Mixup may result in significant degradation of the networks’ generalization performance.
As a result, along with training epochs, the generalization performance of the network measured by its testing error may exhibit a U-shaped curve.
Figure 1 shows such a curve obtained from over-training ResNet18 (He et al., 2016) with Mixup on Cifar10.
As can be seen from Figure 1, when training with Mixup for a long time, both ERM and Mixup keep decreasing their training loss, but the testing accuracy of the Mixup-trained ResNet18 gradually reduces, while that of the ERM-trained ResNet18 keeps decreasing. [Figure]
aiming to better understand the aforementioned behavior of
Mixup training show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data: under label noise, the early phase of training is primarily driven by the clean data pattern, which moves the model parameter closer to the correct solution.
But as training progresses, the effect of label noise accumulates through iterations and gradually over-weighs that of the clean pattern and dominates the training process.
In this phase, the model parameter gradually moves away from the correct solution until it is sufficient apart and approaches a location depending on the noise realization.  Related Work
Mixup Improves Generalization After the initial work of Zhang et al.
(2018), a series of Mixup variants have been proposed (Guo et al., 2019a; Verma et al., 2019; Yun et al., 2019; Guo, 2020; Kim et al., 2020; Greenewald et al., 2021; Han et al., 2022; Sohn et al., 2022). For example, AdaMixup (Guo et al., 2019a) trains an extra network to dynamically determine the interpolation coefficient parameter α . Manifold Mixup (Verma et al., 2019) performs the linear mixing on the hidden states of the neural networks.
Due to its regularization effectiveness, Mixup’s working mechanism and possible limitations are also being explored constantly.
For example, Zhang et al.
(2021) demonstrate that Mixup yields an upper bound of the Rademacher complexity of the class of functions that the network fits, which in turn bounds the generalization error of the network.
Thulasidasan et al.
(2019) show that Mixup helps to improve the calibration of the trained networks.
Zhang et al.
(2022b) theoretically demonstrate that the calibration effect of Mixup is correlated with the capacity of the network.
Guo et al.
(2019a) introduce the concept of manifold intrusion.
It refers to a phenomenon in
Mixup training where the synthetic data “intrude” the data manifolds of the real data.
Training on Random Labels, Epoch-Wise Double Descent and Robust Overfitting The thoughtprovoking work of Zhang et al.
(2017) highlights that neural networks are able to fit data with random labels.
After that, the generalization behavior on corrupted label dataset has been widely investigated (Arpit et al., 2017; Liu et al., 2020; Feng & Tu, 2021; Wang & Mao, 2022; Liu et al., 2022).
Specifically, Arpit et al.
(2017) observe that neural networks will learn the clean pattern first before fitting to data with random labels.
This is further explained by Arora et al.
(2019) where they demonstrate that under the overparameterization regime, the convergence of loss depends on the projections of labels on the eigenvectors of some Gram matrix and these projections are different for true labels and random labels.
As a parallel research line, an epoch-wise double descent behavior of testing loss of deep neural networks is observed in Nakkiran et al.
While the model-wise double descent is largely studied (Belkin et al., 2019; Hastie et al., 2022; Mei & Montanari, 2022; Ba et al., 2020), theoretical works studying the epoch-wise double descent are still very limited (Heckel & Yilmaz, 2021; Stephenson & Lee, 2021; Pezeshki et al., 2022).
Among the related works, we note that the work of Advani et al.
(2020) inspires our theoretical analysis of the U-sharped curve of Mixup training.
Moreover, robust overfitting is also another yet related research line, which is found by Rice et al.
In particular, robust overfitting is referred to a phenomenon in adversarial training that robust accuracy will first increase then decrease after a long training time.
Chen et al.
(2020) shows that robust overfitting is deemed to the early part of epoch-wise double descent due to the implicit label noise induced by adversarial training.
Since Mixup training has been connected to adversarial training or adversarial robustness in the previous works (Archambault et al., 2019;
Zhang et al., 2021), the work of Chen et al.
(2020) indeed motivates us to study the label noise induced by
Mixup training.
Preliminaries
Consider a C -class classification setting with input space X
= R d 0 and label space Y :
= { 1 , 2 , . .
, C } .
Denote by P ( Y ) the space of distributions over Y .
Let S = { ( x i , y i ) } ni =1 be a training set, where each y i ∈ Y may also be treated as a one-hot vector in P ( Y ) .
Suppose the model is parameterized by θ ∈ Θ , and let f θ : X → [0 , 1] C denote the predictive function associated with θ , which maps an input feature to a distribution in P ( Y ) .
For any pair ( x , y ) ∈
X × P ( Y ) , let ℓ ( θ, x , y ) denote the loss of the prediction f θ ( x ) with respect to the target label y .
The empirical risk of the predictor θ on S is defined as
[Equation]
In Mixup, instead of using the original training set S , a synthetic dataset (cid:101) S is generated by repeatedly sampling a random pair of examples (( x , y ) , ( x ′ , y ′ )) from S and creating a synthetic example (˜ x , ˜ y ) by
[Equation]
where λ ∈ [0 , 1] is drawn from some prescribed distribution, independently across all synthesized examples.
The optimization objective, or the “Mixup loss”, is then
[Equation]
Mixup training aims to find a θ ∗ that minimizes the above Mixup loss.
Most often, the interpolating parameter λ is drawn from a symmetric Beta distribution, Beta( α, α ) .
The default option is to take α = 1 .
In this case, the following can be proved.
Lemma 3.1.
Suppose that S is a balanced dataset, ℓ ( · ) is the cross-entropy loss, and { λ } is drawn i.i.d.
from Beta(1 , 1) (or the uniform distribution on [0 , 1] ).
Then
[Equation]
where the equality holds if and only if f θ (˜ x ) = ˜
y for each synthetic example (˜ x , ˜ y ) ∈ (cid:101) S .
The lower bound C −2 C in the lemma allows us to make sense of the Mixup loss during training.
For example, for 10-class classification tasks, the bound has value 0.45.
Then only when the Mixup loss approaches this value, we may conclude that the model parameter is near an optimum (assuming the model has sufficient capacity).
Empirical Observations
We conduct experiments using CIFAR10, CIFAR100 and SVHN.
For CIFAR10 and SVHN, we adopt both the original training set and a balanced subset of it containing 30% of the data.
For
CIFAR100, we only use the original training set, since downsampling CIFAR100 appears to result in high variances for the testing baselines.
We train ResNet networks on the three datasets using both ERM and Mixup, while adopting SGD with weight decay.
No data augmentation is used.
In each training trial, we train the network for in total a fixed number of epochs.
We record the minimal training loss achieved by the network during the training process, and we also record the network’s testing accuracy of the epoch at which the minimal training loss is achieved.
Additionally, we visualize the local loss landscape around the solution found by the network at the aforementioned training epoch. We gradually increase the total number of the training epochs in different trials of training so as to gradually over-train the network. We repeat each training trial for 10 times (using 10 different random seeds) and then average the recorded training losses and testing accuracies. For example, Figure 2a illustrates the results of training ResNet18 on 30% of the CIFAR10 data.
Each point represents the average of the recorded training losses in a training trial, and its label on the horizontal axis denotes the setting of total number of epochs of that trial.
The width of the shade beside each point reflects the deviation of the recorded results in the corresponding trial.
ResNet18 is used for the CIFAR10 and SVHN datasets.
Training is performed for up to 1600 epochs for CIFAR10, and the results are shown in Figure 2.
For both the 30% dataset and the full dataset, we see clearly that after some number of epochs (e.g, epoch 200 for the full dataset), the test accuracy of the Mixup-trained network starts decreasing and this trend continues.
This confirms that over-training with Mixup hurts the network’s generalization.
One would observe a U-shaped curve, as shown in
Figure 1 (right), if we were to plot test error and include results from earlier epochs.
Notably, this phenomenon is not observed in ERM.
Furthermore, we have also visualized the local loss landscape (where “loss” refers to the empirical risk defined using the real data) around the found solution in 2D following Li et al.
The plots are given in Appendix B.
[Figure]
m i n t r a i n i n g l o ss ( m i x u p = 1 )
[Figure]
Training is performed for up to 1000 epochs for SVHN, and the results are presented in Appendix B.
Mixup exhibits a similar phenomenon as it does on CIFAR10.
What differs notably is that overtraining with ERM on the original SVHN training set appears to also lead to worse test accuracy.
However, this does not occur on the 30% SVHN training set 2 .
ResNet34 is used for the more challenging task CIFAR100.
This choice allows Mixup training to drive its loss to lower values, closer to the lower bound given in Lemma 3.1.
Training is performed for up to 1600 epochs.
The results for CIFAR100 are illustrated in Appendix B. Additional results of over-training ResNet34 on CIFAR10 and SVHN are also given in Appendix B.
Theoretical Explanation: Mixup Induces Label Noise
We will use the capital letters X and Y to denote the random variables corresponding to the input feature and output label, while reserving the notations x and y to denote their realizations respectively.
In particular, we consider that each true label y is an element, i.e, a token, in Y , not a one-hot vector in
P ( Y ) .
Let P ( Y | X ) be the ground-truth conditional distribution of the label Y given input feature X .
For simplicity of notation, we will also express P ( Y | X ) using a vector-valued function f : X → R C , where f j ( x ) ≜ P ( Y =
j | X = x ) for each dimension j ∈ Y and input x .
Under this ground truth, the correct hard-assignment of label for x is arg max j ∈Y f j ( x ) .
For simplicity, we consider Mixup with a fixed λ ∈ [0 , 1] ; extension to random λ is straight-forward.
Let (cid:101) X and (cid:101)
Y be the random variables corresponding to the synthetic feature and synthetic label respectively.
Then (cid:101) X ≜
λX + (1 − λ ) X ′ .
Let P ( (cid:101) Y | (cid:101) X ) be the conditional distribution of the synthetic label conditioned on the synthetic feature, induced by Mixup, namely, P ( (cid:101) Y =
j | (cid:101) X ) =
λf j ( X ) + (1 − λ )
f j ( X ′ ) for each j .
Then for a synthetic feature (cid:101) X , there are two ways to assign it a hard label.
The first is based on the ground truth, assigning (cid:101)
Y ∗ h ≜ arg max j ∈Y f j ( (cid:101) X ) .
The second is based on the Mixup-induced conditional P ( (cid:101) Y | (cid:101) X ) , assigning (cid:101) Y h ≜ arg max j ∈Y P ( (cid:101) Y =
j | (cid:101) X ) .
When the two assignments disagree, or (cid:101) Y
h ̸
= (cid:101)
Y ∗ h , we say that the Mixup-assigned label (cid:101)
Y h is noisy.
Theorem 5.1.
Given a synthetic feature (cid:101) X
= λX + (1 − λ )
X ′ for λ ∈ [0 , 1] .
The probability of assigning a noisy label is lower bounded by
[Equation]
, where TV( · , · ) is total variation distance (see Appendix F).
Remark 5.1.
This lower bound hints that label noises induced by Mixup training depends on the distribution of original data P X , the convexity of f ( X ) and the value of λ .
Clearly, Mixup will not create any noisy label almost surely when f j is linear for each j .
Remark 5.2.
In practice, we often use the one-hot vector to denote the real data label, that is to say, we let max j ∈Y f j ( X )
= 1 and (cid:80) Cj =1 f j ( X )
= 1 .
Thus, the probability of assigning noisy label to a given synthetic data can be discussed in three situations: i) if (cid:101)
Y ∗ h / ∈ { Y, Y ′ } , where Y could be the same as Y ′ , then (cid:101)
Y is a noisy label with probability one; ii) if (cid:101)
Y ∗ h ∈
{ Y, Y ′ } where Y ̸ = Y ′ , then a noisy label is assigned with probability at least λ or 1 − λ ; iii) if (cid:101)
Y ∗ h
= Y =
Y ′ , then (cid:101)
Y ∗ h
= (cid:101) Y .
As shown in some previous works (Arpit et al., 2017; Arora et al., 2019), when neural networks are trained with a fraction of random labels, they will first learn the clean data and then will overfit to the data with noisy labels.
In the Mixup training case, we indeed create much more data than traditional ERM training (e.g., n 2 for a fixed λ ).
Thus, Mixup training will give higher testing performance in the first stage of learning (where neural networks learn the true pattern of data (Arpit et al., 2017)), and then performance will be impaired due to overfitting to noisy data.
In some cases, if (cid:101) Y ∗ h / ∈ { Y, Y ′ } happens with a high chance, a phenomenon known as “manifold intrusion” (Guo et al., 2019a), then synthetic dataset contains too many noisy labels, Mixup training may be unable to give better performance than ERM training.
The training dynamics on dataset with noisy labels is illustrated in Appendix C via a simple least squares regression problem, and the theoretical results are empirically verified in Appendix D. Also, we have further investigated the other properties of the aforementioned behavior of Mixup over-training, including that Mixup may generalize well without converging to any stationary points.
The details are in Appendix E.
Concluding Remarks
We empirically discovered a previously unobserved phenomenon in Mixup training: over training with Mixup may give rise to a U-shaped generalization curve.
We further theoretically showed that
Mixup training may introduce undesired data-dependent label noises to the synthesized data, and such noises facilitate the U-shaped generalization curve to occur.
That is, Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise introduced.
Our research here uncovers a unique generalization behavior of the effective Mixup regularizer, which paves ways for several promising directions that are worth further studying.
First, leveraging the
U-shaped generalization behavior identified to devise a training paradigm for Mixup to automatically optimize its regularization effect would be beneficial.
Second, unifying Mixup’s generalization behavior here with that being pointed out by previous works would be useful.
Finally, theoretically verifying that Mixup generalizes well without converging to any stationary points would help improve our understanding on Mixup’s generalization capabilities.
Acknowledgements
This project was supported in part by a National Research Council of Canada (NRC)
Collaborative
R&D grant (AI4D-CORE-07).
A Experimental Setups of Over-training
In each training trial, we train the network for in total a fixed number of epochs.
We record the minimal training loss achieved by the network during the training process, and we also record the network’s testing accuracy of the epoch at which the minimal training loss is achieved.
Additionally, we visualize the local loss landscape around the solution found by the network at the aforementioned training epoch.
We gradually increase the total number of the training epochs in different trials of training so as to gradually over-train the network.
We repeat each training trial for 10 times (using 10 different random seeds) and average the recorded training losses and testing accuracies.
For example, Figure 2a illustrates the results of training ResNet18 on 30% of CIFAR10 data.
Each point represents the average of the recorded training losses in a training trial, and its label on the horizontal axis represents the setting of total number of epochs of that trial.
The width of the shade beside each point reflects the deviation of the recorded results in the corresponding trial.
B Additional Experimental Results of Over-training
The local loss landscapes of Resnet18 trained on CIFAR10 (30% of the dataset and the full dataset) are visualized in Figure 3.
We found that over-training with Mixup tends to force the network to learn a solution located at the sharper local minima on the loss landscape, a phenomenon correlated with degraded generalization performance (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016).
[Figure]
The results of training ResNet18 on SVHN are given in Figure 4. [Figure] m i n t
[Figure]
a i n i n g l o ss ( m i
[Figure]
u p [Figure]
[Figure]
The results of training ResNet34 on CIFAR100 are illustrated in Figure 5.
A U-shaped testing loss curve (obtained from a single trial) is also observed in Figure 5c.
[Figure]
Besides CIFAR100, ResNet34 is also used for the CIFAR10 and the SVHN datasets.
Training is performed on both CIFAR10 and SVHN for in total 200 , 400 and 800 epochs respectively.
The results for CIFAR10 are shown in Figure 6.
For both the 30% dataset and the original dataset, Mixup exhibits a similar phenomenon as it does in training ResNet18 on CIFAR10.
The difference is that over-training ResNet34 with ERM makes the testing accuracy gradually increase on both the 30% dataset and the original dataset.
[Figure]
The results for SVHN with ResNet34 are shown in Figure 7.
These results are consistent with those of training ResNet18 on both 30% and 100% of the SVHN dataset
[Figure]
C Regression Setting With Random Feature Models
To further illustrate the training dynamics on dataset with noisy labels, we now consider a simple least squares regression problem.
Let Y
= R and let f : X → Y be the ground-truth labelling function.
Let ( (cid:101) X, (cid:101) Y ) be a synthetic pair obtained by mixing ( X, Y ) and ( X ′ , Y ′ ) .
Let (cid:101) Y ∗
=
f
( (cid:101) X ) and
Z ≜ (cid:101) Y − (cid:101) Y ∗ .
Then Z can be regarded as noise introduced by Mixup, which may be data-dependent.
For example, if f is strongly convex with some parameter ρ > 0 , then Z ≥ ρ 2 λ (1 − λ )
||
X − X
||
Given a synthesized training dataset (cid:101) S
= { ( (cid:101)
X i , (cid:101) Y i ) } mi =1 , consider a random feature model, θ T ϕ ( X ) , where ϕ : X → R d and θ ∈ R d .
We will train the model using gradient descent on the MSE loss
[Equation]
where (cid:101) Φ =
[ ϕ ( (cid:101) X 1 ) , ϕ ( (cid:101) X 2 ) , . . .
, ϕ ( (cid:101) X m )]
∈ R d × m and (cid:101) Y =
[ (cid:101) Y 1 , (cid:101)
Y 2 , . .
, (cid:101) Y m ] ∈ R m .
Notably, we will consider ϕ as fixed and only update θ .
For a fixed λ , Mixup can create m = n 2 synthesized examples.
Thus it is reasonable to assume m > d (e.g., under-parameterized regime) in Mixup training.
For example, ResNet-50 has less than 30 million parameters while the square of CIFAR10 training dataset size is larger than 200 million without using other data augmentation techniques.
Then the gradient flow is
[Equation]
) where η is the learning rate and (cid:101) Φ † = ( (cid:101) Φ (cid:101) Φ T ) − 1 (cid:101) Φ is the Moore–Penrose inverse of (cid:101) Φ T (only possible when m > d ).
Thus, we have the following important lemma.
Lemma C.1.
Let θ ∗ = (cid:101) Φ † (cid:101)
Y ∗ and θ noise =
(cid:101)
Φ † Z wherein Z
=
[ Z 1 , Z 2 , . . .
, Z m ] ∈ R m , the ODE of
Eq.
(2) has the following closed form solution
[Equation]
Remark C.1.
Lemma C.1 indicates that the dynamics of θ gives a U-shaped curve in each dimension, and the increasing behavior results from the second term that contains the noise Z .
More precisely, the first term in Eq.
(3) is monotonically decreasing and it dominates the dynamics of θ t in the early phase of learning.
Remarkably θ ∗ = (cid:101) Φ † (cid:101)
Y ∗ may be understood as the “clean pattern” of the training data.
Then we see that the model, in the early phase, is learning the “clean pattern”, which generalizes to the unseen data (i.e. ( X, Y ) ).
In the later training phase, the second term in
Eq.
(3) gradually dominates the trajectory of θ t , and the model learns the“noisy pattern”, namely θ noise = (cid:101) Φ † Z .
This then hurts generalization.
For a given synthesized dataset (cid:101) S , the expected population risk as a function of time step t is
[Equation]
The following theorem shows the dynamics of the population risk under mild assumptions.
Theorem C.
(Dynamic of Population Risk) .
Given a synthesized dataset (cid:101) S , assume θ 0 ∼
N (0 , ξ 2 I d ) , || ϕ ( X ) || 2 ≤ C 1 / 2 for some constant C 1 > 0 and | Z | ≤ √ C 2 for some constant
C 2 > 0 , then we have the following upper bound
[Equation]
where R ∗ =
E X,Y (cid:12)(cid:12)(cid:12)(cid:12)
Y − θ ∗ T ϕ
( X ) (cid:12)(cid:12)(cid:12)(cid:12) 2 2 , ζ = (cid:80) dk
=1 max { ξ 2 k + θ ∗ 2 k , C 2 µ k } and µ k is the k th eigenvalue of the matrix 1 m (cid:101) Φ (cid:101) Φ T .
Remark C.2.
The additive noise Z is usually assumed as a zero mean Gaussian in the literature of generalization dynamics analysis (Advani et al., 2020; Pezeshki et al., 2022; Heckel & Yilmaz, 2021).
We note that the boundness assumption of the data-dependent noise in the theorem is easily satisfied as long as the output of f is bounded, while there is no clue to assume Z is Gaussian.
Theorem C.1 indicates that the population risk in each dimension will first convexly decrease due to the first term, (cid:0)
ξ 2 k + θ ∗ 2 k (cid:1) e − 2 η (cid:101) µ k t , then it will concavely grow due to the existence of label noises (i.e. C 2 µ k (1 − e − ηµ k t ) 2 ).
Overall, the population risk will be endowed with the U-shaped behavior.
Notice that the quantity ηµ k plays a key role in the upper bound, the larger of ηµ k , the inflection point of the U-shaped curve comes earlier.
This explains why reducing the learning rate will make the population risk again decrease in a certain interval. That is, smaller learning rate forces the risk to jump from the second stage of the original U-shaped curve to the first stage of a new U-shaped curve with a latter inflection point.
C.1 Results of Mean Square Error Loss
We also conduct Mixup training experiments using mean square error (MSE) loss on CIFAR10 and
SVHN.
Figure 8 shows that the U-shaped behavior also holds for the MSE loss.
Note that the learning rate is divided by 10 at epoch 100 and 150 .
[Figure]
D Empirical Verification of the Theoretical Explanation
In this section [Figure] we present empirical evidences to validate our theoretical results in Section 5 and
Appendix C.
D.1 A Teacher-Student Toy Setup
[Figure]
To empirically verify our theoretical results discussed in
Appendix C, we invoke a simple teacher-student setting.
Consider the original data { X i } ni =1 are drawn i.i.d.
from a standard Gaussian N (0 , I d 0 ) , where the dimension of the input feature d 0 = 10 .
The teacher network is a twolayer neural networks with Tanh as the activation function.
The student network is also a two-layer neural network with Tanh , where we fix the parameters in the first layer and only train the second layer.
The output dimension of the first layer is 100 (i.e. d = 100 ) For the value of λ , we consider two cases: a fixed value with λ = 0 .
5 and random values drawn from a Beta(1 , 1) distribution at each epoch.
We choose n = 20 (so that n < d is the overparameterized regime and m ≥ n 2 > d is the underparameterized regime) and the learning rate is 0 .
We use full-batch gradient descent to train the student network with MSE.
Notice that here the “full-batch” indicates the batch size is equal to n , so that we can fairly compare the fixed λ and random λ .
As a comparison, we also present the result of ERM training in an overparameterized regime (i.e., n < d ).
All the testing loss dynamics are presented in Figure 9.
From Figure 9, we first note that
Mixup still outperforms ERM in this regression problem, but clearly, only Mixup training has a
U-shaped curve while the testing loss of ERM training converges to a constant value.
Furthermore, the testing loss of Mixup training is endowed with a U-shaped behavior for both the two λ scenarios, namely constant value of 0.5 and Beta(1 , 1) .
This indeed justifies that our analysis does not depend on specific λ distribution.
Moreover, Figure 9 indicates that when λ is fixed to 0 . 5 , the increasing stage of the U-shaped curve comes earlier than that of λ with Beta(1 , 1) .
This is again consistent with our theoretical results in Appendix C. That is, owning to the fact that λ with a constant value of 0.5 provides larger noise level for Mixup, the noise domination effect for Mixup training comes earlier.
D.2 Using Mixup Only in the Early Stage of Training
We here aim to empirically verify that Mixup can induce label noises as discussed in Section 5.
If Mixup training learns “clean patterns” in the early stage of the training and then overfits the “noisy patterns” in the latter stage of training, then we can stop using Mixup after a certain number of epochs.
Doing so, we can prevent the training from overfitting the noises induced by Mixup.
We present the results of utilizing such training schema for both CIFAR10 and SVHN in Figure
[Figure]
Results in Figure 10 clearly indicate that switching from Mixup to ERM at a proper time can successfully avoid the generalization degradation of Mixup training.
Figure 10 also suggests that it may not boost the model performance if we change Mixup to ERM before the clean samples created by Mixup have large effect.
In addition, if we change Mixup to ERM too late then the memorization of noisy data may already has negative effect on the generalization.
We note that our results here can be regarded as a complement to (Golatkar et al., 2019), where the authors show that regularization techniques only matter during the early phase of learning.
E Further Investigations
Impact of Data Size on U-shaped Curve Although the U-shaped behavior occurs for using both 100% and 30% of the original data of CIFAR10 and SVHN, we notice that smaller size datasets facilitate the U-shaped behavior to present.
We present such experimental results in Figure 11.
[Figure]
It may be tempting to think that we can apply the usual analysis of generalization dynamics in the existing literature (Liao & Couillet, 2018; Advani et al., 2020;
Stephenson & Lee, 2021), where they utilize some tools from random matrix theory.
For example one can analyze the distribution of the eigenvalues in Theorem C.1.
Specifically, if entries in Φ are independent identically distributed with zero mean, then the eigenvalues { µ k } dk =1 follow the Marchenko-Pasteur (MP) distribution (Marˇcenko & Pastur, 1967) in the limit d, m → ∞ with d/m = γ ∈ (0 , + ∞ ) , which is defined as
[Equation]
where γ ± = (1 ± γ ) 2 .
Note that the P MP are only non-zero when µ = 0 or µ ∈ [ γ − , γ + ] .
When γ is close to one, the probability of extremely small eigenvalues is immensely increased.
From
Theorem C.1, when µ k is small, the second term that contains noise will badly dominate the behavior of population risk and converge to a larger value.
Thus, letting d ≪ m will alleviate the domination of the noise term in Theorem C.1.
However, it is important to note that such analysis is not rigorous enough since columns in Φ are not independent (each two columns might come from the linear combination of the same two original instances).
To apply the similar analysis here, one may need to remove or relax the independence conditions for the MP distribution to hold, for example, by invoking some techniques developed in Bryson et al.
This is beyond the scope of this paper, and we would like to leave it for future study.
Gradient Norm in Mixup Training Does Not Vanish Normally, ERM training will obtain zero gradient norm at the end of training, which indicates that a local minimum is found by SGD.
However,
We observe that the gradient norm of Mixup training does not converge to zero, as shown in Figure 12.
[Figure]
Indeed, gradient norm in the Mixup training even increases until converging to a maximum value instead of converging to zero.
When models are trained on random labels, this increasing trend of gradient norm is also observed in the previous works (Feng & Tu, 2021; Wang & Mao, 2022).
Specifically, in Wang &
Mao (2022), such increasing behavior is interpreted as a sign that the training of SGD enters a “memorization regime”, and after the overparameterized neural networks memorize all the noisy labels, the gradient norm (or gradient dispersion in Wang & Mao (2022)) will decrease again until it converges to zero.
In Mixup training, since the size of synthetic dataset is usually larger than the number of parameters (i.e., m > d ), neural networks may not be able to memorize all the noisy labels in this case.
Notice that m is larger than n 2 in practice since λ is not fixed for all the pairs of original data.
Notably, although ERM training is able to find a local minimum in the first 130 epochs on CIFAR10,
Figure 1 indicates that Mixup training outperforms ERM in the first 400 epochs.
Similar observation also holds for SVHN.
This result indeed suggests that Mixup can generalize well without converging to any stationary points.
Notice that there is a close observation in the recent work of Zhang et al. (2022a), where they show that large-scale neural networks generalize well without vanishing of the gradient norm during training.
Additionally, by switching Mixup training to ERM training, as what we did in Figure 10, the gradient norm will instantly become zero (see Figure 13 in Appendix E.1).
This further justifies that the “clean patterns” are already learned by Mixup trained neural networks at the early stage of training, and the original data may not provide any useful gradient signal.
E.1 Gradient Norm Vanishes When Changing Mixup to ERM
In Figure 12, we know that the gradient norm of Mixup training will not vanish at the end of training and will even explode to a very high value.
In contrast, ERM will have zero gradient norm at the end.
Figure 13 shows that the gradient norm will instantly become zero.
This is because the “clean patterns” are already learned by Mixup trained neural networks and the original data will not provide any useful gradient signal.
This further justifies that the latter stage of Mixup training is only for memorizing noisy data. [Figure] [Equation]
[Figure]
F Omitted Definitions and Proofs
Definition F.1 (Total Variation) .
The total variation between two probability measures P and Q is
TV( P, Q ) ≜ sup E | P ( E ) − Q ( E ) | , where the supremum is over all measurable set E .
Lemma F.1 ((Levin & Peres, 2017, Proposition 4.2)) .
Let P and Q be two probability distributions on X .
If X is countable, then
[Equation]
Lemma F.2 (Coupling Inequality (Levin & Peres, 2017, Proposition 4.7)) .
Given two random variables X and Y with probability distributions P and Q , any coupling ˆ P of P , Q satisfies
[Equation]
F.1 Proof of Lemma 3.
Proof.
We first prove the closed-form of the cross-entropy loss’s lower bound.
For any two discrete distributions P and Q defined on the same probability space Y , the KL divergence of P from Q is defined as follows:
[Equation]
It is non-negative and it equals 0 if and only if P = Q .
Let’s denote the i th element in f θ ( x ) by f θ ( x ) i .
By adapting the definition of the cross-entropy loss, we have
[Equation]
where the equality holds if and only if f θ ( x ) =
y .
Here H ( y ) :
= (cid:80) Ci =
y i log y
i is the entropy of the discrete distribution y .
Particularly in ERM training, since y is one-hot, by definition its entropy is simply 0 .
Therefore the lower bound of the empirical risk is given as follows.
[Equation]
The equality holds if f θ ( x i ) =
y i is true for each i ∈ { 1 , 2 , · · · , n } .
We then prove the lower bound of the expectation of empirical Mixup loss.
From Eq.
(5), the lower bound of the general Mixup loss for a given λ is also given by:
[Equation]
if (˜ x , ˜ y ) is formulated via cross-class mixing.
Recall the definition of the Mixup loss,
[Equation]
we can exchange the computation of the expectation and the empirical average:
[Equation]
[Equation] Note that when α = 1 , Beta ( α, α ) is simply the uniform distribution in the interval [0 , 1] : U (0 , 1) .
Using the fact that the probability density of U (0 , 1) is constantly 1 in the interval [0 , 1] , the lower bound of
E λ ∼ Beta (1 , 1) ℓ ( θ, (˜ x , ˜ y )) where y ̸ = y ′ is given by:
[Equation]
[Equation]
Note that if the synthetic example is formulated via in-class mixing, the synthetic label is still one-hot, thus the lower bound of its general loss is 0 .
In a balanced C -class training set, with probability 1 C the in-class mixing occurs.
Therefore, the lower bound of the overall Mixup loss is given as follows,
[Equation]
The equality holds if f θ (˜ x ) =
˜
y is true for each synthetic example (˜ x , ˜ y ) ∈ (cid:101) S .
This completes the proof.
F.2 Proof of Theorem 5.
Proof.
By the coupling inequality i.e. Lemma F.2, we have
[Equation]
Since TV( P ( (cid:101)
Y h | (cid:101) X ) , P ( Y | X ))
=
TV( P ( (cid:101)
Y h | (cid:101) X ) , P ( (cid:101)
Y ∗ h | (cid:101) X )) , then the first inequality is straightforward.
For the second inequality, by Lemma F.1, we have
[Equation]
This completes the proof
[Equation]
F.3 Proof of Lemma C.
Proof.
The proof here is trivial.
The ordinary differential equation of Eq.
(2) (Newton’s law of cooling) has the closed form solution
[Equation]
Recall that (cid:101) Y =
[Equation]
Y
[Equation]
+ Z
[Equation]
which concludes the proof
[Equation]
F.4 Proof of Theorem C.
Proof.
We first notice that
[Equation]
(13) where the first inequality is by the Cauchy–Schwarz inequality and the second inequality is by the assumption.
Recall Eq.
[Equation]
By eigen-decomposition we have
[Equation]
where { v k } dk =1 are orthonormal eigenvectors and
{ µ k } dk =1 are corresponding eigenvectors.
Then, for each dimension k ,
[Equation]
Taking expectation over θ 0 for both side [Equation] we have
[Equation]
Notich that the RHS in Eq.
14 first monotonically decreases and then monotonically increases, so the maximum value of RHS is achieved either at t = 0 or t → ∞ .
That is,
[Equation]
Plugging Eq.
14 and Eq. 15 into Eq.
13, we have
[Equation]
where ζ = (cid:80) dk =1 max { ξ 2 k + θ ∗ 2 k , C 2 µ k } .
This concludes the proof.