T OWARDS U NDERSTANDING THE D ATA D EPENDENCY OF M IXUP - STYLE T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
In the Mixup training paradigm, a model is trained using convex combinations of data points and their associated labels.
Despite seeing very few true data points during training, models trained using Mixup seem to still minimize the original empirical risk and exhibit better generalization and robustness on various tasks when compared to standard training.
In this paper, we investigate how these beneﬁts of Mixup training rely on properties of the data in the context of classiﬁcation.
For minimizing the original empirical risk, we compute a closed form for the
Mixup-optimal classiﬁcation, which allows us to construct a simple dataset on which minimizing the Mixup loss can provably lead to learning a classiﬁer that does not minimize the empirical loss on the data.
On the other hand, we also give sufﬁcient conditions for Mixup training to also minimize the original empirical risk.
For generalization, we characterize the margin of a Mixup classiﬁer, and use this to understand why the decision boundary of a Mixup classiﬁer can adapt better to the full structure of the training data when compared to standard training.
In contrast, we also show that, for a large class of linear models and linearly separable datasets, Mixup training leads to learning the same classiﬁer as standard training.
I NTRODUCTION
Mixup (Zhang et al., 2018) is a modiﬁcation to the standard supervised learning setup which involves training on convex combinations of pairs of data points and their labels instead of the original data itself.
In the original paper, Zhang et al.
(2018) demonstrated that training deep neural networks using Mixup leads to better generalization performance, as well as greater robustness to adversarial attacks and label noise on image classiﬁcation tasks.
The empirical advantages of Mixup training have been afﬁrmed by several follow-up works (He et al., 2019; Thulasidasan et al., 2019; Lamb et al., 2019; Arazo et al., 2019; Guo, 2020).
The idea of Mixup has also been extended beyond the supervised learning setting, and been applied to semi-supervised learning (Berthelot et al., 2019;
Sohn et al., 2020), contrastive learning (Verma et al., 2021; Lee et al., 2020), privacy-preserving learning (Huang et al., 2021), and learning with fairness constraints (Chuang & Mroueh, 2021).
However, from a theoretical perspective, Mixup training is still mysterious even in the basic multiclass classﬁcation setting – why should the output of a linear mixture of two training samples be the same linear mixture of their labels, especially when considering highly nonlinear models?
Despite several recent theoretical results (Guo et al., 2019; Carratino et al., 2020; Zhang et al., 2020; 2021), there is still not a complete understanding of why Mixup training actually works in practice.
In this paper, we try to understand why Mixup works by ﬁrst understanding when Mixup works: in particular, how the properties of Mixup training rely on the structure of the training data.
We consider two properties for classiﬁers trained with Mixup.
First, even though Mixup training does not observe many original data points during training, it usually can still correctly classify all of the original data points (empirical risk minimization (ERM)).
Second, the aforementioned empirical works have shown how classiﬁers trained with Mixup often have better adversarial robustness and generalization than standard training.
In this work, we show that both of these properties can rely heavily on the data used for training, and that they need not hold in general.
Main Contributions and Related Work.
The idea that Mixup can potentially fail to minimize the original risk is not new; Guo et al.
(2019) provide examples of how Mixup labels can conﬂict with actual data point labels.
However, their theoretical results do not characterize the data and model conditions under which this failure can provably happen when minimizing the Mixup loss.
In
Section 2 of this work, we provide a concrete classiﬁcation dataset on which continuous approximateminimizers of the Mixup loss can fail to minimize the empirical risk.
We also provide sufﬁcient conditions for Mixup to minimize the original risk, and show that these conditions hold approximately on standard image classiﬁcation benchmarks.
With regards to generalization and robustness, the parallel works of Carratino et al.
(2020) and Zhang et al.
(2020) showed that Mixup training can be viewed as minimizing the empirical loss along with a data-dependent regularization term.
Zhang et al.
(2020) further relate this term to the adversarial robustness and Rademacher complexity of certain function classes learned with Mixup.
In Section 3, we take an alternative approach to understanding generalization and robustness by analyzing the margin of Mixup classiﬁers.
Our perspective can be viewed as complementary to that of the aforementioned works, as we directly consider the properties exhibited by a Mixup-optimal classiﬁer instead of considering what properties are encouraged by the regularization effects of the Mixup loss.
In addition to our margin analysis, we also show that for the common setting of linear models trained on high-dimensional Gaussian features both Mixup (for a large class of mixing distributions) and
ERM with gradient descent learn the same classiﬁer with high probability.
Finally, we note the related works that are beyond the scope of our paper; namely the many Mixup-like training procedures such as Manifold Mixup (Verma et al., 2019), Cut Mix (Yun et al., 2019), Puzzle
Mix (Kim et al., 2020), and Co-Mixup (Kim et al., 2021).
M IXUP AND E MPIRICAL R ISK M INIMIZATION
The goal of this section is to understand when Mixup training can also minimize the empirical risk.
Our main technique for doing so is to derive a closed-form for the Mixup-optimal classiﬁer over a sufﬁciently powerful function class, which we do in Section 2.2 after introducing the basic setup in
Section 2.1.
We use this closed form to motivate a concrete example on which Mixup training does not minimize the empirical risk in Section 2.3, and show under mild nondegeneracy conditions that
Mixup will minimize the emprical risk in Section 2.4.
2. S ETUP
We consider the problem of k -class classiﬁcation where the classes 1 , ..., k correspond to compact disjoint sets X 1 , ..., X k ⊂ R n with an associated probability measure P X supported on X
=
(cid:83) ki
=1 X i .
We use C to denote the set of all functions g : R n → [0 , 1] k satisfying the property that (cid:80) ki
=1 g i ( x ) = 1 for all x (where g i represents the i -th coordinate function of g ).
We refer to a function g ∈ C as a classiﬁer , and say that g classiﬁes x as class j if j = argmax i g i ( x ) .
The cross-entropy loss associated with such a classiﬁer g is then:
[Equation]
The goal of standard training is to learn a classiﬁer h ∈ argmin g ∈C J ( g, P X ) .
Any such classiﬁer h will necessarily satisfy h i ( x )
= 1 on X i since the X i are disjoint.
Mixup.
In the Mixup version of our setup, we are interested in minimizing the cross-entropy of convex combinations of the original data and their classes.
These convex combinations are determined according to a probability measure P f whose support is [0 , 1] , and we assume this measure has a density f .
For two points s, t ∈ (cid:83) ki =1 X i , we let z st ( λ ) =
λs + (1 − λ ) t (and use z st when λ is understood) and deﬁne the Mixup cross-entropy on s, t with respect to a classiﬁer g as:
[Equation]
Having deﬁned (cid:96) mix as above
[Equation]
we may write the component of the full Mixup cross-entropy loss corresponding to mixing points from classes i and j as:
[Equation]
The ﬁnal Mixup cross-entropy loss is then the sum of J i,jmix over all i, j ∈ { 1 , ..., k } (corresponding to all possible mixings between classes, including themselves):
[Equation]
Where the coefﬁcient of 2 in front of the second term comes from the fact that J i,jmix =
J j,imix from
Fubini’s Theorem (we consider only classiﬁers g for which the (cid:96) mix terms are deﬁned and integrable).
Relation to Prior Work.
We have opted for a more general deﬁnition of the Mixup loss (at least when constrained to multi-class classiﬁcation) than prior works.
This is not generality for generality’s sake, but rather because many of our results apply to any mixing distribution supported on [0 , 1] .
One obtains the original Mixup formulation of Zhang et al.
(2018) for multi-class classiﬁcation on a ﬁnite dataset by taking the X i to be ﬁnite sets, and choosing P X to be the normalized counting measure (corresponding to a discrete uniform distribution).
Additionally, P f is chosen to have density
Beta( α, α ) , where α is a hyperparameter.
2. M IXUP - OPTIMAL C LASSIFIER
Given our setup, we now wish to characterize the behavior of a Mixup-optimal classiﬁer at a point x ∈ R n .
However, if the optimization of J mix is considered over the class of functions C , this is intractable (to the best of our knowledge) due to the lack of regularity conditions imposed on functions in C .
We thus wish to constrain the optimization of J mix to a class of functions that is sufﬁciently powerful (so as to include almost all practical settings) while still allowing for local analysis.
To do so, we will need the following deﬁnitions, which will also be referenced throughout the results in this section and the next:
[Equation]
has positive measure for every (cid:15) >
[Equation]
The set A i,jx,(cid:15) represents all points in X i × X j that have lines between them intersecting an (cid:15) - neighborhood of x , while the set
A i,j x,(cid:15),δ represents the restriction of A i,jx,(cid:15) to only those points whose connecting line segments intersect an (cid:15) -neighborhood of x with λ values bounded by 1 − δ (used in Section 3).
The set X mix corresponds to all points for which every neighborhood factors into J mix .
The ξ i,jx,(cid:15) term represents the measure of the set
A i,jx,(cid:15) while ξ i,jx,(cid:15),λ represents the expectation of λ over the same set.
To provide better intuition for these deﬁnitions, we provide visualizations in Section B of the appendix.
We can now deﬁne the subset of C to which we will constrain our optimization of J mix .
Deﬁnition 2.1.
Let C ∗ to be the subset of C for which every h ∈ C ∗ satisﬁes
h ( x ) = lim (cid:15) →
0 argmin θ ∈ [0 , 1] k J mix ( θ ) | B (cid:15) ( x ) for all x ∈ X mix when the limit exists.
Here J mix ( θ ) | B (cid:15) ( x ) represents the Mixup loss for a constant function with value θ with the restriction of each term in
J mix to the set
A i,jx,(cid:15) .
We immediately justify this deﬁnition with the following proposition.
Proposition 2.2.
Any function h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) satisﬁes J mix ( h ) ≤ J mix ( g ) for any continuous g ∈ C .
Proof Sketch.
We can argue directly from deﬁnitions by considering points in X mix for which h and g differ.
Proposition 2.2 demonstrates that optimizing over C ∗ is at least as good as optimizing over the subset of C consisting of continuous functions, so we cover most cases of practical interest (i.e. optimizing deep neural networks).
As such, the term “Mixup-optimal” is intended to mean optimal with respect to C ∗ throughout the rest of the paper.
We may now characterize the classiﬁcation of a Mixup-optimal classiﬁer on X mix .
Lemma 2.3.
For any point x ∈ X mix and (cid:15) > 0 , there exists a continuous function h (cid:15) satisfying:
[Equation]
With the property that lim (cid:15) → 0 h (cid:15) ( x ) = h ( x ) for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) when the limit exists.
Proof Sketch.
We set h (cid:15) = argmin θ ∈ [0 , 1] k J mix ( θ ) | B (cid:15) ( x ) and show that this is well-deﬁned, contin- uous, and has the above form using the strict convexity of the minimization problem.
Remark 2.4.
For the important case of ﬁnite datasets, it will be shown that the limit above always exists as part of the proof of Theorem 3.2.
Although the expression for h i(cid:15) looks complicated, it just represents the expected location of the point x on all lines between class i and other classes, normalized by the sum of the expected locations for all classes.
Importantly, we note that while h (cid:15) as deﬁned in Lemma 2.3 is continuous for every (cid:15) > 0 , its pointwise limit h need not be, which we demonstrate below.
Proposition 2.5.
Let X
= { (0 , 1) , (0 , − 1) } and let X
= { (1 , 0) , ( − 1 , 0) } , with P X being discrete uniform over X 1 ∪ X 2 and P f being continuous uniform over [0 , 1] .
Then the Mixup-optimal classiﬁer h is discontinuous at (0 , 0) .
Proof Sketch.
One may explicitly compute for x = (0 , 0)
that h 1 ( x ) =
h 2 ( x ) = 12 .
Proposition 2.5 illustrates our ﬁrst signiﬁcant difference between Mixup training and standard training: there always exists a minimizer of the empirical cross-entropy J that can be extended to a continuous function (since a minimizer is constant on the class supports and not constrained elsewhere), whereas depending on the data the minimizer of J mix can be discontinuous.
2. A M IXUP F AILURE C ASE
With that in mind, several model classes popular in practical applications consist of continuous functions.
For example, neural networks with ReLU activations are continuous, and several works have noted that they are Lipschitz continuous with shallow networks having approximately small
Lipschitz constant (Scaman & Virmaux, 2019; Fazlyab et al., 2019; Latorre et al., 2020).
Given the regularity of such models, we are motivated to consider the continuous approximations h (cid:15) in Lemma 2.3 and see if it is possible to construct a dataset on which h (cid:15) (for a ﬁxed (cid:15) ) can fail to classify the original points correctly.
We thus consider the following dataset:
Deﬁnition 2.6.
[3-Point Alternating Line]
We deﬁne X 23 to be the binary classiﬁcation dataset consisting of the points { 0 , 1 , 2 } classiﬁed as { 1 , 2 , 1 } .
In our setup, this corresponds to X 1 = { 0 , 2 } and X 2 = { 1 } with P X = 13 1 { 0 , 1 , 2 } .
Intuitively, the reason why Mixup can fail on X 23 is that, for choices of P f that concentrate about 12 , we will have by Lemma 2.3 that the Mixup-optimal classiﬁcation in a neighborhood of point 1 should skew towards class 1 instead of class 2 due to the sandwiching of point 1 between points 0 and 2.
The canonical choice of P f corresponding to a mixing density of Beta( α, α ) is one such choice:
Theorem 2.7.
Let P f have associated density
Beta ( α, α ) .
Then for any classiﬁer h (cid:15) on X 23 (as deﬁned in Lemma 2.3), we may choose α such that h (cid:15) does not achieve 0 classiﬁcation error on X 23 .
Proof Sketch.
For any (cid:15) > 0 , we can bound the ξ terms in Equation 1 using the fact that Beta( α, α ) is strictly subgaussian (Marchal & Arbel, 2017), and then choose α appropriately.
[Figure]
Experiments.
The result of Theorem 2.7 leads us to believe that the Mixup training of a continuous model should fail on X 23 for appropriately chosen α .
To verify that the theory predicts the experiments, we train a two-layer feedforward neural network with 512 hidden units and ReLU activations on Xwith and without Mixup.
The implementation of Mixup training does not differ from the theoretical setup; we uniformly sample pairs of data points and train on their mixtures.
Our implementation uses
PyTorch (Paszke et al., 2019) and is based heavily on the open source implementation of Manifold
Mixup (Verma et al., 2019) by Shivam Saboo.
Results for training using (full-batch) Adam (Kingma & Ba, 2015) with the suggested (and common) hyperparameters of β 1 = 0 .
9 , β 2 = 0 .
999 and a learning rate of 0 .
001 are shown in Figure 1.
The class 1 probabilities for each point in the dataset outputted by the learned Mixup classiﬁers from Figure 1 are shown in Table 1 below:
[Table]
We see from Figure 1 and Table 1 that Mixup training fails to correctly classify the points in X 23 for α = 32 , and this misclassiﬁcation becomes more exacerbated as we increase α .
The choice of α for which misclassiﬁcations begin to happen is largely superﬁcial; we show in Section D of the Appendix that it is straightforward to construct datasets in the style of X 23 for which Mixup training will fail even for the very mild choice of α = 1 .
We focus on the case of X 23 here to simplify the theory.
The key takeaway is that, for datasets that exhibit (approximately) collinear structure amongst points, it is possible for inappropriately chosen mixing distributions to cause Mixup training to fail to minimize the original empirical risk.
2. S UFFICIENT C ONDITIONS FOR M INIMIZING THE O RIGINAL R ISK
The natural follow-up question to the results of the previous subsection is: under what conditions on the data can this failure case be avoided?
In other words, when can the Mixup-optimal classiﬁer classify the original data points correctly while being continuous at those points?
Prior to answering that question, we ﬁrst point out that if discontinuous functions are allowed, then
Mixup training always minimizes the original risk on ﬁnite datasets:
Proposition 2.8.
Consider k -class classiﬁcation where the supports X 1 , ...,
X k are ﬁnite and P X corresponds to the discrete uniform distribution.
Then for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) , we have that h
i ( x ) = 1 on X i .
Proof Sketch.
Only the ξ i,ix,(cid:15) term doesn’t vanish in h i(cid:15) ( x ) as (cid:15) → 0 , as the mixing distribution is continuous and cannot assign positive measure to x alone when mixing two points that are not x .
Note that Proposition 2.8 holds for any continuous mixing distribution P f supported on [0 , 1] - we just need a rich enough model class.
In order to obtain the result of Proposition 2.8 with the added restriction of continuity of h on each of the X i , we need to make further assumptions.
Namely, we
[Figure]
need to avoid the collinearity of different class points that occurred in the previous subsection, and we do so with the following assumption which is a function of a class i and a point x :
Assumption 2.9.
For a class i and a point x , there exists an (cid:15) > 0 such that A j,qx,(cid:15) (cid:48) has measure zero for all (cid:15) (cid:48) ≤ (cid:15) when both j (cid:54) =
i and q (cid:54) =
i .
A visualization of Assumption 2.9 is provided in Section B of the appendix.
With this assumption in hand, we obtain the following result as a corollary of Theorem 3.2 which is proved in the next section:
Theorem 2.10.
We consider the same setting as Proposition 2.8 and further suppose that Assumption 2.9 is satisﬁed by all points in (cid:83) ki
=1 X
i .
Then for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) , we have that h
i ( x ) = 1 on X i and that h is continuous on (cid:83) ki =1 X i .
Application of Sufﬁcient Conditions.
The practical take-away of Theorem 2.10 is that if a dataset does not exhibit approximate collinearity between points of different classes, then Mixup training should achieve near-identical training error on the original training data when compared to ERM.
We validate this by training ResNet-18 (He et al., 2015) (using the popular implementation of Kuang Liu) on MNIST (LeCun, 1998), CIFAR-10, and CIFAR-100 (Krizhevsky, 2009) with and without Mixup for 50 epochs with a batch size of 128 and otherwise identical settings to the previous subsection. For Mixup, we consider mixing using Beta( α, α ) for α = 1 , 32 , 128 , and 1024 to cover a wide band of mixing distributions. Our experimental results for the “worst case” of α = 1024 (the other choices are strictly closer to ERM in training accuracy) are shown in Figure 2, while the other experiments can be found in Section D of the Appendix.
We now check that the theory can predict the results of our experiments by verifying Assumption 2. approximately (exact veriﬁcation is too expensive).
We sample one epoch’s worth of Mixup points (to simulate training) from a downsampled version of each train dataset, and then compute the minimum distances between each Mixup point and points from classes other than the two mixed classes.
The minimum over these distances corresponds to an estimate of (cid:15) in Assumption 2.9.
We compute the distances for both training and test data, to see whether good training but poor test performance can be attributed to test data conﬂicting with mixed training points.
Results are shown below in Table 2.
[Table]
To interpret the estimated (cid:15) values in Table 2, we note that unless (cid:15) (cid:28) 1 /L (where L is the Lipschitz constant of the model being considered), a Mixup point cannot conﬂict with an original point (since the function has enough ﬂexibility to ﬁt both).
Due to the estimated large Lipschitz constants of deep networks (Scaman & Virmaux, 2019), our (cid:15) values certainly do not fall in this regime, explaining how the near-identical performance to ERM is possible on the original datasets.
We remark that our results challenge an implication of Guo et al.
(2019), which was that training/test performance on the above benchmarks degrades with high values of α due to conﬂicts between original points and Mixup points.
2. T HE R ATE OF E MPIRICAL R ISK M INIMIZATION U SING M IXUP
Another striking aspect of the experiments in Figure 2 is that Mixup training minimizes the original empirical risk at a very similar rate to that of direct empirical risk minimization.
A priori, there is no reason to expect that Mixup should be able to do this - a simple calculation shows that Mixup training only sees one true data point per epoch in expectation (each pair of points is sampled with probability 1 m 2 and there are m true point pairs and m pairs seen per epoch, where m is the dataset size).
The experimental results are even more surprising given that we are training using α = 1024 , which essentially corresponds to training using the midpoints of the original data points.
This seems to imply that it is possible to recover the classiﬁcations of the original data points from the midpoints alone (not including the midpoint of a point and itself), and similar phenomena has indeed been observed in recent empirical work (Guo, 2021).
We make this rigorous with the following result:
Theorem 2.11.
Suppose { x 1 , ..., x m } with m ≥ 6 are sampled from (cid:83)
ki =1 X i according to P X , and that P X has a density.
Then with probability 1, we can uniquely determine the points { x 1 , ..., x m } given only the (cid:0) m 2 (cid:1) midpoints { x i,j } 1 ≤ i<j ≤ m .
Proof Sketch.
The idea is to represent the problem as a linear system, and show using rank arguments that the sets of m points that cannot be uniquely determined are a measure zero set.
Theorem 2.11 shows, in an information-theoretic sense, that it is possible to obtain the original data points (and therefore also their labels) from only their midpoints.
While this gives more theoretical backing as to why it is possible for Mixup training using Beta(1024 , 1024) to recover the original data point classiﬁcations with very low error, it does not explain why this actually happens in practice at the rate that it does.
A full theoretical analysis of this phenomenon would necessarily require analyzing the training dynamics of neural networks (or another model of choice) when trained only on midpoints of the original data, which is outside the intended scope of this work.
That being said, we hope that such analysis will be a fruitful line of investigation for future work.
G ENERALIZATION P ROPERTIES OF M IXUP C LASSIFIERS
Having discussed how Mixup training differs from standard empirical risk minimization with regards to the original training data, we now consider how a learned Mixup classiﬁer can differ from one learned through empirical risk minimization on unseen test data.
To do so, we analyze the per-class margin of Mixup classiﬁers, i.e. the distance one can move from a class support
X i while still being classiﬁed as class i .
3. T HE M ARGIN OF M IXUP C LASSIFIERS
Intuitively, if a point x falls only on line segments between X i and some other classes X j , ... , and if x always falls closer to X i than the other classes, we can expect x to be classiﬁed according to class i by the Mixup-optimal classiﬁer due to Lemma 2.3.
To make this rigorous, we introduce another assumption in the same vein as Assumption 2.9:
Assumption 3.1.
For a class i and a point x , suppose there exists an (cid:15) > 0 and a 0 < δ < min
j d ( X i ,X j )such that A i,jx,(cid:15) (cid:48) ,δ has measure zero for all (cid:15) (cid:48) ≤ (cid:15) .
Here the measure zero condition on the sets A i,jx,(cid:15) (cid:48) ,δ is codifying the aforementioned idea that the point x falls closer to X i than any other class on every line segment that intersects it.
A visualization of Assumption 3.1 is provided in Section B of the Appendix.
Now for points for which Assumptions 2.9 and 3.1 hold, we can prove:
Theorem 3.2.
Consider k -class classiﬁcation where the supports X 1 , ...,
X k are ﬁnite and P X corresponds to the discrete uniform distribution.
If a point x satisﬁes Assumptions 2.9 and 3.1 with respect to a class i , then for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) , we have that h classiﬁes x as class i and that h is continuous at x .
Proof Sketch.
The limit can be shown to exist using the Lebesgue differentiation theorem, and we can bound the limit below since the A i,jx,(cid:15) (cid:48) ,δ have measure zero.
Any point x ∈ X i is easily seen to satisfy Assumption 3.1 with respect to class i , and hence we get Theorem 2.10 as a corollary of Theorem 3.2 as mentioned in Section 2.
To use Theorem 3.2 to understand generalization, we make the observation that a point x can satisfy Assumptions 2.9 and 3.1 while being a distance of up to min j d ( X i ,X j )from some class i .
This distance can be signiﬁcantly farther than, for example, the optimal linear separator in a linearly separable dataset.
Experiments.
To illustrate that Mixup can lead to more separation between points than a linear decision boundary, we consider the two moons dataset (Buitinck et al., 2013), which consists of two classes of points supported on semicircles with added Gaussian noise.
Our motivation for doing so comes from the work of Pezeshki et al.
(2020), in which it was noted that neural network models trained on a separated version of the two moons dataset essentially learned a linear separator while ignoring the curvature of the class supports.
While Pezeshki et al.
(2020) introduced an explicit regularizer to encourage a nonlinear decision boundary, we expect due to Theorem 3.2 that Mixup training will achieve a similar result without any additional modiﬁcations.
To verify this empirically, we train a two-layer neural network with 500 hidden units with and without
Mixup, to have a 1-to-1 comparison with the setting of Pezeshki et al.
We use α = 1 and α = 1024 for Mixup to capture a wide band of mixing densities.
The version of the two moons dataset we use is also identical to that of the one used in the experiments of Pezeshki et al.
and we are grateful to the authors for releasing their code under the MIT license.
We do full-batch training with all other training, implementation, and compute details remaining the same as the previous section.
Results are shown in Figure 3.
Our results afﬁrm the observations of Pezeshki et al.
and previous work (des Combes et al., 2018) that neural network training dynamics may ignore salient features of the dataset; in this case the “Base Model” learns to differentiate the two classes essentially based on the x -coordinate alone.
On the other hand, the models trained using Mixup have highly nonlinear decision boundaries.
Further experiments for different class separations and values of α are included in Section F of the Appendix.
3. W HEN M IXUP T RAINING L EARNS THE S AME C LASSIFIER
The experiments and theory of the previous sections have shown how a Mixup classiﬁer can differ signiﬁcantly from one learned through standard training.
In this subsection, we now consider the opposing question - when is the Mixup classiﬁer the same as the one learned through standard training?
Prior work (Archambault et al., 2019) has considered when Mixup training coincides with certain adversarial training, and our results complement this line of work.
The motivation for our results comes from the fact that a practitioner need not spend compute on Mixup training in addition to standard training in settings where the performance will be provably the same.
We consider the case of binary classiﬁcation using a linear model θ (cid:62) x on high-dimensional Gaussian data, which is a setting that arises naturally when training using Gaussian kernels.
Speciﬁcally, we consider the dataset X to consist of n points in R d distributed according to N (0 , I d ) with d > n (to be made more precise shortly).
We also consider the mixing distribution to be any symmetric distribution supported on [0 , 1] (thereby including as a special case
Beta( α, α ) ).
We let the labels of points in X be ± 1 (so that the sign of θ (cid:62) x is the classiﬁcation), and use X 1 and X − 1 to denote the individual class points.
We will show that in this setting, the optimal Mixup classiﬁer is the same (up to rescaling of θ ) as the ERM classiﬁer learned using gradient descent with high probability.
To do so we need some additional deﬁnitions.
Deﬁnition 3.3.
We say ˆ θ is an interpolating solution, if there exists k > 0 such that
[Equation]
Deﬁnition 3.4.
The maximum margin solution ˜ θ is deﬁned through:
[Equation]
When the maximum margin solution coincides with an interpolating solution for the dataset X (i.e. all the points are support vectors), we have that Mixup training leads to learning the max margin solution (up to rescaling).
Theorem 3.5.
If the maximum margin solution for X is also an interpolating solution for X , then any θ that lies in the span of X and minimizes the Mixup loss J mix for a symmetric mixing distribution P f is a rescaling of the maximum margin solution.
Proof Sketch.
It can be shown that θ is an interpolating solution using a combination of the strict convexity of J mix as a function of θ and the symmetry of the mixing distribution.
Remark 3.6.
For every θ , we can decompose it as θ = θ X + θ X ⊥ where θ X is the projection of θ onto the subspace spanned by X .
By deﬁnition we have that θ X ⊥ is orthogonal to all possible mixings of points in X .
Hence, θ X ⊥ does not affect the Mixup loss or the interpolating property, so for simplicity we may just assume θ lies in the span of X .
To characterize the conditions on X under which the maximum margin solution interpolates the data, we use a key result of Muthukumar et al.
(2020), restated below.
Note that Muthukumar et al.
(2020) actually provide more settings in their paper, but we constrain ourselves to the one stated below for simplicity.
Lemma 3.7.
[Theorem 1 in Muthukumar et al.
(2020), Rephrased]
Assuming d > 10 n ln n + n − 1 , then with probability at least 1 − 2 /n, the maximum margin solution for X is also an interpolating solution.
To tie the optimal Mixup classiﬁer back to the classiﬁer learned through standard training, we appeal to the fact that minimizing the empirical cross-entropy of a linear model using gradient descent leads to learning the maximum margin solution on linearly separable data (Soudry et al., 2018;
Ji &
Telgarsky, 2018).
From this we obtain the desired result of this subsection:
Corollary 3.8.
Under the same conditions as Lemma 3.7, the optimal Mixup classiﬁer has the same direction as the classiﬁer learned through minimizing the empirical cross-entropy using gradient descent with high probability.
C ONCLUSION
The main contribution of our work has been to provide a theoretical framework for analyzing how
Mixup training can differ from empirical risk minimization.
Our results characterize a practical failure case of Mixup, and also identify conditions under which Mixup can provably minimize the original risk.
They also show in the sense of margin why the generalization of Mixup classiﬁers can be superior to those learned through empirical risk minimization, while again identifying model classes and datasets for which the generalization of a Mixup classiﬁer is no different (with high probability).
We also emphasize that the generality of our theoretical framework allows most of our results to hold for any continuous mixing distribution.
Our hope is that the tools developed in this work will see applications in future works concerned with analyzing the relationship between beneﬁts obtained from Mixup training and properties of the training data.
E THICS S TATEMENT
We do not anticipate any direct misuses of this work due to its theoretical nature.
That being said, the failure case of Mixup discussed in Section 2 could serve as a way for an adversary to potentially exploit a model trained using Mixup to classify data incorrectly.
However, as this requires knowledge of the mixing distribution and other hyperparameters of the model, we do not ﬂag this as a signiﬁcant concern - we would just like to point it out for completeness.
R EPRODUCIBILITY S TATEMENT
Full proofs for all results in the main body of the paper can be found in Sections C and E of the
Appendix.
We have also included all of the code used to generate the plots and experimental results in this paper in the supplementary material.
We have tried our best to organize the code to be easy to use and extend.
Detailed instructions for how to run each type of experiment are provided in the
README ﬁle included with the code.
A R EVIEW OF D EFINITIONS AND A SSUMPTIONS
For convenience, we ﬁrst recall the deﬁnitions and assumptions stated throughout the paper below.
[Equation]
has positive measure for every (cid:15) >
[Equation]
Deﬁnition 2.1.
Let C ∗ to be the subset of C for which every h ∈ C ∗ satisﬁes
h ( x ) = lim (cid:15) →
0 argmin θ ∈ [0 , 1] k J mix ( θ ) | B (cid:15) ( x ) for all x ∈ X mix when the limit exists.
Here J mix ( θ ) | B (cid:15) ( x ) represents the Mixup loss for a constant function with value θ with the restriction of each term in
J mix to the set
A i,jx,(cid:15) .
Deﬁnition 2.6.
[3-Point Alternating Line]
We deﬁne X 23 to be the binary classiﬁcation dataset consisting of the points { 0 , 1 , 2 } classiﬁed as { 1 , 2 , 1 } .
In our setup, this corresponds to X 1 = { 0 , 2 } and X 2 = { 1 } with P X = 13 1 { 0 , 1 , 2 } .
Assumption 2.9.
For a class i and a point x , there exists an (cid:15) > 0 such that A j,qx,(cid:15) (cid:48) has measure zero for all (cid:15) (cid:48) ≤ (cid:15) when both j (cid:54) =
i and q (cid:54) =
i .
Assumption 3.1.
For a class i and a point x , suppose there exists an (cid:15) > 0 and a 0 < δ < min
j d ( X i ,X j )such that A i,jx,(cid:15) (cid:48) ,δ has measure zero for all (cid:15) (cid:48) ≤ (cid:15) .
Deﬁnition 3.3.
We say ˆ θ is an interpolating solution, if there exists k > 0 such that
[Equation]
Deﬁnition 3.4.
The maximum margin solution ˜ θ is deﬁned through:
[Equation]
B V ISUALIZATIONS OF D EFINITIONS AND A SSUMPTIONS
Due to the technical nature of the deﬁnitions and assumptions above, we provide several visualizations in Figures 4 to 7 to help aid the reader’s intuition for our main results.
C F ULL P ROOFS FOR S ECTION
We now prove all results found in Section 2 of the main body of the paper in the order that they appear.
C. P ROOFS FOR P ROPOSITIONS , L EMMAS , AND T HEOREMS 2.2 - 2.
Proposition 2.2.
Any function h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) satisﬁes J mix ( h ) ≤ J mix ( g ) for any continuous g ∈ C .
Proof.
Intuitively, the idea is that when h and g differ at a point x ∈ X mix , there must be a neighborhood of x for which the constant function that takes value h ( x ) has lower loss than g due to the continuity constraint on g .
We formalize this below.
Let h be an arbitrary function in argmin g ∈C ∗ J mix ( g, P X , P f ) and let g be a continuous function in
C .
Consider a point x ∈ X mix such that the limit in Deﬁnition 2.1 exists and that h ( x ) (cid:54) = g ( x ) (if such an x did not exist, we would be done).
Now let θ h
and θ g be the constant functions whose values are h ( x ) and g ( x ) (respectively) on all of R n , and further let θ h δ = argmin θ ∈ [0 , 1] k J mix ( θ )
| B δ ( x ) (this is shown to be a single value in the proof of Lemma 2.3 below).
Finally, as a convenient abuse of notation, we will use J mix ( (cid:15) (cid:48) ) | B δ ( x ) to indicate the result of replacing all log g i terms in the integrands of J mix ( g ) | B δ ( x ) with (cid:15) (cid:48) , as shown below (note that in doing so, we can combine the λ and 1 − λ terms from mixing classes i and j ).
[Equation]
Since θ h (cid:54) = θ g , we have that there exists a δ (cid:48) > 0 such that for δ ≤ δ (cid:48)
we have | θ h δ − θ g | = (cid:15) > 0 .
From this we get that there exists (cid:15) (cid:48) > 0 depending only on (cid:15) and θ g such that:
[Equation]
Now by the continuity of g (and thus the continuity of log g i ), we may choose δ ≤ δ (cid:48) such that J mix ( g ) | B δ ( x ) ∈ J mix ( θ g ) | B δ ( x ) ± J mix ( (cid:15) (cid:48) )
| B δ ( x ) .
This implies J mix ( g ) | B δ ( x ) ≥ J mix ( θ h )
| B δ ( x ) , and since x was arbitrary (within the initially mentioned constraints, as outside of them h is unconstrained) we have the desired result.
Lemma 2.3.
For any point x ∈ X mix and (cid:15) > 0 , there exists a continuous function h (cid:15) satisfying:
[Equation]
With the property that lim (cid:15) → 0 h (cid:15) ( x ) = h ( x ) for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) when the limit exists.
Proof.
Firstly, the condition that x ∈ X mix is necessary, since if (cid:83) i,j A i,j x,(cid:15) has measure zero the
LHS of Equation 1 is not even deﬁned.
Now we simply take h (cid:15) ( x ) = argmin θ ∈ [0 , 1] k J mix ( θ ) | B (cid:15) ( x ) as in Deﬁnition 2.1 and show that h (cid:15) is well-deﬁned.
Since the argmin is over constant functions, we may unpack the deﬁnition of J mix | B (cid:15) ( x ) and pull each of the log θ i ( z st ( λ )) terms out of the integrands and rewrite them simply as log θ i .
Doing so, we obtain:
[Equation]
Plugging the above back into J mix and collecting the log θ i terms as i = 1 , ..., k we get:
[Equation]
Where the ﬁrst part of the summation above is from mixing X i with itself, and the second part of the summation corresponds to the λ and (1 − λ ) components of mixing X i with X j (recall that we get a log θ i term when we mix j, i with j < i ).
Discarding the terms for which the coefﬁcients above are 0 (the associated θ i terms are taken to be 0, as anything else is suboptimal due to the summation constraint), we are left with a linear combination of log θ i 1 , log θ i 2 , ..., log θ i m , where the set { i 1 , i 2 , ..., i m } is a subset of { 1 , ..., k } .
We obviously cannot minimize J mix | B (cid:15) ( x ) by choosing any of the θ i p to be 0, and as a consequence we also cannot choose any of the θ i p to be 1 since they are constrained to add to 1.
As such, we may consider each of the θ i
p ∈
[ δ, 1 − δ ] for some δ > 0 .
With this consideration in mind, J mix | B (cid:15) ( x ) is strictly convex in terms of the θ i p , since the Hessian of a linear combination of log θ i p will be diagonal with positive entries when the arguments of the log terms are strictly greater than 0.
Thus, as [ δ, 1 − δ ] m is compact, there exists a unique solution to h (cid:15) [Equation] ) = argmin [Equation]
[Equation]
mix [Equation] ) | [Equation]
( x ) , justifying the use of equality.
This unique solution is easily computed via Lagrange multipliers, and the solution is given in Equation 1.
We have thus far deﬁned h (cid:15) on X mix , and it remains to check that this construction of h (cid:15) corresponds to the restriction of a continuous function from R n → [0 , 1] k .
To do so, we ﬁrst note that X mix is closed, since any limit point x (cid:48) of X mix must necessarily either be contained in one of the supports
X i (due to compactness) or on a line segment between/within two supports (else it has positive distance from X mix ), and every (cid:15) (cid:48) -neighborhood of x (cid:48) contains points x for which (cid:83) i,j A i,jx,(cid:15) has positive measure for every (cid:15) (immediately implying that (cid:83) i,j A
i,jx (cid:48) ,(cid:15) (cid:48) has positive measure).
Now we can check that h (cid:15) is continuous on X mix as follows.
Consider a sequence x m → x ∗ ; we wish to show that h (cid:15) ( x m ) → h (cid:15) ( x ∗ ) .
Since the codomain of h (cid:15) is compact, the sequence h (cid:15) ( x m ) must have a limit point h ∗ .
Furthermore, since each h (cid:15) ( x m ) is the unique minimizer of J mix | B (cid:15) ( x m ) , we must have that h ∗ is the unique minimizer of J mix | B (cid:15) ( x ∗ ) , implying that h ∗ = h (cid:15) ( x ∗ ) .
We have thus established the continuity of h (cid:15) on X mix , and since X mix is closed (in fact, compact), we have by the Tietze Extension Theorem that h (cid:15) can be extended to a continuous function on all of R n .
Finally, if lim (cid:15) → 0 h (cid:15) ( x ) exists, then it is by deﬁnition h
( x ) for any h ∈ C ∗ and therefore for any h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) .
Proposition 2.5.
Let X
= { (0 , 1) , (0 , − 1) } and let X
= { (1 , 0) , ( − 1 , 0) } , with P X being discrete uniform over X 1 ∪ X 2 and P f being continuous uniform over [0 , 1] .
Then the Mixup-optimal classiﬁer h is discontinuous at (0 , 0) .
Proof.
Considering the point x = (0 , 0) , we note that only ξ 1 , 1 x,(cid:15) and ξ 2 , 2 x,(cid:15) are non-zero for all (cid:15) > 0 .
Furthermore, we have that ξ 1 , 1 x,(cid:15) =
ξ 2 , 2 x,(cid:15)
= (cid:15) 4 since the P f -measure of B (cid:15) ( 12 ) is 2 (cid:15) and the sets
A 1 , 1 x,(cid:15) , A 2 , 2 x,(cid:15) have P X -measure 18 .
From this we have that the limit lim (cid:15) → 0 h (cid:15) ( x ) = (cid:2) 12 12 (cid:3) (cid:62) is the
Mixup-optimal value at x .
On the other hand, every other point on the line segment connecting the points in X 1 will have an (cid:15) -neighborhood disjoint from the line segment between the points in X 2 (and vice versa), so we will have h 1 (cid:15) (cid:48)
( x ) = 1 for all (cid:15) (cid:48) ≤ (cid:15) .
This implies that h (cid:15) (cid:48) ( x ) → [1 0] (and an identical result for points on the X 2 line segment), so we have that the Mixup-optimal classiﬁer is discontinuous at (0 , 0) as desired.
Theorem 2.7.
Let P f have associated density
Beta ( α, α ) .
Then for any classiﬁer h (cid:15) on X 23 (as deﬁned in Lemma 2.3), we may choose α such that h (cid:15) does not achieve 0 classiﬁcation error on X 23 .
Proof.
Fix a classiﬁer h (cid:15) as deﬁned in Lemma 2.3 on X 23 .
Now for Y ∼ Beta( α, α ) we have by the fact that Beta( α, α ) is strictly subgaussian that P (cid:0)(cid:12)(cid:12) Y − 12 (cid:12)(cid:12) ≤ (cid:15) (cid:1) ≥ 1 − 2 exp (cid:0) − (cid:15) 2 / (2 σ 2 ) (cid:1) where σ 2 = 1 4 α +2 is the variance of Beta( α, α ) .
As a result, we can choose α > 12 (cid:16) log4 (cid:15) 2 − 1 (cid:17) to guarantee that P (cid:0)(cid:12)(cid:12) Y − 1 2 (cid:12)(cid:12) ≤ (cid:15) (cid:1) > 12 and therefore that ξ 1 , 1 1 ,(cid:15) > 1 9 = ξ 2 , 2 1 ,(cid:15) .
Now we have by Lemma 2.3 that:
[Equation]
Where the second line followed from the fact that ξ 1 , 2 1 ,(cid:15),λ = ξ 1 , 2 1 ,(cid:15) − ξ 1 , 2 1 ,(cid:15),λ by the symmetry of Beta( α, α ) .
Thus, we have shown that h (cid:15) will classify the point 1 as class 1 despite it belonging to class 2.
Proposition 2.8.
Consider k -class classiﬁcation where the supports X 1 , ...,
X k are ﬁnite and P X corresponds to the discrete uniform distribution.
Then for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) , we have that h
i ( x ) = 1 on X i .
Proof.
The full proof is not much more than the proof sketch in the main body.
For x ∈ X i , we have that ξ j,qx,(cid:15) → 0 and ξ j,qx,(cid:15),λ → 0 as (cid:15) → 0 for every ( j, q ) (cid:54) =
( i, i ) , while ξ i,ix,(cid:15) → 1 | (cid:83) i X i | 2 and ξ i,ix,(cid:15),λ → E P f [ λ ] | (cid:83) i X i | 2 .
As a result, we have lim (cid:15) → 0 h i(cid:15) ( x ) = 1 as desired.
Theorem 2.10.
We consider the same setting as Proposition 2.8 and further suppose that Assumption 2.9 is satisﬁed by all points in (cid:83) ki
=1 X
i .
Then for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) , we have that h
i ( x ) = 1 on X i and that h is continuous on (cid:83) ki =1 X i .
Proof.
Obtained as a corollary of Theorem 3.2.
C. P ROOF FOR T HEOREM 2.
Prior to proving Theorem 2.11, we ﬁrst introduce some additional notation as well as a lemma that will be necessary for the proof.
Notation: Throughout the following m corresponds to the number of data points being considered (as in Theorem 2.11), and as a shorthand we use [ m ] to indicate { 1 , ..., m } .
Additionally, for two matrices A and B , we use the notation [ A, B ] to indicate the matrix formed by the concatenation of the columns of B to A .
We use e i to denote the i -th basis vector in R m , and use e (cid:48)
i to denote the i -th basis vector in R ( m 2 ) .
Let A ∈ R ( m 2 ) ×
m be the “Mixup matrix” where each row has two 1 s and the other entries are 0 , representing a mixture of the two data points whose associated indices have a 1 .
The (cid:0) m 2 (cid:1) rows enumerate all the possible mixings of the m data points.
In this way, A is uniquely deﬁned up to a permutation of rows.
We can pick any representative as our A matrix, and prove the following lemma.
Lemma C.1.
Assume m > 6 , and P ∈ R ( m 2 ) ×
( m 2 ) is a permutation matrix.
If PA is not a permutation of the columns of A , then the rank of [ A, PA ] is larger than m .
Using this lemma we can prove Theorem 2.11.
Theorem 2.11.
Suppose { x 1 , ..., x m } with m ≥ 6 are sampled from (cid:83)
ki =1 X i according to P X , and that P X has a density.
Then with probability 1, we can uniquely determine the points { x 1 , ..., x m } given only the (cid:0) m 2 (cid:1) midpoints { x i,j } 1 ≤ i<j ≤ m .
Proof.
We only need to show that the set of samples that cannot be uniquely determined from their midpoints has Lebesgue measure zero, since P X is absolutely continuous with respect to the Lebesgue measure.
It sufﬁces to show this for the ﬁrst entry (dimension) of the x i ’s, as the result then follows for all dimensions.
Let { x (cid:48) i } be another sample of m points.
For convenience, we group the ﬁrst entries of the data points { x i } mi =1 into a vector w ∗ ∈ R m , and similarly obtain w ∈ R m from { x (cid:48)
i } mi =1 .
Suppose w ∗ ∈
R m is not a permutation of w ∈ R m but that they have the same set of Mixup points.
We only need to show that the set of such w ∗ has measure zero.
Suppose A is a Mixup matrix and P is a permutation matrix.
Suppose PA is not a permutation of the columns in A .
We would need Aw ∗ = PAw , which is equivalent to [ A, PA ][( w ∗ ) (cid:62) , − w (cid:62) ] (cid:62) =(here [( w ∗ ) (cid:62) , − w (cid:62) ] (cid:62) indicates the vector resulting from concatenating − w to w ∗ ).
According to
Lemma C.1, we know the rank of [ A, PA ] is at least m + 1 , which implies that the solution set of w ∗ is at most m − 1 .
So ﬁxing A and P , the set of non-recoverable w ∗ has measure zero.
There are only a ﬁnite number of combinations of A and P .
Thus, considering all of these A and PA , the full set of non-recoverable w ∗ still has measure zero.
C.2. P ROOF OF S UPPORTING L EMMA
Lemma C.1.
Assume m > 6 , and P ∈ R ( m 2 ) ×
( m 2 ) is a permutation matrix.
If PA is not a permutation of the columns of A , then the rank of [ A, PA ] is larger than m .
Proof.
First, we show that both the ranks of A and PA are m .
For all i ∈ [ m − 1] , deﬁne u
i
=
e
+ e i +1 , and deﬁne u m =
e 2 + e 3 .
Note that these m vectors are all rows of A .
The ﬁrst ( m − 1) vectors { u i } m − 1 i =1 are linearly independent because each u i has a unique direction
e i +1 that is not a linear combination of any other vectors in { u i } m − 1 i
=1 .
Besides, we know that the span of { u i } m − 1 i =1 is a subspace of { v ∈ R m : (cid:80) mi =2 v ( i ) = v (1) } where v ( i ) is the i -th entry of v .
Therefore, u m doesn’t lie in the span of { u i } m − 1 i =1 , which implies that these n vectors { u i } ni =1 are linearly independent.
This shows that A is at least rank m .
Since A only has m columns, we know that the rank of A is m .
The matrix PA is also rank m because P is full rank.
Therefore, to show that the rank of [ A, PA ] is larger than m , we only need to ﬁnd a vector v that lies in the column span of A but is not in the column span of PA .
To do this, we need the following claim:
Claim 1.
There exists a row index subset I ⊆
[ (cid:0) m 2 (cid:1) ] with size ( m − 1) such that the sub-matrix composed by the rows of A with indices in I has a column that is all-one vector, but every column of the sub-matrix composed by the rows of PA with indices in I is not all-one vector.
Proof of Claim 1.
By the deﬁnition of A , each column of A has only ( m − 1) ones and the other entries are zero.
Therefore, the position of ( m − 1) ones in a column of A can uniquely determine that column vector.
Besides, for an index set I with size ( m − 1) , the sub-matrix composed by the rows of A with indices in I cannot have two columns that are both all-one vector.
This is because otherwise A will have duplicate rows, which contradicts the deﬁnition of A .
Therefore, there are m possible choices of I , each of which corresponds to the positions of all ones in a column of A .
Assume by contradiction that for these m choices of I , there exist a column of the sub-matrix composed by the rows of PA with indices in I that is all-one vector.
Then these m choices of I also correspond to the positions of all ones in a column of PA .
This means that the columns of A and
PA are the same up to permutations, which contradicts the assumption of our theorem.
This ﬁnishes the proof of this claim.
Now deﬁne B 1 as the sub-matrix composed by the rows of A with indices in I , and C 1 as the sub-matrix composed by the rows of PA with indices in I .
Without loss of generality, suppose
I =
[ m − 1] , and suppose the ﬁrst column of B 1 is all-one vector.
Let u =
−
e
+ (cid:80) mi
=2 e i ∈
R m , we know that
Au = 2 (cid:80) ( m 2 ) i
= m e (cid:48)
i , i.e., the ﬁrst ( m − 1) entries of Au are 0 , and the other entries are 2 .
Deﬁne v (cid:44) Au , we are going to show that v is not in the column span of PA .
Let C 2 be the sub-matrix in PA consisting of the rows that are not in C 1 , then the following claim shows that Chas full column rank.
Claim 2. rank ( C 2 ) = m .
Proof of Claim 2.
In this proof, we will consider each column of C 2 as a vertex.
Since each row of
C 2 has only two 1 s, we view each row as an edge connecting the two columns which correspond to the two 1 s in that row.
From the deﬁnition of A we know that the graph we constructed is a simple undirected graph.
Then we are going to show that we can select m “edges” from C 2 which are linearly independent.
There are (cid:0) m 2 (cid:1) − ( m − 1) edges in C 2 .
From m > 6 we know that (cid:0) m 2 (cid:1) − ( m − 1) > m 2 4 , so from Turán’s theorem, C 2 contains at least one triangle.
Assume this triangle is ( i, j, k ) , then we select edges ( i, j ) , ( j, k ) and ( i, k ) and deﬁne E
= { i, j, k } .
∀ r ∈ [ m ] , r ≥ 3 , we select an edge connecting E r with [ m ] \ E r .
Assume that edge is ( s, t ) where s ∈ E r and t / ∈ E r , we then add t to
E r , i.e., E r +1 =
E r ∪ { t } .
In the next two paragraphs, we are going to show that there are always edges between E r and [ m ]
\ E r
(so we can successfully select m edges in total), and the m edges we selected are linearly independent.
In matrix PA , there are r ( m − r ) edges between E r and [ m ] \ E r .
Since C 2 is constructed by deleting ( m − 1) edges from PA , the number of edges left between E r and [ m ] \ E r is at least r ( m − r ) − ( m − 1) .
When 3 ≤ r ≤ m − 2 , we have r ( m − r ) − ( m − 1) > 0 .
When r = m − 1 , the only case where there is no edge between E r and [ m ] \ E r is when all edges from vertex [ m ] \ E r is in C 1 , which means that C 1 has a column that is all-one vector and is a contradiction.
Therefore, there are always edges between E r and [ m ] \ E r and we can successfully select m edges.
Then we only need to show that these m selected edges are linearly independent.
We use { u i } mi =1 to denote the vectors that correspond to these edges, i.e., u 1 = e i + e j , u 2 =
e j + e k , u 3 =
e i + e k , · · ·
Assume by contradiction that they are linearly independent, then there exists x ∈ R m , x (cid:54) =
such that (cid:80) mi
=1 x ( i ) u i = 0 .
By the selection process of the edges, we know that ∀ r ≥ 4 , u r has a unique direction, so x ( r ) = 0 .
Therefore, x (1) u 1 + x (2) u
+ x (3) u 3 = 0 .
Since { u 1 , u 2 , u 3 } are linearly independent, we have x = 0 , which is a contradiction.
Thus, these m selected edges are linearly independent, proving the claim.
Now assume by contradiction that v is in the column span of PA , then ∃ w ∈ R m such that v = PAw .
Let v 2 ∈ R ( m 2 ) − ( m − 1) be the bottom (cid:0) m 2 (cid:1) − ( m − 1) entries of v , then v 2 = C 2 w .
Deﬁne w 0 to be the all-one vector in R m , we know that w 0 is a valid solution to v
= C 2 w .
Since C 2 has full column rank, w 0 must be the unique solution to v 2 =
C 2 w .
This implies that v = PAw 0 .
However, we know that PAw 0 = 2 (cid:80) ( m 2 ) i
=1 e (cid:48)
i (cid:54) = v , which is a contradiction.
Thus, v is not in the column span of A , which ﬁnishes the proof of the lemma.
D A DDITIONAL E XPERIMENTS FOR S ECTIOND. E XPERIMENTS FOR S ECTION 2.
As noted in the main paper, it is not difﬁcult to extend Deﬁnition 2.6 to construct datasets on which Mixup training empirically fails for small values of α .
The observation to make is that a major part of the proof of Theorem 2.7 is the fact that the same-point mixing probability of point 1 is small relative to the same-class mixing probability of class 0.
As the former probability decreases quadratically with the dataset size m , we are motivated to consider extensions of X 23 consisting of more points and more classes.
Towards that end, we consider the following generalization of X 23 :
Deﬁnition D.1 ( m -Point k -Class
Alternating Line) .
We deﬁne X km to be the k -class classiﬁcation dataset consisting of the points { 0 , 1 , ..., m − 1 } classiﬁed according to their value mod k incremented by 1.
As before, P X is the normalized counting measure on X km .
We now consider training a two-layer feedforward network on X 210 and X 1010 using the same procedure and hyperparameters as in the main paper.
The results are shown in Figure 8 below.
Here we see that even at α = 1 , Mixup training fails to minimize the original empirical risk on
X 210 and X 1010 , whereas in the main paper we noted that at α = 1 Mixup training had no issues minimizing the original risk on X 23 .
Interestingly, we ﬁnd that even standard training does not completely minimize the original empirical risk on X 210 , once again perhaps a result of the regularity of the two-layer network model (although this is merely a hypothesis, and we do not analyze this phenomenon further here). D. E XPERIMENTS FOR S ECTION 2.
Here we include the additional experiments mentioned in Section 2.4, namely the results of training
ResNet-18 on MNIST, CIFAR-10, and CIFAR-100 with and without Mixup for α = 1 , 32 , and(with the other hyperparameters being as described in Section 2.4).
The results are shown in Figure 9.
As mentioned in the main body, these choices of α only lead to Mixup performing more similarly to
ERM than α = 1024 . E F ULL P ROOFS FOR S ECTIONE. P ROOF OF T HEOREM 3.
Theorem 3.2.
Consider k -class classiﬁcation where the supports X 1 , ...,
X k are ﬁnite and P X corresponds to the discrete uniform distribution.
If a point x satisﬁes Assumptions 2.9 and 3.1 with [Figure] respect to a class i , then for every h ∈ argmin g ∈C ∗ J mix ( g, P X , P f ) , we have that h classiﬁes x as class i and that h is continuous at x .
Proof.
For two points p, q in the supports X 1 , ..., X k with a line segment between them intersecting x , let λ ( p, q, x ) denote the value of λ for which λp + (1 − λ ) q = x .
Since the supports are ﬁnite, we have that there exists (cid:15) 1 > 0 such that for all (cid:15) ≤ (cid:15)
we have that:
[Equation]
Where the summations are over all points p, q with line segments containing x .
Now we have by the Lebesgue differentiation theorem applied to the integral term in the summations above (this is possible because P f has a density) that the following limit (as well as each limit for the other
[Figure]
coordinate functions) exists:
[Equation]
Now by Assumption 2.9, we need only consider the ξ i,ix,(cid:15) , ξ i,jx,(cid:15) , ξ i,jx,(cid:15),λ terms (as there are no other line segments that contain x ).
Furthermore, by Assumption 3.1 we have that there exists (cid:15) 2 > 0 such that for all (cid:15) ≤ (cid:15) 2 we have λ ( p, q, x ) − (cid:15) ≥ 12 for p ∈
X i and q ∈ X j .
Now by similar logic to the proof of Theorem 2.7, we have for all (cid:15) ≤ min( (cid:15) 1 , (cid:15) 2 ) that:
[Equation]
Since the above also holds for a sufﬁciently small neighborhood about x , we have the desired result.
E. P ROOF OF T HEOREM 3.
Let us ﬁrst recall the setting of Theorem 3.5, since it is more specialized than that of the previous results.
Setting.
We consider the case of binary classiﬁcation using a linear model θ (cid:62) x on high-dimensional
Gaussian data, which is a setting that arises naturally when training using Gaussian kernels.
Speciﬁcally, we consider the dataset X to consist of n points in R d distributed according to N (0 , I d ) with d > n (to be made more precise shortly).
We let the labels of points in X be ± 1 (so that the sign of θ (cid:62) x is the classiﬁcation), and use X 1 and X − 1 to denote the individual class points.
Additionally, we let n 1 = | X 1 | and n − 1 =
| X − 1 | .
Before introducing the proof of Theorem 3.5, we ﬁrst present a lemma that will be necessary in the proof.
Lemma E.1.
[Strict Convexity of J mix on Data Span] Suppose n 1 = n − 1 = 1 , i.e. there are two data points x, z with opposite labels.
If x and z are linearly independent, then J mix is strictly convex with respect to θ on the span of x and z .
The reason this lemma focuses on the two data point case is that in the proof of Theorem 3.5 we will break up the Mixup loss into the sum over these cases.
With this lemma, we may now prove
Theorem 3.5.
Before doing so, we point out that the deﬁnition of J mix considered in Theorem 3.
and Lemma E.1 appears different from the deﬁnition in the main paper and in Section 2.2 in two respects.
Firstly, because we are now considering binary classiﬁcation with linear models, we write the cross-entropy as the logistic loss applied to the parameter vector θ , which is why J mix is written in the expanded form seen in the proofs below.
Secondly, we use J mix to refer to only the non-ERM terms (i.e. no mixing of identical points) since we need only prove the results for this part of the loss (as an interpolating solution is automatically ERM-optimal).
Theorem 3.5.
If the maximum margin solution for X is also an interpolating solution for X , then any θ that lies in the span of X and minimizes the Mixup loss J mix for a symmetric mixing distribution P f is a rescaling of the maximum margin solution.
Proof.
Since d > n , we have the data sets { x i } n 1 i =1 , { z j } n − 1 j =1 are linearly independent with probability one.
We will now break the proof into two parts.
Part I: n 1 = n − 1 = 1 .
Denote θ (cid:62) x 1 = u and θ (cid:62) z 1 = − v , then
[Equation]
Note λ is sampled from a distribution which is symmetric and has full support on [0 , 1] .
Therefore, we can do the change of variables λ := 1 − λ , and
[Equation]
Combining Eq.(2) and Eq.(3), we know if ( u, − v ) is a global minimum of J mix , then so is ( v, − u ) .
But the strict convexity in Lemma E.1 implies such a global minimum is unique, so we must have u = v =
k ( P f ) , where k ( P f ) is a constant that only depends on P f .
Furthermore, ∀ λ ∈ [0 , 1] , deﬁne
[Equation]
then ∀ k > 0 ,
[Equation]
k )
=
(1 − 2 λ ) log(1 + exp((1 − 2 λ ) k ))
+ (2 λ − 1) log(1 + exp((
λ
[Equation]
1) k ))
= (1 −
[Equation]
λ ) log (cid:18)
[Equation]
[Equation]
λ
[Equation]
) 1 +
[Equation]
[Equation]
[Equation]
) (cid:19)
[Equation] [Equation]
(5) and h
[Equation]
Hence, we must have k ( P f ) > 0 .
Part II: General n 1 and n − 1 .
With the ERM terms ablated and J i,jmix representing the loss incurred from mixing of points x i and z j , the decomposition of the empirical Mixup loss can be written as:
[Equation]
By the proof in the previous part we know if J i,jmix is minimized, then we must have θ (cid:62) x
i = − θ (cid:62) z j = k
( P f ) > 0 .
On the other hand, if J i,jmix ( θ ) are minimized simultaneously for all pairs ( i, j ) , then clearly J mix ( θ ) is also minimized.
This is possible since the datasets are linearly independent, so there exists θ ∈ R d , such that
[Equation]
Now we can conclude that any θ that minimizes the Mixup loss J mix is an interpolating solution.
Restricting θ to the span of X ﬁnishes the proof.
E.2. P ROOF OF S UPPORTING L EMMA
Lemma E.1.
[Strict Convexity of J mix on Data Span] Suppose n 1 = n − 1 = 1 , i.e. there are two data points x, z with opposite labels.
If x and z are linearly independent, then J mix is strictly convex with respect to θ on the span of x and z .
Proof.
Denote
[Equation]
then J mix can be expressed as
[Equation]
Note that the second term in Eq.(10) is linear in θ , hence the Hessian of J mix can be written as
[Equation]
Deﬁne B :=
Span { x, z } .
To show J mix is strictly convex on B , it sufﬁces to show for every non-zero vector a ∈ B , we always have
[Equation]
[Equation]
Note that g ( λ ) is continuous w.r.t.
λ
and that P f has full support on [0 , 1] , it sufﬁces to show either
[Equation]
13) which is equivalent to either
[Equation] [Equation]
This is obvious since a is a non-zero vector in B , and that x and z are linearly independent.
F A DDITIONAL E XPERIMENTS FOR S ECTION
In this section, we consider different class separations and choices of the mixing parameter α when training on the two moons dataset, with all other experimental settings being the same as in Section 3.1.
Upon decreasing the class separation to 0.1, we note that even standard training captures more of the nonlinear aspects of the data, as was observed in the prior work of Pezeshki et al.