k -M IXUP R EGULARIZATION FOR D EEP L EARNING VIA O PTIMAL T RANSPORT
Anonymous authors
Paper under double-blind review
A BSTRACT
Mixup is a popular regularization technique for training deep neural networks that can improve generalization and increase adversarial robustness.
It perturbs input training data in the direction of other randomly-chosen instances in the training set.
To better leverage the structure of the data, we extend mixup to k -mixup by perturbing k -batches of training points in the direction of other k -batches using displacement interpolation, i.e. interpolation under the Wasserstein metric.
We demonstrate theoretically and in simulations that k -mixup preserves cluster and manifold structures, and we extend theory studying the efﬁcacy of standard mixup to the k -mixup case.
Our empirical results show that training with k -mixup further improves generalization and robustness across several network architectures and benchmark datasets of differing modalities.
Standard mixup (Zhang et al., 2018) is a data augmentation approach that trains models on weighted averages of random pairs of training points.
Averaging weights are typically drawn from a beta distribution β
( α, α ) , with parameter α such that the generated training set is vicinal , i.e., it does not stray too far from the original dataset.
Perturbations generated by mixup may be in the direction of any other datapoint instead of being informed by local distributional structure.
As shown in Figure 1, this is a key weakness of mixup that can lead to poor regularization when distributions are clustered or supported on an embedded manifold.
With larger α , the procedure can result in averaged training points with incorrect labels in other clusters or in locations that stray far from the data manifold. (cid:124) (cid:123)(cid:122) (cid:125) No [Table]
(cid:124) (cid:123)(cid:122) (cid:125) (
[Equation] [Table] (cid:124) [Equation]
(cid:125)
[Equation]
[Table]
(cid:124) (cid:123)(cid:122)
[Figure]
To address these issues, we present k -mixup , which averages random pairs of sets of k samples from the training dataset.
This averaging is done using optimal transport, with displacement interpolation .
The sets of k samples are viewed as discrete distributions and are averaged as distributions in a geometric sense.
If k = 1 , we recover standard mixup regularization.
Figures 1 and 2 illustrate how k -mixup produces perturbed training datasets that better match the global cluster or manifold support structure of the original training dataset.
Note that the constraints of optimal transport are crucial, NEW as for instance a nearest-neighbor approach would avoid the cross-cluster matches necessary for smoothing.
In Section 4, we provide empirical results that justify the above intuition.
Our contributions are as follows.
• Empirical results: – We show improved generalization results on standard benchmark datasets showing k -mixup with k > 1 often improves on standard mixup and at worst does no harm.
– We show that k -mixup further improves robustness to adversarial attacks.
• Theoretical characterizations that speak to the claims above in a more precise manner: – We argue that as k increases, one is more and more likely to remain within the data manifold (Section 2.1).
– In the clustered setting, we provide an argument that shows intercluster regularization interpolates nearest points and better smooths interpolation of labels (Section 2.2).
– We extend the theoretical analysis of Zhang et al.
and Carratino et al.
(2020) to our k -mixup setting, showing that it leverages local training distribution structure to make more informed regularizations (Section 3).
Related works.
We tackle issues noted in the papers on adaptive mixup (Guo et al., 2019) and manifold mixup (Verma et al., 2018).
The ﬁrst refers to the problem as “manifold intrusion” and seeks to ﬁx it by training datapoint-speciﬁc weights α and considering convex combinations of more than 2 points.
This requires additional networks to detect the data manifold and to assign weight combinations, adding substantial complexity to their approach.
Manifold mixup deals with the problem by relying on the network to parametrize the data manifold, interpolating in hidden layers of the network.
We show in Section 4 that k -mixup can be performed in hidden layers to boost performance of manifold mixup.
A related approach is that of GAN-mixup (Sohn et al., 2020a)
NEW which trains a conditional GAN and uses it to generate data points between different data manifolds.
As with the other approaches above, the need for training of an additional GAN results in far greater complexity than our k -mixup method.
PuzzleMix (Kim et al., 2020) also combines optimal transport ideas with mixup, extending CutMix (Yun et al., 2019) to combine pairs of images.
PuzzleMix uses transport to shift saliency regions of images, producing meaningful combinations of input training data.
Their use of OT is fundamentally different from ours and does not generalize to non-image data.
A recent follow-on work CoMix (Kim et al., 2021), considers a similar approach based on sub/supermodular optimization.
Performing optimal transport between empirical samples of distributions has been considered in
NEW studies of the sample complexity of Wasserstein distance (e.g. Weed & Bach (2019)).
Ours is the ﬁrst application where the underlying source and target distribution are the same, and a theoretical investigation of the generalized notion of k -variance is in Solomon et al.
In other works, transport between empirical samples has been dubbed minibatch optimal transport and has been used in generative models (Genevay et al., 2018;
Fatras et al., 2020) and domain adaptation (Damodaran et al., 2018; Fatras et al., 2021).
G ENERALIZING M IXUP
Standard mixup.
Mixup uses a training dataset of feature-target pairs { ( x i , y i ) } Ni =1 ; the target y i is a one-hot vector for classiﬁcation.
Weighted averages of training points construct a vicinal dataset:
[Equation]
λ is sampled from a beta distribution, β ( α, α ) , with parameter α > 0 usually small so that the averages are near an endpoint.
Using this vicinal dataset, empirical risk minimization (ERM) becomes:
[Equation]
where f is a proposed feature-target map and (cid:96) is a loss function.
Effectively, one trains on datasets formed by averaging random pairs of training points.
As the training points are randomly selected, this construction makes it likely that the vicinal datapoints may not reﬂect the local structure of the dataset, as in the clustered or manifold-support setting.
k -mixup.
To generalize mixup, we sample two random subsets of k training points { ( x γi , y γi ) } ki = and { ( x ξi , y ξi ) } ki =1 .
For compactness, let x γ :=
{ x γi } ki =1 and y γ :=
{ y γi } ki =1 denote the feature and target sets (and likewise for ξ ).
A weighted average of these subsets is formed with displacement interpolation and used as a vicinal training set.
This concept is from optimal transport (see, e.g., (Santambrogio, 2015)) and considers ( x γ , y γ ) and ( x ξ , y ξ ) as uniform discrete distributions ˆ µ γ , ˆ µ ξ over their support.
In this setting, the optimal transport problem becomes a linear assignment problem (Peyr´e & Cuturi, 2019).
The optimal map is described by a permutation σ ∈ S k minimizing the cost:
[Equation]
Here, σ can be found efﬁciently using the Hungarian algorithm (Bertsimas & Tsitsiklis, 1997).
Figure 2 gives intuition for this identiﬁcation.
When compared to the random matching used by standard mixup, our pairing is more likely to match nearby points and to make matchings that better respect local structure—especially by having cross-cluster matches between nearby points on the two clusters. [Figure] [Equation] [Figure]
[Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
Given a permutation σ and weight λ , the displacement interpolation between ( x γ , y γ ) and ( x ξ , y ξ ) is:
[Equation]
As in standard mixup, we draw λ ∼ β ( α, α ) .
For the loss function, we consider all (cid:0) Nk (cid:1) subsets of k random samples, i.e., {{ ( x γi , y γi ) } ki =1 } ( Nk ) γ =1 :
[Equation]
The localized nature of the matchings makes it more likely that the averaged labels will smoothly interpolate over the decision boundaries.
A consequence is that k -mixup is robust to higher values of α , since it is no longer necessary to keep λ close to 0 or 1 to avoid erroneous labels.
This can be seen in our empirical results in Section 4, and theoretical analysis appears in Section 3.
Pseudocode and Computational Complexity
We would like to emphasize the speed and simplicity NEW of our method (especially relative to other mixup variants). We have included a brief PyTorch pseudocode in supplement Section A, and note that with CIFAR-10 and k = 32 the use of k -mixup added 8-9 seconds per epoch.
Also in that section is a more extended analysis of computational cost, showing that there is little computational downside to generalizing 1-mixup to k -mixup.
M ANIFOLD AND C LUSTER S TRUCTURE P RESERVATION
The use of an optimal coupling for producing vicinal data points makes it likely that vicinal datapoints reﬂect the local structure of the dataset.
Below we argue that as k increases, the vicinal couplings will preserve manifold support, preserve cluster structure, and interpolate labels between clusters.
2. M ANIFOLD SUPPORT
Suppose our training data is drawn from a distribution µ on a d -dimensional embedded submanifold
S in X × Y ⊂ R M , where X and Y denote feature and target spaces.
We deﬁne an injectivity radius:
Deﬁnition 1 (Injectivity radius) .
Let B (cid:15) ( S ) =
{ p ∈ X ×Y | d ( p, S ) < (cid:15) } denote the (cid:15) -neighborhood of S where d ( p, S ) is the Euclidean distance from p to S .
Deﬁne the injectivity radius R S of S to be the inﬁmum of the (cid:15) ’s for which B (cid:15) ( S ) is not homotopy equivalent to S .
As S is embedded, B (cid:15) ( S ) is homotopy equivalent to S for small enough (cid:15) , so R S > 0 .
Essentially,
Deﬁnition 1 grows an (cid:15) neighborhood until the boundary intersects itself.
We have the following:
Proposition 1.
For a training distribution µ supported on an embedded submanifold S with injectivity radius R S , with high probability any constant fraction 1 − δ (for any ﬁxed δ ∈ (0 , 1] ) of the couplings induced by k -mixup will remain within B (cid:15) ( S ) , for k large enough.
[-( i ) (cid:107) 
-]
The proof of this proposition is in supplement Section B. Hence, for large enough k , the interpolation induced by optimal transport will approximately preserve the manifold support structure.
While the theory requires low dimension d and high k to achieve a tight bound, our empirical evaluations show good performance in the small- k regime.
Some schematic examples are shown in Figures 1 and 2.
2. C LUSTERED DISTRIBUTIONS
With a clustered training distribution we preserve global structure in two ways: by mostly matching points within clusters and by matching (approximately) nearest points across clusters.
These characterizations are only true approximately, but are achieved exactly as k → ∞ , as argued below.
To reﬁne the deﬁnition of a clustered distribution, we adopt the ( m, ∆) -clusterable deﬁnition used in Weed & Bach (2019); Solomon et al.
In particular, a distribution µ is ( m, ∆) -clusterable if supp( µ ) lies in the union of m balls of radius at most ∆ .
Now, if our training samples ( x i , y i ) are sampled from such a distribution, where the clusters are sufﬁciently separated, then intracluster matchings will be prioritized over cross-cluster matchings under optimal transport.
Lemma 1.
Draw two batches of samples { p i } Ni =1 and { q i } Ni =1 from a ( m, ∆) -clusterable distribution, where the distance between any pair of covering balls is at least 2∆ .
If r i and s i denote the number of samples in cluster i in batch 1 and 2, respectively, then the optimal transport matching will have
[Equation]
The proof of the above (supplement Section C) involves the pigeonhole principle and basic geometry.
We also argue that the fraction of cross-cluster identiﬁcations approaches zero as k → ∞ and characterize the rate of decrease.
The proof (supplement Section D) follows via Jensen’s inequality:
Theorem 1.
Given the setting of Lemma 1, with probability masses p 1 , . . .
, p m , and two batches NEW of size k matched with optimal transport, the expected fraction of cross-cluster identiﬁcations is
[Equation]
Note that the absolute number of cross-cluster matches still increases as k increases providing more information in the voids between clusters.
We emphasize that the ( m, ∆) assumption is designed
NEW to create a “worst-case scenario” i.e. a setting where clusters are well-separated such that crosscluster identiﬁcations are as unlikely as possible.
Hence in real datasets the fraction of cross-cluster identiﬁcations will be larger than indicated in Theorem 1.
Finally, we show that these cross-cluster matches are of length close to the distance between the clusters with high probability (proved in supplement Section E), i.e., the endpoints of the match lie in the parts of each cluster closest to the other cluster.
This rests upon the fact that we are considering the W 2 cost, for which small improvements to long distance matchings yield large cost reductions in squared Euclidean distance.
Theorem 2.
Suppose density p has support on disjoint compact sets (clusters) A, B whose boundaries are smooth, where p > 0 throughout A and B .
Let D be the Euclidean distance between A and B , and let R A , R B be the radii of A, B respectively.
Deﬁne A (cid:15) to be the subset of set A that is less than
D (1 + (cid:15) ) distance from B , and deﬁne B (cid:15) similarly.
Consider two batches of size k drawn from p and matched with optimal transport.
Then, for k large enough, with high probability all cross-cluster matches will have an endpoint each in A (cid:15) and B (cid:15) , where (cid:15) =
max( R A ,R B ) 2 D.
Theorem 2 implies for large k that the vicinal distribution created by sampling along the cross-cluster matches will almost entirely lie in the voids between the clusters.
If the clusters correspond to different classes, this will directly encourage the learned model to smoothly interpolate between the class labels as one transitions across the void between clusters.
This is in contrast to the randomized matches of 1-mixup, which create vicinal distributions that can span a line crossing any part of the space without regard for intervening clusters.
The behavior noted by Theorems 1 and 2 are visualized in Figures 1 and 2, showing that k -mixup provides smooth interpolation between clusters, and strengthens label preservation within clusters.
R EGULARIZATION E XPANSIONS
Two recent works analyze the efﬁcacy of 1-mixup perturbatively (Zhang et al., 2020; Carratino et al., 2020).
Both consider quadratic Taylor series expansions about the training set or a simple transformation of it, and they characterize the regularization terms that arise in terms of label and
Lipschitz smoothing.
We adapt these expansions to k -mixup and show that the resulting regularization is more locally informed via the optimal transport coupling.
In both works, perturbations are sampled from a globally informed distribution, based upon all other samples in the training distribution.
In k -mixup, these distributions are deﬁned by the optimal transport couplings.
Given a training point x i , we consider all k -samplings γ that might contain it, and all possible k -samplings ξ that it may couple to.
A locally-informed distribution is the following:
[Equation]
where σ γξ denotes the optimal coupling between k -samplings γ and ξ .
This distribution will be more heavily weighted on points that x i is often matched with.
We use “locally-informed” in this sense of upweighting of points closer to the point of interest that are likely to be matched to it by k -mixup.
Zhang et al.
(2020) expand about the features in the training dataset D X := { x 1 , . . .
, x n } , and the perturbations in the regularization terms are sampled from D .
We generalize their characterization to k -mixup, with D replaced by D i .
We assume a loss of the form (cid:96) ( f ( x ) , y )
=
h ( f ( x )) − y
f ( x ) for some twice differentiable h and f .
This broad class of losses includes the cross-entropy for neural networks and all losses arising from Generalized Linear Models.
Theorem 3.
Assuming a loss (cid:96) as above, the k -mixup loss can be written as:
[Equation]
where lim a → 0 φ ( a ) = 0 , E std denotes the standard ERM loss, and the three R
i regularization terms are:
[Equation]
A proof is given in Section F of the supplement and follows from some algebraic rearrangement and a Taylor expansion in terms of 1 − λ .
The higher-order terms are captured by E λ ∼ β
( α +1 ,α )
[(1 − λ ) 2 φ (1 − λ )] .
E std represents the constant term in this expansion, while the regularization terms
R i represent the linear and quadratic terms.
These effectively regularize ∇ f ( x i ) and ∇ 2 f ( x i ) with respect to local perturbations r − x i sampled from D i , ensuring that our regularizations are locallyinformed.
In other words, the regularization terms vary over the support of the dataset, at each point
NEW penalizing the characteristics of the locally-informed distribution rather than a global distribution.
This allows the regularization to adapt better to local data (e.g. manifold) structure. For example, Rand R 3 penalize having large gradients and Hessian respectively along the directions of signiﬁcant variance of the distribution D i of points x i is likely to be matched to. When as k = 1 , this D i will not be locally informed, and will instead effectively be a global variance measure.
As k increases, the
D i will instead be dominated by matches to nearby clusters, better capturing the smoothing needs in the immediate vicinity of x i .
Notably, the expansion is in the feature space alone, yielding theoretical results in the case of 1-mixup on generalization and adversarial robustness.
An alternate approach by Carratino et al.
(2020) characterizes mixup as a combination of a reversion to mean followed by random perturbation.
In supplement Section G we generalize their result to k -mixup via a locally-informed mean and covariance.
E MPIRICAL R ESULTS
We empirically test the efﬁcacy of k -mixup with toy datasets, UCI datasets, image, and speech datasets employing a variety of neural network architectures.
Across these experiments we ﬁnd that k -mixup for k > 1 tends to improve upon (1-)mixup for ﬁxed α , and if not remains comparable.
Additional experiments with CIFAR-10 compare k -mixup with manifold mixup, explore the combination of the two methods, and demonstrate improvements in adversarial robustness from k -mixup.
Experiments were performed in various laptop, cluster, and cloud environments using PyTorch.
Unless otherwise stated, a standard SGD optimizer was used, with learning rate 0.1 decreased at epochs 100 and 150, momentum 0.9, and weight decay 10 − 4 .
Toy datasets.
Results for the toy datasets of Figures 1 and 2 (denoted “One Ring,” “Four Bars,” and “Swiss Roll”) are shown in Figure 3.
We used a fully-connected 3-layer neural network (130 and 120 hidden units).
As the datasets are very clustered and have no noise, smoothing is not needed for generalization and performance without mixup is typically 100%.
Applying mixup to these datasets thus provides a view to the propensity of each variant to oversmooth, damaging performance.
For each dataset and each α , higher k -mixup outperforms (1-)mixup.
For larger α , the performance gap between the baseline (1-)mixup and k > 1 mixup becomes quite signiﬁcant, reaching 8%, 50%, and 40% respectively for k = 16 and α = 64 .
These results quantitatively conﬁrm the intuition built NEW in Figures 1 and 2 that k -mixup regularization more effectively preserves these structures in data, limiting losses from oversmoothing. [Figure]
UCI datasets.
Scaling up from the toy datasets, we tried k -mixup on UCI datasets (Dua & Graff, 2017) of varying size and dimension: Iris (150 instances, dim. 4), Breast Cancer WisconsinDiagnostic (569 instances, dim. 30), Abalone (4177 instances, dim.
8), Arrhythmia (452 instances, dim. 279), HTRU2 (17898 instances, dim. 9), and Phishing (11055 instances, dim. 30).
For Iris, we used a 3-layer network with 120 and 84 hidden units; for Breast Cancer, Abalone, and Phishing, we used a 4-layer network with 120, 120, and 84 hidden units; for Arrhythmia we used a 5-layer network with 120, 120, 36, and 84 hidden units; and lastly, for HTRU2, we used a 5 layer network with 120, 120, 84, 132 hidden units.
Each entry is averaged over 20 Monte Carlo trials.
Test classiﬁcation performance is shown in Figure 4.
k -mixup improves over (1-)mixup in each case, although for Arrhythmia no mixup outperforms both, and for Phishing, no mixup statistically matches the performance of k -mixup.
In these small datasets, the best (or at least good) performance seems to be achieved with relatively small α = 0 .
1 and moderate k (4 or 8).
[Table] [Figure] [Figure] [Figure]
k α = . α = . 1 α = .
2 α = .
5 α =1 α =10 α =10029.
[Figure]
Image datasets. Results for MNIST (LeCun & Cortes, 2010) using a convolutional neural network, i.e., LeNet modiﬁed slightly to accommodate grayscale images, are in Figure 5, with a joint sweep over k and α .
Each table entry is averaged over 20 Monte Carlo trials.
For each generated point in the vicinal distribution, we compute its closest squared distance to the matched pair from which it was generated.
The results are averaged over the vicinal dataset and reported in part (b) of the ﬁgure.
It shows that k -mixup yields signiﬁcantly closer matches as k increases, causing the generated vicinal distribution to deviate less (in terms of squared distance) from the original training set.
This is empirical evidence of our method respecting manifold support and cluster structure, as noted in
Section 2.
In spite of this lower variation, for each α the best generalization performance is for some k > 1 , with α = 100 yielding 99 .
18% accuracy for k = 32 compared to 98 .
64% accuracy for k = 1 .
Generally, for ﬁxed α , increasing k improves performance for this dataset.
[Figure]
Figure 6 shows analogous results for CIFAR-10, using the
PreAct Resnet-18 architecture as in Zhang et al.
Again k -mixup succeeds in ﬁnding matches that lie signiﬁcantly closer together as k increases.
For each α except the small α = .
1 , the best generalization performance is still for some k > 1 , with α = 100 yielding 95 .
165% accuracy for k = 32 compared to 94 .
085% accuracy for k = 1 .
The best performance overall is achieved at k = 8 with parameter α = 1 .
While the best k -mixup performance exceeds that of the best 1-mixup by only 0.17%, recall that in this setting 1-mixup outperforms ERM by 1.4% (Zhang et al., 2018), so when combined with the low overall error rate, small gains are not surprising.
Tables of results for DenseNet and WideResnet architectures can be found in Figure 9, with the best k -mixup outperforming the best 1-mixup by 0.44% and 0.28% respectively.
Note that in the case of Densenet, k = 16 outperforms or (statistically) matches k = 1 for all values of α .
For both MNIST and CIFAR-10 datasets, with coupling distances reported, one could compare
NEW settings with similar average squared distances.
For example, a close inspection of Figures 5 and 6 show that higher k again tends to have the advantage, indicating that performance gains are not simply due to having a closer vicinal distribution.
Additionally, we show training curves (performance as a function of epoch) for Resnet-18 on CIFAR10 in Figure 7, averaged over 20 random trials.
The training speed (test accuracy in terms of epoch) for 1-mixup and 32-mixup shows no loss of convergence speed with k = 32 mixup, with (if anything) k
= 32 showing a slight edge.
The discontinuity at epoch 100 is due to our reduction of the learning [Figure] [Figure] rate at epochs 100 and 150 to aid convergence (used throughout our image experiments).
The train accuracy shows similar convergence proﬁle between 1- and 32-mixup; the difference in absolute accuracy here (and the reason it is less than the test accuracy) is because the training distribution is the mixup-modiﬁed vicinal distribution.
The curve for k = 32 is higher, especially for α = 10 , because the induced vicinal distribution and labels are more consistent with the true distribution, due to the better matches from optimal transport.
The large improvement in train accuracy is remarkable given the high dimension of the CIFAR-10 data space, since it indicates that k = 32 mixup is able to ﬁnd signiﬁcantly more consistent matches than k = 1 mixup.
Figure 8 shows results for CIFAR-100 and SVHN, with a Resnet-18 architecture.
As before, for ﬁxed α , the best performance is achieved for some k > 1 .
The improvement of the best k -mixup over the best 1-mixup is 0.53% for CIFAR-100 and 0.12% for SVHN.
For ﬁxed high α = 10 , the k -mixup improvement over 1-mixup rises to 2.02% for CIFAR-100, possibly indicating that the OT matches yield better interpolation between classes, aiding generalization.
[Table]
test accuracy using LeNet averaged over 20 Monte ( ± .
014 conﬁdence).
Speech dataset .
Performance is tested on a speech dataset:
Google Speech Commands (Warden, 2018) using a LeNet architecture are in Figure 10.
Each table entry is averaged over 20 Monte Carlo trials ( ± .
014 conﬁdence on test performance).
We augmented the data in the same way as
Zhang et al.
The difference between best k -mixup and best 1-mixup is 0.4%.
Manifold mixup.
We compare to Manifold Mixup (Verma et al., 2018), which helps in settings like those in Figure 1 by making interpolations more meaningful.
It performs mixup at random layers (random per minibatch), not only in the input space.
This extends to “manifold k -mixup:” k -mixup in the hidden layers as well as the input layers.
We use settings in Verma et al.
(2018), i.e., for PreAct
Resnet18, the mixup layer is randomized (coin ﬂip) between the input space and the output of the ﬁrst residual block.
Results for CIFAR-10 with a Resnet18 architecture are in Figure 11.
Numbers in this experiment are averaged over 20 Monte Carlo trials ( ± .
03 conﬁdence on test performance).
While manifold 1-mixup outperforms the standard 1-mixup from Figure 6, it is matched by manifold k -mixups with k =and outperformed by standard k -mixup (Figure 6, k = 8 ).
We also tried randomizing over (a) the outputs of all residual blocks and (b) the outputs of (lower-dimensional) deep residual blocks only, but found that performance of both 1-mixup and k -mixup degrades in these cases.
This underscores that mixup in hidden layer manifolds is not guaranteed to be effective and can require tuning.
[Table]
Adversarial robustness.
Figure 12 shows results on white-box adversarial attacks generated by the FGSM method (implementation of Kim (2020) 3 ) for various values of maximum adversarial perturbation.
As in Verma et al.
(2018), we used FGSM over PGD, since the iterative PGD attacks are signiﬁcantly more effective, making any performance improvements seem less relevant in practice.
Additionally, we expect simple gradient attacks to be particularly meaningful for evaluating the success of mixup, whose goal is to approximately linearly interpolate between classes.
We show CIFAR-10 accuracy on NEW white-box FGSM adversarial data (10000 points), where the maximum adversarial perturbation is set to (cid:15)/ 255 ; performance is averaged over 30 Monte Carlo trials ( ± 0 . 6 conﬁdence).
All performances are without adversarial training.
Note that k
= 2 outperforms k = 1 uniformly by as much as 8.64%, and the k > 1 mixups all outperform k = 1 mixup on larger attacks.
Similar results for MNIST are in Figure 12(b), with the FGSM attacks being somewhat less effective.
Here k = 8 is best for smaller attacks, and k = 2 is best for larger attacks.
The improved robustness shown by k -mixup speaks to a key goal of mixup, that of smoothing the predictions in the parts of the data space where no/few labels are available.
This smoothness should make adversarial attacks require greater magnitude to successfully “break” the model.
[Figure]
C ONCLUSIONS AND F UTURE W ORK
The experiments above demonstrate that k -mixup often improves the generalization and robustness gains achieved by (1-)mixup [Figure] and at least does no harm.
This is seen across a diverse range of datasets and network architectures.
It is simple to implement, adds little computational overhead to conventional (1-)mixup training, and may also be combined with related mixup variants.
As k increases, the regularization induced more accurately reﬂects the local structure of the training data, especially in the manifold support and clustered settings, as seen in Sections 2 and 3.
Empirical results show that performance is robust to variations in k , ensuring that extensive tuning is not required.
Our experiments show the greatest improvement from using k -mixup on low-dimensional datasets, while the improvement on high-dimensional datasets remains positive but is smaller (recall that classic mixup also has somewhat small gains over no mixup in these settings).
This difference could be due to the diminishing value of Euclidean distance for characterizing dataset geometry in high dimensions (Aggarwal et al., 2001), but intriguingly this effect was not remedied by doing OT in the possibly lower-dimensional manifolds created by the higher layers in our manifold mixup experiments.
To remedy this issue, in future work we will consider alternative metric learning strategies, with the goal of identifying effective high-dimensional metrics for displacement interpolation of data points.
Mixup has been incorporated into a broad range of learning applications, many of which we have not empirically investigated here.
The original work demonstrated robustness to label corruption and improved GAN training stability, which could easily beneﬁt from k -mixup as well.
Additionally, mixup has been used for augmentation in semi-supervised learning (Berthelot et al., 2019; Sohn et al., 2020b).
Given its simplicity, a k -mixup augmentation may be able to generate more informed shifts of labelled and unlabelled training points.
This same philosophy applies to a recent method for generalizing fair classiﬁers (Chuang & Mroueh, 2021), which already leverages manifold mixup.
Ethics Statement The k -mixup regularization procedure we propose and our experimentation on it do not bring up additional ethical concerns outside of general concerns associated with machine learning as a ﬁeld, and the use of neural network models.
Standard benchmark datasets and tasks were used, with the assumption that dataset originators have done their due diligence with respect to concerns on IRBs, privacy, fairness, and the like.
Reproducibility Statement With regards to reproducibility, we note that k -mixup is exceedingly simple, and may be easily incorporated into almost any neural network as a data perturbation step.
For our experiments, publicly available benchmark datasets have been used and are clearly stated in Section 4.
Additionally, the models used and the relevant training parameters are described for each task considered.
Upon acceptance, we would be glad to make a full implementation of our experiments available on Github under an open source MIT license.
Lastly, for theoretical results, full proofs of our statements are provided in the appendices following the references.
A k - MIXUP P SEUDOCODE & C OMPUTATIONAL C OST
NEW Below is code for one epoch of k -mixup training, in the style of Figure 1 of Zhang et al. # y1, y2 should be one-hot vectors for (x1, y1), (x2, y2) in zip (loader1, loader2): cost =
numpy.inf * [x1.shape[0], x1.shape[0]] for i in range (x1.shape[0] // k): cost[i * k:(i+1)
* k, i * k:(i+1)
* k] = scipy.spatial.
distance_matrix(x1[i * k:(i+1)
* k], x2[i * k:(i+1)
* k]) _, idx = scipy.optimize.linear_sum_assignment(cost) x2[i
* k:(i+1)
* k] = x2[idx, i * k:(i+1)
* k] y2[i * k:(i+1)
* k] = y2[idx, i * k:(i+1)
* k] lam = numpy.random.beta(alpha, alpha) x = Variable(lam
* x1 + (1 - lam) * x2)
y =
Variable(lam * y1 + (1 - lam) * y2)
optimizer.zero_grad() loss(net(x), y).backward() optimizer.step()
Computational cost.
While the cost of the Hungarian algorithm is O ( k 3 ) , it provides k data points for regularization, yielding an amortized O ( k 2 ) complexity per data point.
For the smaller values of k that we empirically consider, approximate Sinkhorn-based methods (Cuturi, 2013) are slower in practice, and the Hungarian cost remains small relative to that of gradient computation (e.g. for
CIFAR-10 and k = 32 , the Hungarian algorithm costs 0.69 seconds per epoch in total).
Computing the distance matrix input to the OT matching costs O ( k 2 d ) where d is dimension, yielding an amortized O ( kd ) complexity per data point.
With the high dimensionality of CIFAR, a naive CPU implementation of this step of the computation adds about 8 seconds per epoch.
Moreover, training convergence speed is unaffected, unlike manifold mixup (Verma et al., 2018), which in our experience converges slower and has larger computational overhead.
Overall, there is little computational downside to generalizing 1-mixup to k -mixup.
B P ROOF OF P ROPOSITION
NEW
Finite-sample convergence of empirical measures implies that for an arbitrary sampling of k points ˆ µ k , we have W 22 ( µ, ˆ µ k ) ≤
O ( k − 2 /d ) with high probability (e.g., 1 − 1 /k 2 probability, from Theorem 9.1 of Solomon et al.
(2020) when combined with results from Weed & Bach (2019)).
The triangle inequality then implies that the Wasserstein-2 distance between our batches of k samples will tend to 0 at the same asymptotic rate: W 22 (ˆ µ γk , ˆ µ ξk ) ≤ O ( k − 2 /d ) with high probability.
Recalling that by the deﬁnition of the optimal coupling permutation σ ( i ) , we have 1 k (cid:80) ki
=1 (cid:107) x γ
{+i − x ξσ ( i ) (cid:107) 
+}
= W 22 (ˆ µ γk , ˆ µ ξk ) ≤
O ( k − 2 /d ) with high probability.
Hence, for any I ⊆
[1 , k ] with (cid:107) x γi − x ξσ
( i ) (cid:107) 22 > k − 1 /d for all
i ∈ I , |I| ≤ O ( k 1 − 1 /d ) ≤
δk for k large enough and any δ ∈ (0 , 1] .
In essence, long-distance identiﬁcations are rare, and their fraction is bounded for large enough k .
Therefore, for k large enough, again with high probability there exists a subset ¯ I with | ¯ I| ≥ (1 − δ ) k
such that (cid:107) x γi − x ξσ
( i ) (cid:107) 2 ≤ R S for all i ∈
¯ I .
Since x γi and x ξσ ( i ) lie in S , the proposition results.
C P ROOF OF L EMMA
We argue by contradiction and prove the result for m = 2 ﬁrst.
Suppose that the number of crosscluster matchings exceeds | r 1 − s 1 | .
Then by the pigeonhole principle, there must be at least two such matchings, which we’ll say are between p i and q i , and p i +
and q i +1 , WLOG.
As the cost of the cross-cluster matchings is ≥ (2∆) 2 , and the cost of intra-cluster matchings is ≤ (2∆) 2 , a more It optimal matching would pair p i and q i +1 , and p i +1 and q i .
This contradicts optimality of the initial pairing.
In the scenario with m clusters, an analogous argument works.
As above, | r i − s i | is the number of cluster i elements that must be matched in a cross-cluster fashion, and half the sum of these quantities is the minimum number of cross-cluster matchings overall.
If this number is exceeded, by the pigeonhole principle, there must be additional cross-cluster matches that form a cycle in the graph over clusters.
As above, the cost would be reduced by matching points within the same clusters, so this contradicts optimality.
D P ROOF OF T HEOREM
We ﬁrst prove the result for two clusters of weight p and 1 − p .
With our clustered assumption, we may assume that all identiﬁcations that can be intracluster will be intracluster.
Thus, if s 1 and sdenote the number of points in cluster 1 from batch 1 and 2, then the resulting number of cross-cluster matches is | s 1 − s 2 | .
As the samples for our batches are i.i.d.
, these random variables follow a simple binomial distribution B ( k, p ) .
We can bound the expectation of this quantity with Jensen’s inequality:
[Equation]
With some algebraic manipulation, our desired rate bound results.
It is also possible to get an exact rate with some hypergeometric identities (Katti, 1960), but these simply differ by a constant factor, so we omit the exact expressions here.
For the general case, our clustered assumption again allows us to assume that OT will prioritize intracluster identiﬁcations.
Thus, if we let r i and s i denote the number of samples in cluster i in batch 1 and 2, respectively, then the number of cross-cluster matchings will be 12 (cid:80)
i
| r i − s i | .
Ultimately, r i and s i are sampled from a binomial distribution B ( k, p i ) , so we may repeat the argument above and note that E
[ r i =
s i ] ≤ (cid:112)
2 kp i (1 − p i ) .
Simple algebraic manipulations lead to the desired result.
E P ROOF OF T HEOREM
By the smooth boundary and positive density assumptions, we know that P ( A δ ) > 0 and P ( B δ ) >for any δ > 0 .
Hence, for ﬁxed δ and k large enough, we know that with high probability the sets A δ and B δ each contain more points than the number of cross-cluster identiﬁcations.
Now consider A (cid:15) and B (cid:15) for (cid:15)
=
δ + (max( R A , R B ) 2 ) / (2 D 2 ) .
All cross-cluster matches need to be assigned.
The cost of assigning a cross-cluster match to a point in A δ and a point in B δ is at most
D 2 (1 + 2 δ ) 2 (since we are using W 2 ).
Furthermore, the cost of assigning a cross-cluster match that contains a point in A outside A (cid:15) and an arbitrary point in B is at least D 2 (1 + (cid:15) ) 2 .
Consider the difference between these two costs:
[Equation]
Since this difference > 0 and we have shown A δ contains sufﬁcient points for handling all assignments, this assignment outside of A (cid:15) will only occur if there is a within-cluster pair which beneﬁts from using the available point in A (cid:15) more than is lost by not giving it to the cross-cluster pair ( > R 2 A ).
The maximum possible beneﬁt gained by the within-cluster pair is the squared radius of A , i.e. R 2 A .
Since we have shown that the lost cost for the cross-cluster pair is bigger than R 2 A , we have arrived at a contradiction.
The proof is similar for the B side.
We have thus shown that for k large enough (depending on δ ), with high probability all cross-cluster matches have an endpoint each in A (cid:15) and B (cid:15) where (cid:15) =
2 δ + (max( R A , R B ) 2 ) / (2 D 2 ) .
Setting δ =
(max( R A , R B ) 2 ) / (4 D 2 ) completes the proof.
F P ROOF OF T HEOREM
We mostly follow the notation and argument of Zhang et al.
(c.f. Lemma 3.1), modifying it for our setting.
There they consider sampling λ ∼ Beta ( α, β ) from an asymmetric Dirichlet distribution.
Here, we assume a symmetric Dirichlet distribution, such that α = β , simplifying most of the expressions.
The analogous results hold in the asymmetric case with simple modiﬁcations.
Consider the probability distribution ˜ D λ with probability distribution:
β ( α + 1 , α ) .
Note that this distribution is more heavily weighted towards 1 across all α , and for α < 1 , there is an asymptote as you approach 1 .
Let us adopt the shorthand notation ˜ x
i,σ γξ ( i ) ( λ ) := λx γi + (1 − λ ) x ξσγξ for an interpolated feature point.
The manipulations below are abbreviated, as they do not differ much for our generalization.
[Equation]
For the third equality above, the ordering of sampling for λ and B has been swapped via conjugacy: λ ∼ β ( α, α ) , B | λ ∼ Bern ( λ ) is equivalent to B ∼ U{ 0 , 1 } , λ | B ∼ β
( α + B, α + 1 − B ) .
This is combined with the fact that ˜ x i,σ γξ ( i ) (1 − λ ) =
˜ x
σ γξ ( i ) ,i ( λ ) to get the last line above.
Now we can swap the sums, grouping over the initial point to express this as the following:
[Equation]
where the probability distribution D i is as described in the text.
The remainder of the argument performs a Taylor expansion of the loss term h
( f ( λx i + (1 − λ ) r )) − y γi f
( λx i + (1 − λ ) r ) in terms of 1 − λ , and is not speciﬁc to our setting, so we refer the reader to
Appendix A.1 of (Zhang et al., 2020).
for the argument.
G k - MIXUP AS M EAN R EVERSION FOLLOWED BY R EGULARIZATION
Theorem 4.
Deﬁne (˜ x γi , ˜ y γi ) as
[Equation]
where ¯ x γi = 1 ( Nk ) (cid:80) ( Nk ) ξ
=1 x ξσ γξ ( i ) and ¯ y γi = 1 ( Nk ) (cid:80) ( Nk ) ξ =1 y ξσ γξ ( i ) are expectations under the matchings and θ ∼ β
[1 / 2 , 1] ( α, α ) .
Further, denote the zero mean perturbations
[Equation]
Then the k -mixup loss can be written as
[Equation]
The mean ¯ x γi being shifted toward is exactly the mean of the locally-informed distribution D i .
Moreover, the covariance structure of the perturbations is detailed in the proof (simpliﬁed in Section
G.1) and is now also derived from the local structure of the distribution, inferred from the optimal transport matchings.
Proof.
This argument is modelled on a proof of Carratino et al.
, so we adopt analogous notation and highlight the differences in our setting and refer the reader to Appendix B.1 of that paper for any omitted details.
First, let us use shorthand notation for the interpolated loss function:
[Equation]
Then the mixup objective may be written as:
[Equation]
As λ ∼ β ( α, α ) , we may leverage the symmetry of the sampling function and use a parameter θ ∼ β [1 / 2 , 1] ( α, α ) to write the objective as:
[Equation]
γ = (cid:96) i , where (cid:96) i
= k (cid:0) Nk (cid:1)
( Nk ) (cid:88)
ξ = k (cid:88) i =
E θ m γξi ( θ )
To obtain the form of the theorem in the text, we introduce auxiliary variables ˜ x γi , ˜ y γi to represent the mean-reverted training points:
[Equation]
and ˜ δ γi , ˜ (cid:15) γi to denote the zero mean perturbations about these points:
[Equation]
These reduce to the simpliﬁed expressions given in the theorem if we recall that θ and ξ are independent random variables.
Note that both the mean-reverted points and the perturbations are informed by the local distribution D i .
G. C OVARIANCE STRUCTURE
As in (Carratino et al., 2020) [Equation] it’s possible to come up with some simple expressions for the covariance structure of the local perturbations, so we write out the analogous result below.
As the argument is very similar, we omit it.
Lemma 2.
Let σ 2 denote the variance of β [1 / 2 , 1] ( α, α ) [Equation] and ν 2 : = σ 2 +(1 − ¯ θ ) 2 .
Then the following expressions hold for the covariance of the zero mean perturbations:
[Equation]
where Σ ˜ x γi ˜ x γi , Σ ˜ y γi ˜ y γi , Σ ˜ x γi ˜ y
γi denote empirical covariance matrices.
Note again, that the covariances above are locally-informed, rather than globally determined.
Lastly, there is also a quadratic expansion performed about the mean-reverted points ˜ x γi , ˜ y γi with terms that regularize f , but we omit this result as the regularization of Theorem 3 is more intuitive
(c.f. Theorem 2 of (Carratino et al., 2020)).