When does mixup promote local linearity in learned representations?
Abstract mixup is a regularization technique that artiﬁcially produces new samples using convex combinations of original training points.
This simple technique has shown strong empirical performance, and has been heavily used as part of semi-supervised learning techniques such as mixmatch [1] and interpolation consistent training (ICT) [17].
In this paper, we look at mixup through a representation learning lens in a semi-supervised learning setup.
In particular, we study the role of mixup in promoting linearity in the learned network representations.
Towards this, we study two questions: (1) how does the mixup loss that enforces linearity in the last network layer propagate the linearity to the earlier layers?; and (2) how does the enforcement of stronger mixup loss on more than two data points affect the convergence of training?
We empirically investigate these properties of mixup on vision datasets such as CIFAR-10, CIFAR-100 and SVHN.
Our results show that supervised mixup training does not make all the network layers linear; in fact the intermediate layers become more non-linear during mixup training compared to a network that is trained without mixup .
However, when mixup is used as an unsupervised loss, we observe that all the network layers become more linear resulting in faster training convergence.
Introduction
While models learned via empirical risk minimization (ERM) [15] tend to perform well on test data that are similar to training data, predictions can change signiﬁcantly when the samples are chosen outside the training distribution.
For improved generalization, typically data augmentation techniques are used to generate new training examples near the neighborhood of the original training samples through simple transformations [10].
Such techniques play a critical role in training deep neural networks that have shown great success [6, 12, 4, 3].
While popular data augmentation techniques such as ﬁltering and cropping on images tend to produce samples near the vicinity of training samples, they are domain-speciﬁc and require expert knowledge in generating augmentations.
On the contrary, mixup [19] – an augmentation technique that generates new training data by linearly interpolating the original training points – is applicable in various domains without requiring expert domain knowledge.
mixup has shown strong performance in both supervised [19] and semisupervised [17, 1] learning setups by allowing the model to learn better network representations .
In the context of supervised learning, the representations learned through mixup regularization are shown to improve generalization of the network and also its robustness to corrupt labels [19].
The general idea of linearly combining the two vectors in mixup can then be extended to the intermediate layers of the network resulting in even better representations [16].
Additionally, mixup training has been shown to improve network calibration for both in- and out-of-distribution data [14].
In the context of semi-supervised learning, mixup regularization, also known as interpolation consistency training ( ICT ), is shown to improve the performance of a learner signiﬁcantly.
Speciﬁcally,
Verma et al.
[17], showed that mixup training encourages consistency regularization which is an effective unsupervised learning signal explored in many works [9, 7, 13, 8].
Other related methods include mixmatch [1] that rely on mixing labeled and unlabeled samples, and methods relying on pseudo-labels [11] and data augmentation [18].
While mixup regularization has brought performance beneﬁts across the board, it is less clear how network representations differ between a network trained with and without mixup .
Some theoretical works have studied the improved generalization of mixup through the lens of regularization effects it brings to the network training.
Carratino et al.
[2] argued that mixup regularizes the Jacobian of the network resulting in a function with a low Lipschitz constant.
Similarly, Zhang et al.
[20] showed that, in a two-layer ReLU network, mixup training reduces the complexity of the hypothesis class leading to better generalization.
While these works point to some regularization effects of mixup , the training dynamics of a network trained under mixup regularization are not fully clear.
Our work is an empirical study that aims to provide some clues as to how network representations evolve during mixup training.
Speciﬁcally, 1 ) we show that mixup tends to make the ﬁrst and the last layer of a network more linear, but does so at the expense of making the intermediate layers more non-linear, compared to a network that is trained without mixup .
We show that when mixup is used as an unsupervised loss, ICT, all layers tend to become more linear.
Finally, 3) we show that enforcing a stronger linearity in mixup , by means of using more than two mixing points, leads to more linear representations that manifest in faster convergence to a speciﬁed test accuracy with less labeled examples in a semi-supervised setup.
Setup
Let X ∈ R D and Y ∈ R C be input and output spaces, respectively.
We adopt a standard semisupervised learning setup where we assume access to a small labeled dataset D s = { ( x i , y i ) } n s i =1 , and a large unlabeled dataset D us = { u i } n us i =1 , with n s (cid:28) n us are the number of examples in the labeled and unlabeled datasets, respectively.
The training objective is to learn a model f θ : X (cid:55)→ Y , parameterized by θ ∈ R P (a neural network in our case), that performs well on a held out test set D te by minimizing the following loss:
[Equation]
where (cid:96) s : Y × Y (cid:55)→ R ≥ 0 is a supervised loss computed on D s (typically a cross-entropy loss for classiﬁcation tasks), (cid:96) us is an unsupervised loss deﬁned on D us , and w ( t ) is the weight of unsupervised loss at the t -th iteration of stochastic gradient descent (SGD).
We deﬁne the mixup operation as a convex operation parameterized by λλλ ∈ R K ≥ 0 , on a set P = { p 1 , · · · , p K } as:
[Equation] [Equation]
such that (cid:80) Ki
=
λ i = 1 , and λ
i ≥ 0 for all i ∈ { 1 , · · · , K } .
A sample from a K -th order Dirichlet distribution with parameters α 1 , · · · , α K > 0 , deﬁnes a valid λλλ for the mixup operation.
When the mixup operation is used on the supervised loss (cid:96) s , the examples x ∈ X and targets y ∈ Y are modiﬁed as:
[Equation]
at each training step of the SGD.
For the case of K = 2 , the supervised loss becomes the standard mixup training as proposed by Zhang et al.
Similarly, we can also use the mixup operation to deﬁne an unsupervised loss on the unlabeled examples.
Concretely, given K unlabeled examples
K = { u 1 , · · · , u K } , the unsupervised loss (cid:96) us is deﬁned as,
[Equation]
where D is some distance metric; L 2 distance is used in this work.
Intuitively, the unsupervised loss enforces consistent predictions on the interpolated unlabeled examples which Verma et al.
[17] found to be a good learning signal for consistency regularization in semi-supervised setups.
Experiments
Setup We assume standard semi-supervised learning datasets (CIFAR-10, CIFAR-100, and SVHN) and experiment with different labeled dataset sizes.
For CIFAR-10 and SVHN, we consider labeled datasets of sizes { 250 , 1000 , 2000 , 40000 } .
For CIFAR-100, we only consider a single labeled dataset of size 10 K.
We use a ‘Wide ResNet-28’ architecture similar to [1].
We denote the output of the initial convolution layer as ‘Layer 0’, and the outputs of the next three Wide ResNet blocks are referred to as ‘Layer 1-3’, followed by the output of average pooling layer that is denoted by ‘Layer 4‘.
Finally the logits layer is referred to as ‘Layer 5’.
Under this deﬁnition of a ‘layer’, we monitor the representation after each layer.
The batch size is set to 128 and the Adam optimizer [5] is used with learning rate = 0 .
002 , β 1 = 0.9 and β 2 = 0.999.
The weight ( w t ) of the unsupervised loss follows a schedule such that its value linearly increases from 0 to a maximum of 100 in 40 % of training iterations that are kept ﬁxed for all datasets and for all dataset sizes.
The linearity of network ‘layers’ is periodically reported throughout the network training on a held out test set.
The α for the Dirichlet distribution is set to 0 .for
CIFAR-10 and CIFAR-100, and 0 . 1 for SVHN.
Each method is run with 5 different random seeds and the performance is averaged across these runs, and error bars are reported for convergence plots.
Baselines We compare 3 different baselines.
ERM refers to standard empirical risk minimization without mixup .
mixup refers to training using two mixup points.
In both these setups, only the supervised labeled set D s is used and D us is not used.
ICT[K] refers to unsupervised mixup , as deﬁned in equation 2, with K ∈ { 2 , 3 , 4 } referring to the number of examples mixed together.
For this setup, the ICT loss is computed on the D us and standard cross-entropy loss computed on D s .
3.1 Non-Linearity in Network Representations
We ﬁrst investigate the effect of mixup on the network representations at different layers during training.
Speciﬁcally, we take a look at the effect of mixup training on the (non-) linearity of network representations.
We speciﬁcally chose to study the linearity of representations because the mixup theory [2, 20] suggests that mixup trained networks tend to be more linear than their non- mixup counterparts.
We deﬁne the non-linearity in a layer l at the t -th training step by:
[Equation]
where D te is a held out test set, and ˆ f lt ( · ) is the normalized output of layer l at the t -th training step.
We normalize the output of a layer to have a unit norm, i.e .
ˆ f lt = f lt (cid:107) f lt (cid:107) , to allow for a fair comparison of the non-linearity across different layers.
Lower values imply a more linear layer .
Results Figure 1 show the evolution of non-linearity throughout network training for different baselines on CIFAR-10.
These ﬁgures use a labeled dataset of size 250 (see Appendix for other datasets and settings).
The total training iterations are remapped between 0 and 1000 and referred to as Train time in the plots.
First, we compare the (non-)linearity of ERM and mixup (both are supervised setups) in different layers.
We see that in the ﬁrst and last layer the representations from mixup are more linear compared to ERM representations.
Intuitively, this could be explained by the mixup operation linearly combining both the inputs and outputs.
Hence, training using mixup keeps the ﬁrst and last functions in the composition f
= f 0 ◦ f 1 · · · ◦ f 5 more linear.
Surprisingly, however, we do not see that mixup representations remain more linear in all the layers compared to those of ERM as suggested by the theory.
For example, in Figure 1, we can see that Layers 2, 3, 4 based on mixup training are more non-linear compared to those trained with ERM.
This suggests that a) the ﬁndings of theory on simpliﬁed setups do not directly translate to practical deep networks, and b) more importantly, without proper regularization in all layers, the network maintains an overall non-linearity from input to output.
Further, this suggests that the linearity enforced by a regularization in some layers is counteracted by increased non-linearity in the other layers.
[Figure] Second, we see that ICT tend to make all the layers more linear compared to ERM and mixup , except for the ﬁrst layer where mixup is slightly more linear.
This suggests that the ICT loss, as deﬁned in equation 2, applied on the network logits, and on a larger unsupervised dataset, is a stronger regularization compared to mixup deﬁned on one-hot encoded vectors.
Further, we see from [-Figure 
 [Figure] that-]Figurethat using more mixup points (ICT4 vs ICT2), leads to stronger linearity across all layers.
In the next section, we will see that this stronger linearity also leads to faster convergence.
3.2 Effect of stronger mixup on convergence
In the previous section, we saw that stronger enforcement of linearity, by using more mixup points, led to the network layers becoming more linear.
We will now empirically show the effect of this linearity on network convergence.
Speciﬁcally, we measure test accuracy given a speciﬁed number of labeled training examples.
For this, we will investigate the performance of different baselines in a semi-supervised setup for varying amounts of labeled data.
Results Figure 2 shows the accuracy on a ﬁxed test set against various labeled dataset sizes for
CIFAR-10 (the results for CIFAR-100 are given in the appendix).
From the Figure, we can see that the ICT-based model generally outperforms the ERM and mixup baselines across all dataset sizes.
This is likely due to the additional unsupervised training performed on the additional unsupervised dataset.
Among the ICT variants we further observe that the test accuracy tends to be higher as we increase the number of points used in the mixup operation.
This is especially true in the regime of very small datasets and the performance of all ICT variants converges to similar performance as more datapoints become available.
For example, to reach 70 % test accuracy on CIFAR-10, mixup requires roughly 1800 labeled examples, whereas ICT4 only requires 500 labeled examples (even less for ICT5).
The stronger regularization achieved by using 4 mixup points leads to the network becoming more linear, which results in more efﬁcient learning from fewer labeled samples.
Conclusion
In this work, we explored when does (or doesn’t) mixup enforce local linearity in the learned representations.
We studied this question in a semi-supervised learning setup for image classiﬁcation tasks.
Our experiments demonstrate that standard supervised mixup training doesn’t make the representations in all the layers locally linear.
In fact, some of the intermediate layers become more
[Figure]
non-linear during supervised mixup training compared to standard empirical risk minimization (ERM) training.
However, when mixup operation is used as an unsupervised loss, on a larger unlabeled dataset, network representations in all layers become more linear.
As a consequence of these smoother representations, the network converges to a given test set performance faster (in terms of labeled samples) than a network that is locally less linear.
These ﬁndings, which we veriﬁed empirically, sheds new light on the training dynamics of a network trained under mixup regularization.
In this appendix we provide more detailed results for various sizes of labeled dataset.
The results here further strengthen the claims made in the main paper.
A More Results
[Figure]
ERM and mixup refer to the cases where only supervised loss is used.
ICT[X] refer to a setup where the ICT loss with X mixup points are used as unsupervised loss.
The supervised loss with the ICT does not use mixup.
[Figure]
a r i t
[Figure]
o n - L i n e a r i t y ( 3 )
[Figure]
(d) Layer
[Figure] [Figure]
N o n - L i n e a r i t y ( 5 )
ERMMIXUPICT2ICT3ICT (f) Layer
[Figure]
i n
[Figure]
a r i t y
[Figure]
[Figure]
ERMMIXUPICT2ICT3ICT (f) Layer
[Figure] [Figure]