H OW D OES M IXUP H ELP W ITH R OBUSTNESS AND G ENERALIZATION ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Mixup is a popular data augmentation technique based on taking convex combinations of pairs of examples and their labels.
This simple technique has been shown to substantially improve both the robustness and the generalization of the trained model.
However, it is not well-understood why such improvement occurs.
In this paper, we provide theoretical analysis to demonstrate how using Mixup in training helps model robustness and generalization.
For robustness, we show that minimizing the Mixup loss corresponds to approximately minimizing an upper bound of the adversarial loss.
This explains why models obtained by Mixup training exhibits robustness to several kinds of adversarial attacks such as Fast Gradient Sign
Method (FGSM).
For generalization, we prove that Mixup augmentation corresponds to a speciﬁc type of data-adaptive regularization which reduces overﬁtting.
Our analysis provides new insights and a framework to understand Mixup.
I NTRODUCTION
Mixup was introduced by Zhang et al.
as a data augmentation technique.
It has been empirically shown to substantially improve test performance and robustness to adversarial noise of state-of-the-art neural network architectures (Zhang et al., 2018; Lamb et al., 2019; Thulasidasan et al., 2019; Zhang et al., 2018; Arazo et al., 2019).
Despite the impressive empirical performance, it is still not fully understood why Mixup leads to such improvement across the different aspects mentioned above.
We ﬁrst provide more background about robustness and generalization properties of deep networks and Mixup.
Then we give an overview of our main contributions.
Adversarial robustness.
Although neural networks have achieved remarkable success in many areas such as natural language processing (Devlin et al., 2018) and image recognition (He et al., 2016a), it has been observed that neural networks are very sensitive to adversarial examples — prediction can be easily ﬂipped by human imperceptible perturbations (Goodfellow et al., 2014; Szegedy et al., 2013).
Speciﬁcally, in Goodfellow et al.
(2014), the authors use fast gradient sign method (FGSM) to generate adversarial examples, which makes an image of panda to be classiﬁed as gibbon with high conﬁdence.
Although various defense mechanisms have been proposed against adversarial attacks, those mechanisms typically sacriﬁce test accuracy in turn for robustness (Tsipras et al., 2018) and many of them require a signiﬁcant amount of additional computation time.
In contrast,
Mixup training tends to improve test accuracy and at the same time also exhibits a certain degree of resistance to adversarial examples, such as those generated by FGSM (Lamb et al., 2019).
Moreover, the corresponding training time is relatively modest.
As an illustration, we compare the robust test accuracy between a model trained with Mixup and a model trained with standard empirical risk minimization (ERM) under adversarial attacks generated by FGSM (Fig. 1a).
The model trained with Mixup loss has much better robust accuracy.
Robustness of Mixup under other attacks have also been empirically studied in Lamb et al.
Generalization.
Generalization theory has been a central focus of learning theory (Vapnik, 1979; 2013; Bartlett et al., 2002; Bartlett & Mendelson, 2002; Bousquet & Elisseeff, 2002; Xu & Mannor, 2012), but it still remains a mystery for many modern deep learning algorithms (Zhang et al., 2016;
Kawaguchi et al., 2017).
For Mixup, from Fig.
(1b), we observe that Mixup training results in better test performance than the standard empirical risk minimization.
That is mainly due to its good generalization property since the training errors are small for both Mixup training and empirical risk minimization (experiments with training error results are included in the appendix).
While there
[Figure] [Figure]
400 epoch 100.3 0.
t e s t l o ss mixup
ERM    400 epoch 100.1 0.10.10.10.
G e n e r a li z a t i
o n g a p mixup
ERM
[Figure]
have been many enlightening studies trying to establish generalization theory for modern machine learning algorithms (Sun et al., 2015; Neyshabur et al., 2015; Hardt et al., 2016; Bartlett et al., 2017; Kawaguchi et al., 2017; Arora et al., 2018; Neyshabur & Li, 2019), few existing studies have illustrated the generalization behavior of Mixup training in theory.
Our contributions.
In this paper, we theoretically investigate how Mixup improves both adversarial robustness and generalization.
We begin by relating the loss function induced by Mixup to the standard loss with additional adaptive regularization terms.
Based on the derived regularization terms, we show that Mixup training minimizes an upper bound on the adversarial loss,which leads to the robustness against single-step adversarial attacks.
For generalization, we show how the regularization terms can reduce over-ﬁtting and lead to better generalization behaviors than those of standard training.
Our analyses provides insights and framework to understand the impact of Mixup.
Outline of the paper.
Section 2 introduces the notations and problem setup.
In Section 3, we present our main theoretical results, including the regularization effect of Mixup and the subsequent analysis to show that such regularization improves adversarial robustness and generalization.
Section 4 concludes with a discussion of future work.
Proofs are deferred to the Appendix.
1. R ELATED WORK
Since its advent, Mixup training (Zhang et al., 2018) has been shown to substantially improve generalization and single-step adversarial robustness among a wide rage of tasks, on both supervised (Lamb et al., 2019; Verma et al., 2019a; Guo et al., 2019), and semi-supervised settings (Berthelot et al., 2019; Verma et al., 2019b).
This has motivated a recent line of work for developing a number of variants of Mixup, including Manifold Mixup (Verma et al., 2019a), Puzzle Mix (Kim et al., 2020), CutMix (Yun et al., 2019), Adversarial Mixup Resynthesis (Beckham et al., 2019), and PatchUp (Faramarzi et al., 2020).
However, theoretical understanding of the underlying mechanism of why Mixup and its variants perform well on generalization and adversarial robustness is still limited.
Some of the theoretical tools we use in this paper are related to Wang & Manning (2013) and Wager et al.
(2013), where the authors use second-order Taylor approximation to derive a regularized loss function for Dropout training.
This technique is then extended to drive more properties of Dropout, including the inductive bias of Dropout (Helmbold & Long, 2015), the regularization effect in matrix factorization (Mianjy et al., 2018), and the implicit regularization in neural networks (Wei et al., 2020).
This technique has been recently applied to Mixup in a parallel and independent work (Carratino et al., 2020) to derive regularization terms.
Compared with the results in Carratino et al.
(2020), our derived regularization enjoys a simpler form and therefore enables the subsequent analysis of adversarial robustness and generalization.
We clarify the detailed differences in Section 3.
To the best of our knowledge, our paper is the ﬁrst to provide a theoretical treatment to connect the regularization, adversarial robustness, and generalization for Mixup training.
P RELIMINARIES
In this section, we state our notations and brieﬂy recap the deﬁnition of Mixup.
Notations.
We denote the general parameterized loss as l ( θ, z ) , where θ ∈ Θ ⊆ R d and z = ( x, y ) is the input and output pair.
We consider a training dataset S =
{ ( x 1 , y 1 ) , · · · , ( x n , y n ) } , where x i ∈
X ⊆ R p and y i ∈ Y ⊆
R m are i.i.d.
drawn from a joint distribution P x,y .
We further denote ˜ x i,j ( λ ) =
λx i + (1 − λ ) x j , ˜ y
i,j ( λ ) = λy i + (1 − λ )
y j for λ ∈ [0 , 1] and let ˜ z
i,j ( λ ) =
(˜ x i,j ( λ ) , ˜ y
i,j ( λ )) .
Let L ( θ ) =
E
z ∼P x,y l ( θ, z ) denote the standard population loss and
L stdn ( θ, S ) =
(cid:80) ni
=1 l ( θ, z i ) /n denote the standard empirical loss.
For the two distributions D 1 and
D 2 , we use p D 1 + (1 − p ) D 2 for p ∈ (0 , 1) to denote the mixture distribution such that a sample is drawn with probabilities p and (1 − p ) from D 1 and D 2 respectively.
For a parameterized function f θ ( x ) , we use ∇ f θ ( x ) and ∇ θ f θ ( x ) to respectively denote the gradient with respect to x and θ .
For two vectors a and b , we use cos ( x, y ) to denote (cid:104) x, y (cid:105) /
( (cid:107) x (cid:107) · (cid:107) y (cid:107) ) .
Mixup.
Generally, for classiﬁcation cases, the output y
i is the embedding of the class of x i , i.e. the one-hot encoding by taking m as the total number of classes and letting y i ∈ { 0 , 1 } m be the binary vector with all entries equal to zero except for the one corresponding to the class of x i .
In particular, if we take m = 1 , it degenerates to the binary classiﬁcation.
For regression cases, y i can be any real number/vector.
The Mixup loss is deﬁned in the following form:
[Equation]
where D λ is a distribution supported on [0 , 1] .
Throughout the paper, we consider the most commonly used D λ –
Beta distribution Beta ( α, β ) for α, β > 0 .
M AIN R ESULTS
In this section, we ﬁrst introduce a lemma that characterizes the regularization effect of Mixup.
Based on this lemma, we then derive our main theoretical results on adversarial robustness and generalization error bound in Sections 3.2 and 3.3 respectively.
3. T HE REGULARIZATION EFFECT OF M IXUP
As a starting point, we demonstrate how Mixup training is approximately equivalent to optimizing a regularized version of standard empirical loss L stdn ( θ, S ) .
Throughout the paper, we consider the following class of loss functions for the prediction function f θ ( x ) and target y :
[Equation]
This function class L includes many commonly used losses, including the loss function induced by
Generalized Linear Models (GLMs), such as linear regression and logistic regression, and also crossentropy for neural networks.
In the following, we introduce a lemma stating that the Mixup training with λ ∼ D λ
=
Beta ( α, β ) induces a regularized loss function with the weights of each regularization speciﬁed by a mixture of Beta distributions
˜ D
λ
= αα + β Beta ( α + 1 , β ) + βα + β Beta ( β + 1 , α ) .
Lemma 3.1.
Consider the loss function l ( θ, ( x, y ))
=
h ( f θ ( x )) − yf θ ( x ) , where h ( · ) and f θ ( · ) for all θ ∈ Θ are twice differentiable.
We further denote ˜ D
λ as a uniform mixture of two Beta distributions, i.e., αα + β Beta ( α + 1 , β ) + βα + β Beta ( β + 1 , α ) , and D X as the empirical distribution of the training dataset S =
( x 1 , · · · , x n ) , the corresponding Mixup loss L mix n ( θ, S ) , as deﬁned in
Eq.
(1) with λ ∼ D λ =
Beta ( α, β ) , can be rewritten as
[Equation]
where lim a → 0 ϕ ( a ) = 0 and
[Equation]
[Equation]
By putting the higher order terms of approximation in ϕ ( · ) , this result shows that Mixup is related to regularizing ∇ f θ ( x i ) and ∇ 2 f θ ( x i ) , which are the ﬁrst and second directional derivatives with respect to x i .
Throughout the paper, our theory is mainly built upon analysis of the quadratic approximation of L mix n ( θ, S ) , which we further denote as
[Equation]
Comparison with related work.
The result in Lemma 3.1 relies on the second-order Taylor expansion of the loss function Eq.
Similar approximations have been proposed before to study the regularization effect of Dropout training, see Wang & Manning (2013); Wager et al.
; Mianjy et al.
; Wei et al.
Recently, Carratino et al.
(2020) independently used similar approximation to study the regularization effect of Mixup.
However, the regularization terms derived in
Carratino et al.
(2020) is much more complicated than those in Lemma 3.1.
For example, in GLM, our technique yields the regularization term as shown in Lemma 3.3, which is much simpler than those in Corollaries 2 and 3 in Carratino et al.
One technical step we use here to simplify the regularization expression is to equalize Mixup with input perturbation, see more details in the proof in the Appendix.
This simpler expression enables us to study the robustness and generalization of
Mixup in the subsequent sections.
Validity of the approximation.
In the following, we present numerical experiments to support the approximation in Eq.
Following the setup of numerical validations in Wager et al.
Carratino et al.
(2020), we experimentally show that the quadratic approximation is generally very accurate.
Speciﬁcally, we train a Logistic Regression model (as one example of a GLM model, which we study later) and a two layer neural network with ReLU activations.
We use the two-moons dataset (Buitinck et al., 2013).
Fig.
2 shows the training and test data’s loss functions for training two models with different loss functions: the original Mixup loss and the approximate Mixup loss.
Both models had the same random initialization scheme.
Throughout training, we compute the test and training loss of each model using its own loss function.
The empirical results shows the approximation of Mixup loss is quite close to the original Mixup loss.
[Figure]
3. M IXUP AND A DVERSARIAL R OBUSTNESS
Having introduced ˜ L mix n ( θ, S ) in Eq.
(3), we are now ready to state our main theoretical results.
In this subsection, we illustrate how Mixup helps adversarial robustness.
We prove that minimizing ˜
L mix n ( θ, S ) is equivalent to minimizing an upper bound of the second order Taylor expansion of an adversarial loss.
Throughout this subsection, we study the logistic loss function
[Equation]
where y ∈ Y = { 0 , 1 }
In addition, let g be the logistic function such that g ( s ) =
e s / (1 + e s ) and consider the case where θ is in the data-dependent space Θ , deﬁned as
[Equation]
Notice that Θ contains the set of all θ with zero training errors: Θ ⊇ { θ ∈ R q : the label prediction ˆ y i = 1 { f θ ( x i ) ≥ 0 } is equal to y i for all i = 1 , . .
. , n } .
In many practical cases, the training error ( 0 - 1 loss) becomes zero in ﬁnite time although the training loss does not.
Equation (4) shows that the condition of θ ∈ Θ is satisﬁed in ﬁnite time in such practical cases with zero training errors.
Logistic regression.
As a starting point, we study the logistic regression with f θ ( x ) = θ (cid:62) x , in which case the number of parameters coincides with the data dimension, i.e. p = d .
For a given ε > 0 , we consider the adversarial loss with (cid:96) 2 -attack of size ε √ d , that is, L advn ( θ, S )
= 1 /n (cid:80) n i =1 max (cid:107) δ i (cid:107) 2 ≤ ε √ d l ( θ, ( x i + δ i , y i )) .
We ﬁrst present the following second order Taylor approximation of L advn ( θ, S ) .
Lemma 3.2.
The second order Taylor approximation of L advn ( θ, S ) is (cid:80) ni
=1 ˜ l adv ( ε √ d, ( x i , y i ))
/n , where for any η > 0 , x ∈ R p and y ∈ { 0 , 1 } ,
[Equation]
adv
[Equation]
By comparing ˜ l adv ( δ, ( x, y )) and ˜ L mix n ( θ, S ) applied to logistic regression, we prove the following.
Theorem 3.1.
Suppose that f θ ( x ) = x (cid:62) θ
and there exists a constant c x > 0 such that (cid:107) x
i (cid:107) 2 ≥ c x √ d for all i ∈ { 1 , . . . , n } .
Then, for any θ ∈ Θ , we have
[Equation]
where ε i =
R i c x E λ ∼ ˜
D
λ
[1 − λ ] with R i =
| cos( θ, x i ) | , and ε mix =
R · c x E λ ∼ ˜ D
λ [1 − λ ] with
R = min i ∈{ 1 ,...,n } | cos( θ, x i ) | .
Theorem 3.1 suggests that ˜ L mix n ( θ, S ) is an upper bound of the second order Taylor expansion of the adversarial loss with (cid:96) 2 -attack of size
ε mix √ d .
Notice ε mix depends on θ , one can think the ﬁnal radius is taken at the minimizer of ˜ L mix n ( θ, S ) .
Therefore, minimizing the Mixup loss would result in a small adversarial loss.
Such proposition leads to the robustness against single-step attacks, which may be interesting for future work to further explore, for instance, to study whether combining
Mixup with adversarial training is able to design efﬁcient adversarial training mechanism against more sophisticated attacks such as attacks generated by iterative projected gradient descent.
In the following we provide more discussion about the range of R = min i ∈{ 1 ,...,n } | cos( θ, x i ) | .
We ﬁrst show that under additional regularity conditions, we can obtain a high probability lower bound that does not depend on sample size.
We then numerically demonstrate that R tends to increase during training for both cases of linear models and neural networks at the end of this subsection.
A constant lower bound for logistic regression.
Now, we show how to obtain a constant lower bound by adding some additional conditions.
Assumption 3.1.
Let us denote ˆΘ n ⊆
Θ as the set of minimizers of ˜ L mix n ( θ, S ) .
We assume there exists a set Θ ∗ 1 , such that for all n ≥ N , where N is a positive integer, ˆΘ n ⊆
Θ ∗ with probability at least 1 − δ n where δ n → 0 as n → 0 .
Moreover, there exists a τ ∈ (0 , 1) such that
[Equation]
[Figure]
Such condition generally holds for regular optimization problems.
Suppose x is uniformly distributed in a ball which also contains Θ , and Θ ∗ is a small neighborhood of minimizers of ˜ L mix n ( θ, S ) , as long as the minimizers are not located too dispersedly, the above condition holds (one can think of the extreme case where there is a unique minimizer and the condition trivially holds).
Theorem 3.2.
Under Assumption 3.1, for f θ ( x ) = x (cid:62) θ , if there exists constants b x , c x > 0 such that c x √ d ≤ (cid:107) x i (cid:107) 2 ≤ b x √ d for all i ∈ { 1 , . . . , n } .
Then, with probability at least 1 − δ n − 2 exp( − np 2 τ / 2) , there exists constants κ > 0 , κ 2 > κ 1 > 0 , such that for any θ ∈ ˆΘ n , we have
[Equation]
where ˜ ε mix = ˜
Rc x E λ ∼ ˜ D
λ [1 − λ ] and ˜ R = min (cid:110)
[Equation]
Neural networks with ReLU / Max-pooling.
The results in the above subsection can be extended to the case of neural networks with ReLU activation functions and max-pooling.
Speciﬁcally, we consider the logistic loss, l ( θ, z )
= log(1 + exp( f θ ( x )))
− yf θ ( x ) with y ∈ { 0 , 1 } , where f θ ( x ) represents a fully connected neural network with ReLU activation function or max-pooling:
[Equation]
Here, σ represents nonlinearity via ReLU and max pooling, each W i is a matrix, and β is a column vector: i.e., θ consists of { W i } N − 1 i
=1 and β .
With the nonlinearity σ for ReLU and max-pooling, the function f θ satisﬁes that f θ ( x ) = ∇ f θ ( x ) (cid:62) x and ∇ 2 f θ ( x ) =
almost everywhere , where the gradient is taken with respect to input x .
Under such conditions, similar to Lemma 3.2, the adversarial loss function (cid:80) n i =1 max
[Equation]
i
[Equation]
2 ≤ ε √ d l ( θ, ( x i + δ i , y i ))
/n can be written as
[Equation]
With a little abuse of notations, we also denote
[Equation]
The following theorem suggests that minimizing the Mixup loss in neural nets also lead to a small adversarial loss.
Theorem 3.3.
Assume that f θ ( x i ) =
∇ f θ ( x i ) (cid:62) x i , ∇ 2 f θ ( x i )
=
and there exists a constant c x >such that (cid:107) x
i (cid:107) 2 ≥ c x √ d for all i ∈ { 1 , . . . , n } .
Then, for any θ ∈ Θ , we have
[Equation]
where ε i =
R i c x E λ ∼ ˜
D
λ
[1 − λ ] , ε mix =
R · c x E λ ∼ ˜ D
λ [1 − λ ] and R i = | cos( ∇ f θ ( x i ) , x i ) | , R = min i ∈{ 1 ,...,n } | cos( ∇ f θ ( x i ) , x i ) | .
Similar constant lower bound can be derived to the setting of neural networsk.
Due to limited space, please see the detailed discussion in the appendix.
On the value of R = min i R i via experiments.
For both linear models and neural networks, after training accuracy reaches 100%, the logistic loss is further minimized when (cid:107) f θ ( x i ) (cid:107) 2 increases.
Since (cid:107) f θ ( x i ) (cid:107)
= (cid:107)∇ f θ ( x i ) (cid:62) x i (cid:107)
=
(cid:107)∇ f θ ( x i ) (cid:107) 2 (cid:107) x
i (cid:107) 2 R i , this suggests that R i and R tend to increase after training accuracy reaches 100% (e.g., ∇ f θ ( x i )
=
θ in the case of linear models).
We conﬁrm this phenomenon in Fig.
In the ﬁgure, R is initially small but tends to increase after training accuracy reaches 100%, as expected.
For example, for ANN, the value of R was initially 2 .
27 × 10 − 5 but increased to 6 .
11 × 10 − 2 after training.
Fig.
3 (c) and (d) also show that R i for each i -th data point tends to increase during training and that the values of R i for many points are much larger than the pessimistic lower bound R :
e.g., whereas R = 6 . 11 × 10 − 2 , we have R i > 0 .
8 for several data points in Fig.
3 (d).
For this experiment, we generated 100 data points as x i ∼ N (0 , I ) and y i
=
1 { x (cid:62) i θ ∗ > 0 } where x i ∈ R 10 and θ ∗ ∼ N (0 , I ) .
We used SGD to train linear models and ANNs with ReLU activations and 50 neurons per each of two hidden layers.
We set the learning rate to be 0.1 and the momentum coefﬁcient to be 0.9.
We turned off weight decay so that R is not maximized as a result of bounding (cid:107)∇ f θ ( x i ) (cid:107) , which is a trivial case from the discussion above.
3. M IXUP AND G ENERALIZATION
In this section, we show that the data-dependent regularization induced by Mixup directly controls the Rademacher complexity of the underlying function classes, and therefore yields concrete generalization error bounds.
We study two models – the Generalized Linear Model (GLM) and two-layer
ReLU nets with squared loss.
Generalized linear model.
A Generalized Linear Model is a ﬂexible generalization of ordinary linear regression, where the corresponding loss takes the following form:
[Equation]
where A ( · ) is the log-partition function, x ∈ R p and y ∈ R .
For instance, if we take A ( θ (cid:62) x ) = log(
+ e θ (cid:62) x ) and y ∈ { 0 , 1 } , then the model corresponds to the logistic regression.
In this paragraph, we consider the case where Θ , X and Y are all bounded.
By further taking advantage of the property of shift and scaling invariance of GLM, we can further simplify the regularization terms in Lemma 3.1 and obtain the following results.
Lemma 3.3.
Consider the centralized dataset S , that is, 1 /n (cid:80) ni
=1 x i = 0 .
and denote ˆΣ X
=
n x i x (cid:62) i .
For a GLM, if A ( · ) is twice differentiable, then the regularization term obtained by the second-order approximation of ˜ L mix n ( θ, S ) is given by
[Equation]
where ˜ D
λ
= αα + β Beta ( α + 1 , β ) + αα + β Beta ( β + 1 , α ) .
Given the above regularization term, we are ready to investigate the corresponding generalization gap.
Following similar approaches in Arora et al.
, we shed light upon the generalization problem by investigating the following function class that is closely related to the dual problem of
Eq.
[Equation]
where α > 0 and Σ X =
E
[ x i x (cid:62) i ] .
Further, we assume that the distribution of x is ρ -retentive for some ρ ∈ (0 , 1 / 2] , that is, if for any non-zero vector v ∈ R d , (cid:2) E
x [ A (cid:48)(cid:48) ( x (cid:62) v )] (cid:3) 2 ≥ ρ · min { 1 , E x
( v (cid:62) x ) 2 } .
Such an assumption has been similarly assumed in Arora et al.
and is satisﬁed by general GLMs when θ has bounded (cid:96) 2 norm.
We then have the following theorem.
Theorem 3.4.
Assume that the distribution of x i is ρ -retentive, and let
Σ X =
E
[ xx (cid:62) ] .
Then the empirical Rademacher complexity of W γ satisﬁes
[Equation]
The above bound on Rademacher complexity directly implies the following generalization gap of
Mixup training.
Corollary 3.1.
Suppose A ( · ) is L A -Lipchitz continuous, X , Y and Θ are all bounded, then there exists constants L, B > 0 , such that for all θ satisfying E x A
(cid:48)(cid:48)
( θ
[Equation]
x ) ·
[Equation]
Σ X θ
[Equation]
γ (the regularization induced by Mixup), we have
[Equation]
with probability at least 1 − δ .
Remark 3.1.
This result shows that the Mixup training would adapt to the intrinsic dimension of x and therefore has a smaller generalization error.
Speciﬁcally, if we consider the general ridge penalty and consider the function class W ridgeγ :
= { x → θ
[Equation]
θ
[Equation]
2 (cid:54) γ } , then the similar technique would yield a Rademacher complexity bound Rad (
[Equation]
(cid:112) p/n , where p is the dimension of x .
This bound is much larger than the result in Theorem 3.
when the intrinsic dimension rank (Σ X ) is small.
Non-linear cases.
The above results on GLM can be extended to the non-linear neural network case with Manifold Mixup (Verma et al., 2019a).
In this section, we consider the two-layer ReLU neural networks with the squared loss L ( θ, S )
= 1 n (cid:80) ni
=
( y i − f θ ( x i ))
2 , where y ∈ R and f θ ( x ) is a two-layer ReLU neural network, with the form of
[Equation]
where W ∈ R p ×
d , θ 1 ∈ R d , and θ 0 denotes the bias term.
Here, θ consists of W , θ 0 and θ 1 .
If we perform Mixup on the second layer (i.e., mix neurons on the hidden layer as proposed by
Verma et al. (2019a)), we then have the following result on the induced regularization.
Lemma 3.4.
Denote ˆΣ σX as the sample covariance matrix of { σ ( Wx i ) } ni =1 , then the regularization term obtained by the second-order approximation of ˜ L mix n ( θ, S ) is given by
[Equation]
where ˜ D λ ∼ αα + β Beta
( α + 1 , β ) βα + β Beta ( β + 1 , α ) .
To show the generalization property of this regularizer, similar to the last section, we consider the following distribution-dependent class of functions indexed by θ :
[Equation]
that
[Equation]
where Σ σX = E [ˆΣ σX ] and α > 0 .
We then have the following result.
Theorem 3.5.
Let µ σ = E
[ σ ( Wx )] and denote the generalized inverse of Σ σX by Σ σ † X .
Suppose X ,
Y and Θ are all bounded, then there exists constants L, B > 0 , such that for all f θ in W NN γ (the regularization induced by Manifold Mixup), we have, with probability at least 1 − δ ,
[Equation]
log
[Equation]
δ
[Equation]
2 n .
C ONCLUSION AND F UTURE W ORK
Mixup is a data augmentation technique that generates new samples by linear interpolation of multiple samples and their labels.
The Mixup training method has been empirically shown to have better generalization and robustness against attacks with adversarial examples than the traditional training method, but there is a lack of rigorous theoretical understanding.
In this paper, we prove that the
Mixup training is approximately a regularized loss minimization.
The derived regularization terms are then used to demonstrate why Mixup has improved generalization and robustness against onestep adversarial examples.
One interesting future direction is to extend our analysis to other Mixup variants, for example,
Puzzle Mix
(Kim et al., 2020) and Adversarial Mixup Resynthesis (Beckham et al., 2019), and investigate if the generalization performance and adversarial robustness can be further improved by these newly developed Mixup methods.
Appendix
In this appendix, we provide proofs of the main theorems and the corresponding technical lemmas.
Additional discussion on the range of R in the case of neural nets, and some further numerical experiments are also provided.
A T ECHNIQUE P ROOFS A. P ROOF OF L EMMA 3.
Consider the following problem with loss function l x
[Equation]
y ( θ ) :
= l ( θ, ( x, y ))
=
h ( f θ ( x )) − yf θ ( x )
[Equation]
that is
[Equation]
The corresponding Mixup version, as deﬁned in Eq.(1)
[Equation]
is
[Equation] [Equation]
where ˜ x i,j ( λ )
=
λx
i + (1 − λ ) x j .
Further transformation leads to
[Equation]
Note that λ ∼ Beta ( α, β ) , B | λ ∼ Bern ( λ ) , by conjugacy, we can exchange them in order and have
[Equation]
As a result,
[Equation]
Using the fact 1 − Beta ( α, β +1) and Beta ( β +1 , α ) are of the same distribution and ˜ x ij (1 − λ ) =
˜ x ji ( λ ) , we have (cid:88) i,j
[Equation] [Equation]
˜
[Equation]
i
[Equation]
λ )))
−
[Equation]
˜
[Equation]
i
[Equation]
λ ))]
= (cid:88)
[Equation]
j
[Equation]
( β
[Equation] [Equation]
[ h ( f θ (˜ x i,j ( λ ))) − y i
f θ (˜ x i,j ( λ ))] .
Thus, let ˜ D
λ
= αα + β Beta ( α + 1 , β ) + βα + β Beta ( β + 1 , α )
L mix n ( θ, S ) = 1 n n (cid:88) i =
E λ ∼ ˜ D
λ E r x ∼ D x h ( f ( θ, λx i + (1 − λ ) r x )))
− y i f
( θ, λx i + (1 − λ ) r x ) =
[Equation] [Equation]
where D x is the empirical distribution induced by training samples, and ˇ x i = λx
i + (1 − λ ) r x .
In the following, denote ˇ S
= { (ˇ x i , y i ) } ni =1 , and let us analyze L stdn ( θ, ˇ S )
= 1 n (cid:80) ni
=
l ˇ x i ,y i ( θ ) , and compare it with L stdn ( θ, S ) .
Let α = 1 − λ and ψ i ( α ) =
l ˇ x i ,y i ( θ ) .
Then, using the deﬁnition of the twice-differentiability of function ψ i ,
[Equation]
where lim z → 0 ϕ
i ( z ) = 0 .
By linearity and chain rule,
[Equation]
where we used ∂ ˇ x i ∂α =
( r x − x i ) .
Since
[Equation]
we have
[Equation]
Thus
[Equation]
By substituting these into equation 9 with ϕ ( α ) = 1 n (cid:80) ni =
1 ϕ i ( α ) , we obtain the desired statement.
A. P ROOFS RELATED TO ADVERSARIAL ROBUSTNESS A.2. P ROOF OF L EMMA 3.
Recall that L advn ( θ, S )
=
1 n (cid:80) ni =1 max (cid:107) δ
i (cid:107) 2 ≤ ε √ d l ( θ, ( x i
+ δ i , y i )) and g ( u )
=
1 / (1 + e − u ) .
Then the second-order Taylor expansion of l ( θ, ( x + δ, t )) is given by
[Equation]
[Equation]
Consequently, for any given η > 0 ,
[Equation]
where the maximum is taken
when δ = sgn( g ( x (cid:62)
[Equation]
) − y )
[Equation]
θ
[Equation]
· η .
A.2. P ROOF OF T HEOREM 3.
Since f θ ( x ) = x (cid:62) θ , we have ∇ f θ ( x i )
=
θ and ∇ 2 f θ ( x i ) = 0 .
Since h ( z ) = log(1 + e z ) , we have h (cid:48)
( z ) =
e z 1+ e z = g ( z ) ≥ 0 and h (cid:48)(cid:48)
( z ) =
e z (1+ e z )
= g ( z )(1 − g
( z )) ≥ 0 .
By substituting these into the equation of Lemma 3.1 with E r x
[ r x ] = 0 ,
[Equation] [Equation]
where
[Equation]
θ where we used E [ z 2 ] =
E [ z ] 2 + Var( z ) ≥ E [ z ] 2 and θ (cid:62) E r x
[( r x − x i )( r x − x i ) (cid:62) ] θ ≥ 0 .
Since E r x [
[Equation]
E
[Equation]
x [
[Equation]
r (cid:62)
[Equation]
x (cid:62)
[Equation]
i ] =
E r x
[ r x r
[Equation]
x ] + x i x (cid:62) i where E r x [ r x r (cid:62) x ] is positive semideﬁnite,
[Equation]
)≥
[Equation]
Now we bound E =
E λ
[(1 − λ )] n (cid:80) ni
=
( y i − g ( x (cid:62) i θ ))
( θ (cid:62) x i ) by using θ ∈ Θ .
Since θ ∈ Θ , we have y i f
θ ( x i )
+ ( y i − 1)
f θ ( x i ) ≥ 0 , which implies that ( θ (cid:62) x i ) ≥
if y i = 1 and ( θ (cid:62) x i ) ≤ 0 if y i = 0 .
Thus, if y i = 1 , ( y i − g ( x (cid:62) i θ ))
( θ (cid:62) x i ) =
(1 − g ( x (cid:62) i θ ))
( θ (cid:62) x i ) ≥ 0 , since ( θ (cid:62) x i ) ≥ 0 and (1 − g ( x (cid:62) i θ )) ≥
due to g ( x (cid:62) i θ ) ∈ (0 , 1) .
If y i = 0 , ( y i − g ( x (cid:62) i θ
[Equation]
since ( θ (cid:62) x i ) ≤ 0 and − g ( x (cid:62) i θ ) < 0 .
Therefore, for all i = 1 , . . . , n ,
[Equation]
y i
[Equation]
g
[Equation]
i θ ))
[Equation]
x i ) ≥ 0 , which implies that, since E λ [(1 − λ )] ≥ 0 ,
[Equation]
By substituting these lower bounds of R 1 ( θ, S ) and R 2 ( θ, S ) into equation 10, we obtain the desired statement.
A.2. P ROOF OF T HEOREM 3.
Recall we assume that ˆ θ n will fall into the set Θ ∗ with probability at least 1 − δ n , and δ n → 0 as n → ∞ .
In addition, deﬁne the set
X Θ ∗ ( τ ) = { x ∈ X : | cos( x, θ ) | (cid:62) τ for all θ ∈ Θ ∗ }
, there is τ ∈ (0 , 1) such that X Θ ∗ ( τ ) (cid:54) = ∅ , and
[Equation]
Let us ﬁrst study
[Equation]
Since we assume Θ ∗ is bounded and c x √ d (cid:54) (cid:107) x i (cid:107) 2 (cid:54) b x √ d for all i , there exists κ > 0 , such that
[Equation]
If we denote ˆ p = { number of x (cid:48)
i s such that x
i ∈
X Θ ∗ ( τ ) } /n .
Then, it is easy to see
[Equation]
For η 2 satisfying
[Equation]
we have
[Equation]
Lastly by Hoeffding’s inequality
[Equation]
if we take ε = p τ /
[Equation]
with probability at least 1 − 2 exp( − 2 nε 2 )
[Equation]
Similarly, if we study
[Equation]
By boundedness of θ
[Equation]
x and y ∈ {
[Equation]
[Equation]
we know there are constants κ 1 , κ 2 > 0 , such that
[Equation]
Similarly, we know
[Equation]
Combined together, we can obtain the result:
[Equation]
A.2. P ROOF OF T HEOREM 3.
From the assumption, we have f θ ( x i ) =
∇ f θ ( x i ) (cid:62) x i and ∇ 2 f θ ( x i )
= 0 .
Since h ( z ) = log(1+ e z ) , we have h (cid:48)
( z ) =
e z 1+ e z = g ( z ) ≥ 0 and h (cid:48)(cid:48)
( z ) =
e z (1+ e z )
= g ( z )(1 − g
( z )) ≥ 0 .
By substituting these into the equation of Lemma 3.1 with E r x
[ r x ] = 0 , ˜ L mix n ( θ, S
[Equation]
=
˜ L mix n ( θ, S )
+ R 1 ( θ,
[Equation]
2 ( θ, S
[Equation]
, (11) where
[Equation]
where we used E [ z 2 ]
=
E [ z ] 2 +Var( z ) ≥ E [ z ] 2 and ∇ f θ ( x i ) (cid:62) E r x
[( r x − x i )( r x − x i ) (cid:62) ] ∇ f θ ( x i ) ≥ 0 .
Since E r x [( r x − x i )( r x − x i ) (cid:62) ] =
E r x
[ r x r (cid:62) x − r x x (cid:62) i − x i r (cid:62) x
+ x i x (cid:62) i ] =
E r x
[ r x r (cid:62) x ] + x i x (cid:62) i where E r x [ r x r (cid:62) x ] is positive semideﬁnite,
R
[Equation] [Equation]
[Equation]
Now we bound E =
E λ
[(1 − λ )] n (cid:80) ni
=
( y i − g
( f θ ( x i )))
f θ ( x i ) by using θ ∈ Θ .
Since θ ∈ Θ , we have y i f
θ ( x i )
+ ( y i − 1)
f θ ( x i ) ≥ 0 , which implies that f θ ( x i ) ≥
if y i = 1 and f θ ( x i ) ≤ 0 if y i = 0 .
Thus, if y i = 1 , ( y i − g
( f θ ( x i ))) ( f θ ( x i ))
= (1 − g
( f θ ( x i )))
( f θ ( x i )) ≥ 0 , since ( f θ ( x i )) ≥ 0 and (1 − g ( f θ ( x i ))) ≥ 0 due to g ( f θ ( x i )) ∈ (0 , 1) .
If y i = 0 , ( y i − g
( f θ ( x i ))) ( f θ ( x i ))
= − g
( f θ ( x i ))
( f θ ( x i )) ≥ 0 , since ( f θ ( x i )) ≤ 0 and − g ( f θ ( x i )) < 0 .
Therefore, for all i = 1 , . . . , n , ( y i − g
( f θ ( x i )))
( f θ ( x i )) ≥ 0 , which implies that, since E λ [(1 − λ )] ≥ 0 ,
R 1 ( θ, S )
=
E λ
[(1 − λ )]
n n (cid:88)
i = |
y
i − g
( f θ ( x i ))
||
f θ ( x i ) | =
E λ [(1 − λ )]
n n (cid:88) i
= | g ( f θ ( x i ))
− y i |(cid:107)∇
f θ ( x i ) (cid:107) 2 (cid:107) x
i (cid:107) 2 | cos( ∇ f θ ( x i ) , x i ) | ≥ Rc x E λ [(1 − λ )] √ d n n (cid:88) i = | g
( f θ (
[Equation]
By substituting these lower bounds of E and F into equation 11, we obtain the desired statement.
A. P ROOFS RELATED TO GENERALIZATION A.3. P ROOF OF L EMMA 3.3 AND L EMMA 3.
We ﬁrst prove Lemma 3.3.
The proof of Lemma 3.4 is similar.
By Eq.
(8), we have L mix n ( θ, S )
=
L stdn ( θ, ˇ S ) , where ˇ S = { (ˇ x i , y i ) } ni =1 with ˇ x i = λx
i + (1 − λ ) r x and λ ∼ ˜ D
λ
= αα + β Beta ( α + 1 , β ) + βα + β Beta ( β + 1 , α ) .
Since for Generalized Linear Model (GLM), the prediction is invariant to the scaling of the training data, so it sufﬁces to consider ˜ S
= { (˜ x i , y i ) } ni =1 with ˜ x i = 1¯ λ ( λx i + (1 − λ ) r x ) .
In the following, we analyze L stdn ( θ, ˜ S ) .
For GLM the loss function is
[Equation]
where A ( · ) is the log-partition function in GLMs.
Denote the randomness (of λ and r x ) by ξ , then the second order Taylor expansion yields
[Equation]
Notice E ξ [˜ x i − x i ] = 0 and V ar ξ (˜ x i ) =
1 n (cid:80) ni
=1 x i x (cid:62) i =
ˆΣ X , then we have the RHS of the last equation equal to
[Equation]
As a result, the second-order Taylor approximation of the Mixup loss L stdn ( θ, ˜ S ) is n
[Equation]
This completes the proof of Lemma 3.3.
For Lemma 3.4, since the Mixup is performed on the ﬁnal layer of the neural nets, the setting is the same as the least square with covariates σ ( w (cid:62) j x ) .
Moreover, since we include both the linear coefﬁcients vector θ 1 and bias term θ 0 , the prediction is invariant to the shifting and scaling of σ ( w (cid:62) j x ) .
Therefore, we can consider training θ
and θ 0 on the covariates { ( σ ( Wx i ) − ¯ σ W ) + 1 − λλ ( σ ( Wr x ) − ¯ σ W ) } ni =1 , where ¯ σ W = 1 n (cid:80) ni =1 σ ( Wx i ) .
Moreover, since we consider the least square loss, which is a special case of GLM loss with A ( u ) = 12 u 2 , we have
A (cid:48)(cid:48) = 1 .
Plugging these quantities into Lemma 3.3, we get the desired result of Lemma 3.4.
A.3. P ROOF OF T HEOREM 3.4 AND C OROLLARY 3.
By deﬁnition, given n i.i.d.
Rademacher rv.
ξ 1 , ..., ξ n , the empirical Rademacher complexity is
[Equation]
Let ˜ x
i = Σ † / 2 X x i , a ( θ ) =
E x
[ A (cid:48)(cid:48)
( x (cid:62) θ )] and v = Σ 1 / 2 X θ , then ρ -retentiveness condition implies a ( θ ) 2 ≥ ρ · min { 1 , E x
( θ (cid:62) x ) 2 } ≥ ρ · min { 1 , θ (cid:62) Σ X θ } and therefore a ( θ ) · θ (cid:62)
Σ X θ ≤ γ implies that (cid:107) v (cid:107)
=
θ (cid:62) Σ X θ ≤ max { ( γ ρ ) 1 / 2 , γ ρ } .
As a result,
[Table]
Consequently,
[Table]
Based on this bound on Rademacher complexity
[Table]
Corollary 3.1 can be proved by directly applying the following theorem.
Lemma A.1 (Result from Bartlett & Mendelson (2002)) .
For any B -uniformly bounded and LLipchitz function ζ , for all φ ∈ Φ , with probability at least 1 − δ ,
[Equation]
A.3. P ROOF OF T HEOREM 3.
To prove Theorem 3.5, by Lemma A.1, it sufﬁces to show the following bound on Rademacher complexity.
Theorem A.1.
The empirical Rademacher complexity of W NNγ satisﬁes
[Equation]
By deﬁnition, given n i.i.d.
Rademacher rv.
ξ 1 , ..., ξ n , the empirical Rademacher complexity is
[Equation]
Let ˜ θ 1 = Σ σ 1 / 2 X θ 1 and µ σ = E [ σ ( Wx )] , then
[Equation]
where the last inequality is obtained by using the same technique as in the proof of Lemma
[Equation]
Combining all the pieces
[Equation]
we get
[Equation]
B D ISCUSSION OF R IN THE N EURAL N ETWORK CASE
(B.1).
On the value of R = min i R i via experiments for neural networks.
After training accuracy reaches 100%
[Equation]
the loss is further minimized when (cid:107) f θ ( x i ) (cid:107) 2 increases.
Since
[Equation]
this suggests that R i and R tend to increase after training accuracy reaches 100%.
We conﬁrm this phenomenon in Figure 3.
In the ﬁgure, R is initially small but tend to increase after training accuracy reaches 100%, as expected.
For example, for ANN, the values of R were initially 2 .
27 × 10 − 5 but increased to 6 .
11 × 10 − 2 after training.
Figure 3 (c) and (d) also show that R i for each i -th data point tends to increase during training and that the values of R i for many points are much larger than the pessimistic lower bound R : e.g., whereas R = 6 .
11 × 10 − 2 , we have R i > 0 .
8 for several data points in Figure 3 (d).
For this experiment, we generated 100 data points as x i ∼ N (0 , I ) and y i
=
1 { x (cid:62) i θ ∗ > 0 } where x i ∈ R 10 and θ ∗ ∼ N (0 , I ) .
We used SGD to train linear models and
ANNs with ReLU activations and 50 neurons per each of two hidden layers.
We set the learning rate to be 0.1 and the momentum coefﬁcient to be 0.9.
We turned off weight decay so that R is not maximized as a result of bounding (cid:107)∇ f θ ( x i ) (cid:107) , which is a trivial case from the above discussion.
(B.2).
A constant lower bound for neural networks.
Similarly, we can obtain a constant lower bound by adding some additional conditions.
Assumption B.1.
Let us denote ˆΘ n ⊆
Θ as the set of minimizers of ˜ L mix n ( θ, S ) .
We assume there exists a set Θ ∗ , such that for all n ≥ N , where N is a positive integer, ˆΘ n ⊆
Θ ∗ with probability at least 1 − δ n and δ n → 0 as n → 0 .
Moreover, there exists τ, τ (cid:48) ∈
(0 , 1) such that
[Equation]
, for all θ ∈ Θ ∗ } , has probability p τ,τ (cid:48) ∈ (0 , 1) .
Theorem B.1.
Deﬁne
[Equation]
almost everywhere ,
[Equation]
Under Assumption B.1, for any f θ ( x ) ∈ F Θ , if there exists constants b x , c x > 0 such that c x √ d ≤ (cid:107) x i (cid:107) 2 ≤ b x √ d for all i ∈ { 1 , . . . , n } .
Then, with probability at least 1 − δ n − 2 exp( − np 2 τ,τ (cid:48) / 2) , there exist constants κ > 0 , κ 2 > κ 1 > 0 , if we further have θ ∈ ˆ Θ n , then
[Equation]
where ˜ ε mix = ˜
Rc x E λ ∼ ˜ D
λ [1 − λ ] and ˜ R = min {
[Equation]
B. P ROOF OF T HEOREM B.
Notice if we assume for
[Equation]
, for all θ ∈ Θ ∗ } , there is τ, τ (cid:48) ∈ (0 , 1) such that X Θ ∗ ( τ, τ (cid:48) )
(cid:54) = ∅ , and
[Equation]
Let us ﬁrst study
[Equation]
By boundedness of θ
[Equation]
x and y ∈ { 0 , 1 }
[Equation]
we know there is κ 1 , κ 2 > 0 , such that
[Equation]
If we denote ˆ p = { number of x (cid:48)
i s such that x
i ∈
X Θ ∗ ( τ, τ (cid:48) ) } /n .
Then
[Equation]
it is easy to see
[Equation]
For η 2 satisfying
[Equation]
we have n n
[Equation]
Besides, if we consider n
[Equation]
Thus, we have
[Equation]
ˆ
[Equation]
With probability at least 1 − 2 exp( − 2 nε 2 ) , for ε = p τ,τ (cid:48) / 2 , we have η
[Equation]
B. P ROOFS OF THE CLAIM f θ ( x ) = ∇ f θ ( x ) (cid:62) x FOR NN WITH R E LU/
M
AX - POOLING
In this section, we prove the claim that for NN with ReLU/Max-pooling f θ ( x ) = ∇ f θ ( x ) (cid:62) x almost everywhere.
In the following, We ﬁrst prove this claim for ReLU.
[Equation]
We ﬁrst prove this claim for N = 2 , and then use the method of induction to prove the case for general N .
When N = 2 , we have f θ ( x )
=
W 2 σ ( W 1 x ) .
We ﬁrst show that for any x ∈ R p such that x j (cid:54) =for all j ∈ [ p ] (which happens almost everywhere w.r.t.
x ), dσ ( x ) x = diag( { 1( x j > 0) } p j =1 ) and therefore σ ( x ) = ( dσ ( x ) x ) (cid:62) x .
In addition, since for any W ∈ R q × p , W 1 x = ( d ( W 1 x ) dx )
[Equation]
x , we then have almost everywhere w.r.t.
x ,
[Equation]
which implies
[Equation]
Now we assume the case with N holds, that is f ( N ) θ
[Equation]
satisﬁes f ( N ) θ
[Equation]
θ
[Equation]
Now, for the case with N + 1 , f ( N +1) ( x ) =
W N +1 σ ( f ( N ) ( x )) , we have
[Equation]
then
[Equation]
1) ( x ) .
This complets the induction.
For NN with Max-pooling, the only additional nonlinear function is η ( x ) = max { x 1 , ..., x p } .
For this η ( x ) , we have almost everywhere dη ( x ) x =
e j with j = arg max k ∈ [ p ] x k and therefore η ( x )
= ( dη ( x ) x ) (cid:62) x .
Using this fact, the proof is similar to the case with ReLU.
C M ORE A BOUT E XPERIMENTS C. A DVERSARIAL A TTACK AND M IXUP
We demonstrate the comparison between Mixup and standard training against adversarial attacks created by FGSM.
We train two WideResNet-16-8 (Zagoruyko & Komodakis, 2016) architectures on the Street View House Numbers SVHN (Netzer et al., 2011)) dataset; one model with regular empirical risk minimization and the other one with Mixup loss ( α = 5 , β = 0 . 5 ).
We create FGSM adversarial attacks (Goodfellow et al., 2014) for 1000 randomly selected test images.
Fig.
(1a) describes the results for the two models.
It can be observed that the model trained with Mixup loss has better robustness.
C. G ENERALIZATION AND M IXUP
Figures 4–7 show the results of experiments for generalization with various datasets that motivated us to mathematically study Mixup.
We followed the standard experimental setups without any modiﬁcation as follows.
We adopted the standard image datasets, CIFAR-10 (Krizhevsky &
Hinton, 2009), CIFAR-100 (Krizhevsky & Hinton, 2009), Fashion-MNIST (Xiao et al., 2017), and
Kuzushiji-MNIST (Clanuwat et al., 2019).
For each dataset, we consider two cases: with and without standard additional data augmentation for each dataset.
We used the standard pre-activation
ResNet with 18 layers (He et al., 2016b).
Stochastic gradient descent (SGD) was used to train the models with mini-batch size = 64, the momentum coefﬁcient = 0.9, and the learning rate = 0.1.
All experiments were implemented in PyTorch (Paszke et al., 2019).
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
i
[Figure]
n
[Figure]