A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective
Abstract
We propose the first unified theoretical analysis of mixed sample data augmentation (MSDA), such as Mixup and CutMix.
Our theoretical results show that regardless of the choice of the mixing strategy, MSDA behaves as a pixel-level regularization of the underlying training loss and a regularization of the first layer parameters.
Similarly, our theoretical results support that the MSDA training strategy can improve adversarial robustness and generalization compared to the vanilla training strategy.
Using the theoretical results, we provide a high-level understanding of how different design choices of MSDA work differently.
For example, we show that the most popular MSDA methods, Mixup and CutMix, behave differently, e.g .,
CutMix regularizes the input gradients by pixel distances, while Mixup regularizes the input gradients regardless of pixel distances.
Our theoretical results also show that the optimal MSDA strategy depends on tasks, datasets, or model parameters.
From these observations, we propose generalized MSDAs, a Hybrid version of
Mixup and CutMix ( HMix ) and Gaussian Mixup ( GMix ), simple extensions of
Mixup and CutMix.
Our implementation can leverage the advantages of Mixup and CutMix, while our implementation is very efficient, and the computation cost is almost neglectable as Mixup and CutMix.
Our empirical study shows that our HMix and GMix outperform the previous state-of-the-art MSDA methods in CIFAR-100 and ImageNet classification tasks.
Source code is available at https://github.com/naver-ai/hmix-gmix .
Introduction
As deep neural networks (DNNs) are data-hungry, the scale of datasets has become a foundation of modern DNN training; recent ground-breaking deep models are built upon gigantic datasets, such as 410B language tokens [6], 3.5B images [47], and 1.8B image-text pairs [33].
While such tremendously large-scale datasets are not always collectible, amplifying the dataset scale by synthesizing more data points by data augmentation techniques is common.
Especially, mixed sample data augmentation (MSDA) [76, 63, 62, 30, 72, 66, 60, 3, 68, 74, 11, 65, 39, 25, 18, 54, 32, 16, 23, 38, 45, 14, 32, 58, 44] has become a standard technique to train a strong deep model by synthesizing a mixed sample from multiple (usually two) samples by combining both of their sample values and labels in a linear combination [76] or a cut-and-paste manner [72].
This simple idea, however, shows surprising performance enhancements in various applications, including image object recognition [72, 39, 25, 64, 16], semi-supervised learning [4, 59] self-supervised learning [35, 40, 42], noisy label training [43], meta-learning [71], semantic segmentation [10, 20], natural language understanding [24, 34], and audio processing [48, 37, 49].
Another advantage of MSDA beyond the performance improvements is that MSDA usually does not need domain-specific knowledge, such as strong image-specific [15] or audio-specific [53] transformations; hence MSDA can be universally employed NAVER
Lab by various applications. However, although MSDA shows excellent benefits in practice, there is still yet not enough understanding of how MSDA works well universally; can MSDA always show better generalization and robustness than the standard training with a theoretical guarantee?
Furthermore, the design choice of MSDA can be significantly varying and the optimal design choice is still ambiguous.
For example, Lee et al .
[42] showed that in self-supervised learning, Mixup is more effective than CutMix, while other studies [56, 57] observed opposite results.
The ambiguity is originated from the fact that we do not have a unified theoretical lens of understanding how different design choices affect the actual learning process; in short, how are Mixup and CutMix different?
There have been several attempts to theoretically understand Mixup, a special case of MSDA [77, 12, 8, 78].
They delve into the effect of Mixup in a loss function perspective, e.g ., Mixup behaves as a regularization of the standard training [77], or in a learning theory perspective, e.g ., Mixup training can provide an upper bound for the true loss [12, 71].
However, their analyses are limited to
Mixup, while other MSDAs, such as CutMix, are poorly understandable through the lens of their analyses.
In this paper, we extend the theoretical results of Zhang et al .
[77] and Chidambaram et al .
[12] to a general MSDA to provide a first unified theoretical lens for understanding how general
MSDAs work by different choices of mixing strategies.
We show that MSDA behaves as an input gradient and Hessian regularization (Theorem 1) as well as a regularizer for the first layer parameters;
MSDA improves adversarial robustness (Theorem 3) and generalization (Theorem 4).
Our theoretical results show that popular MSDA methods, such as Mixup and CutMix, behave differently in terms of regularization effects.
Briefly, CutMix gives a strong regularization in the product of nearby distance pixel-level partial gradient and nearby distance Hessian of the estimated function f , while CutMix gives a weak regularization in the product of long-distance pixel-level partial gradient and long-distance Hessian of the estimated function f .
In contrast, Mixup gives a regularization in gradient or Hessian of the estimated function f regardless of the pixel-level distance.
From our unified theoretical lens for MSDA, we can conclude that there is no one-fit-all optimal
MSDA fit to every data or model parameter .
In other words, the optimal mixing strategy depends on applications, datasets, and model architectures.
It supports previous empirical observations that combining different MSDA methods ( e.g ., alternatively using Mixup and CutMix during training) can outperform using only one MSDA [50, 5, 64, 73].
From these observations, we propose two simple MSDA methods that naturally generalize Mixup and CutMix, so that it can take advantage of both methods.
Our first proposed method, Hybrid version of Mixup and CutMix (HMix) , mixes two samples in both Mixup and CutMix manners; it first cut-and-paste two samples as CutMix, and then it linearly interpolates the out-of-box values of two samples as Mixup.
We let HMix be able to behave as both Mixup and CutMix by introducing a stochastic control parameter.
Our second proposed method, Gaussian Mixup (GMix) , also mixes two samples in both Mixup and Cutmix manners; firstly we select a point, and then we mix two samples gradually using the Gaussian function.
Our empirical results on CIFAR-100 and ImageNet show that HMix and GMix outperform the state-of-the-art
MSDA methods, including Mixup, CutMix, and Stochastic Mixup & CutMix.
A General Framework for Mixed Sample Data Augmentation (MSDA)
In this section, we define the formal definition of MSDA and notations.
We define a training dataset as D =
{ z i = ( x i , y i ) } mi =1 , randomly sampled from a distribution P z .
Here, z = ( x, y ) is the input ( e.g ., an image) and output ( e.g ., the target class label) pair.
Then, for randomly selected two data samples, z i and z j , an augmented sample by MSDA, ˜ z
( MSDA ) i,j , is synthesized as follows
˜ z ( MSDA ) i,j
( λ,
[Equation]
=
[Equation]
MSDA
[Equation]
[Equation] [Equation]
) , ˜ y
[Equation]
MSDA )
[Equation]
( λ, 1 − λ )) where, ˜ x ( MSDA ) i,j
[Equation]
[Equation]
M
[Equation]
λ ) ⊙ [Equation] i +
[Equation]
1 − M
[Equation]
λ ))
⊙ x j and
[Equation]
[Equation]
where λ is the ratio parameter between samples, drawn from D λ (usually Beta distribution).
⊙ means a component-wise multiplication in vector (or matrix).
M ( λ ) is a random variable conditioned on λ that indicates how we mix the input ( e.g ., by linear interpolation [76] or by a pixel mask [72]).
N ( λ ) denotes a random variable conditioned on λ that demonstrates how we combine the output.
We assume that the output y can be one-dimensional data or a matrix; the former means regression or classification task, and the latter means semantic segmentation task.
For the sake of simplicity, we let y be one-dimensional data: N ( λ )
=
λ and E [ M ( λ )]
= λ⃗ 1 .
Remark 1.
If the meaning is not ambiguous, then we sometimes omit λ ( i.e ., M ( λ ) to M ).
For the sake of simplicity, we consider mixing only two samples ( i.e ., ˜ z ( MSDA ) i,j ( λ, 1 − λ ) ), but we can similarly extend these analyses to mixing n -samples data augmentation [60, 32, 23].
If we combine n -samples, the ratio parameter will be a vector in general (See Appendix B).
Remark 2.
As recent studies [68, 39, 38] have shown, M ( λ ) or N ( λ ) can depend on ( z i , z j ) , e.g ., by using a saliency map [39, 38] or the class activation map [68].
Since the proof techniques for our theoretical analysis are invariant to the choice of M ( λ ) and N ( λ ) , our proof techniques also can be applied to the dynamic MSDA methods.
For simplicity, we assume that M is a random variable only depending on λ .
In other words, we assume W as a random sample space, and M : W × Λ → R n is a measurable function.
We left the theoretical analysis of dynamic methods to the future.
Now, we re-write the two most popular MSDA methods, Mixup [76] and CutMix [72], for i -th and j -th samples with λ , drawn from D λ , by using the proposed framework (Equation (1)) as follows:
Mixup: ˜ z ( mixup ) i,j ( λ, 1 − λ ) = (˜ x ( mixup )
[Equation]
( λ, 1 − λ ) , ˜ y ( mixup )
[Equation]
,j ( λ, 1 − λ )) where ˜ x ( mixup ) i,j ( λ, 1 − λ )
=
λx i + (1 − λ ) x j and ˜ y ( mixup ) i,j ( λ, 1 − λ ) = λy i + (1 − λ ) y j .
CutMix: ˜ z ( cutmix ) i,j ( λ, 1 − λ )
= (˜ x ( cutmix ) i,j (
[Equation]
[Equation]
( cutmix )
[Equation]
,j
[Equation]
λ, 1 − λ )) where ˜ x ( cutmix ) i,j ( ˜ M ( cutmix ) , 1 − ˜ M ( cutmix ) ) =
˜ M ( cutmix ) ⊙ x
i + (1 − ˜ M ( cutmix ) ) ⊙ x j and ˜ y ( cutmix ) i,j
[Equation]
Note that Equation (2) is equivalent to Equation (1) by putting M ( λ )
=
λ⃗ 1 .
In Equation (3), ˜ M ( cutmix ) is a binary mask that indicates the location of the cropped box region with a relative area λ .
Similarly, other MSDA variants can be easily formed as Equation (1) by introducing new M ( λ ) and N ( λ ) .
Notations.
We define the loss function as l ( θ, z ) , where θ ∈ Θ ⊆ R d .
We define L ( θ ) =
E z ∼P
z l ( θ, z ) as the non-augmented population loss and L m ( θ ) =
1 m (cid:80) mi =1 l ( θ, z i ) as the empirical loss for the non-augmented population.
For a general MSDA, we can define MSDA loss as
[Equation]
Therefore, the Mixup and CutMix losses can be written as
[Equation]
where [ m ] = { 1 , 2 , . .
, m } and D λ is a distribution supported on [0 , 1] with a conjugate prior.
Throughout this paper, we consider D λ as Beta ( α, β ) , a common selection for λ in practice.
We define D X as the empirical distribution of the training dataset.
A Unified Theoretical Understanding of MSDA
In this section, we provide a unified theoretical lens of how MSDA works.
Specifically, we follow the theoretical results for Mixup provided by Zhang et al .
[77], where Zhang et al .
have shown that Mixup is equivalent to the summation of the original loss function and a Mixup-originated regularization term.
We will give a general approximation form for MSDA using λ ∼ Beta ( α, β ) .
We also show that our analysis can be extended to n -sample mixed augmentation (See Appendix B)
From an MSDA loss to an input gradient and Hessian regularization.
We first consider the following class of loss functions for a twice differentiable prediction function f θ ( x ) ( e.g ., a softmax output of a neural network), a twice differentiable function h , and target y :
[Equation]
This function class L includes the loss function induced by Generalized Linear Models (GLMs) and cross-entropy.
Now, we introduce our first theoretical result that induces an MSDA loss ( i.e .,
Equation (4)) can be re-written as the summation of the original loss (the empirical loss for the non-augmented population loss, L m ( θ ) ) and input gradient-related regularization terms as follows.
Theorem 1.
Consider a loss function l ∈ L .
We define ˜ D λ
as αα + β Beta ( α +1 , β )+ βα + β Beta ( β +1 , α ) .
Assume that E r x ∼D X
[ r x ] = 0 .
Then, we can re-write the general MSDA loss (4) as
[Equation]
where lim a → 0 φ ( a ) = 0 ,
R ( MSDA )
[Equation]
7) and
[Equation]
where
[Equation]
Proof outline of Theorem 1.
Using the definition of ˜ z ij and using the fact that the Binomial distribution and Beta distribution are in the conjugate, we can reformulate L ( MSDA ) m .
In the process of reformulating L ( MSDA ) m , we should define ˜ D λ .
Then, we can make a quadratic Taylor approximation of the loss term.
Here, E r x [ r x ] = 0 is used for not only the simplicity of the results, but also for the fact that using normalization in the dataset.
Details can be found in Appendix A. We also show that
Theorem 1 can be extended to n -sample MSDA methods (Appendix B).
In this case, the combinatorial terms in quadratic multivariate Taylor approximation also come out.
How is our approximation accurate?
We call ˜ L MSDA m [Equation]
as the approximate MSDA loss .
Here, we empirically demonstrate that our quadratic approximation is almost accurate by following numerical validations in [67, 8, 77].
Specifically, we train logistic regression models on two-moons dataset [7] in two ways: (1) by using the original MSDA loss function (2) by using our approximated loss function.
We employ two MSDA examples as below.
• The original Mixup i.e ., λ ∼ Beta (1 , 1) and M = λ⃗• Variants of CutMix i.e ., λ ∼ Beta (1 , 1) and M = ( m 1 , m 2 ) such that m i ∼ Bernoulli ( λ ) .
Figure 1 displays the approximate loss function and the original loss function.
According to empirical findings, we can conclude that the original MSDA loss is fairly close to the approximate MSDA loss.
What makes the difference between various MSDA methods?
In the theorem, as we define E M (1 −
M )
= 1 − λ , R ( MSDA )is the same for every MSDA method.
Namely, the difference between MSDA methods originated from R ( MSDA )and R ( MSDA ).
Note that if we set M = λ⃗ 1 , Theoremindicates a Mixup loss (5), and the result is consistent with Zhang et al . In Equation (7) and
Equation (8), we observe that R 2 is related to the input gradient ∇ f θ ( x i ) and R 3 is related to input [Figure] Hessian ) with mask-dependent coefficients a jk (9).
In other words, different design choice of MSDA ( e.g ., how to design M ) will lead to different magnitudes of regularization on the input gradients and Hessians.
Because the values of input gradients and Hessians are varying by datasets, tasks and model architecture choices, we can conclude that the optimal choice of M is dependent on the applications.
We also describe how the other MSDA methods ( e.g ., dynamic MSDAs [65, 39, 38]) can be interpreted through the lens of our unified analysis in Appendix I.
In addition, to show that MSDA behaves as a regularization on input gradients and Hessians for any desired a jk , we also show that there always exists a MSDA design choice M for any desired regularization coefficient matrix A ( λ ) :
= ( a jk ( λ )) with the regular conditions.
Theorem 2.
For the given λ , we assume A ( λ ) − (1 − λ )
2 ⃗ 1 ⃗ 1 ⊺ is a nonnegative definite matrix.
Then we can construct a real-valued mask M that E (1 − M j )(1 − M k ) =
a jk for all j, k .
Proof.
Setting M = 1 − λ
+( A ( λ ) − (1 − λ ) 2 ⃗ 1 ⃗ 1 ⊺ ) 1 / 2 Z where Z is normal distribution, the theorem holds.
Note that, in the proof, M values are not bounded where typically we choose 0 ≤ M i ≤ 1 .
In other words, the theorem holds if we allow mask values out of [0 , 1] .
To investigate the potentiality of unbounded mask, we explore Mixup with unbounded masks in Figure 2.
Although, allowing negative values to M can be beneficial, we leave a new mask design with unbounded values as a future work.
Unfortunately, as the target loss function (6) is mingled with the choice of mask M , data sample x i , and pixel-level function gradient, the optimal choice of mixing strategy M is not achievable in the closed-form solution.
Instead, Theorem 1 implies that there is no absolute superiority between the design choice of MSDA, but it depends on datasets and the target tasks, as our empirical observation is consistent with the theoretical interpretation.
In Section 4, we will provide more examples of how different M affects the actual coefficients a jk and the input gradients for better understanding.
Using the regularization term R ( MSDA )(7), we can also provide a theoretical connection between
MSDA methods and the notion of flatness where a more flat solution leads to better generalization in applications [36, 21, 31, 19, 9].
Inspired by Ma et al .
[46], we split the parameters by θ =
( θ 1 , θ 2 ) , and then the neural network can be represented by the form f θ ( x ) =
˜
f
θ 2 ( θ 1 x ) .
Therefore, we have
[Equation]
where ∂f ∂ ( θ 1 x ) is the partial derivative of the first layer.
Now, we have
[Equation]
Note that the terms in G (8) can be re-written as follows
[Equation]
In other words, by minimizing the regularization term R ( MSDA )
[Equation]
the regularization effect of flatness at the interpolation solution can be minimized in a sample-wise weighted manner.
Therefore, the regularization term R ( MSDA )also can be interpreted as a regularization of the first layer parameters and their partial derivative of f .
Robustness and generalization properties of MSDA.
As a number of studies [51, 52, 46] have shown that regularizing input gradient and Hessian will give better robustness and generalization to the target network θ , it can be shown that MSDA also has adversarial robustness properties and generalization properties based on Theorem 1.
The full statement of Theorem 3 and Theorem 4 can be found in Appendix C and Appendix D, respectively.
Theorem 3 (Informal) .
With the logistic loss function under the ReLU network, the approximate loss function of MSDA is greater than the adversarial loss with the ℓ 2 attack of size ϵ √ d .
Proof outline of Theorem 3.
Defining adversarial loss function and using second order taylor expansion, we can prove that adversarial loss is less than MSDA loss.
Theorem 4 (Informal) .
Under the GLM model and the regular conditions, and if we use MSDA in training, we have
[Equation]
n with probability at least 1 − δ .
This also holds for the MSE loss and a feature-level MSDA.
Proof outline of Theorem 4.
MSDA regularization can be altered to the original empirical risk minimization problem with a constrained function set, and calculating Radamacher complexity of this function set gives the theorem.
In addition to Theorem 3 and Theorem 4, we can prove that the optimal solution of (4) can achieve a perfect classifier ( i.e ., classifies every augmented sample x correctly) in the logistic classification setting by following Chidambaram et al .
The full statement are in Appendix E.
Summary.
Our unified theoretical lens for MSDA shows that for any MSDA method formed as
Equation (4), the method satisfies that (1) it behaves as a regularizer of input gradients, Hessian, and the first layer parameters (Theorem 1); (2) there exists a mask M for any desired regularization coefficients a jk (Theorem 2) (3) it achieves better adversarial robustness (Theorem 3) and generalization (Theorem 4) than the vanilla training.
Interestingly, Theorem 1 shows the difference between different MSDA design choices ( e.g ., different M , such as linear interpolation [76], cropped box [72]) will lead to different magnitudes of the input gradient regularization (7).
Comparison of Different MSDA Design Choices: The Role of Masks
As we observed in the previous section, different design choices for MSDA ( i.e ., the choice of
M ) affect to the degree of the regularization in Theorem 1 ( i.e ., R ( MSDA )and R ( MSDA )) by the relationships of pixels.
In this section, we show how different MSDA methods lead to different regularization effects by empirical studies; we first show the values of the regularization coefficients [Figure] a jk by varying masks; then we show the input gradient values that are regularized by a jk (9) after the MSDA training; finally, we show that the best choice of the mask design can be varying by the target task settings.
In addition, we propose two generalized versions of Mixup and CutMix, called HMix and GMix, that empirically show the intermediate property of Mixup and CutMix.
Introduction to HMix and GMix.
Recall that the regularization coefficients a jk is determined by
M (Equation (9)).
For example, by choosing M = λ⃗ 1 ( i.e ., Mixup), a jk is always (1 − λ ) 2 .
On the other hand, the result slightly changes for CutMix: a jk depends on how j and k are close.
Informally, due to dependency between M j and M k (as M ’s component is always 0 in the cropped box regions and 1 in others), close j and k give large a jk , but distant j and k give small a jk .
a jk is calculated as
[Equation]
where j = ( j 1 , j 2 ) , k = ( k 1 , k 2 ) , h ( t ) = min( t, n − [ √ 1 − λn ]) , l ( t ) = max( t − [ √ 1 − λn ] , 0) .
We visualize a jk of different MSDA methods in Figure 4.
We compare Mixup, CutMix, Stochastic
Mixup & CutMix.
We also propose two generalized MSDA methods, named HMix and GMix .
Before comparing the methods, we first formally define Stochastic Mixup & CutMix, HMix and
GMix.
These methods can be formed as (1) where the definition of M is varying by the methods.
Stochastic Mixup & CutMix is a practical variant of MSDA by considering Mixup and CutMix at the same time.
By a simple alternation of two augmentations, the state-of-the-art performances on largescale datasets are shown [64, 70].
Stochastic Mixup & CutMix is the same as Equation (1) by setting
M ( λ ) =
(1 − λ )
⃗ 1 with probability q and M ( λ ) =
M cutmix ( λ ) with probability 1 − q .
We choose q = 0 .
5 as [64, 70].
In our loss function perspective, the regularizing coefficient terms ( i.e ., R 2 , R 3 ) become the average of Mixup and CutMix’s regularization coefficient.
Namely, let a mixup ij = (
[Equation] − λ )be a regularization coefficient of Mixup and a cutmix ij be a regularization coefficient of CutMix (10), then the regularization coefficients of Stochastic Mixup & CutMix is qa cutmix ij [Equation]
Here, we additionally propose two MSDA variants, HMix and GMix, that leverage the advantages of
Mixup and CutMix, resulting in showing the intermediate property between Mixup and CutMix.
Hybrid version of Mixup and CutMix (HMix) combines Mixup and CutMix by shrinking the CutMix cropped box region and linearly interpolating two images in the areas out of the box as Mixup.
The shrinking ratio of the cropped box region is determined by the ratio r .
HMix can be written as (1) by setting M by (1) randomly cropped box region with side length √ 1 − λ √ rN where N is the side length of the original image, and make M ’s component in the box region as 0 (2) in the areas other than the box, we set M i as λ 1 − (1 − λ ) r .
We can easily check that E [ M ] = λ⃗ 1 .
As r → 0 , this method goes to Mixup, and as r → 1 , this method goes to CutMix.
Note that the ratio r can be a random variable, such as Beta ( γ, γ ) .
In this case, if we set γ → 0 , as Beta ( γ, γ ) goes to Bernoulli distribution, this is equivalent to Stochastic Mixup & CutMix.
We propose Gaussian Mixup (GMix) to relax the CutMix box condition to a continuous version as the rectangle cropping of CutMix causes implausible augmented data, e.g ., the boundary between two mixed samples.
Therefore, we combine two ideas of Mixup and CutMix.
Firstly, we select a point p from the given input.
Then, we make M i as the related function with ∥ i − p ∥ 2 .
Specifically, we use the Gaussian function for making M : (1) randomly select a point p in image (2) in the areas other than the box, we set M i as 1 − exp (cid:16) − ∥ i − p ∥ 2 π 2(1 − λ ) N 2 (cid:17) .
The proposed GMix has the following a ij
[Equation]
As seen in Equation (11), a ij smoothly goes down when the pixel distance becomes larger.
Figure 3 shows the examples generated Mixup, CutMix, HMix, and GMix.
The proposed methods (HMix and GMix) generate images in a hybrid form with the properties of both Mixup and CutMix.
Comparison in terms of regularization coefficients a jk .
We illustrate the regularization coefficients a jk of the different MSDA methods in Figure 4.
In particular, we fix the mask parameter λ to 0.5 and the input resolution to 64 × 64.
Figure 4 shows the difference between the MSDA methods in terms of how they regularize the input gradients and input Hessians: Mixup has equal weights to every gradient component or Hessian component, while CutMix gives high regularization in close coordinate gradient products or Hessian.
We also observe that the hybrid methods ( e.g ., Stochastic
Mixup & CutMix, HMix, and GMix) show the intermediate coefficient values of Mixup and CutMix.
Comparison in terms of the regularized input gradients after MSDA training.
Equation (8) shows that the regularization term a ij directly affects to the pixel gradients | ∂ i f θ ( x k ) ∂
j f θ ( x k ) | in our approximated loss function.
The purpose of Figure 5 is to show how the pixel gradients are actually regularized after training.
We investigate the amount of the regularized input gradients by | ∂ v f θ ( x ) ∂ v + p f θ ( x ) | with respect to the pixel distance vector p for trained models by different MSDA methods.
Here, if our approximated loss function actually behaves as a regularization, then we can expect that the pixel gradients | ∂ v f θ ( x ) ∂ v + p f θ ( x ) | is small when a ij is large for the given p .
We first define the partial gradient product as follows:
[Equation]
Now, we visualize the pixel-wise maximum values of PartialGradProd ( x, p ) in Figure 5.
We train different models f θ on resized ImageNet (64 x 64) and measure the values on the validation dataset.
The x -axis and y -axis of Figure 5 denote the pixel distance p along each x and y axis, and the scale of the colorbar denotes the value of the maximum partial gradient product.
In the figure, we can observe that CutMix reasonably regularizes effectively in the input gradients products when a pixel distance is small; these results aligned with our previous interpretation, CutMix behaves a pixel-level regularizer where it gives stronger regularization (larger a ij ) to the closer pixels.
Note that we are not discussing the relationship between regularizing effects and accuracy but discussing regularizing coefficients and the optimized function’s pixel gradients.
Understanding application cases when a specific MSDA design choice works better than others.
From our theoretical results and empirical studies, we have shown that the design choice of MSDAs [Figure] [Table] ( i.e ., M ) leads to different regularization effects by regularization coefficient a jk .
Furthermore, as we have shown in Theorem 2, there always exists a mask that can form any desired a jk .
We hypothesize that for the given dataset, if a short distance relation is relatively more important than longer distance relations, then CutMix will be better than Mixup.
On the contrary, in the opposite case, if a short distance relation is relatively less important, then Mixup will be better than CutMix.
Here, we study different task scenarios when different a jk s are required by controlling the pixel-level importance of ImageNet-100 [61] training images.
In particular, we design two different scenarios where each of them needs different regularization strategies due to the different pixel-level importance of each task.
The results are shown in Table 1.
We also report the performance of our proposed methods in both scenarios 1 and 2 in Appendix G.
Scenario 1: Smaller objects by large crop size.
We randomly crop a large region (80% to 100%) of an image and resize to 64 × 64 to train a model.
As the objects in the image become small, a close-distance relationship might be more important than a large-distance relationship.
Here, we expect CutMix performs better than Mixup as shown in Table 1.
Scenario 2: Larger objects by small crop size We randomly crop a small region (25% to 40%) of an image and resize to 64 × 64 to train a model.
Contrary to Scenario 1, the objects in the image would become large in the cropping region and the large-distance relationship might be important, therefore, we expect that Mixup performs better than CutMix.
This hypothesis is aligned to Table 1.
Comparison of Different MSDA Design Choices: An Empirical Validation
In this section, we compare various MSDA methods on two popular large-scale image classification benchmarks: CIFAR-100 [41] and ImageNet-1K [17].
We will confirm that our proposed design choices, HMix and GMix, are not only theoretically interpolating Mixup and CutMix in the toy settings, but also taking benefits of each method by showing great performances in real-world applications.
The implementation details and the hyper-parameter study can be found in Appendix F.
Results on CIFAR-100 classification.
We evaluate HMix and GMix against baseline MSDA methods including Mixup [76], CutMix [72], Stochastic Mixup & CutMix, [64] and PuzzleMix [39] on CIFAR-100 dataset.
Here, we include PuzzleMix, to see the effectiveness of our data-agnostic method against the data-aware mask strategy.
Note that although our theoretical results (Section 3) are based on the data-agnostic mask selection methods, our theoretical results can be easily extended to the data-dependent mask selection methods.
We leave the extension as a future research direction.
[Table]
To see the generalizability of our methods, we train various network architectures including
ResNet-56 (RN56) [26], WideResNet28-2 (WRN28-2) [75], PreActResNet-18 (PreActRN18) [27],
PreActResNet-34 (PreActRN34) [27] and PreActResNet-50 (PreActRN50) [27] with various MSDA methods.
We train networks for 300 epochs using SGD optimizer with a learning rate 0 .
Tableshows the summarized results.
We set the hyper-parameter α for Mixup, CutMix, and Stochastic
Mixup & CutMix to 1 .
α for HMix and GMix were set to 1 and 0 . 5 , respectively.
We use r = 0 .for
HMix.
In the table, HMix and GMix outperform Mixup only and CutMix only counterparts and
Stochastic Mixup & CutMix often show comparable performances to HMix and GMix.
Our methods show comparable performance with the state-of-the-art data-dependent strategy PuzzleMix.
[Table]
Results on ImageNet-1K classification.
Table 3 shows the comparison of various MSDA methods on ImageNet-1K. We train ResNet50 [26] with various MSDA methods forepochs using SGD optimizer with a learning rate 0 .
We set the hyper-parameter α for all methods except Mixup to 1 , while Mixup has α = 0 . 8 .
We use r = 0 .
75 for HMix.
Here, we do not include PuzzleMix because it needs heavy additional computations to compute the input saliencies.
In the table, HMix shows the best performance, while GMix and Stochastic Mixup & CutMix show the second-best performances.
We also include evaluations on various robustness benchmarks in Appendix G.
Conclusion
We analyze MSDA by a unified theoretical framework.
Our unified theoretical results show that any
MSDA method behaves as a regularization on the input gradients and Hessians, where the degree of the regularization is controlled by the design choice of MSDA.
We compare various MSDA methods in (1) regularization coefficient (2) regularized gradients (3) model performances in various scenarios with different pixel-level importance.
We propose two simple MSDA methods, HMix and GMix, which leverage the benefits of Mixup and CutMix by their design.
Our experimental results show that
HMix and GMix outperform popular MSDA methods Mixup and CutMix.
Furthermore, our methods show comparable or outperformed performances than the state-of-the-art MSDA method, Stochastic
Mixup & CutMix, in CIFAR-100 and ImageNet classification tasks.
Author Contributions
This work is done as an internship project by C Park under the supervision of S Yun.
C Park contributed to the theoretical analysis, including theory ideas and proofs.
Theoretical results were verified and interpreted by C Park and S Chun.
HMix and GMix were designed by C Park and S
Yun.
S Yun implemented and conducted CIFAR and ImageNet experiments.
Empirical findings and analyses were contributed by all authors.
Paper presentation, such as paper writing and storyline, is mainly led by S Chun.
All authors significantly contributed to the final manuscript.