D ATA I NTERPOLATING P REDICTION : A LTERNATIVE I NTERPRETATION OF M IXUP
Preferred Networks
Kohei Hayashi Preferred Networks Sosuke Kobayashi Preferred Networks
A BSTRACT
Data augmentation by mixing samples, such as Mixup, has widely been used typically for classiﬁcation tasks.
However, this strategy is not always effective due to the gap between augmented samples during training and clean samples.
This gap may prevent a classiﬁer from learning the optimal decision boundary and increases the generalization error.
To overcome this problem, we propose an alternative framework called Data Interpolating Prediction (DIP).
Unlike common data augmentations, we encapsulate the sample-mixing process in the hypothesis class of a classiﬁer so that train and test samples are treated equally.
We derive the generalization bound and show that DIP reduces the original Rademacher complexity.
Also, we empirically demonstrate that DIP can outperform existing Mixup.
I NTRODUCTION
Data augmentation (Simard et al., 1998) has played an important role in training deep neural networks for the purpose of preventing overﬁtting and improving the generalization performance.
Recently, sample-mixed data augmentation (Zhang et al., 2018; Tokozume et al., 2018a;b; Verma et al., 2018;
Guo et al., 2019; Inoue, 2018) has attracted attention, where we combine two samples linearly to generate augmented samples.
The effectiveness of this approach is shown especially for image classiﬁcation and sound recognition tasks.
Many traditional data augmentations (e.g., slight deformations for image data (Taylor & Nitschke, 2017)) rely on the speciﬁc properties of the target domain such as invariances to some transformations.
On the other hand, sample-mixed augmentation can be applied to any dataset due to its simplicity.
However, its effectiveness depends on the speciﬁed data structure.
There is basically a difference between original clean samples and augmented samples in that augmented samples are not drawn from an underlying distribution directly.
Thus a classiﬁer trained with sample-mix augmentation may learn the biased decision boundary.
In fact, we can easily create a distribution where sample-mix deteriorates the classiﬁcation performance (See Fig. 1).
[Figure]
Although the standard sample-mix training deteriorates classiﬁcation performance due to the biased decision boundary caused by sample-mix training, our method mitigates this problem.
See sectionfor the details of a hyper-parameter of beta distribution (cid:11) and the number of sampling S .
[Figure] To overcome this problem, we propose a novel framework called Data Interpolating Prediction (DIP), where the sample-mixing process is encapsulated in a classiﬁer.
More speciﬁcally, we consider sample-mix as a stochastic perturbation in a function and obtain the prediction by computing the expected value over the random variable.
Note that we apply sample-mix to both train and test samples in our framework.
This procedure is similar to existing work such as monte carlo dropout (Gal & Ghahramani, 2016) and Augmented PAttern Classiﬁcation (Sato et al., 2015).
Furthermore, we establish the generalization error bound for our algorithm via Rademacher complexity and ﬁnd that sample-mix helps to reduce the Rademacher complexity of a hypothesis class.
Through experiments on benchmark image datasets, we conﬁrm the generalization gap can be reduced by samplemix and demonstrate the effectiveness of the proposed method.
P ROPOSED M ETHOD
In this section, we propose a novel framework called Data Interpolating Prediction (DIP), where the sample-mixing process is encapsulated in the hypothesis class of a classiﬁer.
Furthermore, we theoretically establish the generalization bound for proposed method via Rademacher complexity and show that sample-mix is effective to reduce Rademacher complexity.
2. D ATA I NTERPOLATING P REDICTION
Let X (cid:26) R d and Y (cid:26) f 0 ; 1 g K be a d -dimensional input space and a K -dimensional one-hot label space.
Denote a classiﬁer by f : R d ! R K .
The standard goal of classiﬁcation problems is to obtain a classiﬁer which minimizes the classiﬁcation risk deﬁned as follows:
[Equation]
(1) where p ( x ; y ) is the joint density of an underlying distribution and ℓ is a loss function.
Here, we consider a function where the sample-mixing process is encapsulated as a random variable.
We describe sample-mix between x and x ′ with a function : R d (cid:2) R d (cid:2) R !
R d as follows.
[Equation]
where (cid:21) 2 [0 ; 1] is a parameter that controls mixing ratio between two samples.
Let ( x 1 ; y 1 ) ; :::; ( x n ; y n ) be i.i.d.
samples drawn from p ( x ; y ) , b p ( x ) :
= 1 n ∑ ni =1 (cid:14) ( x = x i ) be the density of the empirical distribution of f ( x i ; y i )
g ni
=1 , and h be a speciﬁed classiﬁer.
Using the function , we deﬁne the deterministic function f : R d !
R K by
[Equation]
where p ( (cid:21) ) is some density function over [0 ; 1] .
Note that the function f corresponds to the base function h when we set p ( (cid:21) )
= (cid:14) ( (cid:21) = 1) .
2. P RACTICAL O PTIMIZATION
Since the expected value E (cid:21) (cid:24) p
[Equation]
) ; x ′ (cid:24) p ( x ′ ) [ (cid:1) ] is usually intractable, we train the classiﬁer by minimizing upper the bound of an empirical version of L ( f ) ,
[Equation]
By applying Jensen’s inequality, we have
[Equation]
where S is a positive integer which represents the number of sampling to estimate the expectation
[Equation]
We denote the RHS in equation 5 by b L upper ;S ( f ) .
The tightness of the above bound is related to the value of S as
[Equation]
We can prove this in a similar manner to Burda et al.
Since lim S !
1 b L upper ;S ( f )
=
b L ( f ) , larger S helps a precise risk estimation.
2. L ABEL -M IXING OR L ABEL -P RESERVING
There are two types of sample-mix data augmentation, namely, label-mixing approach and labelpreserving approach.
We can show that the objective functions of both approaches are consistent under some conditions.
Proposition 1.
Suppose that ℓ is a linear function with respect to the second argument and p ( (cid:21) )
=
Beta( (cid:11); (cid:11) ) for some constant (cid:11) > 0 .
Then we have the following equation.
[Equation]
The proof of this theorem can be found in the blog post 1 .
For many label-mixing approaches (Zhang et al., 2018; Verma et al., 2018), they use beta distribution for a prior of (cid:21) .
Thus, the optimization of such approaches can be considered as a special case of our framework because an empirical version of RHS in equation 7 corresponds to b L upper ; 1 where p ( (cid:21) ) is set to Beta( (cid:11) +1 ; (cid:11) ) .
We experimentally investigate behaviors of both label-mixing and label-preserving training in Sec. 3.
2. G ENERALIZATION B OUND VIA R ADEMACHER C OMPLEXITY
In this section, we present a generalization bound for a function equipped with sample-mix.
Let F be a function class of the speciﬁed model and b R n ( F ) be the empirical Rademacher complexity of F .
Then we have the following inequality.
Proposition 2.
Let f ( x i ; y i )
g ni =1 be i.i.d.
random variables drawn from an underlying distribution with the density p ( x ; y ) and ℓ ◦ F := f ℓ ◦ f j f 2 Fg .
Suppose that ℓ is bounded by some constant
B > 0 .
For any (cid:14) > 0 , with the probability at least 1 (cid:0) (cid:14) , the following holds for all f 2 F .
[Equation]
The proof of this theorem can be found in the textbook such as Mohri et al.
Now we analyze a Rademacher complexity of a proposed function class.
Let H be a speciﬁed function class and F := f f ( x [Equation] ; x
[Equation]
(cid:24)
[Equation]
p
[Equation]
x [Equation] [Equation] x ; x ′ ; (cid:21) ))]
j h 2 Hg as deﬁned in equation 3.
Suppose that the empirical Rademacher complexity of H can be bounded with some constant C H as follows.
[Equation]
We can prove this assumption holds for neural network models in a similar manner to Gao & Zhou (2016).
Then we have the following theorem.
Theorem 1.
Suppose that ℓ is a (cid:26) -Lipschitz function with respect to the ﬁrst argument (0 < (cid:26) < 1 ) and H satisﬁes the assumption in equation 9.
Let F = f f : X !
R g be a function class of f deﬁned in equation 3.
Then we have the following inequality
[Equation]
where C (cid:21) =
E (cid:21) [ (cid:21) [Equation] +
[Equation] (cid:21) [Equation] Note that 1 n ∑ ni =1 ∥ x i ∥ 22 (cid:21) ∥ 1 n ∑ ni =1 x i ∥ 22 always holds from Jensen’s inequality and C (cid:21) 2 [0 ; 1] .
Thus, sample-mix can reduce the empirical Rademacher complexity of the function class, which reduces the generalization gap (i.e., L ( f ) (cid:0) b L ( f )) .
For example, when p ( (cid:21) )
=
Beta( (cid:11) + 1 ; (cid:11) ) in equation 3, we have C (cid:21) =
(cid:11) +1 2 (cid:11) +1 , which is a monotonically decreasing function with respect to (cid:11) .
Hence, we claim that larger (cid:11) can be effective for the smaller generalization gap.
We experimentally analyze the behavior with respect to (cid:11) in Sec. 3.
E XPERIMENTS ON CIFAR D ATASETS
In this section, we analyze the behavior of our proposed framework through experiments on CIFAR10/100 datasets (Krizhevsky & Hinton, 2009).
We evaluated the classiﬁcation performances with two neural network architectures, VGG16 (Simonyan & Zisserman, 2015) and PreActResNet18 (He et al., 2016).
The details of the experimental setting are described in Appendix B.
For our proposed DIP, output after ﬁnal fully-connected layer is used as h ( x ) in equation 3 and the expected output is approximated by 500 times monte carlo sampling in test stage.
As we discussed in Sec. 2.3, there are two types of optimization process when S = 1 .
We evaluated both label-preserving and label-mixing style training.
We set p ( (cid:21) )
=
Beta( (cid:11) + 1 ; (cid:11) ) for label-preserving sample-mix training and p ( (cid:21) )
=
Beta( (cid:11); (cid:11) ) for label-mixing sample-mix training.
Note that the prediction was computed with p ( (cid:21) )
=
Beta( (cid:11) +1 ; (cid:11) ) in test stage even when label-mix style was used for training.
For two baseline methods, we trained a classiﬁer with (i) standard training (without sample-mix) and (ii) Mixup (Zhang et al., 2018) training (label-mixing style).
To evaluate the performances of these methods, we computed the prediction only from clean samples.
We used p ( (cid:21) )
=
Beta( (cid:11); (cid:11) ) for Mixup training.
We show the classiﬁcation performances in Table 1 and generalization gap (i.e., the gap between train and test performances) in Fig.
Due to limited space, the magniﬁed versions of experimental results are shown in Appendix C. As can be seen in Table 1, our proposed method is likely to outperform existing Mixup approach.
Remarks:
For all approaches including existing Mixup, the larger (cid:11) leads to the smaller generalization gap, which is consistent with the discussion in Sec. 2.4.
In addition, we found that the larger S is likely to enlarge the gap and deteriorate the performance on test samples.
It might be because the variance of the empirical loss function computed by S times sampling plays a role of regularization
[Table] [Table]
li
[Table]
t i [Table] g
[Table]
a
[Table]
[Table]
[Figure]
C ONCLUSION
In this paper, we proposed a novel framework called Data Interpolating Prediction (DIP), where sample-mix is encapsulated in the hypothesis class of a classiﬁer.
We theoretically established the generalization error bound via Rademacher complexity and showed that sample-mix is effective to reduce the generalization gap.
Through experiments on CIFAR datasets, we demonstrated that our approach can outperform existing Mixup data augmentation.
A P ROOFS OF T HEOREM
In this section, we give a complete proof of theorem 1.
The empirical Rademacher complexity is deﬁned as follows.
Deﬁnition 1.
Let n be a positive integer, x 1 ; :::; x n be i.i.d.
random variables drawn from p ( x ) , F = f f : X !
R g be a class of measurable functions, and ϵ =
( ϵ 1 ; :::; ϵ n ) be Rademacher random variables, namely, random variables taking +1 and (cid:0) 1 with the equal probabilities.
Then the empirical Rademacher complexity of F is deﬁned as
[Equation]
We assume that ℓ is a (cid:26) -lipschitz function with respect to ﬁrst argument.
Here we have the following useful lemma.
The proof of this lemma can be found in (Mohri et al., 2012).
Lemma 1 (Talagrand’s lemma) .
Let (cid:8) : R !
R be an (cid:26) -lipschitz function.
Then for any hypothesis set H of real valued function functions, the following inequality holds:
[Equation]
From this lemma, we have
[Equation]
Let F :=
f f ( x ) =
E
[Equation]
x ′ (cid:24) b p ( x )
[ h ( ( x ; x ′ ; (cid:21) ))]
j h 2 Hg .
In equation 9, we assume that
[Equation]
n vuut n ∑ i = ∥ x i ∥ 22 :
Now we can bound b R n ( F ) as follows.
[Equation]
[Equation] [Equation] [Equation] [Equation]
of
[Equation]
vuut
[Equation]
[Equation]
n vuut
[Equation]
; x ′ (cid:24) b p ( x ) [ n ∑ i = f (cid:21) {+
+} ∥ x i ∥ 22 + 2 (cid:21) (1 (cid:0) (cid:21) ) ⟨ x i ; x ′ ⟩ + (1 (cid:0) (cid:21) )
∥ x ′ ∥
22 g ] =
C H n vuuut
E (cid:21) (cid:24) p ( (cid:21) ) 4n n ∑ i = n ∑ j = f (cid:21) 2 ∥ x i ∥ 22 + 2 (cid:21) (1 (cid:0) (cid:21) ) ⟨ x i ; x j ⟩ + (1 (cid:0) (cid:21) )
∥ x j ∥ 22 g 5 ( ∵ b p ( x ) = 1 n n ∑ i = (cid:14) ( x = x i )) =
[Equation]
n vuuut
E (cid:21) (cid:24) p ( (cid:21) ) 4 (cid:21)n n ∑ i = ∥ x i ∥ 22 + 2 (cid:21) (1 (cid:0) (cid:21) ) ⟨ 1 n n ∑ i = x i ; 1 n n [Table] x j ⟩ + (1 (cid:0) (cid:21) )
[Table]
n
[Table] = [Table]
x
[Table]
=
[Equation]
n vuut E (cid:21) (cid:24) p ( (cid:21) )
[ (cid:21) 2 + (1 (cid:0) (cid:21) ) 2 ]
[Table]
n
[Table]
(cid:20) C H p n vuut C (cid:21) 1 n n ∑
[Table]
x
[Table]
+
[Table]
C
[Table]
[Table]
n
[Table]
By combining the above result and equation 13, we complete the proof of this theorem.
B D ETAILS OF E XPERIMENTAL S ETTING
In this section, we describe the details of training for experiments in Section 3.
B. T RAINING
VGG16 (Simonyan & Zisserman, 2015) and PreActResNet18 (He et al., 2016) was used for experiments.
We did not apply Dropout similarly to Mixup (Zhang et al., 2018).
For all experiments, we trained a neural network for 200 epoch.
Learning rate was set to 0 : 1 in the beginning and multiplied by 0 : 1 at 100 and 150 epoch.
We applied standard augmentation such as cropping and ﬂipping.
The size of mini-batch was set to 128 .
We set p ( (cid:21) )
=
Beta( (cid:11) + 1 ; (cid:11) ) for label-preserving sample-mix training and p ( (cid:21) )
=
Beta( (cid:11); (cid:11) ) for label-mixing sample-mix training.
(cid:21) was generated for each sample in mini-batch, and x ′ was obtained by permutation of samples in mini-batch.
B. P REDICTION
For standard without sample-mix method and Mixup method, we predicted labels of test samples from clean samples.
For proposed method, we predicted labels from the expectation over mixed samples computed by monte carlo approximation.
In the same manner to the training process, we sampled (cid:21) and x ′ 500 times and calculated the average to obtain the ﬁnal output.
In evaluation stage, data augmentation except for sample-mix was turned off.
C M AGNIFIED V ERSIONS OF E XPERIMENTAL R ESULTS
In this section, we present the magnﬁed version of experimental results in Section 3.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]