Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity
Abstract
While deep neural networks show great performance on ﬁtting to the training distribution, improving the networks’ generalization performance to the test distribution and robustness to the sensitivity to input perturbations still remain as a challenge.
Although a number of mixup based augmentation strategies have been proposed to partially address them, it remains unclear as to how to best utilize the supervisory signal within each input data for mixup from the optimization perspective.
We propose a new perspective on batch mixup and formulate the optimal construction of a batch of mixup data maximizing the data saliency measure of each individual mixup data and encouraging the supermodular diversity among the constructed mixup data.
This leads to a novel discrete optimization problem minimizing the diﬀerence between submodular functions.
We also propose an eﬃcient modular approximation based iterative submodular minimization algorithm for eﬃcient mixup computation per each minibatch suitable for minibatch based neural network training.
Our experiments show the proposed method achieves the state of the art generalization, calibration, and weakly supervised localization results compared to other mixup methods.
The source code is available at https://github.com/snu-mllab/Co-Mixup .
Introduction
Deep neural networks have been applied to a wide range of artiﬁcial intelligence tasks such as computer vision, natural language processing, and signal processing with remarkable performance (Ren et al., 2015;
Devlin et al., 2018; Oord et al., 2016).
However, it has been shown that neural networks have excessive representation capability and can even ﬁt random data (Zhang et al., 2016).
Due to these characteristics, the neural networks can easily overﬁt to training data and show a large generalization gap when tested on previously unseen data.
To improve the generalization performance of the neural networks, a body of research has been proposed to develop regularizers based on priors or to augment the training data with task-dependent transforms (Bishop, 2006; Cubuk et al., 2019).
Recently, a new taskindependent data augmentation technique, called mixup , has been proposed (Zhang et al., 2018).
The original mixup, called Input Mixup , linearly interpolates a given pair of input data and can be easily applied to various data and tasks, improving the generalization performance and robustness of neural networks.
Other mixup methods, such as manifold mixup (Verma et al., 2019) or CutMix (Yun et al., 2019), have also been proposed addressing diﬀerent ways to mix a given pair of input data.
Puzzle Mix (Kim et al., 2020) utilizes saliency information and local statistics to ensure mixup data to have rich supervisory signals.
However, these approaches only consider mixing a given random pair of input data and do not fully utilize the rich informative supervisory signal in training data including collection of object saliency, relative arrangement, etc.
In this work, we simultaneously consider mixmatching diﬀerent salient regions among all input data so that each generated mixup example accumulates as many salient regions from multiple input data as possible while ensuring
Correspondence to: Hyun Oh Song.
[Figure]
diversity among the generated mixup examples.
To this end, we propose a novel optimization problem that maximizes the saliency measure of each individual mixup example while encouraging diversity among them collectively.
This formulation results in a novel discrete submodular-supermodular objective.
We also propose a practical modular approximation method for the supermodular term and present an eﬃcient iterative submodular minimization algorithm suitable for minibatch-based mixup for neural network training.
As illustrated in the Figure 1, while the proposed method, Co-Mixup , mix-matches the collection of salient regions utilizing inter-arrangements among input data, the existing methods do not consider the saliency information (Input Mixup & CutMix) or disassemble salient parts (Puzzle Mix).
We verify the performance of the proposed method by training classiﬁers on CIFAR-100,
Tiny-ImageNet, ImageNet, and the Google commands dataset (Krizhevsky et al., 2009;
Chrabaszcz et al., 2017; Deng et al., 2009; Warden, 2017).
Our experiments show the models trained with Co-Mixup achieve the state of the performance compared to other mixup baselines.
In addition to the generalization experiment, we conduct weakly-supervised object localization and robustness tasks and conﬁrm Co-Mixup outperforms other mixup baselines.
Related works
Mixup Data augmentation has been widely used to prevent deep neural networks from over-ﬁtting to the training data (Bishop, 1995).
The majority of conventional augmentation methods generate new data by applying transformations depending on the data type or the target task (Cubuk et al., 2019).
Zhang et al.
(2018) proposed mixup , which can be independently applied to various data types and tasks, and improves generalization and robustness of deep neural networks.
Input mixup (Zhang et al., 2018) linearly interpolates between two input data and utilizes the mixed data with the corresponding soft label for training.
Following this work, manifold mixup (Verma et al., 2019) applies the mixup in the hidden feature space, and CutMix (Yun et al., 2019) suggests a spatial copy and paste based mixup strategy on images.
Guo et al.
(2019) trains an additional neural network to optimize a mixing ratio.
Puzzle Mix (Kim et al., 2020) proposes a mixup method based on saliency and local statistics of the given data.
In this paper, we propose a discrete optimization-based mixup method simultaneously ﬁnding the best combination of collections of salient regions among all input data while encouraging diversity among the generated mixup examples.
Saliency The seminal work from Simonyan et al.
(2013) generates a saliency map using a pre-trained neural network classiﬁer without any additional training of the network.
Following the work, measuring the saliency of data using neural networks has been studied to obtain a more precise saliency map (Zhao et al., 2015; Wang et al., 2015) or to reduce the saliency computation cost (Zhou et al., 2016; Selvaraju et al., 2017).
The saliency information is widely applied to the tasks in various domains, such as object segmentation or speech recognition (Jung and Kim, 2011; Kalinli and Narayanan, 2007).
Submodular-Supermodular optimization
A submodular (supermodular) function is a set function with diminishing (increasing) returns property (Narasimhan and Bilmes, 2005).
It is known that any set function can be expressed as the sum of a submodular and supermodular function (Lovász, 1983), called BP function.
Various problems in machine learning can be naturally formulated as BP functions (Fujishige, 2005), but it is known to be NP-hard (Lovász, 1983).
Therefore, approximate algorithms based on modular approximations of submodular or supermodular terms have been developed (Iyer and Bilmes, 2012).
Our formulation falls into a category of BP function consisting of smoothness function within a mixed output (submodular) and a diversity function among the mixup outputs (supermodular).
Preliminary
Existing mixup methods return { h ( x 1 , x i (1) ) , . . .
, h ( x m , x i ( m ) ) } for given input data { x 1 , . . .
, x m } , where h : X × X
→ X is a mixup function and ( i (1) , . .
, i ( m )) is a random permutation of the data indices.
In the case of input mixup, h ( x, x (cid:48) ) is λx + (1 − λ ) x (cid:48) , where λ ∈ [0 , 1] is a random mixing ratio.
Manifold mixup applies input mixup in the hidden feature space, and CutMix uses h ( x, x (cid:48) )
= 1 B (cid:12) x
+ (1 − 1 B ) (cid:12) x (cid:48) , where 1 B is a binary rectangular-shape mask for an image x and (cid:12) represents the element-wise product.
Puzzle
Mix deﬁnes h ( x, x (cid:48) ) as z (cid:12) Π (cid:124) x
+ (1 − z ) (cid:12) Π (cid:48)
(cid:124) x (cid:48) , where Π is a transport plan and z is a discrete mask.
In detail, for x ∈ R n , Π ∈ { 0 , 1 } n and z ∈ L n for L = { lL | l = 0 , 1 , . . .
, L } .
In this work, we extend the existing mixup functions as h : X m → X m (cid:48) which performs mixup on a collection of input data and returns another collection.
Let x B ∈ R m × n denote the batch of input data in matrix form.
Then, our proposed mixup function is
[Equation]
where z j ∈ L m × n for j = 1 , . .
, m (cid:48) with L = { lL | l = 0 , 1 , . . .
, L } and g : R m × n → R n returns a column-wise sum of a given matrix.
Note that, the k th column of z j , denoted as z j,k ∈ L m , can be interpreted as the mixing ratio among m inputs at the k th location.
Also, we enforce (cid:107)
z
j,k (cid:107) 1 = 1 to maintain the overall statistics of the given input batch.
Given the one-hot target labels y B ∈ { 0 , 1 } m × C of the input data with C classes, we generate soft target labels for mixup data as y (cid:124) B ˜ o j for j = 1 , . . .
, m (cid:48) , where ˜ o
j = 1 n (cid:80) nk =1 z j,k ∈ [0 , 1] m represents the input source ratio of the j th mixup data.
We train models to estimate the soft target labels by minimizing the cross-entropy loss.
Method 4. Objective
Saliency Our main objective is to maximize the saliency measure of mixup data while maintaining the local smoothness of data, i.e. , spatially nearby patches in a natural image look similar, temporally adjacent signals have similar spectrum in speech, etc.
(Kim et al., 2020).
As we can see from CutMix in Figure 1, disregarding saliency can give a misleading supervisory signal by generating mixup data that does not match with the target soft label.
While the existing mixup methods only consider the mixup between two inputs, we generalize the number of inputs m to any positive integer.
Note, each k th location of outputs has m candidate sources from the inputs.
We model the unary labeling cost as the negative value of the saliency, and denote the cost vector at the k th location as c k ∈ R m .
For the saliency measure, we calculate the gradient values of training loss with respect to the input and measure (cid:96) 2 norm of the gradient values across input channels (Simonyan et al., 2013; Kim et al., 2020).
Note that this method does not require any additional architecture dependent modules for saliency calculation.
In addition to the unary cost, we encourage adjacent locations to have similar labels for the smoothness of each mixup data.
In summary, the objective can be formulated as follows:
[Equation]
s u m s u p e
r m o d u l a r un a r y
Diverse z ∗
Salient but not salient but not diverse
O b j ec t i v e v a l u e (a) 1 2 3 4
Number of mixed inputs
C o un t s (b) small τ large τ 1.
no-mix CutMix Co-Mixup Input PuzzleMix
B a t c h s a li e n c y (c) 0 0.2 0.4 0.6 0.8 1.
baselines τ
D i v e r
s
i t y (d) j (cid:48) (cid:54)
(cid:124) where the prior p is given by z j,k ∼ 1 L Multi ( L, λ ) with λ = ( λ 1 , . . .
, λ m ) ∼
Dirichlet ( α, . . . , α ) , which is a generalization of the mixing ratio distribution of Zhang et al.
(2018), and N denotes a set of adjacent locations ( i.e. , neighboring image patches in vision, subsequent spectrums in speech, etc.).
Diversity Note that the naive generalization above leads to the identical outputs because the objective is separable and identical for each output.
In order to obtain diverse mixup outputs, we model a similarity penalty between outputs.
First, we represent the input source information of the j th output by aggregating assigned labels as (cid:80) nk =
z j,k .
For simplicity, let us denote (cid:80) nk =
z j,k as o j .
Then, we measure the similarity between o j ’s by using the inner-product on R m .
In addition to the input source similarity between outputs, we model the compatibility between input sources, represented as a symmetric matrix
A c ∈ R m × m + .
Speciﬁcally, A c [ i 1 , i 2 ] quantiﬁes the degree to which input i
and i 2 are suitable to be mixed together.
In summary, we use inner-product on A =
(1 − ω )
I + ωA c for ω ∈ [0 , 1] , resulting in a supermodular penalty term.
Note that, by minimizing (cid:104)
o j , o j (cid:48) (cid:105)
A
= o (cid:124)
j Ao j (cid:48) , ∀ j (cid:54) = j (cid:48) , we penalize output mixup examples with similar input sources and encourage each individual mixup examples to have high compatibility within.
In this work, we measure the distance between locations of salient objects in each input and use the distance matrix
A c [ i, j ] = (cid:107) argmax k
s i
[ k ] − argmax k s j [ k ] (cid:107) 1 , where s i is the saliency map of the i th input and k is a location index ( e.g. , k is a 2-D index for image data).
From now on, we denote this inner-product term as the compatibility term.
Over-penalization The conventional mixup methods perform mixup as many as the number of examples in a given mini-batch.
In our setting, this is the case when m = m (cid:48) .
However, the compatibility penalty between outputs is inﬂuenced by the pigeonhole principle.
For example, suppose the ﬁrst output consists of two inputs.
Then, the inputs must be used again for the remaining m (cid:48) − 1 outputs, or only m − 2 inputs can be used.
In the latter case, the number of available inputs ( m − 2 ) is less than the outputs ( m (cid:48) − 1 ), and thus, the same input must be used more than twice.
Empirically, we found that the remaining compatibility term above over-penalizes the optimization so that a substantial portion of outputs are returned as singletons without any mixup.
To mitigate the over-penalization issue, we apply clipping to the compatibility penalty term.
Speciﬁcally, we model the objective so that no extra penalty would occur when the compatibility among outputs is below a certain level.
Now we present our main objective as following:
[Equation]
where
[Equation]
In Figure 2, we describe the properties of the BP optimization problem of Equation (1) and statistics of the resulting mixup data.
Next, we verify the supermodularity of the compatibility term.
We ﬁrst extend the deﬁnition of the submodularity of a multi-label function as follows (Windheuser et al., 2012).
Deﬁnition 1.
For a given label set L , a function s : L m × L m → R is pairwise submodular, if ∀ x, x (cid:48) ∈ L m , s ( x, x )+ s ( x (cid:48) , x (cid:48) )
≤ s ( x, x (cid:48) )
+ s ( x (cid:48) , x ) .
A function s is pairwise supermodular, if − s is pairwise submodular.
Proposition 1.
The compatibility term f c in Equation (1) is pairwise supermodular for every pair of ( z j 1 ,k , z
j 2 ,k ) if A is positive semi-deﬁnite.
Proof.
See Appendix B.1.
Finally note that, A = (1 − ω )
I + ωA c , where A c is a symmetric matrix.
By using spectral decomposition, A c can be represented as UDU (cid:124) , where D is a diagonal matrix and U (cid:124) U = UU (cid:124)
=
I .
Then,
A =
U ((1 − ω ) I + ωD ) U (cid:124) , and thus for small ω > 0 , we can guarantee A to be positive semi-deﬁnite.
4. Algorithm
Our main objective consists of modular ( unary, prior ), submodular ( smoothness ), and supermodular ( compatibility ) terms.
To optimize the main objective, we employ the submodularsupermodular procedure by iteratively approximating the supermodular term as a modular function (Narasimhan and Bilmes, 2005).
Note that z j represents the labeling of the j th output and o j represents the aggregated input source information of the j th output, (cid:80) nk =
z j,k .
Before introducing our algorithm, we ﬁrst inspect the simpler case without clipping.
Proposition 2.
The compatibility term f c without clipping is modular with respect to z j .
Proof.
Note, A is a positive symmetric matrix by the deﬁnition.
Then, for an index j 0 , we can represent f c without clipping in terms of o j 0 as (cid:80) m (cid:48)
j =1 (cid:80) m (cid:48)
j (cid:48)
=1 ,j (cid:48) (cid:54) =
j o (cid:124)
j Ao j (cid:48)
= 2 (cid:80) m
[Table]
c , where v - j 0 ∈ R m and c ∈ R are values independent with o j 0 .
Finally, v
[Table]
By Proposition 2, we can apply a submodular minimization algorithm to optimize the objective with respect to z j when there is no clipping.
Thus, we can optimize the main objective without clipping in coordinate descent fashion (Wright, 2015).
For the case with clipping, we modularize the supermodular compatibility term under the following criteria:
The modularized function value should increase as the compatibility across outputs increases.
The modularized function should not apply an extra penalty for the compatibility below a certain level.
[Figure]
i
[Figure]
Computemodularapproximateddiversityfor j
Borrowing the notation from the proof in Proposition 2, for an index j , f c ( z ) = max { τ, v (cid:124) - j o j + c } = max { τ − c, v (cid:124) - j o j } + c .
Note, o j = (cid:80) nk =
z j,k represents the input source information of the j th output and v - j = 2 (cid:80) m (cid:48)
j (cid:48)
=1 ,j (cid:48) (cid:54) =
j Ao j (cid:48) encodes the status of the other outputs.
Thus, we can interpret the supermodular term as a penalization of each label of o j in proportion to the corresponding v - j value (criterion 1), but not for the compatibility below τ − c (criterion 2).
As a modular function which satisﬁes the criteria above, we use the following function:
[Equation]
Note that, by satisfying the criteria above, the modular function reﬂects the diversity and over-penalization desiderata described in Section 4.1.
We illustrate the proposed mixup procedure with the modularized diversity penalty in Figure 3.
Proposition 3.
The modularization given by Equation (2) satisﬁes the criteria above.
Proof.
See Appendix B.2.
Initialize z as z (0) .
Let z ( t ) denote a solution of the t th step.
Φ : modularization operator based on Equation (2).
for t = 1 , . . .
, T do for j = 1 , . . .
, m (cid:48) do f ( t )
j
( z j ) :
= f
( z j ; z ( t ) 1: j − 1 , z
( t − 1) j +1: m (cid:48) ) .
˜ f
( t )
j = Φ( f ( t ) j ) .
Solve z ( t )
j = argmin ˜ f
( t ) j
( z j ) .
end for end for return z ( T )
By applying the modular approximation described in Equation (2) to f c in Equation (1), we can iteratively apply a submodular minimization algorithm to obtain the ﬁnal solution as described in Algorithm 1.
In detail, each step can be performed as follows:
1) Conditioning the main objective f on the current values except z j , denoted as f j ( z j )
=
f
( z j ; z 1: j − 1 , z j +1: m (cid:48) ) .
2) Modularization of the compatibility term of f j as Equation (2), resulting in a submodular function ˜ f j .
We denote the modularization operator as Φ , i.e. , ˜ f
j = Φ( f j ) .
Applying a submodular minimization algorithm to ˜ f j .
Please refer to Appendix C for implementation details.
Analysis Narasimhan and Bilmes (2005) proposed a modularization strategy for general supermodular set functions, and apply a submodular minimization algorithm that can monotonically decrease the original BP objective.
However, the proposed Algorithmbased on Equation (2) is much more suitable for minibatch based mixup for neural network training than the set modularization proposed by Narasimhan and Bilmes (2005) in terms of complexity and modularization variance due to randomness.
For simplicity, let us assume each z j,k is an m -dimensional one-hot vector.
Then, our problem is to optimize m (cid:48) n one-hot m -dimensional vectors.
To apply the set modularization method, we need to assign each possible value of z j,k as an element of { 1 , 2 , . . .
, m } .
Then the supermodular term in Equation (1) can be interpreted as a set function with m (cid:48) nm elements, and to apply the set modularization, O ( m (cid:48) nm ) sequential evaluations of the supermodular term are required.
In contrast, Algorithm 1 calculates v - j in Equation (2) in only O ( m (cid:48) ) time per each iteration.
In addition, each modularization step of the set modularization method requires a random permutation of the m (cid:48) nm elements.
In this case, the optimization can be strongly aﬀected by the randomness from the permutation step.
As a result, the optimal labeling of each z j,k from the compatibility term is strongly inﬂuenced by the random ordering undermining the interpretability of the algorithm.
Please refer to Appendix D for empirical comparison between Algorithm 1 and the method by
Narasimhan and Bilmes (2005).
Experiments
We evaluate our proposed mixup method on generalization, weakly supervised object localization, calibration, and robustness tasks.
First, we compare the generalization performance of the proposed method against baselines by training classiﬁers on CIFAR-100 (Krizhevsky et al., 2009), Tiny-ImageNet (Chrabaszcz et al., 2017), ImageNet (Deng et al., 2009), and the
Google commands speech dataset (Warden, 2017).
Next, we test the localization performance of classiﬁers following the evaluation protocol of Qin and Kim (2019).
We also measure calibration error (Guo et al., 2017) of classiﬁers to verify Co-Mixup successfully alleviates the over-conﬁdence issue by Zhang et al.
In Section 5.4, we evaluate the robustness of the classiﬁers on the test dataset with background corruption in response to the recent problem raised by Lee et al.
(2020) that deep neural network agents often fail to generalize to unseen environments.
Finally, we perform a sensitivity analysis of Co-Mixup and provide the results in Appendix F.3.
5. Classification
We ﬁrst train PreActResNet18 (He et al., 2016), WRN16-8 (Zagoruyko and Komodakis, 2016), and ResNeXt29-4-24 (Xie et al., 2017) on CIFAR-100 for 300 epochs.
We use stochastic gradient descent with an initial learning rate of 0.2 decayed by factor 0.1 at epochsand 200.
We set the momentum as 0.9 and add a weight decay of 0.0001.
With this setup, we train a vanilla classiﬁer and reproduce the mixup baselines (Zhang et al., 2018; Verma et al., 2019; Yun et al., 2019; Kim et al., 2020), which we denote as Vanilla, Input, Manifold,
CutMix, Puzzle Mix in the experiment tables.
Note that we use identical hyperparameters regarding Co-Mixup over all of the experiments with diﬀerent models and datasets, which are provided in Appendix E.
Table 1 shows Co-Mixup signiﬁcantly outperforms all other baselines in Top-1 error rate.
Co-Mixup achieves 19.87% in Top-1 error rate with PreActResNet18, outperforming the best baseline by 0.75%.
We further test Co-Mixup on diﬀerent models (WRN16-8 & ResNeXt294-24) and verify Co-Mixup improves Top-1 error rate over the best performing baseline.
[Table]
We further test Co-Mixup on other datasets; Tiny-ImageNet, ImageNet
[Table]
and the Google commands dataset (Table 1).
For Tiny-ImageNet, we train PreActResNet18 for 1200 epochs
[Figure]
following the training protocol of Kim et al.
As a result, Co-Mixup consistently improves Top-1 error rate over baselines by 0.67%.
In the ImageNet experiment, we follow the experimental protocol provided in Puzzle Mix (Kim et al., 2020), which trains ResNet-
(He et al., 2015) for 100 epochs.
As a result, Co-Mixup outperforms all of the baselines in
Top-1 error rate.
We further test Co-Mixup on the speech domain with the Google commands dataset and VGG-11 (Simonyan and Zisserman, 2014).
We provide a detailed experimental setting and dataset description in Appendix F.1.
From Table 1, we conﬁrm that Co-Mixup is the most eﬀective in the speech domain as well.
5. Localization
We compare weakly supervised object localization (WSOL) performance of classiﬁers trained on ImageNet (in Table 1) to demonstrate that our mixup method better guides a classiﬁer to focus on salient regions.
We test the localization performance using CAM (Zhou et al., 2016), a WSOL method using a pre-trained classiﬁer.
We evaluate localization performance following the evaluation protocol in Qin and Kim (2019), with binarization threshold 0.25 in
CAM.
Table 2 summarizes the WSOL performance of various mixup methods, which shows that our proposed mixup method outperforms other baselines.
5. Calibration
We evaluate the expected calibration error (ECE) (Guo et al., 2017) of classiﬁers trained on CIFAR-100.
Note, ECE is calculated by the weighted average of the absolute diﬀerence between the conﬁdence and accuracy of a classiﬁer.
As shown in Table 2, the Co-Mixup classiﬁer has the lowest calibration error among baselines.
From Figure 4, we ﬁnd that other mixup baselines tend to have under-conﬁdent predictions resulting in higher ECE values even than Vanilla network (also pointed out by Wen et al.
(2020)), whereas Co-Mixup has best-calibrated predictions resulting in relatively 48% less ECE value.
We provide more ﬁgures and results with other datasets in Appendix F.2.
[Table]
5. Robustness
In response to the recent problem raised by Lee et al.
(2020) that deep neural network agents often fail to generalize to unseen environments, we consider the situation where the statistics of the foreground object, such as color or shape, is unchanged, but with the corrupted (or replaced) background.
In detail, we consider the following operations: 1) replacement with another image and 2) adding Gaussian noise.
We use ground-truth bounding boxes to separate the foreground from the background, and then apply the previous operations independently to obtain test datasets.
We provide a detailed description of datasets in
Appendix G.
With the test datasets described above, we evaluate the robustness of the pre-trained classiﬁers.
As shown in Table 3, Co-Mixup shows signiﬁcant performance gains at various background corruption tests compared to the other mixup baselines.
For each corruption case, the classiﬁer trained with Co-Mixup outperforms the others in Top-1 error rate with the performance margins of 2.86% and 3.33% over the Vanilla model.
[Table]
5. Baselines with multiple inputs
To further investigate the eﬀect of the number of inputs for the mixup in isolation, we conduct an ablation study on baselines using multiple mixing inputs.
For fair comparison, we use Dirichlet( α, . . .
, α ) prior for the mixing ratio distribution and select the best performing α in { 0 . 2 , 1 . 0 , 2 . 0 } .
Note that we overlay multiple boxes in the case of CutMix.
Tablereports the classiﬁcation test errors on CIFAR-100 with PreActResNet18.
From the table, we ﬁnd that mixing multiple inputs decreases the performance gains of each mixup baseline.
These results demonstrate that mixing multiple inputs could lead to possible degradation of the performance and support the necessity of considering saliency information and diversity as in Co-Mixup.
# inputs for
[Table]
Conclusion
We presented Co-Mixup for optimal construction of a batch of mixup examples by ﬁnding the best combination of salient regions among a collection of input data while encouraging diversity among the generated mixup examples.
This leads to a discrete optimization problem minimizing a novel submodular-supermodular objective.
In this respect, we present a practical modular approximation and iterative submodular optimization algorithm suitable for minibatch based neural network training.
Our experiments on generalization, weakly supervised object localization, and robustness against background corruption show Co-Mixup achieves the state of the art performance compared to other mixup baseline methods.
The proposed generalized mixup framework tackles the important question of ‘what to mix?’
while the existing methods only consider ‘how to mix?’.
We believe this work can be applied to new applications where the existing mixup methods have not been applied, such as multi-label classiﬁcation, multi-object detection, or source separation.
Acknowledgements
This research was supported in part by Samsung Advanced Institute of Technology, Samsung
Electronics Co., Ltd, Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)
(No. 2020-0-00882, (SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning), and Research Resettlement Fund for the new faculty of Seoul National University.
Hyun Oh Song is the corresponding author.
A Supplementary notes for objective A. Notations
In Table 5, we provide a summary of notations in the main text.
Notation
Meaning m , m (cid:48) , n #
[Table]
[Table] [Table]
c k
[Table]
n ) labeling cost
[Table]
input sources
[Table] [Table]
input
[Table]
ratio at
[Table]
labeling
[Table]
aggregation of
[Table]
output
[Table]
compatibility between inputs
A. Interpretation of compatibility
In our main objective Equation (1), we introduce a compatibility matrix A
=
(1 − ω )
I + ωA c between inputs.
By minimizing (cid:104)
o
j , o j (cid:48) (cid:105) A for j (cid:54) =
j (cid:48) , we encourage each individual mixup examples to have high compatibility within.
Figure 5 explains how the compatibility term works by comparing simple cases.
Note that our framework can reﬂect any compatibility measures for the optimal mixup.
[Figure]
more
A 13 .
Then, the left Co-Mixup result has a larger inner-product value (cid:104)
o 1 , o 2 (cid:105)
A than the right.
Thus the mixup result on the right has higher compatibility than the result on the left within each output example.
B Proofs B. Proof of proposition
Lemma 1.
For a positive semi-deﬁnite matrix A ∈ R m × m +
and x, x (cid:48) ∈ R m , s ( x, x (cid:48) )
=
x (cid:124)
Ax (cid:48) is pairwise supermodular.
Proof.
s
[Equation]
[Equation] [Equation]
[Equation] [Equation]
=
[Equation]
Ax + x
[Equation]
Ax − 2 x (cid:124)
Ax (cid:48)
= ( x − x (cid:48) )
(cid:124)
A ( x − x (cid:48) ) , and because A is positive semi-deﬁnite, ( x − x (cid:48) )
(cid:124)
A ( x − x (cid:48) ) ≥ 0 .
Proposition 1.
The compatibility term f c in Equation (1) is pairwise supermodular for every pair of ( z j 1 ,k , z
j 2 ,k ) if A is positive semi-deﬁnite.
Proof.
For j 1 and j 2 , s.t. , j 1 (cid:54) =
j
[Equation]
, max
[Equation]
By Lemma 1, − z (cid:124)
j 1 ,k Az j 2 ,k is pairwise submodular, and because a budget additive function preserves submodularity (Horel and Singer, 2016), min {− τ, − c − 2 z (cid:124)
[Equation]
1 ,k Az j 2 ,k } is pairwise submodular with respect to ( z j 1 ,k , z
j 2 ,k ) .
B. Proof of proposition
Proposition 3.
The modularization given by Equation (2) satisﬁes the criteria.
Proof.
Note, by the deﬁnition in Equation (1), the compatibility between the j th and j (cid:48)
th outputs is o (cid:124) j (cid:48)
Ao j , and thus, v (cid:124) - j o j represents the compatibility between the j th output and the others.
In addition, (cid:107) o j (cid:107)
=
(cid:107) (cid:80) nk =
z j,k (cid:107) 1 = (cid:80) nk =1 (cid:107) z
j,k (cid:107) 1 = n .
In a local view, for the given o j , let us deﬁne a vector o (cid:48)
j as o (cid:48)
j [ i 1 ]
=
o j
[ i 1 ] + α and o (cid:48)
j [ i 2 ]
=
o j
[ i 2 ] − α for α > 0 .
Without loss of generality, let us assume v - j is sorted in ascending order.
Then, v (cid:124) - j o j ≤ v (cid:124) - j o (cid:48)
j implies i 1 > i 2 , and because the max function preserves the ordering, max { τ (cid:48) , v - j } (cid:124) o
j ≤ max { τ (cid:48) , v - j } (cid:124) o (cid:48)
j .
Thus, the criterion 1 is locally satisﬁed.
Next, for τ (cid:48) > 0 , (cid:107) max { τ (cid:48) , v - j } (cid:124) o j (cid:107) 1 ≥ τ (cid:48) (cid:107) o j (cid:107) 1 = τ (cid:48) n .
Let ∃ i 0 s.t.
for i < i 0 , v - j
[ i ] < τ (cid:48) , and for i ≥ i 0 , v - j
[ i ] ≥ τ (cid:48) .
Then, for o j containing positive elements only in indices smaller than i 0 , max { τ (cid:48) , v - j } (cid:124) o
j = τ (cid:48) n which means there is no extra penalty from the compatibility.
In this respect, the proposed modularization satisﬁes the criterion 2 as well.
C Implementation details
We perform the optimization after down-sampling the given inputs and saliency maps to the speciﬁed size ( 4 × 4 ).
After the optimization, we up-sample the optimal labeling to match the size of the inputs and then mix inputs according to the up-sampled labeling.
For the saliency measure, we calculate the gradient values of training loss with respect to the input data and measure (cid:96) 2 norm of the gradient values across input channels (Simonyan et al., 2013).
In classiﬁcation experiments, we retain the gradient information of network weights obtained from the saliency calculation for regularization.
For the distance in the compatibility term, we measure (cid:96) 1 -distance between the most salient regions.
For the initialization in Algorithm 1, we use i.i.d.
samples from a categorical distribution with equal probabilities.
We use alpha-beta swap algorithm from pyGCO 1 to solve the minimization step in Algorithm 1, which can ﬁnd local-minima of a multi-label submodular function.
However, the worst-case complexity of alpha-beta swap algorithm with |L| = 2 is
O ( m 2 n ) , and in the case of mini-batch with 100 examples, iteratively applying the algorithm can become a bottleneck during the network training.
To mitigate the computational overhead, we partition the mini-batch (each of size 20) and then apply Algorithm 1 independently per each partition.
The worst-case complexity theoretic of the naive implementation of Algorithm 1 increases exponentially as |L| increases.
Speciﬁcally, the worst-case theoretic complexity of the alphabeta swap algorithm is proportional to the square of the number of possible states of z j,k , which is proportional to m |L|− 1 .
To reduce the number of possible states in a multi-label case, we solve the problem for binary labels ( |L| = 2 ) at the ﬁrst inner-cycle and then extend to multi labels ( |L| = 3 ) only for the currently assigned indices of each output in the subsequent cycles.
This reduces the number of possible states to O ( m + ¯ m |L|− 1 ) where ¯ m (cid:28) m .
Here, ¯ m means the number of currently assigned indices for each output.
Based on the above implementation, we train models with Co-Mixup in a feasible time.
For example, in the case of ImageNet training with 16 Intel I9-9980XE CPU cores and
NVIDIA RTX 2080Ti GPUs, Co-Mixup training requires 0.964s per batch, whereas the vanilla training without mixup requires 0.374s per batch.
Note that Co-Mixup requires saliency computation, and when we compare the algorithm with Puzzle Mix, which performs the same saliency computation, Co-Mixup is only slower about 1.04 times.
Besides, as we down-sample the data to the ﬁxed size regardless of the data dimension, the additional computation cost of Co-Mixup relatively decreases as the data dimension increases.
Finally, we present the empirical time complexity of Algorithm 1 in Figure 6.
As shown in the ﬁgure,
Algorithm 1 has linear time complexity over |L| empirically.
Note that we use |L| = 3 in all of our main experiments, including a classiﬁcation task.
[Figure]
The left ﬁgure shows the time complexity of the algorithm over |L| and the right ﬁgure shows the time complexity over the number of inputs m .
Note that the other parameters are ﬁxed equal to the classiﬁcation experiments setting, m = m (cid:48)
= 20 , n = 16 , and |L| = 3 .
D Algorithm Analysis
In this section, we perform comparison experiments to analyze the proposed Algorithm 1.
First, we compare our algorithm with the exact brute force search algorithm to inspect the optimality of the algorithm.
Next, we compare our algorithm with the BP algorithm proposed by Narasimhan and Bilmes (2005).
D. Comparison with Brute Force
To inspect the optimality of the proposed algorithm, we compare the function values of the solutions of Algorithm 1, brute force search algorithm, and random guess.
Due to the exponential time complexity of the brute force search, we compare the algorithms on small scale experiment settings.
Speciﬁcally, we test algorithms on settings of ( m = m (cid:48)
= 2 , n = 4) , ( m = m (cid:48)
= 2 , n = 9) , and ( m = m (cid:48)
= 3 , n = 4) varying the number of inputs and outputs ( m, m (cid:48) ) and the dimension of data n .
We generate unary cost matrix in the objective f by sampling data from uniform distribution.
We perform experiments with 100 diﬀerent random seeds and summarize the results on
Table 6.
From the table, we ﬁnd that the proposed algorithm achieves near optimal solutions over various settings.
We also measure relative errors between ours and random guess, ( f ( z ours ) − f ( z brute )) /
( f ( z random )
− f ( z brute )) .
As a result, our algorithm achieves relative error less than 0 .
D. Comparison with another BP algorithm
We compare the proposed algorithm and the BP algorithm proposed by Narasimhan and
Bilmes (2005).
We evaluate function values of solutions by each method using a random
[Table]
unary cost matrix from a uniform distribution.
We compare methods over various scales by controlling the number of mixing inputs m .
Table 7 shows the averaged function values with standard deviations in the parenthesis.
As we can see from the table, the proposed algorithm achieves much lower function values and deviations than the method by Narasimhan and Bilmes (2005) over various settings.
Note that the method by Narasimhan and Bilmes (2005) has high variance due to randomization in the algorithm.
We further compare the algorithm convergence time in Table 8.
The experiments verify that the proposed algorithm is much faster and eﬀective than the method by Narasimhan and Bilmes (2005).
Algorithm m =m
=m
=m =m
=
Ours
[Table]
[Table] [Table]
Narasimhan
[Table]
Algorithm m =m
=m
=m =m
=
[Table]
E Hyperparameter settings
We perform Co-Mixup after down-sampling the given inputs and saliency maps to the pre-deﬁned resolutions regardless of the size of the input data.
In addition, we normalize the saliency of each input to sum up to 1 and deﬁne unary cost using the normalized saliency.
As a result, we use an identical hyperparameter setting for various datasets; CIFAR-100,
Tiny-ImageNet, and ImageNet.
In details, we use ( β, γ, η, τ ) =
for all of experiments.
Note that τ is normalized according to the size of inputs ( n ) and the ratio of the number of inputs and outputs ( m/m (cid:48) ), and we use an isotropic Dirichlet distribution with α = 2 for prior p .
For a compatibility matrix, we use ω = 0 . 001 .
For baselines, we tune the mixing ratio hyperparameter, i.e., the beta distribution parameter (Zhang et al., 2018), among { 0 . 2 , 1 . 0 , 2 . 0 } for all of the experiments if the speciﬁc setting is not provided in the original papers.
F Additional Experimental Results F. Another Domain: Speech
In addition to the image domain, we conduct experiments on the speech domain, verifying
Co-Mixup works on various domains.
Following (Zhang et al., 2018), we train LeNet (LeCun
[Figure]
et al., 1998) and VGG-11 (Simonyan and Zisserman, 2014) on the Google commands dataset (Warden, 2017).
The dataset consists of 65,000 utterances, and each utterance is about one-second-long belonging to one out of 30 classes.
We train each classiﬁer for 30 epochs with the same training setting and data pre-processing of Zhang et al.
In more detail, we use 160 × 100 normalized spectrograms of utterances for training.
As shown in Table 9, we verify that Co-Mixup is still eﬀective in the speech domain.
[Table]
F. Calibration
In this section, we summarize the expected calibration error (ECE) (Guo et al., 2017) of classiﬁers trained with various mixup methods.
For evaluation, we use the oﬃcial code provided by the TensorFlow-Probability library 2 and set the number of bins as 10.
As shown in Table 10, Co-Mixup classiﬁers have the lowest calibration error on CIFARand Tiny-ImageNet.
As pointed by Guo et al.
, the Vanilla networks have overconﬁdent predictions, but however, we ﬁnd that mixup classiﬁers tend to have under-conﬁdent predictions (Figure 7; Figure 8).
As shown in the ﬁgures, Co-Mixup successfully alleviates the over-conﬁdence issue and does not suﬀer from under-conﬁdence predictions.
[Table]
F. Sensitivity analysis
We measure the Top-1 error rate of the model by sweeping the hyperparameter to show the sensitivity using PreActResNet18 on CIFAR-100 dataset.
We sweep the label smoothness coeﬃcient β ∈ { 0 , 0 .
64 } , compatibility coeﬃcient γ ∈ { 0 . 6 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 4 } , clipping level τ ∈ { 0 . 79 , 0 . 81 , 0 . 83 , 0 . 85 , 0 . 87 } , compatibility matrix parameter ω ∈ { 0 , 5 · 10 − 4 , 10 − 3 , 5 · 10 − 3 , 10 − 2 } , and the size of partition m ∈ { 2 , 4 , 10 , 20 , 50 } .
Table 11 shows that Co-Mixup outperforms the best baseline (PuzzleMix, 20.62%) with a large pool of
[Figure]
A cc u r a c y
Vanilla
[Figure]
Manifold
[Figure]
CutMix
[Figure]
Puzzle
[Figure] [Figure]
A cc u r a c y
Vanilla
[Figure]
Manifold
[Figure]
Puzzle
[Figure]
hyperparameters.
We also ﬁnd that Top-1 error rate increases as the partition batch size m increases until m = 20 .
Smoothness coeﬃcient, β =β
= 0 .β
= 0 .β = 0 .β
= 0 .
64 β 20.
Compatibility coeﬃcient, γ = 0 .γ
= 0 .γ
= 1 .γ = 1 .γ
= 1 .
4 γ 20.
Clipping parameter, τ = 0 .τ
= 0 .τ
= 0 .τ = 0 .τ
= 0 .
87 τ 20.
Compatibility matrix ω =ω = 5 · 10 −ω = 10 −ω = 5 · 10 −ω = 10 −parameter, ω 20.
Partition size, m =m
=m
=m =m
= 50 m 20.
We report the mean values of three diﬀerent random seeds.
F. Comparison with non-mixup baselines
We compare the generalization performance of Co-Mixup with non-mixup baselines, verifying the proposed method achieves the state of the art generalization performance not only for the mixup-based methods but for other general regularization based methods.
One of the regularization methods called VAT (Miyato et al., 2018) uses virtual adversarial loss, which is deﬁned as the KL-divergence of predictions between input data against local perturbation.
We perform the experiment with VAT regularization on CIFAR-100 with PreActResNet18 for 300 epochs in the supervised setting.
We tune α (coeﬃcient of VAT regularization term) in {0.001, 0.01, 0.1}, (cid:15) (radius of (cid:96) -inf ball) in {1, 2}, and the number of noise update steps in {0, 1}.
Table 12 shows that Co-Mixup, which achieves Top-1 error rate of 19.87%, outperforms the VAT regularization method.
G Detailed description for background corruption
We build the background corrupted test datasets based on ImageNet validation dataset to compare the robustness of the pre-trained classiﬁers against the background corruption.
[Table]
ImageNet consists of images { x 1 , ..., x M } , labels { y 1 , ..., y M } , and the corresponding groundtruth bounding boxes { b 1 , ..., b M } .
We use the ground-truth bounding boxes to separate the foreground from the background.
Let z j be a binary mask of image x j , which has valueinside of the ground-truth bounding box b j .
Then, we generate two types of background corrupted sample ˜ x
j by considering the following operations:
Replacement with another image as ˜ x j = x j (cid:12) z
j
+ x i ( j ) (cid:12) (1 − z j ) for a random permutation { i (1) , ..., i ( M ) } .
Adding Gaussian noise as ˜ x j = x j (cid:12)
z j + (cid:15) (cid:12) (1 − z j ) , where (cid:15) ∼ N (0 , 0 .
We clip pixel values of ˜ x
j to [0, 1].
Figure 10 visualizes subsets of the background corruption test datasets.
[Figure]
H Co-Mixup generated samples
In Figure 12, we present Co-Mixup generated image samples by using images from ImageNet.
We use an input batch consisting of 24 images, which is visualized in Figure 11.
As can be seen from Figure 12, Co-Mixup eﬃciently mix-matches salient regions of the given inputs maximizing saliency and creates diverse outputs.
In Figure 12, inputs with the target objects on the left side are mixed with the objects on the right side, and objects on the top side are mixed with the objects on the bottom side.
In Figure 13, we present Co-Mixup generated image samples with larger τ using the same input batch.
By increasing τ , we can encourage
Co-Mixup to use more inputs to mix per each output.