S ALIENCY G RAFTING : I NNOCUOUS A TTRIBUTIONG UIDED M IXUP WITH C ALIBRATED L ABEL M IXING
Anonymous authors
Paper under double-blind review
A BSTRACT
The Mixup scheme of mixing a pair of samples to create an augmented training sample has gained much attention recently for better training of neural networks.
A straightforward and widely used extension is to combine Mixup and regional dropout methods: removing random patches from a sample and replacing it with the features from another sample.
Albeit their simplicity and effectiveness, these methods are prone to create harmful samples due to their randomness.
In recent studies, attempts to prevent such a phenomenon by selecting only the most informative features are gradually emerging.
However, this maximum saliency strategy acts against their fundamental duty of sample diversiﬁcation as they always deterministically select regions with maximum saliency, injecting bias into the augmented data.
To address this problem, we present Saliency Grafting , a novel
Mixup-like data augmentation method that captures the best of both ways.
By stochastically sampling the features and ‘grafting’ them onto another sample, our method effectively generates diverse yet meaningful samples.
The second ingredient of Saliency Grafting is to produce the label of the grafted sample by mixing the labels in a saliency-calibrated fashion, which rectiﬁes supervision misguidance introduced by the random sampling procedure.
Our experiments under CIFAR and
ImageNet datasets show that our scheme outperforms the current state-of-the-art augmentation strategies not only in terms of classiﬁcation accuracy, but is also superior in coping under stress conditions such as data corruption and data scarcity.
The code will be released.
I NTRODUCTION
Modern deep neural networks (DNNs) have achieved unprecedented success in various ﬁelds including computer vision (Krizhevsky et al., 2012), natural language processing (Devlin et al., 2018) and speech processing (Chan et al., 2016).
However, due to their over-parameterized nature, DNNs require an immense amount of training data to generalize well for unseen data.
Otherwise, DNNs are predisposed to memorize the training samples and exhibit lackluster performance on the unseen data - in other words, overﬁt.
Acquiring a sufﬁcient amount of data for a given task is not always possible as it consumes manpower and budget.
One common approach to combat data scarcity is data augmentation, which aims to enlarge the effective size of a dataset by producing virtual samples from the training data by means such as injecting noise (Amodei et al., 2016) or cropping out regions (DeVries & Taylor, 2017).
Datasets fortiﬁed with these augmented samples are shown to effectively improve the generalization performance of the trained model.
Furthermore, data augmentation is proven to be effective not only for promoting generalization but also in boosting the robustness of a model (Hendrycks et al., 2019) and acquiring visual representations without human supervision (Chen et al., 2020).
To this end, conventional augmentation methods focused on creating new images by transforming a given image using means such as ﬂipping, resizing and more.
However, a recently proposed augmentation method called Mixup (Zhang et al., 2017) proposed the idea of crafting a new sample out of a pair of samples by taking a convex combination of them.
Inspired by this pioneering work, Yun et al.
(2019) proposed CutMix, a progeny of Mixup and Cutout (DeVries & Taylor, 2017), which crops a random region of an image and pasting it on another.
These methods are able to generate a wider variety of samples while effectively compensating the information loss by actions such as
[Figure]
cropping.
However, the context-agnostic nature of these methods gives way to creating samples that are potentially harmful.
Since the images are combined randomly without considering their contexts and labels, incorrect augmentation is destined to occur (Figure 1(c)).
For instance, an object can be cropped out and replaced by a different kind of object from another image, or the background part of the image can be pasted on top of an existing object.
Even worse, their labels are naively mixed according only to their mixing proportions, disregarding any information transfer or loss caused by the data mixing.
The harmfulness of semantically unaware label mixing was previously reported in
Guo et al.
This mismatch in data and its supervision signal yields harmful samples.
To address this problem, saliency-guided augmentation methods have been recently proposed (Walawalkar et al., 2020; Kim et al., 2020).
These approaches allegedly refrain from generating harmful samples by preserving the region of maximum saliency based on the saliency maps of the image.
Attentive Cutmix (Walawalkar et al., 2020) preserves the maximum saliency regions of the donor image by locating the k-most salient patches of it and merging them on top of the acceptor image.
PuzzleMix (Kim et al., 2020) tries to salvage the most salient regions of each images by mixing one to another and solving an optimal transport problem and region-wise mixup ratio to maximize the saliency of the created sample.
However, these precautionary measures sacriﬁce sample diversity - the advantage of the previous works.
Unlike CutMix that teaches to attend the whole object by probabilistically choosing diverse regions of the image, maximum saliency methods lose this feature as the most discriminative region is always included in the resulting image.
Moreover, they still overlook making appropriate supervision to describe the augmented image properly, and use semantically inaccurate labels determined by the mixing ratio or the size of the pasted region, which can easily mislead the network (Figure 1(b)).
To solve the drawbacks present in contemporary augmentation methods, we propose Saliency Grafting , a novel data augmentation method that can generate diverse but innocuous augmented data (Figure 1(d)).
Instead of blindingly selecting the maximum saliency region, our method scales and thresholds the saliency map to grant all salient regions equal chance.
The selected regions are then imposed with Bernoulli distribution, and are sampled to generate stochastic patches.
These patches are then ‘grafted’ on top of another image.
To compensate for the side effects of grafting such as label mismatch, we propose a novel label mixing strategy: saliency guided label mixing.
By mixing the labels of the two images according to their saliency instead of their area, potential bad apples are effectively neutralized.
Our contribution is threefold:
• We discuss the potential weaknesses of current Mixup-based augmentation strategies and present a novel data augmentation strategy that is able to generate diverse yet meaningful data through saliency based sampling.
• We present a novel label mixing method to calibrate the generated label to match the information contained in the newly generated data.
• Through extensive experiments, we show that models trained with our method outperforms others - even under data corruption or data scarcity.
R ELATED WORK
Data augmentation Image data augmentation played a formidable role in breakthroughs of deep learning based computer vision (LeCun et al., 1998; Krizhevsky et al., 2012; Simonyan & Zisserman, 2015).
Recently, regional dropout methods such as Cutout (DeVries & Taylor, 2017), Dropblock (Ghiasi et al., 2018) and Random Erasing (Zhong et al., 2020) were proposed to promote generalization by removing selected regions of an image or a feature map to diversify the model’s focus.
However, the removed regions are bound to suffer from information loss.
The recently proposed Mixup (Zhang et al., 2017) and its variants (Verma et al., 2019; Guo et al., 2019), shifted the augmentation paradigm by not only transforming a sample but using a pair of samples to create a new augmented sample via convex combination.
Although successful on multiple domains, Mixup is met with lost opportunities when applied to images as it cannot exploit the spatial locality.
To remedy this issue, Cutmix (Yun et al., 2019), a method combining Cutout and Mixup, was proposed.
By cropping out a region then ﬁlling it with a patch of another image, Cutmix executes regional dropout with less information loss.
However, in Cutmix, a new problem arises as the cut-and-paste strategy incurs semantic information loss and label mismatch.
To ﬁx this issue, methods exploiting maximum saliency regions were proposed.
Attentive Cutmix (Walawalkar et al., 2020) selects the top-k regions to cut and paste to another image, and Puzzlemix (Kim et al., 2020) selects maximum saliency regions of the two images and solves a transportation problem to maximize the saliency of the mixed image.
However, since the maximum saliency region is always pertained, the model is robbed of the opportunities to learn from challenging but beneﬁcial samples present in CutMix.
For text classiﬁcation tasks, Guo (2020) takes a different approach towards this problem by generating mixed data using a nonlinear mixing policy to enlarge the input space.
Also, semantically relevant labels are assigned by separately parametrized labeling function.
Saliency methods In neuroscience literature, Koch & Ullman (1987)
ﬁrst proposed saliency maps as a mean to understand the attention patterns of the human visual cortex.
As contemporary CNNs bear close resemblance to the visual cortex, it is plausible to adapt this tool to observe the inner workings of CNNs.
These saliency techniques inspired by human attention are divided into two groups: bottom-up(backward) and top-down(forward) (Katsuki & Constantinidis, 2014).
For backward methods, saliency is determined in a class-discriminative fashion.
Starting from the output of the network, the saliency signal is back-propagated starting from the label logit and attributed to the regions of the input image.
Simonyan et al.
, Zhou et al.
(2016), Selvaraju et al.
(2017) utilizes the backpropagated gradients to construct saliency maps.
Methods such as Montavon et al.
(2018), Nam et al.
(2020) proposed to backpropagate saliency scores with carefully designed backpropagation rules that preserves the total saliency score across a selected layer.
On the other hand, forward saliency techniques start from the input layer and accumulate the detected signals up the network.
The accumulated signals are then extracted at a higher convolutional layer(often
the last convolutional layer) to obtain a saliency map.
Unlike backward approaches, forward methods are class agnostic as the convolutional layers extract features from all possible objects inside an image to support the last classiﬁer.
These maps are used in a variety of ﬁelds such as classiﬁcation (Oquab et al., 2015) and transfer learning (Zagoruyko & Komodakis, 2017).
P RELIMINARIES
We ﬁrst clarify the notations used throughout the section by describing a general form of Mixupbased augmentation procedures.
Let f θ ( · ) be a Convolutional Neural Network (CNN) parametrized by θ .
For a given batch B of input data { x 1 , . . .
, x m } ∈ X | B | and the corresponding labels [Table] [Table]
[Table] } ∈ Y | B | , a mixed image ˜ x is generated by the augmentation function φ ( · ) and the corresponding label ˜
y is created through the label mixing function ψ ( · ) :
˜ x
= φ ( x i , x j ) and ˜ y
=
ψ ( y i [Table] y j ) for data index i and its random permutation j .
Then, Mixup-based augmentation methods deﬁne their own φ ( · ) as a pixel-wise convex combination of two randomly selected pair, as follows:
[Equation]
where M λ is a mixing matrix controlled by a mixing ratio λ , (cid:12) is the element-wise Hadamard product, and h ( · ) is some pre-processing function.
The vanilla (input) Mixup deﬁnes the augmentation function φ as φ ( x i , x j )
=
λx
i + (1 − λ ) x j .
Manifold Mixup uses similar function φ ( x i , x j )
= λh ( x i ) + (1 − λ ) h ( x j ) but with the latent features.
In CutMix, the augmentation function φ is deﬁned as 1 Rect i (cid:12) x
i + (1 − 1 Rect i ) (cid:12) x j .
This method randomly cuts a rectangular region 1 Rect i from the source image x
i with area proportional to λ and pastes it onto the destination image x j .
PuzzleMix, recent saliency-based Mixup variant, employs the augmentation function φ ( x i , x j ) =
Z ∗ (cid:12) Π ∗ Ti x i + ( 1 − Z ∗ ) (cid:12) Π ∗
Tj x j .
This method exploits the image transportation plan Π and region-wise mask matrix Z to maximize the saliency of the mixed image.
Note that unlike the vanilla Mixup, Z is discretized region-wise mixing matrix that satisﬁes λ = 1 n (cid:80) s (cid:80) t
Z st for given mixing rate λ .
To ﬁnd the optimal transportation plan Π ∗ and region-wise mask Z ∗ for the maximum saliency, PuzzleMix solves additional optimization problems in an alternating fashion, per each iteration.
Although it is a simpler scalar function, the label function ψ ( · ) is also deﬁned in a similar form to the augmentation function φ ( · ) :
[Equation]
where ρ is a label mixing coefﬁcient determined by the sample pair ( x i , y i ) , ( x j , y j ) and the mixing ratio λ from φ .
However, in all methods mentioned above, this ρ simply depends on λ , disregarding the contents of sample pair x i and x j : ˜
y = λy
i + (1 − λ )
y j .
Table 1 summarizes φ and ψ for the augmentation methods described above.
S ALIENCY G RAFTING
We now describe our simple approach, Saliency Grafting , that creates diverse and innocuous Mixup augmentation based on the content of instances being merged.
Two key innovations in Saliency
Grafting are stochastic patch selection (Section 4.1) and label mixing (Section 4.2), both of which utilize the saliency information at the core.
Last but not least, another important element of Saliency
Grafting is choosing a saliency map generation method (Section 4.3) for above two main components while keeping the learning cost to a minimum.
The overall procedure is described in Figure 2.
Now we discuss the details of each component in the subsequent subsections.
4. S TOCHASTIC PATCH SELECTION WITH SALIENCY INFORMATION
The stochastic patch selection of Saliency Grafting aims to choose regions that can create diverse and meaningful instances.
The key question here is how to select regions to be grafted, given a saliency matrix S i for the source image x
i (whose element S st indicates the saliency for a region ( s, t ) of x i ).
As in recent studies (Kim et al., 2020; Walawalkar et al., 2020), if only regions with high intensity of S st are always selected, then these regions - which are already easy to judge by the modelare continuously augmented in the iterative training procedure.
As a result, the model is repeatedly exposed to the same grafting patch, which would iteratively amplify the model’s attention on the selected regions and deprive the opportunity to learn how to attend to other parts and structures of the object.
In order to eliminate this selection bias , the patch selection of Saliency Grafting consists of two steps: i) softmax thresholding and ii) stochastic sampling.
Softmax thresholding To neutralize the selection bias due to the intensity of saliency, we normalize the saliency map by applying the softmax function and then binarize the map with some threshold σ : [Equation]
[Equation]
given the temperature hyperparameter T to control the sharpness of the normalized saliency map.
Here, threshold σ has a variety of options, but we use σ = (cid:80) Hh (cid:80) Ww S (cid:48)
hw , using the mean value of the normalized saliency map.
Stochastic sampling Although the selection bias is signiﬁcantly mitigated by thresholding, the high intensity regions are never removed , as the softmax function preserves the order of the regions.
To address this issue, we stochastically sample the grafting regions based on the binarized saliency map produced above.
The ﬁnal mixing matrix is constructed by taking the Hadamard product of S (cid:48)(cid:48)
i and a region-wise i.i.d.
random Bernoulli matrix of same dimensions P ∼ Bern ( p B ) : M i = P (cid:12)
S (cid:48)(cid:48) i .
Here, the batch-wise sampling probability p B is drawn from a Beta distribution p ∼ Beta ( α, α ) .
The ﬁnal augmentation function φ for Saliency Grafting is M i (cid:12)
x i + (1 − M i ) (cid:12) x j .
4. C ALIBRATED LABEL MIXING BASED ON SALIENCY MAPS
In addition to the method of grafting diverse and innocuous augmentations described in previous section, attaching an appropriate label for supervision to the generated data is also the core of Saliency
Grafting .
Although extreme, to highlight the drawbacks of the existing label mixing strategy used in all baselines, suppose that source image x
i is combined with destination image x j , both of which have saliency concentrated in some small regions.
Suppose further that this region of x i is selected and grafted to the region where the original class of destination x j is concentrated.
Then, most of the information of class y i is retained while most of the information on class y j is lost.
However, the label is determined in proportion to the mixing rate or the size of the area used, as all the baselines do, the generated label will be close to class y j since most areas of it originally came from destination image x j .
To tackle this issue, we propose a novel label mixing procedure that can adaptively mix the labels again based on saliency maps.
Regarding on the destination image x j receiving the graft, the ground truth label y j is penalized according to the degree of occlusion.
Speciﬁcally, the importance of the destination image I ( S j , 1 − M i ) 1 given the mixing matrix M i is calibrated using the saliency values of the remaining part not occluded by the source image, I ( S j , 1 − M i ) =
[Equation]
On the other hand, with regard to the source image x
i , the corresponding label y i is compensated in proportion to the importance of the selected region: I ( S i , M i
[Equation] = [Equation]
The ﬁnal label mixing rate is computed based on the relative importance of x i and x j , so that their coefﬁcients sum to 1 to deﬁne the calibrated label mixing function ψ ( y i , y j )
=
λ ( S i , S j , M i )
y i + (cid:0) 1 − λ
( S i , S j , M i ) (cid:1)
y j where λ ( S i , S j , M i ) =
I ( S i ,M i ) I ( S i ,M i )
+ I ( S j , 1 − M i ) .
4. S ALIENCY MAP GENERATION
Technically, Saliency Grafting can be combined with various saliency generation methods without the dependence on a speciﬁc method.
However, the caveat here is that the performance of Saliency
Grafting is, by design, highly affected by the quality of saliency map, or how accurately the saliency map corresponds to the ground truth label.
From this point of view, the forward saliency methods, which incur less false negatives, may support Salient Grafting more stably than the backward methods (see Section 2 for forward and backward saliency methods).
This is because the backward methods are likely to break down and exclude true salient regions when the model fails to predict the true label, whereas the forward methods preserve all the feature maps inside the saliency map, i.e., they act like a class-agnostic saliency detector (Mahendran & Vedaldi, 2016).
In an environment where there is no separate pre-trained model, another advantage of using forward saliency is gained: Saliency maps can be naturally constructed based on the terms already calculated in the learning process.
In this environment, since the generated maps can be noisy in the early phases of training, we employ warmup epochs without no data augmentation.
We now describe the speciﬁc choice of generating the saliency maps to guide our augmentation process.
We adopt the channel-collapsed absolute feature map of the network as our saliency map, mainly due to its simplicity: S ( l ) =
(cid:80) Cc =1 (cid:12)(cid:12)
A ( l ) c (cid:12)(cid:12) where A ∈ R C × H × W is the feature map at the l -th layer.
Although it is possible to extract saliency maps from any designated layer in the network, we extract the maps from the last convolutional layer of the model as it generally conveys the high-level spatial information (Bengio et al., 2013).
E XPERIMENTS
In this section, we conduct a collection of experiments to test Saliency Grafting against other popular augmentation methods.
First, we test standard prediction performance on the standard image classiﬁcation datasets.
We also conduct multiple stress tests to measure the enhancement in generalization capability.
Finally, we conduct an ablation study to investigate the contributions of each sub-components of Saliency Grafting .
5. C LASSIFICATION TASKS
CIFAR-100 We evaluate our method Saliency Grafting on CIFAR-100 dataset (Krizhevsky et al., 2009) using two neural networks: PyramidNet-200 with widening factor ˜ α
= 240 (Han et al., 2017) and WRN28-10 (Zagoruyko & Komodakis, 2016).
For the PyramidNet-200, we follow the experimental setting of Yun et al.
(2019), which trains PyramidNet-200 for 300 epochs.
The baselines [Table] results on PyramidNet-200 are as reported in Yun et al.
For WRN28-10, the network is trained for 400 epochs as reported in following studies (Kim et al., 2020; Verma et al., 2019).
In this experiment, we reproduce other augmentation baselines (Zhang et al., 2017; Hendrycks et al., 2019;
Yun et al., 2019; Kim et al., 2020) following the original setting of each paper.
Detailed hyperparameter settings are provided in Appendix B.1.
As shown in Table
[Table]
Saliency Grafting exhibits a signiﬁcant improvement with PyramidNet-200 architecture compared to other augmentation baselines where Saliency Grafting achieves 13.59% Top1 error
[Table]
and even outperforms the best performance obtained by combining the two regularization methods.
Furthermore, when used together with Shakedrop, Saliency Grafting achieves additional enhancement - 13.00% Top-1 error.
In Appendix B.1, our method also shows best generalization performance - 15.24% Top-1 error with WRN28-10.
(Table 10)
[Table]
ImageNet We evaluate on large-scale dataset
ImageNet (Russakovsky et al., 2015) using
ResNet-50 (He et al., 2016).
The network is trained for 100 epochs.
For a fair comparison, we follow the training protocol in Kim et al.
; Wong et al.
Detailed experiment settings are in Appendix B.2.
As shown in
Table 3, Saliency Grafting achieves again stateof-the-art performance in both Top-1/Top-5 error rates, +0.37% higher than the best Top-1 error rate of baselines.
Additional experiments Due to the space constraint, two additional experiments are deferred to
Appendix A.
The ﬁrst experiment shows that Saliency Grafting is useful for speech dataset beyond the image classiﬁcation task, and the second experiment implies that the ﬁnal model learned through
Saliency Grafting contains more useful saliency information.
5. S TRESS TESTING
Data scarcity The situation where data augmentation is most required is when data is scarce.
In this condition, it is important to improve the generalization performance by increasing the data volume while preventing overﬁtting.
To this end, we reduce the number of data per class to 50%, 20%, and 10%, respectively, with WRN28-10 model on the CIFAR-100 dataset.
In Table 4, Saliency
Grafting exhibits best performance in every condition from 50% to 10%.
Note that the performance of CutMix deteriorates as the number of data per class decreases due to their randomness occurring label mismatching.
This is in line with the fact that, as investigated by Rolnick et al.
(2017), a number of data are required for performance in conditions where label corruption exists.
Our method [Table] [Table] maintains the diversity of CutMix to prevent overﬁtting, while exploiting adaptive label mixing to reduce the mismatch between data and labels, improving the generalization performance.
Robustness against data corruption Another important stress condition is data corruption.
Commercially deployed DNNs are often met with data corrupted with noise, which may be indecipherable for models trained under immaculate data.
To this end [Table] Hendrycks et al.
(2019) showed that carefully crafted data augmentation can be utilized as a mean to temper the model to gain noise robustness.
Although Saliency Grafting is not speciﬁcally built to counter data corruption
[Table]
we found that our method is able to promote noise robustness by grafting the one AugMix image into another
AugMix image.
Experiments on CIFAR100-C (Hendrycks & Dietterich, 2019) show that Saliency
Grafting was able to outperform other contenders (Table 5) while being computationally efﬁcient, requiring a single additional forward pass, compared to Hendrycks et al.
(2019) requiring 2 additional forward passes and Kim et al.
(2020) requiring one additional backward pass and solving a separate optimization problem.
5. A BLATION STUDY
Stochastic selection VS deterministic selection In Section 4.1, we argued that the deterministic patch selection process of existing methods (Walawalkar et al., 2020; Kim et al., 2020) leads to performance degradation.
Here, we measure the classiﬁcation accuracy on CIFAR-100 with
PyramidNet-200 where the deterministic top- k selection of Walawalkar et al.
(2020) is replaced by our stochastic selection.
For fair comparison, the softmax temperature T is adjusted to satisfy E i [ (cid:80) s
(cid:80) t M i,st ] = k .
Results show that stochastic selection indeed outperforms deterministic selection (Table 6).
The training and validation loss curves (Appendix C) show that stochastic selection resists overﬁtting, showing training curves similar to that of Dropout (Srivastava et al., 2014)
(higher training loss, lower validation loss).
Label mixing strategies In Section 4.2, we discussed the pitfalls of naive area-based label mixing and proposed saliency-based label mixing as a solution.
Here, we compare the two strategies.
We experiment on CIFAR-100 with PyramidNet-200 and replace the mixing strategy of Saliency Grafting with area-based mixing.
Results in Table 6 conﬁrms that saliency-based mixing outperforms area-based mixing.
Forward saliency VS Backward saliency To support our choice of forward saliency maps (Section 4.3), we conduct an additional experiment on CIFAR-100 with WRN28-10 where the forward saliency map of Saliency Grafting is replaced by CAM(Zhou et al., 2016), a backward saliency map.
The detailed settings are kept identical to Section 5.1.
Results show that the classiﬁcation error increases when a backward saliency map is used (Table 7). Table
[Table] [Table]
C ONCLUSION
We have presented Saliency Grafting , a data augmentation method that generates diverse saliencyguided samples via stochastic sampling and neutralizing any induced data-label mismatch with saliency-based label mixing.
Through extensive experiments, we have shown that models equipped with Saliency Grafting outperforms existing mixup-based data augmentation techniques under both normal and extreme conditions while using less computational resources.
A A DDITIONAL EXPERIMENTS
Speech data To test our method on data outside the distribution of natural images, we use the
Google Speech Commands dataset (Warden, 2017).
The training samples were ﬁrst augmented in the time domain by applying random changes in amplitude, speed and pitch and in the frequency domain by stretching and time-shifting the spectrogram.
Then, random background noises clip drawn from the noise compilation in the dataset were added to the samples.
Finally, the samples are transformed into 32x32 mel-spectrograms by using 32 MFCC ﬁlters.
To evaluate our method on this data, we used the WRN28-10 architecture.
As in Table 8, our method was able to outperform other methods in a non-natural image domain.
[Table]
Weakly supervised object localization To examine how our method affects the backward saliency of a model (how a model ’thinks’), we measure the weakly supervised object localization performance on the CUB200-2011 dataset (Wah et al., 2011).
For ResNet-50, we slightly modify the last convolution layer to make featuremap size from 7x7 to 14x14.
We ﬁrst obtain the backward saliency map with CAM (Zhou et al., 2016).
The map is then thresholded using a 15% of the maximum value of CAM and enclosed by the smallest possible bounding box.
We measure the
Intersection-over-Union(IoU) between this estimated bounding box and the ground truth bounding box.
For localization accuracy, IoU between the estimated bounding box and ground truth box is greater than 0.5, and, simultaneously, the predicted class label should be correct.
We used Adam optimizer, and the initial learning rate, weight decay, batch size were 0.001,0.0001, and 32.
The learning rate is decaying by the factor of 0.1 per 150 epochs.
All the experiments was performed three times and the averaged best error rates are reported.
[Table]
Sample diversity To further verify our claim, we conducted another intuitive experiment to compare Saliency Grafting and PuzzleMix in terms of sample diversity.
In this experiment, for every iteration, each method trains the network by generating additional augmented data k times from the mini-batch; each method produces k independent augmented instances with its randomness.
In order to ensure sufﬁcient diversity, the mixing ratio λ is also newly sampled for each augmented data.
While varying k from 1 to 6, we evaluated whether each method can obtain the performance gain due to sample diversity.
We followed Puzzlemix’s WRN28-10 training setting for 200 epochs, and use 20% of the Cifar-100 dataset to better conﬁrm the diversity effect of the augmented data.
The average error rate for 5 random seeds are reported.
As shown in Figure 3, the performance of Saliency
Grafting consistently improves as k increases, whereas PuzzleMix is predisposed to maintain somewhat constant performance even when k increases.
In this sense, we believe that this is the direct evidence that generating PuzzleMix’s samples by sampling the random mixing ratio is insufﬁcient to ensure sample diversity.
However, since Saliency Grafting exploits temperature-scaled thresholding with stochastic sampling, the model easily attends the entire object as k increases.
Also, it is possible to properly supervise the augmented data through calibrated label mixing, sample diversity can be guaranteed innocuity.
[Figure]
[Figure] [Figure]
[Figure]
PuzzleMix Saliency Grafting
Sensitivity to temperature T
The threshold value of our method is determined by the expectation of the temperature - scaled saliency map.
Note that the number of saliency regions greater than the expectation depends on the temperature T .
As T decreases, the softmax distribution becomes sharper and the number of saliency regions above the expectation decreases.
That is, the mixing regions are selected from a smaller range.
On the other hand, as T increases, the distribution ﬂattens so that nearly half the numbers are above the threshold.
To see the sensitivity of model performance with respect to the softmax temperature, we conducted an additional experiment on the CIFARdataset with ResNet-18 by increasing the temperature from 0.01 to 0.30 (Figure 4).
If we set a very small T , such as 0.01, only a small number of regions are mixed, resulting in a relatively small performance improvement.
As we raise the temperature, the number of participating regions increases, resulting in a major increase in performance.
When the temperature is sufﬁciently high, enough number of regions can participate in the mix.
Thus, further increasing the temperature plateaus the performance.
[Figure]
T o pE r
r o r
T e m p
e
r a t u r e
T o p - 1 E r r
B D ETAILED EXPERIMENTAL SETTINGS B. CIFAR-100 C LASSIFICATION
We use stochastic gradient descent (SGD) with momentum 0.9 for both network models.
Mixing ratio λ is sampled from Beta (1,1) with regard to Mixup and CutMix.
In the Manifold Mixup case, we adopt Beta (2,2) for sampling distribution to follow the original paper.
PuzzleMix has four hyperparameters: label smoothness term β , data smoothness term γ , prior term η , and transport cost ξ .
We use ( β, γ, η, ξ ) =
For the classiﬁcation task, our method use a temperature T 0.
and Beta (2,2) for stochastic sampling.
For early convergence, we warmup the model for 5 epochs.
The weight decay of each augmentation method is different with CIFAR dataset, so the results of our paper are reported as having better results among 0.0005 and 0.0001.
For PyramidNet-200 network, the initial learning rate was set to 0.25 and decayed by the factor of 0.1 at 150 and 225 epoch.
For WRN28-10 network, the initial learning rate was set to 0.1 and decayed by the factor of 0.
per 75 epochs.
All the experiments was performed three times and the averaged best error rates are reported.
B. I MAGE N ET C LASSIFICATION
For ImageNet, we follow the training process in Kim et al.
We trained the ResNet-50 model with an image resized to 224 by 224.
Our proposed method does not require the tricky tuning of the learning rate decay, but we followed the cyclic learning rate decay setting for a fair comparison.
Our method adopts temperature T for 0.2 and sampling α for 2.
We use SGD optimizer, and the initial learning rate, momentum, weight deacy, and batch size were 0.1, 0.9, 0.0001, 256.
For ImageNet dataset, we warmup the model only for 1 epoch.
C A DDITIONAL TABLES AND FIGURES
WRN
[Figure] [Figure]
[Figure]
T r a in lo s s
[Figure]
r m
[Figure] [Figure]