RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness
Abstract
We show that the effectiveness of the well celebrated Mixup [Zhang et al., 2018] can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss.
This simple change not only improves accuracy but also signiﬁcantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and out-of-distribution detection experiments.
In fact, we observe that Mixup otherwise yields much degraded performance on detecting out-of-distribution samples possibly, as we show empirically, due to its tendency to learn models exhibiting high-entropy throughout; making it difﬁcult to differentiate in-distribution samples from out-of-distribution ones.
To show the efﬁcacy of our approach (RegMixup 2 ), we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-10/100) and compare it with a suite of recent approaches for reliable uncertainty estimation.
Introduction
In real-world machine learning applications one is interested in obtaining models that can reliably process novel inputs.
However, though deep learning models have enabled breakthroughs in multiple ﬁelds, they are known to be unreliable when exposed to samples obtained from a distribution that is different from the training distribution.
Usually, the larger the extent of this difference between the train and the test distributions, the more unreliable these models are.
This observation has led to a growing interest in developing deep learning models that can provide reliable predictions even when exposed to unseen situations [Liu et al., 2020a,b, Wen et al., 2021, Lakshminarayanan et al., 2017].
Most of these approaches either use expensive ensembles, or propose non-trivial modiﬁcations to the neural network architectures in order to obtain reliable models.
In most cases, they trade in-distribution performance (accuracy) to perform reliably on: (1) out-of-distribution (OOD) samples; and (2) covariate shift (CS)
[Quionero-Candela et al., 2009] samples.
Towards developing practically useful and reliable models, we investigate the well known
Mixup [Zhang et al., 2018] as it is extremely popular in improving both a model’s accuracy and its robustness [Wen et al., 2021, Hendrycks et al., 2020b].
It has already been observed that Mixup can help in retaining good accuracy when the test inputs are affected by superﬁcial variations that do not affect the target label (i.e., they undergo CS)
[Hendrycks et al., 2020b].
However, we ﬁnd that Mixup’s reliability degrades signiﬁcantly when exposed to completely unseen samples with potentially different labels than the ones it was shown during training (OOD).
This is undesirable as in such situations we would like our models to reliably reject these inputs instead of making wrong predictions on them.
We observe that the primary reason for such poor OOD performance of Mixup is due to its tendency to provide high predictive entropy for almost all the samples it receives.
Therefore, it becomes difﬁcult for the model to differentiate in-distribution samples from out-of-distribution ones.
We would like to highlight that our observation is in contrast to the prior work [Thulasidasan et al., 2019] which suggests that Mixup provides reliable uncertainty estimates for OOD data as well.
We propose a simple yet effective ﬁx to the aforementioned issue with Mixup: we suggest to train the model on an approximate data-distribution that is an explicit mixture of both the empirical and the Mixup vicinal approximations to the data-distribution.
We call this approach RegMixup.
In a nutshell, it simply combines the Empirical Risk Minimization (ERM)
[Vapnik, 1991] and the Vicinal Risk
Minimization (VRM)
[Chapelle et al., 2000] objectives together.
Implementation wise, along with the Mixup objective on the interpolated sample, it requires adding an additional cross-entropy loss on one of the the clean samples.
We provide proper justiﬁcations behind this proposal and show that such simple modiﬁcation signiﬁcantly improves the performance of Mixup on a variety of experiments.
We would like to highlight that one of the core strengths of our approach is its simplicity .
As opposed to the recently proposed techniques to improve uncertainty estimation like SNGP [Liu et al., 2020a] and DUQ [van Amersfoort et al., 2020], it does not require any modiﬁcations to the architecture and is extremely simple to implement.
It also does not trade accuracy in order to improve uncertainty estimates , and is a single deterministic model, hence, extremely efﬁcient compared to the highly competitive Deep Ensembles (DE)
[Lakshminarayanan et al., 2017].
Summary of our contributions:
• We provide a simple modiﬁcation to Mixup that signiﬁcantly improves its in-distribution, covariate shift, and out-of-distribution performances.
• Through extensive experiments using ImageNet-1K, CIFAR10/100 and their various CS counterparts along with multiple OOD datasets we show that, overall, RegMixup outperforms recent state-of-the-art single-model approaches.
In most cases, it outperforms the extremely competitive and expensive DE as well.
RegMixup: Mixup as a regularizer
Preliminary on ERM and VRM
The principle of risk minimization [Vapnik, 1991] is to estimate a function f ∈ F that, for a given loss function (cid:96) ( ., . ) , minimizes the expected risk over the datadistribution P ( x , y ) .
The risk to be optimized is deﬁned as R ( f ) =
(cid:82) (cid:96) ( f ( x ) , y ) dP ( x , y ) .
Since the distribution P ( x , y ) is unknown, a crude yet widely used approximation is to ﬁrst obtain a training dataset D
= { ( x i , y i ) } ni =1 sampled from the distribution P and then obtain f by minimizing the empirical risk deﬁned as R e ( f ) = 1 /n (cid:80) ni =1 (cid:96) ( f ( x i ) , y i ) .
This is equivalent to approximating the entire data-distribution space by a ﬁnite n number of Delta distributions positioned at each ( x i , y i ) , written as P e ( x , y )
= 1 /n (cid:80) ni
=
δ x i ( x ) δ y i ( y ) .
This approximation to the risk minimization objective is widely known as the Empirical Risk Minimization (ERM) [Vapnik, 1991].
ERM has been successfully used in a plenty of real-world applications and undoubtedly has provided efﬁcient and accurate solutions to many learning problems.
However, it is straightforward to notice that the quality of such ERM solutions would rely on how closely P e mimics the true distribution P , and also on the capacity of the function class F .
In situations where the function class is extremely rich with high capacity (for example, deep neural networks), and hence prone to undesirable behaviours such as overﬁtting and memorization, a good approximation to P is generally needed to enforce suitable inductive biases in the model.
To this end, for a ﬁxed training dataset, instead of a delta distribution one could potentially ﬁt a richer distribution in the vicinity of each input-output pair to estimate a more informed risk computed in a region around each sample.
This is precisely the principle behind Vicinal Risk Minimization (VRM)
[Chapelle et al., 2000].
The approximate distribution in this case can be written as P v ( x , y )
=
1 /n (cid:80) ni
=1 P x i , y i ( x , y ) , where P x i , y i ( x , y ) denotes the user-deﬁned vicinal distribution around the i -th sample 3 .
Therefore, the vicinal risk boils down to
[Equation]
When the integral in the above summation is intractable, a Monte Carlo estimate with m samples can be used as follows:
[Equation]
Several approaches in deep learning can be seen as a special instance of VRM.
For example, training a neural network with multiple augmentations is a special case where the augmented inputs are the samples from the unknown vicinal distribution.
A widely used application of VRM is the procedure to obtain a robust base classiﬁer to design certiﬁable classiﬁers.
For example, if P x i , y i ( x , y ) =
P x i ( x ) δ y i ( y ) and P x i is a Gaussian distribution centered at x i , Eq.
(2) can be computed by taking the average loss over gaussian perturbed inputs ¯ x while keeping the target labels same.
Minimizing such a risk would lead to a classiﬁer that is robust to additive noise bounded within an (cid:96) 2 ball.
This is exactly the procedure that has been widely adopted in the randomized smoothing literature in order to obtain a base neural network for which a certiﬁable smooth classiﬁer can be obtained [Lecuyer et al., 2019, Cohen et al., 2019] 4 .
Below we discuss another highly effective use case of VRM called Mixup which is the main focus of this work.
Mixup Built on the fundamentals of VRM, the vicinal distribution deﬁned in Mixup [Zhang et al., 2018] is as follows:
[Equation]
where λ ∼ Beta ( α, α ) ∈ [0 , 1] , α ∈ (0 , ∞ ) , ¯ x i =
λ x i + (1 − λ ) x j and ¯ y i
=
λ
y i + (1 − λ )
y j .
Note that the vicinal distribution here not only depends on ( x i , y i ) but also on another input-output pair ( x j , y j ) drawn from the same training dataset.
For a ﬁxed α (parameter of the Beta distribution, refer Figure 1), implementing Mixup would require taking multiple Monte Carlo samples 5 for each datapoint (refer Eq.
(2)) which can be computationally prohibitive.
Therefore, in practice, only one sample ( m = 1 ) per Beta distribution per pair of samples from a batch is considered at a time.
Although this procedure might look like a crude approximation to the original objective, it has resulted in highly promising results in a variety of applications and is very well accepted in the research community.
Without undermining the remarkable success of such a simple approach, below we highlight two of its behavioural characteristics that, as we show, might limit its effectiveness:
• Small cross-validated α (cid:28) 1 : The shape of the vicinal distribution depends on the hyperparameter of the Beta distribution, therefore, the values of α decides the strength of the convex interpolation factor λ .
Since high values of α would encourage λ ≈ 0 .
5 resulting in an interpolated ¯ x that is perceptually different from x ( inducing a mismatch between train and test distributions ), the cross-validated value of α for Mixup in most cases turns out to be very small ( α ≈ 0 . 2 ) in order to provide good generalization.
Such small values of α leads to sharp peaks at 0 and 1 (refer Figure 1).
Therefore, effectively, Mixup ends up slightly perturbing a clean sample in the direction of another sample even though the vicinal distribution has the potential to explore a much larger interpolation space.
• High-entropy behaviour: As mentioned earlier, m = 1 is used in practice, therefore, it is very unlikely that the interpolation factor λ ∈ { 0 , 1 } even for small values of α .
Thus, the model never gets exposed to uninterpolated clean samples during training and hence, it always learns to predict interpolated (or smoothed) labels ¯ y for every input.
Just like DNNs with cross-entropy loss are overconﬁdent because of their high capacity and target samples λ Delta distribution [Guo et al., 2017], DNNs with Mixup turns out to be relatively less conﬁdent because the network retains its high capacity but observes only target smoothed labels.
This underconﬁdent behaviour results in high-entropy for both in-distribution and out-of-distribution samples.
This is undesirable as it will not allow predictive uncertainty to reliably differentiate in-distribution samples from out-of-distribution ones.
We validate the consequences of the above limitations of Mixup with the following simple experiment.
In Figure 3, we provide heat-maps to show how the entropy of the predictive distribution varies when interpolating samples belonging to different classes.
The heat-map is created as follows.
First, we train a WideResNet28-10 (WRN) [Zagoruyko and Komodakis, 2016] using the CIFAR-10 (C10) dataset.
Then, we randomly choose 1K pairs of samples { x i , x j } from the dataset such that y i (cid:54) =
y j 6 .
For each pair, via convex combination, we synthesize 20 samples ¯ x s using equally spaced λ s between 0 to 1 .
The heat-map is then created using all the 20K samples.
The intensity of each ( λ , H) bin in the heat-map indicates the number of samples in that bin.
Note, DNN (trained with vanilla cross-entropy loss) shows low entropy (thus yielding overconﬁdence) irrespective of where the interpolated sample lies.
However, Mixup shows higher entropy for almost all λ s (thus yielding lower conﬁdence on indistribution samples as well).
The impact of this behaviour is shown in Table 1.
Though Mixup provides improved accuracy compared to DNN for in-distribution (IND) and covariate shift experiments, the high entropy behaviour makes it much worse than DNN when exposed to out-ofdistribution detection task.
For example, when SVHN is used as the OOD dataset, the performance of Mixup drops by nearly 8 .
47% compared to DNN.
This clearly shows that the distributions of the per-sample entropies on the in-distribution and out-of-distribution sets are harder to separate.
However, there is a clear improvement of nearly 5% for covariate shift experiments, implying that Mixup augmentations do improve robustness in this aspect.
Note that, in the context of calibration, Mixup’s conﬁdence reducing behaviour (equivalent to providing higher predictive entropy) was also observed by [Wen et al., 2021].
RegMixup We now provide a very simple modiﬁcation to Mixup that not only alleviates its aforementioned limitation on detecting OOD samples, but also signiﬁcantly improves its performance on in-distribution and covariate-shift samples.
We propose to use the following approximation to the data-distribution
[Equation]
where P x i , y i ( x , y ) is the mixup vicinal distribution and γ ∈ [0 , 1] is the mixture weight.
The above approximation is simply an explicit assemble of ERM and VRM based approximations to data-distribution.
Though, in theory, VRM subsumes ERM, we argue that because of the crude approximations needed to prevent increasing the training cost, it does not fully realize its potential.
Therefore, explicitly combining them might result in a practically more expressive model.
We provide extensive experimental evidence to support this hypothesis.
Implementation wise, for each sample x i in a batch, another sample x j is drawn randomly from the same batch to obtain interpolated ¯ x i and ¯ y i , and then the following loss is minimized
[Equation]
where CE(.,.) denotes the standard cross-entropy loss, the hyperparameter η ∈ R ≥ 0 , and p ( . )
the softmax output of a neural network parameterized by θ .
Note, dividing Eq.
(3) by (1 + η ) is exactly the same as using a data-distribution with γ = 1 / 1+ η .
In practice, we observe the model’s performance to not vary much with η and using η = 1 (equivalent to γ = 0 . 5 ) to provide highly effective results.
Refer to Algorithm 1 for an overview of the RegMixup training procedure.
What does this simple modiﬁcation bring to the model?
(1) VRM now subsumes ERM because the vicinal distribution can perfectly represent the ERM training distribution.
This is because of the fact that irrespective of the values of m and α , the model will always be exposed to the clean training samples as well.
The interpolation factor λ can potentially explore a much wider space as the presence of clean samples might help in controlling the performance drop due to the train/test distribution shift.
Therefore, if α (cid:28) 1 was actually the most effective solution, the cross validation would automatically ﬁnd it.
Practical implications of such simple modiﬁcation on the behaviour and the performance of the model
• Large cross-validated α (cid:29) 1 : As anticipated, the model is now able to explore strong interpolations because of the additional cross-entropy term over the unperturbed training data.
Interestingly, the cross-validated α that we obtained in fact is very high ( α ∈ { 10 , 20 } , see Appendix A.3 for cross-validation details) leading to λ ≈ 0 .
Therefore, as opposed to the standard Mixup, RegMixup prefers having strong diverse interpolations during training.
Refer Figure 2 for visualizations.
• Maximizing a soft proxy to entropy (knowingly unknowing the unknown): It is straightforward to notice that a value of λ ≈ 0 . 5 would lead to ¯ x that is a heavy mix of two samples (mimicking OOD samples, refer Fig. 2), and the corresponding target label vector ¯ y would have almost equal masses corresponding to the labels of the interpolating samples.
Therefore, minimizing CE ( p θ ( y | ¯ x i ) , ¯ y i ) in this case is equivalent to maximizing a soft proxy to entropy deﬁned over the label support of y i and
y j
(note, exact entropy maximization would encourage equal probability masses of 0.5 for these labels).
We ﬁnd this observation intriguing as RegMixup naturally obtains a cross-validated α that leads to a maximum likelihood solution having high-entropy on heavily interpolated samples.
This is an extremely desirable property as it allows models to reliably differentiate between in and out distribution samples.
Thus, Mixup automatically acts as a regularizer in this case.
[Figure]
[Table]
The entropy heat-map in Figureclearly shows that as opposed to DNN and Mixup, the entropy for RegMixup is very low for λ close to either 0 or 1 , however, it increases and remains high for all other intermediate interpolation factors, practically creating an entropy barrier .
Also, as shown in Table 1, RegMixup provides improvements in both accuracy (in-distribution and covariate-shift) and out-of-distribution robustness.
An interesting observation is that Mixup’s perfor- mance on OOD (T-ImageNet) dropped by 2.4% compared to DNN whereas RegMixup performed 3.75% better than DNN.
Thus, in this case, it effectively improved the performance of Mixup by nearly 6.15%.
Related Works
Covariate-Shift Robustness
It is well known that neural networks performance can severely degrade when exposed to covariate shift [Taori et al., 2016, Recht et al., 2019, Wang et al., 2019].
In image classiﬁcation, for instance, covariate shift could occur due to changes in environmental conditions [Hendrycks and Dietterich, 2019] and the image capturing process, data sampling [Recht et al., 2019, 2018, Lu et al.]
, stylised representations of the objects (e.g. in paintings or drawings)
[Wang et al., 2019, Hendrycks et al., 2021] etc.
In the ﬁeld of out-of-distribution generalization, many techniques assume the availability of sets of samples presenting such variations at training time [Shi et al., 2021, Yao et al., 2022, Wang et al., 2021], other techniques suggest leveraging augmentations to mimic a set of variations that would allow the model to learn features that generalize better [Hendrycks et al., 2020b,a] or self-supervision
[Hendrycks et al., 2019a, Shen et al., 2022].
Out-Of-Distribution Detection While some authors suggest introducing an OOD reject option [Geifman and El-Yaniv, 2019], the most recent literature tends to focus on leveraging a threshold classiﬁer on an uncertainty measure computed on top of the predictive distribution of a network.
For the latter approach, it is therefore convenient for a model to produce high uncertainty predictions when facing OOD samples, and low uncertainty predictions when fed in-distribution samples [Liu et al., 2020a].
Within this family of approaches, the most popular ones are the expensive
Deep
Ensembles (DE) [Lakshminarayanan et al., 2017].
Note, the computational cost of DE scales linearly with the number of members, and also recent works have observed a few limitations of naively using ensembles directly [Wen et al., 2021, Ashukha et al., 2020].
There are several efﬁcient variants of DE where the ensemble members are obtained either by processing multiple inputs and multiple outputs at the same time [Havasi et al., 2021], adding member-speciﬁc components to the network [Wen et al., 2020], or by ﬁnding optima in distinct basins of low loss [Huang et al., 2017a].
Signiﬁcant attention has also been paid to Bayesian techniques where either stochastic gradient descent is being modiﬁed to perform approximate inference [Chen et al., 2014, Zhang et al., 2020, Durmus et al., 2016] or an approximation to Gaussian Processes is being utilized (speciﬁcally, we focus on SNGP
[Liu et al., 2020a]), or Bayesian Logistic Regression is being used during inference via Laplace approximation (speciﬁcally, we focus on KFAC-LLLA [Kristiadi et al., 2020]).
Techniques utilizing distance functions in the feature space have also been suggested [van Amersfoort et al., 2020].
Model calibration Modern NNs have been shown to be miscalibrated [Guo et al., 2017], i.e. they exhibit a mismatch between the model’s conﬁdence and its accuracy.
It has been observed that temperature scaling [Guo et al., 2017] or replacing the typical cross-entropy loss [Chung et al., 2021,
Thulasidasan et al., 2019, Mukhoti et al., 2020] can be highly effective to reduce this mismatch.
Also, ensemble learning has been observed to help in reducing miscalibration [Lakshminarayanan et al., 2017, Wen et al., 2021, Rahaman and Thiery, 2020].
However, it has been shown that good calibration on in-distribution data does not necessarily imply good calibration performance under covariate-shift [Ovadia et al., 2019].
Experiments
Datasets and Network Architectures We employ the widely used WideResNet28-10 (WRN) [Zagoruyko and Komodakis, 2016] and ResNet50 (RN50)
[He et al., 2016] architectures.
We train them on CIFAR-10 (C10) and CIFAR-100 (C100) datasets.
We employ RN50 to perform experiments on ImageNet-1K
[Deng et al., 2009] dataset.
We report the average of all the metrics computed onseeds.
For further details about the code base and the hyperparameters, refer to Appendix A.
For Covariate Shift (CS) experiments on models trained on C10 and C100, we resort to the widely used CIFAR10-C (C10-C) and CIFAR100-C (C100-C) datasets, corrupted versions of C10 and
C100 [Hendrycks and Dietterich, 2019].
These datasets are made by applying 15 synthetically generated but realistic corruptions at 5 degrees of intensity on the test sets of C10 and C100, respectively.
For CIFAR-10, we also use the CIFAR-10.1 (C10.1)
[Recht et al., 2018] and CIFAR- 10.2 (C10.2)
[Lu et al.
datasets designed to test the generalization of CIFAR-10 classiﬁers to natural covariate shifts .
To the best of our knowledge there is no such analogous dataset for CIFAR100.
For ImageNet-1K experiments, we use the widely considered ImageNet-A (A)
[Hendrycks et al., 2019b], ImageNet-R (R)
[Hendrycks et al., 2021], ImageNetv2 (V2)
[Recht et al., 2019], and
ImageNet-Sketch (SK)
[Wang et al., 2019] datasets for covariate shift experiments.
For Out-of-Distribution (OOD) detection, following SNGP [Liu et al., 2020a], we use C100 and
SVHN Netzer et al.
[2011] as OOD datasets for models trained on C10.
Similarly, for models trained on C100, we use C10 and SVHN as OOD datasets.
Additionally, we also consider the Tiny-ImageNet (T-ImageNet) dataset [Le and Yang, 2015] as OOD set in both the cases.
For models trained on
ImageNet-1K, we use ImageNet-O (O)
[Hendrycks et al., 2019b] as the OOD dataset.
Methods considered for comparisons Besides the natural comparison of our method with
Mixup [Zhang et al., 2018] and networks trained via ERM on the standard cross-entropy loss (which we will refer to as DNN), we consider several other methods from the OOD detection and CS literature.
For models trained on C10 and C100, we consider:
• DNN-SN and DNN-SRN: taking inspiration from [Liu et al., 2020a], we consider DNN models trained with Spectral Normalization (SN)
[Miyato et al., 2018a] and Stable Rank Normalization (SRN)
[Sanyal et al., 2020] to control the bi-Lipschitz constant of the networks, which has been shown to affect the generalization properties of neural networks.
• SNGP:
Spectrally Normalized Gaussian Process [Liu et al., 2020a].
• DUQ: Deterministic Uncertainty Quantiﬁcation [van Amersfoort et al., 2020].
• KFAC-LLLA: KFAC-Laplace Last Layer Approximation [Kristiadi et al., 2020] makes a model Bayesian at test time by taking the Laplace approximation of the last layer
[Ritter et al., 2018].
We provide a simple outline of this approach in Appendix D. • AugMix [Hendrycks et al., 2020b]: A data augmentation technique that applies randomized augmentations to an input while enforcing them to be consistent during training.
With the recommended hyperparameters in the paper, it is almost 4 × slower than DNN during training while having the same inference requirements.
• DE: Deep Ensembles [Lakshminarayanan et al., 2017] with 5 members, requiring 5 × more compute than most single-model approaches such as DNN.
We would like to mention that, compared to vanilla DNN, our approach (RegMixup) is almost 1.5 × slower and Mixup is about 1.2 × slower during training, while having the same inference requirements.
Due to the high compute requirements, for ImageNet-1K we consider DNN, Mixup and the two other strongest baselines: AugMix and DE.
We also cross-validate the hyperparameters on a 10% split of the test set, which is removed at test time.
Few missing experiments: Below we provide extensive experiments for proper benchmarking.
The number of datasets and architectures we use lead to many combinations, a few of which we were not able to produce good results for (even after extensive hyperparameter search), hence, some of the entries in the tables are missing.
For example, we could not make DUQ work on C100 as it exhibited unstable behaviour.
We could not produce promising results for SNGP on RN50 CIFAR experiments using their ofﬁcial implementation.
Similarly for AugMix RN50 experiments on C10 and C100.
We chose not to report these suboptimal numbers.
Further details can be found in Appendix A.
Table Note, we do not consider methodologies requiring access to an external dataset during training (either for CS or OOD ) as not only this would be an unfair comparison 7 , but we believe assuming such knowledge is against the goal of this work, which is to develop models robust to unknown scenarios.
4.1 RegMixup Improves Accuracy on In-distribution and Covariate-shift Samples
Small-scale (CIFAR) Experiments on In-distribution Data
In Table 2, we compare the accuracy of various approaches on the in-distribution test sets of C10 and C100, respectively.
Clearly,
• RegMixup outperforms Mixup in all these experiments.
In fact, RegMixup is the best performing one among all the single-model approaches.
• These improvements are non-trivial.
For instance, on WRN trained on C100, it outperforms DNN and Mixup by 1 .
67% and 0 .
65% , respectively.
It outperforms SNGP with a signiﬁcant margin of 4 .
[Table]
Note how the single-model approaches speciﬁcally designed to provide reliable predictive uncertainty estimations (for example, SNGP, DUQ,
KFAC-LLLA) underperform even compared to the vanilla DNN in terms of in-distribution accuracy.
In order to provide improved uncertainty estimates (as we will soon show), they trade clean data accuracy.
This type of behaviour is not observed in RegMixup.
Small-scale (CIFAR)
Covariate Shift Experiments For C10-C and C100-C, as typical in the literature, we report the accuracy averaged over all the corruptions and degrees of intensities in Table 3.
It is evident that our approach produces a remarkable improvement in the average accuracy compared to all the baselines (except AugMix, we discuss later why that might be the case).
For instance, for C100-C WRN experiments, our method achieves an accuracy improvement of 6.9% over DNN, of 3.86% over DE, and of 2.45% over Mixup.
Similarly, for C10-C WRN, our method achieves an improvement of almost 6.53% over DNN, of 4.81% over DE, and of 1.45% over Mixup.
For natural covariate shift datasets C10.1 and C10.2 as well, RegMixup outperforms all the baselines (including AugMix).
For instance, on C10.2, it obtains an improvement of 3.26% over DNN, of 1.5% over Mixup, and of 2.46% over the expensive DE.
[Table] [Table]
Why AugMix performs extraordinarily well on synthetically corrupted C10-C and C100-C but not on natural distribution shift C10.1 and
C10.2?
Looking at Table 3 one can observe AugMix’s extremely good performance on the C10C and C100-C.
However, at the same time, the model is underperforming with respect to RegMixup on C10.1 and C10.2.
Similarly, AugMix is outperformed by RegMixup on all ImageNet
CS experiments (and OOD as will be shown soon).
This seems to suggest that although the augmentations used during the training of AugMix are not exactly same as that of the corrupted test dataset, they tend to beneﬁt from synthetic forms of covariate shifts, hence the dramatic improvement in these particular scenarios.
Large-scale (ImageNet-1K) Experiments on
In-distribution Data As shown in Table 4,
RegMixup scales to ImageNet-1K and exhibits improved accuracy with respect to both Mixup, DNN, and AugMix.
In particular, it is 1.08% better than DNN, 0.53% better than Mixup and 0.80% better than AugMix.
DE in this case is the best performing one.
Large-scale (ImageNet-1K) Covariate Shift Experiments In Table 4 we report the results for common ImageNet-1K covariate-shift test sets.
As it can be seen, RegMixup is either the best performing one among all the single-model approaches, or it is the absolute winner including DE.
For instance, on ImageNet-A, RegMixup performs 2.67% better than Mixup and 3.20% better than
DNN.
Similarly, on ImageNet-V2 it performs 1.08% better than Mixup and 1.13% more than DNN.
[Table]
RegMixup also outperforms AugMix on all the considered datasets, while is outperformed by DE on
ImageNet-V2 (by 1.02% ) and performs competitively on ImageNet-SK.
These experiments clearly show the strong generalization of RegMixup under various in-distribution and CS experiments.
They also show that it does not trade clean data accuracy to do so.
4.2 Out-of-Distribution Detection Experiments
Following the standard procedure [Liu et al., 2020a], we report the performance in terms of AUROCfor the binary classiﬁcation problem between in- and out-distribution samples.
The predictive uncertainty of the model is typically used to obtain these curves.
Given an uncertainty measure (normally entropy, refer Appendix B for an extensive discussion), it is important for models to be more uncertain on OOD samples than on in-distribution samples to be able to distinguish them accurately.
This behaviour would lead to a better AUROC.
We report the OOD detection results for small-scale experiments (CIFAR 10/100) in Table 5, and for large-scale experiments (ImageNet) in Table 4.
Compared to the single-model approaches, it is clear that RegMixup outperforms all the baselines in both small-scale and large-scale experiments, except only in one situation (C100 as OOD, trained on
WRN) where SNGP outperformed it by an AUROC of 0 .
In fact, on ImageNet experiments, RegMixup outperformed all the baselines including DE.
We would also like to highlight that RegMixup provides signiﬁcantly better performance compared to Mixup on all these experiments.
For example, the improvement is more than 9% when SVHN is treated as the OOD dataset for WRN trained on either C10 or C100 (refer Table 5).
Similarly, we can observe dramatic improvements on other experiments as well.
4.3 Calibration on In-distribution and Covariate-shift datasets
Additionally, we provide the calibration performance of various competitive approaches.
Brieﬂy, calibration quantiﬁes how similar a model’s conﬁdence and its accuracy are [Osborne, 1991]).
To measure it, we employ the recently proposed Adaptive ECE (AdaECE)
[Mukhoti et al., 2020].
For all the methods, the AdaECE is computed after performing temperature scaling [Guo et al., 2017] with a cross-validated temperature parameter.
We also provide the AdaECE without temperature scaling in
Appendix C. For completeness, we also report the ECE in Appendix F.
In terms of calibration on in-domain test sets (refer Tables 6 and 7), our method either remarkably improves the AdaECE with respect to Mixup and DNN, or performs competitively (on ImageNet-1K).
Under covariate shift (refer Tables 7 and 8), on corrupted inputs, RegMixup underperforms with respect to Mixup on C10-C, but not on C100-C.
On all other C10 covariate shift datasets, RegMixup outperforms both Mixup and DNN.
Considering also the other baselines, except for the case of C10-C (in which AugMix signiﬁcantly outperforms any other baseline on WRN28-10), our method provides the best calibration in all other cases.
For example, on C100-C experiments on WRN28-10, in terms 8 Area Under Receiver Operating Characteristic curve. of AdaECE, RegMixup obtains a 4.43% improvement over DE, 2.52% over Mixup, and 2.47% over
SNGP.
Though RegMixup outperformed all other approaches in 12 scenarios out of total 17 presented here, it is clear that there is no single method that outperforms any other in all the considered settings
[Figure]
Conclusive Remarks
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
We proposed RegMixup, an extremely simple approach that combines Mixup with the standard cross-entropy loss.
We conducted a wide range of experiments and showed that RegMixup signiﬁcantly improved the reliability of uncertainty estimates of deep neural networks, while also provided a notable boost in the accuracy.
We showed that RegMixup did not just outperform
Mixup, it also outperformed most recent stateof-the-art approaches in providing reliable uncertainty estimates.
We hope that our work opens possibilities to explore situations where ERM and VRM can explicitly be combined together for practical beneﬁts.
An example would be label smoothing [Müller et al., 2019] that can be seen as an instance of VRM where the vicinal distribution is over the labels and the marginal distribution of the input samples (e.g., images) is approximated using deltas.
In Appendix G we conduct a preliminary analysis using CutMix Yun et al.
[2019] and recent techniques to train Vision Transformers Dosovitskiy et al.
[2021] that alternate between Mixup and CutMix.
Another possible future direction would be to use bi-modal vicinal distribution along with importance sampling in order to ensure that the unperturbed samples as well are used during training with high probability.
The observation that RegMixup ends up acting as a soft proxy to entropy maximizer for interpolated samples, a potential extension of our work regards the possibility of exploring different interpolation techniques to enforce high entropy behaviour on those regions of the input space.
Acknowledgements
This work is supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1.
We would like to thank the Royal Academy of Engineering, FiveAI and Meta
AI.
Francesco Pinto’s PhD is funded by the European Space Agency (ESA).
Meta AI authors are neither supported by the UKRI grants nor have any relationships whatsoever to the grant.
Thanks to John Redford, Jon Sadeghi, Kemal Oksuz, Zygmunt Lenyk, Guillermo Ortiz-Jimenez, Amartya
Sanyal, Pau de Jorge, and David Lopez-Paz for their valuable comments on the paper.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the "Methods considered for Comparison" section (c) Did you discuss any potential negative societal impacts of your work?
[No] Not expected (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code is on github.
The appendix contains an explicative pseudocode
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] They are speciﬁed either in the Experiments section of the main paper or the Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
input Batch B , α , θ t ¯ B ← ∅ λ 0 ∼ Beta ( α, α ) for ∀ ( x i , y i ) ∈
B do randomly select ( x j , y j ) from B\ ( x i , y i )
¯
B ← ¯ B ∪ ( λ 0 x
i
+ (1 − λ 0 ) x
j , λ
y i + (1 − λ 0 )
y j ) end for
L = CE ( B ) + CE ( ¯ B ) //
Loosely speaking, compute cross-entropy loss on both the batches return θ t +1 obtained by updating θ t by optimizing the above loss
A Experimental Details A.1 Code-base
The RegMixup training procedure is outlined in Algorithm 1.
For fair comparisons, when training on C10 and C100, we developed our own code base for all the approaches (except SNGP, DUQ and AugMix) and performed an extensive hyperparameter search to obtain the strongest possible baselines.
We would like to highlight that it was not easy to make a few recent state-of-the-art approaches work in situations different from the ones they reported in their papers as these approaches mostly required non-trivial changes to the architectures and additional sensitive hyperparametes.
We also observed that their performances did not easily translate to new situations.
Below we highlight few of these issues we faced and the measures we took for comparisons.
For DUQ , the original paper did not perform large scale experiments similar to ours.
Unfortunately, we could not manage to make their code work on C100 as the training procedure seemed to be unstable .
For this reason, wherever possible , we borrowed the numbers for DUQ from the SNGP paper.
Please note that the authors of SNGP performed non-trivial modiﬁcations to the original DUQ methodology to make it work on C100.
For SNGP , we used the publicly available code following exactly the same procedure as mentioned in their original paper.
The code diverges slightly from the procedure described in their paper, hence the slight differences in the performance.
The only modiﬁcation we performed to the ofﬁcial code-base was to make the inference procedure consistent with the one described in the paper: indeed, in their code they implement a mean-ﬁeld approximation to estimate the predictive distribution [Lu et al., 2020], while in their paper they use Monte Carlo Integration with a number of samples equal to the number of members in the ensembles they use as a baseline, which provides better calibration.
The rationale is that we could not ﬁnd an obvious way to tune the mean-ﬁeld approximation hyperparameters to improve at the same time both the calibration and OOD detection performance (indeed, the mean-ﬁeld approximation imposes a trade-off between calibration and OOD detection performance ).
Additionally, since the standard KFAC-LLLA uses the same Monte Carlo Integration procedure, we opted for the latter for a fair comparison.
For the SNGP RN50 experiments, we tried running the ofﬁcial implementation on C10 and C100, but we could not make SNGP converge to SOTA accuracy values.
The authors of SNGP did not provide experiment results on C10 and C100 on RN50.
Hence we decided not to report these experiments for SNGP.
For the KFAC-LLLA we leverage the ofﬁcial repository 9 [Hobbhahn et al., 2021] and the Backpack library [Dangel et al., 2020] for the computation of the Kronecker-Factored Hessian.
For AugMix , we used their code base and the exact training procedure.
AugMix seems to be sensitive to hyperparameters of the training procedure as we could could not get the considered architectures to converge to acceptable accuracy levels under the training regime we used for all other baselines.
Even with the recipes provided in the AugMix paper, we could not get it to converge to competitive accuracy levels when using RN50 on C10 and C100 hence we decided not to report these experiments for AugMix.
A.2 Optimization
For C10 and C100 training, we use SGD with Nesterov momentum 0 .
9 for 350 epochs and a weight decay of 5 ×
10 − 4 .
For WRN, we apply a dropout p = 0 .
1 at train time.
For all our experiments we set the batch size to 128 10 .
At training time, we apply standard augmentations random crop and horizontal flip similar to [Liu et al., 2020a]).
The data is appropriately normalized before being fed to the network both at train and test time.
For ImageNet-1K training, we use SGD with momentum for 100 epochs, learning rate 0 .
1 , cosine learning scheduler, weight decay of 1 × 10 − 4 , batch size 128 and image size 224 × 224 .
We use color jitter , random horizontal flip and random crop for augmentation.
We leverage the timm library for training [Wightman, 2019] all the considered methods with Automatic Mixed
Precision to accelerate the training.
A.3 Hyperparameters
• For DNN-SN and DNN-SRN the spectral norm clamping factor (maximum spectral norm of each linear mapping)
c ∈ { 0 . 5 , 0 .
75 , 1 . 0 } and the target of stable rank r ∈ { 0 . 3 , 0 . 5 , 0 .
9 } (as r = 1 for SRN is the same as applying SN with c = 1 . 0 ).
Refer to Miyato et al.
[2018b] and [Sanyal et al., 2020] for details about these hyperparameters.
• For Mixup, we consider a wide range of Beta distribution hyperparameter α ∈ { 0 . 1 , 0 .
• For RegMixup we consider the Beta distribution hyperparameters to be α
∈
5 , 1 , 5 , 10 , 15 , 20 , 30 } , and the mixing weight η ∈ { 0 . 1 , 1 , 2 } .
• For KFAC-LLLA we take 1000 samples from the distribution.
Although the number might seem quite high, we could not notice signiﬁcant improvements using a lower number of samples.
We tuned the prior variance σ 0 needed for the computation of the Laplace approximation minimising the ECE on the validation set.
We also tried using the theoretical value σ 0 = 1 /τ
[Kristiadi et al., 2020], where τ represents the weight decay, but it produced inferior results with respect to our cross-validation procedure.
We provide an overview of the KFAC-LLLA in Appendix D. • For Deep Ensembles we use 5 members.
• When temperature scaling is applied, the temperature T is tuned on the validation set, minimising the ECE (we considered values ranging from 0 .
1 to 10 , with a step size of 0 . 001 ).
For Deep Ensembles, we ﬁrst compute the mean of the logits, then scale it by the temperature parameter before passing it through the softmax.
All the cross-validated hyperparameters are reported in Table 9.
The cross-validation is performed with stratiﬁed-sampling on a 90/10 split of the training set to maximise accuracy 11 on C10 and C100.
For ImageNet, we split the test set using the same proportion to obtain the validation set, which is then removed from the test set during evaluation.
It is important to observe that:
• Cross-validating hyperparameters based solely on the ECE can prefer models with lower accuracy but better calibration.
However, a method improving calibration should avoid degrading accuracy.
• Hyperparameters should not be cross-validated based on CS experiments and OOD detection metrics as they these datasets should be unknown during the training and hyperparameter selection procedure as well.
The results of the cross-validation for RegMixup for CIFAR-10, CIFAR-100 can be found in Table 14.
To support our claim that Mixup tends to prefer lower α values, we do not only report the cross-validation accuracy on Mixup for the experiments in the main paper (Table 12) but we also report them for two other popular architectures on CIFAR-10 and CIFAR-100:
DenseNet-121 Huang et al.
[2017b] and PyramidNet200 Han et al.
[2017] (Table 13).
C C
ImageNet
[Table]
B Existing Uncertainty Measures
There are various uncertainty measures and there is no clear understanding on which one would be more reliable.
In our experiments we considered the following metrics and chose the one best suited for each method in order to create the strongest possible baselines.
Let K denote the number of classes, p i the probability of i -th class, and s i the logit of i -th class.
Then, these uncertainty measures can be deﬁned as:
• Entropy : H ( p ( x ))
= − (cid:80)
Ki =1 p i log p i .
• Dempster-Shafer
[Sensoy et al., 2018]: DS ( x ) =
K / ( K + (cid:80) Ki
=1 exp( s i )) .
• Energy : E ( x ) = − log (cid:80) Ki
=
exp
( s i ) (ignoring the temperature parameter).
This metric was used in [Liu et al., 2020b] for OOD.
• Maximum Probability Score : MPS ( x ) = max i p i .
• Feature Space Density Estimation (FSDE):
Assuming that the features of each class follow a Gaussian distribution, there are several ways one can estimate the belief of a test sample belonging to in-distribution data and treat it as a measure of uncertainty.
One such approach is to compute the Mahalanobis score arg min i ∈
y ( φ ( x ) − µ i ) T Σ − 1 i ( φ ( x ) − µ i ) , where µ i and Σ i are class-wise mean and the covariance matrices of the train data, and φ ( x ) is the feature vector.
In the main paper, we report the OOD detection performance using the DS score (as it provided slightly improved performance in most cases), except when it damages the performance of a method (e.g. Mixup) or when it does not yield improvements (e.g. KFAC-LLLA).
In these situations we use the entropy as the uncertainty measure.
Remarks regarding various metrics: We would like to highlight a few important observations that we made regarding these metrics.
(1) DS and E are equivalent as they are both decreasing functions of (cid:80) Ki
=1 exp( s i ) , and since log does not modify the monotonicity, both will provide the same ordering of a set of samples.
Hence, will give the same AUROC values.
(2) We observed
DS and H to perform similarly to each other except in a few situations where DS provided slightly better results.
(3) MPS , in many situations, was slightly worse.
(4) We found Gaussian assumption based density estimation to be unreliable .
Though it provided extremely competitive results for
C10 experiments, sometimes slightly better than the DS based scores, it performed very poorly on
C100.
We found this score to be highly unstable as it involves large matrix inversions.
We applied the well-known tricks such as perturbing the diagonal elements and the low-rank approximation with high variance-ratio, but the results were sensitive to such stabilization and there is no clear way to cross-validate these hyperparameters.
[Table] [Table]
C Calibration Metrics without Temperature Scaling
For completeness, we report the calibration metrics over all the methods and considered datasets without the temperature scaling [Guo et al., 2017] in Tables 10 and 11.
Details about the crossvalidation procedure used when temperature scaling is applied is provided in Appendix A.
D Bayesian at Test Time: Last Layer Laplace Approximation
A structural problem of using MLE logistic regression is that the produced uncertainties depend on the decision boundary.
On the other hand, replacing the MLE logistic regression with a Bayesian logistic regression and estimating the predictive posterior employing a Laplace approximation allows to produce better uncertainties [Kristiadi et al., 2020].
However, a Bayesian training either requires a modiﬁcation in the architecture [Liu et al., 2020a] or makes the inference procedure very expensive [Kingma et al., 2015, Gal and Ghahramani, 2016].
Since the objective is to utilize the standard MLE training of neural networks, the idea of Kronecker-Factored Last Layer Laplace
Approximation [Kristiadi et al., 2020] is making the network Bayesian at test time with almost no additional cost.
Let w be the parameters of the of the last layer of a neural network, then we seek to obtain the posterior only over w .
Let p ( w | x ) be the posterior, then the predictive distribution can be written as:
[Equation]
where, s is the logit vector and softmax ( s k ) is the k -th index of the softmax output of the network.
The Laplace approximation assumes that the posterior p ( s |D ) ∼ N ( s | µ, Σ) , where µ is a mode of the posterior p ( w |D ) (found via standard optimization algorithms for NNs) and Σ is the inverse of the Hessian H − 1 = − ( ∇ 2 log p ( w |D ) | µ ) − 1 .
For the formulations and deﬁnitions, including the variants with the terms associated to the bias, we refer to [Kristiadi et al., 2020].
For our experiments, we obtain Σ using the Kronecker-factored (KF) approximation [Ritter et al., 2018].
Broadly speaking, the KF approximation allows to reduce the computational complexity of computing the Hessian by factorizing the inverse of the Hessian as H − 1 ≈ V − 1 ⊗ U − 1 , then the covariance of the posterior evaluated at a point x takes following form Σ
= ( φ ( x ) T V φ ( x ))
U .
This procedure can be easily implemented using the Backpack library [Dangel et al., 2020] to compute V and U by performing a single pass over the training set after the end of the training, as detailed in the Appendix of [Kristiadi et al., 2020] and clearly exempliﬁed in the code-base of [Hobbhahn et al., 2021].
Let Σ k be the covariance matrix of the posterior over the last linear layer parameters for the kth class obtained using the Laplace approximation around µ , then, given an input x , we obtain σ k = φ ( x ) (cid:62)
Σ k φ ( x ) representing the variance of k-th logit s k .
Once we obtain the covariance matrix, the Monte Carlo approximation of the predictive distribution (equation (4)) is obtained as:
[Equation]
where, m logit vectors s ( i ) are sampled from a distribution with mean s and a covariance matrix (depending on the approximation used).
Lu et.
al
[Lu et al., 2020] showed that similar performance can be achieved via the mean-ﬁeld approximation which provides an approximate closed form solution of the integration in equation (4) involving the re-scaling of the logits and then taking the softmax of the re-scaled logit.
The re-scaling is deﬁned as follows:
[Equation]
Note, the scaling of the k-th logit depends on its variance (obtained using the Laplace approximation) and a hyperparameter λ .
This approximation is efﬁcient in the sense that it does not require multiple samples as required in the MC approximation (which can become expensive as the number of classes and samples grow).
In our experiments, we use the MC approximation, since we could not ﬁnd an obvious way to ﬁne-tune λ .
Additionally, we observe that the mean-ﬁeld approximation imposes a trade-off between calibration and OOD detection performance.
Increasing λ , indeed, ﬂattens the softmax distribution and improves OOD detection scores; although, as a consequence, harms calibration by making the network underconﬁdent.
E Additional Insights: RegMixup encourages compact and separated clusters
Here we provide additional experiments to show that RegMixup encourages more compact and separated clusters in the feature space We use the well known Fisher criterion [Bishop, 2006,
Chapter 4] to quantify the compactness and separatedness of the feature clusters.
Fisher Criterion: Let C k denotes the indices of samples for k -th class.
Then, the overall within-class covariance matrix is computed as S W = (cid:80)
Kk =1 S k , where S k = (cid:80) n ∈C k ( φ ( x n ) − µ k )( φ ( x n ) − µ k ) (cid:62) , µ k = (cid:80) n ∈C k φ ( x n ) N k , and φ ( x n ) denote the feature vector.
Similarly, the between-class covariance matrix can be computed as S B = (cid:80)
Kk =1 N k ( µ k − µ )( µ k − µ ) (cid:62) ,
where µ = 1 N (cid:80) Kk =1 N k µ k , and N k is the number of samples in k -th class.
Then, the Fisher criterion is deﬁned as α = trace ( S − 1 W S B ) .
Note, α would be high when within-class covariance is small and between-class covariance is high, thus, a high value of α is desirable .
In Figure 4, we compute α over the C10 dataset with varying degrees of domain-shift.
As the amount of corruption increases, α gradually decreases for all the models.
However, RegMixup consistently provides the best α in most cases.
[Table] [Table]
F ECE results
In this section we report the Expected Calibration Error (ECE) values.
While ECE is more popular than AdaECE, the latter uses an adaptive binning scheme that better accounts for the bias introduced by the fact neural networks tend to concentrate most samples in high-conﬁdence bins [Mukhoti et al., 2020].
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
IND
[Table] [Table]
Methods
[Table]
Mixup
[Table]
RegMixup
[Table]
DE (5 × )
[Table] [Table]
[Table] [Table]
[Table] [Table]
Methods
[Table]
G Using other vicinal distributions as regularizers
In this Section we perform a preliminary analysis about using other techniques derived from Mixup as regularizers.
In particular, we consider the following methods:
• CutMix Yun et al.
[2019]: instead of performing the convex combination of two samples, cuts and pastes a patch of an image into another, and the area of the patch is proportional to the interpolation factor λ .
We set the CutMix probability to p = 1 and cross-validate α ∈ { 0 . 1 , 0 . 2 , 0 .
3 , 1 , 10 , 20 } , • RegCutMix: we replace Mixup with CutMix in our formulation.
Same hyperparameters as above.
• Mixup + CutMix: we call this way the Transformers-inspired training procedure that at each iteration randomly decides (with equal probability) whether to apply Mixup or CutMix.
For both Mixup and CutMix we cross-validate α ∈ { 0 . 1 , 0 . 3 , 1 , 10 } .
• RegMixup + CutMix: as above, but using Mixup as a regularizer of the cross-entropy when Mixup is selected.
Hyperparameters as above, η ∈ { 0 . 1 , 1 , 3 }
I.I.D.
[Table]
Shift
[Table]
D.
C C
[Table]
C
[Table]
C
[Table]
C
SVHN
T-ImageNet
[Table]
WRN
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
RegMixup + RegCutMix: as above, but also using CutMix as a regularizer of the crossentropy when CutMix is selected.
Hyperparameters as above, η is kept the same for both Mixup and CutMix
The results are reported in Tables 18 and 19.
As it can be seen: (1) in most cases RegCutMix underperforms with respect to CutMix, (2) in most cases RegMixup outperforms CutMix and RegCutMix, (3) Mixup+CutMix represents an extremely competitive method for in-distribution accuracy, but not as competitive for distribution shift and out-of-distribution detection, (4) RegMixup+CutMix is most of the times inferior to Mixup+CutMix, (5) RegMixup+RegCutMix is an extremely competitive method in several cases.
We leave to future research exploring whether and how it is possible to combine Mixup-inspired techniques as regularizers to further improve the accuracy on i.i.d.
and distribution-shifted data, and the outof-distribution detection performance.
We remark that in all these setups, RegMixup is extremely competitive and several times the best method, thus pointing out that despite its simplicity, it is extremely effective.