C OMBINING E NSEMBLES AND D ATA A UGMENTATION CAN H ARM YOUR C ALIBRATION
A BSTRACT
Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model’s calibration and robustness.
Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness.
In this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration.
This leads to a trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration.
On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy.
We investigate this pathology and identify a compounding under-conﬁdence among methods which marginalize over sets of weights and data augmentation techniques which soften labels.
Finally, we propose a simple correction, achieving the best of both worlds with signiﬁcant accuracy and calibration gains over using only ensembles or data augmentation individually.
Applying the correction produces new state-of-the art in uncertainty calibration across CIFAR-10, CIFAR-100, and ImageNet.
I NTRODUCTION
Many success stories in deep learning (Krizhevsky et al., 2012; Sutskever et al., 2014) are in restricted settings where predictions are only made for inputs similar to the training distribution.
In real-world scenarios, neural networks can face truly novel data points during inference, and in these settings it can be valuable to have good estimates of the model’s uncertainty.
For example, in healthcare, reliable uncertainty estimates can prevent over-conﬁdent decisions for rare or novel patient conditions (Dusenberry et al., 2019).
We highlight two recent trends obtaining state-of-the-art in uncertainty and robustness benchmarks.
Ensemble methods are a simple approach to improve a model’s calibration and robustness (Lakshminarayanan et al., 2017).
The same network architecture but optimized with different initializations can converge to different functional solutions, leading to decorrelated prediction errors.
By averaging predictions, ensembles can rule out individual mistakes (Lakshminarayanan et al., 2017; Ovadia et al., 2019).
Additional work has gone into efﬁcient ensembles such as MC-dropout (Gal and Ghahramani, 2016), BatchEnsemble, and its variants (Wen et al., 2020; Dusenberry et al., 2020; Wenzel et al., 2020).
These methods signiﬁcantly improve calibration and robustness while adding few parameters to the original model.
Data augmentation is an approach which is orthogonal to ensembles in principle, encoding additional priors in the form of invariant feature transformations.
Intuitively, data augmentation enables the model to train on more data, encouraging the model to capture certain invariances with respect to its inputs and outputs; data augmentation may also produce data that may be closer to an out-ofdistribution target task.
It has been a key factor driving state-of-the-art: for example, Mixup (Zhang et al., 2018; Thulasidasan et al., 2019a), AugMix (Hendrycks et al., 2020), and test-time data augmentation (Ashukha et al., 2020).
A common wisdom in the community suggests that ensembles and data augmentation should naturally combine.
For example, the majority of uncertainty models in vision with strong performance are built upon baselines leveraging standard data augmentation (He et al., 2016;
Hendrycks et al., 2020)
(e.g., random ﬂips, cropping); Hafner et al.
cast data augmentation as an explicit prior for
Bayesian neural networks, treating it as beneﬁcial when ensembling; and Hendrycks et al.
(2020) highlights further improved results in AugMix when combined with Deep Ensembles (Hansen and
Salamon, 1990; Krogh and Vedelsby, 1995).
However, we ﬁnd the complementary beneﬁts between data augmentations and ensembels are not universally true.
Section 3.1 illustrates the poor calibration of combining ensembles (MC-dropout, BatchEnsemble and Deep Ensembles) and Mixup on CIFAR: the model outputs excessive low conﬁdence.
Motivated by this pathology, in this paper, we investigate in more detail why this happens and propose a method to resolve it.
Contributions.
In contrast to prior work, which ﬁnds individually that ensembles and Mixup improve calibration, we ﬁnd that combining ensembles and Mixup consistently degrades calibration performance across three ensembling techniques.
From a detailed analysis, we identify a compounding under-conﬁdence, where the soft labels in Mixup introduce a negative conﬁdence bias that hinders its combination with ensembles.
We further ﬁnd this to be true for other label-based strategies such as label smoothing.
Finally, we propose CAMixup to correct this bias, pairing well with ensembles.
CAMixup produces new state-of-the-art calibration on both CIFAR-10/100 (e.g., 0 .
4% and 2 .
3% on
CIFAR-10 and CIFAR-10C), building on Wide ResNet 28-10 for competitive accuracy (e.g., 97.5% and 89.8%) and on ImageNet (1.5%), building on ResNet-50 for competitive accuracy (77.4%).
B ACKGROUND ON C ALIBRATION , E NSEMBLES AND D ATA A UGMENTATION 2. C ALIBRATION
Uncertainty estimation is critical but ground truth is difﬁcult to obtain for measuring performance.
Fortunately, calibration error, which assesses how well a model reliably forecasts its predictions over a population, helps address this.
Let ( ˆ Y , ˆ P ) denote the class prediction and associated conﬁdence (predicted probability) of a classiﬁer.
Expected Calibration Error(ECE) :
One notion of miscalibration is the expected difference between conﬁdence and accuracy (Naeini et al., 2015): E ˆ P [ | P ( ˆ Y = Y | ˆ P = p ) − p | ] .
ECE approximates this by binning the predictions in [0 , 1] under M equally-spaced intervals, and then taking a weighted average of each bins’ accuracy/conﬁdence difference.
Let B m be the set of examples in the m th bin whose predicted conﬁdence falls into interval ( m − 1 M , mM ] .
The bin B m ’s accuracy and conﬁdence are:
[Equation]
where ˆ y i and y i are the predicted and true labels and ˆ p i is the conﬁdence for example x i .
Given n examples, ECE is (cid:80) Mm [-=
-]=
[Equation]
2. E NSEMBLES
Aggregating the predictions of multiple models into an ensemble is a well-established strategy to improve generalization (Hansen and Salamon, 1990; Perrone and Cooper, 1992; Dietterich, 2000).
BatchEnsemble:
BatchEnsemble takes a network architecture and shares its parameters across ensemble members, adding only a rank-1 perturbation for each layer in order to decorrelate member predictions (Wen et al., 2020).
For a given layer, deﬁne the shared weight matrix among K ensemble members as W ∈ R m × d .
A tuple of trainable vectors r k ∈ R m and s k ∈ R n are associated with each ensemble member k .
The new weight matrix for each ensemble member in BatchEnsemble is
[Equation]
F k , where F k = r [Equation] s
[Equation]
where ◦ denotes the element-wise product.
Applying rank-1 perturbations via r and s adds few additional parameters to the overall model.
We use an ensemble size of 4 in all experiments.
MC-Dropout : Gal and Ghahramani (2016) interpret Dropout (Srivastava et al., 2014) as an ensemble model, leading to its application for uncertainty estimates by sampling multiple dropout masks at test time in order to ensemble its predictions.
We use an ensemble size of 20 in all experiments.
Deep Ensembles: Composing an ensemble of models, each trained with a different random initialization, provides diverse predictions (Fort et al., 2019) which have been shown to outperform strong baselines on uncertainty estimation tasks (Lakshminarayanan et al., 2017).
We use an ensemble size of 4 in all experiments.
In this work, we focus on the interaction between data augmentation strategies and BatchEnsemble,
MC-Dropout, and deep ensembles.
Other popular ensembling approaches leverage weight averaging such as Polyak-Ruppert (Ruppert, 1988), checkpointing (Huang et al., 2017), and stochastic weight averaging (Izmailov et al., 2018) to collect multiple sets of weights during training and aggregate them to make predictions with only a single set.
2. D ATA A UGMENTATION
Data augmentation encourages a model to make invariant predictions under desired transformations which can greatly improve generalization performance.
For example, in computer vision, random leftright ﬂipping and cropping are de-facto approaches (He et al., 2016).
We highlight two state-of-the-art techniques which we study.
Mixup : Mixup (Zhang et al., 2018) manipulates both the features and the labels in order to encourage linearly interpolating predictions.
Given an example ( x i , y i ) , Mixup applies
[Equation]
Here, x j is sampled from the training dataset (taken from the minibatch), and λ ∼ Beta( a, a ) for a ﬁxed hyperparameter a > 0 .
Mixup was shown to be effective for generalization and calibration of deep neural networks (Zhang et al., 2018; Thulasidasan et al., 2019b).
Recent work has investigated why Mixup improves generalization (Guo et al., 2018;
Shimada et al., 2019) and adversarial robustness (Beckham et al., 2019; Pang et al., 2020; Mangla et al., 2020).
Given Mixup’s simplicity, many extensions have been proposed with further improvements (Yun et al., 2019; Berthelot et al., 2019; Verma et al., 2019;
Roady et al., 2020; Chou et al., 2020).
AugMix: Searching or sampling over a set of data augmentation operations can lead to signiﬁcant improvement on both generalization error and calibration (Cubuk et al., 2019b;a).
AugMix (Hendrycks et al., 2020) applies a sum of augmentations, each with random weighting, with a Jensen-Shannon consistency loss to encourage similarity across the augmentations.
AugMix achieves state-of-the-art calibration across in- and out-of-distribution tasks.
Let O be the set of data augmentation operations and k be the number of AugMix iterations.
AugMix samples w 1 , . .
w k ∼ Dirichlet( a, . .
, a ) for a ﬁxed hyperparameter a > 0 and op 1 , . . .
, op k from O .
Given an interpolation parameter m , sampled from Beta( a, a ) , the augmented input ˜ x augmix is:
[Equation]
M IXUP -E NSEMBLE P ATHOLOGY
We seek to understand the effect of data augmentations on ensembles.
In particular, we hope to verify the hypothesis of compounding improvements when combining the seemingly orthogonal techniques of data augmentation and ensembles.
To our surprise, we ﬁnd that augmentation techniques can be detrimental to ensemble calibration.
3. T HE S URPRISING M ISCALIBRATION OF E NSEMBLES WITH M IXUP
Ensembles are the most known and simple approaches to improving calibration (Ovadia et al.,
Lakshminarayanan et al., 2017), and Thulasidasan et al.
(2019b) showed that Mixup improves calibration in a single network.
Motivated by this, Fig. 1 applies Mixup to each ensemble member on CIFAR-10/CIFAR-100 with WideResNet 28-10 (Zagoruyko and Komodakis, 2016).
Here, we searched over Mixup’s optimal hyperparameter α (Eq. 3) and found that α = 1 gives the best result, which corroborates the ﬁnding in Zhang et al.
All data points in Fig.
1 are averaged overrandom seeds.
Figs.
1a and 1b demonstrate improved test accuracy (Red (ensembles without Mixup) to Blue (ensembles with Mixup)).
However, if we shift focus to Figs. 1c and 1d’s calibration error, it is evident that combining Mixup with ensembles leads to worse calibration (Red to Blue).
This is counterintuitive as we would expect Mixup, which improves calibration of individual models (Thulasidasan
BE
MC
DE 3.
(a) CIFAR10-Error ( % )
BE
MC
DE 16.
(b) CIFAR100-Error ( % )
BE
MC
DE      (
[Figure]
CIFAR
[Figure]
ECE
[Figure]
BE
MC DE [Figure] et al., 2019a), to also improve the calibration of their ensemble.
Fig.
1 conﬁrms this pattern across
BatchEnsemble ( BE ), MC-dropout ( MC ), and deep ensembles ( DE ).
This pathology also occurs on
ImageNet, as seen in Table 1.
[Figure]
Why do Mixup ensembles degrade calibration?
To investigate this in more detail, Fig.
2 plots a variant of reliability diagrams (DeGroot and Fienberg, 1983) on
BatchEnsemble.
We bin the predictions into M =equally spaced intervals based on their conﬁdence (softmax probabilities) and compute the difference between the average conﬁdence and the average accuracy as in
Eq.
1 for each bin.
Fig.
2 tracks this difference over varying conﬁdence levels.
A positive difference (Acc − Conf) implies under-conﬁdence with respect to the true frequencies; negative implies over-conﬁdence; and zero implies perfect calibration.
The backbone model in Fig.
2 is BatchEnsemble with an ensemble size of 4 (we also found this consistent for
MC-Dropout and Deep-Ensemble).
The ﬁgure presents 4 methods: Single: vanilla WideResNet 28-10; MixupSingle: WideResNet 28-10 model trained with Mixup; BatchEnsemble: vanilla BatchEnsemble WideResNet 28-10 model; MixupBE:
BatchEnsemble WideResNet 28-10 model trained with Mixup.
Fig.
2 shows that only models trained with Mixup have positive (Acc − Conf) values on the test set, which suggests that Mixup encourages under-conﬁdence.
Mixup ensemble’s under-conﬁdence is also greater in magnitude than that of the individual Mixup models.
This suggests that Mixup ensembles suffer from compounding under-conﬁdence, leading to a worse calibration for the ensemble than the individual Mixup models’ calibration.
This is contrary to our intuition that ensembles always improves calibration.
To further visualize this issue, Appendix C’s Fig.
8 investigates the conﬁdence (softmax probabilities) surface of deep ensembles and Mixup when trained on a toy dataset consisting of 5 clusters, each with a different radius.
We ensemble over 4 independently trained copies of 3-layer MLPs.
Deep ensemble’s predictive conﬁdence is plotted over the entire input data space in Fig.
8c.
The resulting predictions are extremely conﬁdent except at the decision boundaries.
Deep Ensemble still displays high conﬁdence in the area nearest to the origin which is expected to have lower conﬁdence level.
On the other hand, Fig.
8d shows that Mixup-Ensemble is only conﬁdent in a very constrained area around the training clusters, leading to an overall under-conﬁdent classiﬁer which conﬁrms our postulation of compounding under-conﬁdence.
3. I S THE PATHOLOGY SPECIFIC TO M IXUP ?
At the core of the issue is that Mixup conﬂates data uncertainty (uncertainty inherent to the data generating process) with model uncertainty.
Soft labels can correct for over-conﬁdence in single models which have no other recourse to improve uncertainty estimates.
However, when combined with ensembles, which incorporate model uncertainty, this correction may be unnecessary.
Because image classiﬁcation benchmarks tend to be deterministic, soft labels encourage predictions on training data to be less conﬁdent about their true targets even if they are correct.
We validate this hypothesis by showing it also applies to label smoothing.
Label Smoothing : Like Mixup, label smoothing applies soft labels: it smoothens decision boundaries by multiplying a data point’s true class by (1 − α ) , with probability α spread equally across other classes.
Using the same experimental setup as before, we apply increasing levels of label smoothing to ensembles of WideResNet 28-10 models trained on CIFAR-10.
Fig.
3 demonstrates the harmful effect of label smoothing on CIFAR-10 ECE, particularly when aggressive (coeff ≥ 0 . 2 ).
In the concurrent work, Qin et al.
(2020) found that label smoothing plus ensemble leads to worse calibration.
They showed that adjusting model conﬁdence successfully corrects the compounding underconﬁdence.
C ONFIDENCE A DJUSTED M IXUP E NSEMBLES (CAM IXUP )
In this section, we aim to ﬁx the compounding under-conﬁdence issue when combining Mixup and ensembles without sacriﬁcing its improved accuracy on both in- and out-of-distribution data
. 4. C LASS B ASED CAM IXUP
Mixup encourages model under-conﬁdence as shown in Fig.
Notice that Mixup assigns a uniform hyperparameter α to all examples in the training set.
To improve Mixup, we start from the intuition that in classiﬁcation, some classes are prone to be more difﬁcult than others to predict.
This can be conﬁrmed by Fig. 4a, which provides examples of per-class test accuracy.
Ideally, we prefer our model to be conﬁdent when it is predicting over easy classes such as cars and ships.
For harder classes like cats and dogs, the model is encouraged to be less conﬁdent to achieve better calibration.
Therefore, instead of a uniform Mixup hyperparameter for all classes, we propose to adjust the Mixup hyperparameter of each class by the difference between its accuracy and conﬁdence.
CAMixup’s intuition is that we want to apply Mixup on hard classes on which models tend to be over-conﬁdent.
On easy examples, we impose the standard data-augmentation without Mixup.
This partially prevents
Mixup models from being over-conﬁdent on difﬁcult classes while maintaining its good calibration on out-of-distribution inputs.
Denote the accuracy and conﬁdence of class i as Acc ( C i ) and Conf ( C i ) .
We adjust Mixup’s λ in
Eqn. 3 by the sign of Acc ( C i ) − Conf ( C i ) , which are deﬁned as Acc ( C i ) = 1 | C
i | (cid:80) x j ∈ C i 1 ( ˆ y j =
[Equation]
and
Conf ( C i ) = {+
+} | C i | (cid:80) x j ∈
[Equation]
[Equation] [Figure] [Figure]
If the model is already under-conﬁdent on class i (Acc ( C i ) > Conf ( C i ) )
[Figure]
Mixup is not applied to examples in the class [Figure] and λ i = 0 . However
[Figure] if Acc ( C i ) ≤ Conf ( C i ) [Figure] the model is over-conﬁdent on this class
[Figure]
and Mixup is applied to reduce model conﬁdence.
We compute the accuracy and conﬁdence on a validation dataset after each training epoch
[Figure]
Notice that λ i is dynamically updated at the end of each epoch.
To understand which classes are more often assigned Mixup operation, Fig. 4 calculates the number of times that λ i > 0 throughout training.
The maximum number of times is the number of total training epochs, which is 250 in the
BatchEnsemble model.
We ﬁnd that CAMixup rarely enables Mixup to easy classes such as cars and ships: the number of times is less than 10% of the total epochs.
For harder classes like cats and dogs, CAMixup assigns Mixup operation almost every epoch, accounting for more than 80% of total epochs.
In summary, Fig.
4 shows that CAMixup reduces model conﬁdence on difﬁcult classes and encourages model conﬁdence on easy classes, leading to better overall calibration.
Appendix D.1’s
Fig.
9a also shows that CAMixup effectively shifts the conﬁdence to the lower region.
Fig.
5 presents results of CAMixup on CIFAR-10 and CIFAR-100 test set, where we compare the effect of Mixup and CAMixup on different ensembling strategies (BatchEnsemble, MC Dropout,
DeepEnsemble).
Adding Mixup to ensembles improves accuracy but worsens ECE.
Adding CAMixup to ensembles signiﬁcantly improves accuracy of ensembles in all cases.
More importantly, the calibration results in Figs.
5c and 5d show that CAMixup ensembles are signiﬁcantly better calibrated than Mixup ensembles, for instance, CAMixup reduces ECE by more than 5X for BatchEnsemble over Mixup.
We observe a minor decrease in test accuracy (at most 0 .
2% ) when comparing CAMixup ensembles with Mixup ensembles, but we believe that this is a worthwhile trade-off given the signiﬁcant improvement in test ECE.
[Table] Table 1 presents similar experiments applied to ResNeton ImageNet, using BatchEnsemble as the base ensembling strategy.
These results are state of the art to the best of our knowledge: Dusenberry et al.
(2020) report 1.7% ECE with Rank-1 Bayesian neural nets and 3.0% with Deep Ensembles;
Thulasidasan et al.
(2019a) report 3.2% for ResNet-50 with Mixup, 2.9% for ResNet-50 with an entropy-regularized loss, and 1.8% for ResNet-50 with label smoothing.
4. P ERFORMANCE UNDER D ISTRIBUTION S
[Figure]
Here, we assess model resilience to covariate shift by evaluating on the CIFAR-10-C and CIFAR-100-C benchmarks (C stands for corruptions) proposed by
Hendrycks and Dietterich (2019a), which applytypes of corruptions each with 5 levels of intensity.
We evaluate the performance of CAMixup vs Mixup when applied to different ensembles, and report average error on ECE across different types of corruptions and intensities.
Fig.
6a shows that Mixup improves accuracy on the corrupted dataset because of its strong regularization effect.
However, the models tend to be over-conﬁdent as one moves further from the original distribution (higher corruption intensities), so encouraging underconﬁdence is not an issue.
This explains why Mixup ensembles maintain low ECE on out-of-distribution test data in Fig.
6b.
Fig.
6b also shows that CAMixup’s calibration on out-of-distribution data (CIFAR-10-C) is also on par with Mixup ensembles.
We observe the same result on CIFAR-100-C (Appendix D.1’s Fig. 9).
Thus, we successfully improve model calibration on in-distribution datasets without sacriﬁcing its calibration on out-of-distribution datasets.
C OMPOUNDING THE BENEFITS OF CAM IXUP WITH A UG M IX E NSEMBLES
We have investigated why certain data augmentation schemes may not provide complementary beneﬁts to ensembling.
We proposed class-adjusted Mixup (CAMixup) which compounds both accuracy and ECE over vanilla ensembles.
We believe that the insights from our work will allow the community and practitioners to compound SOTA performance.
We provide two concrete examples.
5. A UG M IX
We show how CAMixup can compound performance over ensembles of models trained with AugMix, which were shown by Hendrycks et al.
(2020) to achieve state-of-the-art accuracy and calibration on both clean and corrupted benchmarks.
We primarily focus on improving BatchEnsemble and we investigate if adding better data augmentation schemes closes the gap between memory-efﬁcient ensembles (BatchEnsemble) and independent deep ensembles.
As discussed in Section 2.3, AugMix only uses label-preserving transformations.
Therefore AugMix provides complementary beneﬁts to ensembles (and CAMixup).
This is consistent with calibration improvements in the literature with ensemble methods, which apply standard data augmentation such as random ﬂips, which also do not smoothen labels.
We consider a combination of AugMix and Mixup as it allows the model to encounter both diverse label-preserving augmentations and soft labels under a linearly interpolating regime.
The combination
BatchEnsemble
MixupBE
CAMixupBE
AugMix
AugMixup
AugCAMixup     5 Intensity
E rr o r (a) CIFAR-10 Error     5 Intensity
E C E (b) CIFAR-10 ECE     5 Intensity
E rr o r (c) CIFAR-100 Error     5 Intensity
E C E (d) CIFAR-100 ECE
[Table]
AugCAMixup
[Table]
and corrupt accuracy at the cost of calibration decay on testset. CAMixup bridges this gap with only a minor drop in accuracy.
AugMixup (AugMix + Mixup) can be written as
[Equation]
Consistent with earlier results on Mixup, Table 2 shows combining AugMixup with BatchEnsemble improves accuracy but worsens ECE, leading to under-conﬁdence on in-distribution data.
(Appendix D.2’s Fig. 10).
With our proposed ﬁx CAMixup, the combination AugCAMixup (AugMix + CAMixup) improves calibration while retaining the highest accuracy for ensembles. Fig. 7 shows detailed results on CIFAR-10-C and CIFAR-100-C. Similar to Mixup, AugMixup improves calibration under shift but worsens calibration on in-distribution.
However, our proposed AugCAMixup improves accuracy and calibration of ensembles on both clean and corrupted data.
To the best of our knowledge, these results are state-of-the-art in the literature: Dusenberry et al.
(2020) report 0 .
8% ECE and 1 . 8% ECE for CIFAR-10 and CIFAR-100 along with 8% and 11 .
ECE for corruptions; Guo et al.
(2017) report 0 .
54% and 2 .
3% ECE for the smaller Wide ResNeton CIFAR-10 and CIFAR-100 with temperature scaling (93% and 72% accuracy), and Ovadia et al.
(2019) demonstrated that temperature scaling does not extend to distribution shift.
5. T EMPERATURE S CALING
In concurrent work, Rahaman and Thiery (2020) consider the interplay between data augmentation and ensembling on calibration.
They also ﬁnd that Mixup ensembles can be under-conﬁdent, and propose temperature scaling as a solution.
Their core contribution is the same but differ in slight ways: we further this analysis by showing the compounding under-conﬁdence extends to other techniques applying soft labels such as label smoothing, and we propose CAMixup as a solution.
Post-hoc calibration techniques like temperature scaling are complementary to our proposal and do not address the core conﬂation issue with Mixup.
Corroborating ﬁndings of Ovadia et al.
(2019), Appendix G shows combining CAMixup and temperature scaling can further improve test calibration error; it does not improve out-of-distribution calibration.
Another concurrent work showed that calibrated ensemble members do not always lead to calibrated ensemble predictions (Anonymous, 2021).
C ONCLUSION
Contrary to existing wisdom in the literature, we ﬁnd that combining ensembles and Mixup consistently degrades calibration performance across three ensembling techniques.
From a detailed analysis, we identify a compounding under-conﬁdence, where Mixup’s soft labels (and more broadly, label-based augmentation strategies) introduce a negative conﬁdence bias that hinders its combination with ensembles.
To correct this, we propose CAMixup, which applies Mixup to only those classes on which the model tends to be over-conﬁdent, modulated throughout training.
CAMixup combines well with state-of-the-art methods.
It produces new state-of-the-art calibration across CIFAR-10,
CIFAR-100, and ImageNet while obtaining competitive accuracy.
Appendix H points out potential future work and limitations of CAMixup.
A D ATASET D ETAILS
CIFAR & CIFAR-C:
We consider two CIFAR datasets, CIFAR-10 and CIFAR-100 (Krizhevsky, 2009).
Each consists of a training set of size 50K and a test set of size 10K.
They are natural images with 32x32 pixels.
Each class has 5,000 training images and 500 training images on CIFAR-10 and
CIFAR-100 respectively.
In our experiments, we follow the standard data pre-processing schemes including zero-padding with 4 pixels on each sise, random crop and horizon ﬂip (Romero et al., 2015;
Huang et al., 2016; Srivastava et al., 2015).
If a training method requires validation dataset such as
CAMixup, we use separate 2 , 500 images from 50K training images as the validation set.
It’s important to test whether models are well calibrated under distribution shift.
CIFAR-10 corruption dataset (Hendrycks and Dietterich, 2019a) is designed to accomplish this.
The dataset consists of 15 types of corruptions to the images.
Each corruption types have 5 intensities.
Thus, in total
CIFAR-10C has 75 corrupted datasets.
Notice that the corrupted dataset is used as a testset without training on it.
Ovadia et al.
(2019) benchmarked a number of methods on CIFAR-10 corruption.
Similarly, we can apply the same corruptions to CIFAR-100 dataset to obtain CIFAR-100C.
ImageNet & ImageNet-C:
We used the ILSVRC 2012 classiﬁcation dataset (Deng et al., 2009) which consists of a total of 1.2 million training images, 50,000 validation images and 150,000 testing images.
Images span over 1,000 classes.
We follow the data augmentation scheme in He et al.
(2016), such as random crop and random ﬂip, to preprocess the training images.
During testing time, we apply a 224x224 center crop to images.
Similarly to CIFAR-C, we apply 15 corruption types withintensities each to obtain ImageNet-C (Hendrycks and Dietterich, 2019b).
B H YPERPARAMETERS IN S ECTION
We kept the same set of hyperparameters as the BatchEnsemble model in Wen et al.
All hyperparameters can be found in Table 3.
The most sensitive hyperparameter we found is whether to use ensemble batch norm, which applies a separate batch norm layer for each ensemble member; and the value of random_sign_init, which controls the standard deviation of Gaussian distributed initialization of s and r .
We kept BatchEnsemble CIFAR-10 the same as Wen et al.
(2020), which does not deploy ensemble batch norm.
We enable ensemble batch norm on CIFAR-100 and ImageNet.
This allows us to use larger standard deviation in the initialization.
The random_sign_init is − 0 .on
CIFAR-10 and − 0 .
75 on CIFAR-100 and - 0 .
75 on ImageNet.
In the code, we use negative value to denote the standard deviation of Gaussian distribution (positive value instead initializes with a
Bernoulli distribution under that probability).
In our case, we only use negative random_sign_init, which means we only consider Gaussian distributed initialization in this work.
[Table]
C E XCESSIVE UNDER - CONFIDENCE ON SYNTHETIC DATA
To further understand the conﬁdence surface of Mixup + Ensembles, we provided a visualization in
Fig.
We trained on a synthetic dataset consisting of 5 clusters, each with a different radius.
We ensemble over 4 independently trained copies of 3-layer MLPs.
We plotted the softmax probabilities surface of Mixup-Single model, Deep-Ensemble and Mixup-Ensemble. The softmax probabilities
[Figure] [Figure]
[Figure]
represent the model conﬁdence.
Fig.
8c shows that Deep-Ensemble predictions are extremely conﬁdent except at the decision boundaries.
Fig.
8b displays a lower conﬁdence than Deep-Ensemble.
This is beneﬁcial in the single model context because single deep neural networks tend to be over-conﬁdent and Mixup can partially correct this bias.
On the other hand, Fig.
8d shows that MixupEnsemble is only conﬁdent in a very constrained area around the training clusters, leading to an overall under-conﬁdent classiﬁer which conﬁrms our postulation of compounding under-conﬁdence.
D M ORE C ALIBRATION R ESULTS OF M IXUP -B ATCH E NSEMBLE
In Section 3.1, we demonstrated that combining Mixup and ensembles leads to worse calibration on testset.
In this appendix section, we complement the above conclusion with the analysis on corrupted datasets and with data-augmentation techniques like AugMix.
D. S UPPLEMENTARY RESULTS ON CAM IXUP
In this section, we provided supplementary results on CAMixup.
Fig.
2 shows that combining Mixup and BatchEnsemble leads to excessive under-conﬁdence.
In Fig.
9a, we showed that our proposed
CAMixup ﬁxes this issue by correcting the conﬁdence bias.
This explains why CAMixup achieves better calibration on in-distribution testset.
As demonstrated in Section 4.2, Mixup improves model out-of-distribution performance because of its strong regularization effect.
We showed that our proposed CAMixup inherits Mixup’s improvement on CIFAR-10-C. Fig.
9b and Fig.
9c show that this conclusion seamlessly transfers to CIFAR-100-C. We also supplement Fig.
5 with Table 4 and
Table 5, illusrating detailed numbers.
[Figure]
a
[Figure]
f i d
[Figure]
n c
[Figure]
D. SUPPLEMENTARY RESULTS ON AUGMIX
We show that Mixup does not combine with ensembles without sacriﬁcing in-distribution calibration in Section 3.1.
As discussed in Section 2.3, AugMix only uses label-preserving transformations and does not modify the labels.
Intuitively, it does not reduce model conﬁdence.
We support this intuition with Fig.
It shows that AugMix does not lead to under-conﬁdence.
Therefore it can be combined with ensembles without any calibration issue
[Figure] In Table [Figure] we showed that combining AugMix and Mixup leads to worse calibration due to the under-conﬁdence although AugMix itself does not.
To better understand the insights beyond staring at scalars, we provided the reliability diagram analysis as well.
In Figure 10, we showed that the underconﬁdence issue of AugMixup (Augmix + Mixup) still exists.
It suggests that applying CAMixup to Augmix can correct the under-conﬁdence bias as what we showed in Fig.
10a and Fig. 10b.
Our proposed CAMixup allows to compound performance of ensembles and data augmentation to achieve the best possible performance.
[Table] [Table]
[Table]
E D EEP E NSEMBLES WITH M IXUP
We showed that CAMixup improves Mixup BatchEnsemble calibration on testset without undermining its calibration under distribution shift in Section 4.
In this section, we show that the improvement can also be observed on deep ensembles.
In Fig.
11, we showed the under-conﬁdence bias we observed on Mixup + BatchEnsemble also exists on Mixup + deep ensembles, with an even more obvious trend.
Beyond commonly used ECE measure, we also explore other calibration measures.
They further conﬁrmed our under-conﬁdence intuition.
We provide some brief explanation on how to calculate
ACE, SCE and TACE.
ACE measure is the same as ECE except for the binning scheme.
Rather than equally divide the conﬁdence evenly into several bins, ACE choses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions.
SCE is the same as ECE except that it accounts for all classes into calibration measure rather than just looking at the class with maximum probability.
The softmax predictions induce inﬁnitesimal probabilities.
These tiny predictions can wash out the calibration score.
TACE is proposed to set a threshold to only include predictions with large predictive probability, to address the above issue.
We present the results of Mixup, CAMixup, AugMix, AugMixup and AugCAMixup on deep ensembles in Table 6.
We notice that the improvement of CAMixup on deep ensembles is smaller than its improvement on BatchEnsemble.
We postulate that this is because Mixup + deep ensembles is much badly calibrated than Mixup + BatchEnsemble.
For example, AugMixup + deep ensembles achieve 2.71% and 6.86% ECE on CIFAR-10 and CIFAR-100.
In the meanwhile, AugMixup +
BatchEnsemble achieve 1.71% and 4.19%.
Thus, even if CAMixup can improve the calibration of Mixup + deep ensembles, it still cannot beat AugMix + deep ensembles.
As a result, when we say we close the calibration gap between BatchEnsemble and deep ensembles, we are comparing
AugCAMixup BatchEnsemble (BatchEnsemble + CAMixup + Augmix) to AugMix deep ensembles.
This is because AugMix deep ensembles achieve the best calibration among all variants we tried.
How to completely ﬁx the under-conﬁdence in deep ensembles is a natural extension of this work.
Since we focus on bridging the calibration gap between BatchEnsemble and deep ensembles, we delegate the complete ﬁx in deep ensembles to the future work.
F M ETRICS OTHER THAN ECE
ECE is the standard metric in calibration, but it is a biased estimate of true calibration (Vaicenavicius et al., 2019).
Heavily relying on ECE metric might lead to inconsistent conclusion.
In this section, we computed the calibration error with recently proposed calibration estimator which reduces bias in ECE, including debiased calibration estimator (Kumar et al., 2019) (DCE) and SKCE (Widmann et al., 2019).
fig.
12 shows that our conclusion in the main section are also supported by these two recently proposed calibration estimators.
In particular, the improvement of proposed CAMixup over
[Figure]
Method/Metric
BatchEnsemble
Deep-Ensembles
Acc( ↑ ) SKCE( ↓ ) cA/cSKCE Acc( ↑ ) SKCE( ↓ ) cA/cSKCE
Vanilla 96.
3.4 e −77.5/0.
3.4 e −54.1/0.
Mixup 96. 4e −80.0/0. 4e −59.3/0. CAMixup 96.
1.3 e −81.1 / 0.
3e −59.7 / 0. [Figure] [Figure] error to the ranking in Fig.
5 and Fig.
This plot shows that our proposed CAMixup is effective in reducing Mixup calibration error when combined with ensembles.
G CAM IXUP WITH T EMPERATURE S CALING
See Fig. 13.
H L IMITATIONS AND F UTURE W ORK
We describe limitations of our work, signalling areas for future research.
One limitation of CAMixup is that all examples in the same class still share the same Mixup coefﬁcient.
This leaves room for developing more ﬁne-grained adaptive Mixup mechanisms, such as adapting the Mixup coefﬁcient per example.
This relates to an open research question: how do you measure the training difﬁculty of a data point given a deep network?
(Toneva et al., 2018; Agarwal and Hooker, 2020)
Another limitation is we showed that CAMixup still cannot fully ﬁx the miscalibration of Mixup + deep ensembles in
Appendix E, due to the fact that Mixup + deep ensembles leads to even worse calibration than Mixup + BatchEnsemble.
This raises a harder question which CAMixup cannot completely solve but also leaves more research room to further understand why Mixup is worse on deep ensembles and how to address it.
Thus, we leave the question on how to address the above issues to future work.
Next, we determine whether to use Mixup based on the reliability (Mean Accuracy - Mean Conﬁdence) of each class on validation set.
One concern is that CAMixup does not scale well to a large number of classes.
Fortunately, we showed that this works on problems up to 1000 classes (ImageNet).
Additionally,
Mixup has been most successful in the vision domain, hence our focus; and with preliminary success on tabular data and natural language processing (Zhang et al., 2018;
Guo et al., 2019).
Assessing whether CAMixup and ensembling techniques translate to text is an interesting area.
Algorithm
CAMixup initialize prevacc i = 0 , i ∈ D initialize forgetting T [ i ] = 0 ,
i ∈ D initialize MixupCoeﬀ[i]
=while training do
B ∼ D
# sample a minibatch
Apply Mixup on B based on
MixupCoeﬀ for example i ∈ B do compute acc i
if prevacc i >
acc
i then
T [ i ] =
T
[ i ] +prevacc i
=
acc i end if end for gradient update classiﬁer on B rank =
sort ( T ) threshold = rank
[ | D | // 2] for example
i ∈ B do if T [ i ] > threshold then
MixupCoeﬀ[ i ] = a else
MixupCoeﬀ[ i ] =
end if end for end while
We took a ﬁrst step in developing a more ﬁne-grained adaptive Mixup mechanism.
Recall that class based
CAMixup calculates the reliability (Accuracy - Conﬁdence) at the end of each epoch, then it decided whether to apply Mixup in each class (illustrated in
Fig. 4).
This requires extra computation on validation dataset and it assigns uniform Mixup coefﬁcient within one class.
By leveraging recently developed forgetting count (Toneva et al., 2018), we can adjust Mixup coefﬁcient for each example based on its forgetting counts.
The intuition is if an examples is associated with high forgetting counts, it indicates the model tends to forget this example.
To achieve better calibration, we should place low conﬁdence on this example.
The algorithm of forgetting counts based
CAMixup is presented in Algorithm 1.
In summary, we ﬁrst calculate the forgetting counts for each training example and obtain the median of these counts as the threshold.
Then, CAMixup applies Mixup to the training example whose forgetting counts are higher than the median.
We provided a preliminary results on CIFAR-10 in
Fig.
It demonstrates that forgetting counts based
CAMixup outperforms class based CAMixup on most metrics across BatchEnsemble and MC-dropout.
One exception is that it underperforms on test calibration on MC-dropout.
We could not observe the same improvement on CIFAR-100.
We postulate that the reliability of forgetting count on CIFAR-100 is not as good as it is on CIFAR-10, leading to the inconsistent results.
We leave the question on how to improve forgeting count based CAMixup on CIFAR-100 into future work.
[Figure]
Validation
[Figure]
Validation ECE (
[Figure]
[Figure]