P ITFALLS OF I N -D OMAIN U NCERTAINTY E STIMATION AND E NSEMBLING IN D EEP L EARNING
A BSTRACT
Uncertainty estimation and ensembling methods go hand-in-hand.
Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance.
At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation.
In this work, we focus on in-domain uncertainty for image classiﬁcation.
We explore the standards for its quantiﬁcation and point out pitfalls of existing metrics.
Avoiding these pitfalls, we perform a broad study of different ensembling techniques.
To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance.
[Table] [Table]
I NTRODUCTION
Deep neural networks (DNNs) have become one of the most popular families of machine learning models.
The predictive performance of DNNs for classiﬁcation is often measured in terms of accuracy.
However, DNNs have been shown to yield inaccurate and unreliable probability estimates, or predictive uncertainty (Guo et al., 2017).
This has brought considerable attention to the problem of uncertainty estimation with deep neural networks.
There are many faces to uncertainty estimation.
Different desirable uncertainty estimation properties of a model require different settings and metrics to capture them.
Out-of-domain uncertainty of the model is measured on the data that does not follow the same distribution as the training dataset (out-of-domain data).
Out-of-domain data can include images corrupted with rotations or blurring, adversarial attacks (Szegedy et al., 2013) or data points from a completely different dataset.
The model is expected to be resistant to data corruptions and to be more uncertain on out-of-domain data than on in-domain data.
On the contrary, in-domain uncertainty of the model is measured on data taken from the training data distribution, i.e. data from the same domain.
In this setting the model is expected to produce reliable probability estimates, e.g. the model shouldn’t be too overconﬁdent in its wrong predictions.
Pitfalls of metrics We show that many common metrics of in-domain uncertainty estimation (e.g. log-likelihood, Brier score, calibration metrics, etc.) are either not comparable across different models or fail to provide a reliable ranking.
We address some of the stated pitfalls and point out more reasonable evaluation schemes.
For instance, although temperature scaling is not a standard for ensembling techniques, it is a must for a fair evaluation.
With this in mind, the calibrated log-likelihood avoids most of the stated pitfalls and generally is a reasonable metric for in-domain uncertainty estimation task.
Pitfalls of ensembles Equipped with the proposed evaluation framework, we are revisiting the evaluation of ensembles of DNNs—one of the major tools for uncertainty estimation.
We introduce the deep ensemble equivalent (DEE) score that measures the number of independently trained models that, when ensembled, achieve the same performance as the ensembling technique of interest.
The
DEE score allows us to compare ensembling techniques across different datasets and architectures using a uniﬁed scale.
Our study shows that most of the popular ensembling techniques require averaging predictions across dozens of samples (members of an ensemble), yet are essentially equivalent to an ensemble of only few independently trained models.
Missing part of ensembling In our study, test-time data augmentation (TTA) turned out to be a surprisingly strong baseline for uncertainty estimation and a simple way to improve ensembles.
Despite being a popular technique in large-scale classiﬁcation, TTA seems to be overlooked in the community of uncertainty estimation and ensembling.
S COPE OF THE PAPER
We use standard benchmark problems of image classiﬁcation which comprise a common setting in research on learning ensembles of neural networks.
There are other relevant settings where the correctness of probability estimates can be a priority, and ensembling techniques are used to improve it.
These settings include, but are not limited to, regression, language modeling (Gal, 2016), image segmentation (Gustafsson et al., 2019), active learning (Settles, 2012) and reinforcement learning (Buckman et al., 2018; Chua et al., 2018).
We focus on in-domain uncertainty, as opposed to out-of-domain uncertainty.
Out-of-domain uncertainty includes detection of inputs that come from a completely different domain or have been corrupted by noise or adversarial attacks.
This setting has been thoroughly explored by (Ovadia et al., 2019).
We only consider methods that are trained on clean data with simple data augmentation.
Some other methods use out-of-domain data (Malinin & Gales, 2018) or more elaborate data augmentation, e.g. mixup (Zhang et al., 2017) or adversarial training (Lakshminarayanan et al., 2017) to improve accuracy, robustness and uncertainty.
We use conventional training procedures.
We use the stochastic gradient descent (SGD) and use batch normalization (Ioffe & Szegedy, 2015), both being the de-facto standards in modern deep learning.
We refrain from using more elaborate optimization techniques including works on superconvergence (Smith & Topin, 2019) and stochastic weight averaging (SWA) (Izmailov et al., 2018).
These techniques can be used to drastically accelerate training and to improve the predictive performance.
Thus, we do not comment on the training time of different ensembling methods since the use of these and other more efﬁcient training techniques would render such a comparison obsolete.
A number of related works study ways of approximating and accelerating prediction in ensembles.
The distillation mechanism allows to approximate the prediction of an ensemble by a single neural network (Hinton et al., 2015; Balan et al., 2015; Tran et al., 2020), whereas fast dropout (Wang & Manning, 2013) and deterministic variational inference (Wu et al., 2018) allow to approximate the predictive distribution of speciﬁc stochastic computation graphs.
We measure the raw power of ensembling techniques without these approximations.
All of the aforementioned alternative settings are orthogonal to the scope of this paper and are promising points of interest for further research.
P ITFALLS OF IN - DOMAIN UNCERTAINTY ESTIMATION
No single metric measures all the desirable properties of uncertainty estimates obtained by a model of interest.
Because of this, the community is using many different metrics in an attempt to capture the quality of uncertainty estimation, such as the Brier score (Brier, 1950), log-likelihood (Quinonero-Candela et al., 2005), metrics of calibration (Guo et al., 2017; Nixon et al., 2019), performance of misclassiﬁcation detection (Malinin & Gales, 2018), and threshold–accuracy curves
[Figure]
Log - li k e li hood ( s o li d )
C a li b r a t ed l og - li k e li hood ( da s hed )
[Figure] [Figure]
(Lakshminarayanan et al., 2017).
In the section we highlight the pitfalls of the aforementioned metrics, and demonstrate that these pitfalls can signiﬁcantly affect evaluation, changing the ranking of the methods.
Notation We consider a classiﬁcation problem with a dataset that consists of N training and n testing pairs ( x i , y ∗ i ) ∼ p ( x, y ) , where x i is an object and y ∗ i ∈ { 1 , . .
, C } is a discrete class label.
A probabilistic classiﬁer maps an object x i into a predictive distribution ˆ p ( y | x i ) .
The predictive distribution ˆ p ( y | x i ) of a deep neural network is typically deﬁned by the softmax function ˆ p ( y | x ) =
Softmax( z ( x ) /T ) , where z ( x ) is a vector of logits and T is a scalar parameter standing for the temperature of the predictive distribution.
This scalar parameter is usually set to T = 1 or is tuned on a validation set (Guo et al., 2017).
The maximum probability max c ˆ p ( y = c
| x i ) is called the conﬁdence of a classiﬁer ˆ p on an object x i .
I [ · ] denotes the indicator function throughout the text.
3. L OG - LIKELIHOOD AND B RIER SCORE
The average test log-likelihood LL = 1 n (cid:80) n i =1 log ˆ p ( y = y ∗ i | x i ) is a popular metric for measuring the quality of in-domain uncertainty of deep learning models.
It directly penalizes high probability scores assigned to incorrect labels and low probability scores assigned to the correct labels y ∗ i .
LL is sensitive to the softmax temperature T .
The temperature that has been implicitly learned during training can be far from optimal for the test data.
However, a nearly optimal temperature can be found post-hoc by maximizing the log-likelihood on validation data.
This approach is called temperature scaling or calibration (Guo et al., 2017).
Despite its simplicity, the temperature scaling results in a notable improvement in the LL.
While ensembling techniques tend to have better temperature than single models, the default choice of T = 1 is still suboptimal.
Comparing the LL with suboptimal temperatures—that is often the case in practice—can potentially produce an arbitrary ranking of different methods.
Comparison of the log-likelihood should only be performed at the optimal temperature.
Empirically, we demonstrate that the overall ordering of methods and also the best ensembling method according to the LL can vary depending on temperature T .
While this applies to most ensembling techniques (see Figure 10), this effect is most noticeable on experiments with data augmentation on ImageNet (Figure 1).
We introduce a new metric called the calibrated log-likelihood that is the log-likelihood at the optimal temperature.
The the calibrated log-likelihood considers pre-training and post-training calibration as a uniﬁed system, targeting to measure all models in equal condition of optimal temperature.
That allows to avoid measuring calibration error that can be eliminated by simple temperature scaling.
The metric signiﬁcantly affect results of comparison and changes ﬁnding.
For example in Figure 10 the difference between Bayesian (VI, K-FAC, SWAG, dropout) methods and conventional non-Bayesian network become much less pronounced, and in most cases making a conventional non-Bayesian network to match performance of Bayesian ones (VI, K-FAC, Dropout) on ResNet110, ResNet164, and WideResNet.
We show how to obtain an unbiased estimate of the calibrated log-likelihood without a held-out validation set in Section 3.5.
LL also demonstrates a high correlation with accuracy ( ρ > 0 . 86 ), that in case of calibrated LL becomes even stronger ( ρ > 0 . 95 ).
That suggests that while (calibrated) LL measures the uncertainty of the model, it still has a signiﬁcant dependence on the accuracy and vice versa.
A model with higher accuracy would likely have a higher log-likelihood.
See Figure 9 in Appendix C for more details.
Brier score BS = 1 n 1 C (cid:80) ni =1 (cid:80) Cc
{+=
 ( I [ y ∗ i =+}
c ] − ˆ p ( y = c
| x i )) 2 has also been known for a long time as a metric for veriﬁcation of predicted probabilities (Brier, 1950).
Similarly to the log-likelihood, the Brier score penalizes low probabilities assigned to correct predictions and high probabilities assigned to wrong ones.
It is also sensitive to the temperature of the softmax distribution and behaves similarly to the log-likelihood.
While these metrics are not strictly equivalent, they show a high empirical correlation for a wide range of models on CIFAR-10, CIFAR-100 and ImageNet datasets (see
Figure 8 in Appendix C ).
3. M ISCLASSIFICATION DETECTION
Detection of wrong predictions of the model, or misclassiﬁcations, is a popular downstream problem relevant to the problem of in-domain uncertainty estimation.
Since misclassiﬁcation detection is essentially a binary classiﬁcation problem, some papers measure its quality using conventional metrics for binary classiﬁcation such as AUC-ROC and AUC-PR (Malinin & Gales, 2018; Cui et al., 2019; Mo˙zejko et al., 2018).
These papers use an uncertainty criterion like conﬁdence or predictive entropy H [ˆ p ( y | x i )] as a prediction score.
While these metrics can be used to assess the misclassiﬁcation detection performance of a single model, they cannot be used to directly compare misclassiﬁcation performance across different models.
Correct and incorrect predictions are speciﬁc for every model, therefore, every model induces its own binary classiﬁcation problem.
The induced problems can differ signiﬁcantly, since different models produce different conﬁdences and misclassify different objects.
In other words, comparing such metrics implies a comparison of performance of classiﬁers that solve different classiﬁcation problems.
Such metrics are therefore incomparable.
cannot be directly compared between different models.
While comparing AUCs is incorrect in the setting of misclassiﬁcation detection, it is correct to compare these metrics in many out-of-domain data detection problems.
In that case, both objects and targets of the induced binary classiﬁcation problems remain the same for all models.
All outof-domain objects have a positive label and all in-domain objects have a negative label.
Note that this condition does not necessarily hold in the problem of detection of adversarial attacks.
Different models generally have different inputs after an adversarial attack, so such AUC-based metrics might still be ﬂawed.
3. C LASSIFICATION WITH REJECTION
Accuracy-conﬁdence curves are another way to measure the performance of misclassiﬁcation detection.
These curves measure the accuracy on the set of objects with conﬁdence max c ˆ p ( y = c
| x i ) above a certain threshold τ (Lakshminarayanan et al., 2017) and ignoring or rejecting the others.
The main problem with accuracy-conﬁdence curves is that they rely too much on calibration and the actual values of conﬁdence.
Models with different temperatures have different numbers of objects at each conﬁdence level which does not allow for a meaningful comparison.
To overcome this problem, one can swhich from thresholding by the conﬁdence level to thresholding by the number of rejected objects.
The corresponding curves are then less sensitive to temperature scaling and thus allow to compare the rejection ability in a more meaningful way.
Such curves have been known as accuracyrejection curves (Nadeem et al., 2009).
In order to obtain a scalar metric for easy comparisons, one can compute the area under this curve, resulting in AU-ARC (Nadeem et al., 2009).
3. C ALIBRATION METRICS
Informally speaking, a probabilistic classiﬁer is calibrated if any predicted class probability is equal to the true class probability according to the underlying data distribution (see (Vaicenavicius et al., 2019) for formal deﬁnitions).
Any deviation from the perfect calibration is called miscalibration.
For brevity, we will use ˆ p i,c to denote ˆ p ( y = c
| x i ) in the current section.
Expected calibration error (ECE) (Naeini et al., 2015) is a metric that estimates model miscalibration by binning the assigned probability scores and comparing them to average accuracies inside these bins.
Assuming B m denotes the m -th bin and M is overall number of bins, the ECE is deﬁned as follows:
[Equation]
where acc( B ) = | B | − 1 (cid:80) i ∈
B I [arg max c ˆ p i,c = y ∗ i ] and conf( B ) =
| B | − 1 (cid:80) i ∈
B ˆ p
i,y ∗ i .
A recent line of works on measuring calibration in deep learning (Vaicenavicius et al., 2019; Kumar et al., 2019; Nixon et al., 2019) outlines several problems of the ECE score.
Firstly, ECE is a biased estimate of the true calibration.
Secondly, ECE-like scores cannot be optimized directly since they are minimized by a model with constant uniform predictions, making the inﬁnite temperature
T = + ∞ its global optimum.
Thirdly, ECE only estimates miscalibration in terms of the maximum assigned probability whereas practical applications may require the full predicted probability vector to be calibrated.
Finally, biases of ECE on different models may not be equal, rendering the miscalibration estimates incompatible.
Similar concerns are also discussed by Ding et al.
Thresholded adaptive calibration error (TACE) was proposed as a step towards solving some of these problems (Nixon et al., 2019).
TACE disregards all predicted probabilities that are less than a certain threshold (hence thresholded ), chooses the bin locations adaptively so that each bin has the same number of objects (hence adaptive ), and estimates miscalibration of probabilties across all classes in the prediction (not just the top-1 predicted class as in ECE).
Assuming that B TA m denotes the m -th thresholded adaptive bin and M is the overall number of bins, TACE is deﬁned as follows:
[Equation]
where objs( B TA , c ) =
| B TA | − 1 (cid:80) i ∈
B TA I [ y ∗ i =
c ] and
[Equation]
Although TACE does solve several problems of ECE and is useful for measuring calibration of a speciﬁc model, it still cannot be used as a reliable criterion for comparing different models.
Theory suggests that it is still a biased estimate of true calibration with different bias for each model (Vaicenavicius et al., 2019).
In practice, we ﬁnd that TACE is sensitive to its two parameters, the number of bins and the threshold, and does not provide a consistent ranking of different models, as shown in Figure 2.
3. C ALIBRATED LOG - LIKELIHOOD AND TEST - TIME CROSS - VALIDATION
There are two common ways to perform temperature scaling using a validation set when training on datasets that only feature public training and test sets (e.g. CIFARs).
The public training set might be divided into a smaller training set and validation set, or the public test set can be split into test and validation parts (Guo et al., 2017; Nixon et al., 2019).
The problem with the ﬁrst method is that the resulting models cannot be directly compared with all the other models that have been trained on the full training set.
The second approach, however, provides an unbiased estimate of metrics such as log-likelihood and Brier score but introduces more variance.
In order to reduce the variance of the second approach, we perform a “test-time cross-validation”.
We randomly divide the test set into two equal parts, then compute metrics for each half of the test set using the temperature optimized on another half.
We repeat this procedure ﬁve times and average the results across different random partitions to reduce the variance of the computed metrics.
A STUDY OF ENSEMBLING & DEEP ENSEMBLE EQUIVALENT
Ensembles of deep neural networks have become a de-facto standard for uncertainty estimation and improving the quality of deep learning models (Hansen & Salamon, 1990; Krizhevsky et al., 2009; Lakshminarayanan et al., 2017).
There are two main directions of training ensembles of
DNNs: training stochastic computation graphs and obtaining separate snapshots of neural network parameters.
Methods based on the paradigm of stochastic computation graphs introduce some kind of random noise over the weights or activations of deep learning models.
When the model is trained, each sample of the noise corresponds to a member of the ensemble.
During test time, the predictions are averaged across the noise samples.
These methods include (test-time) data augmentation, dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016), variational inference (Blundell et al., 2015;
Kingma et al., 2015; Louizos & Welling, 2017), batch normalization (Ioffe & Szegedy, 2015; Teye et al., 2018; Atanov et al., 2019), Laplace approximation (Ritter et al., 2018) and many more.
Snapshot-based methods aim to obtain sets of weights for deep learning models and then to average the predictions across these weights.
The weights can be trained independently (e.g. deep ensembles (Lakshminarayanan et al., 2017)), collected on different stages of a training trajectory (e.g. snapshot ensembles (Huang et al., 2017) and fast geometric ensembles (Garipov et al., 2018)), or obtained from a sampling process (e.g. MCMC-based methods (Welling & Teh, 2011; Zhang et al., 2019)).
These two paradigms can be combined.
Some works suggest construction of ensembles of stochastic computation graphs (Tomczak et al., 2018), while others make use of the collected snapshots to construct a stochastic computation graph (Wang et al., 2018;
Maddox et al., 2019).
In this paper we consider the following ensembling techniques: deep ensembles (Lakshminarayanan et al., 2017), snapshot ensembles (SSE by Huang et al.
(2017)), fast geometric ensembling (FGE by
Garipov et al.
, SWA-Gaussian (SWAG by Maddox et al.
cyclical SGLD (cSGLD by Zhang et al.
, variational inference (VI by Blundell et al.
, K-FAC Laplace approximation (Ritter et al., 2018), dropout (Srivastava et al., 2014) and test-time data augmentation (Krizhevsky et al., 2009).
These techniques were chosen to cover a diverse set of approaches keeping their predictive performance in mind.
All these techniques can be summarized as distributions q m ( ω ) over parameters ω of computation graphs, where m stands for the technique.
During testing, one can average the predictions across parameters ω ∼ q m ( ω ) to approximate the predictive distribution
[Equation]
For example, a deep ensemble of S networks can be represented in this form as a mixture of S
Dirac’s deltas q DE ( ω ) = 1 S (cid:80) Ss =1 δ ( ω − ω s ) , centered at independently trained snapshots ω s .
Similarly, a Bayesian neural network with a fully-factorized Gaussian approximate posterior distribution over the weight matrices and convolutional kernels ω is represented as q VI ( ω ) =
N ( ω | µ, diag( σ 2 )) , µ and σ 2 being the optimal variational means and variances respectively.
If one considers data augmentation as a part of the computational graph, it can be parameterized by the coordinates of the random crop and the ﬂag for whether to ﬂip the image horizontally or not.
Sampling from the corresponding q aug ( ω ) would generate different ways to augment the data during inference.
However, as data augmentation is present by default during the training of all othe [Figure] mentioned ensembling techniques, it is suitable to study it in combination with these methods and not as a separate ensembling technique.
We perform such an evaluation in Section 4.3.
Typically, the approximation (equation 3) requires K independent forward passes through a neural network, making the test-time budget directly comparable across all methods.
4. D EEP ENSEMBLE EQUIVALENT
Most ensembling techniques under consideration are either bounded to a single mode, or provide positively correlated samples.
Deep ensemble, on the other hand, is a simple technique that provides independent samples from different modes of the loss landscape, which, intuitively, should result in a better ensemble.
Therefore deep ensembles can be considered a strong baseline for the performance of other ensembling techniques given a ﬁxed test-time computation budget.
Comparing the performance of ensembling techniques is, however, a challenging problem.
Different models on different datasets achieve different values of metrics; their dependence on the number of samples is non-trivial, and varies depending on a speciﬁc model and dataset.
Values of the metrics are thus lacking in interpretability as the gain in performance has to be compared against a modeland dataset-speciﬁc baseline.
Aiming to introduce perspective and interpretability in our study, we introduce the deep ensemble equivalent score that employs deep ensembles to measure the performance of other ensembling techniques.
Speciﬁcally, the deep ensemble equivalent score answers the following question:
What size of deep ensemble yields the same performance as a particular ensembling method?
Following the insights from the previous sections, we base the deep ensemble equivalent on the calibrated log-likelihood (CLL).
Formally speaking, we deﬁne the deep ensemble equivalent (DEE) for an ensembling method m and its upper and lower bounds as follows:
[Equation]
[Figure]
where CLL mean / std m ( l ) are the mean and the standard deviation of the calibrated log-likelihood achieved by an ensembling method m with l samples.
We compute CLL meanDE ( l ) and CLL stdDE ( l ) for natural numbers l ∈ N > 0 and use linear interpolation to deﬁne them for real values l ≥ 1 .
In the following plots we report DEE m ( k ) for different methods m with different numbers of samples k , and shade the area between the respective lower and upper bounds DEE lower m ( k ) and DEE upper m ( k ) .
4. E XPERIMENTS
We compute the deep ensemble equivalent (DEE) of various ensembling techniques for four popular deep architectures: VGG16 (Simonyan & Zisserman, 2014), PreResNet110/164 (He et al., 2016), and WideResNet28x10 (Zagoruyko & Komodakis, 2016) on CIFAR-10/100 datasets (Krizhevsky et al., 2009), and ResNet50 (He et al., 2016) on ImageNet dataset (Russakovsky et al., 2015).
We use PyTorch (Paszke et al., 2017) for implementation of these models, building upon available public implementations.
Our implementation closely matches the quality of methods that has been reported in original works.
Technical details on training, hyperparameters and implementations can be found in Appendix B.
The source code and all computed metrics are available on GitHub 1 .
As one can see on Figure 3, ensembling methods clearly fall into three categories.
SSE and cSGLD outperform all other techniques except deep ensembles and enjoy a near-linear scaling of DEE with the number of samples on CIFAR datasets.
The investigation of weight-space trajectories of cSGLD and SSE (Huang et al., 2017; Zhang et al., 2019) suggests that these methods can efﬁciently explore different modes of the loss landscape.
In terms of the deep ensemble equivalent, these methods do not saturate unlike other methods that are bound to a single mode.
We found SSE to still saturate on ImageNet.
This is likely due to suboptimal hyperparameters of the cyclic learning rate schedule.
More verbose results are presented in Figures 11–13 and in Table 5 and Table 8 in Appendix C.
In our experiments SSE typically outperforms cSGLD.
This is mostly due to the fact that SSE has a much larger training budget.
The cycle lengths and learning rates of SSE and cSGLD are comparable, however, SSE collects one snapshot per cycle while cSGLD collects three snapshots.
This makes samples from SSE less correlated with each other while increasing the training budget threefold.
Both SSE and cSGLD can be adjusted to obtain a different trade-off between the training budget and the DEE-to-samples ratio.
We reused the schedules provided in the original papers (Huang et al., 2017; Zhang et al., 2019).
[Figure] Being more “local” methods, FGE and SWAG perform worse than SSE and cSGLD, but still signiﬁcantly outperform “single-snapshot” methods like dropout, K-FAC Laplace approximation and variational inference.
We hypothesize that by covering a single mode with a set of snapshots, FGE and SWAG provide a better ﬁt for the local geometry than models trained as stochastic computation graphs.
This implies that the performance of FGE and SWAG should be achievable by singlesnapshot methods.
However, one might need more elaborate posterior approximations and better inference techniques in order to match the performance of FGE and SWAG by training a stochastic computation graph end-to-end (as opposed to SWAG that constructs a stochastic computation graph post-hoc).
The deep ensemble equivalent curves allow us to notice the common behaviour of different methods, e.g. the relation between deep ensembles, snapshot methods, advanced local methods and singlesnapshot local methods.
They also allow us to notice inconsistencies that may indicate a suboptimal choice of hyperparameters.
For example, we ﬁnd that SSE on ImageNet quickly saturates, unlike
SSE on CIFAR datasets (Figure 3).
This may indicate that the hyperparameters used on ImageNet are not good enough for efﬁcient coverage of different modes of the loss landscape.
We also ﬁnd that SSE on WideResNet on CIFAR-10 achieves a DEE score of 100 on approx.
70 samples (Figure 12).
This may indicate that the members of the deep ensemble for this dataset-architecture pair are underﬁtted and may beneﬁt from longer training or a different learning rate schedule.
Such inconsistencies might be more difﬁcult to spot using plain calibrated log-likelihood plots.
4. T EST - TIME D ATA A UGMENTATION I MPROVES E NSEMBLES FOR F REE
Data augmentation is a time-honored technique that is widely used in deep learning, and is a crucial component for training modern DNNs.
Test-time data augmentation has been used for a long time to improve the performance of convolutional networks.
For example, multi-crop evaluation has been a standard procedure for the ImageNet challenge (Simonyan & Zisserman, 2014; Szegedy et al., 2015;
He et al., 2016).
It is, however, often overlooked in the literature on ensembling techniques in deep learning.
In this section, we study the effect of test-time data augmentation on the aforementioned ensembling techniques.
To keep the test-time computation budget the same, we sample one random augmentation for each member of an ensemble.
Figure 5 reports the calibrated log-likelihood on combination of ensembles and test-time data augmentation for ImageNet.
Other metrics and results on CIFAR-10/100 datasets are reported in Appendix C. We have used the standard data augmentation: random horizontal ﬂips and random padded crops for CIFAR-10/100 datasets, and random horizontal ﬂips and random resized crops for ImageNet
Test-time data augmentation (Figure 4) consistently improves most ensembling methods, especially on ImageNet, where we see a clear improvement across all methods (Figure 5 and Table 7).
The performance gain for powerful ensembles (deep ensembles, SSE and cSGLD) on CIFAR datasets is not as dramatic (Figures 14–15 and Table 4).
This is likely due to the fact that CIFAR images are small, making data augmentation limited, whereas images from ImageNet allow for a large number of diverse samples of augmented images.
On the other hand, while the performance of “single-snapshot” methods (e.g. variational inference, K-FAC Laplace and dropout) is improved signiﬁcantly, they perform approximately as good as an augmented version of a single model across all datasets.
Interestingly, test-time data augmentation on ImageNet improves accuracy but decreases the uncalibrated log-likelihood of deep ensembles (Table 7 in Appendix C).
Test-time data augmentation breaks the nearly optimal temperature of deep ensembles and requires temperature scaling to reveal the actual performance of the method, as discussed in Section 3.1.
The experiment demonstrates that ensembles may be highly miscalibrated by default while still providing superior predictive performance after calibration.
We would like to note that test-time data augmentation does not always break the calibration of an ensemble, and, on the contrary, test-time data augmentation often improves the calibration of an ensemble.
In our experiments, decalibration was caused by the extreme magnitude of a random crop, that is conventionally used for ImageNet augmentation.
Using less extreme magnitude of the random crop ﬁxes decalibration , that makes test-time data augmentation a more practical method that provides out-of-the-box calibration.
Although, as we demonstrated earlier, there is no guarantee that any ensemble is calibrated out-of-the-box.
If we are willing to apply post-hoc calibration, the ﬁnal performance can be much better with more severe augmentations.
D ISCUSSION & C ONCLUSION
We have explored the ﬁeld of in-domain uncertainty estimation and performed an extensive evaluation of modern ensembling techniques.
Our main ﬁndings can be summarized as follows:
• Temperature scaling is a must even for ensembles.
While ensembles generally have better calibration out-of-the-box, they are not calibrated perfectly and can beneﬁt from the procedure.
A comparison of log-likelihoods of different ensembling methods without temperature scaling might not provide a fair ranking, especially if some models happen to be miscalibrated.
• Many common metrics for measuring in-domain uncertainty are either unreliable (ECE and analogues) or cannot be used to compare different methods (AUC-ROC, AUC-PR for misclassiﬁcation detection; accuracy-conﬁdence curves).
In order to perform a fair comparison of different methods, one needs to be cautious of these pitfalls.
• Many popular ensembling techniques require dozens of samples for test-time averaging, yet are essentially equivalent to a handful of independently trained models.
Deep ensembles dominate other methods given a ﬁxed test-time budget.
The results indicate, in particular, that exploration of different modes in the loss landscape is crucial for good predictive performance.
• Methods that are stuck in a single mode are unable to compete with methods that are designed to explore different modes of the loss landscape.
Would more elaborate posterior approximations and better inference techniques shorten this gap?
• Test-time data augmentation is a surprisingly strong baseline for in-domain uncertainty estimation.
It can signiﬁcantly improve other methods without increasing training time or model size since data augmentation is usually already present during training.
Our takeaways are aligned with the take-home messages of Ovadia et al.
(2019) that relate to indomain uncertainty estimation.
We also observe a stable ordering of different methods in our experiments, and observe that deep ensembles with few members outperform methods based on stochastic computation graphs.
A large number of unreliable metrics inhibits a fair comparison of different methods.
Because of this, we urge the community to aim for more reliable benchmarks in the numerous setups of uncertainty estimation.
A CKNOWLEDGMENTS
Dmitry Vetrov and Dmitry Molchanov were supported by the Russian Science Foundation grant no. 19-71-30020.
This research was supported in part through computational resources of HPC facilities at NRU HSE.
A I S ” EFFICIENT ” TRAINING OF ENSEMBLES EFFICIENT AT ALL ?
Yes!
If you care most about training time efﬁciency.
All snapshot based methods (SSE, cSGLD and FGE) on average (Figure 6) tend to achieve the same performance as deep ensembles using 2 - 5 × less training epochs on CIFAR datasets.
The gain comes at a cost of inference efﬁciency and memory consumption. ”
Efﬁcient” snapshotbased methods require to store much more samples of weights compared to deep ensembles making inference signiﬁcantly more expensive (up to × 25 ) given the same predictive performance.
You need to get lucky with hyperparameter choice.
While on average ”efﬁcient” snapshot-based methods require less training resources, they might be completely inefﬁcient if your hyperparameter choice is sub-optimal (see Figure 7).
Such hyperparameters as the maximum learning rate, the length of learning rate decay cycle, the snapshot saving schedule can all impact the performance signiﬁcantly.
This analysis assumes that we use only conventional training procedure.
Many models can most likely be trained, stored and executed much more efﬁciently with methods like super-convergence, stochastic weight averaging, compression, distillation and others.
These methods are out of the scope of the paper but are interesting topics for future research.
The current choice of hyperparameters may also not be optimal.
We reuse the hyperparameters used in the original papers.
B E XPERIMENTAL DETAILS
Implementations of deep ensembles, SWAG, FGE and K-FAC Laplace are heavily based on the original PyTorch implementations of stochastic weight averaging (SWA) 2 and SWAG 3 .
Implementations of cyclical MCMC and snapshot ensembles are based on the original implementation of cyclical MCMC 4 .
We hypothesize that the optimal hyperparameters of ensembling methods may vary widely depending on the computational budget and the number of samples in the ensemble.
Searching for the optimal values for each conﬁguration is outside the scope of this paper so we stick to the originally proposed hyperparameters whenever possible.
Implied probabilistic model Conventional neural networks for classiﬁcation are usually trained using the average cross-entropy loss function with weight decay regularization hidden inside an optimizer in a deep learning framework like PyTorch.
The underlying optimization problem can be written as follows:
[Equation]
where { ( x i , y ∗ i ) }
Ni =1 is the training dataset of N objects x i with corresponding labels y ∗
i , λ is the weight decay scale and ˆ p ( j | x i , w ) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object x i .
The cross-entropy loss deﬁnes a likelihood function p ( y ∗ | x, w ) and weight decay regularization, or
L 2 regularization corresponding to a certain Gaussian prior distribution p ( w ) .
The whole optimization objective then corresponds to the maximum a posteriori inference in the following probabilistic model:
[Equation]
In order to make the results comparable across all ensembling techniques, we used the same probabilistic model for all methods, choosing ﬁxed weight decay parameters for each architecture.
We used the softmax-based likelihood for all models.
We also use the fully-factorized zero-mean Gaussian prior distribution with variances σ 2 =
( Nλ ) − 1 where the number of objects N and the weight decay scale λ are dictated by the particular datasets and neural architectures as deﬁned in the following paragraph.
Conventional networks To train a single network on CIFAR-10/100, we used SGD with batch size of 128 , momentum 0 .
9 and model-speciﬁc parameters, i.e. the initial learning rate ( lr init ), the weight decay coefﬁcient ( wd ), and the number of optimization epochs ( epoch ).
Speciﬁc hyperparameters are shown in Table 1.
The models were trained with a uniﬁed learning rate scheduler that is shown in equation 10.
All models have been trained using data augmentation that consists of horizontal ﬂips and a random crop of 32 pixels with a padding of 4 pixels 5 .
The standard data normalization has also been applied.
Weight decays, initial learning rates, and the learning rate scheduler were taken from (Garipov et al., 2018) paper.
Compared with hyperparameters of (Garipov et al., 2018), we increased the number of optimization epochs since we found that all models were underﬁtted.
While the original WideResNet28x10 network includes a number of dropout layers with p = 0 . 3 and is trained for 200 epochs, we ﬁnd that the WideResNet28x10 underﬁts in this setting and requires a longer training.
Therefore, we used p = 0 which reduces training time while bearing
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
otherwise (
[Table]
On ImageNet dataset we used ResNet50 with default hyperparameters taken from PyTorch examples.
Speciﬁcally, we used SGD with momentum 0 .
9 , batch size of 256 , initial learning rate 0 .
1 , weight decay 1 e − 4 .
Training included data augmentation 7 (scaling, random crops of size 224 × 224 , horizontal ﬂips), normalization and learning rate scheduler lr = lr init · 0 .
1 epoch // 30 where // denotes integer division.
We only deviated from standard parameters by increasing the number of training epochs from 90 to 130 .
Or models achieve top-1 error of 23 .
81 ± 0 .
15 that closely matches the accuracy of ResNet50 provided by PyTorch which is 23 .
Training of one model on a single
NVIDIA Tesla V100 GPU takes approximately 5.5 days.
Deep ensembles Deep ensembles (Lakshminarayanan et al., 2017) average the predictions across networks trained independently starting from different initializations.
To obtain a deep ensemble we repeat the described procedure of training standard networks 128 times for all architectures on
CIFAR-10 and CIFAR-100 datasets (1024 networks over all) and 50 times for ImageNet dataset.
Every member of deep ensembles was trained with exactly the same hyperparameters as conventional models of the same architecture.
Dropout Binary dropout (or MC dropout) (Srivastava et al., 2014; Gal & Ghahramani, 2016) is one of the most widely known ensembling techniques.
It involves putting a multiplicative Bernoulli noise with a parameter p over the activations of either a fully connected layer or a convolutional layer, averaging predictions of the network w.r.t.
noise at test-time.
Dropout layers were applied to VGG and WideResNet networks on CIFAR-10 and CIFAR-100 datasets.
Dropout for VGG was applied to fully connected layers with p = 0 .
Two dropout layers were applied: one before the ﬁrst fully connected layer and one before the second one.
While the original version of VGG for
CIFARs (Zagoruyko, 2015) exploits more dropout layers, we observed that any additional dropout layer deteriorates the performance of the model in ether deterministic or stochastic mode.
Dropout for WideResNet was applied in accordance with the original paper (Zagoruyko & Komodakis, 2016) with p = 0 .
Dropout usually increases the time needed to achieve convergence.
Because of this,
WideResNet networks with dropout were trained for 400 epochs instead of 300 epochs for deterministic case, and VGG networks have always been trained with dropout.
All the other hyperparameters were the same as in the case of conventional models.
Variational Inference Variational Inference (VI) approximates the true posterior distribution over weights p ( w | Data ) with a tractable variational approximation q θ ( w ) by maximizing a so-called variational lower bound L (eq. 11) w.r.t.
the parameters of variational approximation θ .
We used a fully-factorized Gaussian approximation q ( w ) and Gaussian prior distribution p ( w ) .
[Equation]
[Equation]
w | 0 , diag ( σ 2 p )) , where σ 2 p =
( N · wd) −(12)
[Table]
In the case of such a prior, the probabilistic model remains consistent with the conventional training which corresponds to MAP inference in the same probabilistic model.
We used variational inference for both convolutional and fully-connected layers where variances of the weights were parameterized by log σ .
For fully-connected layers we applied the local reparameterization trick (LRT, (Kingma et al., 2015)).
While variational inference provides a theoretically grounded way to approximate the true posterior, it tends to underﬁt deep learning models in practice (Kingma et al., 2015).
The following tricks are applied to deal with it: pre-training (Molchanov et al., 2017) or equivalently annealing of β
(Sønderby et al., 2016), and scaling β down (Kingma et al., 2015; Ullrich et al., 2017).
During pre-training we initialize µ with a snapshot of weights of a pre-trained conventional model, and initialize log σ with a model-speciﬁc constant log σ init .
The KL-divergence – except for the term corresponding to the weight decay – is scaled with a model-speciﬁc parameter β .
The weight decay term is implemented as a part of the optimizer.
We used a fact that the KL-divergence between two Gaussian distributions can be rewritten as two terms, one of which is equivalent to the weight decay regularization.
On CIFAR-10 and CIFAR-100 we used β equal to 1e-4 for VGG, ResNet100 and ResNet164 networks, and β equal to 1e-5 for WideResNet.
Log-variance log σ init was initialized with − 5 for all models.
Parameters µ were optimized with SGD in the same manner as in the case of conventional networks except that the initial learning rate lr init was set to 1e-3.
We used a separate Adam optimizer with a constant learning rate of 1e-3 to optimize log-variances of the weights log σ .
Pretraining was done for 300 epochs, and after that the remaining part of training was done forepochs.
On ImageNet we used β
= 1e-3, lr init = 0 .
01 , log σ init = − 6 , and trained the model for 45 epochs after pre-training.
K-FAC Laplace
The Laplace approximation uses the curvature information of the appropriately scaled loss function to construct a Gaussian approximation to the posterior distribution.
Ideally, one would use the Hessian of the loss function as a covariance matrix and use the maximum a posteriori estimate w MAP as a mean of the Gaussian approximation:
[Equation]
In order to keep the method scalable, we use the Fisher Information Matrix as an approximation to the true Hessian (Martens & Grosse, 2015).
For K-FAC Laplace, we use the whole dataset to construct an approximation to the empirical Fisher Information Matrix, and use the π correction to reduce the bias (Ritter et al., 2018; Martens & Grosse, 2015).
Following (Ritter et al., 2018), we ﬁnd the optimal noise scale for K-FAC Laplace on a held-out validation set by averaging across ﬁve random initializations.
We then reuse this scale for networks trained without a hold-out validation set.
We report the optimal values of scales in Table 2.
Note that the optimal scale is different depending on whether we use test-time data augmentation or not.
Since the data augmentation also introduces some amount of additional noise, the optimal noise scale for K-FAC Laplace with data augmentation is lower.
Snapshot ensembles Snapshot ensembles (SSE) (Huang et al., 2017) is a simple example of an array of methods which collect samples from a training trajectory of a network in weight space to construct an ensemble.
Samples are collected in a cyclical manner: during each cycle the learning rate goes from a large value to near-zero and snapshot of weights of the network is taken at the end of the cycle.
SSE uses SGD with a cosine learning schedule deﬁned as follows:
[Equation]
where α 0 is the initial learning rate, T is the total number of training iterations and M is the number of cycles.
For all datasets and models hyperparameters from the original SSE paper are reused.
For CIFAR10/100 length of the cycle is 40 epochs, maximum learning rate is 0.2, batch size is 64.
On ResNet and ImageNet length of the cycle is 45 epochs, maximum learning rate is 0.1, batch size is 256.
Cyclical SGLD Cyclical Stochastic Gradient Langevin Dynamics (cSGLD)
(Zhang et al., 2019) is a state-of-the-art ensembling method for deep neural networks pertaining to stochastic Markov
Chain Monte Carlo family of methods.
It bears similarity to SSE, e.g. it employs SGD with a learning rate schedule described with the equation 16 and training is cyclical in the same manner.
Its main differences from SSE are the introduction of gradient noise and the capturing of several snapshots per cycle, both of which can aid in sampling from posterior distribution over neural network weights efﬁciently.
Some parameters from the original paper are reused: length of cycle is 50 epochs, maximum learning rate is 0.5, batch size is 64.
Number of epochs with gradient noise per cycle is 3 epochs.
This was found to yield much higher predictive performance and better uncertainty estimation compared to the original paper
Finally, the results of cyclical Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), (Zhang et al., 2019) which reportedly has marginally better performance compared with cyclical SGLD, could not be reproduced with any value of SGD momentum term.
Because of this, we only include cyclical SGLD in our benchmark.
FGE Fast Geometric Ensembling (FGE) is an ensembling method that is similar to SSE in that it collects weight samples from a training trajectory to construct an ensemble.
Its main differences from SSE are pretraining, short cycle length and a piecewise-linear learning rate schedule:
[Equation]
Hyperparameters of the original implementtion of FGE are reused.
Model pretraining is done with
SGD for 160 epochs according to the standard learning rate schedule described in equation 10 with maximum learning rates from Table 1.
After that, a desired number of FGE cycles is done with one snapshot per cycle collected.
For VGG the learning rate is changed with parameters α 1 = 1 e − 2 , α 2 = 5 e − 4 in a cycle with cycle length of 2 epochs.
For other networks the learning rate is changed with parameters α 1 = 5 e − 2 , α 2 = 5 e − 4 with cycle length of 4 epochs.
Batch size is 128.
SWAG SWA-Gaussian (SWAG)
(Maddox et al., 2019) is an ensembling method based on ﬁtting a Gaussian distribution to model weights on the SGD training trajectory and sampling from this distribution to construct an ensemble.
Like FGE, SWAG has a pretraining stage which is done according to the standard learning rate schedule described in equation 10 with maximum learning rates from Table 1.
After that, training continues with a constant learning rate of 1e-2 for all models except for PreResNet110 and PreResNet164 on CIFAR-100 where it continues with a constant learning rate of 5e-2 in accordance with the original paper.
Rank of the empirical covariance matrix which is used for estimation of Gaussian distribution parameters is set to be 20.
C A DDITIONAL EXPERIMENTAL RESULTS
[Figure]
[Figure] [Figure]
[Figure] [Figure]
SSE outperforms deep ensembles on CIFAR-10 on the WideResNet architecture.
It possibly indicates that the cosine learning rate schedule and longer training of SSE are more suitable for this architecture than the piecewise-linear learning rate schedule and the number of epochs used in deep ensembles.
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
WideResNet
[Table]
DE
[Table] [Table]
[Table] [Table]
c a li b r a t ed l og - li k e li hood
WideResNet
CIFAR
[Figure]