Why Are Bootstrapped Deep Ensembles Not Better?
Abstract
Ensemble methods have consistently reached state of the art across predictive, uncertainty, and out-of-distribution robustness benchmarks.
One of the most popular ways to construct an ensemble is to independently train each model on a resampled (bootstrapped) version of the dataset.
Bootstrapping is popular in the literature on decision trees and frequentist statistics, with strong theoretical guarantees, but it is not used often in practice for deep neural networks.
We investigate a common hypothesis for bootstrap’s weak performance—percentage of unique points in the subsampled dataset—and ﬁnd that even when adjusting for it, bootstrap ensembles of deep neural networks yield no beneﬁt over simpler baselines.
This brings to question the role of data randomization as a source of uncertainty in deep learning.
[Figure]
Introduction
Let x n ∈ R D denote D -dimensional features and y n ∈ [1 , . .
, K ] denote the corresponding class label.
Assume we have a parametric model p ( y | x , θ ) for the conditional distribution, and a prior p ( θ ) over parameters.
The Bayesian posterior over parameters is given by p ( θ |{ x n , y n
[Equation]
Computing the posterior over θ is computationally challenging when p ( y n | x n , θ ) is parametrized using a deep neural network.
While computing the posterior is challenging, it is usually easy to compute the MAP solution, which corresponds to the mode of the posterior.
The MAP solution can be written as the minimizer of the following loss (negative log likelihood + negative log prior):
[Equation]
The MAP solution is computationally efﬁcient, but only gives a point estimate and not a distribution over parameters.
There are two popular ways to extend MAP to get multiple solutions:
• Multiple random seeds : Initialize at M different values for each ensemble member, possibly with different minibatch ordering, and repeat the minimization in equation 2.
This can produce M different solutions if the loss is non-convex.
• Bootstrap aggregation a.k.a.
Bagging [Breiman, 1996]: Train members on M different bootstrap samples of the original dataset.
Concretely, bagging ﬁrst samples a “bootstrap dataset” of N points with replacement from the original training set of N unique points, and then trains a neural network on this resampled dataset.
Conceptually, it is equivalent to drawing a set of instance weights w =
[ w 1 , . .
. , w N ] from a Multinomial distribution N times with mean parameter [1 /N, ..., 1 /N ] .
Note that (cid:80) n w n = N .
Training a neural network on the bootstrap dataset is equivalent to minimizing the weighted loss function:
[Equation]
where ˆ θ WMAP denotes the “weighted MAP” solution.
Note that setting w n = 1 ∀ n in equationrecovers the usual unweighted case in equation 2.
Bootstrap is theoretically well-motivated [Efron, 1992] and popular in the literature on both ensembles of decision trees (cf.
random forests [Breiman, 2001]) and in uncertainty estimation for calibrated conﬁdence intervals, but it has been found to hurt performance for ensembles of deep neural networks.
Lakshminarayanan et al.
[2017], Lee et al.
[2015] report that using the entire dataset for all the networks in the ensemble, which corresponds to a ﬁxed value of w =
, 1] , works better than training the individual networks on bootstrap versions of the original dataset.
Lakshminarayanan et al.
[2017] hypothesize that the poor empirical performance could be due to the fact that the bootstrap sample only contains 63.2% of unique points.
1 Earlier work on random forests such as
Breiman’s random forests [Breiman, 2001] also used bootstrap, while later work such as extremely randomized trees (ERT) [Geurts et al., 2006] found that bootstrap was not necessary when using other sources of randomization.
We investigate this hypothesis by implementing an extension of bootstrap that increases the percentage of unique points.
We then evaluate the ensembles’ accuracy and calibration on MNIST,
Fashion-MNIST, CIFAR-10, and CIFAR-100.
We ﬁnd that increasing the percentage does improve bootstrap’s performance.
Ultimately, it is not the sole factor: even when close to using 100% unique points, bootstrap does not provide any improvements over well-tuned baselines.
Experiments
We evaluate a Wide-Resnet [Zagoruyko and Komodakis, 2016] ensemble on CIFAR-100 and
CIFAR-10 [Krizhevsky, 2009], as well as a small CNN ensemble on MNIST and Fashion-MNIST
[Table]
[Figure]
[Xiao et al., 2017].
We evaluate accuracy and calibration of individual networks as well as ensembles.
All ensembles trained have 10 members, unless indicated otherwise.
2.1 Evaluation
We evaluate calibration using the expected calibration error (ECE)
[Naeini et al., 2015].
Expected Calibration Error (ECE).
To approximate the calibration error in expectation, ECE
Naeini et al.
[2015] discretizes the probability interval into a ﬁxed number of bins, and assigns each predicted probability to the bin that encompasses it.
The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (conﬁdence).
Intuitively, the accuracy estimates P ( Y =
y
| ˆ p = p ) , and the average conﬁdence is a setting of p .
ECE computes a weighted average of this error across bins:
[Equation]
where n b is the number of predictions in bin b , N is the total number of data points, and acc( b ) and conf( b ) are the accuracy and conﬁdence of bin b , respectively.
2.2 Supersampling the dataset does not lead to improved performance
We evaluate a Wide Resnet on data from bootstrap draws from a dataset as well as on the standard dataset.
As mentioned earlier, the bootstrap sampled data contains 63 .
2% unique points.
We evaluate reweighting by supersampling the training dataset for each ensemble member.
The bootstrap is modiﬁed to sample data with replacement until a given percentage of the training dataset has been sampled.
We sample up to 70% , 80% , 90% , and 99% of the training data.
In this case diversity is in the form of a reweighting of the dataset which comes out of the randomness in the datapoint selection.
Finally, we also report results for deep ensembles [Lakshminarayanan et al., 2017] which correspond to the case where each ensemble member is trained on the entire 100% of the dataset, and the source of diversity is random initialization and randomness in SGD [Fort et al., 2019].
In Figure 1 we show the distribution of ensemble member performance alongside ensemble performance in a box-and-whisker plot on CIFAR-10 and CIFAR-100.
The median ensemble member’s accuracy is marked by the orange line, with the minimum and maximum ensemble members marked by a dot or an end line depending on whether they are within 1.5 * the Inter Quartile Range (25th75th percentile range) of the data.
Models trained on bootstrap dataset (left most point corresponding to 0.632 marker) achieve a much smaller accuracy on their validation set than models trained on the full dataset, which is to be expected.
While the ensemble of the bootstrap sample members performs better than the single model, the gains from ensembling do not overcome the drop in accuracy due to dataset size, underperforming an ensemble trained on the full dataset.
As the number of unique datapoints in the training dataset increases, both the individual ensemble member performance and overall ensemble performance consistently improve peaking at the use of the full dataset.
In Figure 3 we investigate ensemble size and its impact on calibration error and accuracy for varying fraction of unique datapoints.
The trend is that ensembles trained with the bootstrap are dominated in accuracy by ensembles trained without it at every number of ensemble members from 2 up to 10.
We also observe that deep ensembles require fewer networks to achieve the same performance.
With calibration, the impact of full dataset training is not as dramatic.
While the general trend is also for ensembles trained with more unique training datapoints to outperform models trained with a heavier bootstrap, CIFAR 10 trained with > 4 ensemble members and Fashion-MNIST trained with >members show a negligible difference between ECE of ensembles trained on a varying number of unique samples.
We also notice an interesting phenomenon with ECE on CIFAR-100 merits further investigation: where deep ensembles are better calibrated with increasing members but bootstrap ensembles are better calibrated for M = 2 but their ECE becomes worse for higher M .
2.3 Further analysis of CIFAR-100 calibration results
On CIFAR-100 we observe a surprising calibration phenomena in Figure 3, where the model becomes increasingly overconﬁdent as the ensemble size increases for supersampled datasets, especially in the standard bootstrap setting where the fraction of unique datapoints is 63 .
To understand this surprising phenomenon, we evaluate calibration using metrics other than ECE.
Concretely, we evaluate calibration using proper scoring rules [Gneiting and Raftery, 2007] such as Brier score [Brier, 1950] and negative log-likelihood (also known as the cross-entropy loss).
We observe in Figure 4 that as the number of unique data points sampled and trained on by our ensemble members increases (with supersampling) we see improvements in the Cross Entropy loss and in the
Brier score.
Our proper scoring rules also improve as the number of ensemble members improve, in contrast with the ECE on CIFAR-100 in Figure 1.
This suggests that the phenomenon could be due to how ECE measures calibration.
In Figure 5, we report class-conditional ECE as well as reliability diagrams where we plot accuracy - conﬁdence, where the ideal value corresponds to 0.
We observe that class-conditional ECE, where we compute
ECE for each class independently and then average the values, behaves differently than ECE.
This suggests that the surprising phenomenon of ECE becoming worse with increasing ensemble size could have been caused due to the probabilities from each class being on separate scales.
The reliability diagrams show that bootstrapped ensembles may become more underconﬁdent compared to baseline ensembles.
2.4 Supersampling vs. Standard Sampling
The consequences of supersampling include changing the epoch length, with likely implications for the ideal learning rate schedule and other hyperparameters of the learning process.
We show that these are not drivers of our results by comparing to a sampling method that simply selects a random k% of the original training dataset.
Thist ’Standard Sampling’ method is evaluated in Figure 6.
2.5 Ablation: Sources of Randomness
One natural question about using the bootstrap as a source of variability for ensembling is to ask whether the diversity vale from the bootstrap is already captured by existing forms of variability.
There are three other major sources of variability - random initializaiton, stochastic gradient de- scent’s batch choices, and stochasticity in tensorﬂow operations used in training.
In Figure 7 we show the surprisingly high quality performance of ensemble models trained using the same initialization. [Figure]
Discussion
We investigate the effect of bootstrapping strategies on the accuracy and calibration of ensembles.
We ﬁnd that the number of unique points signiﬁcantly affects the accuracy of the ensemble as well as individual members.
Using the entire dataset typically performs best w.r.t.
calibration, and interestingly, bootstrap ensembles with fewer unique data points can still be well-calibrated.
For future work, we plan to look at calibration under dataset shift [Ovadia et al., 2019] and detecting out-ofdistribution inputs [Hendrycks and Gimpel, 2017, Lakshminarayanan et al., 2017].