The Best Deep Ensembles Sacrifice Predictive Diversity
Abstract
Ensembling remains a hugely popular method for increasing the performance of a given class of models.
In the case of deep learning, the benefits of ensembling are often attributed to the diverse predictions of the individual ensemble members.
Here we investigate a tradeoff between diversity and individual model performance, and find that—surprisingly—encouraging diversity during training almost always yields worse ensembles.
We show that this tradeoff arises from the Jensen gap between the single model and ensemble losses, and show that Jensen gap is a natural measure of diversity for both the mean squared error and cross entropy loss functions.
Our results suggest that to reduce the ensemble error, we should move away from efforts to increase predictive diversity and instead we should construct ensembles from less diverse (but more accurate) component models
Introduction
Ensembling, or combining the predictions of multiple models, yields lower error than using a single model in isolation [4, 7, 14, 22, 32].
In many instances, this phenomenon is a natural consequence of
Jensen’s inequality (though this connection is not often made in the literature).
If error is measured by a strongly convex loss function ℓ (e.g. cross entropy, mean squared error, etc), and predictions from models f 1 , . .
. , f M are combined through averaging, we have:
[Equation]
The degree to which ensembling helps depends on the looseness of the bound in Eq.
On one end of the spectrum, if all models make the same errors (i.e. f 1 ( x ) = f 2 ( x ) = . . .
=
f M ( x ) ∀ x ), then the ensemble won’t perform any better than the single models (i.e. Eq.
(1) holds with equality).
On the other end of the spectrum, if the component models make uncorrelated errors, the difference between ensembles and their average component model will be quite large [4, 21].
In this sense, the
Jensen gap between terms in Eq.
(1) quantifies a notion of “predictive diversity,” which is touted as a necessary criterion for effective ensembles [7, 8, 22, 23].
All else being equal, training mechanisms which increase predictive diversity (widening the gap in Eq. 1) will improve ensemble performance.
Of course, any training mechanism that could widen the Jensen gap will also influence the performance of component models (i.e. the right side of Eq.
Ultimately, both terms together determine ensemble performance (Fig. 1).
Moreover, it is unlikely that there is some “magic” training procedure that simultaneously increases predictive diversity and lowers average single model loss, or even leaves single model loss unchanged (Fig. 1, left).
Therefore, to improve ensemble performance, increases in the Jensen gap (predictive diversity) must outweigh any incurred increase in single model loss (as depicted in Fig.
1, center).
I Can’t Believe Not Better Workshop NeurIPS 2022.
Jensen gap (pred. diversity)
Pred. diversity
Pred. diversity helpful
Pred. diversity harmful
A v g .
s i n g l e
m o d e l l o ss
Baseline ens.
performance
Avg.
sing.
model
Increasing pred.
diversity
A v g .
s i n g l e
m o d e l l o ss
A v g .
s i n g l e
m o d e l l o ss
Jensen gap (pred. diversity)
[Figure] [Figure] diversity) Within the context of modern deep ensembles, this paper offers evidence that, surprisingly, increasing the Jensen gap (i.e. predictive diversity) almost always yields worse ensemble performance, as any
Jensen gap increases are outweighted by increased component model loss (as depicted in Fig.
1, right).
This finding implies that—counterintuitively— the path towards better deep ensembles involves less predictive diversity, not more .
In particular, we make the following contributions:
We show that standard ensemble training is equivalent to empirical risk minimization with a regularization term that minimizes the Jensen gap in Eq.
i e. a term that penalizes predictive diversity .
Surprisingly, reducing this diversity penalty to increase the Jensen gap significantly harms ensemble performance.
Even more surprisingly, upweighting this diversity penalty improves performance.
Analyzing these results, we find that encouraging predictive diversity (by reducing the penalty) incurs a substantial increase in component model loss, resulting in net worse ensemble performance.
Across a variety of model architectures, we find the the best deep ensembles are often the least diverse, instead relying on strong component models.
These results hold for standard as well as heterogeneous deep ensembles (combinations of multiple architectures/training procedures).
In the full version of this paper, we will extend these results to other training procedures that explicitly encourage diversity [e.g. 11, 23, 25, 27–29, 35, 36, 39, 41].
Together, these results suggest that the best strategy to improve deep ensembles is reducing the error of component models, not increasing predictive diversity.
Quantifying Predictive Diversity Via Jensen Gap
Here we examine the Jensen gap (Eq. 1) in further depth as a notion of diversity for two common loss functions.
Though previous works propose alternative metrics for diversity [e.g. 20, 21, 29], the
Jensen gap is exactly the performance advantage of an ensemble over the average single component model, and thus definitionally demonstrates that predictive diversity is good for ensembles.
Moreover we find correspondences between the Jensen gap and other diversity metrics, suggesting that these metrics are interchangeable to a large extent.
Example 1: Mean squared error (MSE) loss.
If we take ℓ to be the standard mean squared error loss (or equivalently, the Brier Score [6]), the Jensen gap in Eq.
(1) captures variance across model predictions.
Given M models f 1 , . . .
, f M :
[Equation]
where Var[ f i ( x )] is the sample variance (see Appx.
A.1 for a derivation).
Thus the Jensen gap coincides with a natural notion of diversity used in previous work [e.g. 18, 29].
Example 2: Cross entropy (CE) loss.
If we instead take ℓ to be the cross entropy loss, the Jensen gap does not correspond to a term that is as immediately recognizable:
[Equation]
where f ( y ) i is the probability assigned the correct class by model f i .
Nevertheless, the right hand side of Eq.
(3) can be interpreted as an information theoretic quantification of ensemble diversity (see
Appx.
A.2), which—as we demonstrate in Appx.
D—correlates well with other popular diversity metrics like average pairwise model correlation [e.g. 7, 8, 21].
Standard Ensemble Training Penalizes Predictive Diversity
A standard deep ensemble consists of M independently trained models.
With the Jensen gap notion of predictive diversity, we immediately notice something troubling about this standard setup.
Independently optimizing f 1 , . . .
f M is equivalent to optimizing L ens.
≜ 1 M (cid:80) Mi
=1 E p ( x ,y )
[ ℓ ( f i ( x ) , y )] .
Importantly, this form differs from the risk used to evaluate the ensemble, E p ( x ,y )
[ ℓ ( 1 M (cid:80) Mi
=
f i ( x ) , y )] .
The difference between the training and evaluation risk is exactly the Jensen gap.
Example 3: Training a deep ensemble with MSE penalizes variance.
In particular, if our goal is to apply deep ensembles to a regression task, our training objective can be written as:
[Equation]
where ¯ f ( x ) ≜ 1 M (cid:80) Mi =1 f i ( x ) is the ensemble prediction.
In other words, standard ensemble training amounts to penalizing the variance between ensemble members.
More generally, we can write:
[Equation]
Eq.
(5) frames standard ensemble training as regularized empirical risk minimization.
The first term in Eq.
(5) measures the risk of the ensemble, which is ultimately what we care about minimizing.
The second term is the Jensen gap, which is always greater than zero and thus can be interpreted as a regularizer.
Surprisingly then, although diversity (a large Jensen gap) is beneficial for ensemble performance, the standard ensemble training objective favors a small Jensen gap.
Encouraging Predictive Diversity During Training Hurts Performance
Hypothesis: downweighting the predictive diversity penalty should improve ensembles.
Eq.
(5) shows that standard ensemble training appears to be regularizing against diversity inherent in our choice of ensemble risk.
Eq.
(5) further suggests a straightforward way to increase ensemble diversity: simply reduce the strength of this regularization.
We would hope the corresponding increase in the
Jensen gap would then contribute directly to an improvement in ensemble performance.
Experimental setup.
We consider multiclass classification problems with inputs x ∈ R D and targets y ∈ [1 , . . .
, C ] .
To measure the effect of downweighting the predictive diversity penalty, we train ensembles of M = 4 networks on the CIFAR10 training dataset [19] with an additional hyperparameter γ , which controls for the strength of the Jensen gap diversity regularizer:
[Equation]
If we set γ = 1 , then Eq.
(6) corresponds to the unmodified ensemble training (Eq. 3) objective.
If we instead set γ < 1 , we expect that the resulting ensemble will have a larger Jensen gap (i.e. more predictive diversity) than a standard ensemble, which ideally should result in improved predictive performance (Fig. 1, middle).
Conversely, we would expect that setting γ > 1 would further hurt ensemble performance, as it would yield a smaller Jensen gap and less predictive diversity.
[Figure]
ResNet 18 Ensemble (M = 4)
[Figure]
DenseNet 121 Ensemble (M = 4)
Weighted Training:
L γ − ens
[Figure]
We evaluate ensembles of two different network architectures—ResNet18 [15] and DenseNet [17], for γ ∈ [0 , 5] .
We initialize each ensemble’s component networks using different random seeds.
During training iterations, we optimize a standard minibatched approximation of Eq.
Because
Eq.
(6) trains ensemble members jointly, each component network sees the same minibatch of data and the same per sample data augmentations.
We note that this protocol differs from standard deep ensemble training procedures [e.g. 22, 8], where component networks are trained with different augmentations and minibatch orderings - a point we will return to at the end of this section.
A full description of experimental details can be found in Appx.
B.
In-distribution (InD) performance.
We first evaluate our trained ensembles on the CIFAR test set, which we indicate as the InD test set.
The blue line and bands in the top row of Fig.show the mean ± two standard error for trained ensemble performance, corresponding to models trained following Eq.
(6) with γ = 1 .
The black line and bands correspondingly gives the mean ± two standard error for trained single model performance.
Against these baselines, Fig.
2 shows the performance of individual ensembles trained with different values of γ .
Surprisingly, ensembles with a downweighted diversity penalty ( γ < 1 ) have much worse performance than the ensemble trained with unmodified objective ( γ = 1 ).
When γ is close to zero (i.e. training directly on the ensemble risk), performance drops below that of a single model with the same architecture.
This finding suggests that regularizing against predictive diversity is a critical component of deep ensemble performance- without this regularization, deep ensembles are easily outperformed by a single component model.
Although not shown here, we found that further reducing the value of γ < 0 leads to divergent losses.
In striking contrast, Fig.
2 shows that further penalizing diversity
[Figure]
(i.e. 1 < γ < 2 ) does not harm performance, but surprisingly can yield better accuracy than standard ensembles, with performance dropping off far more slowly as γ is increased further.
Out-of-distribution (OOD) performance.
Beyond InD test performance, deep ensembles have often been discussed as a benchmark method in settings where test data shifts significantly away from the training distribution [12, 22, 30].
Although this benefit is disputed by more recent work [1], we evaluated all of our trained ensembles and individual models on CIFAR10.1, a shifted test dataset for models trained on CIFAR10 [31].
The bottom row of Fig.
2 emphasizes the same conclusions we saw on InD data: a stark drop in ensemble performance with a lower diversity penalty, and an increase in performance with an increased penalty for 1 > γ > 2 , relative to γ = 1 unmodified training.
This result supports the conclusions of [1], which suggests that ensemble performance OOD is very tightly coupled to performance InD.
Comparison with standard ensemble training.
Compared to the γ = 1 ensembles which we train here, standard ensemble training introduces several additional sources of stochasticity: independent orderings of minibatches to each ensemble member during training, and independent data augmentations.
In Fig.
5, we show that these additional sources of randomness offer a boost in performance over our γ = 1 models shown in Fig.
However, we can still find settings of γ which generate ensembles that match or outperform these standard ensembles, across models and data types.
Analysis: The Trade-off Between Diversity and Single Model Performance
Having shown that encouraging predictive diversity in a training objective leads to worse ensemble performance, we next use the decomposition in Eq.
(5) to more directly evaluate the contributions of single model performance and predictive diversity to this ensemble performance.
From the terms in
Eq.
(5), we can see that are level sets of the ensemble loss: balanced tradeoffs between Jensen gap (i.e. predictive diversity) and single model loss that give ensembles of equivalent performance.
These level sets determine the transition between the desired scenario of helpful predictive diversity (Fig. 1, middle), and the worst-case scenario of harmful diversity (Fig. 1, right).
Unfortunately, our results show that the behavior of deep ensembles is best described by this worst case scenario.
Fig.
3 shows the predictive diversity (Jensen gap) versus average single model cross entropy loss in Eq.
(6) for the models in Fig. 2, evaluated on the CIFAR10 test set.
The cross in Fig.depicts the standard ensemble ( γ = 1 ).
We first observe that for ensembles trained with γ < 1 (blue dots), there is indeed an increase in the diversity of predictions among trained ensemble members, and a decrease in diversity of predictions for ensembles trained with γ > 1 (red dots).
Across all regularization strengths and model architectures, we find that any meaningful increase in predictive
[Figure]
A v g .
s i n g l e
m o d e l l o ss (
[Figure]
diversity comes at a cost of a large decrease in average single model performance, negating any diversity benefits.
Conversely, a moderate sacrifice in predictive diversity yield net improvements to ensemble performance, due to an outsize improvement in average single model performance.
Generalizing to deep ensembles used in practice.
The results shown thus far strictly control the diversity of ensemble models through a single parameter, γ .
In practice, it could be the case that ensembles based on different architectures and training paradigms trade off predictive diversity and average single model loss in qualitatively different ways to achieve their performance.
To test this hypothesis, we conducted a large scale study with 45 sets of standard deep ensembles, across different architectures and training hyperparameters (See Appx.
B for additional details).
Fig.illustrates the average single model cross entropy versus predictive diversity for these ensembles as blue dots.
Surprisingly, the best performing ensembles are well predicted by average single model performance, and have the lowest levels of ensemble diversity.
The orange dots in Fig.
4 show the result of forming heterogeneous ensembles, i.e. combining models of different architectures and training to create deep ensembles.
Because heterogeneous ensembles introduce greater diversity across model architectures without changing the training objective each individual model, we might hope that there is a greater potential for diversity based improvements.
It is somewhat surprising, therefore that our conclusions hold for these heterogeneous deep ensembles as well: average single model performance is the dominant factor in determining heterogeneous ensemble performance, and the best performing ensembles have very low levels of predictive diversity.
Discussion and Open Questions
Throughout this paper, all of our attempts to increase predictive diversity in deep ensembles end up harming the component models so severely that the resulting ensembles underperform the standard ensemble baseline.
In the full paper, we will apply these same analytical tools to models that have been trained against other diversity-based regularizers [e.g. 27, 29, 35, 41].
We have reason to believe that we will arrive at similar results in these settings: previous attempts to increase other notions of predictive diversity in ensemble training have yielded worse ensemble performance in large neural network models [23, 29].
More broadly, our results suggest a qualitative difference between deep ensembles and classical ensembles of weak learners.
Looking beyond models trained with a regularized Jensen gap, we find that the best performing deep ensembles used in practice are also the least diverse.
In stark contrast, increasing predictive diversity generates improvements in ensembles of weak learners [e.g.
We have reason to believe that the overparameterized nature of neural networks contributes to this phenomenon: recent work suggests that overparameterized models have similar inductive biases [2, 24], and large single models already perform similarly to deep ensembles [1, 3].
In the full version of this paper, we plan to analyze how the “strength” of the base model class affects the diversity-single model performance tradeoff.
A Derivation of the Jensen Gap for Common Loss Functions A.1 The Mean Squared Error Jensen Gap
In Sec.
[Equation]
we claim that the Jensen gap for the mean squared error loss is a scaled version of the sample variance (Eq. 2).
The following is a derivation of that fact:
[Equation]
where ¯ f ( x ) = 1 M (cid:80) Mi =1 f i ( x ) is the sample mean
and Var is the sample variance
[Equation]
A.2 Decomposition of Cross Entropy Jensen Gap
In Sec. [Equation] we claim (Eq. 3) that the Jensen gap is an information theoretic quantification of diversity.
The Jensen gap is given by:
[Equation]
Eq.
(7) can be interpreted as the D KL divergence between two categorical distributions.
The first distribution represents the probability of sampling an ensemble member uniformly at random ( 1 /M ).
The second distribution represents the probability of sampling an ensemble member proportional to its correct class prediction.
Eq.
(7) will be minimized when these two distributions are equal, which will only happen if all the ensemble members predict the correct class equally.
Conversely, Eq.
(7) will be larger when the component model predictions differ from one another.
B Experimental details B.1 Code and data
Code to train models with a diversity regularization and plot Figure 2 can be found in https://anonymous.4open.science/r/ensemble_attention-6178/README.md .
Model training can be performed by running the script: scripts/run.py .
Model and regularization strength can be changed as parameters, for example by running: python scripts/run.py ++classifier=resnet18 ++gamma=0.5 .
In this repository, the code to generate Figure 2 is located in:
• scripts/vis\_scripts/kl\_weight\_resnet18\_cifar.py • scripts/vis\_scripts/kl\_weight\_densenet121\_cifar.py Code to train all other models, and plot all remaining figures can be found in https://anonymous. 4open science/r interp ensembles
B /README. md where model training code is based on implementations from https://github.com/huyvnphan/PyTorch CIFAR10 and https:/ github.com/FreeformRobotics/Divide-and-Co-training .
In this repository, the code to create Figure 3, 4, 5, and 6 can be found at the following locations:
• scripts/paperfigs/estimate\_biasvar\_ce\_resnet\_zoom.py (Figure 3,5) • scripts/paperfigs/estimate\_biasvar\_ce\_densenet\_zoom.py (Figure 3,5) • scripts/paperfigs/estimate\_biasvar\_ce\_allmodels.py (Figure 4) • scripts/paperfigs/estimate\_biasvar\_ce\_allmodels\_compare.py (Figure 6)
Finally, relevant data can be found in this Zenodo repository: https://zenodo.org/record/ 6582653#.Yo7R0y-B3fZ , as previously shared by [1].
This data provides the logit outputs from individual models (and some ensembles) on the in and out of distribution data that we consider.
These data are referenced in the code above.
B.2 Figures 2, 3, and
All ensembles and single networks depicted in Figures 2,3 and 5 were trained with the same training schedule and hyperparameters: 100 epochs of training, with batch size 256, learning rate 1 e − 2 , weight decay 1 e − 2 .
The optimizer used was SGD with momentum, with a linear warmup ofepochs and cosine decay.
The best performing checkpoint on validation data was selected after training and used for further evaluation.
In all experiments, we created ensembles of size ( M = 4 ).
The SEM bands in Fig.
2 and Fig.are generated using ( N = 4) independently trained ensembles or samples; no models are reused between ensembles and single models in Fig.
2, while in Fig.
5 ensembles of size M = 4 are created by subselecting models from a collection of 5 independently trained models.
B.3 Figures 4 and
Fig.
4 includes three separate subsets of ensemble types.
The same models are used to generate
Fig.
The first subset consists of models that were trained with the same hyperparameters as in Appx.
B.2.
These models are all of size M = 5 , of the following architectures:
• ResNet 18 [15] • WideResNet 18-2, 18-4, 28-10 [38] • GoogleNet, Inception v3 [34] • VGG with 11 and 19 layers [33] • DenseNet 121 and 169 [17]
The second subset consists of model implementations derived from the code released with [41], found here https://github.com/FreeformRobotics/Divide-and-Co-training .
These ensembles of larger models are of the following architectures:
• Pyramidnet110 (M=4)
[13] • SeResNet-164 (M=5)
• ShakeShake26_2x96d (M=5) [10] • ResNexst50_4x16d (M=5) [37, 40, 41]
For these models, we changed the learning rate to 1 e − 1 , and the weight decay to 1 e − 4 .
Finally, we used 36 separate sets of ensembles from Miller et al.
[26], and we thank the authors for graciously sharing these results with us.
To generate heterogeneous ensembles, we first recorded the distribution of ensemble sizes among these separate subsets of ensembles.
We then randomly shuffled all of our individual models together, and divided them into new deep ensembles, respecting the distribution of ensemble sizes seen in the original set of homogeneous ensembles.
We computed all metrics of interest on 10 random shuffles of ensemble members, and plotted the spread across those random shuffles as the orange dots in Figs.and 6.
C Performance comparison to standard ensemble training
In Fig.
2, we compare models regularized with γ < 1 or γ > 1 directly to models trained with γ = 1 : i.e., an ensemble trained in parallel with Eq.
Here in Fig.
5, we compare these same γ regularized models to standard ensemble training, where each ensemble member is trained in serial.
In particular, standard ensemble training presents data to each ensemble member in a different minibatch ordering, with different augmentations.
As we can see in Fig.
5, this results in a slightly stronger baseline.
Nevertheless, we find that the same conclusions largely hold as in Fig.
2: ensembles trained with 1 < γ < 2 can match or outperform standard ensemble training, on both InD and OOD data.
D Comparison of CE Jensen gap to a standard diversity measure
In Eq.
(3), we describe the Jensen gap notion of predictive diversity for the cross entropy loss.
As this is a nonstandard notion of predictive diversity, we directly compare the diversity computed from this metric to that of a more standard one for classification tasks, based on average pairwise correlation between ensemble members.
This notion of diversity has previously been discussed in [e.g. 7, 21, 8].
In particular, given a set of models , f i , i ∈ [1 . . .
M ] we define this diversity measure based on the average pairwise correlation as:
[Equation]
Where pred ( f i ( x )) returns the index of the class predicted as the label for image x by network f i .
In Fig.
6, we apply this diversity measure, as well as the cross entropy instantiation of the Jensen gap to deep ensembles as constructed for Fig. 4.
Altogether, we see a strong correlation between these notions of diversity (black line gives linear fit).
[Figure]
a
[Figure]
ResNet 18 Ensemble (M = 4)
Weighted
[Figure]
DenseNet 121 Ensemble (M = 4)
[Figure]
[Figure]
CE Jensen gap (pred. diversity)
[Figure]
1A v g .
p a i r w
i s e c o rr e l a t i o n ( p r e d .
d i v e r
s
i t y )
CIFAR linear ﬁt heterogeneous homogeneous