G REEDY B AYESIAN P OSTERIOR A PPROXIMATION WITH D EEP E NSEMBLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixture of delta functions.
The training of ensembles relies on non-convexity of the loss landscape and random initialization of their individual members, making the resulting posterior approximation uncontrolled.
This paper proposes a novel and principled method to tackle this limitation, minimizing an f -divergence between the true posterior and a kernel density estimator in a function space.
We analyze this objective from a combinatorial point of view, and show that it is submodular with respect to mixture components for any f .
Subsequently, we consider the problem of greedy ensemble construction, and from the marginal gain of the total objective, we derive a novel diversity term for ensemble methods.
The performance of our approach is demonstrated on computer vision out-of-distribution detection benchmarks in a range of architectures trained on multiple datasets.
The source code of our method is made publicly available.
I NTRODUCTION
Estimation of predictive uncertainty is one of the most important challenges to solve in Deep Learning (DL).
Applications in ﬁnance, medicine and self-driving cars are examples where reliable uncertainty estimation may help to avoid substantial ﬁnancial losses, improve patient outcomes, or prevent fatal accidents (Gal, 2016).
However, to date, despite rapid progress, there is a lack of principled methods that reliably estimate the predictive uncertainty of deep neural networks (DNNs).
Numerous attempts have been made to develop Bayesian techniques for uncertainty estimation in
DL (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Wilson & Izmailov, 2020; Maddox et al., 2019; Izmailov et al., 2020; Van Amersfoort et al., 2020; Wenzel et al., 2020b).
One of the most practical and empirically best-performing approaches is based on training a series of independent DNNs (Lakshminarayanan et al., 2017; Wilson & Izmailov, 2020; Ashukha et al., 2020;
Lu et al., 2020; Wenzel et al., 2020b).
The main method in this category, Deep Ensembles (DE) (Lakshminarayanan et al., 2017), is used as a reference approach in the context of this paper.
Recent studies, e.g. Wilson & Izmailov (2020) interpret ensembles as an approximation of predictive posterior.
While this interpretation is correct from a Bayesian point of view, obtaining individual ensemble members via maximum a posteriori probability (MAP) estimation, as e.g. done in DE (Lakshminarayanan et al., 2017), may not lead to obtaining good coverage of the full support of the posterior distribution, and has arbitrary bad approximation guarantees.
For example, the resulting approximation can be poor in the case when the true posterior distribution is unimodal.
In this work, we argue that enforcing coverage of posterior (i.e. ensemble diversity) is non-trivial, and needs a specialized principled approach.
We highlight this graphically in Figure 1.
Another important line of work in modern Bayesian DL (BDL) is a paradigm of performing Bayesian inference in the weight space.
While distributions over weights induce distributions over functions (Wilson & Izmailov, 2020), it is rather unclear what the properties of such functions are, and whether Bayesian posteriors obtained in the weight space yield good quality approximations in the function space.
For example, it is known that diverse weights do not necessary yield diverse functions, thus sampling from weight-based posteriors may yield poor quality uncertainty estimation, e.g. in detecting out-of-distribution (OOD) data (Garipov et al., 2018; Hafner et al., 2020).
[Figure]
Recent studies (Wang et al., 2019; D’Angelo & Fortuin, 2021) show the promise of particle variational inference (POVI) done in the function space, however, the performance of those methods is not state-of-the-art due to the use of BNN priors.
In practice, a weight decay prior combined with batch normalization, data augmentation and dropout are often employed due to empirical improvement, and theoretical guarantees are traded for practical performance.
Furthermore, particle-based function space VI requires training an ensemble of BNNs simultaneously, which is not only difﬁcult to implement, but also requires extensive resources for parallelization.
Summary of the contributions.
In this paper, we propose a novel and principled methodology for approximate function space posterior inference for DNNs.
Contrary to the mainstream approach in BDL, which is based on deﬁning a posterior distribution over the model parameters (Maddox et al., 2019), we take a functional view, which allows us to treat the problem of training ensembles as optimization over sets.
Namely, selecting a set of functions to approximate the Bayesian posterior is fundamentally similar to problems such as facility location and set cover, which are frequently solved using submodular optimization (Krause et al., 2008).
Speciﬁcally, our contributions are:
We show that ﬁtting a kernel density estimator to a distribution using an f -divergence is a cardinality-ﬁxed non-monotone submodular maximization problem.
Inspired by the Random Greedy algorithm for submodular maximization (Buchbinder et al., 2014), we design a new method for the function space Bayesian posterior approximation via greedy training of ensembles with a justiﬁed coverage-promoting diversity term.
We demonstrate the effectiveness and competitiveness of our approach compared to DE in the OOD detection task on MNIST, CIFAR, and SVHN, benchmarks on different architectures and ensemble sizes.
Furthermore, we show that our method can use state-of-the-art training technqiues, compared to the existing Bayesian approaches, and yeilds state-of-theart performance.
P RELIMINARIES 2. P ROBLEM STATEMENT
Consider an ensemble to be parameterized by a set of functions Z
= { z m } Mm =1 ⊂ F , where F is a class of continuous functions, z m : R d → R c , with d the dimensionality of the input data, and c the dimensionality of the output.
When training ensembles, we generally want to solve the following optimization problem:
[Equation]
where R ( Z ) = 1 N (cid:80) Ni =1 (cid:96) (cid:16) 1 M (cid:80) Mm =
z m ( x i ) , y i (cid:17) is the empirical risk of the ensemble, (cid:96) :
Y×Y →
R + is a loss function, D
= { x i , y i } Ni =1 is a training dataset of size N , and Ω λ M ( Z ) is some diversitypromoting term, with diversity regularization strength λ M .
Empirical observations in the earlier works on ensembles (Lakshminarayanan et al., 2017; Wilson & Izmailov, 2020; Fort et al., 2019) have shown that one can simply ignore Ω λ M ( Z ) , and rely on non-convexity of the loss landscape, optimizing the risks of individual ensemble members.
From a variational inference (VI) perspective (Zhang et al., 2019), this can be seen as a mode-seeking method, and it has been shown experimentally that every ensemble member may discover different modes of the posterior distribution in the function space p ( z |D ) .
Approximation of p ( z |D ) is the ultimate aim of this paper, and we argue that randomization-based mode-seeking is insufﬁcient to obtain a good quality approximation of p ( z |D ) , as this procedure does not maximize the coverage of the support of the posterior.
By contrast, we aim to ﬁnd an Ω λ M ( Z ) such that the posterior coverage is also maximized.
Taking a VI perspective again (Zhang et al., 2019), Ω λ M ( Z ) needs to enforce that min Z, | Z | =
M R ( Z ) − Ω λ M ( Z ) has rather mean-seeking behavior, while still discovering the high density modes.
2. A COMBINATORIAL VIEW OF ENSEMBLE CONSTRUCTION
Having now deﬁned the main criteria for (1), we highlight that the problem of constructing an ensemble can be seen from a combinatorial point of view.
We therefore treat ensemble construction as subset selection from some ground set of functions, and introduce the main notions of submodular analysis, a powerful tool that enables the analysis of the optimization of set functions (Bach, 2013;
Fujishige, 2005).
Deﬁnition 1 (Submodularity) .
A set function g : 2 V → R , for the power set of a base set V , is submodular if for all A ⊆ B ⊂ V and x ∈ V \ B
[Equation]
Deﬁnition 2 (Supermodularity and modularity) .
A set function is called supermodular if its negative is submodular, and modular if it is both submodular and supermodular.
Consider now problem (1).
Assuming that the loss function (cid:96) is convex, we can derive an upper-bound on the risk R ( Z ) using Jensen’s inequality, and obtain a method, which generalizes DE
[Equation]
If M is ﬁxed during optimization, 1 M (cid:80) Mm =1 R ( z m ) contributes a positive modular term to the overall objective.
Adding a positive modular function to any set function does not change its submodularity or supermodularity, thus we focus on Ω λ M ( Z ) .
A trivial approach would be to enforce pair-wise diversity by computing a norm of the pairwise differences between functions, i.e. setting Ω λ M ( Z ) =
λ M (cid:80) i (cid:54) =
j (cid:107) z
i − z j (cid:107) 2 ∗ .
However, this is a cardinality-ﬁxed submodular minimization problem.
It is known that it is strongly NP-hard, i.e. there exists no general polynomial time approximation algorithm for it (Svitkina & Fleischer, 2011).
The poor quality approximation of this approach is highlighted in Figure 1.
We therefore conclude that the choice of Ω λ M ( Z ) has a direct impact on the approximabilty of the objective.
S UBMODULAR ANALYSIS OF f - DIVERGENCES 3. f - DIVERGENCES ARE SUPERMODULAR FUNCTIONS
Main result.
We now consider the problem of approximating a Bayesian posterior via minimization of an f -divergence.
Here, we speciﬁcally aim our optimization procedure to have both mode and mean-seeking behaviors, i.e. cover the posterior distribution as much as possible, ending up in its mode.
We furthermore aim to obtain a polynomial time algorithm that yields a good quality approximation guarantee.
In this paper, we leverage classic deﬁnitions of approximation algorithm and approximation guarantees.
Deﬁnition 3 (Approximation algorithm and guarantees (Williamson & Shmoys, 2011)) .
A γapproximation algorithm for an optimization problem is a polynomial-time algorithm that for all instances of the problem produces a solution whose value is within a factor γ of the optimal solution.
Let us now formally introduce f -divergences.
Deﬁnition 4 ( f -divergence) .
Let f : R + → R be a convex function such that f (1) = 0 .
The f -divergence between distributions P z and Q z ( P z being absolutely continuous with respect to Q z ), admitting densities p ( z ) and q ( z ) is deﬁned as
[Equation]
Consider some density p ( z ) over continuous functions.
We deﬁne q M (
[Equation]
= 1 M
[Equation]
Mm =1 K
[Equation]
K j ( z ) :=
K ( z, z j ) is a kernel centered at z j used to approximate the modes of p ( z ) .
Theorem 1.
Any f -divergence
[Equation]
between a distribution p ( z ) and a normalized mixture of M kernels with equal weights is supermodualar in a cardinality-ﬁxed setting, assuming that ∀ z max q M D f ( p ( z )
||
q M ( z )) < ∞ .
Proof.
The proof is shown in Appendix A.1.
Minimization of (5) is equivalent to a cardinality-constrained maximization of a non-monotone submodular function of Z = { z 1 , . .
z M } .
Approximation guarantees for problems of this form are given for non-negative submodular functions (Buchbinder et al., 2014).
One can convert (5) to a non-negative function by deﬁning:
[Equation]
where C = max Z D ( p || q M ) is a pre-deﬁned constant, which is important for understanding approximation guarantees, but does not need to be computed in practice.
After the described transformation, which leads to (6), we obtain F ( Z ) , a non-negative non-monotone submodular function.
Approximation guarantees for f -divergences.
In the context of submodular functions, there exists an inapproximability result, obtained in (Gharan & Vondrák, 2011), which states that no general polynomial time algorithm with guarantees better than 0 .
491 exists to solve a submodular maximization problem with constrained cardinality.
We, however, that in practice, it might still be possible to obtain a better approximation factor than 0 .
491 for some speciﬁc types of divergences, as these guarantees are deﬁned for all instances of the optimization problem (Deﬁnition 3).
Let us consider the approximation guarantees for (6), denoting by q ∗ M the optimal solution, and by ˆ q M a solution found by some algorithm.
For an approximation factor γ , we have
[Equation]
Simple algebra shows that this implies
[Equation]
where F is a family of distributions over the spaces of candidate functions.
The derived result indicates that the upper bound on the approximate solution found by minimizing an f -divergence can be substantially dominated by (1 − γ ) max q M ∈F D f ( p || q M ) if F is chosen poorly.
The impact of the choice of function space, determined largely by the neural network architecture, is therefore unavoidable when designing algorithms for approximating the Bayesian posterior.
3. G REEDY MINIMIZATION OF f - DIVERGENCES
Random greedy algorithm.
Although submodular optimization has natural parallel extensions and associated approximation guarantees, due to the simplicity of presentation, we focus in this paper on forward greedy selection and use Algorithm 1 for optimizing submodular functions.
This algorithm has approximation guarantee of 1 /e in general.
The only required step for this greedy algorithm is a computation of the marginal gain on the objective function F ( Z ) , i.e. ∆( z k | Z ) =
F ( Z ∪ { z k } ) − F ( Z ) .
We argue that due to randomized initialization in neural networks, one can use a naive greedy algorithm instead of the sampling procedure in lines 7-8, and we show the implementation of the resulting training procedure in Appendix B.1.
Algorithm
1: Input: V – Ground set 2: Input: F – Arbitrary submodular function 3: Input: M – Cardinality of the solution 4: Z ← ∅ 5: for m = 1 to M do 6:
R ← arg max T ⊂
V \ Z : | T | =
M (cid:80) z (cid:48) ∈
T ∆( z (cid:48) | Z ) 7: u i ← Uniform ( R ) 8:
Z ← Z ∪ { u i } 9: end for 10: return Z
Marginal gain.
At each step of a greedy algorithm, a marginal gain ∆( z k | Z )
=
F ( Z ∪ { z k } ) −
F ( Z ) of adding a new element z k to an existing mixture (cid:80) k − 1 j =1 K j ( z ) is maximized.
For fdivergences, we thus formulate the following proposition:
Proposition 1.
Consider C = max D f ( p || q M ) , where D f ( p || q M ) is an arbitrary f -divergence between some distribution p ( z ) and a mixture of kernels q M ( z )
= 1 M (cid:80) Mj
=1 K j ( z ) , and D f ( p || q M ) < ∞ .
Then, maximization of a marginal gain for set function
[Equation]
at a step k of a greedy algorithm corresponds to
[Equation]
Proof.
The proof is shown Appendix A.2.
The case of reverse KL divergence.
Having mean-seeking behavior is useful to ﬁt the kernel density estimator using forward divergences.
However, if one wants to optimize marginal gains, they need to have a mode-seeking behavior, which is achieved via optimizing reverse divergences (Zhang et al., 2019).
If we consider the generator for the reverse KL-divergence, f ( x ) = − log x , the minimization simpliﬁes to
[Equation]
In the sequel, we leverage the standard reverse KL-divergence generator, which aims to discover the modes of p ( z ) , but also enforces the coverage of the approximation.
Finally, we note that it is costly to compute the expectation in practice; we thus use the following point estimate of (11):
[Equation]
G REEDY A PPROXIMATION OF THE B AYESIAN POSTERIOR
Objective function.
In this section, we consider parametric functions z θ : R d → R c , where θ denotes the parameters determining the function.
We now use Bayes’ theorem and a mean-ﬁeld approximation to express the posterior of an ensemble of such functions as
[Equation]
where θ m are parameters, p ( D| z
θ ) the likelihood and p ( z θ ) the prior; p
[Equation]
D| z
[Equation]
) ∝ (cid:81) ni =1 exp( − (cid:96) (
z θ ( x i ) , y i ) ), and p ( z θ ) ∝ exp( − λ (cid:107) θ (cid:107) 22 ) .
To solve the problem deﬁned by (12) in the context of Bayesian posterior approximation, we deﬁne the kernel density components via generalized exponential kernels K j ( z θ ) ∝ exp( − λ M d ( z θ , z θ j ) 2 ) , where λ M is proportional to the kernel width.
Therefore, at a k th greedy step, we minimize
[Equation]
which is similar to the marginal gain on our originally deﬁned high-level objective (1), except that the diversity term Ω has a different form based on our submodular f -divergence optimization.
Sampling-based approximation of the diversity term.
When minimizing (14), one needs to be able to compute the diversity term Ω = log (cid:80)
k − 1 j =1 K j ( − λ M d ( z θ , z θ j ) 2 ) , which is derived from a kernel density estimator we aim to ﬁt to the true posterior.
We note that this needs to be done in the function space , which makes this computation non-trivial.
We earlier deﬁned K j ( z ) to be an individual kernel in a mixture 1 M (cid:80) Mj
=1 K j ( z ) .
In order to be able to use the f -divergence, the individual components K j ( z ) must be density functions centered at z j , which implies the need of a notion of similarity or the existence of a function norm, which are known to be NP-hard to compute for any neural network with depth greater than 3 (Rannen-Triki et al., 2019).
We note that this intrinsic hardness result applies to all methods that deﬁne a meaningful posterior distribution in function space through a kernel density estimator.
We thus use here a method from (Rannen-Triki et al., 2019) to approximate (cid:107) z (cid:107) 22 , via i.i.d.
samples x i ∼ P ∗ , where P ∗ is a weighting distribution, which is required to ensure that Monte Carlo integration yields a reasonable approximation.
This leads to the following sampling-based approximation of the diversity term:
[Equation]
A note on practical implementation.
To this point, we deﬁned all the main components of our method, except the weighting distribution in (15).
The desirable behavior, which we expect an ensemble to exhibit, is that it must be uncertain on the OOD data and certain in the regions where the training data are available.
This implies that p ∗ ( x ) must include OOD samples.
One can use OOD data in training explicitly, however, we resort to a setting when OOD data are unknown, and use a simple heuristic, which ﬁts a Gaussian to every data dimension with the variance × 5 larger than the variance of the data.
We specify further details about the full algorithm in Appendix B.1.
R ELATED WORK
Randomization-based ensembles.
Generally, ensembles have been studied in Machine Learning over several decades for different classes of models (Hansen & Salamon, 1990; Breiman, 1996;
Freund & Schapire, 1997; Lakshminarayanan et al., 2017).
One can distinguish several main methods for diverse ensemble construction (Pearce et al., 2020): randomization of training initialization and hyperparameters (Hansen & Salamon, 1990; Wenzel et al., 2020b; Wilson & Izmailov, 2020; Zaidi et al., 2020), bagging (Breiman, 1996; 2001), boosting (Freund & Schapire, 1997), and explicit diversity training (Kuncheva & Whitaker, 2003; Ross et al., 2020;
Yang et al., 2020; Brown et al., 2005; Kariyappa & Qureshi, 2019; Sinha et al., 2020; Melville & Mooney, 2005).
Randomizationbased ensemble construction has shown good results in in-domain uncertainty (Ashukha et al., 2020) estimation, but also in the detection of OOD data (Lakshminarayanan et al., 2017).
Submodular ensemble pruning and greedy ensemble construction Submodularity in ensemble learning has previously been discussed in the context of ensemble pruning (Sha et al., 2014).
The goal of ensemble pruning is to trim a large ensemble of models so that the accuracy of the ensemble remains the same.
We note that recently, inspired by the submodular pruning approach, a greedy algorithm has been applied to randomization based ensembles (Wenzel et al., 2020b; Zaidi et al., 2020).
However, we also note that neither of these works approached the problem of ensemble construction from a Bayesian posterior approximation point of view.
However, they provide extensive experimental evidence on the plausibility of greedy approach.
Our work sheds light on why in Wenzel et al.
(2020b)
; Zaidi et al.
(2020) ensembles constructed greedily worked well in OOD detection tasks.
Diversity-promoting regularization for ensemble training.
The ensemble literature contains a line of work focusing on explicitly promoting regularization in ensembles (Kuncheva & Whitaker, 2003; Ross et al., 2020;
Yang et al., 2020; Brown et al., 2005; Kariyappa & Qureshi, 2019; Sinha et al., 2020; Melville & Mooney, 2005; Havasi et al., 2021).
In terms of promoting diversity outside training data, the closest work to ours is (Ross et al., 2020; Rame & Cord, 2021), and in terms of the form of diversity regularization it is (Kariyappa & Qureshi, 2019).
However, none of these works takes the perspective of approximating the Bayesian posterior.
Another limitation of most of these approaches is that they use either out-of-distribution data in training, adversarial examples, or expensive generative models, thus making those methods difﬁcult to scale to large datasets.
POVI A problem of learning diverse ensembles can be seen from a POVI perspective: in particular
Stein Variational Gradient Descent (SVGD) (Wang & Liu, 2019).
SVGD aims to learn a diverse set of functions Z , approximating arbitrary distributions via a set of particles using a reverse KL divergence.
This approach is similar to ours, however, we tackle the problem of optimizing a general f -divergence in the function space.
Furthermore, to our knowledge, the present work is the ﬁrst that takes a submodular minimization perspective in BDL.
Greedy POVI Another line of work in the POVI family also considers greedy approximations (Futami et al., 2019; Jerfel et al., 2021).
In both of these works, the authors perform particle based inference by also optimizing the kernel widths of the KDE, which is different to our method.
Furthermore, these works do not tell what the diversity term for ensemble training should look like, and how to compute it in practice.
Function space POVI Conventionally, Bayesian inference in DL is thought of in the weight space (MacKay, 1992; Blundell et al., 2015; Gal & Ghahramani, 2016; Izmailov et al., 2020; Wilson & Izmailov, 2020;
Pearce et al., 2020; Wenzel et al., 2020a).
However, recent studies point out that despite the fact that simple priors over weights may imply complex posteriors in the function space, the connection between two of these is difﬁcult to establish (Sun et al., 2019; Hafner et al., 2020).
Recent papers on function-space POVI (Wang et al., 2019; D’Angelo & Fortuin, 2021) point out that one can do Bayesian inference in the function space by optimizing an objective function with a repulsive term.
Notably, they draw connection to the reverse KL-divergence minimization, and we thus consider our method to be closely connected to function space POVI.
E XPERIMENTS 6. S ETUP
Datasets and models.
We ran our main experiments on CIFAR10, CIFAR100 (Krizhevsky, 2009) and SVHN (Netzer et al., 2011) in-distribution datasets.
Our OOD detection benchmark included
CIFAR10, CIFAR100, DTD (Cimpoi et al., 2014), SVHN (Netzer et al., 2011), LSUN (Yu et al., 2015), TinyImageNet (Le & Yang, 2015), Places 365 (Zhou et al., 2017), Bernoulli noise images,
Gaussian noise, random blobs image, and uniform noise images.
The composition of the benchmark was inspired by the work of Hendrycks et al.
We excluded the in-distribution datasets for each of the settings, resulting in a total of 10 OOD datasets for each in-distribution dataset.
The full description of the benchmark is shown in Appendix B.2.
The experiments were conducted using ResNet164 (pre-activated version; denoted as PreResNet164) (He et al., 2016), VGG16 (with batch normalization (Ioffe & Szegedy, 2015); denoted as VGG16BN)
(Simonyan & Zisserman, 2015), and WideResNet28x10 (Zagoruyko & Komodakis,
43113(a)
Deep Ensembles    43113(b)
λ
M = 0 .
43113(c)
λ M =    43113(d)
λ M =
All our models in the ensembles were trained for 100 epochs using PyTorch (Paszke et al., 2019), each ensemble on a single NVIDIA V100 GPU.
In the case of CIFAR and SVHN experiments, we trained ensembles of size M = 11 , and report the results across 5 different random seeds.
For the
CIFAR experiments, we selected λ M ∈ { 0 . 001 , 0 .
In addition to the CIFAR and SVHN experiments, we used MNIST (LeCun et al., 1998) with ResNet8.
The details of those experiments are shown in Appendix B.2.
Model selection and metrics.
We used mutual information (MI) between the distribution of the predicted label ˆ y for the point ˆx and the posterior distribution over functions p ( f |D ) , to evaluate the epistemic uncertainty (Malinin & Gales, 2018; Depeweg et al., 2018) and reported the area under the ROC curve (AUC) and area under the precision-recall (PR) curve, i.e. average precision (AP) to quantify the OOD detection performance.
Furthermore, we computed the false positive rate at 95% true positive rate (FPR95).
Details on the computation of epistemic uncertainty can be found in Appendix B.3.
6. R ESULTS
Illustrative examples.
Figure 2 illustrates how our method performs on the two moons dataset.
Here, we used a two-layer fully-connected network with ReLU activations (Fukushima, 1988).
Having a high λ M is important to obtain good uncertainty estimation.
As expected, compared to our method, the DE method does not explicitly maximize the coverage of the posterior, and thus fails to be uncertain outside the training data.
Out-of-distribution detection.
We present aggregated results for all the models and in-distribution datasets in Table 1.
It is clear that on average (across OOD datasets), our method is substantially better than DE.
This holds for all the architectures and in-distribution datasets.
We show the expanded version of all the OOD detection results in Appendix C.3.
An example of these results is shown in Figure 3 for all the models trained on CIFAR 100.
Here, one can see that our method is at least similar to DE, and substantially better overall.
Finally, some examples of OOD detection by both
DE and our method are shown in Figure 4.
Here, we computed optimal thresholds for each of the methods by optimizing the trade-off between the true positive and true negative rates.
D ISCUSSION
In this paper, we have introduced a novel paradigm for Bayesian posterior approximation in Deep
Learning using greedy ensemble construction via submodular optimization.
We have proven a new general theoretical result, which shows that minimization of an f -divergence between some distribution and a kernel density estimator has approximation guarantees, and can be done greedily.
We then derived a novel coverage promoting diversity term for ensemble construction.
The results presented in this paper, as well as in Appendix C.4, demonstrate that our method outperforms the
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
This study has some limitations, which outline several directions for the future work.
Firstly, we did not compare our approach to a variety of existing methods for ensemble generation, e.g. snapshot ensembles (Huang et al., 2017), batch ensembles (Wen et al., 2020) or hyperparameter ensembles (Wenzel et al., 2020b).
However, these methods are heuristic, and as discussed in the related work, our method can be used in conjunction with them to make them more principled.
Furthermore, we note that the main contribution of this paper is novel theory.
The second limitation of this work is that it does not compare to f-POVI Wang et al.
; D’Angelo & Fortuin (2021).
However, as noted earlier, those methods are in a different class, as they focus on training models without state-of-the-art training techniques such as batch normalization (Ioffe &
Szegedy, 2015) and data augmentation.
The third limitation is that we did not fully explore different techniques of generating weighting distributions for the diversity term.
However, we still evaluated the possibility of using real images for this purpose in an outlier exposure (Appendix C.4).
The ﬁnal limitation is related to our method, as it lacks parallelization possibilities compared to DE.
We note, however, that there is a class of parallel submodular optimization algorithms that enable parallelization and retain approximation guarantees (Ene & Nguyen, 2020).
To conclude, this paper provides a novel foundational framework for Bayesian Deep Learning.
We hope for the wide adaption of the proposed method by practitioners across the ﬁelds.
Our code is available at https://anonymous.4open.science/r/greedy_ensembles_ training-0CD6/ .
A P ROOFS A. P ROOF OF T HEOREM
To prove this proposition we make use of the following theorem:
Theorem A1 (Theorem 1.4 from (Mitrinovic & Vasic, 1970)) .
A function f : D → R is convex on
D =
[ a, b ] if and only if ∀ x 1 < x 2 < x 3 ∈ D (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Theorem 1.
Any f -divergence
[Equation]
between a distribution p ( z ) and a mixture of M kernels with equal weights is supermodualar in a cardinality-ﬁxed setting, assuming that ∀ z max q M D f ( p ( z )
||
q M ( z )) < ∞ .
Proof.
In order to prove that f -divergences are supermodular, we need to show that ∀ α > 0 , xf ( αx ) is convex, because q M ( z ) is a positive modular function (Bach, 2019, Proposition 6.1), due to M being ﬁxed.
From Theorem A1, ∀ x 1 < x 2 < x 3 in [ a, b ] f ( x ) is convex if an only if
[Equation]
[Equation]
We know that x 1 ≤ x 2 ≤ x 3 and x 1 , α > 0 .
We divide each i th row by x i :
[Equation]
After the division, we denote new variables y
= 1 x 3 , y 2 = 1 x 2 , and y
= 1 x 2 .
One can see that y 1 < y 2 < y 3 , because x 1 < x 2 < x 3 .
We then get
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Changing the ﬁrst and the third row of the determinant will change the sign.
Changing the third and the ﬁrst columns will also change the sign.
Therefore
[Equation]
(21) and thus we get that
[Equation]
Consider now a mapping 1 α y , which preserves convexity:
[Equation]
Division of the ﬁrst and the second column by α does not change the sign of the determinant, therefore
[Equation] [Equation]
which concludes the proof.
A. P ROOF OF P ROPOSITION
Proposition 1.
Consider C = max D f ( p || q M ) , where D f ( p || q M ) is an arbitrary f -divergence between some distribution p ( z ) and a mixture of kernels q M ( z )
= 1 M (cid:80) Mj
=1 K j ( z ) , and D f ( p || q M ) < ∞ .
Then, maximization of a marginal gain for set function
[Equation]
at a step k of a greedy algorithm corresponds to
[Equation]
Proof.
We aim to derive a marginal gain of adding an element deﬁned by z k to 1 M (cid:80) k − 1 j =1 K j ( z ) 1 .
Let us denote G ( z ) = f
[Equation]
Then −
[Equation]
One can observe that the ﬁrst term of (27) is upper-bounded by a constant, which is not dependent on z k :
[Equation]
therefore, to maximize the marginal gain, one needs to maximize the second term.
Consequently, we write the objective corresponding to a marginal gain as
[Equation]
maximization of which is equivalent to
[Equation]
which concludes the proof.
B I MPLEMENTATION DETAILS B. P RACTICAL IMPLEMENTATION OF THE ALGORITHM
Weighting distribution for the diversity term We propose the following simple heuristic, deﬁning p ∗ ( x ) as a normal distribution N ( µ D , α · Σ D ) of dimensionality, corresponding to the training data.
The covariance Σ D for this distribution is set to be diagonal, such that the variance for every dimension j is Σ D [ j, j ] =
( α · σ j ) 2 , where α > 1 is a scaling parameter, and σ 2 j is a variance of the dimension j computed from samples of the training dataset D .
Similarly, µ D , the vector of expected values for every dimension, is also computed from the training data.
Finally, the hyperparameter α = 5 was found to work well, and we thus report all the experimental results with it ﬁxed.
We note that a similar technique, but for in-distribution data generation has been used earlier in (Melville & Mooney, 2005).
The resulting algorithm
The resulting, computationally tractable optimization algorithm for ensembles, which minimizes marginal gains (10), is shown in Algorithm B2.
For simplicity, we omit the snapshot selection step, i.e. early stopping.
We note that contrary to the general Random Greedy method, shown in the main text, we can resort to a method with complexity of O ( k ) .
This is achieved through the fact that a uniform selection of the elements maximizing the marginal gain can be avoided, since at each greedy step, we initialize the new models randomly before maximizing the marginal gain.
Another performance improvement can be gained by storing the evaluations z j ( x i ) ∀ j = 1 , . . .
, k − 1 in memory before executing each k th step.
We report here also one important practical trick, which we found important during the training.
Speciﬁcally, freezing the batch normalization layers (Ioffe & Szegedy, 2015) before computing the diversity term turned out to help the convergence substantially.
We anticipate that the diversity term weighting distribution approximated as a simple multivariate Gaussian with diagonal covariance may be corrupting the batch norm statistics.
We thus think that using other, more sophisticated techniques for generating the weighting distribution samples might provide better results.
B. OOD DETECTION BENCHMARKS
MNIST The MNIST dataset benchmark included 6 datasets: Fashion MNIST Xiao et al.
DTD Cimpoi et al.
(2014), Omniglot Lake et al.
(2015), Gaussian noise, Bernoulli noise, and uniform noise datasets (see Table B1).
We re-scaled all the images to the range of [0 , 1] .
Subsequently, we applied the same mean and standard deviation as we have applied to the original images before feeding them to the network.
6: Z
7: D ∗ ← {
∼ N ( µ D α ·
8: while | Z | < M do 9: k ← | Z |
Randomly initialize z θ k ; 11: for i = 1 to N do 12:
D i ← { x b , y b ) } N b b =1 ∼ D ; 13:
D ∗ i ← { x ∗ b } N b b =1 ∼ D ∗ ; 14:
L ← E
∼D z θ k x y )
+ λ (cid:107) θ k
15: Ω ←16:
if k > 1 then 17: for j = 1 to k − 1 do 18: d i j ← E x ∗
∼D ∗
i (cid:107) z
θ k ( x ∗ ) − z θ j ( x ∗ ) (cid:107)19: end for 20: Ω ← log exp( i m ) 21: end if 22:
Update θ using ∇ θ ( L + Ω) 23: end for 24:
Z ← Z ∪ { z θ k } 25: end while 26: return Z
CIFAR and SVHN The CIFAR and SVHN OOD benchmark included 10 different datasets.
Here, we also DTD, Bernoulli noise, Gaussian noise and uniform noise datasets, and added Places(Zhou et al., 2017), Tiny ImageNet (Le & Yang, 2015; Deng et al., 2009), and LSUN (Yu et al., 2015) datasets to the benchmark.
For CIFAR10 as in-domain data, we added CIFAR100 and SVHN (Netzer et al., 2011) to the benchmark.
For CIFAR100 – CIFAR10 and SVHN.
Finally, for SVHN, we added
CIFAR10 and CIFAR100 as OOD datasets, making a total of 10 OOD datasets per 1 in-distribution dataset.
The details about each of the datasets are shown in Table B1.
Before feeding the images to the network, we applied re-scaling similarly to the MNIST setting.
We also used the same mean and standard deviation normalization procedure, as for the in-domain data.
[Table]
B. E PISTEMIC UNCERTAINTY COMPUTATION
We used the epistemic uncertainty , i.e. mutual information (MI) between the distribution of predicted label ˆ y for the point ˆx and the posterior distribution over functions p ( f |D ) , to evaluate the uncertainty (Malinin & Gales, 2018; Depeweg et al., 2018).
As a distribution over weights induces a distribution over functions, we approximate the MI as:
[Equation]
where H [ · ] denotes the entropy.
One can see that this metric can be efﬁciently computed from the predictions of an ensemble.
C E XPERIMENTS C. E XPERIMENTAL DETAILS
Model selection Contrary to the commonly used practice, we did not use CIFAR10/100 and SVHN test set sets for model selection.
Neither did we use any OOD data.
Instead, we used validation set accuracy ( 10% of the training data; randomly chosen stratiﬁed split) to select the models when optimizing the marginal gain.
The best snapshot was found using the validation data, was then selected for ﬁnal testing.
When selecting the models for evaluation on OOD data, we ﬁrst evaluated ensembles on the in-distribution test set (Appendix C.2).
Subsequently, we selected the highest λ M that did not harm the test set (in-domain) performance (no overlap of conﬁdence intervals deﬁned as mean ± standard error).
To provide additional information, we also analyzed adaptive calibration error (ACE) with 30 bins (Nixon et al., 2019).
Two moons dataset For the synthetic data experiments, we used scikit-learn (Pedregosa et al., 2011), and generated a two-moons dataset with 300 points in total, having the noise parameter ﬁxed to 0 .
Here, we used a two-layer neural network with ReLU (Krizhevsky, 2009) activations and hidden layer size of 128 .
CIFAR10/100 and SVHN The main training hyper-parameters were adapted from (Maddox et al., 2019)
(see Table C2), but with additional modiﬁcations inspired by (Malinin & Gales, 2018; Smith &
Topin, 2019), which helped to train the CIFAR models to state-of-the-art performance in onlyepochs.
As such, we ﬁrst employed a warm-up of the learning rate ( LR ) from a value 10 times lower than the initial LR ( LR init in Table C2) for 5 epochs.
Subsequently, after 50% of the training budget, we linearly annealed the LR to the value of LR × lr scale until 90% of the training budget is reached, after which we kept the value of LR constant.
All models were trained using stochastic gradient descent with momentum of 0 .
9 and a total batch size of 128 .
We employed standard training augmentations – horizontal ﬂipping, reﬂective padding to 34 × 34 , and random crop to 34 × 34 pixels.
[Table]
MNIST In addition to the CIFAR10/100 experiments, we also trained our method on MNIST (LeCun et al., 1998) with PreResNet8 architecture.
As OOD, we used FashionMNIST (Xiao et al., 2017) and Omniglot (Lake et al., 2015) datasets.
We also tested other architectures, such as PreResNet20, but the models with higher depth than 8 already gave nearly perfect scores on MNIST.
Hyper-parameter-wise, we trained all the models for 20 epochs without warmup with the batch size of 256 using plain SGD with momentum.
The weight decay was set to 1 e − 5 .
We used LR annealing similarly as for CIFAR experiments, but used lr scale = 0 . 0001 .
No data augmentations were used in any of the MNIST experiments.
λ M was searched in range { 0 . 0001 , 0 .
11 , 7 } for
[Figure]
M ∈ { 3 , 5 , 9 , 15 } .
This series of experiments was re-run 3 times, as the MNIST dataset is rather simple, and the test scores have low variance between the runs.
C. CIFAR10/100 AND SVHN IN - DOMAIN PERFORMANCE
CIFAR10/100 in-distribution performance vs. diversity.
Figure C1 provides an illustration of how the test set performance changes with λ M on CIFAR data.
One can see a general trend that when λ M approaches M , the models lose the ability to make accurate predictions, which results in lower accuracy and poorer calibration.
Interestingly, performance on the VGG model degrades much slower with λ M compared to other architectures.
Similar ﬁndings were also obtained for the SVHN dataset.
Based on the test performance, we selected the models for further evaluation on OOD benchmark.
CIFAR and SVHN: best models’ performance Table C3 shows the results of all the trained models on the in-domain data.
One can see that the results between Deep Ensembles (DE) (Lakshminarayanan et al., 2017) do not differ signiﬁcantly.
We trained all these models according to the earlier speciﬁed hyper-parameters and the learning rate schedule.
Models selected in Table C3 are used to report the results in the main experiments.
C. D ETAILED CIFAR, SVHN RESULTS
Detalized versions of the results presented in the main text are shown in Table C4.
The corresponding λ M coefﬁcients are the same as in Table C3.
C.3. MNIST DETAILED RESULTS
In-domain performance Model selection scheme on MNIST was exactly the same as for the
CIFAR10/100.
We illustrate the relationship between λ M , accuracy, and the Adaptive Calibration error (ACE) in Figure C2.
One can see that λ M remains the same even when M is increasing.
Table C7 shows the detailed in-domain performance for the best λ M , equal to 0 .
[Figure]
[Table]
Out of distribution detection The ﬁnal OOD benchmark results on MNIST are summarized in Table C8.
One can see that on MNIST, our method yields a performance boost for any size of an ensemble, even for the very small ones ( M = 3 ).
C. A DDITIONAL RESULTS
Does the choice weighting distribution for the diversity term affect the results?
We considered
PreResNet164 trained on CIFAR100 with CIFAR10 as a weighting distribution dataset.
This approach can be seen as a form of outlier exposure technique, earlier proposed by Hendrycks et al.
The results on OOD benchmark (excluding CIFAR10) are shown in Table C9.
One can observe that the relationship between changing to a dataset of real images and the OOD detection performance is non-trivial.
While for some datasets outlier exposure did bring beneﬁt, for some other datasets, it did not.
Notably, the datasets on which the beneﬁt of outlier exposure was the best were datasets of real images, and we think that in case unlabeled images with non-overlapping class distribution are available, using them can bring beneﬁt.
Effect of model capacity While the main experiments in the paper were conducted using large models, we also investigated whether ensembles of smaller models can beneﬁt from our method.
We followed the same λ M selection procedure as for the main experiments in the paper, and report results for PreResNet20 in Table C10 for an ensemble of size M = 11 .
We found that the optimal λ M in this case is smaller compared to λ M for PreResNet164, and our method here does not yield any substantial boost over Deep Ensembles on these data.
We found that all the models with small capacity trained on CIFAR diverged with high λ M
Robustness to distribution shift As an additional evaluation, we investigated whether our method performs on par with DE under the distribution shift, to make sure that introduction of additional regularization did not affect the robustness properties.
We thus use a corrupted version of CIFAR test set released by (Hendrycks & Dietterich, 2019), and it has been recently shown that DE outperform many other methods on this benchmark (Ovadia et al., 2019).
Here, we report the results for
[Table]
VGG16BN and PreResNet164, as they yielded OOD performance gain in both SVHN and LSUN datasets.
One can see from Figure C3 that our method performs on-par with DE, as no statistical signiﬁcance in difference between methods can be concluded from this plot.
This further supports our claims that the developed greedy ensemble training approach works the same or on par with DE.
Effect of ensemble size on CIFAR We ran our experiments using PreResNet164 with on CIFAR10/100, having M ∈ { 3 , 5 , 7 } and λ M { 0 . 1 , 0 . 5 , 0 . 8 , 1 , 1 . 5 , 2 , 3 } .
Both in-domain accuracy, and the OOD detection on LSUN and SVHN are shown in Table C11.
The results in that table show that with small ensemble size, our method may yield better performance than DE on SVHN, abut even with an ensemble of size M = 3 , it has a substantial boost over DE in detecting LSUN.
Qualitative results on CIFAR100.
In Figure C4, we further illustrate the capabilities of uncertainty estimation of our method for the PreResNet164 trained on CIFAR100 ( M = 11 ).
Our method has a better true positive rate (as can also be seen from the histograms), and slightly better precision when the threshold for the recall is high.
We note that the histograms of epistemic uncertainties still overlap signiﬁcantly, however, with our method, the model does not have a high number of overconﬁdent predictions coming from the OOD data anymore.
[Table]
WideResNet28x
[Table]
[Figure]
[Table] [Table]
[Table]
[Table] [Table]
[Figure]