Original: D EEP E NSEMBLE AS A G AUSSIAN P ROCESS  P OSTERIOR
Revised: D EEP E NSEMBLE AS A G AUSSIAN P ROCESS A PPROXIMATE P OSTERIOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Ensemble(DE) is a ﬂexible, feasible, and effective alternative to Bayesian neural networks(BNNs) for uncertainty estimation in deep learning.
Revised: Deep Ensemble is a ﬂexible and effective alternative to Bayesian neural networks for uncertainty estimation in deep learning.

Original: However, DE is broadly criticized for lacking a proper Bayesian justiﬁcation.
Revised: However, Deep Ensemble is broadly criticized for lacking a proper Bayesian justiﬁcation.

Original: Some attempts try to ﬁx this issue, while they are typically coupled with a regression likelihood or rely on restrictive assumptions.
Revised: Some attempts try to ﬁx this issue, while they are typically coupled with the regression likelihood or rely on restrictive assumptions.

Original: In this work, we propose to deﬁne a Gaussian process (GP) approximate posterior with the ensemble members, based on which we perform variational inference directly inthe function space.
Revised: In this work, we propose to deﬁne a Gaussian process (GP) approximate posterior with Deep Ensemble, based on which we perform variational inference directly in function space.

Original: We further develop a function-space posterior regularization mechanism to properly incorporate prior knowledge.
Revised: We further develop a function-space posterior regularization mechanism to properly incorporate prior knowledge.

Original: We  demonstrate the algorithmic beneﬁts of variational inference in the GP family, and provide strategies to make the training feasible.
Revised: We provide strategies to make the training feasible, and demonstrate the algorithmic beneﬁts of variational inference in the GP family.

Original: As a result, our method consumes only marginally added training cost than the standard Deep Ensemble.
Revised: As a result, our method consumes only marginally added training cost than the standard Deep Ensemble.

Original: Empirically, our approach achieves better uncertainty estimation thanthe existing Deep Ensemble and its variants across diverse scenarios.
Revised: Empirically, our approach achieves better uncertainty estimation than Deep Ensemble and its variants across diverse scenarios.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bayesian treatment of deep neural networks (DNNs) is promised to enjoy principled Bayesian uncertainty while unleashing the capacity of DNNs, with Bayesian neural networks (BNNs) as popular examples (MacKay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).
Revised: Bayesian treatment of deep neural networks (DNNs) is promised to enjoy principled Bayesian uncertainty while unleashing the capacity of DNNs, with Bayesian neural networks (BNNs) as popular examples (MacKay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).

Original: Nevertheless, despite the surge of advance in BNNs (Louizos & Welling, 2016; Zhang et al., 2018), many of existing BNNs still face challenges in accurate and scalable inference (Sun et al., 2019), and exhibit limitations in uncertainty estimation and out-of-distribution robustness (Ovadia et al., 2019).
Revised: Nevertheless, despite the surge of advance in BNNs (Louizos & Welling, 2016; Zhang et al., 2018), many of existing BNNs still face challenges in accurate and scalable inference (Sun et al., 2019), and exhibit limitations in uncertainty estimation and out-of-distribution robustness (Ovadia et al., 2019).

Original: Alternatively, Deep Ensemble (DE) (Lakshminarayanan et al., 2017)independently trains multiple  randomly-initialized DNNs(referred to as basis functions hereinafter) for ensemble, presenting higher ﬂexibility, feasibility, and performance than standard Bayesian inference for DNNs. Despite the ability to explicitly explore the non-convex, multimodal loss landscape of DNNs (Fort et al., 2019;
Revised: Alternatively, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) trains multiple independent, randomly-initialized DNNs for ensemble, presenting higher ﬂexibility and effectiveness than BNNs. However, the “uncertainty” yielded by DE totally stems from the randomness in DNNs initialization and stochastic gradient descent (SGD), and hence it is just some kind of stochasticity instead of the

Original: Wilson & Izmailov, 2020), DE lacks a rigorous Bayesian justiﬁcation (He et al., 2020; Pearce et al., 2020).
Revised: Bayesian uncertainty. So, there is no guarantee that the uncertainty estimates given by DE are reliable.

Original: Some recent works like RMS (Lu & Van Roy, 2017; Osband et al., 2018; Pearce et al., 2020) and NTKGP (He et al., 2020) reﬁne DE to be a “sample-then-optimize” approachwhich directly yields posterior samples after training (Matthews et al., 2017).
Revised: To chase a Bayesian justiﬁcation for DE, recent works like RMS (Lu & Van Roy, 2017; Osband et al., 2018; Pearce et al., 2020) and NTKGP (He et al., 2020) reﬁne DE to be a “sample-then-optimize” approach (Matthews et al., 2017).

Original: However, such methods rely on a regression likelihood and often make strong assumptions like linearised and/or inﬁnite-width models.
Revised: Nonetheless, these works rely on the regression likelihood and often make strong assumptions like linearised and/or inﬁnite-width models.

Original: In this work, we endow DE with aprincipled learning criterion under the function-space Bayesian inference paradigm. In essence, the mixture ofdeltas (MoD) distribution over weights in DE induces a distribution over functions (i.e., a stochastic process). However, the singularity issue of MoD can yield an ill-deﬁned objective for Bayesian learning (Hron et al., 2018). To remediate the issue, we instead specify a Gaussian process (GP)with DE’s basis functions, analogous to deep kernel learning (Wilson et al., 2016), and expect it to approximate the true posterior.
Revised: To build a Bayesian reﬁnement of DE without reliance on restrictive assumptions, we propose to use DE to deﬁne a Gaussian process (GP) approximate posterior, which is dubbed as DE-GP for short.

Original: The imposition of a GP form on the approximate posterior is empowered by the equivalence between GPs andboth ﬁnite- and inﬁnite-width BNNs (Khan et al., 2019;Rudner et al., 2021; Neal, 1996; Lee et al., 2017;
Revised: The imposition of a GP form on the approximate posterior is empowered by the equivalence between GPs and BNNs (Khan et al., 2019; Neal, 1996; Lee et al., 2018; Garriga-Alonso et al., 2018;

Original: Garriga-Alonso et al., 2018; Matthews et al., 2018; Novak et al., 2018;
Revised: Matthews et al., 2018; Novak et al., 2018;

Original: He et al., 2020), as well as the associated computational advantages when adopting a GP prior without loss of generality.
Revised: He et al., 2020), as well as the associated computational beneﬁts when adopting a GP prior without loss of generality.

Original: Inspired by the Monte Carlo (MC) estimates for kernels (Rahimi et al., 2007), especially for neural network Gaussian process (NN-GP) kernels (Novak et al., 2018), we evaluate the empirical mean and covariance of the basis functions and use them to specify  the GP approximate posterior.
Revised: Concretely, we evaluate the empirical mean and covariance of the ensemble members and use them to specify DE-GP. This is inspired by the formula of neural network Gaussian process (NN-GP) (Novak et al., 2018).

Original: Namely, we deﬁne a parameteric and learnable GP kernel with ﬁnitely many and adaptive basis functions, in contrast to thetypical kernels like radial basis function (RBF) and NN-GP kernels built with inﬁnitely 
Revised: Nevertheless, we clarify that we deﬁne a parameteric and learnable GP kernel with ﬁnitely many and adaptive basis functions, in contrast to the NN-GP kernels built with inﬁnitely many yet ﬁxed basis functions.

Revised: The ﬂexibility of DE-GP is assured by the expressiveness of DNN ensemble members – by tuning them it is enabled to evolve over a rich variety of kernels.

Revised: Given these setups, we perform function-space variational inference (VI) for learning.

Revised: We maximize the functional evidence lower bound (fELBO) (Sun et al., 2019; Rudner et al., 2021) to push DE-GP towards the true posterior over functions.

Revised: We further present a posterior regularization scheme in

Original: T i m e p e r it e r a ti on ( s )
Revised: T i m e p e r it e r a ti on ( s )

Original: NN-GP
Revised: NN-GP

Original: DE-GP (our)
Revised: DE-GP (our)

Original: DE
Revised: DE

Original: RMS 0.
Revised: RMS 0.

Original: T i m e p e r it e r a ti on ( s )
Revised: T i m e p e r it e r a ti on ( s )

Original: DE-GPDERMS
Revised: DE-GPDERMS

Original: Training speed many yet ﬁxed basis functions.

Original: The ﬂexibility of the GP is assured by the expressiveness of DNN basis functions – by learning the basis functions it is enabled to evolve over a rich variety of kernels.

Original: Given these setups, we perform variational inference (VI) directly in function space.

Original: We maximize the functional evidence lower bound (fELBO) (Sun et al., 2019; Rudner et al., 2021) to push the Deep

Original: Ensemble based Gaussian process (DE-GP) towards the true posterior over functions.

Original: We further present a posterior regularization scheme in function space to conveniently incorporate structural priors, and present a concrete example on how to control function complexity.
Revised: Training speed function space to conveniently incorporate structural priors, and present a concrete example on how to control function complexity.

Original: By VI, we are enabled to handle classiﬁcation problems directly and exactly, without casting them into regression ones (He et al., 2020).
Revised: By VI, we are enabled to handle classiﬁcation problems directly and exactly, without casting them into regression ones (He et al., 2020).

Original: In practice, DE-GP can be easily implemented on top of a standard DE, while only introducing minimally added cost.
Revised: In practice, DE-GP can be easily implemented on top of a standard DE, while only introducing minimally added cost.

Original: Compared to DE and RMS, DE-GP overcomes the pathologies of posterior inference in the overparameterized weight space containing many symmetries (Sun et al., 2019; Wang et al., 2019), hence offering better uncertainty estimates, especially when adopting large network (see Fig. 1).
Revised: The function-space Bayesian inference enables DE-GP to ameliorate the pathologies induced by the over-parameterization nature of DNNs (Sun et al., 2019; Wang et al., 2019). Thus DE-GP can offer calibrated uncertainty estimates, especially when adopting large networks (see Fig. 1).

Original: Further, we empirically demonstrate that DE-GP outperforms strong baselines on various regression datasets, and presents superior uncertainty estimates and out-of-distribution robustness  in challenging classiﬁcation tasks.
Revised: We empirically demonstrate that DE-GP outperforms strong baselines on various regression datasets, and presents superior uncertainty estimates and out-of-distribution robustness without compromising accuracy in challenging classiﬁcation tasks.

Original: DE-GP also shows promise in solving contextual bandit problems, where the uncertainty plays a vital role in guiding exploration.  
Revised: DE-GP also shows promise in solving contextual bandit problems, where the uncertainty plays a vital role in guiding exploration.  R ELATED W ORK

Original: B ACKGROUND 2. B AYESIAN N EURAL N ETWORKS AND D EEP E NSEMBLE
Revised: Bayesian neural networks (BNNs). Bayesian treatment of DNNs is an emerging topic in deep learning yet with a long history (Mackay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011). BNNs can be learned by virtue of variational inference (Blundell et al., 2015; HernándezLobato & Adams, 2015; Louizos & Welling, 2016; Zhang et al., 2018; Khan et al., 2018; Deng et al., 2020), Laplace approximation (Mackay, 1992; Ritter et al., 2018), Markov chain Monte

Revised: Carlo (Welling & Teh, 2011; Chen et al., 2014; Zhang et al., 2019), particle-optimization based variational inference (Liu & Wang, 2016), Monte Carlo dropout (Gal & Ghahramani, 2016), and other methods (Maddox et al., 2019; Izmailov et al., 2020).

Revised: To avoid the difﬁculties of posterior inference in weight space, some recent works advocate performing Bayesian reasoning in function space (Sun et al., 2019; Rudner et al., 2021; Wang et al., 2019).

Revised: In function space, BNNs of inﬁnite or even ﬁnitely width equal to Gaussian processes (GPs) (Neal, 1996; Lee et al., 2018; Novak et al., 2018;

Revised: He et al., 2020; Khan et al., 2019), which provides valuable insights for this work.

Revised: Deep Ensemble (DE) (Lakshminarayanan et al., 2017) is a qualiﬁed alternative to BNNs for uncertainty estimation (Ovadia et al., 2019), yet lacks a proper Bayesian interpretation.

Revised: Wilson & Izmailov (2020) conceptually interpreted DE as a method that approximates the Bayesian posterior predictive distribution, but it is hard to judge whether the approximation is reliable or not in practice.

Revised: Recently, a suite of works based on the notion of “sample-then-optimize” (Matthews et al., 2017) have been developed to make DE Bayesian.

Revised: For example, RMS (Lu & Van Roy, 2017; Osband et al., 2018;

Revised: Pearce et al., 2020) regularizes the ensemble members towards randomised priors to obtain posterior samples after training, while it typically assumes linear data likelihood which is impractical for deep models and classiﬁcation tasks.

Revised: He et al.

Revised: (2020) proposed to add a randomised function to each ensemble member to realise a function-space Bayesian interpretation, but the method is asymptotically exact in the inﬁnite width limit and is limited to regressions.

Revised: By contrast, DE-GP works without restrictive assumptions.

Revised: In parallel, D’Angelo & Fortuin (2021) proposed to add a repulsive term to

Revised: DE to achieve Bayesian inference, which yet relies on unscalable gradient estimators.

Revised:  B ACKGROUND 3. B AYESIAN N EURAL N ETWORKS

Original: Consider learning on a dataset D
Revised: Consider learning on a dataset D

Original: =
Revised: =

Original: ( X , Y )
Revised: ( X , Y )

Original: = { ( x i , y i ) } ni =1 , with x i ∈ X as data and y i as
Revised: = { ( x i , y i ) } ni =1 , with x i ∈ X as data and y i as

Original: C -dimensional targets.
Revised: C -dimensional targets.

Original: Let g ( · , w ) :
Revised: Let g ( · , w ) :

Original: X → R C denote the function represented by a DNN with weights w .
Revised: X → R C denote the function represented by a DNN with weights w .

Original: Bayesian neural networks (BNNs) treat w as random variables and chase the posterior p ( w |D ) by imposing a prior p ( w ) .
Revised: Bayesian neural networks (BNNs) treat w as random variables and chase the posterior p ( w |D ) by imposing a prior p ( w ) .

Original: Among the family, variational BNNs (VBNNs) are particularly appealing as they cast posterior inference as an optimization problem (Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2016), where a variational distribution q ( w ) is introduced to approximate the true posterior by maximizing the evidence lower bound (ELBO):
Revised: Among the family, variational BNNs (VBNNs) are particularly appealing as they cast posterior inference as an optimization problem (Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2016), where a variational distribution q ( w ) is introduced to approximate the true posterior by maximizing the evidence lower bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original: Other popular BNN methods include Laplace approximation (Mackay, 1992), MC dropout (Gal &

Original: Ghahramani, 2016), Markov chain Monte Carlo (MCMC) (Neal, 1995; Welling & Teh, 2011), etc.

Original: BNNs marginalize over the posteriors to obtain the posterior predictive for new data x ∗ :
Revised: BNNs marginalize over the posteriors to obtain the posterior predictive for new data x ∗ :

Original: [Equation]
Revised: [Equation]

Original: where w s ∼ q ( w ) , s = 1 , ..., S .
Revised: where w s ∼ q ( w ) , s = 1 , ..., S .

Original: As expected, such a procedure propagates the embedded model uncertainty into the prediction.
Revised: Such a procedure is expected to propagate the embedded model uncertainty into the prediction.

Original: However in practice, most of the existing BNN approaches face obstacles in precise posterior inference due to non-trivial and convoluted posterior dependencies (Louizos & Welling, 2016; Zhang et al., 2018; Shi et al., 2018; Sun et al., 2019), and deliver unsatisfactory uncertainty estimation and out-of-distribution (OOD) robustness (Ovadia et al., 2019).
Revised: However in practice, most of the existing BNN approaches face obstacles in precise posterior inference due to non-trivial and convoluted posterior dependencies (Louizos & Welling, 2016; Zhang et al., 2018; Shi et al., 2018; Sun et al., 2019), and deliver unsatisfactory uncertainty estimation and out-of-distribution (OOD) robustness (Ovadia et al., 2019).

Revised: 3. D EEP E NSEMBLE

Original: As a workaround of BNNs, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) deploys a set ofM
Revised: As a workaround of BNNs, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) deploys a set of

Original:  DNNs(referred to as basis functions ) with weights {  w i  } Mi =1 for ensemble.
Revised: M DNNs { g ( · , w i ) } Mi =1 for ensemble.

Original: { w i } Mi =1 are independently trained to interpret the data from different angles, under maximum likelihood estimation (MLE) principle in standard DE, or maximum a posteriori (MAP) principle in regularized DE (rDE):
Revised: { w i } Mi =1 are independently trained to interpret the data from different angles, under maximum likelihood estimation (MLE) principle in standard DE, or maximum a posteriori (MAP) principle in regularized DE (rDE):

Original: [Equation]
Revised: [Equation]

Original: Due to the randomness in network initialization and stochastic gradient descent (SGD), DE can effectively explore the complex, non-convex loss landscape of DNNs (Fort et al., 2019; Wilson &
Revised: Due to the randomness in network initialization and stochastic gradient descent (SGD), DE can effectively explore the non-convex, multimodal loss landscape of DNNs (Fort et al., 2019; Wilson &

Original: Izmailov, 2020), thus shows appealing potential in various uncertainty estimation scenarios (Lakshminarayanan et al., 2017; Ovadia et al., 2019).
Revised: Izmailov, 2020), thus has shown promise in various uncertainty estimation tasks (Lakshminarayanan et al., 2017; Ovadia et al., 2019).

Original: However, DE lacks arigorous Bayesian justiﬁcation. To this end, a suite of works based on the notion of “sample-then-optimize” (Matthews et al., 2017) are developed. For example, RMS (Lu & Van Roy, 2017; Osband et al., 2018; Pearce et al., 2020) regularizes { w i } Mi =1 towards randomised priors to obtain posterior samples after training, while it typically assumes linear data likelihood which is impractical for deep models and classiﬁcation tasks. He et al.
Revised: However, due to the lack of a Bayesian justiﬁcation, there is no guarantee that the uncertainty estimates given by DE are reliable.

Original: (2020) proposed to add a randomised function to each basis function to realise a function-space Bayesian interpretation, while the method is asymptotically exact in the inﬁnite width limit and is limited to regressions.

Original: So, a proper Bayesian exposition for DE is still a paramount need.

Original: 2. R ELATION OF  B AYESIAN N EURAL N ETWORKS TO G AUSSIAN P ROCESSES
Revised: 3. R ELATION OF THE B AYESIAN P OSTERIORS OF DNN S TO G AUSSIAN P ROCESSES

Original: It is early shown that shallow BNNs converge to GPs in the inﬁnite width limit (Neal, 1996).
Revised: It is early shown that shallow BNNs converge to Gaussian processes (GPs) in the inﬁnite width limit (Neal, 1996).

Original: Recently, there is ongoing effort to extend the result to multiple layers (Lee et al., 2017; Matthews et al., 2018), convolutional architectures (Garriga-Alonso et al., 2018; Novak et al., 2018), and beyond (Yang, 2019).
Revised: Recently, there is ongoing effort to extend the result to multiple layers (Lee et al., 2018; Matthews et al., 2018), convolutional architectures (Garriga-Alonso et al., 2018; Novak et al., 2018), and beyond (Yang, 2019).

Original: Brieﬂy, an inﬁnitely wide BNN amount to the NN-GP: 
Revised: Brieﬂy, an inﬁnitely wide BNN amount to the NN-GP: GP

Original: [Equation]
Revised: [Equation]

Original: where p ( w ) is a layerwise isotropic Gaussian and the DNN specifying g has inﬁnite width.
Revised: [Equation]

Original: Notably, k is a matrix-valued kernel, with values in the space of C × C matrices.
Revised: [Equation] κ [Equation]

Original: Due to the i.i.d. weights, the kernel evaluation k ( x ∗ , x ∗ ) of some data x ∗ is a scaled identity matrix.
Revised: [Equation]

Original: The hierarchical nature of DNNs makes k a compositional kernel, which can be analytically estimated in some cases (Novak et al., 2018; 2019). Yet, the current implementations routinely suffer from scalability issues and struggle to handle modern network conﬁgurations. Thus, Novak et al.
Revised: [Equation]

Original: (2018) developed an MC estimate of k using ﬁnitely wide NNs, in a similar spirit to the classic random feature approaches (Rahimi et al., 2007).

Original: Formally, supposing that g comprises a feature projector h ( · , w ) :

Original: X → R ˆ C and a linear readout layer with weight variance σ 2 w and bias variance σ 2 b , the MC NN-GP kernel takes the form of k ( x , x (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ≈ [Equation] w [Equation]
Revised: where p ( w ) is a layerwise isotropic Gaussian and the DNN specifying g has inﬁnite width.

Original: k [Equation] [Equation] +
Revised: Notably, κ is a compositional, matrix-valued kernel, with values in the space of C × C matrices, and can be analytically estimated in some cases (Novak et al., 2018; 2019).

Original: [Equation] b [Equation] I C , with
Revised: Due to the i.i.d.

Original: [Equation] [Equation] where I C refers to the indentity matrix of size C × C and w s ∼ p ( w ) , s = 1 , ..., S .
Revised: weights, the kernel evaluation κ ( x ∗ , x ∗ ) of some data x ∗ is a scaled identity matrix.

Original: Based on the canonical results of GP regression, we know the posteriors of inﬁnitely wide BNNs are still GPs, corresponding to the Bayesian training of the last readout layer (Arora et al., 2019; Lee et al., 2019).
Revised: Based on the canonical results of GP regression, we know the posteriors of inﬁnitely wide BNNs are still GPs (in regression tasks), corresponding to the Bayesian training of the last readout layer (Arora et al., 2019; Lee et al., 2019).

Original: Despite a golden standard for small regression problems, analytical posterior inference for NN-GP faces challenges when handling contemporary DNN architectures (Novak et al., 2018), large data (Shi et al., 2019) and classiﬁcation problems.
Revised: Despite a golden standard for small regression problems, analytical posterior inference for NN-GP faces challenges when handling contemporary DNN architectures (Novak et al., 2018), large data (Shi et al., 2019) and classiﬁcation problems.

Original: Instead, we use DE to deﬁne a parametric, adaptive kernel in a similar vein to above MC kernels, and perform variational inference in function space, which is more ﬂexible and can unleash the potential of practically sized SGD-trained DNNs.

Original: It is also noteworthy that ﬁnitely wide BNNs with Gaussian posteriors on weights can  be connected toNeural Tangent Kernel (NTK)
Revised: Interestingly, ﬁnitely wide BNNs with Gaussian posteriors on weights can also be connected to

Original: (Jacot et al., 2018)

Original: based GPs (Khan et al., 2019; Rudner et al., 2021), providing supports for dictating a GP approximate posterior in this work. Yet, these methods make strong assumptions like local linearization, while our method bypasses this.
Revised: GPs (Khan et al., 2019; Rudner et al., 2021). We then arrive at the hypothesis that the Bayesian posteriors of DNNs are closely related to GPs. So, we specify a GP approximate posterior with DE.

Original:  M ETHODOLOGY 3. D EEP E NSEMBLE AS A G AUSSIAN P ROCESS  P OSTERIOR
Revised:  M ETHODOLOGY 4. D EEP E NSEMBLE AS A G AUSSIAN P ROCESS A PPROXIMATE P OSTERIOR

Original: Let g i denote g ( · , w i ) . DE actually deﬁnes a distribution q ( f ; w i ,..., w M ) = 1 M (cid:80) i δ ( f − g i ) over functions f : X → R C .
Revised: Basically, we utilize DE’s ensemble members (referred to as basis functions hereinafter) to deﬁne a

Original: To make DE a Bayesian method, a natural idea is to push q towards the true posterior over functions. However, the mixture of delta (MoD) distribution can yield ill-deﬁned objectives for approximate inference (Hron et al., 2018). To address this, we instead utilize the basis functions to deﬁne a kernel as well as a GP variational, in a similar spirit to NN-GP, as follows:q
Revised: GP approximate posterior as follows:

Original: [Equation]
Revised: [Equation]

Revised: where g i refers to g ( · , w i ) .

Original: As shown, the empirical mean and central covariance of ﬁnitely many, adaptive basis functions are leveraged to specifythe DE-GP.
Revised: Namely, the empirical mean and central covariance of ﬁnitely many, adaptive basis functions are leveraged to specify DE-GP.

Original: k q ( x , x (cid:48) ) is a linear , matrix-valued kernel with a small scaled identity matrix λ I C appended to avoid singularity.
Revised: k q ( x , x (cid:48) ) is a linear , matrix-valued kernel with a small scaled identity matrix λ I C appended to avoid singularity.

Original: The variations in k q ( x , x (cid:48) ) are conﬁned to having up to M − 1 rank, which is likely to be reasonable given the previous investigations showing that low-rank approximate posteriors for DNNs conjoin effectiveness and efﬁciency (Maddox et al., 2019; Izmailov et al., 2020; Dusenberry et al., 2020).
Revised: The variations in k q ( x , x (cid:48) ) are conﬁned to having up to M − 1 rank, echoing the recent investigations showing that low-rank approximate posteriors for DNNs conjoin effectiveness and efﬁciency (Maddox et al., 2019; Izmailov et al., 2020; Dusenberry et al., 2020).

Original: Note that the popular Nyström method (Williams & Seeger, 2001) also uses a low-rank matrix to approximate the original kernel matrix.
Revised: Note that the popular Nyström method (Williams & Seeger, 2001) also uses a low-rank matrix to approximate the original kernel matrix.

Original: Akin to the kernels in (Wilson et al., 2016), the DE-GP kernels are highly ﬂexible, and may automatically discover the underlying structures of high-dimensional data without manual participation.
Revised: Akin to the kernels in (Wilson et al., 2016), the DE-GP kernels are highly ﬂexible, and may automatically discover the underlying structures of high-dimensional data without manual participation.

Original: 3. V ARIATIONAL I NFERENCE IN F UNCTION S PACE
Revised: 4. V ARIATIONAL I NFERENCE IN F UNCTION S PACE

Original: Given these setups, we perform variational inference directly in function space.
Revised: We do function-space variational inference to push DE-GP towards the true posterior over functions.

Revised: Prior We have high freedom to determine which prior to use attributed to the variational inference paradigm.

Original: We specify the functional prior as the aforementioned NN-GPs p ( f ) =
Revised: Without loss of generality, we use the MC estimate of NN-GP (MC NN-GP) (Novak et al., 2018) as the prior due to its accessibility. Concretely, supposing a ﬁnitely wide DNN composed of a feature projector h ( · , w ) : X → R ˆ C and a linear readout layer with weight variance σ 2 w and bias variance σ 2 b , the MC NN-GP is deﬁned as p ( f ) =

Original: GP ( f | 0 , k
Revised: GP ( f | 0 , k

Original: ( x , x 
Revised: ( x , x (cid:48) )) , where k [Equation] , x

Original: [Equation]
Revised: [Equation]

Original: )) toenjoy the inductive bias carried by speciﬁc NN architectures.
Revised: , with [Equation] I C refers to the indentity matrix of size C × C and w s are i.i.d.

Original: 1 As will be shown later, the variational inference in  the GP family is much simpler than that using general stochastic processes.
Revised: samples from the Gaussian prior on weights p ( w ) . Similar setups can be found in some related works like (Wang et al., 2019). There may be a subtle difference between the MC NN-GP and the NN-GP, so what we are actually doing is not approximating the exact NN-GP posterior with DE-GP. Also of note that the NN-GP prior and the DE-GP posterior can be deﬁned with various architectures.

Original: Formally, DE-GP evolves by maximizing the functional ELBO (fELBO) (Sun et al., 2019):
Revised: fELBO Following Sun et al. (2019), we maximize the functional ELBO (fELBO) to optimize DE-GP:

Original: [Equation]
Revised: [Equation]

Original: fELBO involves the KL divergence between two GPs, which, on its own, is challenging to cope with, yet can be replaced by  the KL divergence between the marginal distributions of function evaluations. As proved by (Sun et al., 2019), with a measurement set ˜ X including all training inputs X , we have
Revised: Notably, there is a KL divergence between two GPs, which, on its own, is challenging to cope with. Fortunately, as proved by Sun et al. (2019), we can take the KL divergence between the marginal distributions of function evaluations as a substitute for it, giving rise to a more tractable objective:

Original: [Equation]
Revised: [Equation]

Original: where  f ˜ X is the concatenation of the vectorized outputs of f for ˜ X , i.e., f ˜ X ∈ R | ˜ X | C .In this way, we can obtain a tractable lower bound L for the log marginal likelihood.
Revised: where ˜ X denotes a measurement set including all training inputs X , and f ˜ X is the concatenation of the vectorized outputs of f for ˜ X , i.e., f ˜ X ∈ R | ˜ X | C .

Original: 3. P OSTERIOR R EGULARIZATION IN F UNCTION S PACE
Revised: 4. P OSTERIOR R EGULARIZATION IN F UNCTION S PACE

Original: Though Bayesian learning in function space enables a direct imposition of prior knowledge on function properties like smoothness and periodicity (Sun et al., 2019), sometimes the prior knowledge cannot be trivially designed as a prior distribution over functions.
Revised: Though Bayesian learning in function space enables a direct imposition of prior knowledge on function properties like smoothness and periodicity (Sun et al., 2019), sometimes the prior knowledge cannot be trivially designed as a prior distribution over functions.

Original: For example, the functions are periodical only over a limited input range, or the functions are Lipschitz continuous.
Revised: For example, the functions are periodical only over a limited input range, etc.

Original: In these cases, posterior regularization (Ganchev et al., 2010; Zhu et al., 2014) provides a principled workaround to impose structural constraints.
Revised: In these cases, posterior regularization (Ganchev et al., 2010; Zhu et al., 2014) provides a principled workaround to impose structural constraints.

Original: Brieﬂy, we apply posterior regularization to functional variational inference by solving:
Revised: Brieﬂy, we apply posterior regularization to functional variational inference by solving:

Original: [Equation]
Revised: [Equation]

Original: Q = { q ( f ) | E q ( f ) Ω( f ) ≤ 0 } is a valid set deﬁned in terms of a functional Ω which delivers some statistic of interest of a function.
Revised: Q = { q ( f ) | E q ( f ) Ω( f ) ≤ 0 } is a valid set deﬁned in terms of a functional Ω which delivers some statistic of interest of a function.

Original: 2 For tractable optimization, we slack the constraint as a penalty:
Revised: 1 For tractable optimization, we slack the constraint as a penalty:

Original: [Equation]
Revised: [Equation]

Original:  β is a trade-off coefﬁcient.
Revised: where β is a trade-off coefﬁcient.

Original: We next present an example on how to induce prior knowledge on function (hypothesis) complexity.
Revised: We next present an example on how to impose prior knowledge on function (hypothesis) complexity on the DE-GP approximate posterior using this paradigm.

Original: For the sake of brevity, we assume a binary classiﬁcation scenario in the following discussion where y ∈ {− 1 , 1 } and f, g i : X → R .
Revised: For the sake of brevity, we assume a binary classiﬁcation scenario in the following discussion where y ∈ {− 1 , 1 } and f, g i : X → R .

Original: We use 0 - 1 loss (cid:96) ( f ( x ) , y )
Revised: We use 0 - 1 loss (cid:96) ( f ( x ) , y )

Original: = 1 y (cid:54) =
Revised: = 1 y (cid:54) =

Original: sign ( f ( x )) to measure the classiﬁcation error on one datum.
Revised: sign ( f ( x )) to measure the classiﬁcation error on one datum.

Original: We assume an underlying distribution µ = µ ( x , y ) supported on X × {− 1 , 1 } for generating the training data D , based on which we can deﬁne the true risk of a function (hypothesis) f : R ( f ) :
Revised: We assume an underlying distribution µ = µ ( x , y ) supported on X × {− 1 , 1 } for generating the training data D , based on which we can deﬁne the true risk of a function (hypothesis) f : R ( f ) :

Original: = E ( x ,y ) ∼ µ (cid:96) ( f ( x ) , y ) .
Revised: = E ( x ,y ) ∼ µ (cid:96) ( f ( x ) , y ) .

Original: We set E q ( f ) Ω( f ) :
Revised: We set E q ( f ) Ω( f ) :

Original: =
Revised: =

Original: E q ( f ) R ( f ) in the seek of a posterior over functions that can generalize well.
Revised: E q ( f ) R ( f ) in the seek of a posterior over functions that can generalize well.

Original: It is easy to see that a hypothesis sample f ∼ q ( f )
Revised: By deﬁnition, a hypothesis sample f ∼ q ( f )

Original: =
Revised: =

Original: GP ( m q ( x ) , k q ( x , x (cid:48) )) can be decomposed asf ( x ) = 1 M (cid:80) Mi
Revised: GP ( m q ( x ) , k q ( x , x (cid:48) )) can be decomposed as

Original: =1 g i ( x ) + ζ ( x ) with ζ ( x ) ∼ GP (0 , k q ( x , x (cid:48) )) .
Revised: [Equation] + ζ ( x ) with ζ ( x ) ∼ GP (0 , k q ( x , x (cid:48) )) .

Original: Then, given the fact that a voting is incorrect when at least one of the individuals makes a mistake, we have
Revised: If sign ( f ( x )) (cid:54) = y , it is impossible that sign ( g 1 ( x )) = y , ..., sign ( g M ( x ))

Original: [Equation]
Revised: = y , and sign ( ζ ( x )) =

Original: , y 
Revised: y .

Revised: In other words,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Note that ζ ( x )can be re-parameterized as ζ ( x ) =
Revised: We can further re-parameterize ζ ( x ) as ζ ( x ) =

Original: [Equation]
Revised: [Equation]

Original: N (0 , 1) , i = 0 , ..., M , which is essentially a real-valued random function symmetric around 0.
Revised: N (0 , 1) , i = 0 , ..., M , which is essentially a real-valued random function symmetric around 0.

Original: Thus, for any ( x , y ) ∼ µ , we have E q ( f ) (cid:96) ( ζ ( x ) , y ) 
Revised: Thus, for any ( x , y ) ∼ µ , we have E q ( f ) (cid:96) ( ζ ( x ) , y ) =

Original: =
Revised: E

Original: E (cid:15) 0 ,...,(cid:15) M (cid:96) ( ζ ( x ) , y ) = 1 / 2 .
Revised: [Equation] [Equation] M (cid:96) ( ζ ( x ) , y ) = 1 / 2 .

Original: As a result,
Revised: As a result,

Original: [Equation]
Revised: [Equation]

Original: Namely, the expected generalization error of the approximately posteriori functions can be bounded from above by those of the DNN basis functions.
Revised: Namely, the expected generalization error of the approximately posteriori functions can be bounded from above by those of the DNN basis functions.

Original: 3 Recalling the theoretical and empirical results showing that DNNs’ generalization error R ( g i ) can be penalized by controlling model capacity in terms of norm-based regularization min w i
Revised: 3 Recalling the theoretical and empirical results showing that DNNs’ generalization error R ( g i ) can be decreased by controlling model capacity in terms of norm-based regularization min w i

Original: ||
Revised: ||

Original: w i
Revised: w i

Original: || 

Revised: ||

Original: (Neyshabur et al., 2015; 2017; Bartlett et al., 2017;
Revised: (Neyshabur et al., 2015; 2017; Bartlett et al., 2017;

Original: Jiang et al., 2019), we opt to maximize the following reﬁned fELBO for learning DE-GP:
Revised: Jiang et al., 2019), we opt to maximize the following reﬁned fELBO for learning DE-GP:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table] [Table]
Revised: [Equation]

Original: [Table]
Revised: [Figure]

Original: settles

Original: Intuitively, the weight space of an over-parameterized
Revised: Intuitively, the weight space of an over-parameterized

Original: DNN contains many redundant interpretations for the data, thus pushing the weights towards zero would not hurt data ﬁtting but boost generalization.
Revised: DNN contains many redundant interpretations for the data, thus pushing the weights towards zero would not hurt data ﬁtting but boost generalization.

Original: In practice, when processing with shallow architectures, β can be set as 0 to remove the constraint on model capacity; while when using deep architectures (e.g., ResNet (He et al., 2016)), we can set β according to commonly used weight decay coefﬁcient for better generalization.
Revised: In practice, when processing with shallow architectures, β can be set as 0 to remove the constraint on model capacity; while when using deep architectures (e.g., ResNet (He et al., 2016)), we can set β according to commonly used weight decay coefﬁcient for better generalization.

Original: Comparisons in Fig.
Revised: Comparisons in Fig.

Original: 2 and Table 1 support this.
Revised: 2 and Table 1 support this.

Original: 3. T HE A LGORITHM
Revised: 4. T HE A LGORITHM

Original: We outline the training procedure of DE-GP in Algorithm 1, and elaborate some details below. 
Revised: We outline the training procedure of DE-GP in Algorithm 1, and elaborate some details below. [Table]

Original: 3.4. M INI - BATCH T RAINING
Revised: 4.4. M INI - BATCH T RAINING

Original: In deep learning scenarios, DE-GP should proceed by mini-batch training.
Revised: In deep learning scenarios [Table] DE-GP should proceed by mini-batch training.

Original: At each step, we manufacture a stochastic measurement set with a mini-batch D s =
Revised: At each step [Table] we manufacture a stochastic measurement set with a mini-batch D s =

Original: ( X s , Y s ) from the training data D and random samples X ν from a continuous distribution ν
Revised: ( X s , Y s ) from the training data D and random samples X ν from a continuous distribution ν

Original: (e.g., a uniform distribution) supported on X .
Revised: (e.g., a uniform distribution) supported on X .

Original: Then, we adapt the objective deﬁned in Eq.
Revised: Then, we adapt the objective deﬁned in Eq.

Original: (13) to the following form:
Revised: (13) to the following form:

Original: [Equation]
Revised: [Equation]

Original: where ˜ X s indicates the union of X s and X ν .
Revised: where ˜ X s indicates the union of X s and X ν .

Original: When α = 1 and β = 0 , L DE-GP is a lower bound of log p ( D s ) according to (Sun et al., 2019).
Revised: When α = 1 and β = 0 , L DE-GP is a lower bound of log p ( D s ) according to (Sun et al., 2019).

Original: Yet, given that we do not perform intensive hyper-parameter tuning for the NN-GP prior, it is reasonable to make α a tunable hyper-parameter to better trade off between the data ﬁtting and the priori inductive bias.[Table] [Table]
Revised: Yet, given that we do not perform intensive hyper-parameter tuning for the NN-GP prior, it is reasonable to make α a tunable hyper-parameter to better trade off between the data ﬁtting and the priori inductive bias.

Original:  A N E XACT AND E FFICIENT E STIMATION OF THE M ARGINAL KL D IVERGENCE
Revised: 4.4. A N E XACT AND E FFICIENT E STIMATION OF THE M ARGINAL KL D IVERGENCE

Original: The marginal distributions of function evaluations are multivariate Gaussian by the deﬁnition of GP:
Revised: The marginal distributions of function evaluations are multivariate Gaussian by the deﬁnition of GP:

Original: [Equation]
Revised: [Equation]

Original: with the kernel matrices k ˜ X s , ˜ X s q , k ˜ X s , ˜ X s ∈ R | ˜
Revised: with the kernel matrices k ˜ X s , ˜ X s q , k ˜ X s , ˜ X s ∈ R | ˜

Original: X s | C ×|
Revised: X s | C ×|

Original: ˜
Revised: ˜

Original: X s | C as the joints of pair-wise outcomes.
Revised: X s | C as the joints of pair-wise outcomes.

Original: Therefore, the marginal KL divergence and its gradients can be estimated exactly without resorting to some approximations (Rudner et al., 2021).
Revised: Therefore, the marginal KL divergence and its gradients can be estimated exactly without resorting to some approximations (Rudner et al., 2021).

Original: What’s more, as discussed in Section 2.2, there is a simple structure in the prior kernel matrices, so we can write them in the form of Kronecker product:
Revised: What’s more, as discussed in Section 3.3, there is a simple structure in the prior kernel matrices, so we can write them in the form of Kronecker product:

Original: [Equation]
Revised: [Equation]

Original: where ˆk ˜ X s , ˜ X s ∈ R | ˜ X s
Revised: where ˆk ˜ X s , ˜ X s ∈ R | ˜ X s

Original: |×| ˜ X
Revised: |×| ˜ X

Original: s | corresponds to the evaluation of kernel ˆ k .
Revised: s | corresponds to the evaluation of kernel ˆ k .

Original: Hence we can exploit the property of Kronecker product to inverse k ˜ X s , ˜ X s in O ( | ˜ X s | 3 ) complexity.
Revised: Hence we can exploit the property of Kronecker product to inverse k ˜ X s , ˜ X s in O ( | ˜ X s | 3 ) complexity.

Original: Besides, as k ˜ X s , ˜ X s q is low-rank, we can leverage the Woodbury matrix identity (Woodbury, 1950) and matrix determinant lemma (Harville, 1998) to efﬁciently compute the inverse and determinant of k ˜ X s ,
Revised: Besides, as k ˜ X s , ˜ X s q is low-rank, we can leverage the Woodbury matrix identity (Woodbury, 1950) and matrix determinant lemma (Harville, 1998) to efﬁciently compute the inverse and determinant of k ˜ X s ,

Original: ˜ X s q in O ( | ˜ X s | CM 2 ) time complexity given that usually M (cid:28) | ˜
Revised: ˜ X s q in O ( | ˜ X s | CM 2 ) time complexity given that usually M (cid:28) | ˜

Original: X s | C
Revised: X s | C

Original: (e.g., 10 (cid:28) 256 C ).
Revised: (e.g., 10 (cid:28) 256 C ).

Original: 3. D ISCUSSION
Revised: 4. D ISCUSSION

Original: Diversity.
Revised: Diversity.

Original: The diversity among the ensemble members in function space is explicitly encouraged by the KL divergence between variational and the prior in fELBO.
Revised: The diversity among the ensemble members in function space is explicitly encouraged by the KL divergence between variational and the prior in fELBO.

Revised: Nonetheless, the expected loglikelihood in fELBO enforces each ensemble member to yield the same, correct outcomes for the training data.

Original: Arguably, the diversity on the points far away from the training datais further fostered, giving rise to calibrated uncertainty (seeFig.
Revised: Thereby, the diversity mainly exists in the regions far away from the training data (see

Original:  1 and Fig. 9 in Appendix B.2).
Revised: Fig. 1 and Fig. 9 in Appendix B.2).

Original: However, the diversity in DE does not have a clear theoretical support.
Revised: Yet, the diversity in DE does not have a clear theoretical support.

Original: Efﬁciency.
Revised: Efﬁciency.

Original: Compared to the overhead introduced by DNNs, the effort of estimating the KL divergence between Gaussians is negligible.
Revised: Compared to the overhead introduced by DNNs, the effort of estimating the KL divergence between Gaussians is negligible.

Original: The added cost of DE-GP primarily arises from the extra measurement points and the evaluation of the prior kernels.
Revised: The added cost of DE-GP primarily arises from the extra measurement points and the evaluation of the prior kernels.

Original: In practice, we use a small batch size  for the extra measurement points.
Revised: In practice, we use a small batch size [Figure] for the extra measurement points.

Original: We build the prior NN-GP kernels with cheap architectures and perform MC estimation in parallel.
Revised: We build the prior NN-GP kernels with cheap architectures and perform MC estimation in parallel.

Original: Eventually, DE-GP is only marginally slower than DE in training.
Revised: Eventually, DE-GP is only marginally slower than DE in training.

Original: Weight sharing.
Revised: Weight sharing.

Original: DE-GP does not care about how the basis functions are parameterized, so we can perform weight sharing among the basis functions, for example, using a shared feature extractor and
Revised: DE-GP does not care about how the basis functions are parameterized, so we can perform weight sharing among the basis functions, for example, using a shared feature extractor and

Original: M independent MLP classiﬁers to construct M basis functions (Deng et al., 2021).
Revised: M independent MLP classiﬁers to construct M basis functions (Deng et al., 2021).

Original: With shared weights, DE-GP is still likely to be reliable because our learning principle induces diversity in function space regardless of the weights.
Revised: With shared weights, DE-GP is still likely to be reliable because our learning principle induces diversity in function space regardless of the weights.

Original: Experiments in Section 5.3 validate this.
Revised: Experiments in Section 5.3 validate this.

Original: Limitations.
Revised: Limitations.

Original: Despite being Bayesian in principle, DE-GP is likely to be less ﬂexible than DE and other variants because all the involved basis functions need to be updated simultaneously.
Revised: Despite being Bayesian in principle, DE-GP is likely to be less ﬂexible than DE and other variants because all the involved basis functions need to be updated simultaneously.

Original: So, the ML practitioners need to make a trade-off between theoretical soundness and ﬂexibility in practical usage.

Original: R ELATED W ORK

Original: Bayesian neural networks (BNNs).

Original: Bayesian treatment of DNNs is an emerging topic in deep learning yet with a long history (Mackay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).

Original: BNNs can be learned by virtue of variational inference (Blundell et al., 2015; HernándezLobato & Adams, 2015; Louizos & Welling, 2016; Zhang et al., 2018; Khan et al., 2018;

Original: Deng

Original: [Figure]

Original: et al., 2020), Laplace approximation (Mackay, 1992; Ritter et al., 2018), Markov chain Monte

Original: Carlo (Welling & Teh, 2011; Chen et al., 2014; Zhang et al., 2019), particle-optimization based variational inference (Liu & Wang, 2016), Monte Carlo dropout (Gal & Ghahramani, 2016), and other methods (Maddox et al., 2019; Izmailov et al., 2020).

Original: To avoid the difﬁculties of posterior inference in weight space, some recent works advocate performing Bayesian reasoning in function space (Sun et al., 2019; Rudner et al., 2021; Wang et al., 2019).

Original: In function space, BNNs of inﬁnite or even ﬁnitely width equal to Gaussian processes (GPs) (Neal, 1996; Lee et al., 2017; Novak et al., 2018;

Original: He et al., 2020; Khan et al., 2019), which provides valuable insights for this work.

Original: Deep Ensemble (DE) (Lakshminarayanan et al., 2017) is a qualiﬁed alternative to BNNs for uncertainty estimation (Ovadia et al., 2019), yet lacks a proper Bayesian interpretation.

Original: Some works regard DE as a “sample-then-optimize” (Matthews et al., 2017) method, but usually rely on restrictive assumptions (Lu & Van Roy, 2017; Osband et al., 2018; Pearce et al., 2020;

Original: He et al., 2020).

Original: Having identiﬁed that DE speciﬁes a GP, we offer a more proper Bayesian exposition for DE.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate DE-GP in multiple scenarios to prove that DE-GP yields better uncertainty estimates than competing baselines, while preserving non-degraded predictive performance.
Revised: We perform extensive evaluation to prove that DE-GP yields better uncertainty estimates than the baselines, while preserving non-degraded predictive performance.

Original: The baselines include DE, rDE, NN-GP, RMS, etc.
Revised: The baselines include DE, rDE, NN-GP, RMS, etc.

Original: In all experiments, we estimate the prior kernel with 10 MC samples and set the sampling distribution for extra measurement points ν as the uniform distribution over the data region.
Revised: In all experiments, we estimate the prior kernel with 10 MC samples and set the sampling distribution for extra measurement points ν as the uniform distribution over the data region.

Original: Code is provided in the supplementary material.

Original: 5. I LLUSTRATIVE R EGRESSION
Revised: 5. I LLUSTRATIVE R EGRESSION

Original: We build two regression problems with data from y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) and y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , respectively.
Revised: We build two regression problems with data from y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) and y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , respectively.

Original: We consider two architectures: MLPs with 2 hidden layers ofunits and MLPs with 3 hidden layers of 512 units, where ReLU activation is used.
Revised: We consider two architectures: MLPs with 2 hidden layers ofunits and MLPs with 3 hidden layers of 512 units, where ReLU activation is used.

Original: For NN-GP, we analytically estimate the GP kernel and perform GP regression without training MLPs.
Revised: For NN-GP, we analytically estimate the GP kernel and perform GP regression without training MLPs.

Original: For DE-GP,
Revised: For DE-GP,

Original: DE, rDE, and RMS, we train 50 MLPs.
Revised: DE, rDE, and RMS, we train 50 MLPs.

Original: For MC dropout, we train a MLP with the commonly used 0.2 dropout rate.
Revised: For MC dropout, we train a MLP with the commonly used 0.2 dropout rate.

Original: We use the default values for α and β , i.e., α = 1 and β = 0 .
Revised: We use the default values for α and β , i.e., α = 1 and β = 0 .

Original: Fig.
Revised: Fig.

Original: 1 and Appendix B.2 show the results on the two problems.
Revised: 1 and Appendix B.2 show the results on the two problems.

Original: We also provide the comparison on training speed in Fig.
Revised: We also provide the comparison on training speed in Fig.

Original: As shown, DE-GP delivers calibrated uncertainty estimates across settings, on par with the non-parametric Bayesian baseline NN-GP.
Revised: As shown, DE-GP delivers calibrated uncertainty estimates across settings, on par with the non-parametric Bayesian baseline NN-GP.

Original: Yet, the baselines like DE, rDE, and RMS suffer from degeneracy issue as the dimension of weights increases.
Revised: Yet, the baselines like DE, rDE, and RMS suffer from degeneracy issue as the dimension of weights increases.

Original: Though NN-GP outperforms other methods, the involved analytical GP regression may have scalability and effectiveness issues when facing modern architectures (Novak et al., 2018), while DE-GP does not suffer from them.
Revised: Though NN-GP outperforms other methods, the involved analytical GP regression may have scalability and effectiveness issues when facing modern architectures (Novak et al., 2018), while DE-GP does not suffer from them.

Original: 5. UCI R EGRESSION
Revised: 5. UCI R EGRESSION

Original: Suggested by existing works (Hernández-Lobato & Adams, 2015; Pearce et al., 2020), we then assess

Original:  DE-GP on 5 UCI real-valued regression problems.
Revised: We then assess DE-GP on 5 UCI real-valued regression problems.

Original: The used architecture is a MLP with 2 hidden layers of 256 units and ReLU activation.
Revised: The used architecture is a MLP with 2 hidden layers of 256 units and ReLU activation.

Original: 10 networks are trained for DE, DE-GP and other variants.
Revised: 10 networks are trained for DE, DE-GP and other variants.

Original: For DE-GP, we set β = [-
 (stated in Section 3.3)-] and tune α according to validation sets.
Revised: For DE-GP, we set β = [-
 (stated in Section 3.3)-]0 and tune α according to validation sets.

Original: We perform cross validation with 5 splits.
Revised: We perform cross validation with 5 splits.

Original: Fig.
Revised: Fig.

Original: 3 shows the results.
Revised: 3 shows the results.

Original: DE-GP surpasses or approaches the baselines across scenarios in aspects of both test negative log-likelihood (NLL) and test root mean square error (RMSE).
Revised: DE-GP surpasses or approaches the baselines across scenarios in aspects of both test negative log-likelihood (NLL) and test root mean square error (RMSE).

Original: Notably, DE-GP’s merits are prominent on the large-scale Protein dataset.
Revised: DE-GP even beats NN-GP, which is probably attributed to that the variational family speciﬁed by DE enjoys the beneﬁcial inductive bias of practically sized SGD-trained DNNs, and DE-GP can ﬂexibly trade off between the likelihood and the prior by tuning α .

Original: 5. I MAGE C LASSIFICATION ON F ASHION -MNIST AND CIFAR- 
Revised: 5. I MAGE C LASSIFICATION ON F ASHION -MNIST AND CIFAR-

Original: In the classiﬁcation experiments, we augment the data log-likelihood (i.e., the ﬁrst term in Eq.
Revised: In the classiﬁcation experiments, we augment the data log-likelihood (i.e., the ﬁrst term in Eq.

Original: (14)) with a trainable temperature to tackle oversmoothing and avoid underconﬁdence.
Revised: (14)) with a trainable temperature to tackle oversmoothing and avoid underconﬁdence.

Revised: Fashion-MNIST .

Revised: We use a widened LeNet5 architecture with batch normalizations (BNs) (Ioffe &

Revised: Szegedy, 2015) for the Fashion-MNIST dataset (Xiao et al., 2017).

Revised: Considering the inefﬁciency of

Original: 2 3 4 5 6 7 8 9 10 Ensemble size 6.
Revised: 2 3 4 5 6 7 8 9 10 Ensemble size 6.

Original: C l a ss i f
Revised: C l a ss i f

Original: i c a t i o n
Revised: i c a t i o n

Original: E rr o r ( % )
Revised: E rr o r ( % )

Original: Test Error vs. Ensemble Size DE-GP
Revised: Test Error vs. Ensemble Size DE-GP

Original: DErDERMS 2 3 4 5 6 7 8 9 10 Ensemble size 0.
Revised: DErDERMS 2 3 4 5 6 7 8 9 10 Ensemble size 0.

Original: N e g a t
Revised: N e g a t

Original: i v e L o g - li k e li h
Revised: i v e L o g - li k e li h

Original: oo d
Revised: oo d

Original: Test NLL vs. Ensemble Size DE-GP
Revised: Test NLL vs. Ensemble Size DE-GP

Original: DErDERMS 1.
Revised: DErDERMS 1.

Original: 0.2 Uncertainty Threshold 0.
Revised: 0.2 Uncertainty Threshold 0.

Original: E rr
Revised: E rr

Original: o r o n e x a m p l e s w i
Revised: o r o n e x a m p l e s w i

Original: t
Revised: t

Original: h un c e r t a i n t y
Revised: h un c e r t a i n t y

Original: FashionMNIST+MNIST Error vs. Uncertainty
Revised: FashionMNIST+MNIST Error vs. Uncertainty

Original: DE-GP DE rDE RMS
Revised: DE-GP DE rDE RMS

Original: [Table]
Revised: [Table]

Original: Fashion-MNIST .

Original: We use a widened LeNet5 architecture with batch normalizations (BNs) (Ioffe &

Original: Szegedy, 2015) for the Fashion-MNIST dataset (Xiao et al., 2017).

Original: Considering the inefﬁciency of

Original: NN-GP, we mainly compare DE-GP to DE, rDE, and RMS.
Revised: NN-GP, we mainly compare DE-GP to DE, rDE, and RMS.

Original: We set α as well as the regularization coefﬁcients for rDE and RMS all as 0 .
Revised: We set α as well as the regularization coefﬁcients for rDE and RMS all as 0 .

Original: 1 according to validation accuracy.
Revised: 1 according to validation accuracy.

Original: For DE-GP, we use β =given the limited capacity of the architecture.
Revised: For DE-GP, we use β =given the limited capacity of the architecture.

Original: The in-distribution performance is averaged overruns.
Revised: The in-distribution performance is averaged over 3 runs.

Original: Fig. 4-(Left) and Fig. 4-(Middle) display how ensemble size impacts the test results.
Revised: Fig. 4-(Left) and Fig. 4-(Middle) display how ensemble size impacts the test results.

Original: Notably, the test error of DE-GP is consistently superior to the baselines with statistical evidence.
Revised: Surprisingly, the test error of DE-GP is even lower than the baselines.

Original: Besides, to compare the quality of uncertainty estimates, we use the trained models to make prediction and quantify epistemic uncertainty for both the in-distribution test set and the out-of-distribution (OOD) MNIST test set.
Revised: Besides, to compare the quality of uncertainty estimates, we use the trained models to make prediction and quantify epistemic uncertainty for both the in-distribution test set and the out-of-distribution (OOD) MNIST test set.

Original: All predictions on OOD data are regarded as wrong.
Revised: All predictions on OOD data are regarded as wrong.

Original: The epistemic uncertainty is estimated by the mutual information between the prediction and the variable function:
Revised: The epistemic uncertainty is estimated by the mutual information between the prediction and the variable function:

Original: [Equation]
Revised: [Equation]

Original: where H indicates Shannon entropy, with f s = g ( · , w s ) for DE, rDE, and RMS, and f s ∼ q ( f ; w 1 , ..., w M ) for DE-GP.
Revised: where H indicates Shannon entropy, with f s = g ( · , w s ) for DE, rDE, and RMS, and f s ∼ q ( f ; w 1 , ..., w M ) for DE-GP.

Original: We normalize the uncertainty estimates into [0 , 1] .
Revised: We normalize the uncertainty estimates into [0 , 1] .

Original: For each threshold τ ∈ [0 , 1] , we plot the average test error for data with ≤ τ uncertainty in Fig. 4-(Right).
Revised: For each threshold τ ∈ [0 , 1] , we plot the average test error for data with ≤ τ uncertainty in Fig. 4-(Right).

Original: It is prominent that under various uncertainty thresholds, DE-GP makes fewer mistakes than the baselines, implying that DE-GP succeeds to assign relatively higher uncertainty for the OOD data.
Revised: It is prominent that under various uncertainty thresholds, DE-GP makes fewer mistakes than the baselines, implying that DE-GP succeeds to assign relatively higher uncertainty for the OOD data.

Original: CIFAR-10 .
Revised: CIFAR-10 .

Original: Next, we apply DE-GP to the real-world image classiﬁcation task CIFAR-10 (Krizhevsky et al., 2009).
Revised: Next, we apply DE-GP to the real-world image classiﬁcation task CIFAR-10 (Krizhevsky et al., 2009).

Original: We consider the popular ResNet architectures (He et al., 2016) including ResNetand ResNet-56.
Revised: We consider the popular ResNet architectures (He et al., 2016) including ResNetand ResNet-56.

Original: The ensemble size is ﬁxed as 10 .
Revised: The ensemble size is ﬁxed as 10 .

Original: We split the data as training set, validation set, and test set of size 45000, 5000, 10000, respectively.
Revised: We split the data as training set, validation set, and test set of size 45000, 5000, 10000, respectively.

Original: We set β = 0 .
Revised: We set β = 0 .

Original: 1 , equivalent with the regularization coefﬁcient on weight in rDE.
Revised: 1 , equivalent with the regularization coefﬁcient on weight in rDE.

Original: We select α from { 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 } according to validation performance.

Original: We use a lite ResNet-20 architecture without BNs and residual connections to set up the prior NN-GP kernel for both the ResNet-20 and ResNet-56 based variational posteriors.
Revised: We use a lite ResNet-20 architecture without BNs and residual connections to set up the prior NN-GP kernel for both the ResNet-20 and ResNet-56 based variational posteriors.

Original: We present the in-distribution test accuracy in Table 1 and the error versus uncertainty plots on the combination of CIFAR-10 and SVHN test sets in Fig.
Revised: We present the in-distribution test accuracy in Table 1 and the error versus uncertainty plots on the combination of CIFAR-10 and SVHN test sets in Fig.

Original: It is noteworthy that DE-GP isbetter than or on par with the practically-used, competing rDE in aspect of test accuracy.
Revised: It is noteworthy that DE-GP is on par with the practically-used, competing rDE in aspect of test accuracy.

Original: DE-GP ( β = 0 ) shows unsatisfactory test accuracy, verifying the necessity of performing posterior regularization to penalize function complexity when using ultra-deep networks.
Revised: DE-GP ( β = 0 ) shows unsatisfactory test accuracy, verifying the necessity of performing posterior regularization to penalize function complexity when using ultra-deep networks.

Original: The error versus uncertainty plots are similar to those for Fashion-MNIST, substantiating the universality of DE-GP.
Revised: The error versus uncertainty plots are similar to those for Fashion-MNIST, substantiating the universality of DE-GP.

Original: We further test the trained methods on CIFAR-10 corruptions (Hendrycks & Dietterich, 2019), a challenging OOD generalization/robustness benchmark for deep models.
Revised: We further test the trained methods on CIFAR-10 corruptions (Hendrycks & Dietterich, 2018), a challenging OOD generalization/robustness benchmark for deep models.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6 and
Revised: 6 and

Original: Appendix B.3, DE-GP reveals smaller Expected Calibration Error (ECE) (Guo et al., 2017) and lower NLL at various levels of skew, reﬂecting its ability to make conservative predictions under corruptions and good OOD robustness.
Revised: Appendix B.3, DE-GP reveals smaller Expected Calibration Error (ECE) (Guo et al., 2017) and lower NLL at various levels of skew, reﬂecting its ability to make conservative predictions under corruptions and good OOD robustness.

Original: More results for the deeper ResNet-110 architecture are provided in Fig.
Revised: More results for the deeper ResNet-110 architecture are provided in Fig.

Original: 12 in Appendix B.3.
Revised: 12 in Appendix B.3.

Original: [Figure]
Revised: [Figure]

Revised: x

Revised: [Figure]

Original: p l
Revised: p l

Original: [Figure]
Revised: [Figure]

Original: i t
Revised: i t

Original: [Figure]
Revised: [Figure]

Original: t a
Revised: t a

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: r

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Weight Sharing.
Revised: Weight Sharing.

Original: We build a ResNet-20 with 10 classiﬁcation heads and a shared feature extraction module to evaluate the methods under weight sharing.
Revised: We build a ResNet-20 with 10 classiﬁcation heads and a shared feature extraction module to evaluate the methods under weight sharing.

Original: We set a larger value for α for DE-GP to induce higher magnitudes of functional diversity.
Revised: We set a larger value for α for DE-GP to induce higher magnitudes of functional diversity.

Original: The test accuracy  and error versus uncertainty plots on CIFAR-10 are illustrated in Fig.
Revised: The test accuracy (over 6 trials) and error versus uncertainty plots on CIFAR-10 are illustrated in Fig.

Original: We exclude RMS from the comparison as it assumes i.i.d.
Revised: We exclude RMS from the comparison as it assumes i.i.d.

Original: basis functions which may be incompatible with weight sharing.
Revised: basis functions which may be incompatible with weight sharing.

Original: DE-GP beneﬁts from the diversity in function space, hence performs better than DE and rDE, which purely hinge on the diversity in weight space.
Revised: DE-GP beneﬁts from the diversity in function space, hence performs better than DE and rDE, which purely hinge on the diversity in weight space.

Original: 5. C ONTEXTUAL B ANDIT
Revised: 5. C ONTEXTUAL B ANDIT

Original: [Figure]
Revised: [Figure]

Original: Finally, we apply DE-GP to contextual bandit, an important decision-making task where the uncertainty helps to guide exploration.
Revised: Finally, we apply DE-GP to contextual bandit, an important decision-making task where the uncertainty helps to guide exploration.

Original: Following (Wang et al., 2019), we plug
Revised: Following (Wang et al., 2019), we plug

Original: DE-GP into the BNNs
Revised: DE-GP into the BNNs

Original: based Thompson sampling algorithm.
Revised: based Thompson sampling algorithm.

Original: We reuse most of the settings for UCI regression.
Revised: We reuse most of the settings for UCI regression.

Original: We leverage the GenRL library to build two contextual bandit problems Covertype and Mushroom (Riquelme et al., 2018).
Revised: We leverage the GenRL library to build two contextual bandit problems Covertype and Mushroom (Riquelme et al., 2018).

Original: The cumulative reward is depicted in Fig.
Revised: The cumulative reward is depicted in Fig.

Original: As desired, DE-GP offers better uncertainty estimates and hence beats the baselines by clear margins.
Revised: As desired, DE-GP offers better uncertainty estimates and hence beats the baselines by clear margins.

Original: The potential of DE-GP in more reinforcement learning and Bayesian optimization scenarios deserves future investigation.
Revised: The potential of DE-GP in more reinforcement learning and Bayesian optimization scenarios deserves future investigation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we attempt to establish aproper Bayesian justiﬁcation for Deep Ensemble by drawing inspiration from the connection between BNNs and GPs.
Revised: In this work, we attempt to build a Bayesian reﬁnement of Deep Ensemble by drawing inspiration from the connection between BNNs and GPs.

Original: We propose to leverage the ensemble members to specify an adaptive GP approximate posterior, and perform variational inference directly in function space.
Revised: We propose to leverage the ensemble members to specify an adaptive GP approximate posterior, and perform variational inference directly in function space.

Original: We inherit some theoretical results from existing works on functional variational inference, and further develop a posterior regularization scheme to conveniently induce prior knowledge on function properties.
Revised: We inherit some theoretical results from existing works on functional variational inference, and further develop a posterior regularization scheme to conveniently induce prior knowledge on function properties.

Original: We demonstrate how to penalize the function complexity based on this scheme and empirically show the necessity of doing so.
Revised: We demonstrate how to penalize the function complexity based on this scheme and empirically show the necessity of doing so.

Original: The whole algorithm can be implemented easily and efﬁciently.
Revised: The whole algorithm can be implemented easily and efﬁciently.

Original: Extensive experiments validate the effectiveness of our method.
Revised: Extensive experiments validate the effectiveness of our method.

Original: We hope this work may shed light on the development of more useful and more graceful BNNs.
Revised: We hope this work may shed light on the development of better Bayesian deep learning approaches.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This work proposes a Bayesian reﬁnement of Deep Ensemble.
Revised: This work proposes a Bayesian reﬁnement of Deep Ensemble.

Original: Its potential positive impacts in the society are evident: its ability to enable better uncertainty estimation while maintaining predictive performance is crucial in industry, e g., automatic driving, disease analysis, and ﬁnancial applications.
Revised: Its potential positive impacts in the society are evident: its ability to enable better uncertainty estimation while maintaining predictive performance is crucial in industry, e g., automatic driving, disease analysis, and ﬁnancial applications.

Original: In this scenarios the uncertainty estimates could be used to reject uncertain predictions and raise the requirement of inviting humans into the decision process.
Revised: In this scenarios the uncertainty estimates could be used to reject uncertain predictions and raise the requirement of inviting humans into the decision process.

Original: As a fundamental research in machine learning, the negative consequences are not obvious.
Revised: As a fundamental research in machine learning, the negative consequences are not obvious.

Original: Though in theory any technique can be misused, it is not likely to happen at the current stage
Revised: Though in theory any technique can be misused, it is not likely to happen at the current stage

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: It is easy to reproduce the experiments given the code in the supplementary material.
Revised: It is easy to reproduce the experiments given the code in the supplementary material.

Original: A P ENALIZE F UNCTION C OMPLEXITY IN M ULTI - CLASS C LASSIFICATION
Revised: A P ENALIZE F UNCTION C OMPLEXITY IN M ULTI - CLASS C LASSIFICATION

Original: In the multi-class classiﬁcation scenario where y ∈ { 1 , 2 , ..., C } and f, g i : X → R C , we use the loss (cid:96) ( f ( x ) , y )
Revised: In the multi-class classiﬁcation scenario where y ∈ { 1 , 2 , ..., C } and f, g i : X → R C , we use the loss (cid:96) ( f ( x ) , y )

Original: =
Revised: =

Original: 1 f ( x )[ y ] < max y (cid:48)(cid:54) =
Revised: 1 f ( x )[ y ] < max y (cid:48)(cid:54) =

Original: y f ( x )[ y (cid:48) ] to measure prediction error where f ( x )[ j ] denotes j -th coordinate of f ( x ) .
Revised: y f ( x )[ y (cid:48) ] to measure prediction error where f ( x )[ j ] denotes j -th coordinate of f ( x ) .

Original: The distinct difference between this scenario and the binary classiﬁcation scenario is that in this setting, ζ ( x ) is a vector-valued function:
Revised: The distinct difference between this scenario and the binary classiﬁcation scenario is that in this setting, ζ ( x ) is a vector-valued function:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) 0 ∼ N ( 0 , I C ) and (cid:15) i ∼ N (0 , 1) , i = 1 , ..., M .
Revised: where (cid:15) 0 ∼ N ( 0 , I C ) and (cid:15) i ∼ N (0 , 1) , i = 1 , ..., M .

Original: We then make a mild assumption to simplify the analysis.
Revised: We then make a mild assumption to simplify the analysis.

Original: Assumption 1.
Revised: Assumption 1.

Original: For any ( x , y ) ∈ µ , the elements on the diagonal of k q ( x , x ) have the same value.
Revised: For any ( x , y ) ∈ µ , the elements on the diagonal of k q ( x , x ) have the same value.

Original: This assumption implies that for any j, j (cid:48) ∈ { 1 , ..., C } ,
Revised: This assumption implies that for any j, j (cid:48) ∈ { 1 , ..., C } ,

Original: [Equation]
Revised: [Equation]

Original: I.e.,
Revised: I.e.,

Original: [Equation]
Revised: [Equation]

Original: Therefore, ζ ( x ) possesses the same variance across its output coordinates and becomes a random guess classiﬁer.
Revised: Therefore, ζ ( x ) possesses the same variance across its output coordinates and becomes a random guess classiﬁer.

Original: Based on this, we have E q ( f ) (cid:96) ( ζ ( x ) , y )
Revised: Based on this, we have E q ( f ) (cid:96) ( ζ ( x ) , y )

Original: =
Revised: =

Original: E (cid:15) 0 ,...,(cid:15) M (cid:96) ( ζ ( x ) , y )
Revised: E (cid:15) 0 ,...,(cid:15) M (cid:96) ( ζ ( x ) , y )

Original: =
Revised: =

Original: ( C − 1) /C .
Revised: ( C − 1) /C .

Original: We can then derive a similar conclusion to that in Section 3.3.
Revised: We can then derive a similar conclusion to that in Section 4.3.

Original: The validity of Assumption 1.
Revised: The validity of Assumption 1.

Original: When the data dimension |X| is high and the number of training data n is ﬁnitely large, with zero probability the sampled data ( x , y ) ∼ µ resides in the training set.
Revised: When the data dimension |X| is high and the number of training data n is ﬁnitely large, with zero probability the sampled data ( x , y ) ∼ µ resides in the training set.

Original: Therefore, only the KL divergence term of the fELBO explicitly affects the predictive uncertainty at x .
Revised: Therefore, only the KL divergence term of the fELBO explicitly affects the predictive uncertainty at x .

Original: Because the NN-GP prior possesses a diagonal structure, it is hence reasonable to make the above assumption.
Revised: Because the NN-GP prior possesses a diagonal structure, it is hence reasonable to make the above assumption.

Original: B M ORE OF E XPERIMENTS
Revised: B M ORE OF E XPERIMENTS

Original: We provide more experimental details and results in this section.
Revised: We provide more experimental details and results in this section.

Original: In all experiments, the number of
Revised: In all experiments, the number of

Original: MC samples for estimating the expected log-likelihood (i.e., U in Line 7 of Algorithm 1) is set as 256 .
Revised: MC samples for estimating the expected log-likelihood (i.e., U in Line 7 of Algorithm 1) is set as 256 .

Original: Unless otherwise stated, we set the regularization constant λ as 0 .
Revised: Unless otherwise stated, we set the regularization constant λ as 0 .

Original: 05 times of the average eigenvalue of the central covariance matrices, and set the weight and bias variance for deﬁning the NN-GP prior kernel at each layer as 2 /fan _ in and 0 .
Revised: 05 times of the average eigenvalue of the central covariance matrices, and set the weight and bias variance for deﬁning the NN-GP prior kernel at each layer as 2 /fan _ in and 0 .

Original: 01 , where fan _ in is the number of input features, as suggested by He initialization (He et al., 2015).
Revised: 01 , where fan _ in is the number of input features, as suggested by He initialization (He et al., 2015).

Original: B. D ETAILED S ETTINGS
Revised: B. D ETAILED S ETTINGS

Original: Illustrative regression.
Revised: Illustrative regression.

Original: For the problem on y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) , we randomly sampledata points from [ − 1 . 5 , − 0 . 6] ∪ [0 . 6 , 1 . 5] .
Revised: For the problem on y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) , we randomly sampledata points from [ − 1 . 5 , − 0 . 6] ∪ [0 . 6 , 1 . 5] .

Original: For the problem on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , we use 6 data points {− 0 .
Revised: For the problem on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , we use 6 data points {− 0 .

Original: 8 , − 0 .
Revised: 8 , − 0 .

Original: 8 } following (Pearce et al., 2020).
Revised: 8 } following (Pearce et al., 2020).

Original: For optimizing the
Revised: For optimizing the

Original: DNN basis functions, we use an Adam (Kingma & Ba, 2014) optimizer with 0 .
Revised: DNN basis functions, we use an Adam (Kingma & Ba, 2015) optimizer with 0 .

Original: 003 learning rate.
Revised: 003 learning rate.

Original: The optimization takes 300 iterations.
Revised: The optimization takes 300 iterations.

Original: We set hidden size as 64 for the architecture of the prior NN-GP kernel for speedup.
Revised: We set hidden size as 64 for the architecture of the prior NN-GP kernel for speedup.

Original: The regularization constant λ is set as 1 e − 4 times of the average eigenvalue of the central covariance matrices.
Revised: The regularization constant λ is set as 1 e − 4 times of the average eigenvalue of the central covariance matrices.

Original: UCI regression.
Revised: UCI regression.

Original: We pre-process the UCI data by standard normalization.
Revised: We pre-process the UCI data by standard normalization.

Original: We set the variance for data noise and the weight variance for the prior kernel following (Pearce et al., 2020).
Revised: We set the variance for data noise and the weight variance for the prior kernel following (Pearce et al., 2020).

Original: The batch size for stochastic training is 256 .
Revised: The batch size for stochastic training is 256 .

Original: We use an Adam optimizer to optimize for 1000 epochs.
Revised: We use an Adam optimizer to optimize for 1000 epochs.

Original: The learning rate is initialized as 0 . 01 and decays by 0 .
Revised: The learning rate is initialized as 0 . 01 and decays by 0 .

Original: 99 every 5 epochs.
Revised: 99 every 5 epochs.

Original: Fashion-MNIST classiﬁcation.
Revised: Fashion-MNIST classiﬁcation.

Original: The used architecture is Conv(32, 3, 1)-BN-ReLU-MaxPool(2)-
Revised: The used architecture is Conv(32, 3, 1)-BN-ReLU-MaxPool(2)-

Original: Conv(64, 3, 0)-BN-ReLU-MaxPool(2)-Linear(256)-ReLU-Linear(10), where Conv( x , y , z ) represents 
Revised: Conv(64, 3, 0)-BN-ReLU-MaxPool(2)-Linear(256)-ReLU-Linear(10), where Conv( x , y , z ) represents [Figure] [Figure] N

Original: [Figure]
Revised: [Figure]

Revised: g a t i v e

Revised: L o g

Original: [Figure]
Revised: [Figure]

Original: g a t i v
Revised: li k e li h oo d

Original: [Figure]
Revised: [Figure]

Original: a 2D convolution with x output channels, kernel size y , and padding z .
Revised: a 2D convolution with x output channels, kernel size y , and padding z .

Original: The batch size for training data is 64 .
Revised: The batch size for training data is 64 .

Original: The batch size for extra measurement points is 0 .
Revised: The batch size for extra measurement points is 0 .

Original: We use an SGD optimizer to optimize for 24 epochs.
Revised: We use an SGD optimizer to optimize for 24 epochs.

Original: The learning rate is initialized as 0 . 1 and follows a cosine decay schedule.
Revised: The learning rate is initialized as 0 . 1 and follows a cosine decay schedule.

Original: We use an
Revised: We use an

Original: Adam optimizer with 1 e − 3 learning rate to optimize the temperature.
Revised: Adam optimizer with 1 e − 3 learning rate to optimize the temperature.

Original: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty, because the involved computation is only the cheap softmax transformation on the sampled function values.
Revised: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty, because the involved computation is only the cheap softmax transformation on the sampled function values.

Original: CIFAR-10 classiﬁcation.
Revised: CIFAR-10 classiﬁcation.

Original: We perform data augmentation including random horizontal ﬂip and random crop.
Revised: We perform data augmentation including random horizontal ﬂip and random crop.

Original: The batch size for training data is 128 .
Revised: The batch size for training data is 128 .

Original: The batch size for extra measurement points is 0 .
Revised: The batch size for extra measurement points is 0 .

Original: We use a SGD optimizer with 0 .
Revised: We use a SGD optimizer with 0 .

Original: 9 momentum to optimize for 200 epochs.
Revised: 9 momentum to optimize for 200 epochs.

Original: The learning rate is initialized as 0 . 1 and decays by 0 .
Revised: The learning rate is initialized as 0 . 1 and decays by 0 .

Original: 1 at 100 -th and 150 -th epochs.
Revised: 1 at 100 -th and 150 -th epochs.

Original: We use an Adam optimizer with 1 e − 3 learning rate to optimize the temperature.
Revised: We use an Adam optimizer with 1 e − 3 learning rate to optimize the temperature.

Original: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty.
Revised: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty.

Original: Contextual bandit.
Revised: Contextual bandit.

Original: We use MLPs with 2 hidden layers of 256 units.
Revised: We use MLPs with 2 hidden layers of 256 units.

Original: The batch size for training data is 512.
Revised: The batch size for training data is 512.

Original: The batch size for extra measurement points is 0.
Revised: The batch size for extra measurement points is 0.

Original: We update the model (i.e., the agent) for 100 epochs with an Adam optimizer every 50 rounds.
Revised: We update the model (i.e., the agent) for 100 epochs with an Adam optimizer every 50 rounds.

Original: We set α = 1 and β = 0 without tuning.
Revised: We set α = 1 and β = 0 without tuning.

Revised: DE, rDE, and RMS all randomly choose an ensemble member at per iteration, but our method randomly draws a sample from the GP for decision.

Revised: “Random” baseline corresponds to the Uniform algorithm.

Original: B. M ORE R ESULTS ON I LLUSTRATIVE R EGRESSION
Revised: B. M ORE R ESULTS ON I LLUSTRATIVE R EGRESSION

Original: We depict the results on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) problem in Fig.
Revised: We depict the results on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) problem in Fig.

Original: 9, which are consistently with those in Fig.
Revised: 9, which are consistently with those in Fig.

Original: B. M ORE R ESULTS ON CIFAR-10 C LASSIFICATION
Revised: B. M ORE R ESULTS ON CIFAR-10 C LASSIFICATION

Original: We plot the negative log-likelihood and test accuracy on CIFAR-10 corruptions for models trained with ResNet-20 and ResNet-56 in Fig.
Revised: We plot the negative log-likelihood and test accuracy on CIFAR-10 corruptions for models trained with ResNet-20 and ResNet-56 in Fig.

Original: 10 and Fig.
Revised: 10 and Fig.

Original: As shown, DE-GP outperforms the baselines in aspect of negative log-likelihood, but yields similar test accuracy to the baselines.
Revised: As shown, DE-GP outperforms the baselines in aspect of negative log-likelihood, but yields similar test accuracy to the baselines.

Original: Recapping the results in Fig.
Revised: Recapping the results in Fig.

Original: 6, DE-GP indeed has improved OOD robustness, but may still face problems in OOD generalization.
Revised: 6, DE-GP indeed has improved OOD robustness, but may still face problems in OOD generalization.

Original: We then conduct experiments with the deeper ResNet-110 architecture.
Revised: We then conduct experiments with the deeper ResNet-110 architecture.

Original: Due to resource constraint, we use 5 ensemble members.
Revised: Due to resource constraint, we use 5 ensemble members.

Original: The other settings are roughly the same as those for ResNet-56.
Revised: The other settings are roughly the same as those for ResNet-56.

Original: The results are offered in Fig.
Revised: The results are offered in Fig.

Original: 12, which validate the effectiveness of DE-GP for large networks.
Revised: 12, which validate the effectiveness of DE-GP for large networks.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: B. I N - DISTRIBUTION T EST A CCURACY ON CIFAR-

Original: Ensemble size is as 5 for these experiments.
Revised: We further perform experiments on the more challenging CIFAR-100 dataset. We present the indistribution test accuracy of DE-GP as well as the baselines in Table 2. We can see that DE-GP ( β = 0 . 1 ) is still on par with rDE. [Table]

Revised: B. A BLATION S TUDY ON α

Revised: We have conducted an ablation study on α (using ResNet-20 on CIFAR-10).

Revised: The results are presented in Table 3.

Revised: We can see that DE-GP is not sensitive to the value of α .

Revised: We in practice set α = 0 .

Revised: 01 in the CIFAR experiments.

Revised: We did not use a smaller α as it may result in colder posteriors and in turn worse uncertainty estimates.

Revised: [Table]

Original: B. A BLATION S TUDY ON THE A RCHITECTURE OF P RIOR K ERNEL
Revised: B. A BLATION S TUDY ON THE A RCHITECTURE OF P RIOR K ERNEL

Original: We perform an ablation study on the architecture for deﬁning the prior  NN-GP kernel, with the results listed in Table 2.
Revised: We perform an ablation study on the architecture for deﬁning the prior MC NN-GP kernel, with the results listed in Table 4.

Original: Surprisingly, usinga prior kernel with the cheap ResNet-20 architecture results in DE-GP with better test accuracy.
Revised: Surprisingly, using the cheap ResNet-20 architecture results in DE-GP with better test accuracy.

Original: We deduce this is because a deeper prior architecture induces more complex, black-box correlation for the function, which may lead to over-regularization.
Revised: We deduce this is because a deeper prior architecture induces more complex, black-box correlation for the function, which may lead to over-regularization

Original: [Table]
Revised: [Table]

Revised: [Table]