Original: D EEP E NSEMBLE AS A G AUSSIAN P ROCESS A PPROXIMATE P OSTERIOR
Revised: D EEP E NSEMBLE AS A G AUSSIAN P ROCESS A PPROXIMATE P OSTERIOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Ensemble is a ﬂexible and effective alternative to Bayesian neural networks for uncertainty estimation in deep learning.
Revised: Deep Ensemble is a ﬂexible and effective alternative to Bayesian neural networks for uncertainty estimation in deep learning.

Original: However, Deep Ensemble is broadly criticized for lacking a proper Bayesian justiﬁcation.
Revised: However, Deep Ensemble is broadly criticized for lacking a proper Bayesian justiﬁcation.

Original: Some attempts try to ﬁx this issue, while they are typically coupled with the regression likelihood or rely on restrictive assumptions.
Revised: Some attempts try to ﬁx this issue, while they are typically coupled with the regression likelihood or rely on restrictive assumptions.

Original: In this work, we propose to deﬁne a Gaussian process (GP) approximate posterior with Deep Ensemble, based on which we perform variational inference directly in function space.
Revised: In this work, we propose to deﬁne a Gaussian process (GP) approximate posterior with Deep Ensemble, based on which we perform variational inference directly in function space.

Original: We further develop a function-space posterior regularization mechanism to properly incorporate prior knowledge.
Revised: We further develop a function-space posterior regularization mechanism to properly incorporate prior knowledge.

Original: We provide strategies to make the training feasible, and demonstrate the algorithmic beneﬁts of variational inference in the GP family.
Revised: We provide strategies to make the training feasible, and demonstrate the algorithmic beneﬁts of variational inference in the GP family.

Original: As a result, our method consumes only marginally added training cost than the standard Deep Ensemble.
Revised: As a result, our method consumes only marginally added training cost than the standard Deep Ensemble.

Original: Empirically, our approach achieves better uncertainty estimation than Deep Ensemble and its variants across diverse scenarios.
Revised: Empirically, our approach achieves better uncertainty estimation than Deep Ensemble and its variants across diverse scenarios.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bayesian treatment of deep neural networks (DNNs) is promised to enjoy principled Bayesian uncertainty while unleashing the capacity of DNNs, with Bayesian neural networks (BNNs) as popular examples (MacKay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).
Revised: Bayesian treatment of deep neural networks (DNNs) is promised to enjoy principled Bayesian uncertainty while unleashing the capacity of DNNs, with Bayesian neural networks (BNNs) as popular examples (MacKay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).

Original: Nevertheless, despite the surge of advance in BNNs (Louizos & Welling, 2016; Zhang et al., 2018), many of existing BNNs still face challenges in accurate and scalable inference (Sun et al., 2019), and exhibit limitations in uncertainty estimation and out-of-distribution robustness (Ovadia et al., 2019).
Revised: Nevertheless, despite the surge of advance in BNNs (Louizos & Welling, 2016; Zhang et al., 2018), many of existing BNNs still face challenges in accurate and scalable inference (Sun et al., 2019), and exhibit limitations in uncertainty estimation and out-of-distribution robustness (Ovadia et al., 2019).

Original: Alternatively, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) trains multiple independent, randomly-initialized DNNs for ensemble, presenting higher ﬂexibility and effectiveness than BNNs.
Revised: Alternatively, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) trains multiple independent, randomly-initialized DNNs for ensemble, presenting higher ﬂexibility and effectiveness than BNNs.

Original: However, it is hard to interpret DE as a Bayesian approach which seeks for the Bayesian posterior of a certain model, and there is no guarantee that the uncertainty estimates given by DE are reliable.
Revised: However, it is hard to interpret DE as a Bayesian approach which seeks for the Bayesian posterior of a certain model, and there is no guarantee that the uncertainty estimates given by DE are reliable.

Original: To chase a Bayesian justiﬁcation for DE, recent works like RMS (Lu & Van Roy, 2017; Osband et al., 2018; Pearce et al., 2020) and NTKGP (He et al., 2020) reﬁne DE to be a “sample-then-optimize” approach (Matthews et al., 2017).
Revised: To chase a Bayesian justiﬁcation for DE, recent works like RMS (Lu & Van Roy, 2017; Osband et al., 2018; Pearce et al., 2020) and NTKGP (He et al., 2020) reﬁne DE to be a “sample-then-optimize” approach (Matthews et al., 2017).

Original: Nonetheless, these works rely on the regression likelihood and often make strong assumptions like linearised and/or inﬁnite-width models.
Revised: Nonetheless, these works rely on the regression likelihood and often make strong assumptions like linearised and/or inﬁnite-width models.

Original: To build a Bayesian reﬁnement of DE without reliance on restrictive assumptions, we propose to use DE to deﬁne a Gaussian process (GP) approximate posterior, which is dubbed as DE-GP for short.
Revised: To build a Bayesian reﬁnement of DE without reliance on restrictive assumptions, we propose to use DE to deﬁne a Gaussian process (GP) approximate posterior, which is dubbed as DE-GP for short.

Original: The imposition of a GP form on the approximate posterior is empowered by the equivalence between GPs and BNNs (Khan et al., 2019; Neal, 1996; Lee et al., 2018; Garriga-Alonso et al., 2018;
Revised: The imposition of a GP form on the approximate posterior is empowered by the equivalence between GPs and BNNs (Khan et al., 2019; Neal, 1996; Lee et al., 2018; Garriga-Alonso et al., 2018;

Original: Matthews et al., 2018; Novak et al., 2018;
Revised: Matthews et al., 2018; Novak et al., 2018;

Original: He et al., 2020), as well as the associated computational beneﬁts when adopting a GP prior without loss of generality.
Revised: He et al., 2020), as well as the associated computational beneﬁts when adopting a GP prior without loss of generality.

Original: Concretely, we evaluate the empirical mean and covariance of the ensemble members and use them to specify DE-GP.
Revised: Concretely, we evaluate the empirical mean and covariance of the ensemble members and use them to specify DE-GP.

Original: This is inspired by the formula of neural network Gaussian process (NN-GP)
Revised: This is inspired by the formula of neural network Gaussian process (NN-GP)

Original: (Novak et al., 2018).
Revised: (Novak et al., 2018).

Original: Nevertheless, we clarify that we deﬁne a parameteric and learnable GP kernel with ﬁnitely many and adaptive basis functions, in contrast to the NN-GP kernels built with inﬁnitely many yet ﬁxed basis functions.
Revised: Nevertheless, we clarify that we deﬁne a parameteric and learnable GP kernel with ﬁnitely many and adaptive basis functions, in contrast to the NN-GP kernels built with inﬁnitely many yet ﬁxed basis functions.

Original: The ﬂexibility of DE-GP is assured by the expressiveness of DNN ensemble members – by tuning them it is enabled to evolve over a rich variety of kernels.
Revised: The ﬂexibility of DE-GP is assured by the expressiveness of DNN ensemble members – by tuning them it is enabled to evolve over a rich variety of kernels.

Original: Given these setups, we perform function-space variational inference (VI) for learning.
Revised: Given these setups, we perform function-space variational inference (VI) for learning.

Original: We maximize the functional evidence lower bound (fELBO) (Sun et al., 2019; Rudner et al., 2021) to push DE-GP towards the true posterior over functions.
Revised: We maximize the functional evidence lower bound (fELBO) (Sun et al., 2019; Rudner et al., 2021) to push DE-GP towards the true posterior over functions.

Original: We further present a posterior regularization scheme in function space to conveniently incorporate structural priors, and present a concrete example on how
Revised: We further present a posterior regularization scheme in function space to conveniently incorporate structural priors, and present a concrete example on how

Original: T i m e p e r it e r a ti on ( s )
Revised: T i m e p e r it e r a ti on ( s )

Original: NN-GP
Revised: NN-GP

Original: DE-GP (our)
Revised: DE-GP (our)

Original: DE
Revised: DE

Original: RMS 0.
Revised: RMS 0.

Original: T i m e p e r it e r a ti on ( s )
Revised: T i m e p e r it e r a ti on ( s )

Original: DE-GPDERMS
Revised: DE-GPDERMS

Original: Training speed to control function complexity.
Revised: Training speed to control function complexity.

Original: By VI, we are enabled to handle classiﬁcation problems directly and exactly, without casting them into regression ones (He et al., 2020).
Revised: By VI, we are enabled to handle classiﬁcation problems directly and exactly, without casting them into regression ones (He et al., 2020).

Original: In practice, DE-GP can be easily implemented on top of a standard DE, while only introducing minimally added cost.
Revised: In practice, DE-GP can be easily implemented on top of a standard DE, while only introducing minimally added cost.

Original: The function-space Bayesian inference enables DE-GP to ameliorate the pathologies induced by the over-parameterization nature of DNNs (Sun et al., 2019;
Revised: The function-space Bayesian inference enables DE-GP to ameliorate the pathologies induced by the over-parameterization nature of DNNs (Sun et al., 2019;

Original: Wang et al., 2019).
Revised: Wang et al., 2019).

Original: Thus DE-GP can offer calibrated uncertainty estimates, especially when adopting large networks (see Fig. 1).
Revised: Thus DE-GP can offer calibrated uncertainty estimates, especially when adopting large networks (see Fig. 1).

Original: We empirically demonstrate that DE-GP outperforms strong baselines on various regression datasets, and presents superior uncertainty estimates and out-of-distribution robustness without compromising accuracy in challenging classiﬁcation tasks.
Revised: We empirically demonstrate that DE-GP outperforms strong baselines on various regression datasets, and presents superior uncertainty estimates and out-of-distribution robustness without compromising accuracy in challenging classiﬁcation tasks.

Original: DE-GP also shows promise in solving contextual bandit problems, where the uncertainty plays a vital role in guiding exploration.
Revised: DE-GP also shows promise in solving contextual bandit problems, where the uncertainty plays a vital role in guiding exploration.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Bayesian neural networks (BNNs).
Revised: Bayesian neural networks (BNNs).

Original: Bayesian treatment of DNNs is an emerging topic in deep learning yet with a long history (Mackay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).
Revised: Bayesian treatment of DNNs is an emerging topic in deep learning yet with a long history (Mackay, 1992; Hinton & Van Camp, 1993; Neal, 1995; Graves, 2011).

Original: BNNs can be learned by virtue of variational inference (Blundell et al., 2015; HernándezLobato & Adams, 2015; Louizos & Welling, 2016; Zhang et al., 2018; Khan et al., 2018;
Revised: BNNs can be learned by virtue of variational inference (Blundell et al., 2015; HernándezLobato & Adams, 2015; Louizos & Welling, 2016; Zhang et al., 2018; Khan et al., 2018;

Original: Deng et al., 2020), Laplace approximation (Mackay, 1992; Ritter et al., 2018), Markov chain Monte
Revised: Deng et al., 2020), Laplace approximation (Mackay, 1992; Ritter et al., 2018), Markov chain Monte

Original: Carlo (Welling & Teh, 2011; Chen et al., 2014; Zhang et al., 2019), particle-optimization based variational inference (Liu & Wang, 2016), Monte Carlo dropout (Gal & Ghahramani, 2016), and other methods (Maddox et al., 2019; Izmailov et al., 2020).
Revised: Carlo (Welling & Teh, 2011; Chen et al., 2014; Zhang et al., 2019), particle-optimization based variational inference (Liu & Wang, 2016), Monte Carlo dropout (Gal & Ghahramani, 2016), and other methods (Maddox et al., 2019; Izmailov et al., 2020).

Original: To avoid the difﬁculties of posterior inference in weight space, some recent works advocate performing Bayesian reasoning in function space (Sun et al., 2019; Rudner et al., 2021; Wang et al., 2019).
Revised: To avoid the difﬁculties of posterior inference in weight space, some recent works advocate performing Bayesian reasoning in function space (Sun et al., 2019; Rudner et al., 2021; Wang et al., 2019).

Original: In function space, BNNs of inﬁnite or even ﬁnitely width equal to Gaussian processes (GPs) (Neal, 1996; Lee et al., 2018; Novak et al., 2018;
Revised: In function space, BNNs of inﬁnite or even ﬁnitely width equal to Gaussian processes (GPs) (Neal, 1996; Lee et al., 2018; Novak et al., 2018;

Original: He et al., 2020; Khan et al., 2019), which provides valuable insights for this work.
Revised: He et al., 2020; Khan et al., 2019), which provides valuable insights for this work.

Original: Deep Ensemble (DE) (Lakshminarayanan et al., 2017) is a qualiﬁed alternative to BNNs for uncertainty estimation (Ovadia et al., 2019), yet lacks a proper Bayesian interpretation.
Revised: Deep Ensemble (DE) (Lakshminarayanan et al., 2017) is a qualiﬁed alternative to BNNs for uncertainty estimation (Ovadia et al., 2019), yet lacks a proper Bayesian interpretation.

Original: Wilson & Izmailov (2020) conceptually interpreted DE as a method that approximates the Bayesian posterior predictive distribution, but it is hard to judge whether the approximation is reliable or not in practice.
Revised: Wilson & Izmailov (2020) conceptually interpreted DE as a method that approximates the Bayesian posterior predictive distribution, but it is hard to judge whether the approximation is reliable or not in practice.

Original: Recently, a suite of works based on the notion of “sample-then-optimize” (Matthews et al., 2017) have been developed to make DE Bayesian.
Revised: Recently, a suite of works based on the notion of “sample-then-optimize” (Matthews et al., 2017) have been developed to make DE Bayesian.

Original: For example, RMS (Lu & Van Roy, 2017; Osband et al., 2018;
Revised: For example, RMS (Lu & Van Roy, 2017; Osband et al., 2018;

Original: Pearce et al., 2020) regularizes the ensemble members towards randomised priors to obtain posterior samples after training, while it typically assumes linear data likelihood which is impractical for deep models and classiﬁcation tasks.
Revised: Pearce et al., 2020) regularizes the ensemble members towards randomised priors to obtain posterior samples after training, while it typically assumes linear data likelihood which is impractical for deep models and classiﬁcation tasks.

Original: He et al.
Revised: He et al.

Original: (2020) proposed to add a randomised function to each ensemble member to realise a function-space Bayesian interpretation, but the method is asymptotically exact in the inﬁnite width limit and is limited to regressions.
Revised: (2020) proposed to add a randomised function to each ensemble member to realise a function-space Bayesian interpretation, but the method is asymptotically exact in the inﬁnite width limit and is limited to regressions.

Original: By contrast, DE-GP works without restrictive assumptions.
Revised: By contrast, DE-GP works without restrictive assumptions.

Original: In parallel, D’Angelo & Fortuin (2021) proposed to add a repulsive term to
Revised: In parallel, D’Angelo & Fortuin (2021) proposed to add a repulsive term to

Original: DE to achieve Bayesian inference, which yet relies on unscalable gradient estimators.
Revised: DE to achieve Bayesian inference, which yet relies on unscalable gradient estimators.

Original:  B ACKGROUND 3. B AYESIAN N EURAL N ETWORKS
Revised:  B ACKGROUND 3. B AYESIAN N EURAL N ETWORKS

Original: Consider learning on a dataset D
Revised: Consider learning on a dataset D

Original: =
Revised: =

Original: ( X , Y )
Revised: ( X , Y )

Original: = { ( x i , y i ) } ni =1 , with x i ∈ X as data and y i as
Revised: = { ( x i , y i ) } ni =1 , with x i ∈ X as data and y i as

Original: C -dimensional targets.
Revised: C -dimensional targets.

Original: Let g ( · , w ) :
Revised: Let g ( · , w ) :

Original: X → R C denote the function represented by a DNN with weights w .
Revised: X → R C denote the function represented by a DNN with weights w .

Original: Bayesian neural networks (BNNs) treat w as random variables and chase the posterior p ( w |D ) by imposing a prior p ( w ) .
Revised: Bayesian neural networks (BNNs) treat w as random variables and chase the posterior p ( w |D ) by imposing a prior p ( w ) .

Original: Among the family, variational BNNs (VBNNs) are particularly appealing as they cast posterior inference as an optimization problem (Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2016), where a variational distribution q ( w ) is introduced to approximate the true posterior by maximizing the evidence lower bound (ELBO):
Revised: Among the family, variational BNNs (VBNNs) are particularly appealing as they cast posterior inference as an optimization problem (Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2016), where a variational distribution q ( w ) is introduced to approximate the true posterior by maximizing the evidence lower bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original: BNNs marginalize over the posteriors to obtain the posterior predictive for new data x ∗ :
Revised: BNNs marginalize over the posteriors to obtain the posterior predictive for new data x ∗ :

Original: [Equation]
Revised: [Equation]

Original: where w s ∼ q ( w ) , s = 1 , ..., S .
Revised: where w s ∼ q ( w ) , s = 1 , ..., S .

Original: Such a procedure is expected to propagate the embedded model uncertainty into the prediction.
Revised: Such a procedure is expected to propagate the embedded model uncertainty into the prediction.

Original: However in practice, most of the existing BNN approaches face obstacles in precise posterior inference due to non-trivial and convoluted posterior dependencies (Louizos & Welling, 2016; Zhang et al., 2018; Shi et al., 2018; Sun et al., 2019), and deliver unsatisfactory uncertainty estimation and out-of-distribution (OOD) robustness (Ovadia et al., 2019).
Revised: However in practice, most of the existing BNN approaches face obstacles in precise posterior inference due to non-trivial and convoluted posterior dependencies (Louizos & Welling, 2016; Zhang et al., 2018; Shi et al., 2018; Sun et al., 2019), and deliver unsatisfactory uncertainty estimation and out-of-distribution (OOD) robustness (Ovadia et al., 2019).

Original: 3. D EEP E NSEMBLE
Revised: 3. D EEP E NSEMBLE

Original: As a workaround of BNNs, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) deploys a set of
Revised: As a workaround of BNNs, Deep Ensemble (DE) (Lakshminarayanan et al., 2017) deploys a set of

Original: M DNNs { g ( · , w i ) } Mi =1 for ensemble.
Revised: M DNNs { g ( · , w i ) } Mi =1 for ensemble.

Original: { w i } Mi =1 are independently trained to interpret the data from different angles, under maximum likelihood estimation (MLE) principle in standard DE, or maximum a posteriori (MAP) principle in regularized DE (rDE):
Revised: { w i } Mi =1 are independently trained to interpret the data from different angles, under maximum likelihood estimation (MLE) principle in standard DE, or maximum a posteriori (MAP) principle in regularized DE (rDE):

Original: [Equation]
Revised: [Equation]

Original: Due to the randomness in network initialization and stochastic gradient descent (SGD), DE can effectively explore the non-convex, multimodal loss landscape of DNNs (Fort et al., 2019; Wilson &
Revised: Due to the randomness in network initialization and stochastic gradient descent (SGD), DE can effectively explore the non-convex, multimodal loss landscape of DNNs (Fort et al., 2019; Wilson &

Original: Izmailov, 2020), thus has shown promise in various uncertainty estimation tasks (Lakshminarayanan et al., 2017; Ovadia et al., 2019).
Revised: Izmailov, 2020), thus has shown promise in various uncertainty estimation tasks (Lakshminarayanan et al., 2017; Ovadia et al., 2019).

Original: However, due to the lack of a Bayesian justiﬁcation, there is no guarantee that the uncertainty estimates given by DE are reliable.
Revised: However, due to the lack of a Bayesian justiﬁcation, there is no guarantee that the uncertainty estimates given by DE are reliable.

Original: 3. R ELATION OF THE B AYESIAN P OSTERIORS OF DNN S TO G AUSSIAN P ROCESSES
Revised: 3. R ELATION OF THE B AYESIAN P OSTERIORS OF DNN S TO G AUSSIAN P ROCESSES

Original: It is early shown that shallow BNNs converge to Gaussian processes (GPs) in the inﬁnite width limit (Neal, 1996).
Revised: It is early shown that shallow BNNs converge to Gaussian processes (GPs) in the inﬁnite width limit (Neal, 1996).

Original: Recently, there is ongoing effort to extend the result to multiple layers (Lee et al., 2018; Matthews et al., 2018), convolutional architectures (Garriga-Alonso et al., 2018; Novak et al., 2018), and beyond (Yang, 2019).
Revised: Recently, there is ongoing effort to extend the result to multiple layers (Lee et al., 2018; Matthews et al., 2018), convolutional architectures (Garriga-Alonso et al., 2018; Novak et al., 2018), and beyond (Yang, 2019).

Original: Brieﬂy, an inﬁnitely wide BNN amount to the NN-GP:
Revised: Brieﬂy, an inﬁnitely wide BNN amount to the NN-GP:

Original: GP
Revised: GP

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: κ
Revised: κ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where p ( w ) is a layerwise isotropic Gaussian and the DNN specifying g has inﬁnite width.
Revised: where p ( w ) is a layerwise isotropic Gaussian and the DNN specifying g has inﬁnite width.

Original: Notably, κ is a compositional, matrix-valued kernel, with values in the space of C × C matrices, and can be analytically estimated in some cases (Novak et al., 2018; 2019).
Revised: Notably, κ is a compositional, matrix-valued kernel, with values in the space of C × C matrices, and can be analytically estimated in some cases (Novak et al., 2018; 2019).

Original: Due to the i.i.d.
Revised: Due to the i.i.d.

Original: weights, the kernel evaluation κ ( x ∗ , x ∗ ) of some data x ∗ is a scaled identity matrix.
Revised: weights, the kernel evaluation κ ( x ∗ , x ∗ ) of some data x ∗ is a scaled identity matrix.

Original: Based on the canonical results of GP regression, we know the posteriors of inﬁnitely wide BNNs are still GPs (in regression tasks), corresponding to the Bayesian training of the last readout layer (Arora et al., 2019; Lee et al., 2019).
Revised: Based on the canonical results of GP regression, we know the posteriors of inﬁnitely wide BNNs are still GPs (in regression tasks), corresponding to the Bayesian training of the last readout layer (Arora et al., 2019; Lee et al., 2019).

Original: Despite a golden standard for small regression problems, analytical posterior inference for NN-GP faces challenges when handling contemporary DNN architectures (Novak et al., 2018), large data (Shi et al., 2019) and classiﬁcation problems.
Revised: Despite a golden standard for small regression problems, analytical posterior inference for NN-GP faces challenges when handling contemporary DNN architectures (Novak et al., 2018), large data (Shi et al., 2019) and classiﬁcation problems.

Original: Interestingly, ﬁnitely wide BNNs with Gaussian posteriors on weights can also be connected to
Revised: Interestingly, ﬁnitely wide BNNs with Gaussian posteriors on weights can also be connected to

Original: GPs (Khan et al., 2019; Rudner et al., 2021).
Revised: GPs (Khan et al., 2019; Rudner et al., 2021).

Original: We then arrive at the hypothesis that the Bayesian posteriors of DNNs are closely related to GPs.
Revised: We then arrive at the hypothesis that the Bayesian posteriors of DNNs are closely related to GPs.

Original: So, we specify a GP approximate posterior with DE.
Revised: So, we specify a GP approximate posterior with DE.

Original:  M ETHODOLOGY 4. D EEP E NSEMBLE AS A G AUSSIAN P ROCESS A PPROXIMATE P OSTERIOR
Revised:  M ETHODOLOGY 4. D EEP E NSEMBLE AS A G AUSSIAN P ROCESS A PPROXIMATE P OSTERIOR

Original: Basically, we utilize DE’s ensemble members (referred to as basis functions hereinafter) to deﬁne a
Revised: Basically, we utilize DE’s ensemble members (referred to as basis functions hereinafter) to deﬁne a

Original: GP approximate posterior as follows:
Revised: GP approximate posterior as follows:

Original: [Equation]
Revised: [Equation]

Original: where g i refers to g ( · , w i ) .
Revised: where g i refers to g ( · , w i ) .

Original: Namely, the empirical mean and central covariance of ﬁnitely many, adaptive basis functions are leveraged to specify DE-GP.
Revised: Namely, the empirical mean and central covariance of ﬁnitely many, adaptive basis functions are leveraged to specify DE-GP.

Original: k q ( x , x (cid:48) ) is a linear , matrix-valued kernel with a small scaled identity matrix λ I C appended to avoid singularity.
Revised: k q ( x , x (cid:48) ) is a linear , matrix-valued kernel with a small scaled identity matrix λ I C appended to avoid singularity.

Original: The variations in k q ( x , x (cid:48) ) are conﬁned to having up to M − 1 rank, echoing the recent investigations showing that low-rank approximate posteriors for DNNs conjoin effectiveness and efﬁciency (Maddox et al., 2019; Izmailov et al., 2020; Dusenberry et al., 2020).
Revised: The variations in k q ( x , x (cid:48) ) are conﬁned to having up to M − 1 rank, echoing the recent investigations showing that low-rank approximate posteriors for DNNs conjoin effectiveness and efﬁciency (Maddox et al., 2019; Izmailov et al., 2020; Dusenberry et al., 2020).

Original: Note that the popular Nyström method (Williams & Seeger, 2001) also uses a low-rank matrix to approximate the original kernel matrix.
Revised: Note that the popular Nyström method (Williams & Seeger, 2001) also uses a low-rank matrix to approximate the original kernel matrix.

Original: Akin to the kernels in (Wilson et al., 2016), the DE-GP kernels are highly ﬂexible, and may automatically discover the underlying structures of high-dimensional data without manual participation.
Revised: Akin to the kernels in (Wilson et al., 2016), the DE-GP kernels are highly ﬂexible, and may automatically discover the underlying structures of high-dimensional data without manual participation.

Original: 4. V ARIATIONAL I NFERENCE IN F UNCTION S PACE
Revised: 4. V ARIATIONAL I NFERENCE IN F UNCTION S PACE

Original: We do function-space variational inference to push DE-GP towards the true posterior over functions.
Revised: We do function-space variational inference to push DE-GP towards the true posterior over functions.

Original: Prior We have high freedom to determine which prior to use attributed to the variational inference paradigm.
Revised: Prior We have high freedom to determine which prior to use attributed to the variational inference paradigm.

Original: Without loss of generality, we use the MC estimate of NN-GP (MC NN-GP)
Revised: Without loss of generality, we use the MC estimate of NN-GP (MC NN-GP)

Original: (Novak et al., 2018) as the prior due to its accessibility.
Revised: (Novak et al., 2018) as the prior due to its accessibility.

Original: Concretely, supposing a ﬁnitely wide DNN composed of a feature projector h ( · , w ) :
Revised: Concretely, supposing a ﬁnitely wide DNN composed of a feature projector h ( · , w ) :

Original: X → R ˆ C and a linear readout layer with weight variance σ 2 w and bias variance σ 2 b , the MC NN-GP is deﬁned as p ( f ) =
Revised: X → R ˆ C and a linear readout layer with weight variance σ 2 w and bias variance σ 2 b , the MC NN-GP is deﬁned as p ( f ) =

Original: GP ( f | 0 , k
Revised: GP ( f | 0 , k

Original: ( x , x (cid:48) )) , where k
Revised: ( x , x (cid:48) )) , where k

Original: [Equation]
Revised: [Equation]

Original: , x
Revised: , x

Original: [Equation]
Revised: [Equation]

Original: , with
Revised: , with

Original: [Equation]
Revised: [Equation]

Original: I C refers to the indentity matrix of size C × C and w s are i.i.d.
Revised: I C refers to the indentity matrix of size C × C and w s are i.i.d.

Original: samples from the Gaussian prior on weights p ( w ) .
Revised: samples from the Gaussian prior on weights p ( w ) .

Original: Similar setups can be found in some related works like (Wang et al., 2019).
Revised: Similar setups can be found in some related works like (Wang et al., 2019).

Original: There may be a subtle difference between the MC NN-GP and the NN-GP, so what we are actually doing is not approximating the exact NN-GP posterior with DE-GP.
Revised: There may be a subtle difference between the MC NN-GP and the NN-GP, so what we are actually doing is not approximating the exact NN-GP posterior with DE-GP.

Original: Also of note that the NN-GP prior and the DE-GP posterior can be deﬁned with various architectures.
Revised: Also of note that the NN-GP prior and the DE-GP posterior can be deﬁned with various architectures.

Original: fELBO Following Sun et al.
Revised: fELBO Following Sun et al.

Original: (2019), we maximize the functional ELBO (fELBO) to optimize DE-GP:
Revised: (2019), we maximize the functional ELBO (fELBO) to optimize DE-GP:

Original: [Equation]
Revised: [Equation]

Original: Notably, there is a KL divergence between two GPs, which, on its own, is challenging to cope with.
Revised: Notably, there is a KL divergence between two GPs, which, on its own, is challenging to cope with.

Original: Fortunately, as proved by Sun et al.
Revised: Fortunately, as proved by Sun et al.

Original: (2019), we can take the KL divergence between the marginal distributions of function evaluations as a substitute for it, giving rise to a more tractable objective:
Revised: (2019), we can take the KL divergence between the marginal distributions of function evaluations as a substitute for it, giving rise to a more tractable objective:

Original: [Equation]
Revised: [Equation]

Original: where ˜ X denotes a measurement set including all training inputs X , and f ˜ X is the concatenation of the vectorized outputs of f for ˜ X , i.e., f ˜ X ∈ R | ˜ X | C .
Revised: where ˜ X denotes a measurement set including all training inputs X , and f ˜ X is the concatenation of the vectorized outputs of f for ˜ X , i.e., f ˜ X ∈ R | ˜ X | C .

Original: 4. P OSTERIOR R EGULARIZATION IN F UNCTION S PACE
Revised: 4. P OSTERIOR R EGULARIZATION IN F UNCTION S PACE

Original: Though Bayesian learning in function space enables a direct imposition of prior knowledge on function properties like smoothness and periodicity (Sun et al., 2019), sometimes the prior knowledge cannot be trivially designed as a prior distribution over functions.
Revised: Though Bayesian learning in function space enables a direct imposition of prior knowledge on function properties like smoothness and periodicity (Sun et al., 2019), sometimes the prior knowledge cannot be trivially designed as a prior distribution over functions.

Original: For example, the functions are periodical only over a limited input range, etc.
Revised: For example, the functions are periodical only over a limited input range, etc.

Original: In these cases, posterior regularization (Ganchev et al., 2010; Zhu et al., 2014) provides a principled workaround to impose structural constraints.
Revised: In these cases, posterior regularization (Ganchev et al., 2010; Zhu et al., 2014) provides a principled workaround to impose structural constraints.

Original: Brieﬂy, we apply posterior regularization to functional variational inference by solving:
Revised: Brieﬂy, we apply posterior regularization to functional variational inference by solving:

Original: [Equation]
Revised: [Equation]

Original: Q = { q ( f ) | E q ( f ) Ω( f ) ≤ 0 } is a valid set deﬁned in terms of a functional Ω which delivers some statistic of interest of a function.
Revised: Q = { q ( f ) | E q ( f ) Ω( f ) ≤ 0 } is a valid set deﬁned in terms of a functional Ω which delivers some statistic of interest of a function.

Original: 1 For tractable optimization, we slack the constraint as a penalty:
Revised: 1 For tractable optimization, we slack the constraint as a penalty:

Original: [Equation]
Revised: [Equation]

Original: where β is a trade-off coefﬁcient.
Revised: where β is a trade-off coefﬁcient.

Original: We next present an example on how to impose prior knowledge on function (hypothesis) complexity on the DE-GP approximate posterior using this paradigm.
Revised: We next present an example on how to impose prior knowledge on function (hypothesis) complexity on the DE-GP approximate posterior using this paradigm.

Original: For the sake of brevity, we assume a binary classiﬁcation scenario in the following discussion where y ∈ {− 1 , 1 } and f, g i : X → R .
Revised: For the sake of brevity, we assume a binary classiﬁcation scenario in the following discussion where y ∈ {− 1 , 1 } and f, g i : X → R .

Original: We use 0 - 1 loss (cid:96) ( f ( x ) , y )
Revised: We use 0 - 1 loss (cid:96) ( f ( x ) , y )

Original: = 1 y (cid:54) =
Revised: = 1 y (cid:54) =

Original: sign ( f ( x )) to measure the classiﬁcation error on one datum.
Revised: sign ( f ( x )) to measure the classiﬁcation error on one datum.

Original: We assume an underlying distribution µ = µ ( x , y ) supported on X × {− 1 , 1 } for generating the training data D , based on which we can deﬁne the true risk of a function (hypothesis) f : R ( f ) :
Revised: We assume an underlying distribution µ = µ ( x , y ) supported on X × {− 1 , 1 } for generating the training data D , based on which we can deﬁne the true risk of a function (hypothesis) f : R ( f ) :

Original: = E ( x ,y ) ∼ µ (cid:96) ( f ( x ) , y ) .
Revised: = E ( x ,y ) ∼ µ (cid:96) ( f ( x ) , y ) .

Original: We set E q ( f ) Ω( f ) :
Revised: We set E q ( f ) Ω( f ) :

Original: =
Revised: =

Original: E q ( f ) R ( f ) in the seek of a posterior over functions that can generalize well.
Revised: E q ( f ) R ( f ) in the seek of a posterior over functions that can generalize well.

Original: By deﬁnition, a hypothesis sample f ∼ q ( f )
Revised: By deﬁnition, a hypothesis sample f ∼ q ( f )

Original: =
Revised: =

Original: GP ( m q ( x ) , k q ( x , x (cid:48) )) can be decomposed as
Revised: GP ( m q ( x ) , k q ( x , x (cid:48) )) can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: + ζ ( x ) with ζ ( x ) ∼ GP (0 , k q ( x , x (cid:48) )) .
Revised: + ζ ( x ) with ζ ( x ) ∼ GP (0 , k q ( x , x (cid:48) )) .

Original: If sign ( f ( x )) (cid:54) =
Revised: If sign ( f ( x )) (cid:54) =

Original: y , it is impossible that sign ( g 1 ( x ))
Revised: y , it is impossible that sign ( g 1 ( x ))

Original: =
Revised: =

Original: y , ..., sign ( g M ( x ))
Revised: y , ..., sign ( g M ( x ))

Original: =
Revised: =

Original: y , and sign ( ζ ( x ))
Revised: y , and sign ( ζ ( x ))

Original: =
Revised: =

Original: y all hold.
Revised: y all hold.

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: We can further re-parameterize ζ ( x ) as ζ ( x ) =
Revised: We can further re-parameterize ζ ( x ) as ζ ( x ) =

Original: [Equation]
Revised: [Equation]

Original: N (0 , 1) , i = 0 , ..., M , which is essentially a real-valued random function symmetric around 0.
Revised: N (0 , 1) , i = 0 , ..., M , which is essentially a real-valued random function symmetric around 0.

Original: Thus, for any ( x , y ) ∼ µ , we have E q ( f ) (cid:96) ( ζ ( x ) , y )
Revised: Thus, for any ( x , y ) ∼ µ , we have E q ( f ) (cid:96) ( ζ ( x ) , y )

Original: =
Revised: =

Original: E (cid:15) 0 ,...,
Revised: E (cid:15) 0 ,...,

Original: [Equation]
Revised: [Equation]

Original: M (cid:96) ( ζ ( x ) , y ) = 1 / 2 .
Revised: M (cid:96) ( ζ ( x ) , y ) = 1 / 2 .

Original: As a result,
Revised: As a result,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ) R
Revised: ) R

Original: [Equation]
Revised: [Equation]

Original: Namely, the expected generalization error of the approximately posteriori functions can be bounded from above by those of the DNN basis functions.
Revised: Namely, the expected generalization error of the approximately posteriori functions can be bounded from above by those of the DNN basis functions.

Original: 3 Recalling the theoretical and empirical results showing that DNNs’ generalization error R ( g i ) can be decreased by controlling model capacity in terms of norm-based regularization min w i
Revised: 3 Recalling the theoretical and empirical results showing that DNNs’ generalization error R ( g i ) can be decreased by controlling model capacity in terms of norm-based regularization min w i

Original: ||
Revised: ||

Original: w i
Revised: w i

Original: || 

Revised: ||

Original: (Neyshabur et al., 2015; 2017; Bartlett et al., 2017;
Revised: (Neyshabur et al., 2015; 2017; Bartlett et al., 2017;

Original: Jiang et al., 2019), we opt to maximize the following reﬁned fELBO for learning DE-GP:
Revised: Jiang et al., 2019), we opt to maximize the following reﬁned fELBO for learning DE-GP:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Intuitively, the weight space of an over-parameterized
Revised: Intuitively, the weight space of an over-parameterized

Original: DNN contains many redundant interpretations for the data, thus pushing the weights towards zero would not hurt data ﬁtting but boost generalization.
Revised: DNN contains many redundant interpretations for the data, thus pushing the weights towards zero would not hurt data ﬁtting but boost generalization.

Original: In practice, when processing with shallow architectures, β can be set as 0 to remove the constraint on model capacity; while when using deep architectures (e.g., ResNet (He et al., 2016)), we can set β according to commonly used weight decay coefﬁcient for better generalization.
Revised: In practice, when processing with shallow architectures, β can be set as 0 to remove the constraint on model capacity; while when using deep architectures (e.g., ResNet (He et al., 2016)), we can set β according to commonly used weight decay coefﬁcient for better generalization.

Original: Comparisons in Fig.
Revised: Comparisons in Fig.

Original: 2 and Table 1 support this.
Revised: 2 and Table 1 support this.

Original: 4. T HE A LGORITHM
Revised: 4. T HE A LGORITHM

Original: We outline the training procedure of DE-GP in Algorithm 1, and elaborate some details below.
Revised: We outline the training procedure of DE-GP in Algorithm 1, and elaborate some details below.

Original: [Table]
Revised: [Table]

Original: 4.4. M INI - BATCH T RAINING
Revised: 4.4. M INI - BATCH T RAINING

Original: In deep learning scenarios
Revised: In deep learning scenarios

Original: [Table]
Revised: [Table]

Original: DE-GP should proceed by mini-batch training.
Revised: DE-GP should proceed by mini-batch training.

Original: At each step
Revised: At each step

Original: [Table]
Revised: [Table]

Original: we manufacture a stochastic measurement set with a mini-batch D s =
Revised: we manufacture a stochastic measurement set with a mini-batch D s =

Original: ( X s , Y s ) from the training data D and random samples X ν from a continuous distribution ν
Revised: ( X s , Y s ) from the training data D and random samples X ν from a continuous distribution ν

Original: (e.g., a uniform distribution) supported on X .
Revised: (e.g., a uniform distribution) supported on X .

Original: Then, we adapt the objective deﬁned in Eq.
Revised: Then, we adapt the objective deﬁned in Eq.

Original: (13) to the following form:
Revised: (13) to the following form:

Original: [Equation]
Revised: [Equation]

Original: where ˜ X s indicates the union of X s and X ν .
Revised: where ˜ X s indicates the union of X s and X ν .

Original: When α = 1 and β = 0 , L DE-GP is a lower bound of log p ( D s ) according to (Sun et al., 2019).
Revised: When α = 1 and β = 0 , L DE-GP is a lower bound of log p ( D s ) according to (Sun et al., 2019).

Original: Yet, given that we do not perform intensive hyper-parameter tuning for the NN-GP prior, it is reasonable to make α a tunable hyper-parameter to better trade off between the data ﬁtting and the priori inductive bias.
Revised: Yet, given that we do not perform intensive hyper-parameter tuning for the NN-GP prior, it is reasonable to make α a tunable hyper-parameter to better trade off between the data ﬁtting and the priori inductive bias.

Original: 4.4. A N E XACT AND E FFICIENT E STIMATION OF THE M ARGINAL KL D IVERGENCE
Revised: 4.4. A N E XACT AND E FFICIENT E STIMATION OF THE M ARGINAL KL D IVERGENCE

Original: The marginal distributions of function evaluations are multivariate Gaussian by the deﬁnition of GP:
Revised: The marginal distributions of function evaluations are multivariate Gaussian by the deﬁnition of GP:

Original: [Equation]
Revised: [Equation]

Original: with the kernel matrices k ˜ X s , ˜ X s q , k ˜ X s , ˜ X s ∈ R | ˜
Revised: with the kernel matrices k ˜ X s , ˜ X s q , k ˜ X s , ˜ X s ∈ R | ˜

Original: X s | C ×|
Revised: X s | C ×|

Original: ˜
Revised: ˜

Original: X s | C as the joints of pair-wise outcomes.
Revised: X s | C as the joints of pair-wise outcomes.

Original: Therefore, the marginal KL divergence and its gradients can be estimated exactly without resorting to some approximations (Rudner et al., 2021).
Revised: Therefore, the marginal KL divergence and its gradients can be estimated exactly without resorting to some approximations (Rudner et al., 2021).

Original: What’s more, as discussed in Section 3.3, there is a simple structure in the prior kernel matrices, so we can write them in the form of Kronecker product:
Revised: What’s more, as discussed in Section 3.3, there is a simple structure in the prior kernel matrices, so we can write them in the form of Kronecker product:

Original: [Equation]
Revised: [Equation]

Original: where ˆk ˜ X s , ˜ X s ∈ R | ˜ X s
Revised: where ˆk ˜ X s , ˜ X s ∈ R | ˜ X s

Original: |×| ˜ X
Revised: |×| ˜ X

Original: s | corresponds to the evaluation of kernel ˆ k .
Revised: s | corresponds to the evaluation of kernel ˆ k .

Original: Hence we can exploit the property of Kronecker product to inverse k ˜ X s , ˜ X s in O ( | ˜ X s | 3 ) complexity.
Revised: Hence we can exploit the property of Kronecker product to inverse k ˜ X s , ˜ X s in O ( | ˜ X s | 3 ) complexity.

Original: Besides, as k ˜ X s , ˜ X s q is low-rank, we can leverage the Woodbury matrix identity (Woodbury, 1950) and matrix determinant lemma (Harville, 1998) to efﬁciently compute the inverse and determinant of k ˜ X s ,
Revised: Besides, as k ˜ X s , ˜ X s q is low-rank, we can leverage the Woodbury matrix identity (Woodbury, 1950) and matrix determinant lemma (Harville, 1998) to efﬁciently compute the inverse and determinant of k ˜ X s ,

Original: ˜ X s q in O ( | ˜ X s | CM 2 ) time complexity given that usually M (cid:28) | ˜
Revised: ˜ X s q in O ( | ˜ X s | CM 2 ) time complexity given that usually M (cid:28) | ˜

Original: X s | C
Revised: X s | C

Original: (e.g., 10 (cid:28) 256 C ).
Revised: (e.g., 10 (cid:28) 256 C ).

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: Diversity.
Revised: Diversity.

Original: The diversity among the ensemble members in function space is explicitly encouraged by the KL divergence between variational and the prior in fELBO.
Revised: The diversity among the ensemble members in function space is explicitly encouraged by the KL divergence between variational and the prior in fELBO.

Original: Nonetheless, the expected loglikelihood in fELBO enforces each ensemble member to yield the same, correct outcomes for the training data.
Revised: Nonetheless, the expected loglikelihood in fELBO enforces each ensemble member to yield the same, correct outcomes for the training data.

Original: Thereby, the diversity mainly exists in the regions far away from the training data (see
Revised: Thereby, the diversity mainly exists in the regions far away from the training data (see

Original: Fig. 1 and Fig. 9 in Appendix B.2).
Revised: Fig. 1 and Fig. 9 in Appendix B.2).

Original: Yet, the diversity in DE does not have a clear theoretical support.
Revised: Yet, the diversity in DE does not have a clear theoretical support.

Original: Efﬁciency.
Revised: Efﬁciency.

Original: Compared to the overhead introduced by DNNs, the effort of estimating the KL divergence between Gaussians is negligible.
Revised: Compared to the overhead introduced by DNNs, the effort of estimating the KL divergence between Gaussians is negligible.

Original: The added cost of DE-GP primarily arises from the extra measurement points and the evaluation of the prior kernels.
Revised: The added cost of DE-GP primarily arises from the extra measurement points and the evaluation of the prior kernels.

Original: In practice, we use a small batch size
Revised: In practice, we use a small batch size

Original: [Figure]
Revised: [Figure]

Original: for the extra measurement points.
Revised: for the extra measurement points.

Original: We build the prior NN-GP kernels with cheap architectures and perform MC estimation in parallel.
Revised: We build the prior NN-GP kernels with cheap architectures and perform MC estimation in parallel.

Original: Eventually, DE-GP is only marginally slower than DE in training.
Revised: Eventually, DE-GP is only marginally slower than DE in training.

Original: Weight sharing.
Revised: Weight sharing.

Original: DE-GP does not care about how the basis functions are parameterized, so we can perform weight sharing among the basis functions, for example, using a shared feature extractor and
Revised: DE-GP does not care about how the basis functions are parameterized, so we can perform weight sharing among the basis functions, for example, using a shared feature extractor and

Original: M independent MLP classiﬁers to construct M basis functions (Deng et al., 2021).
Revised: M independent MLP classiﬁers to construct M basis functions (Deng et al., 2021).

Original: With shared weights, DE-GP is still likely to be reliable because our learning principle induces diversity in function space regardless of the weights.
Revised: With shared weights, DE-GP is still likely to be reliable because our learning principle induces diversity in function space regardless of the weights.

Original: Experiments in Section 5.3 validate this.
Revised: Experiments in Section 5.3 validate this.

Original: Limitations.
Revised: Limitations.

Original: Despite being Bayesian in principle, DE-GP is likely to be less ﬂexible than DE and other variants because all the involved basis functions need to be updated simultaneously.
Revised: Despite being Bayesian in principle, DE-GP is likely to be less ﬂexible than DE and other variants because all the involved basis functions need to be updated simultaneously.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We perform extensive evaluation to prove that DE-GP yields better uncertainty estimates than the baselines, while preserving non-degraded predictive performance.
Revised: We perform extensive evaluation to prove that DE-GP yields better uncertainty estimates than the baselines, while preserving non-degraded predictive performance.

Original: The baselines include DE, rDE,
Revised: The baselines include DE, rDE,

Original: NN-GP, RMS, etc.
Revised: NN-GP, RMS, etc.

Original: In all experiments, we estimate the prior kernel with 10 MC samples and set the sampling distribution for extra measurement points ν as the uniform distribution over the data region.
Revised: In all experiments, we estimate the prior kernel with 10 MC samples and set the sampling distribution for extra measurement points ν as the uniform distribution over the data region.

Original: 5. I LLUSTRATIVE R EGRESSION
Revised: 5. I LLUSTRATIVE R EGRESSION

Original: We build two regression problems with data from y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) and y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , respectively.
Revised: We build two regression problems with data from y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) and y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , respectively.

Original: We consider two architectures: MLPs with 2 hidden layers ofunits and MLPs with 3 hidden layers of 512 units, where ReLU activation is used.
Revised: We consider two architectures: MLPs with 2 hidden layers ofunits and MLPs with 3 hidden layers of 512 units, where ReLU activation is used.

Original: For NN-GP, we analytically estimate the GP kernel and perform GP regression without training MLPs.
Revised: For NN-GP, we analytically estimate the GP kernel and perform GP regression without training MLPs.

Original: For DE-GP,
Revised: For DE-GP,

Original: DE, rDE, and RMS, we train 50 MLPs.
Revised: DE, rDE, and RMS, we train 50 MLPs.

Original: For MC dropout, we train a MLP with the commonly used 0.2 dropout rate.
Revised: For MC dropout, we train a MLP with the commonly used 0.2 dropout rate.

Original: We use the default values for α and β , i.e., α = 1 and β = 0 .
Revised: We use the default values for α and β , i.e., α = 1 and β = 0 .

Original: Fig.
Revised: Fig.

Original: 1 and Appendix B.2 show the results on the two problems.
Revised: 1 and Appendix B.2 show the results on the two problems.

Original: We also provide the comparison on training speed in Fig.
Revised: We also provide the comparison on training speed in Fig.

Original: As shown, DE-GP delivers calibrated uncertainty estimates across settings, on par with the non-parametric Bayesian baseline NN-GP.
Revised: As shown, DE-GP delivers calibrated uncertainty estimates across settings, on par with the non-parametric Bayesian baseline NN-GP.

Original: Yet, the baselines like DE, rDE, and RMS suffer from degeneracy issue as the dimension of weights increases.
Revised: Yet, the baselines like DE, rDE, and RMS suffer from degeneracy issue as the dimension of weights increases.

Original: Though NN-GP outperforms other methods, the involved analytical GP regression may have scalability and effectiveness issues when facing modern architectures (Novak et al., 2018), while DE-GP does not suffer from them.
Revised: Though NN-GP outperforms other methods, the involved analytical GP regression may have scalability and effectiveness issues when facing modern architectures (Novak et al., 2018), while DE-GP does not suffer from them.

Original: 5. UCI R EGRESSION
Revised: 5. UCI R EGRESSION

Original: We then assess DE-GP on 5 UCI real-valued regression problems.
Revised: We then assess DE-GP on 5 UCI real-valued regression problems.

Original: The used architecture is a MLP with 2 hidden layers of 256 units and ReLU activation.
Revised: The used architecture is a MLP with 2 hidden layers of 256 units and ReLU activation.

Original: 10 networks are trained for DE, DE-GP and other variants.
Revised: 10 networks are trained for DE, DE-GP and other variants.

Original: For DE-GP, we set β = 0 and tune α according to validation sets.
Revised: For DE-GP, we set β = 0 and tune α according to validation sets.

Original: We perform cross validation with 5 splits.
Revised: We perform cross validation with 5 splits.

Original: Fig.
Revised: Fig.

Original: 3 shows the results.
Revised: 3 shows the results.

Original: DE-GP surpasses or approaches the baselines across scenarios in aspects of both test negative log-likelihood (NLL) and test root mean square error (RMSE).
Revised: DE-GP surpasses or approaches the baselines across scenarios in aspects of both test negative log-likelihood (NLL) and test root mean square error (RMSE).

Original: DE-GP even beats NN-GP, which is probably attributed to that the variational family speciﬁed by DE enjoys the beneﬁcial inductive bias of practically sized SGD-trained DNNs, and DE-GP can ﬂexibly trade off between the likelihood and the prior by tuning α .
Revised: DE-GP even beats NN-GP, which is probably attributed to that the variational family speciﬁed by DE enjoys the beneﬁcial inductive bias of practically sized SGD-trained DNNs, and DE-GP can ﬂexibly trade off between the likelihood and the prior by tuning α .

Original: 5. I MAGE C LASSIFICATION ON F ASHION -MNIST AND CIFAR- 
Revised: 5. I MAGE C LASSIFICATION ON F ASHION -MNIST AND CIFAR-

Original: In the classiﬁcation experiments, we augment the data log-likelihood (i.e., the ﬁrst term in Eq.
Revised: In the classiﬁcation experiments, we augment the data log-likelihood (i.e., the ﬁrst term in Eq.

Original: (14)) with a trainable temperature to tackle oversmoothing and avoid underconﬁdence.
Revised: (14)) with a trainable temperature to tackle oversmoothing and avoid underconﬁdence.

Original: Fashion-MNIST .
Revised: Fashion-MNIST .

Original: We use a widened LeNet5 architecture with batch normalizations (BNs) (Ioffe &
Revised: We use a widened LeNet5 architecture with batch normalizations (BNs) (Ioffe &

Original: Szegedy, 2015) for the Fashion-MNIST dataset (Xiao et al., 2017).
Revised: Szegedy, 2015) for the Fashion-MNIST dataset (Xiao et al., 2017).

Original: Considering the inefﬁciency of
Revised: Considering the inefﬁciency of

Original: 2 3 4 5 6 7 8 9 10 Ensemble size 6.
Revised: 2 3 4 5 6 7 8 9 10 Ensemble size 6.

Original: C l a ss i f
Revised: C l a ss i f

Original: i c a t i o n
Revised: i c a t i o n

Original: E rr o r ( % )
Revised: E rr o r ( % )

Original: Test Error vs. Ensemble Size DE-GP
Revised: Test Error vs. Ensemble Size DE-GP

Original: DErDERMS 2 3 4 5 6 7 8 9 10 Ensemble size 0.
Revised: DErDERMS 2 3 4 5 6 7 8 9 10 Ensemble size 0.

Original: N e g a t
Revised: N e g a t

Original: i v e L o g - li k e li h
Revised: i v e L o g - li k e li h

Original: oo d
Revised: oo d

Original: Test NLL vs. Ensemble Size DE-GP
Revised: Test NLL vs. Ensemble Size DE-GP

Original: DErDERMS 1.
Revised: DErDERMS 1.

Original: 0.2 Uncertainty Threshold 0.
Revised: 0.2 Uncertainty Threshold 0.

Original: E rr
Revised: E rr

Original: o r o n e x a m p l e s w i
Revised: o r o n e x a m p l e s w i

Original: t
Revised: t

Original: h un c e r t a i n t y
Revised: h un c e r t a i n t y

Original: FashionMNIST+MNIST Error vs. Uncertainty
Revised: FashionMNIST+MNIST Error vs. Uncertainty

Original: DE-GP DE rDE RMS
Revised: DE-GP DE rDE RMS

Original: [Table]
Revised: [Table]

Original: NN-GP, we mainly compare DE-GP to DE, rDE, and RMS.
Revised: NN-GP, we mainly compare DE-GP to DE, rDE, and RMS.

Original: We set α as well as the regularization coefﬁcients for rDE and RMS all as 0 .
Revised: We set α as well as the regularization coefﬁcients for rDE and RMS all as 0 .

Original: 1 according to validation accuracy.
Revised: 1 according to validation accuracy.

Original: For DE-GP, we use β =given the limited capacity of the architecture.
Revised: For DE-GP, we use β =given the limited capacity of the architecture.

Original: The in-distribution performance is averaged over 3 runs.
Revised: The in-distribution performance is averaged over 3 runs.

Original: Fig. 4-(Left) and Fig. 4-(Middle) display how ensemble size impacts the test results.
Revised: Fig. 4-(Left) and Fig. 4-(Middle) display how ensemble size impacts the test results.

Original: Surprisingly, the test error of DE-GP is even lower than the baselines.
Revised: Surprisingly, the test error of DE-GP is even lower than the baselines.

Original: Besides, to compare the quality of uncertainty estimates, we use the trained models to make prediction and quantify epistemic uncertainty for both the in-distribution test set and the out-of-distribution (OOD) MNIST test set.
Revised: Besides, to compare the quality of uncertainty estimates, we use the trained models to make prediction and quantify epistemic uncertainty for both the in-distribution test set and the out-of-distribution (OOD) MNIST test set.

Original: All predictions on OOD data are regarded as wrong.
Revised: All predictions on OOD data are regarded as wrong.

Original: The epistemic uncertainty is estimated by the mutual information between the prediction and the variable function:
Revised: The epistemic uncertainty is estimated by the mutual information between the prediction and the variable function:

Original: [Equation]
Revised: [Equation]

Original: where H indicates Shannon entropy, with f s = g ( · , w s ) for DE, rDE, and RMS, and f s ∼ q ( f ; w 1 , ..., w M ) for DE-GP.
Revised: where H indicates Shannon entropy, with f s = g ( · , w s ) for DE, rDE, and RMS, and f s ∼ q ( f ; w 1 , ..., w M ) for DE-GP.

Original: We normalize the uncertainty estimates into [0 , 1] .
Revised: We normalize the uncertainty estimates into [0 , 1] .

Original: For each threshold τ ∈ [0 , 1] , we plot the average test error for data with ≤ τ uncertainty in Fig. 4-(Right).
Revised: For each threshold τ ∈ [0 , 1] , we plot the average test error for data with ≤ τ uncertainty in Fig. 4-(Right).

Original: It is prominent that under various uncertainty thresholds, DE-GP makes fewer mistakes than the baselines, implying that DE-GP succeeds to assign relatively higher uncertainty for the OOD data.
Revised: It is prominent that under various uncertainty thresholds, DE-GP makes fewer mistakes than the baselines, implying that DE-GP succeeds to assign relatively higher uncertainty for the OOD data.

Original: CIFAR-10 .
Revised: CIFAR-10 .

Original: Next, we apply DE-GP to the real-world image classiﬁcation task CIFAR-10 (Krizhevsky et al., 2009).
Revised: Next, we apply DE-GP to the real-world image classiﬁcation task CIFAR-10 (Krizhevsky et al., 2009).

Original: We consider the popular ResNet architectures (He et al., 2016) including ResNetand ResNet-56.
Revised: We consider the popular ResNet architectures (He et al., 2016) including ResNetand ResNet-56.

Original: The ensemble size is ﬁxed as 10 .
Revised: The ensemble size is ﬁxed as 10 .

Original: We split the data as training set, validation set, and test set of size 45000, 5000, 10000, respectively.
Revised: We split the data as training set, validation set, and test set of size 45000, 5000, 10000, respectively.

Original: We set β = 0 .
Revised: We set β = 0 .

Original: 1 , equivalent with the regularization coefﬁcient on weight in rDE.
Revised: 1 , equivalent with the regularization coefﬁcient on weight in rDE.

Original: We set α = 0 .
Revised: We set α = 0 .

Original: 1 according to an ablation study in Appendix B.5.
Revised: 1 according to an ablation study in Appendix B.5.

Original: We use a lite ResNet-20 architecture without BNs and residual connections to set up the prior NN-GP kernel for both the ResNet-20 and ResNet-56 based variational posteriors.
Revised: We use a lite ResNet-20 architecture without BNs and residual connections to set up the prior NN-GP kernel for both the ResNet-20 and ResNet-56 based variational posteriors.

Original: We present the in-distribution test accuracy in Table 1 and the error versus uncertainty plots on the combination of CIFAR-10 and SVHN test sets in Fig.
Revised: We present the in-distribution test accuracy in Table 1 and the error versus uncertainty plots on the combination of CIFAR-10 and SVHN test sets in Fig.

Original: It is noteworthy that DE-GP is on par with the practically-used, competing rDE in aspect of test accuracy.
Revised: It is noteworthy that DE-GP is on par with the practically-used, competing rDE in aspect of test accuracy.

Original: DE-GP ( β = 0 ) shows unsatisfactory test accuracy, verifying the necessity of performing posterior regularization to penalize function complexity when using ultra-deep networks.
Revised: DE-GP ( β = 0 ) shows unsatisfactory test accuracy, verifying the necessity of performing posterior regularization to penalize function complexity when using ultra-deep networks.

Original: The error versus uncertainty plots are similar to those for Fashion-MNIST, substantiating the universality of DE-GP.
Revised: The error versus uncertainty plots are similar to those for Fashion-MNIST, substantiating the universality of DE-GP.

Original: We further test the trained methods on CIFAR-10 corruptions (Hendrycks & Dietterich, 2018), a challenging OOD generalization/robustness benchmark for deep models.
Revised: We further test the trained methods on CIFAR-10 corruptions (Hendrycks & Dietterich, 2018), a challenging OOD generalization/robustness benchmark for deep models.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6 and
Revised: 6 and

Original: Appendix B.3, DE-GP reveals smaller Expected Calibration Error (ECE) (Guo et al., 2017) and lower NLL at various levels of skew, reﬂecting its ability to make conservative predictions under corruptions and good OOD robustness.
Revised: Appendix B.3, DE-GP reveals smaller Expected Calibration Error (ECE) (Guo et al., 2017) and lower NLL at various levels of skew, reﬂecting its ability to make conservative predictions under corruptions and good OOD robustness.

Original: More results for the deeper ResNet-110 architecture and the more challenging CIFAR-100 benchmark are provided in Appendix B.3 and Appendix B.4.
Revised: More results for the deeper ResNet-110 architecture and the more challenging CIFAR-100 benchmark are provided in Appendix B.3 and Appendix B.4.

Original: [Figure]
Revised: [Figure]

Original: x
Revised: x

Original: [Figure]
Revised: [Figure]

Original: p l
Revised: p l

Original: [Figure]
Revised: [Figure]

Original: i t
Revised: i t

Original: [Figure]
Revised: [Figure]

Original: t a
Revised: t a

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Weight Sharing.
Revised: Weight Sharing.

Original: We build a ResNet-20 with 10 classiﬁcation heads and a shared feature extraction module to evaluate the methods under weight sharing.
Revised: We build a ResNet-20 with 10 classiﬁcation heads and a shared feature extraction module to evaluate the methods under weight sharing.

Original: We set a larger value for α for DE-GP to induce higher magnitudes of functional diversity.
Revised: We set a larger value for α for DE-GP to induce higher magnitudes of functional diversity.

Original: The test accuracy (over 6 trials) and error versus uncertainty plots on CIFAR-10 are illustrated in Fig.
Revised: The test accuracy (over 6 trials) and error versus uncertainty plots on CIFAR-10 are illustrated in Fig.

Original: We exclude RMS from the comparison as it assumes i.i.d.
Revised: We exclude RMS from the comparison as it assumes i.i.d.

Original: basis functions which may be incompatible with weight sharing.
Revised: basis functions which may be incompatible with weight sharing.

Original: DE-GP beneﬁts from the diversity in function space, hence performs better than DE and rDE, which purely hinge on the diversity in weight space.
Revised: DE-GP beneﬁts from the diversity in function space, hence performs better than DE and rDE, which purely hinge on the diversity in weight space.

Original: 5. C ONTEXTUAL B ANDIT
Revised: 5. C ONTEXTUAL B ANDIT

Original: [Figure]
Revised: [Figure]

Original: Finally, we apply DE-GP to contextual bandit, an important decision-making task where the uncertainty helps to guide exploration.
Revised: Finally, we apply DE-GP to contextual bandit, an important decision-making task where the uncertainty helps to guide exploration.

Original: Following (Osband et al., 2016), we use
Revised: Following (Osband et al., 2016), we use

Original: DE-GP to achieve efﬁcient exploration inspired by Thompson sampling.
Revised: DE-GP to achieve efﬁcient exploration inspired by Thompson sampling.

Original: We reuse most of the settings for UCI regression.
Revised: We reuse most of the settings for UCI regression.

Original: We leverage the GenRL library to build two contextual bandit problems Covertype and Mushroom (Riquelme et al., 2018).
Revised: We leverage the GenRL library to build two contextual bandit problems Covertype and Mushroom (Riquelme et al., 2018).

Original: The cumulative reward is depicted in Fig. 8.
Revised: The cumulative reward is depicted in Fig. 8.

Original: As desired, DE-GP offers better uncertainty estimates and hence beats the baselines by clear margins.
Revised: As desired, DE-GP offers better uncertainty estimates and hence beats the baselines by clear margins.

Original: The potential of DE-GP in more reinforcement learning and Bayesian optimization scenarios deserves future investigation.
Revised: The potential of DE-GP in more reinforcement learning and Bayesian optimization scenarios deserves future investigation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we attempt to build a Bayesian reﬁnement of Deep Ensemble by drawing inspiration from the connection between BNNs and GPs.
Revised: In this work, we attempt to build a Bayesian reﬁnement of Deep Ensemble by drawing inspiration from the connection between BNNs and GPs.

Original: We propose to leverage the ensemble members to specify an adaptive GP approximate posterior, and perform variational inference directly in function space.
Revised: We propose to leverage the ensemble members to specify an adaptive GP approximate posterior, and perform variational inference directly in function space.

Original: We inherit some theoretical results from existing works on functional variational inference, and further develop a posterior regularization scheme to conveniently induce prior knowledge on function properties.
Revised: We inherit some theoretical results from existing works on functional variational inference, and further develop a posterior regularization scheme to conveniently induce prior knowledge on function properties.

Original: We demonstrate how to penalize the function complexity based on this scheme and empirically show the necessity of doing so.
Revised: We demonstrate how to penalize the function complexity based on this scheme and empirically show the necessity of doing so.

Original: The whole algorithm can be implemented easily and efﬁciently.
Revised: The whole algorithm can be implemented easily and efﬁciently.

Original: Extensive experiments validate the effectiveness of our method.
Revised: Extensive experiments validate the effectiveness of our method.

Original: We hope this work may shed light on the development of better Bayesian deep learning approaches.
Revised: We hope this work may shed light on the development of better Bayesian deep learning approaches.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This work proposes a Bayesian reﬁnement of Deep Ensemble.
Revised: This work proposes a Bayesian reﬁnement of Deep Ensemble.

Original: Its potential positive impacts in the society are evident: its ability to enable better uncertainty estimation while maintaining predictive performance is crucial in industry, e g., automatic driving, disease analysis, and ﬁnancial applications.
Revised: Its potential positive impacts in the society are evident: its ability to enable better uncertainty estimation while maintaining predictive performance is crucial in industry, e g., automatic driving, disease analysis, and ﬁnancial applications.

Original: In this scenarios the uncertainty estimates could be used to reject uncertain predictions and raise the requirement of inviting humans into the decision process.
Revised: In this scenarios the uncertainty estimates could be used to reject uncertain predictions and raise the requirement of inviting humans into the decision process.

Original: As a fundamental research in machine learning, the negative consequences are not obvious.
Revised: As a fundamental research in machine learning, the negative consequences are not obvious.

Original: Though in theory any technique can be misused, it is not likely to happen at the current stage
Revised: Though in theory any technique can be misused, it is not likely to happen at the current stage

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: It is easy to reproduce the experiments given the code in the supplementary material.
Revised: It is easy to reproduce the experiments given the code in the supplementary material.

Original: A P ENALIZE F UNCTION C OMPLEXITY IN M ULTI - CLASS C LASSIFICATION
Revised: A P ENALIZE F UNCTION C OMPLEXITY IN M ULTI - CLASS C LASSIFICATION

Original: In the multi-class classiﬁcation scenario where y ∈ { 1 , 2 , ..., C } and f, g i : X → R C , we use the loss (cid:96) ( f ( x ) , y )
Revised: In the multi-class classiﬁcation scenario where y ∈ { 1 , 2 , ..., C } and f, g i : X → R C , we use the loss (cid:96) ( f ( x ) , y )

Original: =
Revised: =

Original: 1 f ( x )[ y ] < max y (cid:48)(cid:54) =
Revised: 1 f ( x )[ y ] < max y (cid:48)(cid:54) =

Original: y f ( x )[ y (cid:48) ] to measure prediction error where f ( x )[ j ] denotes j -th coordinate of f ( x ) .
Revised: y f ( x )[ y (cid:48) ] to measure prediction error where f ( x )[ j ] denotes j -th coordinate of f ( x ) .

Original: The distinct difference between this scenario and the binary classiﬁcation scenario is that in this setting, ζ ( x ) is a vector-valued function:
Revised: The distinct difference between this scenario and the binary classiﬁcation scenario is that in this setting, ζ ( x ) is a vector-valued function:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) 0 ∼ N ( 0 , I C ) and (cid:15) i ∼ N (0 , 1) , i = 1 , ..., M .
Revised: where (cid:15) 0 ∼ N ( 0 , I C ) and (cid:15) i ∼ N (0 , 1) , i = 1 , ..., M .

Original: We then make a mild assumption to simplify the analysis.
Revised: We then make a mild assumption to simplify the analysis.

Original: Assumption 1.
Revised: Assumption 1.

Original: For any ( x , y ) ∈ µ , the elements on the diagonal of k q ( x , x ) have the same value.
Revised: For any ( x , y ) ∈ µ , the elements on the diagonal of k q ( x , x ) have the same value.

Original: This assumption implies that for any j, j (cid:48) ∈ { 1 , ..., C } ,
Revised: This assumption implies that for any j, j (cid:48) ∈ { 1 , ..., C } ,

Original: [Equation]
Revised: [Equation]

Original: I.e.,
Revised: I.e.,

Original: [Equation]
Revised: [Equation]

Original: Therefore, ζ ( x ) possesses the same variance across its output coordinates and becomes a random guess classiﬁer.
Revised: Therefore, ζ ( x ) possesses the same variance across its output coordinates and becomes a random guess classiﬁer.

Original: Based on this, we have E q ( f ) (cid:96) ( ζ ( x ) , y )
Revised: Based on this, we have E q ( f ) (cid:96) ( ζ ( x ) , y )

Original: =
Revised: =

Original: E (cid:15) 0 ,...,(cid:15) M (cid:96) ( ζ ( x ) , y )
Revised: E (cid:15) 0 ,...,(cid:15) M (cid:96) ( ζ ( x ) , y )

Original: =
Revised: =

Original: ( C − 1) /C .
Revised: ( C − 1) /C .

Original: We can then derive a similar conclusion to that in Section 4.3.
Revised: We can then derive a similar conclusion to that in Section 4.3.

Original: The validity of Assumption 1.
Revised: The validity of Assumption 1.

Original: When the data dimension |X| is high and the number of training data n is ﬁnitely large, with zero probability the sampled data ( x , y ) ∼ µ resides in the training set.
Revised: When the data dimension |X| is high and the number of training data n is ﬁnitely large, with zero probability the sampled data ( x , y ) ∼ µ resides in the training set.

Original: Therefore, only the KL divergence term of the fELBO explicitly affects the predictive uncertainty at x .
Revised: Therefore, only the KL divergence term of the fELBO explicitly affects the predictive uncertainty at x .

Original: Because the NN-GP prior possesses a diagonal structure, it is hence reasonable to make the above assumption.
Revised: Because the NN-GP prior possesses a diagonal structure, it is hence reasonable to make the above assumption.

Original: B M ORE OF E XPERIMENTS
Revised: B M ORE OF E XPERIMENTS

Original: We provide more experimental details and results in this section.
Revised: We provide more experimental details and results in this section.

Original: In all experiments, the number of
Revised: In all experiments, the number of

Original: MC samples for estimating the expected log-likelihood (i.e., U in Line 7 of Algorithm 1) is set as 256 .
Revised: MC samples for estimating the expected log-likelihood (i.e., U in Line 7 of Algorithm 1) is set as 256 .

Original: Unless otherwise stated, we set the regularization constant λ as 0 .
Revised: Unless otherwise stated, we set the regularization constant λ as 0 .

Original: 05 times of the average eigenvalue of the central covariance matrices, and set the weight and bias variance for deﬁning the NN-GP prior kernel at each layer as 2 /fan _ in and 0 .
Revised: 05 times of the average eigenvalue of the central covariance matrices, and set the weight and bias variance for deﬁning the NN-GP prior kernel at each layer as 2 /fan _ in and 0 .

Original: 01 , where fan _ in is the number of input features, as suggested by He initialization (He et al., 2015).
Revised: 01 , where fan _ in is the number of input features, as suggested by He initialization (He et al., 2015).

Original: B. D ETAILED S ETTINGS
Revised: B. D ETAILED S ETTINGS

Original: Illustrative regression.
Revised: Illustrative regression.

Original: For the problem on y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) , we randomly sampledata points from [ − 1 . 5 , − 0 . 6] ∪ [0 . 6 , 1 . 5] .
Revised: For the problem on y = x 3 / 4 + (cid:15), (cid:15) ∼ N (0 , 0 . 1) , we randomly sampledata points from [ − 1 . 5 , − 0 . 6] ∪ [0 . 6 , 1 . 5] .

Original: For the problem on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , we use 6 data points {− 0 .
Revised: For the problem on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) , we use 6 data points {− 0 .

Original: 8 , − 0 .
Revised: 8 , − 0 .

Original: 8 } following (Pearce et al., 2020).
Revised: 8 } following (Pearce et al., 2020).

Original: For optimizing the
Revised: For optimizing the

Original: DNN basis functions, we use an Adam (Kingma & Ba, 2015) optimizer with 0 .
Revised: DNN basis functions, we use an Adam (Kingma & Ba, 2015) optimizer with 0 .

Original: 003 learning rate.
Revised: 003 learning rate.

Original: The optimization takes 300 iterations.
Revised: The optimization takes 300 iterations.

Original: We set hidden size as 64 for the architecture of the prior NN-GP kernel for speedup.
Revised: We set hidden size as 64 for the architecture of the prior NN-GP kernel for speedup.

Original: The regularization constant λ is set as 1 e − 4 times of the average eigenvalue of the central covariance matrices.
Revised: The regularization constant λ is set as 1 e − 4 times of the average eigenvalue of the central covariance matrices.

Original: UCI regression.
Revised: UCI regression.

Original: We pre-process the UCI data by standard normalization.
Revised: We pre-process the UCI data by standard normalization.

Original: We set the variance for data noise and the weight variance for the prior kernel following (Pearce et al., 2020).
Revised: We set the variance for data noise and the weight variance for the prior kernel following (Pearce et al., 2020).

Original: The batch size for stochastic training is 256 .
Revised: The batch size for stochastic training is 256 .

Original: We use an Adam optimizer to optimize for 1000 epochs.
Revised: We use an Adam optimizer to optimize for 1000 epochs.

Original: The learning rate is initialized as 0 . 01 and decays by 0 .
Revised: The learning rate is initialized as 0 . 01 and decays by 0 .

Original: 99 every 5 epochs.
Revised: 99 every 5 epochs.

Original: Fashion-MNIST classiﬁcation.
Revised: Fashion-MNIST classiﬁcation.

Original: The used architecture is Conv(32, 3, 1)-BN-ReLU-MaxPool(2)-
Revised: The used architecture is Conv(32, 3, 1)-BN-ReLU-MaxPool(2)-

Original: Conv(64, 3, 0)-BN-ReLU-MaxPool(2)-Linear(256)-ReLU-Linear(10), where Conv( x , y , z ) represents
Revised: Conv(64, 3, 0)-BN-ReLU-MaxPool(2)-Linear(256)-ReLU-Linear(10), where Conv( x , y , z ) represents

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N e g a t
Revised: N e g a t

Original: i ve
Revised: i v

Original: L

Original: [Figure]
Revised: [Figure]

Original: k
Revised: L o

Original: [Figure]
Revised: [Figure]

Original: li 
Revised: li k e li

Original: [Figure]
Revised: [Figure]

Original: a 2D convolution with x output channels, kernel size y , and padding z .
Revised: a 2D convolution with x output channels, kernel size y , and padding z .

Original: The batch size for training data is 64 .
Revised: The batch size for training data is 64 .

Original: The batch size for extra measurement points is 0 .
Revised: The batch size for extra measurement points is 0 .

Original: We use an SGD optimizer to optimize for 24 epochs.
Revised: We use an SGD optimizer to optimize for 24 epochs.

Original: The learning rate is initialized as 0 . 1 and follows a cosine decay schedule.
Revised: The learning rate is initialized as 0 . 1 and follows a cosine decay schedule.

Original: We use an
Revised: We use an

Original: Adam optimizer with 1 e − 3 learning rate to optimize the temperature.
Revised: Adam optimizer with 1 e − 3 learning rate to optimize the temperature.

Original: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty, because the involved computation is only the cheap softmax transformation on the sampled function values.
Revised: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty, because the involved computation is only the cheap softmax transformation on the sampled function values.

Original: CIFAR-10 classiﬁcation.
Revised: CIFAR-10 classiﬁcation.

Original: We perform data augmentation including random horizontal ﬂip and random crop.
Revised: We perform data augmentation including random horizontal ﬂip and random crop.

Original: The batch size for training data is 128 .
Revised: The batch size for training data is 128 .

Original: The batch size for extra measurement points is 0 .
Revised: The batch size for extra measurement points is 0 .

Original: We use a SGD optimizer with 0 .
Revised: We use a SGD optimizer with 0 .

Original: 9 momentum to optimize for 200 epochs.
Revised: 9 momentum to optimize for 200 epochs.

Original: The learning rate is initialized as 0 . 1 and decays by 0 .
Revised: The learning rate is initialized as 0 . 1 and decays by 0 .

Original: 1 at 100 -th and 150 -th epochs.
Revised: 1 at 100 -th and 150 -th epochs.

Original: We use an Adam optimizer with 1 e − 3 learning rate to optimize the temperature.
Revised: We use an Adam optimizer with 1 e − 3 learning rate to optimize the temperature.

Original: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty.
Revised: We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty.

Revised: Suggested by (Ovadia et al., 2019; He et al., 2020), we train models on CIFAR-10, and test them on the combination of CIFAR-10 and SVHN test sets.

Revised: This is a standard benchmark for evaluating the uncertainty on OOD data.

Original: Contextual bandit.
Revised: Contextual bandit.

Original: We use MLPs with 2 hidden layers of 256 units.
Revised: We use MLPs with 2 hidden layers of 256 units.

Original: The batch size for training data is 512.
Revised: The batch size for training data is 512.

Original: The batch size for extra measurement points is 0.
Revised: The batch size for extra measurement points is 0.

Original: We update the model (i.e., the agent) forepochs with an Adam optimizer every 50 rounds.
Revised: We update the model (i.e., the agent) forepochs with an Adam optimizer every 50 rounds.

Original: We set α = 1 and β = 0 without tuning.
Revised: We set α = 1 and β = 0 without tuning.

Original: DE, rDE, and RMS all randomly choose an ensemble member at per iteration, but our method randomly draws a sample from the GP for decision.
Revised: DE, rDE, and RMS all randomly choose an ensemble member at per iteration, but our method randomly draws a sample from the GP for decision.

Original: This is actually emulating Thompson Sampling and advocated by
Revised: This is actually emulating Thompson Sampling and advocated by

Original: Bootstrapped DQN (Osband et al., 2016).
Revised: Bootstrapped DQN (Osband et al., 2016).

Original: “Random” baseline corresponds to the Uniform algorithm.
Revised: “Random” baseline corresponds to the Uniform algorithm.

Original: B. M ORE R ESULTS ON I LLUSTRATIVE R EGRESSION
Revised: B. M ORE R ESULTS ON I LLUSTRATIVE R EGRESSION

Original: We depict the results on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) problem in Fig.
Revised: We depict the results on y = x sin 5 x + (cid:15), (cid:15) ∼ N (0 , 0 . 2) problem in Fig.

Original: 9, which are consistently with those in Fig.
Revised: 9, which are consistently with those in Fig.

Original: B. M ORE R ESULTS ON CIFAR-10 C LASSIFICATION
Revised: B. M ORE R ESULTS ON CIFAR-10 C LASSIFICATION

Original: We plot the negative log-likelihood and test accuracy on CIFAR-10 corruptions for models trained with ResNet-20 and ResNet-56 in Fig.
Revised: We plot the negative log-likelihood and test accuracy on CIFAR-10 corruptions for models trained with ResNet-20 and ResNet-56 in Fig.

Original: 10 and Fig.
Revised: 10 and Fig.

Original: As shown, DE-GP outperforms the baselines in aspect of negative log-likelihood, but yields similar test accuracy to the baselines.
Revised: As shown, DE-GP outperforms the baselines in aspect of negative log-likelihood, but yields similar test accuracy to the baselines.

Original: Recapping the results in Fig.
Revised: Recapping the results in Fig.

Original: 6, DE-GP indeed has improved OOD robustness, but may still face problems in OOD generalization.
Revised: 6, DE-GP indeed has improved OOD robustness, but may still face problems in OOD generalization.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We then conduct experiments with the deeper ResNet-110 architecture.
Revised: We then conduct experiments with the deeper ResNet-110 architecture.

Original: Due to resource constraint, we use 5 ensemble members.
Revised: Due to resource constraint, we use 5 ensemble members.

Original: The other settings are roughly the same as those for ResNet-56.
Revised: The other settings are roughly the same as those for ResNet-56.

Original: The results are offered in Fig.
Revised: The results are offered in Fig.

Original: 12, which validate the effectiveness of DE-GP for large networks.
Revised: 12, which validate the effectiveness of DE-GP for large networks.

Original: B. R ESULTS ON CIFAR- 
Revised: B. R ESULTS ON CIFAR-

Original: We further perform experiments on the more challenging CIFAR-100 benchmark.
Revised: We further perform experiments on the more challenging CIFAR-100 benchmark.

Original: We present the in-distribution test accuracy of DE-GP as well as the baselines in Table 2.
Revised: We present the in-distribution test accuracy of DE-GP as well as the baselines in Table 2.

Original: We can see that DE-GP ( β = 0 . 1 ) is still on par with rDE.
Revised: We can see that DE-GP ( β = 0 . 1 ) is still on par with rDE.

Original: We depict the error versus uncertainty plots on the combination of CIFAR-100 and SVHN test sets in Fig.
Revised: We depict the error versus uncertainty plots on the combination of CIFAR-100 and SVHN test sets in Fig.

Original: It is shown that the uncertainty estimates yielded by
Revised: It is shown that the uncertainty estimates yielded by

Original: DE-GP for OOD data are more calibrated than the baselines.
Revised: DE-GP for OOD data are more calibrated than the baselines.

Original: We further test the trained methods on
Revised: We further test the trained methods on

Original: CIFAR-100 corruptions (Hendrycks & Dietterich, 2018), and present the comparisons in aspects of test accuracy and NLL in Fig.
Revised: CIFAR-100 corruptions (Hendrycks & Dietterich, 2018), and present the comparisons in aspects of test accuracy and NLL in Fig.

Original: It is evident that DE-GP reveals lower NLL than the baselines at various levels of skew.
Revised: It is evident that DE-GP reveals lower NLL than the baselines at various levels of skew.

Original: [Table]
Revised: [Table]

Original: B. A BLATION S TUDY ON α
Revised: B. A BLATION S TUDY ON α

Original: We have conducted an ablation study on α (using ResNet-20 on CIFAR-10).
Revised: We have conducted an ablation study on α (using ResNet-20 on CIFAR-10).

Original: The results are presented in Table 3.
Revised: The results are presented in Table 3.

Original: We can see that DE-GP is not sensitive to the value of α .
Revised: We can see that DE-GP is not sensitive to the value of α .

Original: We in practice set α = 0 .
Revised: We in practice set α = 0 .

Original: 01 in the CIFAR experiments.
Revised: 01 in the CIFAR experiments.

Original: We did not use a smaller α as it may result in colder posteriors and in turn worse uncertainty estimates.
Revised: We did not use a smaller α as it may result in colder posteriors and in turn worse uncertainty estimates.

Original: [Table]
Revised: [Table]

Original: . A BLATION S TUDY ON THE A RCHITECTURE OF P RIOR K ERNEL
Revised: . A BLATION S TUDY ON THE A RCHITECTURE OF P RIOR K ERNEL

Original: We perform an ablation study on the architecture for deﬁning the prior MC NN-GP kernel, with the results listed in Table 4.
Revised: We perform an ablation study on the architecture for deﬁning the prior MC NN-GP kernel, with the results listed in Table 4.

Original: Surprisingly, using the cheap ResNet-20 architecture results in DE-GP with
Revised: Surprisingly, using the cheap ResNet-20 architecture results in DE-GP with

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: r t a i n t y
Revised: r t a i n t y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t i v
Revised: t i v

Original: [Figure]
Revised: [Figure]

Original: li k
Revised: li k

Original: [Figure]
Revised: [Figure]

Original: li
Revised: li

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original: better test accuracy.
Revised: better test accuracy.

Original: We deduce this is because a deeper prior architecture induces more complex, black-box correlation for the function, which may lead to over-regularization.
Revised: We deduce this is because a deeper prior architecture induces more complex, black-box correlation for the function, which may lead to over-regularization.

Original: [Table]
Revised: [Table]