Original: G ENERATIVE P OSTERIOR N ETWORKS FOR A PPROXIMATELY B AYESIAN E PISTEMIC U NCERTAINTY E STIMATION
Revised: G ENERATIVE P OSTERIOR N ETWORKS FOR A PPROXIMATELY B AYESIAN E PISTEMIC U NCERTAINTY E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Ensembles of neural networks are often used to estimate epistemic uncertainty in high-dimensional problems because of their scalability and ease of use.
Revised: Ensembles of neural networks are often used to estimate epistemic uncertainty in high-dimensional problems because of their scalability and ease of use.

Original: These methods, however, are expensive to sample from as each sample requires a new neural network to be trained from scratch.
Revised: These methods, however, are expensive to sample from as each sample requires a new neural network to be trained from scratch.

Original: We propose a new method, Generative
Revised: We propose a new method, Generative

Original: Posterior Networks (GPNs), a generative model that, given a prior distribution over functions, approximates the posterior distribution directly by regularizing the network towards samples from the prior.
Revised: Posterior Networks (GPNs), a generative model that, given a prior distribution over functions, approximates the posterior distribution directly by regularizing the network towards samples from the prior.

Original: This allows our method to quickly sample from the posterior and construct conﬁdence bounds.
Revised: This allows our method to quickly sample from the posterior and construct conﬁdence bounds.

Original: We prove theoretically that our method indeed approximates the Bayesian posterior and show empirically that it improves epistemic uncertainty estimation over competing methods.
Revised: We prove theoretically that our method indeed approximates the Bayesian posterior and show empirically that it improves epistemic uncertainty estimation over competing methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning systems are becoming ubiquitous in many industries, speciﬁcally in applications with large datasets and where prediction mistakes are inexpensive.
Revised: Deep learning systems are becoming ubiquitous in many industries, speciﬁcally in applications with large datasets and where prediction mistakes are inexpensive.

Original: However, for many other industries, such as autonomous manufacturing, where datasets are smaller and mistakes can be catastrophic, the adoption of deep learning techniques has been slow.
Revised: However, for many other industries, such as autonomous manufacturing, where datasets are smaller and mistakes can be catastrophic, the adoption of deep learning techniques has been slow.

Original: One of the critical challenges to applying deep learning models in these safety-critical domains is the lack of uncertainty estimation.
Revised: One of the critical challenges to applying deep learning models in these safety-critical domains is the lack of uncertainty estimation.

Original: If a model makes a prediction in a safety critical environment, we would like that model to be highly conﬁdent that prediction is correct.
Revised: If a model makes a prediction in a safety critical environment, we would like that model to be highly conﬁdent that prediction is correct.

Original: Uncertainty prediction can also provide signiﬁcant beneﬁt in data-poor domains by guiding exploration towards areas of uncertainty.
Revised: Uncertainty prediction can also provide signiﬁcant beneﬁt in data-poor domains by guiding exploration towards areas of uncertainty.

Original: Most deep learning models, however, are not able to estimate epistemic uncertainty – uncertainty deriving from lack of data samples.
Revised: Most deep learning models, however, are not able to estimate epistemic uncertainty – uncertainty deriving from lack of data samples.

Original: In this work, we introduce a new method for modeling epistemic uncertainty in neural networks.
Revised: In this work, we introduce a new method for modeling epistemic uncertainty in neural networks.

Original: We formulate this problem as a Bayesian Inference problem: given a prior distribution over functions and a set of training data, we want to model the posterior function distribution.
Revised: We formulate this problem as a Bayesian Inference problem: given a prior distribution over functions and a set of training data, we want to model the posterior function distribution.

Original: There are many existing approaches to constructing such a posterior, such as Gaussian Processes (GPs) or Variational
Revised: There are many existing approaches to constructing such a posterior, such as Gaussian Processes (GPs) or Variational

Original: Inference (VI) methods, but these methods can require signiﬁcant engineering to work in practice and can be expensive in high-dimensional settings.
Revised: Inference (VI) methods, but these methods can require signiﬁcant engineering to work in practice and can be expensive in high-dimensional settings.

Original: For this reason, many practitioners turn to using  ensembles of neural networks to approximate epistemic uncertainty for their ease of use in highdimensional settings.
Revised: For this reason, many practitioners turn to using [Figure] ensembles of neural networks to approximate epistemic uncertainty for their ease of use in highdimensional settings.

Original: Pearce et al.
Revised: Pearce et al.

Original: (2020) illustrated that, given the correct regularization, these ensemble members approximate samples from the posterior distribution of functions.
Revised: (2020) illustrated that, given the correct regularization, these ensemble members approximate samples from the posterior distribution of functions.

Original: However, using this technique, every new sample of the posterior requires training a new neural network from scratch, which can be prohibitively expensive for large problems.
Revised: However, using this technique, every new sample of the posterior requires training a new neural network from scratch, which can be prohibitively expensive for large problems.

Original: To address this challenge, we introduce Generative Posterior Networks (GPNs), a generative neural network model that directly approximates the posterior distribution by regularizing the output of the network towards samples from the prior distribution.
Revised: To address this challenge, we introduce Generative Posterior Networks (GPNs), a generative neural network model that directly approximates the posterior distribution by regularizing the output of the network towards samples from the prior distribution.

Original: By learning a low-dimensional latent representation of the posterior, our method can quickly sample from the posterior and construct conﬁdence intervals.
Revised: By learning a low-dimensional latent representation of the posterior, our method can quickly sample from the posterior and construct conﬁdence intervals.

Original: We prove that our method approximates the Bayesian posterior over functions and show empirically that our method can improvesepistemic uncertainty estimation over competing methods.
Revised: We prove that our method approximates the Bayesian posterior over functions and show empirically that our method can improves uncertainty estimation over competing methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There are two categories of uncertainty in the context of probabilistic modeling: aleatoric uncertainty and epistemic uncertainty.
Revised: There are two categories of uncertainty in the context of probabilistic modeling: aleatoric uncertainty and epistemic uncertainty.

Original: Aleatoric uncertainty refers to uncertainty inherent in a system, even when the true parameters of the system are known, while epistemic uncertainty refers to modeling uncertainty that can be reduced through collecting more data (Der Kiureghian & Ditlevsen, 2009;
Revised: Aleatoric uncertainty refers to uncertainty inherent in a system, even when the true parameters of the system are known, while epistemic uncertainty refers to modeling uncertainty that can be reduced through collecting more data (Der Kiureghian & Ditlevsen, 2009;

Original: Gal, 2016).
Revised: Gal, 2016).

Original: Deep learning models have shown impressive performance in quantifying aleatoric uncertainty when given enough data.
Revised: Deep learning models have shown impressive performance in quantifying aleatoric uncertainty when given enough data.

Original: On the other hand, although there have been many methods seeking to estimate epistemic uncertainty due to its well-motivated applications in Bayesian Optimization (Snoek et al., 2012; Springenberg et al., 2016), Reinforcement Learning (Curi* et al., 2020), and adversarial robustness (Stutz et al., 2020), quantifying it remains a challenging problem for deep learning models.
Revised: On the other hand, despite many methods seeking to estimate epistemic uncertainty due to its well-motivated applications in Bayesian Optimization (Snoek et al., 2012; Springenberg et al., 2016), Reinforcement Learning (Curi* et al., 2020), and adversarial robustness (Stutz et al., 2020), quantifying it remains a challenging problem for deep learning models.

Original: Gaussian Processes (GPs) remain one of the most effective methods for epistemic uncertainty estimation in low-dimensional, low-data settings.
Revised: Gaussian Processes (GPs) remain one of the most effective methods for epistemic uncertainty estimation in low-dimensional, low-data settings.

Original: While scalability can be improved with interpolation methods (Hensman et al., 2013; Titsias, 2009; Wilson & Nickisch, 2015) and kernel learning methods (Wilson et al., 2016), GPs tend to still require signiﬁcant tuning and domain knowledge to achieve good performance.
Revised: While scalability can be improved with interpolation methods (Hensman et al., 2013; Titsias, 2009; Wilson & Nickisch, 2015) and kernel learning methods (Wilson et al., 2016), GPs tend to perform worse than neural network-based methods on high-dimensional problems.

Original: Another way to formulate epistemic uncertainty estimation is as a Bayesian Inference problem.
Revised: Another way to formulate epistemic uncertainty estimation is as a Bayesian Inference problem.

Original: Speciﬁcally, given a prior distribution over functions and some labeled data, the goal is to construct a posterior distribution in the Bayesian sense.
Revised: Speciﬁcally, given a prior distribution over functions and some labeled data, the goal is to construct a posterior distribution in the Bayesian sense.

Original: Because this posterior distribution is usually intractable to compute exactly, approximate sampling methods are used instead.
Revised: Because this posterior distribution is usually intractable to compute exactly, approximate sampling methods are used instead.

Original: Markov-Chain
Revised: Markov-Chain

Original: Monte Carlo (MCMC) (Tanner & Wong, 1987) and Variational Inference (VI) (Blei et al., 2017) are some of the most common methods for posterior sampling.
Revised: Monte Carlo (MCMC) (Tanner & Wong, 1987) and Variational Inference (VI) (Blei et al., 2017) are some of the most common methods for posterior sampling.

Original: These methods, however, tend to struggle in high dimensional problems.
Revised: These methods, however, tend to struggle in high dimensional problems.

Original: The computational complexity of MCMC scales poorly with the dimension of the data and VI methods tend to require restrictive function families with strong independence assumptions.
Revised: The computational complexity of MCMC scales poorly with the dimension of the data and VI methods tend to require restrictive function families with strong independence assumptions.

Original: Bayesian Neural Networks (BNNs) (Blundell et al., 2015) are another way to perform Bayesian Inference a neural networks by modeling each parameter in the network with an independent Gaussian distribution.
Revised: Bayesian Neural Networks (BNNs) (Blundell et al., 2015) are another way to perform Bayesian Inference a neural networks by modeling each parameter in the network with an independent Gaussian distribution.

Original: While BNNs perform well in low dimensional settings, their performance tends to degrade in high-dimensional settings that require large networks.
Revised: While BNNs perform well in low dimensional settings, their performance tends to degrade in high-dimensional settings that require large networks.

Original: Gal & Ghahramani (2016) proved that dropout can also be used as a form of Bayesian Inference.
Revised: Gal & Ghahramani (2016) proved that dropout can also be used as a form of Bayesian Inference.

Original: Speciﬁcally, if a network is trained using dropout and L2 regularization, posterior samples can be collected by applying dropout at test-time.
Revised: Speciﬁcally, if a network is trained using dropout and L2 regularization, posterior samples can be collected by applying dropout at test-time.

Original: This method has surprisingly good out-of-distribution detection and can be applied to any network.
Revised: This method has surprisingly good out-of-distribution detection and can be applied to any network.

Original: However, it can be hard to tune the dropout parameter to have good uncertainty prediction without signiﬁcantly degrading performance.
Revised: However, it can be hard to tune the dropout parameter to have good uncertainty prediction without signiﬁcantly degrading performance.

Original: Neural network ensembling is another commonly used epistemic uncertainty estimation technique, which is closely related to our proposed method in this work.
Revised: Neural network ensembling is another commonly used epistemic uncertainty estimation technique, which is closely related to our proposed method in this work.

Original: This technique involves training a number of completely separate neural networks on the same data to form an ensemble.
Revised: This technique involves training a number of completely separate neural networks on the same data to form an ensemble.

Original: Because of the stochasticity of initialization and the training process, each member of the ensemble should be a slightly different function.
Revised: Because of the stochasticity of initialization and the training process, each member of the ensemble should be a slightly different function.

Original: Speciﬁcally, the ensemble members should produce similar outputs in data-dense regions of the state-space and dissimilar outputs in data-sparse regions.
Revised: Speciﬁcally, the ensemble members should produce similar outputs in data-dense regions of the state-space and dissimilar outputs in data-sparse regions.

Original: Pearce et al.
Revised: Pearce et al.

Original: (2020) proved that, by using the correct regularization, each ensemble member becomes an approximate sample from the function posterior.
Revised: (2020) proved that, by using the correct regularization, each ensemble member becomes an approximate sample from the function posterior.

Original: Ensembling methods are particularly attractive to practitioners because they require very little ﬁne-tuning and work surprisingly well in practice.
Revised: Ensembling methods are particularly attractive to practitioners because they require very little ﬁne-tuning and work surprisingly well in practice.

Original: However, these methods have one main drawback, namely that each new sample from the posterior requires training a new network from scratch.
Revised: However, these methods have one main drawback, namely that each new sample from the posterior requires training a new network from scratch.

Original: Our method, on the other hand, seeks to construct a generative posterior model using similar regularization techniques to Pearce et al.
Revised: Our method, on the other hand, seeks to construct a generative posterior model using similar regularization techniques to Pearce et al.

Original: (2020), allowing for quick sampling from the posterior.
Revised: (2020), allowing for quick sampling from the posterior.

Original:  P ROBLEM S TATEMENT AND B ACKGROUND
Revised:  P ROBLEM S TATEMENT AND B ACKGROUND

Original: We will focus on the problem of Bayesian Inference on the parameters of θ of some function f ( · ; θ ) .
Revised: We will focus on the problem of Bayesian Inference on the parameters of θ of some function f ( · ; θ ) .

Original: We assume we have some known prior over parameters P prior ( θ ) ∼ N ( µ prior , Σ prior ) as well as access to a noisy dataset ( x obs , y obs ) = { ( x 1 obs , y 1 obs ) . . .
Revised: We assume we have some known prior over parameters P prior ( θ ) ∼ N ( µ prior , Σ prior ) as well as access to a noisy dataset ( x obs , y obs ) = { ( x 1 obs , y 1 obs ) . . .

Original: , ( x N obs , y N obs ) } , where y i obs = f
Revised: , ( x N obs , y N obs ) } , where y i obs = f

Original: ( x i obs ; θ ) + (cid:15) , (cid:15) ∼ N (0 , σ (cid:15) ) for some unknown parameter θ .
Revised: ( x i obs ; θ ) + (cid:15) , (cid:15) ∼ N (0 , σ (cid:15) ) for some unknown parameter θ .

Original: We can then deﬁne the data-likelihood as
Revised: We can then deﬁne the data-likelihood as

Original: P like

Original: [Equation]
Revised: [Equation]

Original: The goal is to approximate samples from the posterior,
Revised: The goal is to approximate samples from the posterior,

Original: [Equation]
Revised: [Equation]

Revised: For ease of notation, we will drop the x obs from the likelihood.

Original: For our analysis, it is convenient if the posterior distribution is Gaussian.
Revised: For our analysis, it is convenient if the posterior distribution is Gaussian.

Original: To show this, we will use the fact that the product of two Gaussian PDFs is proportional to a Gaussian PDF.
Revised: To show this, we will use the fact that the product of two Gaussian PDFs is proportional to a Gaussian PDF.

Original: We make a distinction here between the data-likelihood, a function of the observed data y obs , and the parameterlikelihood, a function of the parameters θ , deﬁned as follows:
Revised: We make a distinction here between the data-likelihood, a function of the observed data y obs , and the parameterlikelihood, a function of the parameters θ , deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: The parameter-likelihood PDF is equivalent to the data-likelihood PDF for all values of y obs and θ , but is notably a function of the parameters θ .
Revised: The parameter-likelihood PDF is equivalent to the data-likelihood PDF for all values of y obs and θ , but is notably a function of the parameters θ .

Original: For a more detailed explanation of the distinction between the parameter and data-likelihoods, see Deﬁnition 1 from Pearce et al.
Revised: For a more detailed explanation of the distinction between the parameter and data-likelihoods, see Deﬁnition 1 from Pearce et al.

Original: When the function f is linear, it can be shown that the parameter-likelihood is also Gaussian,
Revised: When the function f is linear, it can be shown that the parameter-likelihood is also Gaussian,

Original: [Equation]
Revised: [Equation]

Original: While this is not true in general, Pearce et al.
Revised: While this is not true in general, Pearce et al.

Original: (2020) assume that the parameter likelihood is Gaussian so that the posterior also becomes Gaussian,
Revised: (2020) assume that the parameter likelihood is Gaussian so that the posterior also becomes Gaussian,

Original: [Equation]
Revised: [Equation]

Original: where the posterior covariance and mean are given by
Revised: where the posterior covariance and mean are given by

Original: [Equation]
Revised: [Equation]

Original: Note, however, that after the reformulation of the problem in Section 4.1, we no longer require this assumption for our theoretical analysis.
Revised: Note, however, that after the reformulation of the problem in Section 4.1, we no longer require this assumption for our theoretical analysis.

Original: 3. R ANDOMIZED MAP SAMPLING (RMS)
Revised: 3. R ANDOMIZED MAP SAMPLING (RMS)

Original: Since the values µ like and Σ like are intractible to compute in practice, the challenge of computing the posterior distribution remains.
Revised: Since the values µ like and Σ like are intractible to compute in practice, the challenge of computing the posterior distribution remains.

Original: However, Pearce et al.
Revised: However, Pearce et al.

Original: (2020) showed how we can use a method called Randomized MAP sampling (RMS) to approximately sample from the posterior distribution.
Revised: (2020) showed how we can use a method called Randomized MAP sampling (RMS) to approximately sample from the posterior distribution.

Original: With RMS we ﬁrst assume access to a mechanism to compute the maximum a posteriori (MAP) solution for our posterior.
Revised: With RMS we ﬁrst assume access to a mechanism to compute the maximum a posteriori (MAP) solution for our posterior.

Original: Then, by randomly shifting the prior distribution we use in our MAP solver, we can estimate samples from the posterior.
Revised: Then, by randomly shifting the prior distribution we use in our MAP solver, we can estimate samples from the posterior.

Original: Speciﬁcally, RMS samples “anchor points” θ anc from some distribution θ anc ∼ N ( µ anc , Σ anc ) .
Revised: Speciﬁcally, RMS samples “anchor points” θ anc from some distribution θ anc ∼ N ( µ anc , Σ anc ) .

Original: Each anchor point deﬁnes a shifted prior distribution P anc ( θ anc ) =
Revised: Each anchor point deﬁnes a shifted prior distribution P anc ( θ anc ) =

Original: N ( θ anc , Σ prior ) .
Revised: N ( θ anc , Σ prior ) .

Original: Now we deﬁne the
Revised: Now we deﬁne the

Original: MAP function which ﬁnds the MAP solution using this shifted prior:
Revised: MAP function which ﬁnds the MAP solution using this shifted prior:

Original: [Equation]
Revised: [Equation]

Original: By sampling different anchor points, we can then construct a probability distribution over MAP solutions P ( θ MAP ( θ anc )) , which can be shown to be Gaussian.
Revised: By sampling different anchor points, we can then construct a probability distribution over MAP solutions P ( θ MAP ( θ anc )) , which can be shown to be Gaussian.

Original: Using moment matching, Pearce et al.
Revised: Using moment matching, Pearce et al.

Original: (2020) proved, in Theorem 1, that if we choose µ anc = µ prior and Σ anc =
Revised: (2020) proved, in Theorem 1, that if we choose µ anc = µ prior and Σ anc =

Original: Σ prior + Σ prior Σ − 1 like Σ prior , then P ( θ MAP ( θ anc ))
Revised: Σ prior + Σ prior Σ − 1 like Σ prior , then P ( θ MAP ( θ anc ))

Original: =
Revised: =

Original: P post ( θ | y obs ) .
Revised: P post ( θ | y obs ) .

Original: Using the approximation Σ anc ≈ Σ prior , a sample from posterior, θ post , can be collected by sampling θ anc ∼ N ( µ anc , Σ anc ) and optimizing the following:
Revised: Using the approximation Σ anc ≈ Σ prior , a sample from posterior, θ post , can be collected by sampling θ anc ∼ N ( µ anc , Σ anc ) and optimizing the following:

Original: [Equation]
Revised: [Equation]

Original:  G ENERATIVE P OSTERIOR N ETWORKS
Revised:  G ENERATIVE P OSTERIOR N ETWORKS

Original: While RMS can be used to construct samples from the posterior, every new sample requires solving a new optimization problem to ﬁnd the MAP solution.
Revised: While RMS can be used to construct samples from the posterior, every new sample requires solving a new optimization problem.

Original: Instead, we propose learning the MAP function itself.
Revised: Instead, we propose learning the MAP function itself.

Original: Let g be a neural network parameterized by some vector, φ , that takes as input a sample from the anchor distribution, θ anc ∼ P anc ( θ anc ) , and outputs θ MAP ( θ anc ) .
Revised: Let g be a neural network parameterized by some vector, φ , that takes as input a sample from the anchor distribution, θ anc ∼ P anc ( θ anc ) , and outputs θ MAP ( θ anc ) .

Original: If we assume that g has enough expressive power to represent the MAP function, then g ( θ anc ; φ ) =
Revised: If we assume that g has enough expressive power to represent the MAP function, then g ( θ anc ; φ ) =

Original: θ MAP ( θ anc ) if:
Revised: θ MAP ( θ anc ) if:

Original: [Equation]
Revised: [Equation]

Original: In practice, however, this is a very hard optimization problem for two main reasons.
Revised: In practice, however, this is a very hard optimization problem for two main reasons.

Original: (1) Optimizing in parameter space tends to be challenging for gradient-descent optimization.
Revised: (1) Optimizing in parameter space tends to be challenging for gradient-descent optimization.

Original: The space of all parameters θ anc is not only very large, but there are also many parameters θ anc that map to the same outputs for all inputs.
Revised: The space of all parameters θ anc is not only very large, but there are also many parameters θ anc that map to the same outputs for all inputs.

Original: Moreover, we would expect the posterior parameters to be highly correlated and, therefore, need less expressive power to be represented.
Revised: Moreover, we would expect the posterior parameters to be highly correlated and, therefore, need less expressive power to be represented.

Original: To address these two problems, we will ﬁrst change the problem deﬁnition slightly to ﬁnd the posterior in output space as opposed to parameter space.
Revised: To address these two problems, we will ﬁrst change the problem deﬁnition slightly to ﬁnd the posterior in output space as opposed to parameter space.

Original: We prove in the next section that, by changing our loss function, we can still use RMS to approximate the posterior in output space.
Revised: We prove in the next section that, by changing our loss function, we can still use RMS to approximate the posterior in output space.

Original: Next, we show practically how we construct a low-dimensional representation of the anchor distribution.
Revised: Next, we show practically how we construct a low-dimensional representation of the anchor distribution.

Original: 4. T HEORETICAL R ESULTS
Revised: 4. T HEORETICAL R ESULTS

Original: As mentioned above, optimizing a generative model to output parameters can be challenging.
Revised: As mentioned above, optimizing a generative model to output parameters can be challenging.

Original: Instead, we reformulate the Bayesian Inference problem slightly and consider the posterior in output space.
Revised: Instead, we reformulate the Bayesian Inference problem slightly and consider the posterior in output space.

Original: Concretely, let ˆY = { ˆ Y 1 , . . , ˆ Y M } be a transformation of the random variable θ , where each element is the output of the function f parameterized by θ evaluated at points x sample = { x 1 sample , . . 
Revised: Concretely, consider some discretization of the input space, x sample = { x 1 sample , . . .

Original: , x M sample } , ˆ Y i =
Revised: , x M sample } . Let ˆ Y i =

Original: f ( x i sample ; θ )  .
Revised: f ( x i sample ; θ ) be a set of sample points and ˆY = { ˆ Y 1 , . . , ˆ Y M } be a transformation of the random variable θ , where each element is the output of the function f parameterized by θ evaluated at points x sample .

Original: The prior and likelihood for this transformed random variable ˆY can be expressed in terms of the prior and likelihood of θ :
Revised: The prior and likelihood for this transformed random variable ˆY can be expressed in terms of the prior and likelihood of θ :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that samples from the prior of ˆY can be easily obtained by sampling θ ∼ P like ( θ ) and computing ˆ Y i =
Revised: Note that samples from the prior of ˆY can be easily obtained by sampling θ ∼ P like ( θ ) and computing ˆ Y i =

Original: f ( x i sample ; θ ) for all i .
Revised: f ( x i sample ; θ ) for all i .

Original: The goal in this reformulated problem is to ﬁnd the posterior
Revised: The goal in this reformulated problem is to ﬁnd the posterior

Original: P ( ˆY | y obs ) ∝ P [Equation]
Revised: P ( ˆY | y obs ) ∝ P ( y obs | ˆY ) P ( ˆY ) .

Original: As before, we estimate the posterior using RMS and deﬁne a new MAP function in the output space:
Revised: As before, we estimate the posterior using RMS and deﬁne a new MAP function in the output space:

Original: [Equation]
Revised: [Equation]

Original: for some anchor distribution ˆY anc ∼ P anc ( ˆY anc ) =
Revised: for some anchor distribution ˆY anc ∼ P anc ( ˆY anc ) =

Original: N ( µ anc , Σ anc ) .
Revised: N ( µ anc , Σ anc ) .

Original: We now need to show that, with the correct choice of µ anc and Σ anc , P ( ˆ Y MAP ( ˆY anc ))
Revised: We now need to show that, with the correct choice of µ anc and Σ anc , P ( ˆ Y MAP ( ˆY anc ))

Original: =
Revised: =

Original: P ( ˆY | y obs ) .
Revised: P ( ˆY | y obs ) .

Original: To simplify the analysis, we assume that the output prior is normally distributed P ( ˆY ) =
Revised: To simplify the analysis, we assume that the output prior is normally distributed P ( ˆY ) =

Original: N ( µ ˆY , Σ ˆY ) .
Revised: N ( µ ˆY , Σ ˆY ) .

Original: Since the parameters are normally distributed, this approximation becomes exact when either f is linear or the ﬁnal layer is inﬁnitely wide.
Revised: Since the parameters are normally distributed, this approximation becomes exact when either f is linear or the ﬁnal layer is inﬁnitely wide.

Original: Next, we need to show that the likelihood function P ( y obs | ˆY ) is roughly normally distributed with respect to the vector ˆY .
Revised: Next, we need to show that the likelihood function P ( y obs | ˆ Y ) is roughly normally distributed with respect to the vector ˆY .

Original: Recall that each observed data point, y i obs , is equal to a noisy output of f for some parameter θ , y i obs = f
Revised: Recall that each observed data point, y i obs , is equal to a noisy output of f for some parameter θ , y i obs = f

Original: ( x i obs ; θ ) + (cid:15) .
Revised: ( x i obs ; θ ) + (cid:15) .

Original: In the case where x obs ⊂ x sample , y i obs is evaluated on the same points as ˆY , so the likelihood of P ( y obs | θ ) becomes equivalent to P ( y obs | ˆY ) .
Revised: In the case where x obs ⊂ x sample , y i obs is evaluated on the same points as ˆY , so the likelihood of P ( y obs | θ ) becomes equivalent to P ( y obs | ˆY ) .

Original: By the same reasoning, if we choose x sample to contain all the points in the domain X , P ( y obs | θ ) becomes equivalent to P ( y obs | ˆY ) .
Revised: By the same reasoning, if we choose x sample to contain all the points in the domain X , P ( y obs | θ ) becomes equivalent to P ( y obs | ˆY ) .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let f be some neural network parameterized by θ .
Revised: Let f be some neural network parameterized by θ .

Original: Let x obs , y obs correspond to vectors of observed labeled data.
Revised: Let x obs , y obs correspond to vectors of observed labeled data.

Original: Let x sample be some discretization over the input space of M d points such that for any x ∈ X there exists an x (cid:48) ∈ x sample where (cid:107) x − x (cid:48) (cid:107) 2 ≤ lM for some constant l .
Revised: Let x sample be some discretization over the input space of M d points such that for any x ∈ X there exists an x (cid:48) ∈ x sample where (cid:107) x − x (cid:48) (cid:107) 2 ≤ lM for some constant l .

Original: Deﬁne ˆY :=
Revised: Deﬁne ˆY :=

Original: { f ( x ; θ ) : ∀ x ∈ x sample } be a vector of evaluations of f on all points in x sample for a given parameter vector θ .
Revised: { f ( x ; θ ) : ∀ x ∈ x sample } be a vector of evaluations of f on all points in x sample for a given parameter vector θ .

Original: Then, if we take the limit as M → ∞ , P ( y obs | ˆY ) =
Revised: Then, if we take the limit as M → ∞ , P ( y obs | ˆY ) =

Original: P ( y obs | θ ) .
Revised: P ( y obs | θ ) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.1.
Revised: See Appendix A.1.

Original: It turns out that, since the observation noise is additive, the likelihood in parameter space P ( y obs | θ ) is also Gaussian as a function of the output vector ˆY .
Revised: It turns out that, since the observation noise is additive, the likelihood in parameter space P ( y obs | θ ) is also Gaussian as a function of the output vector ˆY .

Original: Thus, if we take | x sample | → ∞ and apply
Revised: Thus, if we take | x sample | → ∞ and apply

Original: Lemma 1, we get that the output likelihood P ( y obs | ˆY ) is Gaussian as a function of ˆY .
Revised: Lemma 1, we get that the output likelihood P ( y obs | ˆY ) is Gaussian as a function of ˆY .

Original: Now, since we have that both the prior and likelihood of ˆY can be expressed as Gaussian functions of ˆY , we also have that the posterior is Gaussian.
Revised: Now, since we have that both the prior and likelihood of ˆY can be expressed as Gaussian functions of ˆY , we also have that the posterior is Gaussian.

Original: As before, we just need to show that the distribution of MAP solutions P ( ˆ Y MAP ( ˆY anc )) is also Gaussian, then apply moment matching to ﬁnd the right values of µ anc and Σ anc such that P ( ˆ Y MAP ( ˆY anc )))
Revised: As before, we just need to show that the distribution of MAP solutions P ( ˆ Y MAP ( ˆY anc )) is also Gaussian, then apply moment matching to ﬁnd the right values of µ anc and Σ anc such that P ( ˆ Y MAP ( ˆ Y anc )))

Original: =
Revised: =

Original: P ( ˆY | y obs ) .
Revised: P ( ˆ Y | y obs ) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f be some neural network parameterized by θ .
Revised: Let f be some neural network parameterized by θ .

Original: Let x sample be some vector of inputs sampled uniformly from X M .
Revised: Let x sample be some vector of inputs sampled uniformly from X M .

Original: Deﬁne ˆY :=
Revised: Deﬁne ˆY :=

Original: [ f ( x 1 ; θ ) , . .
Revised: [ f ( x 1 ; θ ) , . .

Original: f ( x M ; θ )] as a transformation of the random variable θ .
Revised: f ( x M ; θ )] as a transformation of the random variable θ .

Original: Assume the prior of ˆY follows a multivariate normal distribution, P ( ˆY ) =
Revised: Assume the prior of ˆY follows a multivariate normal distribution, P ( ˆY ) =

Original: N ( µ ˆY , Σ ˆY ) .
Revised: N ( µ ˆY , Σ ˆY ) .

Original: We assume access to some function which takes as input the center of a shifted distribution and outputs the MAP estimate of ˆY , ˆ Y MAP ( ˆY anc ) .
Revised: We assume access to some function which takes as input the center of a shifted distribution and outputs the MAP estimate of ˆY , ˆ Y MAP ( ˆY anc ) .

Original: If we choose the distribution over ˆY anc to be P ( ˆY anc ) =
Revised: If we choose the distribution over ˆY anc to be P ( ˆY anc ) =

Original: N ( µ anc , Σ anc ) , where µ anc = µ ˆY and Σ anc =
Revised: N ( µ anc , Σ anc ) , where µ anc = µ ˆY and Σ anc =

Original: Σ ˆY + Σ ˆY Σ − 1 like Σ ˆY , then lim M →∞ P ( ˆ Y MAP ( ˆY anc ))
Revised: Σ ˆY + Σ ˆY Σ − 1 like Σ ˆY , then lim M →∞ P ( ˆ Y MAP ( ˆY anc ))

Original: =
Revised: =

Original: P ( ˆY | y obs ) .
Revised: P ( ˆY | y obs ) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.2.
Revised: See Appendix A.2.

Original: 4. P RACTICAL I MPLEMENTATION
Revised: 4. P RACTICAL I MPLEMENTATION

Original: In practice, as was done in Pearce et al.
Revised: In practice, as was done in Pearce et al.

Original: (2020), we will instead use the approximation Σ anc ≈ Σ ˆY to make it tractable to sample from the anchor distribution.
Revised: (2020), we will instead use the approximation Σ anc ≈ Σ ˆY to make it tractable to sample from the anchor distribution.

Original: Pearce et al.
Revised: Pearce et al.

Original: (2020) argue that this approximation, in general, causes RMS to over-estimate the posterior variance.
Revised: (2020) argue that this approximation, in general, causes RMS to over-estimate the posterior variance.

Original: With this assumption, the anchor distribution P anc ( ˆY anc ) becomes approximately equal to the prior distribution P ( ˆY ) .
Revised: With this assumption, the anchor distribution P anc ( ˆY anc ) becomes approximately equal to the prior distribution P ( ˆY ) .

Original: Recall that, to sample from the output prior, we simply sample a set of parameters θ ∼ P prior ( θ ) and evaluate the function f at all evaluation points x sample .
Revised: Recall that, to sample from the output prior, we simply sample a set of parameters θ ∼ P prior ( θ ) and evaluate the function f at all evaluation points x sample .

Original: Now, if we want to learn the MAP function in the output space, we can construct a neural network g , parameterized by some vector φ , that takes a sample from the output anchor distribution, ˆY anc ∼ N ( µ anc , Σ anc ) , along with the evaluation points x sample , and outputs θ MAP ( θ anc ) .
Revised: Now, if we want to learn the MAP function in the output space, we can construct a neural network g , parameterized by some vector φ , that takes a sample from the output anchor distribution, ˆY anc ∼ N ( µ anc , Σ anc ) , along with the evaluation points x sample , and outputs θ MAP ( θ anc ) .

Original: If we assume that g has enough expressive power to represent the MAP function, then g ( ˆY anc ; φ )
Revised: If we assume that g has enough expressive power to represent the MAP function, then g ( ˆY anc ; φ )

Original: =
Revised: =

Original: ˆ Y MAP ( ˆY anc ) , if:
Revised: ˆ Y MAP ( ˆY anc ) , if:

Original: [Equation]
Revised: [Equation]

Original: This can be re-written as:
Revised: This can be re-written as:

Original: [Equation]
Revised: [Equation]

Original:  where δ j = g ( x j sample , θ anc ; φ ) − f ( x j sample ; θ anc ) .
Revised: 14) where δ j = g ( x j sample , θ anc ; φ ) − f ( x j sample ; θ anc ) .

Original: See Appendix A.3 for a step-by-step derivation.
Revised: See Appendix A.3 for a step-by-step derivation.

Original: In practice, of course, we can only sample a subset of x sample .
Revised: In practice, of course, we cannot compute the regularization term, [Equation] N [Equation]

Revised: Instead, we approximate this term using a ﬁnite number of sample points from the input space X .

Original: Because of this, for high-dimensional problems, the off-diagonal terms of Σ ˆY are near zero almost always.
Revised: Because of this, for high-dimensional problems, the off-diagonal terms of Σ ˆY are near zero almost 1.5 MCMC - Ground Truth GP Bayesian Dropout

Revised: Ensemble

Revised: GPN (Ours) always.

Original: However, we found that, in practice, assuming that the samples of ˆY are independent did not signiﬁcantly degrade performance.
Revised: However, we found that, in practice, assuming that the samples of ˆY are independent did not signiﬁcantly degrade performance.

Revised: Using this approximation, our optimization problem becomes:

Original:  In order for g to have enough expressive power to represent the full MAP function, it may require the size of the parameters φ to be much larger than θ .
Revised: [Equation] where δ = g ( x sample , θ anc ; φ ) − f ( x sample ; θ anc ) and β is a hyper-parameter. In order for g to have enough expressive power to represent the full MAP function, it may require the size of the parameters φ to be much larger than θ .

Original: However, after applying the low-dimensional approximation of θ anc described below, in our experiments, we only used roughly double the number of parameters compared to each member of the ensemble method.
Revised: However, after applying the low-dimensional approximation of θ anc described below, in our experiments, we only used roughly double the number of parameters compared to each member of the ensemble method.

Original: And since the ensemble we used has 10 members, the total number of parameters needed for our method is signiﬁcantly smaller.
Revised: And since the ensemble we used has 10 members, the total number of parameters needed for our method is signiﬁcantly smaller.

Original: 4. L OW -D IMENSIONAL E MBEDDING
Revised: 4. L OW -D IMENSIONAL E MBEDDING

Original: As mentioned above, optimizing the generative function g over the the space of all anchor parameters θ anc did not yield great results because the space was too large.
Revised: As mentioned above, optimizing the generative function g over the the space of all anchor parameters θ anc did not yield great results because the space was too large.

Original: Instead, we sought to learn a lowdimensional representation of the anchor parameters.
Revised: Instead, we sought to learn a lowdimensional representation of the anchor parameters.

Original: There are two key reasons why we would expect to be able to decrease the representational power of the anchor distribution and maintain the same of ﬁdelity in the posterior estimation.
Revised: There are two key reasons why we would expect to be able to decrease the representational power of the anchor distribution and maintain the same of ﬁdelity in the posterior estimation.

Original: (1) Because of the nature of neural networks, there are many settings of parameters θ that would result in the same output vector ˆY .
Revised: (1) Because of the nature of neural networks, there are many settings of parameters θ that would result in the same output vector ˆY .

Original: Because we are[Figure] focusing on the posterior of ˆY , we need less representational power to model ˆY .
Revised: Because we are focusing on the posterior of ˆY , we need less representational power to model ˆY .

Original: And, maybe more importantly, because the posterior parameters are highly correlated, we would expect to need less expressive power to represent the posterior distribution than the prior distribution.
Revised: And, maybe more importantly, because the posterior parameters are highly correlated, we would expect to need less expressive power to represent the posterior distribution than the prior distribution.

Original: Thus, we would like to construct a simpler estimate of the prior space using a low-dimensional embedding vector, z .
Revised: Thus, we would like to construct a simpler estimate of the prior space using a low-dimensional embedding vector, z .

Original: That is, we want our generative model to take as input the embedding vector z ∼ N (0 , I ) , instead of θ anc , in order to estimate the MAP function.
Revised: That is, we want our generative model to take as input the embedding vector z ∼ N (0 , I ) , instead of θ anc , in order to estimate the MAP function.

Original: To do this, we need to learn a mapping from anchor parameters θ anc to embedding vectors z .
Revised: To do this, we need to learn a mapping from anchor parameters θ anc to embedding vectors z .

Original: For our experiments, we used a 1-1 embedding scheme where we sample k parameters from the true prior θ 1 , . . .
Revised: For our experiments, we used a 1-1 embedding scheme where we sample k parameters from the true prior θ 1 , . . .

Original: , θ k ∼ P prior ( θ ) and k independent samples from our embedding z 1 , . . .
Revised: , θ k ∼ P prior ( θ ) and k independent samples from our embedding z 1 , . . .

Original: , z k .
Revised: , z k .

Original: We then jointly optimize over φ and z 1 , . .
Revised: We then jointly optimize over φ and z 1 , . .

Original: z k as follows:
Revised: z k as follows:

Original: [Equation]
Revised: [Equation]

Original: where δ j = g ( x j [Equation]
Revised: where δ j = g ( x j sample , z j + (cid:15) ; φ ) − f ( x j sample ; θ j ) , (cid:15) ∼ N (0 , I ) is a noise injection vector that improves the smoothness of interpolations between embedding vectors, and L reg ( z 1 , . . .

Original: , z j ; φ ) − f ( x j
Revised: , z k ) is a regularizer to keep z 1 , . . .

Original: [Equation]
Revised: , z k roughly normally distributed, which allows us to easily sample from the embedding space.. For our experiments we use the KL divergence between z and the normal distribution, L reg ( z 1 , . . . , z k ) =

Original: ; θ j ) , (cid:15) ∼ N (0 , I ) is a noise injection vector that improves the smoothness of interpolations between embedding vectors, and L reg (  z1 , . . . ,  zk )is a regularizer to keep z 1 , . . . , z k roughly normally distributed.
Revised: D KL ( N (¯ z , s z ) , N (0 , 1)) . Figure 1 illustrates how we can sample from this embedding space to construct posterior functions.

Original:  4. C LASSIFICATION
Revised: [Figure] [Figure] 4. C LASSIFICATION

Original: Our method uses the assumption that the prior over ˆY is approximately normally distributed.
Revised: Our method uses the assumption that the prior over ˆY is approximately normally distributed.

Original: While this approximation is exact, assuming the prior parameters are normally distributed, only when either the network is linear or the ﬁnal layer is inﬁnitely wide and does not have an activation function, for most deep networks, this is a good approximation.
Revised: While this approximation is exact, assuming the prior parameters are normally distributed, only when either the network is linear or the ﬁnal layer is inﬁnitely wide and does not have an activation function, for most deep networks, this is a good approximation.

Original: However, if we add a soft-max to the ﬁnal output, as is usually done for classiﬁcation tasks, this assumption is violated.
Revised: However, if we add a soft-max to the ﬁnal output, as is usually done for classiﬁcation tasks, this assumption is violated.

Original: To get around this, for classiﬁcation tasks, we add the anchor loss to the pre-softmax outputs.
Revised: To get around this, for classiﬁcation tasks, we add the anchor loss to the pre-softmax outputs [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The goal of our experiments is to illustrate the ability of our method to accurately model epistemic uncertainty for out of distribution data while retaining high performance on in distribution data.
Revised: The goal of our experiments is to illustrate the ability of our method to accurately model epistemic uncertainty for out of distribution data while retaining high performance on in distribution data.

Original: We will test our method on two problems: (1)  Bayesian Optimization (BO) and (2) Classiﬁcation with out of distribution examples.
Revised: We will test our method on three problems: (1) a small-scale regression task, (2) Bayesian Optimization (BO), and (3) Classiﬁcation with out of distribution examples.

Original: We compare our method against 3 competing Bayesian methods: Gaussian Processes (GPs),
Revised: We compare our method against 3 competing Bayesian methods: Gaussian Processes (GPs),

Original: Bayesian Dropout (Gal & Ghahramani, 2016), and anchor-regularized neural-network ensembles (Pearce et al., 2020) with 10 ensemble members.
Revised: Bayesian Dropout (Gal & Ghahramani, 2016), and anchor-regularized neural-network ensembles (Pearce et al., 2020) with 10 ensemble members.

Original: For our BO experiments we use exact GP with a static kernel since the problems are sufﬁciently low-dimensional, but for our classiﬁcation experiments we need to use approximate GPs.
Revised: For our BO experiments we use exact GP with a static kernel since the problems are sufﬁciently low-dimensional, but for our classiﬁcation experiments we need to use approximate GPs.

Original: Speciﬁcally, we use a grid-interpolated GP with Deep
Revised: Speciﬁcally, we use a grid-interpolated GP with Deep

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Kernel Learning (Wilson et al., 2016) implemented with the GPyTorch library (Gardner et al., 2018).

Original: T r u e P o s i t i v e R a t e [Figure]
Revised: For the anchor-regularized ensembles, since every member requires a unique set of parameters (and anchor points) the number of ensemble members was limited by the memory capacity of our GPUs.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 5. S MALL S CALE R EGRESSION

Original: [Figure] [Figure] u e P
Revised: Our ﬁrst experiment is a small 1-dimensional regression problem where we can easily compute the ground truth.

Original: [Figure] s i t i [Figure]
Revised: We provide each method with the same 6 observations.

Original: Kernel Learning (Wilson et al., 2016) implemented with the GPyTorch library (Gardner et al., 2018). For  the anchor-regularized ensembles, since every member requires a unique set of parameters (and anchor points) the number of ensemble members was limited by the memory capacity of our GPUs.
Revised: For this experiment, we compute 100 samples from the approximate posterior for each method (except for the Ensemble where we only use 10 samples). For ground truth, we use the Metropolis-Hastings MCMC algorithm.

Original:  5. B AYESIAN O PTIMIZATION
Revised: Figure 2 shows the results. 5. B AYESIAN O PTIMIZATION

Original: The goal of Bayesian Optimization is to ﬁnd the maximum of a function in as few samples as possible.
Revised: The goal of Bayesian Optimization is to ﬁnd the maximum of a function in as few samples as possible.

Original: This is commonly done by ﬁrst constructing a posterior over possible functions given the samples we have collected so far, then, through various methods, choosing sample points that might maximize the true function.
Revised: This is commonly done by ﬁrst constructing a posterior over possible functions given the samples we have collected so far, then, through various methods, choosing sample points that might maximize the true function.

Original: By using different methods for predicting the posterior in BO, we can evaluate how well these methods capture epistemic uncertainty. 
Revised: By using different methods for predicting the posterior in BO, we can evaluate how well these methods capture epistemic uncertainty. [Figure]

Original:  For all methods we use Thompson Sampling and implemented the experiments using the BOTorch library by Balandat et al.
Revised: [Table] For all methods we use Thompson Sampling and implemented the experiments using the BOTorch library by Balandat et al.

Original: We tested each method on ﬁve standard benchmark BO functions: the Branin, Hartmann-6D, Shekel, Michalewicz-10D, and Powell functions.
Revised: We tested each method on ﬁve standard benchmark BO functions: the Branin, Hartmann-6D, Shekel, Michalewicz-10D, and Powell functions.

Original: We normalize the output of each function between a minimum value (found by random sampling) and the optimal value.
Revised: We normalize the output of each function between a minimum value (found by random sampling) and the optimal value.

Original: To make sure that each method had a large enough sample variance, we chose hyper-parameters such that 100 prior samples from each method have a range of roughly [ − 1 , 1] .
Revised: To make sure that each method had a large enough sample variance, we chose hyper-parameters such that 100 prior samples from each method have a range of roughly [ − 1 , 1] .

Original: To choose the best regularization hyper-parameter, we ran each method with 5 different regularization parameters, then chose the hyper-parameters with the highest average performance across all BO problems.
Revised: To choose the best regularization hyper-parameter, we ran each method with 5 different regularization parameters, then chose the hyper-parameters with the highest average performance across all BO problems.

Original: Figure 1 shows the best normalized function value sampled so far during optimization averaged across the 5 BO problems each ran for 5 trials.
Revised: Figure 3 shows the best normalized function value sampled so far during optimization averaged across the 5 BO problems each ran for 5 trials.

Original: Our method signiﬁcantly out-performs Random sampling and even slightly outperforms GPs, one of the best BO techniques for low-dimensional problems.
Revised: Our method signiﬁcantly out-performs Random sampling and even slightly outperforms GPs, one of the best BO techniques for low-dimensional problems.

Original: Even though our method trails slightly behind Bayesian Dropout, these results still illustrate the uncertainty estimation power of our method, even on low-dimensional problems.
Revised: Even though our method trails slightly behind Bayesian Dropout, these results still illustrate the uncertainty estimation power of our method, even on low-dimensional problems.

Original: 5. C LASSIFICATION
Revised: 5. C LASSIFICATION

Original: We also look at the problem of image classiﬁcation.
Revised: We also look at the problem of image classiﬁcation.

Original: Along with our method having high accuracy, we are also interested in how well our method (and the baseline methods) is able to capture epistemic uncertainty.
Revised: Along with our method having high accuracy, we are also interested in how well our method (and the baseline methods) is able to capture epistemic uncertainty.

Original: To measure this we split the dataset into two sets based on class labels: the in distribution dataset and the out of distribution dataset.
Revised: To measure this we split the dataset into two sets based on class labels: the in distribution dataset and the out of distribution dataset.

Original: During training, we only provide the labels for the in-distribution dataset.
Revised: During training, we only provide the labels for the in-distribution dataset.

Original: During testing, along with recording the accuracy on the in distribution dataset, we also sample our models on the out of distribution dataset.
Revised: During testing, along with recording the accuracy on the in distribution dataset, we also sample our models on the out of distribution dataset.

Original: For such out of distribution examples, we expect the models to show high epistemic uncertainty and, thus, the samples for these out of distribution examples should have high variance.
Revised: For such out of distribution examples, we expect the models to show high epistemic uncertainty and, thus, the samples for these out of distribution examples should have high variance.

Original: Note that we still provide the full set of unlabeled training data to our agent.
Revised: Note that we still provide the full set of unlabeled training data to our agent.

Original: This obviously favors our method over the baseline methods as our method is the only method that is able to take advantage of this unlabeled data.
Revised: This obviously favors our method over the baseline methods as our method is the only method that is able to take advantage of this unlabeled data.

Original: However, from a practical perspective this is a reasonable setup as many realworld datasets have far more unlabeled data than labeled data.In Distribution Out Of Distribution 0. D ee p G P 0.
Revised: However, from a practical perspective this is a reasonable setup as many realworld datasets have far more unlabeled data than labeled data.

Original: B a y e s i a n

Original: D r o p o u t 0.

Original: E n s e m b l e 0 1 2 3 4 5 6 7 8 9 0.

Original: G P N ( O u r s )

Original: [Figure]

Original: In Distribution

Original: Out Of Distribution 0.

Original: D ee p G P 0.

Original: B a y e s i a n

Original: D r o p o u t 0.

Original: E n s e m b l e a i r p

Original: [Figure]

Original: a n

Original: [Figure]

Original: a

Original: [Figure]

Original: il e

Original: [Figure]

Original: i

Original: [Figure]

Original: a t

Original: [Figure]

Original: f

Original: [Figure]

Original: o

Original: [Figure]

Original: h i p t

Original: r u c k 0.

Original: [Figure]

Original: [Figure]

Original: [Figure] u r s ) [Figure] One desired property of an epistemic uncertainty estimator is in constructing conﬁdence sets.
Revised: One desired property of an epistemic uncertainty estimator is in constructing conﬁdence sets.

Original: For the classiﬁcation setting, the conﬁdence interval (CI) we consider is the set of possible class predictions.
Revised: For the classiﬁcation setting, the conﬁdence interval (CI) we consider is the set of possible class predictions.

Original: We would expect that this conﬁdence interval contains the true class with high probability.
Revised: We would expect that this conﬁdence interval contains the true class with high probability.

Original: We also expect that, on in-distribution examples, the conﬁdence interval only contains the true label with high probability.
Revised: We also expect that, on in-distribution examples, the conﬁdence interval only contains the true label with high probability.

Original: To construct these conﬁdence intervals, we take 100 samples from each model on every data point in the test data.
Revised: To construct these conﬁdence intervals, we take 100 samples from each model on every data point in the test data.

Original: We deﬁne the metrics CI-Any as the proportion of data points for which at least one sample contains the correct label and CI-All as the proportion of data points for which every sample predicts the correct label.
Revised: We deﬁne the metrics CI-Any as the proportion of data points for which at least one sample contains the correct label and CI-All as the proportion of data points for which every sample predicts the correct label.

Original: Another desired property of an epistemic uncertainty estimator is in predicting when a data point is outside the training distribution and, thus, any classiﬁcation prediction made on such a data point will likely be incorrect.
Revised: Another desired property of an epistemic uncertainty estimator is in predicting when a data point is outside the training distribution and, thus, any classiﬁcation prediction made on such a data point will likely be incorrect.

Original: For out of distribution data points, we expect high sample variance from the posterior model and low sample variance from in distribution data points.
Revised: For out of distribution data points, we expect high sample variance from the posterior model and low sample variance from in distribution data points.

Original: Thus, variance can be used as a metric to predict which test data points are outside the training distribution.
Revised: Thus, variance can be used as a metric to predict which test data points are outside the training distribution.

Original: To quantify each method’s ability to perform out of distribution detection using sample variance, we construct
Revised: To quantify each method’s ability to perform out of distribution detection using sample variance, we construct

Original: ROC curves for each method and measure the area under the curve (AUC).
Revised: ROC curves for each method and measure the area under the curve (AUC).

Original: [Table]

Original: We tested all methods on both the MNIST dataset and the CIFAR-10 dataset.
Revised: We tested all methods on both the MNIST dataset and the CIFAR-10 dataset.

Original: For MNIST, the in distribtuion classes were [0 , 1 , 2 , 3 , 4 , 5] and the out of distribution classes were [6 , 7 , 8 , 9] .
Revised: For MNIST, the in distribtuion classes were [0 , 1 , 2 , 3 , 4 , 5] and the out of distribution classes were [6 , 7 , 8 , 9] .

Original: For
Revised: For

Original: CIFAR-10, the in distribtuion classes were [ “airplane”, “automobile”, “bird”, “deer”, “dog”, “ship” ] and the out-of-distribution classes were [ “cat”, “frog”, “horse”, “truck” ] .
Revised: CIFAR-10, the in distribtuion classes were [ “airplane”, “automobile”, “bird”, “deer”, “dog”, “ship” ] and the out-of-distribution classes were [ “cat”, “frog”, “horse”, “truck” ] .

Original: To make sure that each method had a large enough sample variance, we chose hyper-parameters such that prior distributions had near 100% CI-All.
Revised: To make sure that each method had a large enough sample variance, we chose hyper-parameters such that prior distributions had near 100% CI-All.

Original: To choose the best regularization hyper-parameter, as in the BO experiments, we ran each method with 5 different regularization parameters, then chose the model with both a high validation accuracy and CI-All.
Revised: To choose the best regularization hyper-parameter, as in the BO experiments, we ran each method with 5 different regularization parameters, then chose the model with both a high validation accuracy and CI-All.

Original: Note that, unlike the other methods tested, ensembles are not a generative model.

Original: Thus, when trying to sample from the ensemble method, we can only sample as many functions as there are members of the ensemble; for our experiments this number is 10.

Original: For all other methods, we take 100 samples for measuring CI-Any, CI-All, and sample variance.
Revised: For all other methods, we take 100 samples for measuring CI-Any, CI-All, and sample variance.

Original: Table 1 summarizes the performance of each model using the metrics of test accuracy, CI-Any,
Revised: Table 1 summarizes the performance of each model using the metrics of test accuracy, CI-Any,

Original: CI-All, and the ROC AUC for out of distribution detection.
Revised: CI-All, and the ROC AUC for out of distribution detection.

Original: Full ROC curves for each method are plotted in Figure 2.
Revised: Full ROC curves for each method are plotted in Figure 4.

Original: While all methods are able to achieve high prediction accuracy and high CI-All on in distribution examples, our method is the only one to achieve high CI-Any on out of distribution examples, illustrating our method’s ability to construct valid conﬁdence intervals.
Revised: While all methods are able to achieve high prediction accuracy and high CI-All on in distribution examples, our method is the only one to achieve high CI-Any on out of distribution examples, illustrating our method’s ability to construct valid conﬁdence intervals.

Original: Additionally, our method signiﬁcantly out performs all other methods on out of distribution detection.
Revised: Additionally, our method signiﬁcantly out performs all other methods on out of distribution detection.

Original: Figure 3 shows two informative test images, one from the in distribution dataset and one from the out of distribution dataset, for both MNIST and CIFAR-10, along with box plots of sampled PMFs from each of the learned models.
Revised: Figure 5 shows two informative test images, one from the in distribution dataset and one from the out of distribution dataset, for both MNIST and CIFAR-10, along with box plots of sampled PMFs from each of the learned models.

Original: While all methods predict the correct label with high precision on the in distribution example, ours has a uniquely wide prediction distribution on the out of distribution example, illustrating high predicted epistemic uncertainty.
Revised: While all methods predict the correct label with high precision on the in distribution example, ours has a uniquely wide prediction distribution on the out of distribution example, illustrating high predicted epistemic uncertainty.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we introduce Generative Posterior Networks (GPNs), a method to learn a generative model of the Bayesian posterior distribution by regularizing the outputs of the network towards samples of the prior.
Revised: In this paper we introduce Generative Posterior Networks (GPNs), a method to learn a generative model of the Bayesian posterior distribution by regularizing the outputs of the network towards samples of the prior.

Original: We prove that under mild assumptions, GPNs approximate samples from the true posterior.
Revised: We prove that under mild assumptions, GPNs approximate samples from the true posterior.

Original: We then show empirically that our method is not only competitive with the best uncertainty predictions techniques on small Bayesian Optimization problems, it signiﬁcantly outperforms these competing methods on high-dimensional classiﬁcation tasks.
Revised: We then show empirically that our method is not only competitive with the best uncertainty predictions techniques on small Bayesian Optimization problems, it signiﬁcantly outperforms these competing methods on high-dimensional classiﬁcation tasks.

Original: A P ROOFS A. P ROOF OF L EMMA
Revised: A P ROOFS A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: It is a standard result that linear feed-forward neural networks have bounded Lipschitz constants.
Revised: It is a standard result that linear feed-forward neural networks have bounded Lipschitz constants.

Original: We’ll denote L ( θ ) as the Lipschitz constant of f for a given set of parameters θ .
Revised: We’ll denote L ( θ ) as the Lipschitz constant of f for a given set of parameters θ .

Original: Recall that this likelihood is deﬁned as follows:
Revised: Recall that this likelihood is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Let x sample (cid:48)
Revised: Let x sample (cid:48)

Original: ⊂ x sample be the subset of points in our discretization, x sample , that are closest to the observed points, x obs ; in other words, for each element x i obs ∈ x obs , there exists an element x i ∈ x sample (cid:48) where:
Revised: ⊂ x sample be the subset of points in our discretization, x sample , that are closest to the observed points, x obs ; in other words, for each element x i obs ∈ x obs , there exists an element x i ∈ x sample (cid:48) where:

Original: [Equation]
Revised: [Equation]

Original: We denote the elements of x sample (cid:48) as [ x (cid:48) 1 , . . .
Revised: We denote the elements of x sample (cid:48) as [ x (cid:48) 1 , . . .

Original: , x (cid:48) N ] .
Revised: , x (cid:48) N ] .

Original: Let ˆY (cid:48)
Revised: Let ˆY (cid:48)

Original: ⊂ ˆY be the corresponding elements in ˆY ; in other words, ˆ Y (cid:48)
Revised: ⊂ ˆY be the corresponding elements in ˆY ; in other words, ˆ Y (cid:48)

Original: i =
Revised: i =

Original: f ( x (cid:48)
Revised: f ( x (cid:48)

Original: i ; θ ) for i ∈ [1 , . .
Revised: i ; θ ) for i ∈ [1 , . .

Original: . , N ] .
Revised: . , N ] .

Original: By deﬁnition of our discretiztaion, we know that for all i ∈ [1 , . . .
Revised: By deﬁnition of our discretiztaion, we know that for all i ∈ [1 , . . .

Original: , N ] , (cid:107) x
Revised: , N ] , (cid:107) x

Original: i obs − x (cid:48)
Revised: i obs − x (cid:48)

Original: i (cid:107) ≤ lM .
Revised: i (cid:107) ≤ lM .

Original: Using the Lipschitz-continuity of f , for any setting of parameters θ , (cid:107) f
Revised: Using the Lipschitz-continuity of f , for any setting of parameters θ , (cid:107) f

Original: ( x i obs ; θ ) − ˆ Y (cid:48)
Revised: ( x i obs ; θ ) − ˆ Y (cid:48)

Original: i (cid:107) ≤ lM L ( θ ) .
Revised: i (cid:107) ≤ lM L ( θ ) .

Original: If we take the limit as M → ∞ , f ( x i obs ; θ )
Revised: If we take the limit as M → ∞ , f ( x i obs ; θ )

Original: =
Revised: =

Original: ˆ Y i and
Revised: ˆ Y i and

Original: P ( y obs | ˆY )
Revised: P ( y obs | ˆY )

Original: [Equation]
Revised: [Equation]

Original: Now we know that P ( y obs | ˆY ) ∝ P ( y obs | θ ) .
Revised: Now we know that P ( y obs | ˆY ) ∝ P ( y obs | θ ) .

Original: But since both P ( y obs | ˆY ) and P ( y obs | θ ) are both probability distributions of y obs , then P ( y obs | ˆY )
Revised: But since both P ( y obs | ˆY ) and P ( y obs | θ ) are both probability distributions of y obs , then P ( y obs | ˆY )

Original: =
Revised: =

Original: P ( y obs | θ ) .
Revised: P ( y obs | θ ) .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We will start by showing that P ( ˆY | y obs ) is normally distributed.
Revised: We will start by showing that P ( ˆY | y obs ) is normally distributed.

Original: By Lemma 1, we know that lim M →∞ P
Revised: By Lemma 1, we know that lim M →∞ P

Original: [Equation]
Revised: [Equation]

Original: y obs | ˆY ) =
Revised: y obs | ˆY ) =

Original: P ( y obs | ˆY ) .
Revised: P ( y obs | ˆY ) .

Original: Thus, in the limit as M → ∞ ,
Revised: Thus, in the limit as M → ∞ ,

Original: [Equation]
Revised: [Equation]

Original: for some mean and covariance µ like , Σ like .
Revised: for some mean and covariance µ like , Σ like .

Original: Since the product of two multivariate Gaussians is a
Revised: Since the product of two multivariate Gaussians is a

Original: Gaussian, we know that:
Revised: Gaussian, we know that:

Original: [Equation]
Revised: [Equation]

Original: 16) where
Revised: 17) where

Original: [Equation]
Revised: [Equation]

Original: Now we consider at the distribution P ( ˆ Y MAP ( ˆY anc )) where ˆY anc ∼ N ( µ anc , Σ anc ) .
Revised: Now we consider at the distribution P ( ˆ Y MAP ( ˆY anc )) where ˆY anc ∼ N ( µ anc , Σ anc ) .

Original: We will show that, when we set
Revised: We will show that, when we set

Original: [Equation]
Revised: [Equation]

Original: the distribution P ( ˆ Y MAP ( ˆY anc )) becomes equal to the posterior distribution N ( µ post , Σ post ) .
Revised: the distribution P ( ˆ Y MAP ( ˆY anc )) becomes equal to the posterior distribution N ( µ post , Σ post ) .

Original: There are three steps needed to show equality between these distributions:
Revised: There are three steps needed to show equality between these distributions:

Original: Show that P ( ˆ Y MAP ( ˆY anc )) is normally distributed for some mean and variance, µ RMSpost , Σ RMSpost .
Revised: Show that P ( ˆ Y MAP ( ˆY anc )) is normally distributed for some mean and variance, µ RMSpost , Σ RMSpost .

Original: Show that µ RMSpost = µ post . 3.
Revised: Show that µ RMSpost = µ post . 3.

Original: Show that Σ RMSpost = Σ post .
Revised: Show that Σ RMSpost = Σ post .

Original: Using the same reasoning as above, in the limit as M → ∞ ,
Revised: Using the same reasoning as above, in the limit as M → ∞ ,

Original: [Equation]
Revised: [Equation]

Original: Since the max of a Gaussian is the mean, then
Revised: Since the max of a Gaussian is the mean, then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where we deﬁne:
Revised: ) where we deﬁne:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We will now show that E [ ˆ Y MAP ( ˆY anc )] = µ post .
Revised: We will now show that E [ ˆ Y MAP ( ˆY anc )] = µ post .

Original: Because we set µ anc = µ ˆY :
Revised: Because we set µ anc = µ ˆY :

Original: [Equation]
Revised: [Equation]

Original: Finally, we will show that V ar [ ˆ Y MAP ( ˆY anc )]
Revised: Finally, we will show that V ar [ ˆ Y MAP ( ˆY anc )]

Original: = Σ post .
Revised: = Σ post .

Original: [Equation]
Revised: [Equation]

Original: So, P ( ˆ Y MAP ( ˆY anc )) ∼ N ( µ RMSpost , Σ RMSpost )
Revised: So, P ( ˆ Y MAP ( ˆY anc )) ∼ N ( µ RMSpost , Σ RMSpost )

Original: where µ RMSpost = µ post and Σ RMSpost = Σ post .
Revised: where µ RMSpost = µ post and Σ RMSpost = Σ post .

Original: Thus, in the limit as M → ∞ , P ( ˆ Y MAP ( ˆY anc )) =
Revised: Thus, in the limit as M → ∞ , P ( ˆ Y MAP ( ˆY anc )) =

Original: P ( ˆY | y obs ) .
Revised: P ( ˆY | y obs ) .

Original: A. D ERIVATION OF LOSS FUNCTION
Revised: A. D ERIVATION OF LOSS FUNCTION

Original: We start with
Revised: We start with

Original: [Equation]
Revised: [Equation]

Original: If we choose µ anc = µ ˆY and Σ anc =
Revised: If we choose µ anc = µ ˆY and Σ anc =

Original: Σ ˆY , we can simplify the logarithm of the likelihood and anchor distributions as follows:
Revised: Σ ˆY , we can simplify the logarithm of the likelihood and anchor distributions as follows:

Original: [Equation]
Revised: [Equation]

Original: where δ j = g ( x j sample , θ anc ; φ ) − f ( x j sample ; θ anc ) .
Revised: where δ j = g ( x j sample , θ anc ; φ ) − f ( x j sample ; θ anc ) .

Original: Putting these together [Equation] we get:
Revised: Putting these together, we get:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]