Understanding Variational Inference in Function-Space
Abstract
Recent work has attempted to directly approximate the ‘function-space’ or predictive posterior distribution of Bayesian models, without approximating the posterior distribution over the parameters.
This is appealing in e.g. Bayesian neural networks, where we only need the former, and the latter is hard to represent.
In this work, we highlight some advantages and limitations of employing the Kullback-Leibler divergence in this setting.
For example, we show that minimizing the KL divergence between a wide class of parametric distributions and the posterior induced by a (non-degenerate)
Gaussian process prior leads to an ill-deﬁned objective function.
Then, we propose (featurized)
Bayesian linear regression as a benchmark for ‘function-space’ inference methods that directly measures approximation quality.
We apply this methodology to assess aspects of the objective function and inference scheme considered in Sun et al.
(2018), emphasizing the quality of approximation to
Bayesian inference as opposed to predictive performance.
1. Introduction
While neural networks oﬀer a successful parametric representation of functions, performing
Bayesian inference over the parameters is challenging.
Since only the predictions matter, a recent line of work has focused on directly approximating the posterior predictive distribution in ‘function-space’
(Sun et al., 2018; Ma et al., 2019; Wang et al., 2019), with the intent of reducing the inﬂuence of the speciﬁc parameterization on the quality of inference.
Similar to approximate Gaussian processes, a variational approach can be taken, where a discrepancy (e g. a KL divergence) is minimized between approximate and posterior predictive distributions (Matthews et al., 2016).
However, we are dealing with measures on functions, so care is needed to ensure that the divergence is well-deﬁned and a useful objective function.
In this work, we investigate situations where this may not form a useful objective.
Sun et al.
(2018) hint that: “the function space KL divergence may be inﬁnite, for instance if the prior assigns measure zero to the set of functions representable by a neural network”.
We give examples of ill-deﬁned objective functions that arise in the context of variational inference between approximate and exact predictive posteriors.
We give a proof that variational inference using the KL divergence (or any f -divergence) does not lead to a sensible objective function when the prior is a non-degenerate Gaussian process and the variational family contains only ‘nice’ parametric models (or vice versa).
We show similar results for single hidden layer (1HL) Bayesian neural networks (BNNs) in the case when the prior has a diﬀerent width than the approximate posterior and ReLU activation functions are used.
Our proofs are contained in the appendix.
Having established that issues with the objective function can arise; we consider how to assess the quality of the approximation to the posterior achieved by methods motivated by performing variational inference in function-space.
We propose Bayesian linear regression (BLR) as a benchmark for measuring the eﬀectiveness of functional variational inference schemes.
Since the exact posterior, as well as the KL divergence in function-space, can be computed in this case, we can directly assess the quality of approximate inference separately from the quality of predictive performance, which may be good even in cases where inference is not accurate.
We show how this benchmark can be used to decouple the impacts of further approximations to the objective function made by Sun et al.
This approach gives a principled starting point for assessing future improvements in functional variational inference.
2. Background
We begin by introducing some measure theory notation, which is required to handle the distributions on functions that we perform inference with.
We review the data processing inequality, which is needed for the proofs, and discuss the work of Sun et al.
(2018) which we analyze further in this work.
Notation
A probability measure, P , is a function from subsets (‘events’) to [0 , 1], such that the probability of the event E is given by P ( E ).
In the case when the subsets are contained in R k , this can sometimes be related to a (Lebesgue) density p : R k
→ [0 , ∞ ) by P ( E ) = (cid:82) z ∈ E p ( z ) dz .
A random variable, Z , is said to have distribution P (and we write Z ∼ P ) if for all events E , Prob( Z ∈ E )
=
P ( E ).
For measures deﬁned on the same event space, we write Q (cid:28) P if for all E such that P ( E ) = 0, we have Q ( E ) = 0.
The
Kullback-Leibler (KL) divergence between two probability measures Q and P is given by,
[Equation]
where dQdP denotes the Radon-Nikodym derivative, which is simply the ratio of the densities of these measures when the densities exist.
1 While we focus on the KL divergence in the main text due to its wide-spread use in variational inference, our results can be naturally extended to other f -divergences as described in appendix B.
The data processing inequality
Given a random variable Z ∼ P , we can transform it by a function g to ﬁnd a new random variable g ( Z ).
We refer to its distribution as the pushforward measure of P , which we denote g ∗ P .
The data processing inequality states that if two random variables are transformed in this way, they cannot become easier to tell apart.
Proposition 1 (Data processing inequality Polyanskiy and Wu, 2014, Thm 6.2)
Let g be a measurable function, then D KL ( g ∗ P, g ∗ Q ) ≤ D KL ( P, Q ) ,
Background on Functional Variational Inference
We consider the application of variational inference to regression.
In particular, we assume data D
= { ( x n , y n ) }
Nn =1 has been observed, with x n ∈ X =
R d and y n ∈ R .
We assume an additive noise model, i.e ˆ y ( x ∗ ) =
ˆ f ( x ∗ ) + (cid:15) ∗ , where ˆ f is a stochastic process indexed by X and each (cid:15) ∗ is an independent mean-zero random variable.
We assume a priori that ˆ f ∼ P .
The goal is to approximate the posterior distribution of ˆ f given the data D , P D .
Deﬁne (cid:96) D : R X
→ R to be the likelihood function given the observed data D .
Given an approximate distribution
Q , variational inference (Blei et al., 2017) gives us the evidence lower bound (ELBO)
[Equation]
Maximizing the RHS of eq.
(2) over Q ∈ Q is equivalent to minimizing D KL ( Q, P D ).
Moreover, the RHS of eq.
(2) can often be estimated: (cid:82) log (cid:96) D dQ can be estimated with Monte
Carlo methods so long as we can evaluate (cid:96) D and sample from Q .
In the context of variational inference in parameter space, Q is generally constrained to be from some family such that D KL ( Q, P ) is tractable (e.g. if P is an isotropic Gaussian distribution, then Q is often chosen to be Gaussian so that this KL divergence can be evaluated in closed form).
Sun et al.
(2018) proposed using eq.
(2) with Q and P the approximate predictive and prior predictive distributions, in which case D KL ( Q, P ) is a divergence between measures on the inﬁnite product space R X .
The starting point for their work is:
Proposition 2 (Sun et al.
2018, Theorem 1) For measures Q, P on (the product σalgebra of) R X ,
[Equation]
where Q X , P X are the marginals of the measures Q and P on the set X .
In other words, the KL divergence between the stochastic processes is equal to the supremum of the KL divergence between the measures restricted to ﬁnite marginals.
Substituting eq.
(3) into eq.
(2) introduces two sources of intractability.
First, the supremum is over uncountably many subsets, and will be generally intractable.
Second, the distributions
Q X , P X are often deﬁned implicitly through a tractable sampling procedure, which does not provide closed form densities (with the notable exception when either Q X or P X is a
Gaussian measure).
Sun et al.
(2018) propose replacing the supremum with an expectation, and using this in eq.
(2) to address the ﬁrst intractability.
This involves deﬁning a distribution over ﬁnite subsets of X , e.g. sampling points from the data as well as uniformly from a subset of X , and using the KL divergence between the approximate posterior and the prior restricted to this index set.
The second intractability can be addressed using any form of implicit inference (Husz´ar, 2017).
Sun et al.
(2018) use the spectral Stein gradient estimator (SSGE; Shi et al.
to obtain estimates of the gradient of the KL.
3. Properties of the KL divergence in function-space
In this section we discuss properties of KL divergences in function-space, noting that our results generalize with minor modiﬁcations to all f -divergences (appendix B).
We focus on ﬁnding conditions under which eq.
(2) is a well-deﬁned objective.
We discuss the case when parametric models are used to describe both the prior and the approximate posterior, and then move to the case when a Gaussian process is used as either the approximate posterior or the prior.
Parametric distributions
We call a distribution parametric if it is described by a probability distribution over a parameter space, which we assume is R k , as well as a mapping from parameters Θ to functions, i.e. ˆ f ( x ) =
h ( x, Θ) for all x ∈ X = R d .
Note that ˆ f is a random function (stochastic process) indexed by X .
Moreover, from its deﬁnition, we see that given Θ ∼ P Θ , upon deﬁning g (Θ)( x ) =
h ( x, Θ), we have ˆ f = g (Θ) ∼ g ∗ P Θ .
We call such a distribution parameterized by the pair ( P Θ , g ).
The function g : R k
→ R X is the mapping from parameters to functions and is assumed to be measurable.
From the data processing inequality (proposition 1), we make the following observation:
Proposition
Suppose that the approximate posterior is parameterized by ( Q θ , g ) and that the prior is parameterized by ( P θ , g ) .
Deﬁne Q = g ∗ Q θ and P = g ∗ P θ to be the approximate posterior predictive and the prior predictive respectively.
Then,
[Equation]
If g is injective (each set of parameters leads to a unique predictive function) equality holds.
Remark 1 Proposition 3 implies that in cases where we can perform variational inference in parameter space, variational inference in function-space is also well-deﬁned.
While the
ELBO using KL-divergence in parameter-space depends on the choice of parameterization (unless g is injective), the ELBO in function-space does not.
Ma et al.
(2019) observed the inequality in eq.
(4) in the context of performing inference with stochastic processes; it is an immediate consequence of the data processing inequality.
The equality can be derived by noting that if g is injective, it has a left inverse and the data-processing inequality can be applied in the opposite direction.
We note that it is possible that for speciﬁc Q, P equality can hold even if g is not injective: in other words, the converse is not generally true.
When applied to variational inference in function-space, this inequality tells us that the
KL divergence in function-space is no larger than the KL divergence in parameter space, implying that the evidence lower bound (ELBO) obtained in function-space must be closer to the log marginal likelihood than the ELBO in parameter space.
Bayesian neural networks
The above discussion applies immediately to variational inference with Bayesian neural networks (after establishing measurability of g ), given the approximate posterior and the prior have the same architecture.
This can be seen as motivation for using functional variational inference, particularly since BNNs are highly nonidentiﬁable (e.g. permuting neurons leads to identical predictions), and equality between parameter-space and function-space KLs does not generally hold in these models.
A natural question is whether variational inference can be applied when the prior and candidate approximate posteriors have diﬀerent architectures.
We show in appendix C that in the case of single hidden ReLU networks, if the prior and approximate posterior have diﬀerent widths, and both Q Θ and P Θ have densities (with respect to Lebesgue measure), then the KL divergence between the approximate posterior and the prior is inﬁnite.
We conjecture that this result is true much more broadly for Bayesian neural networks of diﬀerent architectures, excluding cases where architectures are trivially the same (e.g. if a neuron is added but the outgoing weight is 0 with probability 1 so that the additional neuron is always pruned).
Remark 2 Proposition 9 (appendix C) shows that variational inference in function space is not always well-deﬁned when both the prior and approximate posteriors are deﬁned using neural networks, if the architectures are not the same.
Parametric distributions and Gaussian processes
A Gaussian process (GP) is a random function such that when the function is indexed at any ﬁnite collection of points, the distribution of the function values is multivariate Gaussian.
We call a Gaussian process non-degenerate if there exist arbitrarily large collections of points where, when we evaluate the function at these points, the resulting multivariate Gaussian has a full-rank covariance matrix (equivalently has a density with respect to the appropriate Lebesgue measure).
Gaussian processes have been proposed for use in functional inference schemes both as priors (Sun et al., 2018) and as approximate posteriors (Ma et al., 2019).
However, under quite general conditions, we show that the KL divergence between Gaussian processes and parametric models is not a useful objective.
Proposition 4 Let ( Q θ , g ) parameterize the approximate posterior of a parametric model and let P be a non-degenerate Gaussian process.
Assume that g ( · )( x ) is locally Lipschitz for all x .
Let Q = g ∗ Q θ .
Then, D KL ( Q, P ) = ∞ and D KL ( P, Q ) = ∞ .
The assumption that g is locally Lipschitz in each output is very weak; it holds for most commonly used Bayesian machine learning models, including (deep) BNNs with ReLU, tanh or sigmoid non-linearities.
Remark 3 Proposition 4 tells us that using KL divergences as an objective function to approximate Gaussian processes with parametric models does not lead to a useful objective.
However, this does not mean that parametric models cannot approximate Gaussian processes well.
This is evidently false from the success of methods such as Random Fourier features (Rahimi and Recht, 2007) and Subset of Regressors (Wahba, 1990, § 7).
While propositions 4 and 9 highlight limitations of variational objective deﬁned in function space, we believe that the overall idea of approximating the predictive posterior as opposed to the parameter-space posterior is well-motivated.
Propositions 4 and 9 suggest the need for other objective functions for this task, as well as carefully assessing whether the predictive posterior obtained by variational inference in function space resembles the exact predictive posterior.
[Figure]
4. Benchmarking Functional Approaches to Variational Inference
Bayesian linear regression with Gaussian priors can be used as a tool for assessing the quality of variational inference in function-space.
We consider the model,
[Equation]
where φ : X → R k is a feature mapping, and Θ is the (random) vector of weights.
Recall the notation introduced in Section 3, g (Θ)( x ) =
Θ T φ ( x ).
In this case, we can verify that g is injective for a given set of features by ﬁnding a set of k inputs A
= { a 1 , . . .
, a k } such that the vectors { φ ( a 1 ) , . . .
, φ ( a k ) } are linearly independent.
Therefore, proposition 3 implies that the KL divergence in parameter and function-space are exactly the same .
We can therefore expect that successful inference methods with identical variational families should obtain the same approximate posteriors, regardless of whether they are represented in the parameter space or function-space.
Since the exact solution for BLR is Gaussian and can be computed analytically, we can compare diﬀerent function-space inference methods by seeing which method ﬁnds the solution with the smallest KL divergence to the exact posterior.
This allows us to assess the quality of inference while avoiding potential issues of model mis-speciﬁcation, whereby it is possible to achieve good test performance with a poor model by using poor inference.
Following Sun et al.
, we consider the modiﬁed variational objective,
[Equation]
(6) where P A , Q A denote marginals of P, Q at points indexed by A and µ is a measure on subsets of X .
We note that eq.
(6) may be ﬁnite, even in cases where eq.
(2) is not.
We compare four algorithms using both full-covariance Gaussians ( Full ) and fully-factorised
Gaussians ( FFG ) as the approximating families (noting that Full will contain the true posterior).
As a baseline, we consider using the Exact KL divergence, i.e. D KL ( Q, P ), which we can obtain since the KLs are the same in weight and function-space.
In FixedA , we randomly select a set of input points A and keep it ﬁxed throughout training in eq.
For RandA , we use the approach proposed in Sun et al.
and sample a diﬀerent set A at each iteration, so that we Monte Carlo evaluate E A ∼ µ [D KL ( Q A , P A )] in eq.
However,
Full
FFG 0.
K L vs .
t r ue po s t e r i o r × 10boston
Full
FFG   × 10concrete
Full
FFG 0.
× 10energy
Full
FFG 2.
T e s t nega t
i v e l og den s i t y boston
Full
FFG 3.
concrete
Full
FFG   energy
Exact FixedA
RandA SSGE as we can evaluate the KL exactly in this case, we do not use SSGE, instead leaving it for the ﬁnal algorithm, SSGE , which uses the random sampling scheme as well.
Therefore,
SSGE is similar to the implementation in Sun et al.
(2018), although we do not use their heuristic for re-scaling the KL term to reduce over-ﬁtting (as they note that the modiﬁed objective will underestimate the KL term since it cannot achieve the supremum over all ﬁnite inputs).
We consider two experiments, for which we provide additional experimental details in appendix F.
Toy experiment
We generate a synthetic 1D dataset by sampling a 20-dimensional weight vector from the prior and applying it to 20 radial basis function features.
We then use this model to sample 40 noisy ( x, y ) pairs with a noise standard deviation of 0 . 1.
We perform full-covariance inference with each method and plot the predictive posteriors in Figure 1.
The posteriors found using the approximate KL divergence are prone to over-ﬁtting; in the case of FixedA it can be shown the optimal mean behaves like a combination of maximum likelihood estimation ( MLE ) and maximum a posteriori ( MAP ) inference (appendix E).
UCI regression task
We ﬁt a sparse variational GP (Titsias, 2009) with a squaredexponential kernel, with a separate length-scale for each input dimension.
The learned kernel parameters and inducing points form a good representation of the training data (Wahba, 1990, § 7).
We use these features for linear regression.
We show the negative log predictive densities (NLPDs) and KLs to the true posterior for the boston , concrete , and energy datasets in Figure 2, providing additional results in appendix F.
Conclusions
In general, we observe from both experiments that, of the approximate methods RandA , performs the best in terms of matching the true posterior.
However, we note that all the approximate methods exhibit some amount of overﬁtting, which we would expect since the true functional KL divergence is obtained by taking the supremum over all ﬁnite marginals.
Finally, we note that SSGE tends to perform worse than RandA , which we would expect as it introduces an additional approximation.
It is our hope that using this benchmark will help researchers ﬁnd ways of improving these methods; for example, using other methods of implicit inference to reduce the discrepancy between RandA and a version of inference that does not rely on Gaussianity. Acknowledgments The authors would like to thank Andrew Y.K.
Foong and Wessel Bruinsma for providing feedback on a draft of this work as well as Christos Louizos for useful conversations related to this research.
DRB would like to acknowledge funding from the Qualcomm innovation fellowship and the Herchel Smith fellowship. SWO acknowledges the support of the Gates Cambridge Trust in funding his doctoral studies. Appendix A. Measure-theoretic Deﬁnitions and Lemmas
In this section we recall several deﬁnitions and lemmas that will be useful in formalizing the results in the main text
We say two measures P, Q on a common measurable space are equivalent and write
P ∼ Q if P (cid:28) Q and Q (cid:28) P .
We say P and Q are mutually singular and write P ⊥ Q if there exists a (measurable) event E such that P ( E ) = 0 and Q ( E c )
=
(where E c denotes the complement of E ).
In the case of probability measures, this is the same as P ( E ) =and Q ( E ) = 1.
Lemma
Let Q, Q (cid:48) , and P be three measures on the measurable space ( A, Σ A ) .
Then
Q ∼ Q (cid:48) and Q (cid:48) ⊥ P implies that Q ⊥ P .
Proof By the assumption Q (cid:48) ⊥ P , there exists an event E ∈
Σ
A such that Q (cid:48)
( E ) =
and
P ( E c ) = 0.
Since Q ∼ Q (cid:48) , Q (cid:48)
( E ) =
implies Q ( E )
= 0.
Deﬁnition 1 (Canonical projection) For any A ⊂ X , let π A : R X
→ R A denote the canonical projection onto A , i.e. π A ( f ) =
( f ( a )) a ∈ A .
A.1. Product σ -algebra
For any ﬁnite A ⊂ X , we let λ A denote Lebesgue measure on R A , restricted to the Borel σ -algebra.
As in Sun et al.
, we consider the product σ -algebra on R X i.e. the coarsest σ -algebra on R X such that π { x } : R X
→ R is measurable as a map to R equipped with the
Borel σ -algebra for all x ∈ X .
For arbitrary A ⊂ X the map π A is measurable when R A and
R X are both equipped with their respective product σ -algebras (see Tao (2011, Exercise 2.4.1.2)).
Lemma
Let Q, P denote measures on Σ .
Suppose there exists an A ⊂ X such that π A ∗ Q ⊥
π A ∗ P .
Then Q ⊥ P .
Proof Let E ∈ Σ
A denote a witness to the orthogonality of π A ∗ P ⊥ π A ∗ Q .
Then π − 1 A ( E ) ∈ Σ by the measurability of projections.
It follows from the deﬁnition of a pushforward measure that π − 1 A ( E ) is a witness to the orthogonality of P and Q .
A.2. Topological Lemmas
As we work in the product σ -algebra generated by the Borel σ -algebra, we use several lemmas from topology in order to prove sets are measurable.
We note that in general the product σ -algebra on R X generated by the Borel σ -algebra on R is not the same as the
Borel σ -algebra generated by the product topology on R X when X is uncountable (Tao, 2011, Exercise 2.4.1.5-6).
Lemma 3 (Closed Mapping Lemma (Conrad, Theorem 2.6))
Suppose X is compact and Y is Hausdorﬀ. Let φ : X → Y continuous, then φ is a closed map.
Lemma
Suppose X can be written as a countable union of compact subspaces and Y is
Hausdorﬀ. Let φ : X → Y continuous.
Let A ⊂ X be a closed set, the φ ( A ) is a countable union of closed sets.
Proof As X is countable union of compact spaces, we can write X =
(cid:83)
i ∈ N X i , with X i compact.
Deﬁne A
i =
A ∩ X i , and note that A i is closed in the subspace topology of X i .
Deﬁne φ i : X i → Y to be the restriction of φ to X i ; it follows from the deﬁnition of the subspace topology φ i is continuous.
Then,
[Equation]
By the closed mapping lemma φ i ( A i ) is closed for all i .
Appendix B. f -divergences
In this appendix, we brieﬂy recall the deﬁnition of an f -divergence, as well as the necessary results to generalize our claims from the Kullback-Leibler divergence to other f -divergences.
We use the deﬁnition from Polyanskiy and Wu (2014),
Deﬁnition
Given a measurable space (Ω , Σ) and a convex function f : [0 , ∞ )
→ R satisfying f (1) = 0 which is strictly convex at 1 .
For any two probability measures P, Q on Σ ,
[Equation]
with p ( z ) =
dP dµ ( z ) and q ( z ) =
dQ dµ ( z ) , µ is an arbitrary dominating measure (e.g. ( P + Q ) / 2 ), f (cid:48) ( ∞ ) =
lim
z → 0 + zf (1 /z ) and the understanding that if Q ( { z : p ( z ) = 0 } )
the second term is 0 (even if f (cid:48) ( ∞ ) = ∞ ).
Examples of f -divergences include KL divergence, total variation distance, squared
Hellinger distance and α -divergence.
The data processing inequality holds for general fdivergences:
Proposition 5 (Data processing inequality Polyanskiy and Wu (2014, Thm 6.2))
Let ( A, Σ A ) and ( B, Σ B ) measurable spaces and g :
A → B a (Σ A , Σ B ) -measurable function, then for any f -divergence D f ,
[Equation]
where g ∗ P indicates the pushforward measure of P by g .
From this the analogue of proposition 3 holds for general f -divergences.
Proposition
Suppose the approximate posterior is parameterized by ( Q θ , g ) and the prior is parameterized by ( P θ , g ) .
Deﬁne Q = g ∗ Q θ and P = g ∗ P θ to be the approximate posterior predictive and the prior predictive respectively.
Then for any f -divergence D f ,
[Equation]
Moreover, if g is injective (each set of parameters corresponds to a unique predictive function) then equality holds in eq.
Proof For the inequality, using proposition 5,
[Equation]
Suppose g is injective, then there exists a g (cid:48) :
R X
→ R k such that g (cid:48) ◦ g ( θ )
= θ for all θ ∈ R k
( g has a left inverse).
Then,
[Equation]
If P ⊥ Q , then Q ( p = 0) = 1, so that D f ( Q, P ) =
f (cid:48) ( ∞ ).
Note that this value is the same for all P ⊥ Q (and by the convexity of f is the maximum value that can be obtained by the f -divergence), so that if all Q ∈ Q are mutually singular to P , then any f -divergence is entirely independent of which Q ∈ Q is selected.
This leads to the generalizations of propositions 4 and 9 for other f -divergences:
Proposition 7 Suppose the approximate posterior is parameterized by ( Q Θ 1 , g 1 ) and the prior is parameterized by ( P Θ 2 , g 2 ) , where both Q θ and P θ have densities (with respect to
Lebesgue measure).
Further suppose that g 1 (Θ 1 ) is the mapping deﬁned by a 1HL BNN with ReLU activation functions, k neurons and parameters Θ
and that g 2 (Θ 2 ) is deﬁned similarly, but with j (cid:54) =
k neurons.
Let Q = g 1 ∗ Q Θ 1 and P = g 2 ∗ P Θ 2 denote the approximate posterior predictive and the prior predictive respectively, then
[Equation]
Proposition 8 Let ( Q Θ , g ) parameterize the approximate posterio and P be a (non-degenerate)
Gaussian process.
Assume that g ( · )( x ) is locally Lipschitz for all x .
Let Q = g ∗ Q θ .
Then,
D
[Equation]
( Q, P
[Equation]
= f (cid:48) (
[Equation]
and
[Equation]
By choosing f ( z ) =
z log( z ), we obtain the KL divergence used above.
In this case f (cid:48) ( ∞ ) =
lim
z → 0 + log(1 /z )
= ∞ .
Appendix C. ReLU BNNs are mutually singular
Proposition 9 Suppose the approximate posterior is parameterized by ( Q Θ 1 , g 1 ) and the prior is parameterized by ( P Θ 2 , g 2 ) , where both Q Θ 1 and P Θ 2 have densities (with respect to the appropriate Lebesgue measures).
Further suppose that g 1 (Θ 1 ) is the mapping deﬁned by a 1HL BNN with ReLU activation functions, k neurons and parameters Θ 1 and g 2 (Θ 2 ) is deﬁned similarly, but with j (cid:54) =
k neurons.
Let Q = g 1 ∗ Q Θ 1 and P = g 2 ∗ P Θ 2 denote the approximate posterior predictive and the prior predictive respectively, then D KL ( Q, P ) = ∞ .
C.1. Preliminary Deﬁnitions and Lemmas
We will construct a measurable event which one neural network assigns probability 1 to and the other probability 0.
This event will roughly be functions f ∈ R X such that f ( x, 0 , . .
. , 0) is continuous with k + 1 linear pieces.
There are two steps: constructing an event that is measurable and captures this behavior; and showing that a ReLU network with a distribution over parameters with Lebesgue density and k neurons assigns probability 1 to this event.
Deﬁnition 3 Let f ∈ R R , we say f is continuous with k -linear pieces if there exists an x 1 ≤ x 2 ≤ . . .
≤ x k − 1 and b, a 1 , . .
. , a k such that
[Equation]
with the understanding
that x 0 = −∞ , x k = ∞ , b 0 = b , and where b i is selected so that the resulting function is continuous for 0 < i ≤ k .
We say a function ˜ f ∈ R Q is continuous with k -linear pieces if it can be extended to a function f ∈ R R that is continuous with k -linear pieces.
Proposition 10 Deﬁne E k := { f ∈ R R : π Q ( f ) is continuous with k -linear pieces } .
Then
E k is measurable in the product σ -algebra on R R induced by the Borel σ -algebra.
Proof By the measurability of π Q , it suﬃces to show that
[Equation]
is measurable in the product σ -algebra on R Q .
We will use the previous lemma to show that ˜ E k is a countable union of closed sets and is hence measurable in the Borel σ -algebra induced by the product topology on R Q .
As this coincides with the product σ -algebra for countable products of R (Tao, 2011, Exercise 2.4.1.5), this suﬃces.
We will do this by showing that ˜ E k is the image of R 2 k +1 under a continuous function, and applying lemma 4.
As R 2 k +1 is closed, R Q is a product of Hausdorﬀ spaces, hence
Hausdorﬀ and R 2 k +
=
(cid:83)
i ∈ N [ − i, i ]
2 k +
(i.e it is a countable union of compact set), all that remains is to construct a continuous φ : R 2 k +1 → R Q such that φ ( R 2 k +1 ) =
˜
E k .
Let
[Equation]
with ˜ f deﬁned as in deﬁnition 3 (restricted to Q ) with b = ˜ s 0 , a i =
˜ s
i and x 1 = ˜ s k
+ and x i +
= x
i + | ˜ s i + k +1 | .
From this deﬁnition, it is clear that φ ( R 2 k +1 ) ⊂ ˜
E k .
The reverse inclusion follows from the noting that any function of the form in deﬁnition 3 can be written in this form.
It remains to show φ is continuous.
By the universal property of the product topology (Munkres, 2000, Theorem 19.6), we need only show φ { q } is continuous for all q ∈ Q , which can be shown from the metric space deﬁnition of continuity (with some care for cases when q is on the boundary of two linear regions).
C.2. Proof in the case when input space is one-dimensional
We ﬁrst prove proposition 9 under the assumption that X =
R ; the generalization to multidimensional inputs is straightforward.
Proof All that remains to show that the implied measures for two 1HL ReLU BNNs mapping from R → R are orthogonal is showing that if a 1HL ReLU BNN has k neurons, then it produces a function in ˜ E k +1 \ ˜ E k with probability 1.
We ﬁrst show that with probability 1, the implied function is in E k +1 (in fact, this holds surely).
Let w 1 , w 2 , b 1 ∈ R k and b 2 ∈ R be an arbitrary realization of weights and biases, then f ( x )
=
b (2) + (cid:104) w (2) , max(0 , w (1) ◦ x + b (1) ) (cid:105)
=
b (2) , where ◦ denotes an element-wise vector product.
We can rewrite this as
[Equation]
Note that this is piecewise linear, with boundaries at ˜ x i = − b i w i for i ≤ k , and is continuous in x as it can be written as a composition of continuous functions.
As this holds for arbitrary realizations of parameters, f is surely in E k +1 .
On the other hand, for f to be in E k it must be the case that either:
• ˜ x i = ˜ x j for some i (cid:54) =
j (i.e. boundaries coincide).
• There exists an i such that w (1) i w (2) i =
(adjacent regions have the same slope).
The above conditions deﬁne a Lebesgue null set; hence under the assumption that the distribution over parameters has density with respect to Lebesgue measure, this is a probability 0 event.
C.3. Extension to multidimensional inputs
The extension to multidimensional inputs is almost immediate up considering the set E (cid:48)
k :=
{ f ∈ R R d : f ( · , 0 , 0 , . . . , 0) is continuous with k linear pieces } .
Deﬁning ˜ E (cid:48)
k :=
{ f ∈ R Q ×
R R d − 1 : f ( · , 0 , 0 , . .
. , 0) is continuous with k linear pieces on rational x } , we see that ˜ E (cid:48)
k is measurable for the same reason ˜
E k is measurable.
Moreover, viewed along this slice of input space, the neural network is identical to a 1HL network mapping from R → R , so the proof in the previous subsection holds without modiﬁcation.
Appendix D. Parametric models and non-degenerate Gaussian measures are mutually singular D.1. Preliminaries
The main ingredient in the result is the following lemma:
Lemma 5 (Rudin (1966, Lemma 7.25))
Let E ⊂ R k a Lebesgue null set.
Suppose f : R k
→ R k satisﬁes, for all x ∈ E there exists a δ > 0 and M > 0 such that
[Equation]
for all y ∈ E ∩ B ( x ; δ ) where B ( x ; δ ) denotes the ball of radius δ centered at x .
Then λ k ( f ( E ))
= 0 .
Note that the condition on f holds if f is locally Lipschitz.
Moreover, if a function from
R k
→ R k is locally Lipschitz for every output index, it is locally Lipschitz as a function from R k
→ R k .
D.2. Statement and Proof
Proposition 4 Let ( Q θ , g ) parameterize the approximate posterior of a parametric model and let P be a non-degenerate Gaussian process.
Assume that g ( · )( x ) is locally Lipschitz for all x .
Let Q = g ∗ Q θ .
Then, D KL ( Q, P ) = ∞ and D KL ( P, Q ) = ∞ .
Proof Fix a set
A ⊂ X , | A | =
k +
so that π A ∗ P ∼ λ A .
Such an A exists by the assumption that P is non-degenerate and X is inﬁnite.
By lemma 1 and lemma 2 it then suﬃces to show π A Q ⊥ λ A .
Deﬁne the event E =
( π A ◦ g )
( R k ).
As π A ◦ g is continuous, by the closed mapping lemma, E is a countable union of closed sets (lemma 4), hence Borel measurable.
Also,
[Equation]
The inequalities follow from the pre-image of the image of a set containing the original set.
All that is left to show is that E is a Lebesgue null set.
Let ˜ g
A : R k +
→ R A be deﬁned by ˜
g A ( x 1 , . . . , x k +1 ) =
( π A ◦ g )( x 1 , . . .
, x k ).
Then ˜ g
A ( R k × { 0 } )
=
E .
As π A ◦ g is locally
Lipschitz, so is ˜ g
A .
The proof is then completed by lemma 5, noting that R k × { 0 } is a
Lebesgue null set in R k +1 .
Appendix E. Bayesian Linear Regression and Variational Inference
Consider the Bayesian linear regression model eq.
(5) and the modiﬁed variational objective,
[Equation]
which is a special case of eq.
(6) when µ is a point mass on A (i
[Equation]
e. FixedA ).
Let φ X be the n × k feature matrix with [ φ X ] ij = φ j ( x i ).
Let A =
{ a 1 , · · · , a m } and φ A be the m × k feature matrix with [ φ A ] ij = φ j ( a i ).
Without loss of generality, we assume that φ A has linearly independent rows, so that Q A , P A are non-degenerate.
Then,
[Equation]
For Q = N ( µ Q , Σ Q )
[Equation]
we have
[Equation]
The gradient of this term with respect to both µ Q is,
[Equation]
We now turn out attention to the KL divergence between Q A and P A .
We assume
P = N (0 , I ).
We then have Q A = N ( φ A µ Q , φ A Σ Q φ T A ) and P A = N (0 , φ A φ T A ).
The KL divergence is then,
[Equation]
and its gradient with respect to µ Q is,
[Equation]
Note that φ T A ( φ A φ T A ) − 1 φ A is the projection of µ Q onto the column space of φ A .
We can ﬁnd the optimal µ Q solutions to our optimization problem by setting gradient equal to 0.
This yields,
[Equation]
Rearranging,
[Equation]
A solution for µ Q (though this solution is not in general unique) is given by,
[Equation]
where we use † to denote the pseudo-inverse.
The maximum likelihood solution could be found by removing the term φ T A ( φ A φ T A ) − 1 φ A , while map inference is recovered when φ T A ( φ A φ T A ) − 1 φ
A
= I .
As φ T A ( φ A φ T A ) − 1 φ A is a projection matrix, we see this has the intuitive interpretation of regularizing the MLE solution, but only in the directions that eﬀect predictions on the points in A .
Appendix F. Experimental Details
In this section, we present details on the experiments we performed.
F.1. Toy experiment
We generate the data using 20 linearly-spaced radial basis function features betweenand 2, with each feature having lengthscale 0.2.
The input data are sampled by taking 20 samples from N ( − 1 . 2 , 0 .
3 2 ) and 20 from N (1 . 2 , 0 .
We sample the weights from a standard normal prior and generate the output data as described earlier.
For each method, we optimize the ELBO using Adam (Kingma and Ba, 2015) using 5000 gradient steps with a learning rate of 0.01.
We evaluate the likelihood term using the full dataset every iteration without minibatching.
For the FixedA , RandA , and SSGE methods we use 10 points for A , with 5 from selected from the dataset, and the othersampled uniformly from a box with bounds determined by the bounds of the training data.
F.2. UCI experiments
We begin by training sparse Gaussian process regression (SGPR)
(Titsias, 2009) for each split of the dataset we use, using 100 inducing points initialized by sampling from the dataset.
We use the GPﬂow (Matthews et al., 2017) implementation of SGPR, initializing the ARD lengthscales to 1 and the noise variance to 1e-4.
We use the built-in Scipy optimizer with a maximum of 5000 iterations.
After the SGPR model is trained, we use theinducing points to obtain 100 features to use for Bayesian linear regression (Wahba, 1990), with a standard normal prior over the weights.
We then train each of the methods for estimating the KL term, keeping the learned noise variance from the SGPR models ﬁxed.
For Exact , we simply use the closed-form solutions to the inference problem, which exist for both Full and FFG .
For the algorithms that make use of eq.
(6), we set | A | =and choose half of its points from the training set (note this diﬀers slightly from Sun et al.
(2018), who always include the current mini-batch used to evaluate the expected log likelihood term in A ), and the other half uniformly randomly over the bounding box of the training set.
For FixedA , we optimize using L-BFGS (Nocedal, 1980), whereas for RandA and SSGE we optimize using Adam.
In each case we optimize for (up to) 15000 steps.
We note that we add jitter as necessary to ensure that the Cholesky decompositions we perform succeed.
Finally, we average each run over 20 seeds.
Extended results are displayed in ﬁgures 3 and 4.
K L vs .
t r ue po s t e r i o r ×
[Figure]
boston
Batch size and |A| =
Batch size and |A| = Batch size and |A| =
Batch size and |A| =0.
[Figure] ×
[Figure]
concrete
[Figure]
T e s t neg .
l og den s i t y
[Figure]
ExactFull FixedAFullRandAFull SSGEFull ExactFFG FixedAFFGRandAFFG SSGEFFG
[Figure]
T e s t
[Figure] M
SE
ExactFull FixedAFullRandAFull SSGEFull ExactFFG FixedAFFGRandAFFG SSGEFFG
[Figure]
ue po
[Figure]
r i
[Figure]
T e s t neg .
l og den s i t y
[Figure]
ExactFull FixedAFullRandAFull SSGEFull ExactFFG FixedAFFGRandAFFG SSGEFFG
[Figure]
s t
R M
SE
ExactFull FixedAFullRandAFull SSGEFull ExactFFG FixedAFFGRandAFFG SSGEFFG
[Figure] [Figure]