G AUSSIAN P ROCESS M ETA -R EPRESENTATIONS FOR N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian inference offers a theoretically grounded and general way to train neural networks and can potentially give calibrated uncertainty.
It is, however, challenging to specify a meaningful and tractable prior over the network parameters.
More crucially, many existing inference methods assume mean-ﬁeld approximate posteriors, ignoring interactions between parameters in high-dimensional weight space.
To this end, this paper introduces two innovations: (i) a Gaussian process-based hierarchical model for the network parameters based on recently introduced unit embeddings that can ﬂexibly encode weight structures, and (ii) input-dependent contextual variables for the weight prior that can provide convenient ways to regularize the function space being modeled by the network through the use of kernels.
Furthermore, we develop an efﬁcient structured variational inference scheme that alleviates the need to perform inference in the weight space whilst retaining and learning non-trivial correlations between network parameters.
We show these models provide desirable test-time uncertainty estimates, demonstrate cases of modeling inductive biases for neural networks with kernels and demonstrate competitive predictive performance of the proposed model and algorithm over alternative approaches on a range of classiﬁcation and active learning tasks.
I NTRODUCTION
Bayesian neural networks (BNNs) (see e.g. MacKay, 1992; Neal, 1993; Ghahramani, 2016) are one of the research frontiers on combining Bayesian inference and deep learning, potentially offering ﬂexible modelling power with calibrated predictive performance.
In essence, applying probabilistic inference to neural networks allows all plausible network parameters, not just the most likely, to be used for predictions.
Despite some earlier success in supervised learning tasks (Guyon et al., 2005; Quiñonero-Candela et al., 2006), BNNs has enjoyed limited recent practical applications.
This could be attributed to: (i) the need to choose an appropriate weight prior as commonly modelers care about functions priors rather than weight priors but require the power and expressivity of weight-based models, and (ii) recently developed approximate inference methods are not able to capture complicated posterior correlations in large networks.
As such, addressing these challenges is an active research area.
This paper attempts to remedy the aforementioned limitations for Bayesian neural networks by extending previous work on unit-based representations for neural networks with a unit-based Gaussian process hierarchical prior over network parameters, which ﬂexibly models the correlations between weights in a layer and across layers while forgoing the need for a parametric hyper-network.
Moreover, input-dependent contextual variables are used for modulating the GP-function and in turn specifying input-dependent weight priors which draw from the tools from the kernel literature to imbue neural networks with inductive biases.
This may yield more desirable uncertainty estimates at test time, or restrict function spaces to generalize in particular ways.
A structured variational inference approach is employed that side-steps the need to do inference in the weight space whilst retaining weight correlations in the approximate posterior.
These theoretical beneﬁts translate to competitive practical performance on a range of supervised and active learning tasks.
The paper is organized as follows: in the next section, we will brieﬂy review mean-ﬁeld variational inference for BNNs and a recently proposed hierarchical model for BNNs.
The proposed model and inference scheme are discussed in section 3 and section 4, respectively, followed by a discussion on related work in section 5.
We then validate the performance of the proposed model and algorithm in section 6.
B ACKGROUND
In this section, we lay out the groundwork for the paper by providing a concise summary of variational approaches to Bayesian neural networks and a recent approach to learning meta-representations of neural networks.
2. B AYESIAN NEURAL NETWORKS AND VARIATIONAL INFERENCE
Consider a training set comprising of N input-output pairs, D = { x n , y n } Nn =1 , and a neural network parameterized by weights and biases, w , that describes the distribution over an output y n given an input x n , p ( y n | w , x n ) .
We follow a Bayesian approach by placing a prior distribution over the network parameters, p ( w ) , and obtaining the posterior distribution p ( w |D ) , which involves calulation of the marginal likelihood p ( D ) =
(cid:82) d w p ( w ) p ( D| w ) .
However, obtaining p ( w |D ) and p ( D ) exactly is intractable when N is large or when the network is large and as such, approximation methods are often required.
In particular, mean-ﬁeld Gaussian variational inference (MFVI) has recently become a method of choice for approximate inference for Bayesian neural networks due to its simplicity and the recently popularized reparameterization trick (Salimans & Knowles, 2013; Kingma & Welling, 2013; Titsias & Lázaro-Gredilla, 2014; Blundell et al., 2015).
MFVI sidesteps the intractability by positing a diagonal Gaussian approximation q ( w ) =
N ( w ; µµµ, diag( σσσ 2 )) and optimising an approximate lower bound to the marginal likelihood
L MFVI ( q ( w ))
≈ − KL[ q ( w ) || p ( w )]
+ 1 K (cid:80)
Kk =1 (cid:80) Nn =1 log p ( y n | w k , x n ) , where w k = µµµ + σσσ (cid:12) (cid:15) k and (cid:15)
k ∼ N ( 0 , I ) , i.e. w k is a sample from q ( w ) .
Note that the mean-ﬁeld variational Gaussian approximation with a standard normal prior, presented in is often outperformed by point estimation in certain settings (Trippe & Turner, 2018).
Despite being practical and able to give reasonable uncertainty estimates, improving MFVI is still an active research area, and the main focuses of which are (i) improving the reparameterization gradient estimator to enable faster convergence (Miller et al., 2017; Wu et al., 2018), (ii) replacing the typical standard Normal prior, p ( w ) =
N ( w ; 0 , I ) by a structured prior that better models the structures present in the weight a-priori (Ghosh et al., 2018;
Neal, 2012; Blundell et al., 2015), and (iii) using structured variational approximations that can potentially capture weight correlations in the posterior (Louizos & Welling, 2016; Zhang et al., 2017).
This paper builds on the two latter themes and proposes a hierarchical model for the prior and a structured variational scheme that explicitly model and infer weight structures.
2. M ETA - REPRESENTING WEIGHTS AND NETWORKS
Our work builds on and expands the class of hierarchical neural network models based on the concept of latent variables associated with units in a network as proposed in Karaletsos et al.
In these models, each unit (visible or hidden) of the l -th layer of the network has a corresponding latent hierarchical variable z l,i , possibly of high dimensions D z , where i denotes the index of the unit in a layer.
Note that these latent variables do not describe the activation of units, but rather constitute latent features associated with a unit.
These latent variables are then used to construct the weights in the network.
Their construction is judiciously chosen such that a weight in the l -th weight layer, w l,i,j is linked to the latent variables
z ’s of the i -th input unit and the j -th output unit of the weight layer.
We can summarize this relationship by introducing the concept of weight codes C w ( z ) consisting of the set of weight encodings for each individual weight in the network c w l,i,j = (cid:2)
z l +1 ,i , z l,j (cid:3) , which can be deterministically constructed from the collection of unit latent variables z by concatenating them correctly.
Mathematically, the probabilistic description of the relationship between the weight codes (summarizing the structured latent variables) and the weights w is:
[Equation]
where l denotes a visible or hidden layer and H l is the number of units in that layer, and w denotes all the weights in this network.
The central motivations for this parameterization are three-fold.
First , weights in a layer and across layers are explicitly correlated in the modelling stage, as they share latent variables if they are connected to the same unit, unlike the standard normal prior typically used as described in the last section.
Second , the number of visible and hidden units in a neural network is typically much smaller than the number of weights.
For example, for the l -th weight layer, there are H l × H l +1 weights compared to H l + H l +1 associated latent variables.
This encourages the development of inference schemes that can work in the lower-dimensional hierarchical latent space directly without the need to infer the weights.
Third , as hierarchical latent variables are associated with units in the network and have the same cardinalities, inferring these latent variables conditioning on observed data could be thought of as a form of network representation learning.
In Karaletsos et al.
(2018), a small parametric neural network regression model (conceptually a structured hyper-network ) is chosen to map the latent variables to the weights.
The network either assumes a Gaussian noise model which factorizes over weights, p ( w l,i,j | z l +1 ,i , z l,j , θ ) =
N ( w l,i,j ; µ l,i,j , σ 2 l,i,j ) , where ( µ l,i,j , log σ l,i,j ) =
NN θ ([ z l +1 ,i , z l,j ]) , or models dependence between weights implicitly as p ( w | C w ( z ) , θ ) ∝ (cid:82) p ( (cid:15) )
(cid:81)
[Equation]
[Equation] [Equation]
[Equation] [Equation]
, z
[Equation] [Equation]
, (cid:15) ]) d(cid:15) , where (cid:15) is a random variate, w l,i,j = NN θ ([ z l +1 ,i , z l,j , (cid:15) ]) , and θ denotes the parameters of the neural network mapping.
We will call this network a meta mapping .
Note that given the collection of sampled unit variables z , the weights (or theirs mean and variance) can be obtained efﬁciently in parallel.
A prior over the latent variables z , p ( z ) =
N ( z ; 0 , I ) , completes the model speciﬁcation.
We can thus write down the joint density of the resulting hierarchical model as follows,
[Equation]
Variational inference was employed in prior work to infer z (and w implicitly), and to obtain a point estimate of θ , as a by-product of optimising the variational lower bound.
M ETA - REPRESENTATIONS WITH G AUSSIAN PROCESS MAPPINGS AND CONTEXTUAL LATENT VARIABLES
In this section, we present two novel extensions of the hierarchical model in section 2.2 that aim to increase the robustness in the small data settings and improve its out-of-sample uncertainty estimates.
3. M ETA - REPRESENTING WEIGHTS WITH G AUSSIAN PROCESSES
Notice that in section 2.2, the meta mapping from the hierarchical latent variables to the weights is a parametric non-linear function, speciﬁed by a neural network.
We replace the parametric neural network by a probabilistic functional mapping and place a nonparametric Gaussian process (GP) prior over this function.
That is,
[Equation]
where we have assumed a zero-mean GP, k γ ( · , · ) is a covariance function and γ is a small set of hyper-parameters.
The representational effect is that the latent function introduces correlations for the individual weight predictions,
[Equation]
Here, we present a homoscedastic noise model for the weights, but the model is readily adaptable to a heteroscedastic noise model which we omit for clarity.
For a more detailed discussion on GPs and their applications, please see Rasmussen & Williams (2005).
It is worth noting that whilst the number of latent variables and weights can be large, the input dimension to the GP mapping is only 2 D z , where D z is the dimensionality of each latent variable z .
The GP mapping effectively performs one-dimensional regression from latent variables to individual weights while capturing their correlations.
We will refer to this mapping as a GPMetaPrior ( metaGP ).
In general, the form of the covariance function and the values of the hyperparameters encapsulate prior knowledge about the unknown function.
We note that GPs beneﬁt from the fact that kernels are compositional and can thus be used to represent correlation structures explicitly.
We exploit this fact and deﬁne the following factorized kernel at the example of two weights in the network,
[Equation]
In this section and what follows, we will use the popular exponentiated quadratic kernel with ARD lengthscales, k ( x 1 , x
[Equation]
(cid:17) , where { l d } 2 D z d =1 are the kernel lengthscales and σ 2 k is the kernel variance.
For the model considered here, this admits the following separation in the kernel computation:
[Equation]
that is, the computation of the covariance matrix between all the weights can be computed efﬁciently as a product of two sub-covariance matrices that are computed using the latent variables.
3. C ONTEXTUAL VARIABLES FOR MODULATING FUNCTION PRIORS
We ﬁrst note that while the hierarchical latent variables and meta mappings introduce non-trivial coupling between the weights a priori, the weights and latent variables are inherently global.
That is, a function drawn from the model, represented by a set of weights, does not take into account the inputs at which the function will be evaluated.
Inspired by recent work on using contextual information or dataset summary to drive supervised learning predictions (Edwards & Storkey, 2016; Garnelo et al., 2018), we introduce the input variable into the weight codes c w l,i,j = (cid:2)
z l +1 ,i , z l,j , x (cid:3) .
In turn, this yields input-conditional weight models p ( w n,l,i,j | f, z l +1 ,i , z l,j , x n ) .
We again turn to compositional kernels and introduce a new input kernel K x which we use as follows,
[Equation]
As a result of having private contextual inputs to the meta mapping, the weights are now also local to each data point.
In effect, each function drawn from this model explicitly needs the input locations we are interested in, in a similar vein to how functions are drawn from a Gaussian process.
What effects should we expect from such a modulation?
As usual, the modeler has to choose the input kernel.
Consider the use of an exponentiated quadratic kernel: we would expect data which lies far away from training data to receive small kernel values from K x .
This, in turn, would modulate the entire kernel K c w for that data point to small values, leading to a weight model that reverts increasingly to the prior.
We would hope such a model would help with modeling uncertainty by resetting weights to uninformative distributions away from training data.
However, one may also want to use this mechanism to model inductive biases for the network, such as adding structure to the weight prior that can be captured with a kernel function.
This is a potentially appealing avenue, as multiple useful kernels have been found in the GP literature that allow modelers to describe relationships between data, but was not a tool that was previously accessible to neural network modelers.
We consider this a novel form of functional regularization, as the entire network can be given structure that will constrain its function space.
Unfortunately, we cannot easily apply this method directly to high dimensional inputs as commonly used kernel functions fail in this setting.
To overcome this limitation, we introduce { (cid:15)(cid:15)(cid:15) n } Nn =1 , one for each training instance, which can be then used to as an extra input dimension for the meta mapping to generate weights instead of the actual inputs.
In detail, each auxiliary input is obtained via a (potentially nonlinear) transformation applied to an input: (cid:15)(cid:15)(cid:15) n = g ( Vx n ) , where V ∈ R D e × D x , and D e and D x are the dimensionality of (cid:15)(cid:15)(cid:15) n and x n , respectively, and g ( · ) is an arbitrary transformation.
We may also layer these transformations in general.
We typically set D e (cid:28) D x so this transformation could be thought of as a dimensionality reduction operation.
For low dimensional inputs, we can directly set (cid:15)(cid:15)(cid:15) n = x n .
The auxiliary variable is then augmented to the input of the meta mapping as follows,
[Equation]
that is, the input dimension of the meta mapping is now 2 D z + D e .
The covariance matrices can be decomposed and computed efﬁciently as Kronecker products K c w = K out ⊗ K in ⊗ K x , as K x maps over data ( N × N ) instead of over weights ( W × W ) like the previous kernels and so their product requires efﬁcient approximation to avoid calculating an object of size ( WN × WN ).
Additionally, we also place a prior over the linear transformation: p ( V ) =
N ( V ; 0 , I ) .
I NFERENCE AND LEARNING USING STOCHASTIC STRUCTURED VARIATIONAL INFERENCE
Performing inference is challenging due to the non-linearity of the neural network and the need to infer an entire latent function f .
To address these problems, we derive a structured variational inference scheme that makes use of innovations from inducing point GP approximation literature (Titsias, 2009;
Hensman et al., 2013; Quiñonero-Candela & Rasmussen, 2005; Matthews et al., 2016;
Bui et al., 2017) and previous work on inferring meta-representations (Karaletsos et al., 2018).
As a reminder, we write down the joint density of all variables in the model speciﬁed in section 3: p ( y , w , z , f, V | x ) = p ( z ) p ( V ) p ( f ) p ( w | f, z , V , x ) p
( y | w , x ) =
p ( z ) p ( V ) p ( f ) (cid:34) N (cid:89) n = p ( w n | f, C w ( z , x n , V )) p ( y n | w , x n ) (cid:35) .
We ﬁrst partition the space Z of inputs to the function f into a ﬁnite set of M variables called inducing inputs z u and the remaining inputs, Z = { z u , Z (cid:54) =
z u } .
The function f is partitioned identically, f = { u , f (cid:54) =
u } , where u
= f ( z u ) .
We can then rewrite the GP prior as follows, p ( f )
= p ( f (cid:54) =
u | u , z u ) p ( u | z u ) .
The inducing inputs and outputs, { z u , u } , will be used to parameterize the approximation.
In particular, a variational approximation is judiciously chosen to mirror the form of the joint density:
[Equation]
where the variational distribution over w is made to explicitly depend on remaining variables through the conditional prior, q ( z ) and q ( V ) are chosen to be diagonal (mean-ﬁeld) Gaussian densities, q ( z ) =
N ( z ; µµµ z , diag( σσσ 2 z )) and q ( V ) =
N ( V ; µµµ V , diag( σσσ 2 V )) , respectively, and q ( u ) is chosen to be a correlated multivariate Gaussian, q ( u ) =
N ( u ; µµµ u , Σ u ) .
This approximation allows convenient cancellations yielding a tractable variational lower bound as follows,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
where the last expectation has been partly approximated using simple Monte Carlo with the reparameterization trick, i.e. z k ∼ q ( z ) and V k ∼ q ( V ) .
We will next discuss how to approximate the expectation F k = (cid:82) w ,f q ( w , f | z k , V k , x ) log p ( y | w , x ) .
Note that we split f into f (cid:54) = u and u , and that we can integrate f (cid:54) =
u out exactly to give, q ( w | z k
[Equation]
At this point, we can either (i) sample u from q ( u ) , or (ii) integrate u out analytically.
We opt for the second approach, which gives
[Equation]
In contrast to GP regression and classiﬁcation in which the likelihood term is factorized point-wise w.r.t.
the parameters and thus their expectations only involve a low dimensional integral, we have to integrate out w in this case, which is of much higher dimensions.
When necessary or practical, we resort to Kronecker factored models or make an additional diagonal approximation as follows,
[Equation]
Whilst the diagonal approximation above might look poor from the ﬁrst glance, it is conditioned on a sample of the latent variables z k and thus the weights’ correlations are retained after integrating out z .
Such correlation is illustrated in ﬁg. 2 where we show the marginal and conditional covariance structures for the weights of a small neural network, separated into diagonal and full covariance models.
The diagonal approximation above has been observed to give pathological behaviours in the GP regression case (Bauer et al., 2016), but we did not observe these in practice.
A comparison of this diagonal approximation to alternatives is provided in the appendix.
F k is approximated by
F k ≈ (cid:82) w ˆ q ( w | z k , V k , x ) log p ( y | w , x ) which can be subsequently efﬁciently estimated using the local reparameterization trick (Kingma et al., 2015).
The ﬁnal lower bound is then optimized to obtain the variational parameters of q ( u ) , q ( z ) and q ( V ) , and estimates for the noise in the meta-GP model, the kernel hyper-parameters and the inducing inputs.
R ELATED WORK
Bayesian neural networks have recently garnered a resurgence of interests in the community.
There is, however, a long history of research on developing approximate Bayesian inference methods for these networks.
Notable work include extended Kalman ﬁltering (Singhal & Wu, 1989), Laplace’s approximation (MacKay, 1992), Hamiltonian Monte Carlo (Neal, 1993; 2012), variational inference (Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011; Blundell et al., 2015; Gal &
Ghahramani, 2016), and approximate expectation propagation (Hernández-Lobato & Adams, 2015;
Li et al., 2015; Hernández-Lobato et al., 2016).
The work in this paper is orthogonal to these as the model employs a hierarchical prior, and inference is done in a lower-dimensional latent space instead of the weight space.
The variational approximation is chosen such that the marginal distribution over the weights is non-Gaussian and the correlations between weights are retained, in contrast to the popular mean-ﬁeld Gaussian approximation.
Additionally, a standard multivariate Normal distribution is typically used as the prior over the network weights and biases.
Imposing additional structures over the weights a prior or carefully choosing the right prior variances have been observed to improve the predictive performance (Ghosh et al., 2018; Neal, 2012; Blundell et al., 2015).
In this vein, the proposed model in this paper imposes a class of ﬂexible structures over the network parameters and learn a posterior over them conditioned on observed data.
Another related theme is hyper-networks, the core idea of which is to generate network parameters using another network (see e.g. Ha et al., 2016; Stanley et al., 2009).
The proposed model could be thought of as a GP hyper-network, but utilizes latent variables for nodes as introduced in (Karaletsos et al., 2018), thus avoiding over-parametrized, monolithic models over weight tensors.
Instead, the model uses hierarchical modeling, conditional probabilities, and the unit nodes to perform one-dimensional regression to weights maintaining modeling correlations.
E XPERIMENTS
In this section, we evaluate the proposed model and inference scheme on several toy and benchmark datasets.
We compare the proposed approach primarily to maximum a posteriori estimation (MAP) and MFVI, both of which use a standard Normal prior over the weights.
These were implemented using PyTorch (Paszke et al., 2017) and the code will be available upon acceptance.
Additional results are included in the appendices.
We use M = 50 inducing points for all experiments in this section.
6. T OY REGRESSION AND CLASSIFICATION EXAMPLES
2 x404y    2 x404y
We ﬁrst illustrate the performance of the proposed model on two toy examples, one regression and one classiﬁcation.
In both cases, a Bayesian neural network with one hidden layer of 50 hidden units is used.
Additionally, the hierarchical latent variables have two dimensions.
For the regression case, we generate 100 data points from a toy function suggested by Louizos et al.
For classiﬁcation, a four-class dataset of 100 data points is used.
Figure 3 shows the predictive performance of the proposed models, MFVI and a meta-NN on both tasks.
Notably, the proposed model with contextual inputs (and thus input dependent network weights) gives uncertainty estimates that are reminiscent to that of a GP regression model, despite being a neural network under the hood.
Following Bradshaw et al.
, we also show the uncertainty for data further from the training instances.
The model with auxiliary inputs remains uncertain as expected for these points whilst MFVI produces arguably overconﬁdent predictions.
We also attempt to summarize the distribution of the weights in the network in appendix A.7 in the appendices.
6. I NPUT D EPENDENT N EURAL N ETWORKS F OR U NCERTAINTY Q UANTIFICATION
Motivated by the performance of the proposed metaGP model with auxiliary inputs in the toy examples in ﬁg. 3, this section tests the ability of this model class to produce calibrated predictive uncertainty to out-of-distribution samples.
That is, for test samples that do not come from the same training distribution, a robust and well-calibrated model should produce uncertain predictive distribution and thus high predictive entropy.
Such a model could ﬁnd applications in safety-critical tasks where the cost of making conﬁdent but wrong predictions is high, or in area where detecting unfamiliar inputs is crucial such as active learning or reinforecement learning.
In this work, we ﬁrst train a neural network classiﬁer with one hidden layer of 100 rectiﬁed linear units on the MNIST dataset, and apply the metaGP prior only to the last layer of the network.
The dimensions of the latent variables and the auxiliary inputs are both 2.
After training, we compute the entropy of the predictions on various test sets, including notMNIST, fashionMNIST, Kuzushiji-MNIST, and uniform and Gaussian noise inputs.
Following (Lakshminarayanan et al., 2017; Louizos & Welling, 2017), the CDFs of the predictive entropies for various methods are shown in ﬁg. 4.
A calibrated classiﬁer should give a CDF that bends towards the top-left corner of the plot for in-distribution examples and, vice versa, towards the bottom-right corner of the plot for out-of-distribution inputs.
In most out-ofdistribution sets considered, except Gaussian random noise, metaGP and metaGP with local auxiliary variables demonstrate competitive performance to Gaussian MFVI.
Notably, MAP estimation, often deployed in practice, tends to give wildly poor uncertainty estimates on out-of-distribution samples.
We illustrate this behaviour and that of other models and methods on representative inputs of the
MNIST and Kuzushiji-MNIST datasets in ﬁg. 4.
6. M ODELING I NDUCTIVE B IASES F OR N EURAL N ETWORKS W ITH I NPUT -D EPENDENT K ERNELS
We further explore the utility of the contextual variable towards modeling inductive biases for neural networks and evaluate on predictive performance on a regression example.
In particular, we generate 100 training points from a synthetic sinusoidal function and create two test sets that contains in-sample inputs and out-of-sample inputs, respectively.
We test an array of models and inference methods, including BNN with MFVI, metaGP and metaGP with contextual variables.
In particular, as noted in section 3.1 we can choose the covariance function to be used for the auxiliary variables to encode our belief about how the weights should be modulated by the input.
We pick exponentiated quadratic and periodic kernels (MacKay, 1998) in this example.
The predictive performance are measured by the root mean squared error (RMSE) and the log-likelihood (LL) on the test examples.
Figuresummarizes the results and illustrate the qualitative difference between models.
It could be noted that the periodic kernel allows the model to discover and encode periodicity, allowing for more long-range conﬁdent predictions compared to that of the exponentiated quadratic kernel.
In this example, MetaGP with contextual variables is superior to other methods, demonstrating good RMSE and LL on both in-distribution and out-of-distribution examples.
6. A CTIVE LEARNING
We next stress-test the performance of the proposed model in a pool-based active learning setting for real-valued regression, where limited training data is provided initially and the target is to sequentially select points from a pool set to add to the training set.
The criterion to select the next best point from
[Figure]
the pool set is based on the entropy of the predictive distribution,
i
[Figure]
e. we pick one with the highest entropy.
In practice, we approximate the predictive density by a Gaussian density, which results in a tractable entropy computation.
Note that this selection procedure can be interpreted as selecting points that maximally reduce the posterior entropy of the network parameters (Houlsby et al., 2011).
Four UCI regression datasets were considered, where each with 40 random train/test/pool splits.
For each split, the initial train set has 20 data points, the test set has 100 data points, and the remaining points are used for the pool set, similar to the active learning set-up in Hernández-Lobato & Adams (2015).
We compare the performance of the proposed model and inference scheme to that of Gaussian mean-ﬁeld variational inference and show the average results in ﬁg. 6.
Across all runs, we observe that active learning is superior to random selection and more crucially using the proposed model and inference scheme seems to yield comparable or better predictive errors with a similar number of queries.
We also illustrate the procedure on a toy classiﬁcation dataset and visualize the predictive uncertainty as more points are picked and used for training in ﬁg. 17 in the appendix.
This simple setting quantitatively reveals the inferior performance of MFVI, compared to MAP and metaGP.
S UMMARY
We have developed a ﬂexible hierarchical neural network model and a structured variational inference scheme that together (i) enable ﬂexible modelling of the correlations between network parameters, (ii) allow weight structures to be retained in the approximate posterior without having to perform direct inference in the high-dimensional weight space, and (iii) allow input-dependent contextual variables to be used to generate private weights per data point which translates to more desirable uncertainty estimates in practice and the ability to generalize out of sample with inductive biases such as periodicity.
We plan to evaluate the performance of the model on more challenging decision making tasks such as contextual bandits or reinforcement learning, and to extend the inference scheme to handle continual learning.
A E XTRA EXPERIMENTAL RESULTS A. A N EMPIRICAL EVALUATION OF VARIOUS APPROXIMATIONS FOR q ( w | z k , V k , x )
In this experiment, we analyze the impact of different approximations to the covariance matrix of q ( w | z k , V k , x ) :
[Equation]
If we use the exact, fully correlated Gaussian distribution above, it is necessary to sample from this distribution to evaluate the lower bound.
This step costs O ( W 3 ) where W is the number of parameters in the network.
The complexity can be greatly improved by making a diagonal approximation to B ( k ) as follows,
[Equation]
Sampling from this distribution can be done in O ( WM 2 ) where M is the number of pseudo-points.
This can be further approximated by assuming a diagonal covariance matrix,
[Equation]
The variational bound can then be evaluated by drawing samples from ˆ q diag as in the above approximation, or by drawing activity samples by employing the local reparameterization trick (Kingma et al., 2015).
We evaluate the performance of using the exact and approximate conditional distributions above in a range of toy regression and classiﬁcation, and show representative results in ﬁgs.
7 and 8.
We note that the diagonal approximation is fast and gives qualitatively similar performance compared to more structured approximation or the exact case, in both cases where there is a singla GP for all weights in the network and there is multiple GPs, one for each weight layer in the network.
A. R OBUSTNESS IN VARIOUS DATA REGIMES FOR A TOY REGRESSION PROBLEM
In this experiment, we evaluate the qualitative performance of various methods, including MFVI, NN-Metarep, GP-Metarep and GP-Metarep with local, input-dependent kernel, on a toy regression problem, in different data regimes.
In particular, we considers 10, 20, and 50 training points respectively, and plot the predictions in ﬁg. 9.
GP-metarep demonstrates consistent performance across all data regimes, comparable to that of NN-metarep.
The input-dependent kernel helps the performance further in the out-of-distribution area.
A. E FFECT OF INPUT - DEPENDENT KERNEL AND APPLICATIONS TO FORWARD TRANSFER / FAST ADAPTATION / MULTI - TASK LEARNING
In this section, we investigate the effect of the input-dependent kernel by probing the learnt kernel hyper- parameters.
In particular, we ﬁrst train a model with an periodic input-dependent kernel on a sinusoidal 1D data set, as show in the main text.
We then vary the period hyper-parameter in the kernel whilst keeping other hyper- parameters and variational parameters ﬁxed.
The predictions for a few hyperparameters are shown in ﬁg. 10.
We note the variation/period in the data is captured by weight modulation, governed by the input-dependent kernel.
Changing the period hyperparameter affects how fast or slow the weights are changing wrt the input.
We further investigate using the proposed model for multi-task learning.
In particular, the latent variable z and corresponding hyper-parameters and variational parameters can be shared across different tasks whilst the meta mapping and the input-dependent kernel are private to each individual task.
We ﬁrst train the model on four regression tasks, each corresponds to a sinusoid of a particular frequency.
At test time, a novel test set is shown to the model.
The hyper-parameters of the input kernel and variational parameters corresponding to this new test set are optimised while other hyper-parameters and the latent variables are kept ﬁxed.
We evaluate the performance of the model on the novel test sets to see how the latent variables can be reused and shared across tasks to facilitate fast adaptation to new settings.
The performance of the model on the tasks used for training and new tasks at test time is shown in ﬁg. 11.
This result demonstrates the ability of the model trained with multiple similarly related tasks to faithfully and quickly adapt to new settings.
A. MNIST EXPERIMENT : A COMPARISON TO DEEP KERNEL LEARNING AND FULL FIGURES
In this section, we include the full ﬁgures of the MNIST out-of-distribution uncertainty experiment, as well as additional results using deep kernel learning (Wilson et al., 2016). In particular, we employ the same network architecture with the last layer being replaced by multiple independent GPs, one for each class (output dimension). As exact inference is intractable, variational inference based on inducing points is employed – we used 50 inducing points for each output.
The full results of all models/methods considered are shown in ﬁg. 12.
For clarify, the results of deep kernel learning and GP-metarep are shown in ﬁg. 13.
GP-metarep with the input-dependent kernel shows good performance, outperforming deep kernel learning in all cases.
In addition, we include the full ﬁgures for the predictive distributions on representative test examples in ﬁgs.
14 and 15.
[Figure]
A. R OBUSTNESS OF GP- METAREP WITH NETWORK ARCHITECTURES
In this experiment, we compare the performance of GP-metarep for various numbers of hidden units (20, 50 and 100) and two activation functions (tanh and ReLU) on a toy regression problem.
The observation noise is ﬁxed in this experiment.
We observe that the performance of the models is in general consistent across different activation functions and numbers of hidden units.
We show the results in ﬁg. 16.
A. A TOY ACTIVE LEARNING SETTING
Please see ﬁg. 17 and the associated caption.
A. P ROBABILITY DENSITIES OF WEIGHTS IN THE TOY 1 D REGRESSION EXPERIMENT
Please see ﬁgs.
18 and 19 and the associated captions.
[Figure]
d a t a p o i n t s d a t a p o i n t s d a t a p o i n t s
M F V I
N N - M e t a r e p G PM e t a r e p
G PM e t a r e p + l o c a l
F i gu r e: P e rf o r m a n ce o f m ea n - ﬁ e l d v a r i a ti on a l i n f e r e n ce ,
NNM e t a r e p w it h v a r i a ti on a l i n f e r e n ce a nd
G PM e t a r e p w it h v a r i a ti on a l i n f e r e n ce on a t oy r e g r e ss i on p r ob l e m w it h v a r i ou s nu m b e r o f t r a i n i ng
po i n t s .
B e s t
v i e w e d i n c o l ou r .
T = period period = 3*T period =
T /
[Table]
F i gu r e 11 :
T r a i n i ng on m u lti p l e r e l a t e d t a s k s a nd a d a p t a ti on t o nov e l t a s k s a t t
e
s t ti m e .
I n t h i s ca s e , t h e l a t e n t v a r i a b l e s ( a s w e ll a s w e i gh t c od e hyp e r p a r a m e t e r s ) a r e s h a r e d ac r o ss t
a s k s w h il e eac h
i nd i
v i du a l h
a
s it s o w n i npu t - d e p
e
nd e n t k e r n e l .
A t t e s t ti m e , on l y t
h e p r
i v a t e p a r a m e t e r s f o r t
h e n e w t a s k a r
e
r
e - i n iti a li s e d a nd op ti
m i s e d .
B e s t
v i e w e d i n c o l ou r .
[Figure]
nat
[Figure] MAP MFVI metaGP metaGP + local dkl mnist kmnist fmnist uniform gauss [Figure]
mnist kmnist fmnist uniform gauss
M A P 0.
M F V
I 0.
m e t a G P 0.
m e t a G P + l o c a l 9 0.
{+d k l 
+}
M A P 0.
M F V
I 0.
m e t a G P 0.
m e t a G P + l o c a l 9 0.
{+d k l 
+}
[Figure]
F i gu r e 18 :
L e f t:
T r a i n i ng d a t a a nd p r e d i c ti on s u s i ng a
B a y e s i a n n e u r a l n e t w o r k w it
h
a m ea n - ﬁ e l d v a r i a ti on a l G a u ss i a n a pp r ox i m a ti on .
T h
e n e t w o r k h
a s on e h i dd e n l a y e r o f 50 r ec ti ﬁ e d li n ea r un it s , h e n ce t
h e r e a r e
i npu t
w e i gh t s a nd ou t pu t w e i gh t s .
R i gh t: t
h e m a r g i n a l v a r i a ti on a l a pp r ox i m a ti on s ov e r t
h e w e
i
gh t s a f t e r t r a i n i ng .
M o s t
h i dd e n un it s a r e p r un e d ou t a f t e r t r a i n i ng , a s ob s e r v e d i n
T r i pp e & T u r n e r
I n t h
i s ca s e , t
h e r e a r e ac ti v e h
i dd e n un it s .
W e i n c l ud e t h e h
i s t og r a m o f t
h e w e
i
gh t s i n eac h l a y e r i n ﬁ g .
F i gu r e 19 :
M i dd l e :
T r a i n i ng d a t a a nd p r e d i c ti on s u s i ng a
B a y e s i a n n e u r a l n e t w o r k w it h a n i npu t - d e p e nd e n t m e t a
G P p r i o r .
S i m il a r t o ﬁ g .
18 , t h e n e t w o r k h
a s on e h i dd e n l a y e
r
o f un it s .
T h
e d i s t
r
i bu ti on s o f t
h e w e
i gh t
s i n t h e n e t w o r k ,
g i v e n a n i npu t , a r e i n c l ud e d .
N o t e t h a t , f o r eac
h
w e i gh t , w e ﬁ r s t s a m p l e t h e l a t e n t v a r i a b l e z , a nd t
h e n p r op a g a t e t
h
i s s a m p l e t h r ough t
h e m e t a
G P m a
pp
i ng t o ob t a i n a d
i s t
r
i bu ti on o f eac h
w e i gh t .
I n eac h s ubp l o t , w e p l
o t d i s t
r
i bu ti on s c o
rr e s pond
i ng t
o d i ff e r e n t z s a m p l e s .
O n t h e l e f t ﬁ gu r e s , a s p a r t o f t h e i npu
t t o t h e
G P
i s f a r a w a y fr o m t
h e t r a i n i ng i npu t s , t h e w e
i
gh t s b ec o m e un ce r t a
i n w h
i c h l ea d s t o h
i gh un ce r t a i n t
y i n t
h e p r e d i c ti on .
O n t h e c on t r a r y , a r ound t h e t r a i n i ng po i n t s , t h e
G P ou t pu t s a r e m o r e ce r t a i n a nd m o r e d i v e r s e .
N o t e t h a t , t
h
e m a r
g
i n a l o f eac h
w e i
gh t
i s a n i n ﬁ n it e m i x t u r e o f G a u ss i a n d e n s iti e s .
B e s t
v i e w e d i n c o l ou r .