Sparse Gaussian Process Hyperparameters: Optimize or Integrate?
Abstract
The kernel function and its hyperparameters are the central model selection choice in a Gaussian process [Rasmussen and Williams, 2006].
Typically, the hyperparameters of the kernel are chosen by maximising the marginal likelihood, an approach known as Type-II maximum likelihood (ML-II).
However, ML-II does not account for hyperparameter uncertainty, and it is well-known that this can lead to severely biased estimates and an underestimation of predictive uncertainty.
While there are several works which employ a fully Bayesian characterisation of GPs, relatively few propose such approaches for the sparse GPs paradigm.
In this work we propose an algorithm for sparse Gaussian process regression which leverages MCMC to sample from the hyperparameter posterior within the variational inducing point framework of [Titsias, 2009].
This work is closely related to Hensman et al.
[2015b], but side-steps the need to sample the inducing points, thereby significantly improving sampling efficiency in the Gaussian likelihood case.
We compare this scheme against natural baselines in literature along with stochastic variational GPs (SVGPs) along with an extensive computational analysis.
Introduction
Gaussian processes (GPs) are a prominent class of models for supervised learning which can quantify uncertainty and incorporate inductive biases in function space via the kernel function.
Hand-crafting a kernel function is a powerful way to incorporate prior knowledge.
In many instances not all properties of a kernel function can be specified from prior knowledge alone, and parameters are chosen via MLII.
However, defining a complex kernel function with a large number of hyperparameters can make the marginal likelihood prone to multiple local optima and overfitting.
Further, several local optima may correspond to priors that do not sensibly model the data.
Weakly identified hyperparameters can manifest in flat ridges in the marginal likelihood surface 1 making gradient based optimisation extremely sensitive to starting values [Warnes and Ripley, 1987].
Overall, the ML-II point estimates for the hyperparameters are subject to high variability and underestimate prediction uncertainty.
The problem of ridges in the marginal likelihood surface also does not necessarily go away as more observations are collected.
For example, if f 1 and f 2 are Brownian motions, σf 1 ( x/ℓ ) is equal in distribution to √ ασf 2 ( x/αℓ ) , which means observations do not provide any information about the product σℓ .
More generally, for a greater class of kernels, including the Mat´ern– 1 / 2 kernel, σf 1 ( x/ℓ ) is equivalent to √ ασf 2 ( x/αℓ ) , which means that is not possible to consistently estimate the product σℓ from data, no matter how many observations are collected in a fixed domain [Chapter 6, Stein, 1999].
This implies that one cannot estimate the individual hyperparameters ( σ, ℓ ) consistently.
It also motivates why there can be benefits to estimating the hyperparameter posterior even in large data regimes, and ML-II may be insufficient.
A more satisfactory treatment of hyperparameters involves placing a prior over the hyperparameters and performing Bayesian inference to compute a (hyper)posterior.
For large datasets, this motivates using scalable GP inference (e.g. sparse methods) in conjunction with Markov chain Monte Carlo (MCMC) for the hyperparametrs.
The Bayesian treatment of weakly identified hyperparameters may also be fraught with difficulties.
Gradient-based samplers like Hamiltonian Monte Carlo [Neal et al., 2011] and its variants have difficulty navigating regions of high curvature and flat ridges where the gradient offers no information for transition [Betancourt, 2017].
This leads to over-concentration of samples from the flat region. (Usually, this can at least partially be rectified with informative priors.)
These pathologies are also typical of other hierarchical models [Betancourt and Girolami, 2015].
Figure 1 shows that the GP marginal likelihood surface can manifest these pathologies.
The evidence lower bound (ELBO) used in hyperparameter selection for variational sparse GPs relying on inducing points inherits similar, or even less favourable, characteristics to the exact marginal likelihood.
As a result, for weakly informative priors, gradient based samplers are susceptible to getting stuck at the boundary of these pathological regions hence biasing the sample estimates.
The effective sample size metric, used for diagnosing mixing in MCMC, is indicative of this behaviour when directly observing the phase space of the target distribution, but is infeasible in high-dimensions.
Historical justification for ML-II (also called the evidence framework) comes from [MacKay, 1994] which highlighted several conditions for ML-II to yield reasonable estimates for the hyperparameters.
Crucially, the evidence is unlikely to manifest multiple local optima for a model well-matched to the data and with a high signal-to-noise ratio.
Transferring this insight to the Gaussian process paradigm we show how the evidence can have a significant tail or no well-defined maximum in settings with a low signal-to-noise ratio which arises with high aleatoric uncertainty or sparse data, frequently both.
In these settings, a point estimate may not adequately summarise the hyperparameter posterior and the benefits of marginalisation stand out.
The situation is only exacerbated in high-dimensions (i.e. when there are many hyperparameters) where increasingly more volume of the posterior is captured in a thin shell making the density peak extremely unrepresentative of the posterior.
Unlike the likelihood of parameters, the marginal likelihood inherently contains a trade-off between the data-fit and complexity penalty term.
This is one of the main properties that makes the marginal likelihood objective a viable choice for model selection [Rasmussen and Williams, 2006].
For example, for the Gaussian process regression model,
[Equation]
the marginal likelihood takes the form,
[Equation]
[Table]
[Equation]
[Table]
penalty (cid:122) (cid:125)(cid:124)
[Equation]
where c is a constant, p ( y | f ) denotes the data likelihood and θ denotes kernel hyperparameters.
This trade-off is a well-established idea which embodies the automatic Occam’s razor effect [Rasmussen and Ghahramani, 2001] where models well-suited to the data are automatically selected just by using the marginal likelihood objective.
This may seem to contradict earlier claims regarding overfitting, but by shying away from dealing with the hyperparameter posterior we risk overfitting even with the marginal likelihood approach.
In other words, the evidence framework subdues the overfitting effect induced by the canonical maximum likelihood approach which does not have any complexity penalty term.
The parameters in the canonical approach are free to fit the data as well as possible making them prone to overfitting and poor generalisation.
Overparameterized kernels based on neural networks like deep kernel models [Wilson et al., 2016] are well known to exacerbate overfitting [Ober et al., 2021].
The main motivation for this work is to highlight that fully Bayesian schemes in sparse Gaussian process models are practically beneficial.
While several works in the literature employ fully Bayesian scheme of integrating out the hyperparameters (see table 1), in this work we attempt to analyse them in an orthogonal direction, focusing on comparison with the evidence framework and other benchmarks by extending the main sparse variational formulation in the literature Titsias [2009].
We present a generalised inference scheme for fully Bayesian GP regression and counteract some of the computational cost of sampling both inducing variables and hyperparameters by deriving a doubly collapsed bound which selects the optimal distribution over the inducing points analytically and targets the kernel hyperparameters with HMC.
Related Work
[Table]
Fully Bayesian Gaussian processes have been used by several authors spawning several variants.
In early accounts, Neal [1998], Williams and Rasmussen [1996] explore the integration over covariance hyperparameters using HMC in the regression setting.
Barber and Williams [1997] extend this to the classification setting using HMC for sampling in the hyperparameter space and Laplace approximation for the integrand over function values.
Murray and Adams [2010a] and Filippone et al.
[2013] focused on MCMC schemes to sample covariance hyperparameters in conjunction with latent function values, mainly mitigating the coupling effect through reparameterisation.
Hensman et al.
[2015b] considered joint sampling of inducing variables and hyperparameters from the optimal variational posterior distribution while [Bui et al., 2018] consider inference schemes for fully Bayesian sparse GPs in a streaming setting.
More recently, Rossi et al.
[2021] studied fully Bayesian sparse GPs using
SG-HMC.
Rossi et al.
[2021] modify the generative model by adding a prior over the inducing inputs, and peform inference using SG-HMC over the joint ( Z , u , θ ) space.
We list the most recent works in Table 1.
Background
Let f ∼ GP (0 , k θ ) be a Gaussian process prior with kernel function k θ depending on hyperparameters θ .
We are given noisy observations y
=
( y n ) Nn =1 ⊆ R of f = ( f ( x n ))
Nn =1 at input data X = ( x n ) Nn =1 ⊆ R D .
We consider a Gaussian likelihood which factorises over the data, p ( y | f ) = (cid:81)
Nn =1 N ( y n | f n , σ 2 ) .
We wish to compute the posterior p ( f | y , θ ) .
In this section, we recapitulate the canonical inducing variable approximation of p ( f | y , θ ) by Titsias [2009] and its extension to a Bayesian treatment of the hyperparameters.
3.1 Sparse variational inference in Gaussian processes
Following Titsias [2009], we consider a set of inducing variables u
= { f ( z m ) } Mm =1 ⊆ R at inducing inputs Z = { z m } Mm =1 , z m ∈ R d .
The complete generative model can then be factored as,
[Equation]
We approximate the posterior p ( f, u | y , θ ) with a variational distribution:
[Equation]
where q ( u ) is chosen to minimise the Kullback–Leibler divergence KL( q ( f | θ ) ∥ p ( f | y , θ )) .
Minimising this KL divergence corresponds to maximising the evidence lower bound [Matthews et al., 2016], henceforth called the ELBO:
[Equation]
Because the ELBO still depends on q ( u ) , this bound is called uncollapsed .
Hensman et al.
[2013, 2015a] let q ( u ) be a Gaussian, which is optimal if the likelihood is Gaussian [Titsias, 2009], approximate the expectation using Monte Carlo, and maximise the ELBO using stochastic optimisation.
On the other hand, if the likelihood is Gaussian, Titsias [2009] computes the optimal form for q ( u ) directly:
[Equation]
Plugging q ∗ ( u | θ ) back into equation (5), the resulting bound is called collapsed , because it now only depends on θ and Z .
The collapsed bound, denoted L θ ,Z , is the objective that Titsias [2009] proposes:
[Equation]
where K nn is the prior covariance matrix of f , K mm is the prior covariance matrix over u and K nm is crosss-covariance matrix formed by f and u .
Using the collapsed bound, approximate ML-II consists of finding,
[Equation]
Predictions at new functions values can be made in O ( M 2 ) after an initial cost of O ( NM 2 ) .
Under certain assumptions on the data generating process, even when M ≪ N , the approximate posterior closely resembles the posterior, and equation (7) is a provably accurate approximation to equation (2) [Burt et al., 2020].
3.2 Bayesian treatment of hyperparameters and sparse methods
The extension of the sparse variational framework to a Bayesian treatment of the hyperparameters has been previously considered by Hensman et al.
[2015b].
Extend the generative model with a prior p ( θ ) over the hyperparameters θ , and let the variational approximation of the posterior p ( f, u , θ | y ) be q ( f, u , θ )
= p ( f | u , θ ) q ( u , θ ) .
The analogue of equation (5) is
[Equation]
and the optimal form for q ( u , θ ) can again be determined:
[Equation]
The distribution q ∗ ( u , θ ) does not have a closed form, and for general likelihoods, Hensman et al. [2015b] propose to approximate the expectation in (10) with quadrature and to sample from q ∗ ( u , θ ) using HMC.
While this approach is quite general, in the case of Gaussian regression, it vastly increases the dimensionality of the state space over which HMC must be run relative to HMC in GPR, since the u are sampled in addition to the θ .
This increases the cost of the procedure, and impacts the success of the sampler.
An alternative approach to approximately inferring hyperparameters is to assume a parametric form for q ( u , θ ) and maximise equation (9) with respect to the variational parameters.
Bui et al.
[Table] took such an approach, assuming that q ( u , θ ) =
q ( u ) q ( θ ) , with both distributions Gaussian.
Similar approaches have been applied to variational inference in state-space modelling, sometimes leveraging the optimal form of q ( u | θ ) discussed earlier
[Table]
The QUALITY column in Table 2 indicates the ability of the method to faithfully represent the hyperparameter posterior.
If VI is run to convergence, a potentially significant amount of error will be incurred by the Gaussian approximation to the non-Gaussian posterior over the hyperparameters (red).
On the opposite extreme, if no sparsity assumption is made, MCMC over the hyperparameters without sparse approximations is asymptotically consistent (green).
The inducing point approximations combined with MCMC lie somewhere in-between these methods yellow).
3.3 Making predictions
The predictive posterior distribution for unknown test inputs X ∗ integrates over the joint posterior,
[Equation]
where we have suppressed the conditioning over inputs X, X ∗ for brevity.
The inner integral simplifies to (cid:82) p ( f ∗ | f , u , θ ) p ( f | u , θ ) d f = p ( f ∗ | u , θ ) .
We discuss the predictive posterior in such models in section 4.3.
Fully Bayesian SGPR with HMC: Doubly collapsed formulation
In the previous section, we observed that a major drawback of the approach taken in Hensman et al. [2015b] is the need to sample u , which for high-dimensional inputs or in cases where many inducing points are needed could introduce thousands of additional variables to sample.
In this section, we leverage the optimal form of q ( u | θ ) derived in Titsias [2009] to alleviate this sampling problem.
4.1 Collapsing the evidence lower bound (again)
We first derive the lower bound for this formulation and provide pseudo-code for the algorithm in
Algorithm 1.
Following the usual derivation of the ELBO,
[Equation]
[Equation] where log p ( y | θ ) ≥ L θ ,Z with L θ ,Z defined in equation (7), and where we assign M θ ,Z = e L θ ,Z .
4.1.1 Deriving q ∗ ( θ )
We can interpret L ∗ Z ( q ( θ )) as a negative KL divergence as long as we account for a normalisation constant C Z = (cid:82) M θ ,Z p ( θ ) d θ for the un-normalised numerator M θ ,Z p ( θ ) .
Hence, we can re-write L ∗ Z ( q ( θ )) as,
[Equation]
where q ∗ ( θ ) =
M θ ,Z p ( θ ) /C Z .
By inspecting equation (14), we observe that the optimal variational distribution over θ is given by q ∗ ( θ ) .
2 Crucially, by sampling directly from q ∗ ( θ ) using MCMC we eliminate the need to sample the variables u .
By evaluating L ∗ Z at q ∗ ( θ ) , we find the doubly collapsed ELBO L ∗∗ Z := L ∗ Z ( q ∗ ( θ )) =
log C Z . Although the value of L ∗∗ Z is computationally intractable, given samples ( θ j ) Jj =1 from q ∗ ( θ ) , gradients of L ∗∗ Z with respect to Z can be estimated using the stochastic estimate of the canonical ELBO equation (7): using the chain rule,
[Equation]
where δδq L ∗ Z ( q ) is the functional derivative of L ∗ Z ( q ) with respect to q , which is zero at q = q ∗ ( θ ) , because q ∗ optimises L ∗ Z (it is a critical point, so the derivative is zero).
Further, the partial derivative of L ∗ Z with respect to Z concerns just the first term of the LHS of equation (13) as the KL term
KL( q ( θ ) ∥ p ( θ )) is independent of Z .
4.2 Performing approximate inference
We deploy HMC to (approximately) sample from the optimal variational posterior q ∗ ( θ ) along with optimising the inducing inputs Z in a hybrid scheme.
We alternate between the two steps allocating longer intervals for optimising Z for every HMC sampling run for the hyperparameters.
We note that this hybrid scheme is much more computationally efficient than sampling u and θ jointly where one has to tackle the coupling between inducing variables and hyperparameters in joint space.
Further, joint sampling is only feasible for moderate number of inducing variables while this scheme can scale to much larger datasets as the efficiency of sampling in the hyperparameter space is only dependent on the dimensionality of the hyperparameter space rather than the number of inducing variables.
The entire inference scheme is summarized in Algorithm 1.
The warm-start strategy (of optimizing both ( Z, θ ) jointly for a few gradient steps) is used to find a good region for the sampler to initialise θ .
1: Input: ELBO objective L θ,
Z :=
L ( θ , Z ) (equation (7)), gradient based optimiser optim() 2: 3: procedure W ARM -S TART 4: for fixed number of steps do 5:
Gradient step: Z, θ ←− optim ( L ( θ , Z ) ) 6: return initial values Z init and θ init 7:8: procedure T RAIN 9: ## Initialisation protocol 10: ▶ Initialise L θ,Z at warm-start values L ( θ init , Z init ) , lets call this ˆ L 11: ▶ Freeze kernel hyperparameters in the ELBO objective by setting requires grad=
False .
12:13: while not converged do 14: for t = 1 . . .
T do ## start of training loop 15:16: • Gradient step: Z opt ←− optim (
L ) (# equation (15) shows the validity of taking the derivative of the stochastic ELBO) 17: if t mod L == 0 then 18: ( gradient steps) 19: 20:
21: log q ∗ ( θ ) ∝ L ( θ , Z opt ) + log p ( θ ) keeping Z opt fixed.
( θ j , p j ) HMC ←−−− H ( θ , p ) , (where H is the Hamiltonian) 22: where p denotes the zero-mean auxilliary momentum variable in phase space with 23: the same dimensionality as θ .
( θ j , Z
) , where θ q ∗ ( θ ) 25: return
4.3 Predictive posterior
It is ultimately the posterior predictive (PP) distribution that is of interest rather than point estimates or the hyperparameter posterior.
The Bayesian sparse GP predictive posterior entails integrating out the posterior over inducing variables u and hyperparameters θ .
Once we have performed inference, we can approximate this directly, p ( f
[Equation]
where q ∗ ( u | θ , y ) =
N ( m ∗ , S ∗ ) is the optimal Gaussian variational distribution and
A =
K ∗ m K − 1 mm under the SGPR scheme (Section 4) is available in closed form with m ∗ =
σ − 2 ( K mm + σ − 2 K mn K nm ) and S ∗ =
K mm ( K mm + σ − 2 K mn K nm ) − 1 K mm .
In either case, the internal integral with respect to the inducing variables is analytic and the outer integral can be estimated using the samples collected from HMC to perform Monte Carlo estimation.
This yields a mixture of
Gaussians,
[Equation] [Equation]
where J samples are (approximately) drawn from q ∗ ( θ ) via HMC.
The distribution inside the summation is the Gaussian posterior predictive distribution for fixed hyperparameters with identical mixing proportions.
The compute cost for the predictive posterior scales the sparse GPR cost linearly in the number of samples.
The Monte Carlo approximation costs O ( JNM 2 ) for M inducing points and J hyperparameter samples.
Experiments
In the previous section, we discussed a hybrid scheme which leverages MCMC within the variational sparse inducing variable formulation leading to fully Bayesian sparse Gaussian processes.
In the experiments we demonstrate the feasibility of this scheme relative to several benchmarks and assess regression performance on a 1-dimensional illustrative example and a range of other datasets.
We also compare with exact GPs where the cost of using a gradient based sampler is prohibitive for even moderately large datasets requiring several inversions of the full covariance matrix.
We show that using the doubly collapsed scheme proposed in this work is a much more attractive alternative for large datasets as compared to direct HMC in GPR, and sampling is more efficient than in the uncollapsed bound used in Hensman et al.
[2015b].
We henceforth refer to benchmark methods as follows: SGPR + HMC refers to Bayesian GPs with doubly collapsed variational inference with NUTS, as described in section 3 (compatible with Gaussian likelihoods); we benchmark this model against sparse GPs ( SGPR )
[Titsias, 2009] and Stochastic Variational GPs ( SVGP ) both using approximate ML-II [Hensman et al., 2015a].
Additionally, we consider the sparse, joint sampling inference scheme proposed in [Hensman et al., 2015b] which gives a natural benchmark.
We call this model JointHMC .
The FBGP method extends the Bayesian treatment to the inducing locations similar to Rossi et al.
[2021]; we use NUTS to sample from the joint posterior over ( Z, θ ) .
It is not a direct comparison to Rossi et al.
[2021] as the latter explores free-form sampling of u along with ( Z, θ ) while we work with the collapsed bound incorporating the optimal Gaussian variational distribution q ∗ ( u ) .
We also present analysis where we fix inducing point locations at a random subset of the training data (as opposed to interleaving as per Algorithm 1) and only learn hyperparameters using NUTS.
We provide several details about the experimental set-up in the supplementary.
5.1 One dimensional synthetic data
[Table]
We sample noisy observations from f ( x ) = sin(3 x ) + 0 .
cos( πx ) with the constraint ( x < − 2) and ( x > 2) .
Figure 2 shows the results for
SGPR along with the fully Bayesian schemes.
We keep data split and noise identical across the three models to facilitate a comparison.
While there is significant data to identify the hyperparameters we notice that the models mainly differ in their extrapolation abilities away from the training data.
SGPR with ML-II overfits to the training data and recovers a low lengthscale, low noise solution while the SGPR + HMC scheme recovers a more moderate fit and performs significantly better in terms of RMSE and NLPD on unseen data Table 3.
We note that the JointHMC scheme which samples both ( u , θ ) overfits in the central missing data region.
We use M =inducing locations across all methods which are optimised according to the protocol of each method and recover a very similar spatial distribution. [Table]
5.2 UCI regression benchmarks
We compare our approach across methods on 5 standard small to medium-sized UCI benchmark datasets.
Following common practice, we use a 20 % randomly selected held out test-set
[Rossi et al., 2021, Havasi et al., 2018] and scale the inputs and outputs to zero mean and unit standard deviation within the training set (we restore the output scaling for evaluation) [Salimbeni and Deisenroth, 2017].
While we could use any kernel, we choose the RBF-ARD kernel with a lengthscale for each dimension.
For consistency we initialise all the inducing locations ( Z ) identically across the methods, i.e. by using the same random subset of training data split.
We note that adapting the inducing locations brings serious gains in prediction performance versus keeping them fixed (Figure 3).
Further, the
JointHMC scheme underperforms SGPR (with ML-II) and SGPR + HMC.
This is not surprising given that the JointHMC bound equation (10) does not incorporate the optimal setting for q ( u ) and was originally motivated by the need for a fully Bayesian scheme for generalised likelihoods.
The method SGPR + HMC significantly improves upon JointHMC in the specific Gaussian likelihood case.
Sensitivity to M : We benchmark SGPR+HMC and JointHMC on the Elevator dataset ( N = 16599 , D = 18 ) which demands a larger M .
SGPR+HMC outperforms JointHMC for this dataset across different M but the advantage is more pronounced at smaller M .
Further, our method took 1248 vs. 2109 wall clock sec.
for the joint scheme for the same number of hyperparameter samples and 500 inducing points.
5.3 Ablation study
In order to understand the efficacy of Algorithm 1 we conduct an ablation study where we perform inference in the same manner, but keeping inducing locations fixed.
Algorithmically, this implies that we don’t need to compute the stochastic ELBO ˆ L and just conduct a single sampling run for the hyperparameters.
The results across 10 splits are summarised in Table 5.
[Table]
5.4 Runtimes
While it is possible to train exact GPR with HMC for datasets of this size (in terms of N ) it is important to look at the trade-off in terms of compute cost.
In Figure 4 we record the average number of wall clock seconds to draw 500 samples under each method.
The cost of sampling is O ( N ) for SGPR + HMC but O ( N 3 ) for Exact GPR + HMC.
JointHMC deals with a higher dimensional phase space ( u , θ ) hence requires more tuning.
We don’t include tuning time for a fair comparison.
For further context we report the total training run-time for our scheme alongside ML-II, GPR + HMC and FBGP in Table 6.
The hybrid scheme we propose is significantly cheaper to canonical alternatives with virtually no degradation in predictive performance
[Table]
[Table]
Discussion
The evidence framework continues to be the pre-dominant method for training Gaussian processes since their inception into modern machine learning [Rasmussen and Williams, 2006].
While the marginal likelihood is a compelling model selection objective as it offers an inherent trade-off between data-fit and complexity [Table] it is susceptible to overfitting and other pathologies leading to biased inference.
This work builds on existing methods that combine sparse Gaussian process regression based on inducing variables with Bayesian hyperparameter inference.
Bayesian hyperparameter inference in GPs is however intractable and one has to consider balancing between objectives of computational cost, prediction accuracy and robustness of uncertainty intervals.
While in straightforward conditions the fully Bayesian approach might be counter-productive
[Table] most real-world applications of GPs rely on engineering sophisticated hand-crafted kernels involving many hyperparameters where there risk of overfitting is pronounced and further [Table] harder to detect.
A more robust solution is to incorporate prediction intervals that reflect these uncertainties in the model choice.
Studying full Bayesian inference in more sophisticated GP models like deep [Damianou and Lawrence, 2013], warped [Snelson et al., 2004] and convolutional GPs [Van der Wilk et al., 2017] will offer greater insight to this question and is an imminent direction of future work.
Acknowledgements
This research was conducted while WPB and DRB were students at the University of Cambridge.
During that time, WPB was supported by the Engineering and Physical Research Council (studentship number 10436152).
VL acknowledges funding from the Qualcomm Innovation Fellowship (Europe).
A Experimental Set-up
For methods SGPR, SGPR + HMC, JointHMC and Ablation experiment we use the Adam [Kingma and Ba, 2014] optimizer with a learning rate set at 0.01 (we didn’t extensively tune for learning rates and 0.01 seemed to give a reasonable performance).
We do maintain consistency over the data splits and initialisation values for the inducing locations and hyperparameters across all the methods. Further, all the sparse models use M = 100 inducing variables to aid in run-time analysis.
All the hyperparameters are initialised at the gpytorch default of log(2) and inducing locations at a random subset of the training data split.
SGPR + HMC: We place individual priors over the set of hyperparameters {{ l d } Dd =1 , σ f , σ n } shown in the code block below.
During the warm-up phase we optimize both the inducing locations and hyperparamters. We use J = 100 samples to construct the stochastic ELBO for the first sampling window along with 500 steps of tuning, thereafter just 10 samples are used every 50 gradient steps.
At the end of training we again draw J = 100 samples.
The intermediate sampling windows do not require elaborate tuning as we persist good initial step-size values from the penultimate chains.
Despite this we do expend a few tuning steps in each sampling window as it improved the overall performance of the sampler. The inducing locations are kept fixed during sampling and are only optimized through the stochastic ELBO.
JointHMC:
As recommended by the authors we use a warm-up phase of 100 gradient steps to optimize inducing locations.
Subsequent training happens through the HMC sampler which targets the joint variables ( v , θ ) (where v is a whitened representation of u ) with a target acceptance rate of 0.8, path length (number of leapfrog steps) to 10 and an initial step-size of 0.01 with an adaptation rate of 0.1. We use tfd. Gamma(2.0, 1.0) for each inidividual kernel hyperparamter.
The software for all the methods is largely written in gpytorch [Gardner et al., 2018].
For sampling we resort to the auto-tuning NUTS sampler in pymc3 [Salvatier et al., 2016].
The JointHMC model uses the SGPMC class from gpflow [Van der
Wilk et al., 2020].
The source code for all the models and experiments is attached with the supplementary. The code-snippet below shows the straight-forward pymc3 sampling loop which is triggered at pre-specified intervals.
with pm.
Model() as model_pymc3:
[Equation]
# Z_opt is the intermediate inducing points from the optimisation stage y_ = gp.marginal_likelihood(
[Equation]
self
[Equation]
train_x.numpy
[Equation]
Xu=Z_opt, \\ y=self.train_y.numpy(), noise=sig_n) if sampler_params is not None: step = pm.
NUTS(step_scale = sampler_params[’step_scale’]) else:step = pm.
NUTS() trace = pm.sample(n_samples, tune=tune, chains=1, step=step, \\ return_inferencedata=False) return trace
We additionally compare the performance of our algorithm to 2, 3 and 4 layer deep GPs (DGP 2–4), inducing points and point estimation [Damianou and Lawrence, , [Salimbeni and Deisenroth, 2017].
We also compare to a two-layer Bayesian neural network with ReLu activations, 50 hidden units, with inference by probabilistic backpropagation (PBP). The results were taken from Salimbeni and Deisenroth [2017] and Hern´andez-Lobato and Adams [2015] respectively and follow a very similar data processing scheme for the datasets. We learn the inducing locations Z through optimisation but keep the number of inducing points fixed across all methods. B.2 NUTS Sampling Summary
In the tables below we include the summary statistics of the NUTS sampler for split 4 for each dataset for the SGPR + HMC model. The statistics were computed based on the trace of the final sampling window. The columns hdi 3% and hdi 97% calculate the highest posterior density interval based on marginal posteriors.
ess [Equation]
{+1 + 2 (cid:80) Tt =
 ˆ ρ t computes effective sample size where M is the number of chains, N is the number of samples in each chain and ρ t denotes auto-correlation at lag t . For the results reported below N = 100+} and M = 1 . Each chain was run with 500 warm-up iterations for the sampler to adapt to an optimal step-size.
ess bulk refers to the effective sample size normalized draws and is a useful indicator of sampling efficiency tail the of the [Vehtari et al., 2021]. B.3 Boston hyper mean sd hdi 3% hdi 97% mcse mean mcse sd ess bulk ess tail ls[0] 2.
ls[1] 7. ls[2] 5. ls[3] 8. ls[4] 3.
ls[5] 3.
ls[6] 5.
ls[7] 3.
ls[8] 6.
ls[9] 2.
ls[10] 5.
ls[11] 6. ls[12] 1.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
B.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
ls[4] 2.
ls[5] 25. ls[6] 1. ls[7] 21.
sig
f 1.
sig n 0.
B.7 WineRed
[Table]