Original: Improved Inverse-Free Variational Bounds for Sparse Gaussian Processes
Revised: Improved Inverse-Free Variational Bounds for Sparse Gaussian Processes

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: The need for matrix decompositions (inverses) is often named as a major impediment to scaling Gaussian process (GP) models, even in eﬃcient approximations.
Revised: The need for matrix decompositions (inverses) is often named as a major impediment to scaling Gaussian process (GP) models, even in eﬃcient approximations.

Original: To address this,
Revised: To address this,

Original: Van der Wilk et al.
Revised: Van der Wilk et al.

Original: (2020) introduced a variational lower bound that can be computed without these costly operations.
Revised: (2020) introduced a variational lower bound that can be computed without these costly operations.

Original: We improve this bound by 1) simplifying it by removing the need for iterative procedures, and 2) making it more numerically stable.
Revised: We improve this bound by 1) simplifying it by removing the need for iterative procedures, and 2) making it more numerically stable.

Original: While these improvements do not result in a procedure that is faster in wall-clock time than existing variational bounds, they are  necessary steps along the way.
Revised: While these improvements do not result in a procedure that is faster in wall-clock time than existing variational bounds, they are likely to be necessary steps along the way

Original: 1. Introduction
Revised: 1. Introduction

Original: Gaussian processes (GPs) (Rasmussen and Williams, 2006) are distributions on functions with many properties that make them convenient for Bayesian modelling.
Revised: Gaussian processes (GPs) (Rasmussen and Williams, 2006) are distributions on functions with many properties that make them convenient for Bayesian modelling.

Original: In particular, their nonparametric properties improve uncertainty estimation and simplify hyperparameter tuning.
Revised: In particular, their nonparametric properties improve uncertainty estimation and simplify hyperparameter tuning.

Original: Variational approximations (Titsias, 2009) were originally introduced to address the
Revised: Variational approximations (Titsias, 2009) were originally introduced to address the

Original: O ( N 3 ) computational cost of manipulating GPs when considering N datapoints.
Revised: O ( N 3 ) computational cost of manipulating GPs when considering N datapoints.

Original: However, these approximations also enabled Gaussian processes to be used in much more sophisticated models.
Revised: However, these approximations also enabled Gaussian processes to be used in much more sophisticated models.

Original: Deep Gaussian Processes (DGPs) in particular are intriguing as they present a way to merge the beneﬁts of GPs with Neural Networks (NNs).
Revised: Deep Gaussian Processes (DGPs) in particular are intriguing as they present a way to merge the beneﬁts of GPs with Neural Networks (NNs).

Original: While DGPs can be seen as direct
Revised: While DGPs can be seen as direct

Original: Bayesian analogues of deep NNs (Dutordoir et al., 2021), DGPs are still not a convenient way to perform Bayesian Deep Learning.
Revised: Bayesian analogues of deep NNs (Dutordoir et al., 2021), DGPs are still not a convenient way to perform Bayesian Deep Learning.

Original: One contributing factor to this, is the high cost of each training iteration, despite the development of minibatching (Hensman et al., 2013).
Revised: One contributing factor to this, is the high cost of each training iteration, despite the development of minibatching (Hensman et al., 2013).

Original: The cause is the need to compute an M × M matrix inverse and determinant, making the cost for each layer
Revised: One cause is the need to compute an M × M matrix inverse and determinant, making the cost for each layer

Original: O
Revised: O

Original: ( M 3 + BM 2 ), where M is analogous to the number of neurons in a layer and B is the minibatch size.
Revised: ( M 3 + BM 2 ), where M is analogous to the number of neurons in a layer and B is the minibatch size.

Original: These matrix operations are costly and limit the eﬀectiveness of minibatching for speeding up each training iteration.
Revised: These matrix operations are costly and limit the eﬀectiveness of minibatching for speeding up each training iteration.

Original: In addition, these matrix operations are serial and require high-precision ﬂoating point operations, which is poorly suited to modern day hardware.
Revised: In addition, these matrix operations are serial and require high-precision ﬂoating point operations, which is poorly suited to modern day hardware.

Original: It therefore seems likely that these operations will need to be removed before Deep Gaussian Processes will truly scale.
Revised: It therefore seems likely that these operations will need to be removed for Deep Gaussian Processes to truly scale.

Original: We investigate variational approximations that do not require the computation of costly matrix operations at every iteration.
Revised: We investigate variational approximations that do not require the computation of costly matrix operations at every iteration.

Original: Van der Wilk et al.
Revised: Van der Wilk et al.

Original: (2020) introduced such an “inversefree” approximation, which could be computed without performing matrix decompositions or inverses to completion, while also being a drop-in replacement for schemes based on
Revised: (2020) introduced such an “inversefree” approximation, which could be computed without performing matrix decompositions or inverses to completion, while also being a drop-in replacement for schemes based on

Original: Hensman et al.
Revised: Hensman et al.

Original: (e.g. Salimbeni and Deisenroth (2017) for DGPs).
Revised: (e.g. Salimbeni and Deisenroth (2017) for DGPs).

Original: However, no wall-clock speed-up was provided.

Original: We make the inverse-free bound 1) less cumbersome, by removing iterative procedures, and 2) more numerically stable.
Revised: We make the inverse-free bound 1) less cumbersome, by removing iterative procedures, and 2) more numerically stable.

Original: While issues with training still impede a wall-clock speed-up, we believe our improvements are necessary steps on the way.
Revised: While issues with training still prevent a wall-clock speed-up, we believe our improvements are necessary steps on the way.

Original: 2. Variational Inference for Gaussian Process Models
Revised: 2. Variational Inference for Gaussian Process Models

Original: For the sake of brevity, we consider the simplest possible model to introduce our new bound.
Revised: For the sake of brevity, we consider the simplest possible model to introduce our new bound.

Original: We aim to learn some function f : X → R in a Bayesian manner by placing a GP prior on f ( · ), and observing data y ∈ R N through some arbitrary pointwise factorised likelihood:[Equation]
Revised: We aim to learn some function f : X → R in a Bayesian manner by placing a GP prior on f ( · ), and observing data y ∈ R N through some arbitrary pointwise factorised likelihood:

Original: [Equation]
Revised: [Equation]

Original: 2.1. Marginal Parameterisation
Revised: 2.1. Marginal Parameterisation

Original: Variational inference for GPs 1 decouples the size of matrix inverses from the dataset size (Titsias, 2009), allows minibatching (Hensman et al., 2013), and addresses non-conjugacy (Hensman et al., 2015).
Revised: Variational inference for GPs 1 decouples the size of matrix inverses from the dataset size (Titsias, 2009), allows minibatching (Hensman et al., 2013), and addresses non-conjugacy (Hensman et al., 2015).

Original: The approximation introduces a set of tractable approximate posteriors by conditioning the prior on M inducing variables u ∈ R M , e.g. observations f ( Z ) for the M inputs collected in Z ∈ X M , and specifying the marginal distribution on u as q ( u ) =
Revised: The approximation introduces a set of tractable approximate posteriors by conditioning the prior on M inducing variables u ∈ R M , e.g. observations f ( Z ) for the M inputs collected in Z ∈ X M , and specifying the marginal distribution on u as q ( u ) =

Original: N ( u ; m , S ).
Revised: N ( u ; m , S ).

Original: This results in the approximate posterior for arbitrary inputs X ∗ :
Revised: This results in the approximate posterior for arbitrary inputs X ∗ :

Original: [Equation]
Revised: [Equation]

Original: where the subscripts of K matrices determine its elements as e.g. [ K uu ] ij =
Revised: where the subscripts of K matrices determine its elements as e.g. [ K uu ] ij =

Original: Cov[ u i , u j ] =
Revised: Cov[ u i , u j ] =

Original: k ( z i , z j ).
Revised: k ( z i , z j ).

Original: The KL between approximation and posterior (Matthews et al., 2016) can be minimised by maximising the ELBO (Hensman et al., 2013, 2015):
Revised: The KL between approximation and posterior (Matthews et al., 2016) can be minimised by maximising the ELBO (Hensman et al., 2013, 2015):

Original: [Equation]
Revised: [Equation]

Original: 2.2. Likelihood Parameterisation
Revised: 2.2. Likelihood Parameterisation

Original: While the former parameterisation is the most common [Equation] it is numerically unstable as K uu can be arbitrarily badly conditioned (e
Revised: While the former parameterisation is the most common, it is numerically unstable as K uu can be arbitrarily badly conditioned (e

Original: [Equation]
Revised: [Equation]

Original: g. singular for repeated inducing inputs).
Revised: g. singular for repeated inducing inputs).

Original: Panos et al.
Revised: Panos et al.

Original: (2018) noted this in passing
Revised: (2018) noted this in passing

Original: [Equation]
Revised: [Equation]

Original: and suggested using the likelihood parameterisation :
Revised: and suggested using the likelihood parameterisation :

Original: [Equation]
Revised: [Equation]

Original: This can simply be seen as a straightforward reparameterisation of q ( u ) =
Revised: This can simply be seen as a straightforward reparameterisation of q ( u ) =

Original: N ( u ; m , S ) (note the equalities to m , S above) 2 , but with the matrix operations now performed on K uu + Σ :q ( f ( x n )
Revised: N ( u ; m , S ) (note the equalities to m , S above) 2 , but with the matrix operations now performed on K uu + Σ :

Original: [Equation]
Revised: [Equation]

Original: = N (cid:0) f ( [Equation] [Equation] This has one big advantage: by enforcing a small minimum diagonal term in Σ , we only need to decompose matrices with lower bounded minimum eigenvalues.
Revised: This has one big advantage: by enforcing a small minimum diagonal term in Σ , we only need to decompose matrices with lower bounded minimum eigenvalues.

Original: In the marginal parameterisation, this is typically ensured in an ad-hoc manner by adding “jitter” to K uu directly.
Revised: In the marginal parameterisation, this is typically ensured in an ad-hoc manner by adding “jitter” to K uu directly.

Original: In the likelihood parameterisation, numerical stability is ensured through a welldeﬁned variational parameter.
Revised: In the likelihood parameterisation, numerical stability is ensured through a welldeﬁned variational parameter.

Original: If it has to be constrained to be larger than what is optimal because of limited ﬂoating-point precision, this is penalised in the ELBO, in a manner that is uniﬁed with all other properties of the approximation.
Revised: If it has to be constrained to be larger than what is optimal because of limited ﬂoating-point precision, this is penalised in the ELBO, in a manner that is uniﬁed with all other properties of the approximation.

Original: Typically though, the uncertainty on u will be many times larger than jitter matrices, leading to much more stable inverses.
Revised: Typically though, the uncertainty on u will be many times larger than jitter matrices, leading to much more stable inverses.

Original: Numerical stabilityand the choice of parameterisation is even more crucial for inverse-free variational bounds.
Revised: This improved numerical stability is even more crucial for inverse-free variational bounds.

Original: 3. Improved Inverse-Free Variational Bounds for Gaussian Processes 3.1. Inverse-Free Marginal Parameterisation
Revised: 3. Improved Inverse-Free Variational Bounds for Gaussian Processes 3.1. Inverse-Free Marginal Parameterisation

Original: Van der Wilk et al.
Revised: Van der Wilk et al.

Original: (2020) noted that, for concave log-likelihoods, using an over-estimate of the predictive variance of eq.
Revised: (2020) noted that substituting an over-estimate of the predictive variance of eq.

Original: (2) in the ELBO calculation, would yield a further lower bound to L sv .
Revised: (2) into the ELBO calculation resulted in a further lower bound to L sv , for concave log-likelihoods.

Original: This helps with removing the inverse term k n u K − 1 uu k u n , which can not be removed by simple substitution, from eq. (2), as it can be replaced by a bound
Revised: This allows further inverses to be removed by bounding k n u K − 1 uu k u n using

Original: [Equation]
Revised: [Equation]

Original: with equality when a n = K − 1 uu k u n .
Revised: with equality when a n = K − 1 uu k u n .

Original: In principle, this could lead to a method where one a n per datapoint would be optimised alongside with the existing variational parameters.
Revised: In principle, this could lead to a method where one a n per datapoint would be optimised alongside with the existing variational parameters.

Original: To avoid adding a parameter for each datapoint, we can reparameterise a n = Tk u n , where T ∈
Revised: To avoid adding a parameter for each datapoint, we can reparameterise a n = Tk u n , where T ∈

Original: R M × M is now a free matrix which takes T
Revised: R M × M is now a free matrix which takes T

Original: = K − 1 uu when optimised to a maximum.
Revised: = K − 1 uu when optimised to a maximum.

Original: Van der
Revised: Van der

Original: Wilk et al.
Revised: Wilk et al.

Original: (2020) suggested to optimise T along with the other variational parameters.
Revised: (2020) suggested to optimise T along with the other variational parameters.

Original: Herein lies the problem of numerical stability of this parameterisation: As  K uu becomes increasingly badly conditioned, entries in K − 1 uu tend to inﬁnity.
Revised: Herein lies the problem of numerical stability of this parameterisation: As the conditioning of K uu worsens, entries in K − 1 uu tend to inﬁnity.

Original: Gradient-based optimisers like Adam, which have limited step sizes, can not be expected to successfully optimise over these wildly varying scales.
Revised: Gradient-based optimisers (e.g. Adam) have limited step sizes and cannot be expected to successfully recover such solutions.

Original: The second problem of using this parameterisation lies in the log-determinants that need to be computed for the KL term (eq.
Revised: The second problem of using this parameterisation lies in the log-determinants that need to be computed for the KL term (eq.

Original: These terms were not removed by the reparameterisation, and areactually equally hard as inverses.
Revised: These terms were not removed by the reparameterisation, and are equally costly as inverses.

Original: In many implementations, the same Cholesky decomposition that is  used for computing the inverse is used for computing log-determinant terms.
Revised: The same Cholesky decomposition that is often used for computing the inverse is used for computing log-determinant terms.

Original: This is addressed by using randomly truncated Conjugate Gradients (CG) to compute unbiased estimates of the gradient of the log-determinant (Filippone and Engler, 2015).
Revised: To still obtain an inverse-free method, this is addressed by using randomly truncated Conjugate Gradients (CG) to compute unbiased estimates of the gradient of the logdeterminant (Filippone and Engler, 2015).

Original: To avoid needing to run M iterations of CG and ending back at the O ( M 3 ) computational cost, T was used as a preconditioner, which at least would ensure convergence within a single CG iteration at the optimal value T = K − 1 uu .
Revised: To avoid needing to run M iterations of CG and ending back at the O ( M 3 ) computational cost, T was used as a preconditioner, which at least would ensure convergence within a single CG iteration at the optimal value T = K − 1 uu .

Original: Despite some nice mathematical properties, this procedure is cumbersome as it introducestwo nestedsets of iterations: one inner loop foractually running CG to compute the bound, and an outer loop for optimising all the parameters.
Revised: This procedure is cumbersome as it introduces nested iterations: one inner loop for running CG to compute the bound, and an outer loop for optimising all the parameters.

Original: This is diﬃcult to implement, and slow to run.
Revised: This is diﬃcult to implement, and slow to run.

Original: 3.2. Deriving the Inverse-Free Likelihood Parameterisation
Revised: 3.2. Deriving the Inverse-Free Likelihood Parameterisation

Original: Applying the same inverse-free trick in the likelihood parameterisation solves both problems with the former approach.
Revised: Applying the same inverse-free trick in the likelihood parameterisation solves both problems of the former approach.

Original: We begin in the same way: by lower-bounding the predictive variance, only  in the likelihood parameterisation of eq.(7), and reparameterising using T :
Revised: We begin in the same way: by lower-bounding the predictive variance and introducing T , only we do so in the likelihood parameterisation of eq.

Original: [Equation]
Revised: [Equation]

Original: In this case, equality will be obtained when T =
Revised: In this case, equality will be obtained when T =

Original: ( K uu + Σ ) − 1 , which is much more numerically stable, as discussed in section 2.2.
Revised: ( K uu + Σ ) − 1 , which is much more numerically stable, as discussed in section 2.2.

Original: Next, our goal is to remove log-determinant terms from the KL[ q ( u ) || p ( u )] computation.
Revised: Next, our goal is to remove log-determinant terms from the KL[ q ( u ) || p ( u )] computation.

Original: We do this by upper bounding it using properties from our inverse-free parameterisation of the variance.
Revised: Our likelihood parameterisation allows us to derive an upper bound to the KL without matrix decompositions.

Original: We start by considering whether the upper bound on the predictive variance in eq.
Revised: We start by considering whether the upper bound on the predictive variance in eq.

Original: (11) could have been obtained by a particular choice of variational distribution q ( u ).
Revised: (10) could have been obtained by a particular choice of variational distribution q ( u ).

Original: We ﬁnd this by equating the upper bound in eq.
Revised: We ﬁnd this by equating the upper bound in eq.

Original: (11), with the predictive variance term in the marginal parameterisation from eq.
Revised: (10), with the predictive variance term in the marginal parameterisation from eq.

Original: (2), and solving for S :
Revised: (2), and solving for S :

Original: [Equation]
Revised: [Equation]

Original: We ﬁnd the S that results in our upper bounded predictive variance in two forms:
Revised: We ﬁnd the S that results in our upper bounded predictive variance in two forms:

Original: [Equation]
Revised: [Equation]

Original: We obtain one compact form
Revised: We obtain one compact form

Original: [Equation]
Revised: [Equation]

Original: and one form that shows that it is equivalent to the likelihood parameterisation, but with an additional PSD term added
Revised: and one form that shows that it is equivalent to the likelihood parameterisation

Original: [Equation]
Revised: [Equation]

Revised: but with an additional PSD term added.

Original: We now ﬁnd KL[ q ( u ) || p ( u )] for our new inverse-free parameterisation by substituting in eqs.
Revised: We now ﬁnd KL[ q ( u ) || p ( u )] for our new inverse-free parameterisation by substituting in eqs.

Original: (13) and (14) and m = K uu µ from eq.
Revised: (12) and (13) and m = K uu µ from eq.

Original: (6) into eq.
Revised: (5) into eq.

Original: [Equation]
Revised: [Equation]

Original: Using the fact that log | A + ∆ | ≥ log | A | if A , ∆ are PSD
Revised: By using the fact that log | A + ∆ | ≥ log | A | if A [Equation] ∆ are PSD

Original: [Equation]
Revised: [Equation]

Original: we can bound the KL:
Revised: we can bound the KL:

Original: [Equation]
Revised: [Equation]

Original: Simply by reparameterisation, the inverse in the trace from eq.
Revised: Simply by reparameterisation, the inverse in the trace from eq.

Original: (8) has been removed.
Revised: (7) has been removed.

Original: Now, we apply the bound log | A | ≤
Revised: Now, we apply the bound log | A | ≤

Original: Tr( A − I ) to remove the ﬁnal problematic log | K uu + Σ | term:
Revised: Tr( A − I ) to remove the ﬁnal problematic log | K uu + Σ | term:

Original: [Equation]
Revised: [Equation]

Original: By applying the bound to ( K uu + Σ ) T , we ensure that it becomes tight at the optimum when T =
Revised: By applying the bound to ( K uu + Σ ) T , we ensure that equality holds when T =

Original: ( K uu + Σ ) − 1 .
Revised: ( K uu + Σ ) − 1 , at which point we recover the original likelihood parameterisation.

Original: 3.3. Summarising the Inverse-Free Likelihood Parameterisation
Revised: Summarising the Inverse-Free Likelihood Parameterisation

Original: Through our derivation, we obtained new lower bound on the marginal likelihood  for a GP model.
Revised: Through our derivation, we obtained a marginal likelihood bound for a GP model that 1) is more numerically stable, and 2) does not require iterative procedures for computing the KL.

Original: It can be obtained in two steps.
Revised: It can be obtained in two steps.

Original: First [Equation] we use eqs.
Revised: First, we use eqs.

Original: (7) and (13) to reparameterise the approximate posterior, which also implies an inverse-free predictive distribution:
Revised: (6) and (12) to reparameterise the approximate posterior, which also implies an inverse-free predictive distribution:

Original: [Equation]
Revised: [Equation]

Original: Next, we obtain a marginal likelihood lower bound that does not require matrix decompositionsor iterative procedures by applying additional bounds:
Revised: Next, we remove matrix decompositions by applying additional bounds to the KL:

Original: [Equation]
Revised: [Equation]

Original: Training now requires optimising T along with the existing variational parameters µ , Σ .
Revised: Training now requires optimising T along with the existing variational parameters µ , Σ .

Revised: To ensure the ﬁnal log determinants are eﬃciently computable, we parameterise T and Σ using their Cholesky decompositions.

Original: We note thatat the additional slack in the bound is removed when T is at its optimal value of ( K uu + Σ ) − 1 ,which allows it to recover the original solution from Hensman et al.
Revised: We note that when T is optimised to its optimal value of ( K uu + Σ ) − 1 , the predictive variance and KL bounds are both equalities. This implies that if optimisation is successful, no performance is lost by using this inverse-free bound.

Original: 4. Results
Revised: 4. Results

Original: To test the behaviour of the method, we train a GP regression model with the original SVGP bound, and the our inverse free bound (RSVGP).
Revised: To test the behaviour of the method, we train a GP regression model with the original SVGP bound, and the our inverse free bound (RSVGP).

Original: To train the model, we follow the common procedure of using autodiﬀ for ﬁnding gradients, and Adam to perform optimisation.
Revised: To train the model, we follow the common procedure of using autodiﬀ for ﬁnding gradients, and Adam to perform optimisation.

Original: We note that our new RSVGP bound was signiﬁcantly easier to implement than that of van der
Revised: We note that our new RSVGP bound was signiﬁcantly easier to implement than that of van der

Original: Wilk et al.
Revised: Wilk et al.

Original: (2020), because the latter required custom gradient ops with CG inner loops.
Revised: (2020), because the latter required custom gradient ops with CG inner loops.

Original: Figure 1 shows a ﬁt on a toy 1D dataset, together with a plot of the ELBO objective with time.
Revised: Figure 1 shows a ﬁt on a toy 1D dataset, together with a plot of the ELBO objective with time.

Original: We see that the approximations are all very similar, indicating that the inverse free bound is behaving similarly to the commonly used SVGP approximation (Hensman et al., 2013).
Revised: We see that the approximations are all very similar, indicating that the inverse free bound is behaving similarly to the commonly used SVGP approximation (Hensman et al., 2013).

Original: However, even for this simple example, we note that our inverse-free method takes more iterations to converge than the existing SVGP.
Revised: However, even for this simple example, we note that our inverse-free method takes more iterations to converge than the existing SVGP.

Original: Signiﬁcant problems arise with training when running RSVGP on the more realistic kin40nm UCI dataset (ﬁg. 2).
Revised: Signiﬁcant problems arise with training when running RSVGP on the more realistic kin40nm UCI dataset (ﬁg. 2).

Original: We notice that hyperparameters converge much more slowly for RSVGP than SVGP, which leads to poor performance.
Revised: We notice that hyperparameters converge much more slowly for RSVGP than SVGP, which leads to poor performance.

Original: In particular, there is very strange behaviour of the hyperparameters moving in the correct direction, to only reverse direction suddenly.
Revised: In particular, there is very strange behaviour of the hyperparameters moving in the correct direction, to only reverse direction suddenly.

Original: This is reﬂected by a metric of how good the variational parameter
Revised: This is reﬂected by a metric of how good the variational parameter

Original: T approximates ( K uu + Σ ) − 1 .
Revised: T approximates ( K uu + Σ ) − 1 .

Original: It seems that very sudden changes in the quality of the approximation occur, which signiﬁcantly hinders optimisation.
Revised: It seems that very sudden changes in the quality of the approximation occur, which signiﬁcantly hinders optimisation.

Original: 5. Discussion
Revised: 5. Discussion

Original: We introduced a new and more convenient lower bound on the GP marginal likelihood that does not require matrix decompositions to compute.
Revised: We introduced a new and more convenient lower bound on the GP marginal likelihood that does not require matrix decompositions to compute.

Original: While our method works on simple examples, its optimisation behaviour on more realistic datasets is currently not acceptable for a practically useful method.
Revised: While our method works on simple examples, its optimisation behaviour on more realistic datasets is currently not acceptable for a practically useful method.

Original: The existence of these bounds hints at exciting possibilities, but signiﬁcant optimisation challenges do need to be overcome.
Revised: The existence of these bounds hints at exciting possibilities, but signiﬁcant optimisation challenges do need to be overcome.

Original: One ray of hope, is that currently the most naive optimisation procedure is being used.
Revised: One ray of hope, is that currently the most naive optimisation procedure is being used.

Original: Perhaps a more tailored approach is needed.
Revised: Perhaps a more tailored approach is needed.