Matrix Inversion free variational inference in Conditional Student’s T Processes
1. Introduction
Gaussian Processes (GP) are data-eﬃcient Bayesian non-parametric models that oﬀer calibrated uncertainty quantiﬁcation and are robust to overﬁtting, recently ﬁnding applicability in data-scarce domains (Timonen et al., 2019;
Wang et al., 2020) or where the uncertainty is of utmost importance (Chen et al., 2014) .Their
drawback resides in the computational complexity of inverting the covariance matrix, which is cubic in computation and quadratic in memory.
This has motivated research on sparse GP (SGP) methods (Seeger et al., 2003;
Quinonero-Candela and Rasmussen, 2005).
Titsias (2009) has addressed this problem by leaving the prior distribution of the GP unchanged, with sparsity being enforced in the posterior through inducing points learnt by variational inference.
Hensman et al.
(2013) proposed an inducing point framework scalable to large data, obtaining posterior formulas conditioned on these artiﬁcial points.
However, this scales supralinearly with regards to inducing point numbers, resulting in O ( M 2 N + M 3 ) computation and O ( MN + M 2 ) storage complexity, where M is the number of inducing points.
Therefore, a major obstacle towards the wider adoption of GP in large scale datasets is related to the computational cost of matrix inversion and log determinants.
With this in mind, in van der Wilk et al.
(2020) the authors propose a lower bound that can be computed without computationally expensive matrix operations such as inversion.
Similar in scope, we propose to learn variational approximations of the covariance matrix, implicitly also over inverses, of inducing and training points, thereby proposing a computationally eﬃcient approximate posterior over covariance matrices within the probabilistic framework of Student’s T Processes (STP) (Shah et al., 2014).
Compared to van der Wilk et al.
(2020), where the authors solve the issue of inversefree predictive equations by using a highly structured posterior over U , we obtain similar properties by virtue of our Bayesian hierarchical process, with an additional KL divergence term over our approximation of K − 1 uu , thus favouring the retrieval of the exact solution, alongside showing that it works on large scale datasets, a task which was not tackled in the latter work due to training instability.
2. Generative process of conditional sparse Student’s T Processes
We follow the hierarchical Bayesian construction in prior space introduced in Shah et al.
(2014) for the non-sparse scenario in a regression scenario:
[Equation]
In Shah et al.
, the authors show that the latent function follows a MV T ( v p , 0 , K ff ), where MV T stands for multivariate t-distribution, which is a generalization to random vectors of the Student’s t-distribution.
For completeness, we provide in Appendix A a more detailed proof of this statement.
We adapt the generative process to the sparse scenario.
In order to achieve prior conditional matching in the Inverse Wishart prior over covariance matrices we add another layer in our Bayesian hierarchical generative process, namely:
[Equation]
, where v f,p = v p + m + 1 for notation purposes.
Motivation behind the change of scaling for mean covariance matrices is given in Appendix B.
We denote K ff · u = K ff +
K fu T K uu T K uf −
2 K fu T K uf ≥ K ff · u , with equality if and only
if T = K − 1 uu (Davies, 2015).
As a consequence of Theorem 3 in Bodnar and Okhrin (2008), we can further express Σ ff · u |
T ∼ W − 1 n (cid:0) v f,p + n, v f,p K ff · u (cid:1) and Σ − 1 uu Σ uf | Σ ff · u , T ∼ MN (cid:16) T K uf , Σ ff · u ⊗ Tv f,p (cid:17) , which are present in equation (7), where ⊗ denotes the Kronecker product and MN denotes the matrix variate distribution.
Lastly, we notice that f no longer follows a MV T ( v p , 0 , K ff ) distribution.
Instead we have f | T ∼ MV T ( v p , 0 , K ff · u + K fu T K uf ), which converges to
MV T ( v p , 0 , K ff ) as v → ∞ , implicitly T ≈ K − 1 uu .
2.1. Variational Posteriors over Inverse-Wishart Processes
To perform inference over our hierarchical Bayesian model, we need a way of deﬁning a valid variational mean covariance matrix for the posterior Inverse-Wishart Process over the space of possibly inﬁnitely large covariance matrices: q (Σ fu,fu )
=
W − 1 n + m (cid:16) v f,q + n, v f,q
[Equation]
, where ˜ K fu,fu (cid:31) 0 positive deﬁnite matrix deﬁned as
[Equation]
To see this is true, we make use of the following proposition.
Proposition 1 (Proposition 2.1 in Gallier et al. (2010))
For any symmetric matrix
M of the form: M = (cid:18)
A
B
B (cid:62) C (cid:19) and if C is invertible then M (cid:31)⇐⇒
A (cid:31) 0 and
A − BC − 1 B (cid:62) (cid:31) 0 .
We note that T − 1 (cid:31) 0 and K ff · u (cid:31) 0 by construction, thereby ˜ K fu,fu (cid:31) 0, so that it satisﬁes the condition for deﬁning mean covariance matrices for Inverse Wishart distributions.
3. Variational lower bound of conditional inverse free sparse Student’s T Processes
We defer the full derivation of the ELBO to Appendix C.
[Equation]
, where we deﬁne constituent terms of the factorized approximate posterior as follows:
[Equation]
Expectation over data ﬁt term in SVI setting.
We can re-express expectations over individual terms as follows:
[Equation]
, where ˜ U (Σ f k u,f k u ) =
Σ f k u Σ − 1 uu m and ˜Σ(Σ
[Equation]
k
[Equation]
k
[Equation]
= Σ f [Equation] deﬁne distributions over scalars.
Hence, the aforementioned multivariate formulas can now be rewritten as:
[Equation]
, where IG represents in the inverse gamma distribution.
Kullback-Leibler Divergences.
In the KL term present in our lower bound we can separate the fraction terms that with respect to integrants that they exclusively depend on:
[Equation]
, where the ﬁrst and third terms cancel out due to conditional prior matching.
The second term can be written as:
[Equation]
We remind ourselves that we can obtain samples from Σ − 1 uu via following Bayesian hierarchical construction:
[Equation]
No matrix inversion is required for this divergence term.
Moreover, T is obtained via the Bartlett decomposition
[Equation]
, where A T is a lower triangular matrix deﬁned in Appendix D. Then, we can apply again the Bartlett decomposition to obtain samples from Σ − 1 uu by noticing that [Equation] represents a lower triangular matrix.
Finally, samples are obtained via
[Equation]
˜
[Equation]
Hence, Σ − 1 uu is a product of lower triangular matrices L K − 1 uu A T A Σ − 1 uu , with log | Σ − 1 uu | having an analytic formula.
We now focus on KL [ q ( T ) (cid:107) p ( T )], which has an analytic formula:
[Equation]
, where ψ m denotes the multivariate digamma function.
From the above equation we can notice that no matrix inverses are required in the computation of the KL divergence.
Directly computing log | K uu | would incur an O (cid:0) m 3 (cid:1) computational cost.
To bypass it, we use the following proposition.
Proposition 2 For a p.s.d.
matrix K ∈ M n × n , x being a conjugate gradient solution to
Kx = g , where g ∈ M n × k are independent standard normal samples and ˜ K (cid:31)
we have the following lower bound on the log determinant of K :
[Equation]
with the bound being tight if and only if ˜ K ≈ K .
A proof of this proposition can be found in Appendix E. Whereas in current work we use standard preconditioned conjugate gradients, one could use the Unbiased LInear Systems
SolvEr (ULISSE) (Filippone and Engler, 2015), a randomly truncated CG run, to compute unbiased estimate of x i .
Remark 3 Translating Proposition 2 to our case, we have log | K uu |≥ m + 1 k k (cid:80)
[Equation] log | T | .
In the computation of x i we precondition with T .
Hence if T ≈ K − 1 uu the CG routine will ﬁnish in one step.
We empirically prove on UCI datasets that this method is close to O (cid:0) m 2 (cid:1) (Appendix F.4).
4. Experiments Convergence
to Matrix Inversion based counterparts.
We are interested to ﬁnd out whether our proposed model (denoted as IF-CSTP) is capable to recover the testing
[Figure]
IF-STP is capable of almost recovering the testing time predictive behaviour of both
STP and SVGP on the “snelson” dataset (Snelson and Ghahramani, 2006) (Figure 1).
We train IF-CSTP and SVGP for varying number of inducing points on “banana” dataset (Figure 2).
Whereas SVGP brings the inducing points’ locations Z exactly on the classes’ demarcation lines, IF-CSTP does not exhibit similar behaviour.
Initial results on large-scale datasets.
We evaluate our proposed model on selected
UCI datasets (Figure 3).
The gap between ELBOs is predominantly caused by the KLdivergence pertaining to T (eqn. 19).
IF-CSTP’s testing time behaviour almost approaches
SVGP.
However, in some cases, such as on “Protein” for 250 inducing points, the gap widens.
5. Discussion
Using our matrix inversion-free variational lower bound we showed similar behaviour to matrix inversion-dependent counterparts on toy tasks.
However, scaling our proposed method to work on large-scale data still remains a research avenue as there is a tendency of overstability of Z for large numbers of inducing points due to large penalty stemming from
KL [ q ( T ) (cid:107) p ( T )].
Imposing low tolerance for Proposition 2 does not guarantee that our objective remains a lower bound.
Nevertheless, we have empirically shown that it’s not a problem in practice.
Moreover, a drawback is the lack of competitiveness in terms of speed with SVGP.
For Proposition 2 we used the standard implementation of PCG, which was shown to not be optimal on GPU hardware (Gardner et al., 2018).
Truncated conjugate gradients (Filippone and Engler, 2015; Potapczynski et al., 2021) should be used in future work to provide bias-free solutions.
Appendix A. LATENT FUNCTIONS F ARE STUDENT-T DISTRIBUTED
In Shah et al.
[Equation]
the authors provide a derivation showing that by marginalizing out Σ zz one obtains a multivariate t-distribution.
We provide here a more detailed derivation taking into consideration that we have the general result:
[Equation]
where A = XX (cid:62) .
Adapting this result to our scenario we get that:
[Equation]
Focusing on the determinant terms we can rearrange:
[Equation]
[Equation]
We remind the matrix determinant lemma : | A + uv (cid:62) | =
( I + v (cid:62) A − 1 u ) |
A | .
We apply it to the denonimator of the previous equation:
[Equation]
Plugging this into equation 22 we get that:
[Equation]
which we can now recognize as a MV T ( v , 0 , K zz ), where MV T stands for multivariate tdistribution, which is a generalization to random vectors of the Student’s t-distribution.
The p.d.f.
of this distribution is deﬁned as
[Equation]
where µ is the location, Σ is a positive-deﬁnite m ∗ m matrix and v represent the degrees of freedom.
For v ≥ 1 the mean is µ and for v ≥ 2 the variance is deﬁned as vv −
Σ. Therefore, by starting from a Inverse-Wishart distributed prior over the covariance of a GP evaluated at X , we have implicitly imposed a Student-t distribution over our latent function.
Appendix B. Properties of diﬀerent mean covariance matrix scalings
Under the parametrization introduced in Shah et al.
(2014), respectively:
[Equation]
which implies the following probabilistic formulas and their expectations:
[Equation]
We can notice that for increasingly larger number of inducing points, the expectations over Σ − 1 uu and Σ ff · u get increasingly upscaled, respectively downscaled with respect to their true solution.
For v p → ∞ , the expectations will converge towards the real solution and their variance will collapse to zero
[Equation]
Under the parametrization introduced in section 2:
[Equation]
we have:
[Equation]
In stark contrast to the parameterization introduced in Shah et al.
(2014), we now have an upscaling of expectations over covariance matrices, which only reverts to the real solution for very large values of v p .
However, in the interest of obtaining predictions at testing time from our sparse Student’s T Process, we are only interested in obtaining unbiased samples from Σ − 1 uu and Σ f k f k · u , for which we can notice that our scaling factor is more appropiate.
Appendix C. Derivation of evidence lower bound
We consider our training data D
=
{ X i , Y i } i =1 , ··· ,n .
Our goal is to approximate the true posterior distribution p ( F, U, Σ fu,fu , T | D ) by utilising the following approximate posterior q ( F, U, Σ fu,fu , T ) = [Equation]
[Equation]
| U, Σ fu,fu , T ) q ( U ) p (Σ fu,fu | T ) q ( T ).
Our minimization goal can be expressed as:
[Equation] [Equation]
which after some rearrangements becomes:
[Equation]
We can now re-express this derivation as:
[Equation]
Hence, minimizing the intractable KL [ q ( F, U, Σ fu,fu , T ) | p ( F, U, Σ fu,fu , T | D )] translates to maximizing a tractable lower bound on the log marginal likelihood
[Equation]
respectively the r
[Equation]
h
[Equation]
s. of the previous equation, which can be expressed as follows:
[Equation]
Appendix D. Sampling Wishart Distributions
For this operation we use the Bartlett decomposition of the matrix R ∼ W p ( v, Σ), which can be written in the following factorized manner K − 1 zz
= LAA (cid:62) L (cid:62) , where A has the following matrix form A =   c. . .
n p. . .
c p   , where n ij ∼ N (0 , 1) and c 2 i ∼ χ 2 v − i +1 is the chi-squared distribution and LL (cid:62) = Σ.
Appendix E. Proof of Proposition
We start with a lower bound on log | K | , which can be expressed as:
[Equation]
Hutchinson (1989) introduced a stochastic estimation method for computing matrix traces.
It is deﬁned as follows:
[Equation]
which was shown to converge towards T r ( A ) as m → ∞ , where A ∈ M n × n .
Earlier work by Girard (1987) suggests taking g
i ∈ M n × 1 with individual elements of the column matrix stemming from standard normal, whereas Hutchinson (1989) advocated for taking the individual elements as Rademacher random variables, which translates into equally probable {− 1 , 1 } splits.
We can rewrite our objective as follows log | K | = log | K ˜ K − 1 | + log | ˜ K | for any arbitrary p.s.d.
matrix ˜ K .
We are interested in obtaining reliable estimates of the following bound without performing the matrix inversion operation:
[Equation]
Rather than obtaining lower/upper bounds on the part in involving matrix inversion, we can reliably estimate the solution via conjugate gradients.
The solution will converge in one iteration if the condition number is close to 1, which will happen if ˜ K − 1 is an accurate estimate of K − 1 .
We brieﬂy summarize the theory behind conjugate gradients and how to compute them.
We are interested in solution to Kx
=
y , where the optimal solution x opt =
K − 1 y , where
K ∈ M n × n and y ∈ M n × 1 .
Conjugate gradients start with an initial solution x 0 , which progressively gets reﬁned and is guaranteed to converge in the worst case scenario in n iterations.
We provide the algorithm in pseudocode 1.
Input: K ∈ M n × n , y ∈ M n ×
Preconditioning matrix: M ∈ M n × n
Initial estimate: x 0 ∈ M n ×
Compute ﬁrst search direction and basis r
= Kx
− y z
=
M − 1 rp [-
-]0 = z1.
For i = 1 to n
[Equation]
For the purposes of our problem at hand, we have y =
g i and the CG-based solution x i ≈ K − 1 g i .
Then, the lower bound translates to:
[Equation]
The tightness of the lower bound in the case ˜ K ≈ K can be shown as follows:
[Equation]
where we used the fact that x i = K − 1 g i since the condition number is 1 in this scenario.
Appendix F. Additional results F.1. Lower bound recovers true approximate posteriors despite erroneous initialization
We aim to quantify the inﬂuence of ”accurate” initializations of variational parameters on the converged solution.
Whereas one would expect to initialize L Σ − 1 uu with their respective values given initial estimates of Z and kernel hyperparameters θ , we instead initialize them with identity matrices to determine if the bound can recover optimal posteriors.
The lower
[Figure]
bound manages to recover an accurate estimate of
[Equation]
The inﬂuence of the initial values of v q is of paramount importance as it can make any change to K fu,fu impossible to take, possibly leading to scenarios where the optimization prefers to keep Z and kernel hyperparamters ﬁxed.
We explore this scenario and see if the inducing points’ locations Z will get optimized to ideal locations despite starting from values outside the training set range.
From Figure 5 we can easily notice that the optimized
Z values are brought back to locations similar to ones obtained from running k-means.
[Figure]
F.2. Reliability of uncertainty estimates
We are interested to stress test our proposed model in three diﬀerent scenarios, which will individually point towards biases or discrepancies in the ﬁnal estimates for Σ ff · u and Σ fu Σ − 1 uu .
For this we use the “snelson” dataset, with the training set taken to comprise the intervals between 0.0 and 2.0, respectively 4.0 and 6.5.
Thereby, in an ideal scenario we would expect our model to oﬀer high uncertainty estimates between 2.0 and 4.0.
From
Figure 6 we can notice that indeed IF-STP does not suﬀer from pathologies.
Subsampling the “snelson” dataset allows us to discern if our method is capable to increase its uncertainty.
For this we only subsample data between 0.0 and 2.5.
Naturally, one should expect an increase in uncertainty just in that interval, with the remainder of the data range reverting back to levels seen for the full “snelson” dataset (Figure 7).
[Figure] [Figure]
F.3. Additional ﬁgures for “snelson” and “banana”
F.4. Conjugate Gradient steps during optimization