Decoupled Sparse Gaussian Processes Components : Separating Decision Making from Data Manifold Fitting
1. Introduction
Gaussian processes (GPs) are data-eﬃcient Bayesian non-parametric models that oﬀer calibrated uncertainty quantiﬁcation and are robust to overﬁtting.
Their drawback resides in the computational complexity for inverting the covariance matrix, which is cubic in computation and quadratic in memory.
To solve this issue, Hensman et al.
(2013) proposed an “inducing point” framework scalable to large data, obtaining posterior formulas conditioned on these artiﬁcial points.
However, this also scales supralinearly with regards to inducing point numbers.
Van der Wilk (2019) have shown that the parametric mean can only have
M degrees of freedom, where M is the number of inducing points.
On this note, recent work (Cheng and Boots, 2017; Salimbeni et al., 2018) have introduced a dual representation of Sparse GP (SGPs) in Reproducing Kernel Hilbert Space (RKHS), choosing a diﬀerent set of basis for the mean, respectively the variance component.
The degrees of freedom of the mean posterior equation can be increased at a linear cost with regards to number of inducing points.
Similar in scope, Shi et al.
(2020) decompose a SGP into two orthogonal components in prior function space and impose variational posteriors with diﬀerent sets of inducing points for each component.
In this paper we make the following contributions:
• Propose a new RKHS parametrization of SGPs that separates the basis for the parametric, respectively non-parametric components on a SGPs as introduced in Hensman et al.
• Empirically show that this formulation results in a set of inducing points that has the task of devising the decision boundaries and another set which exclusively focuses on ﬁtting the training data manifold.
Show the equivalence between Bayesian Kernel Ridge Regression (BKRR) and SGPs in posterior function space.
Based on insights from BKRR we propose a parametrization of the inducing points’ variance as a function of their location.
2. Background
We defer the notation conventions and introduction to SGPs to the supplementary material, while introducing the necessary theoretical background for our method.
2.1. Disentangling Uncertainties in Sparse Gaussian Processes
As noted by Hensman et al.
(2017), the variational GP posterior over function values can be divided into two components:
[Equation]
The variance pertaining to g ( · ) will be denoted as within-data uncertainty as it is given by the basis functions that control the shape of the mean, thereby it could be interpreted as the variance stemming from the parametric side of the SGP.
The variance of h ( · ) captures the shift from within to outside the data manifold as it is given by the current number of inducing points and will be denoted as distributional uncertainty.
2.2. Decoupled GPs
Cheng and Boots (2017) introduced the dual formulation of Sparse Gaussian Processes in
RKHS.
The authors motivated the Decoupled GPs by the O ( M 3 ) complexity.
They argue that the posterior mean can be decoupled from the posterior variance, when the goal is rather enhancing the predictive mean rather than obtaining better approximations to the real posterior variance of the full GP.
We deﬁne an RKHS H to be a Hilbert space with the reproducing property: ∀ x ∈ X , ∃ φ x ∈ H such that ∀ f ∈ H , f ( x ) =
φ Tx f. A GP
( m, k ) is equivalent to a Gaussian measure ν on a Banach space B which has a RKHS H .
There is a mean functional ν ∈ H and a bounded positive semi-deﬁnite linear operator Σ : H → H such that for any x, x ∗ ∈ X , ∃ φ x , φ x ∗ ∈ H , we can then write m ( x ) =
φ Tx ν and k ( x, x ∗ ) =
φ Tx Σ φ x ∗
With the deﬁnition of the dual of GPs in mind, we can now proceed to characterize a subspace parametrization as follows: µ = φ Z a and Σ
= I + φ Z Aφ TZ , where a ∈ R M and
A ∈ R M ∗ M .
3. Decoupled GP Components
Our motivation is to get a better approximation of the true distributional variance of a full
GP with minimum computational and memory overhead.
With this in mind, our functional basis parametrization for h ( · ) is:
[Equation]
whereas for the g ( · ) component we impose the following parametrization:
[Equation]
where Z = { α, β } .
The additional β inducing point locations are introduced to obtain a better ﬁt in terms of variance to the data manifold.
This comes without the introduction of additional variational parameters m β , S β that need to be optimized.
Consequently, we have the following maximization objective:
[Equation]
where q ( f ) = N ( h + g | µ h , Σ h +Σ g ), respectively the Kullback-Liebler divergence has the the following analytic form:
[Equation]
Through this RKHS parametrization, we have decoupled the inducing points of h ( · ) and g ( · ).
In all experiments we use the same kernel hyperparameters for both components.
4. Equivalence between Bayesian Kernel Ridge Regression and Sparse Gaussian Processes
Kanagawa et al.
(2018) provide an in-depth analysis on connections between kernel methods and GPs.
In this section, we seek to establish connections between BKRR and SGPs.
At testing time, the parameterization for Decoupled SGP Components (DSGPC) has the following form:
[Equation] [Equation]
[Equation]
BKRR assumes the following model
[Equation]
, where β ∼ N (0 , σ 2 λ ) and we deﬁne (cid:15) ∼ N (0 , σ 2 ).
One can obtain an analytic expression for the posterior p ( β | D ) ∼ N ( ˜ β, ˜Σ) with ˜ β
= ( X T X
+ λ I ) − 1 X T Y and ˜Σ = σ 2 ( X T X
+ λ I ) − 1 .To obtain the predictive mean and variance for a new point x ∗ we need to integrate out p ( y ∗ | x ∗ , D ) =
(cid:82) p ( y ∗ | x ∗ , β ) p ( β | D ) dβ =
N ( φ ( x ∗ ) T ˜ β, σ 2 + φ ( x ∗ ) T Σ φ ( x ∗ )).
After introducing the analytic equations for p ( β | D ) we obtain the ﬁnal form:
[Equation]
[Equation]
One can easily notice that the above equations correspond to equations 11 and 13.
A variant of equation 12 can be obtained as follows:
[Equation]
[Figure]
4.1. BKRR inspired parameterization of inducing points’ variance
From equation 16 we can notice that σ 2 X ( X T X + λ I ) − 1 X T + σ
I basically stands for S in the conditional posterior variance (see eqn.23 in supplement).
Rather than introducing additional parameters that need to be optimized, we can instead use this formulation of within-data manifold variance stemming from BKRR.
This translates into the the following posterior for inducing point values u :
[Equation]
17) where λ can be interpreted as the additional jitter added for a stable Cholesky decomposition.
5. DSGPC separate decision making from ﬁtting the data manifold
Since the α variational parameters are essentially tasked to construct the decision boundaries, the α inducing point locations are optimized to be in locations closer to the decision boundaries.
Conversely, the β inducing points locations are only tasked to ensure low distributional variance within the data manifold, hence their locations will be spread out evenly across the training set, thereby resulting in a behaviour more similar to that of a full GP (Figure 1).
The α and β inducing points’ locations of Solve-GP are concentrated around the decision boundaries.
6. Results
We evaluate our models on a range of regression and classiﬁcation benchmark tasks from the UCI machine learning dataset repository, alongside MNIST and Fashion-MNIST.
All of the experiments were run with the same initializations and with 50 and 100 inducing points with the goal of comparing our method to SGP (Hensman et al., 2013) and Solve-GP (Shi et al., 2020).
[Figure]
For large scale classiﬁcation datasets, such as Avila (n=20,867; d=10), MNIST(n=10,000; d=784) and Fashion-MNIST(n=10,000; d=784), we can observe that the DSGPC is surpassing its counterparts in all scenarios, whereas for Letters (n=20,000; d=16), Solve-GP provides improved performance.
For large scale regression we use the ”YearPredictionMSD” dataset (n=515,345; d=90).
DSGPC and SGP obtain relatively similar results with
Solve-GP lagging behind.
Furthermore, we investigative the convergence of models to the full GP log likelihood on the testing set based on number of inducing points (Figure 3).
For ”Protein”, DSGPC and Solve-GP achieve faster convergence rates compared to SGP.
Lastly, we were not able to successfully train Solve-GP on ”YearPredictionMSD”.
DSGPC manages to be closer to the Full GP solution on this dataset as well.
[Figure]
L o g
L i k e li h
oo d
[Figure]
7. Discussion
We have presented a dual formulation of SGPs in RKHS that is capable of separating the task of devising decision boundaries from the task of ﬁtting the data manifold from the point of view of inducing point locations.
This results in α inducing point locations which are situated close to the decision boundaries, whereas the β inducing point locations are centred in high density areas.
This comes in stark contrast to the case of coupled inducing points, where the inducing point locations are a mix of the aforementioned scenarios.
The introduction of an additional amount of inducing points does not add up to a doubling of variational parameters, as the β inducing points do not warrant values in function space.
Furthermore, explicitly linking the inducing points’ variance to their location results in a model where we only have to learn the mean variational parameters.
Future work should explore subspace basis formulations for learning the β inducing point locations to further reduce parameter numbers.
Another prospective research avenue resides in applying DSGPC in the context of hierarchical GPs with Wasserstein-2 kernels (Popescu et al., 2020).
The separation of inducing points’ tasks should in theory improve out-of-distribution detection due to the enhanced data manifold ﬁt in the ﬁrst layer.
In the supplementary material, we have also included an initial extension of our work to DGPs and intuitively show why the hidden layers of DGPs do not require a high number of β inducing points
Supplementary Materials Sparse Gaussian Processes
We denote the output vector Y , where each entry Y i is a noisy observation of the function
F ( x i ) for all input points X
=
( x i ) ni =1 .
We place a GP ( m, k ) prior on the stochastic function F .
We introduce inducing points Z
=
( z i ) mi =1 with inducing point function values
U =
( u i ) mi =1 .
Under standard Gaussian identities we have
[Equation]
By the deﬁnition of a sparse GP, the joint density is p ( Y, F, U ) =
p ( F | U ) p
( U ) (cid:81) n i =1 p
( Y i | F i ).
We follow the variational inference framework introduced in Hensman et al.
and maximize the lower bound on the marginal likelihood
[Equation] [Equation]
where the variational posterior is choosen as q ( F, U )
= p ( F | U ; X, Z ) q ( U ), where q ( U ) =
N ( U | m, S ).
Here, m and S are free variational parameters.
Due to the Gaussian nature of both terms we can marginalize U to arrive at q ( F ) =
(cid:82) p ( F | U ) q ( U ) =
N ( F | ˜ U, ˜Σ) where
[Equation]
Then the evidence lower bound (ELBO) can be rewritten as:
[Equation]
Derivation of Kullback-Liebler Divergence
We remind that the Kullback-Liebler divergence between two Gaussian measures q =
N ( m q , Σ q ) and p = N ( m p , Σ p ) is given by:
[Equation]
We now adapt this formula to suit our posterior q = N ( φ Zα K − 1 α m, I − φ Z K − 1 Z φ TZ + φ Zα K − 1 α S K − 1 α φ TZα ), respectively the prior p =
N (0 , I )
[Equation]
Trace term:
[Equation] [Equation]
Quadratic term:
[Equation]
Log Determinants term:
[Equation]
Derivation of posterior for BKRR
BKRR assumes the following model p ( Y | β, X ) =
Xβ + (cid:15) , where w β ∼ N (0 , σ 2 λ ) and we deﬁne (cid:15) ∼ N (0 , σ 2 )
[Equation]
From a parameter space view
[Equation]
we are trying to solve the following problem:
[Equation]
[Equation] [Equation]
Finally, one can recognize the last equation as being a Gaussian with mean ˜ β
= ( X T X
+ λ I ) − 1 X T Y , respectively variance Σ =
σ 2 ( X T X
+ λ I ) −
Derivation of variance of BKRR predictive mean
[Equation]
Comparison between BKRR and DSGPC
We can observe a similar behaviour both in within-data and distributional variance between
BKRR (Figure 4A) and DSGPC(Figure 4C).
The kernel hyperparameters of the BKRR were
[Figure]
not optimised.
Additionally, we have included a variant of DSGPC (Figure 4B) where we keep the S term free-ﬂoating in a similar manner to SGP (Hensman et al., 2013).
There are no visible diﬀerences in the two methods, motivating the usage of the parameterised S based on inducing points locations to further reduce variational parameters that need to be optimized
[Equation]
Collapsed Lower bound for Decoupled Sparse Gaussian Process Components
We now derive the collapsed DSGPC lower bound by obtaining an analytic expression for the optimal q ( u ) ∗ .
[Equation]
By expanding the ﬁrst term we obtain:
[Equation] [Equation]
[Equation]
We now plug equation 46 into equation 43 to obtain the collapsed bound
[Equation]
[Equation]
The optimal q ( u ) ∗ can be represented as N ( y | h, σ 2 ) p ( h ).
By change of variable, respectively h
= K fz K − 1 zz
u we get the equivalent:
[Equation]
One can notice that they have the same form as the ones in Titsias (2009).
[Equation]
We can notice that the optimal varaitional mean and variance are with respect to Z = { α, β } .
Decoupling extended to hierarchical architectures Motivation:
In the case of DGPs, separating the inducing point locations for g ( · ) and h (˙) is motivated by the fact that most sampled realisations of intermediate layers are centred around 0.
Hence, for inliers it suﬃces a small amount of inducing points to capture the data manifold at these intermediate layers (Figure 5).
In contrast, there is a need of signiﬁcantly more inducing point locations for the parametric part ( h ( · ))
so as to devise complex patterns.
We commence by laying the foundations of DGPs, subsequently introduce the new
RKHS parametrization for intermediate layers followed by a derivation of the ELBO for this model which we entitle Decoupled Deep Gaussian Processes Components (DDGPC).
A Deep Gaussian Process (Damianou and Lawrence, 2013) is deﬁned as a stack of shallow GPs acting as the prior:
[Equation]
) where for brevity of notation we denote Z
= X .
The sparse GPs between hidden layers are treated as being noiseless.
As the prior is analytically intractable to integrate, Salimbeni and Deisenroth (2017) have suggested to sample from each hidden layer of the DGP in order to obtain unbiased stochastic gradients.
We introduce a factorized posterior between layers and dimensions of the following form:
[Equation]
where q ( U l ) is taken to be a multivariate Gaussian with mean m l and variance S l = σ 2 Z l ( Z Tl Z l + λ I m ) − 1 Z Tl + σ 2 I M , where λ can be interpreted as the jitter noise added for stable Cholesky decomposition.
Considering that most sampled points in the hidden layers of a DGP are closely centred around 0, we propose an alternative basis parameterization for the hidden layers of a DGP.
The parametrization for the l-th layer h l ( · ) is:
[Equation]
whereas for the g ( · ) GP component we impose the following parametrization:
[Equation]
We can immediately notice that at testing time the computation cost of Σ g l is greatly reduce from O ( M α l + M β l ) 3 to O (( M β l ) 3 .
In our experiments we only consider M α l >> M β l
The KL divergence for the l-th layer is given by:
[Equation]
For the ﬁrst GP layer in the hierarchy we keep the parametrization introduced in section 3 so as to capture the full extent of the data manifold.
Lastly, the ELBO is the following:
[Equation]
,..,L [ logp θ
[Equation]
L
[Equation]
] −
[Equation]
We provide some exploratory results on UCI regression tasks (Figure 6).
We can notice a relative under performance of DDGPC and DGP-SolveGP in relation to DGPs.
Experiments were performed under a similar training routine, limited to 50,000 iterations to gain an intuition of their convergence under a strict training regime.
Future work should expand on these initial results to accommodate for adaptive training regimes until convergence.
[Figure]
Additional details for UCI experiments
For all datasets we randomly selected 20% as the testing set, with the remainder being used for training.
All implementations use the RBF kernel with automatic relevance determination.
For the shallow GP experiments in the main paper we use 50 and 100 inducing points for each model.
For the DGP experiments, in terms of model architecture, we use 2 hidden units for each hidden level, with 50, respectively 100 inducing points.
For DDGPC we use 10 β inducing points for the hidden layers.
All models are optimized for 50,000 iterations with a mini-batch of size 32 and the learning rate is set to 0.001.
Results are provided for 2, 3 and 4 layers.
Kernel
All Euclidean space kernels used the standard ARD RBF, using a lengthscale parameter per input dimension, initialized to 1.351.
We initialize the variance of the kernel with the same value.
Likelihood
For regression tasks the likelihood variance was initialized to 1.
[Table]
Orthogonality
Task
Orthogonality
SGP (Hensman et al., 2013)
M + M ( M +1)
M (cid:55) (cid:55)
SOLVE-GP (Shi et al., 2020)
M α
+
M β +
M α ( M α +1)+
M β ( M β +1)
M α + M β (cid:88) (cid:55)
[Table]
M α
M α + M β
[Table]
Prior; (cid:55) Posterior (cid:88)
Inducing points
We initialize the inducing point locations to the k-means of the training data for the ﬁrst layer, whereas for hidden layers the locations are uniformly sampled in the interval [-1.0,1.0].
Variational parameters
We initialize the mean to a zero column vector, whereas the variance is given by the lower triangular Cholesky decomposition which is initialized by the identity matrix for the last layer.
For intermediate layers, the Cholesky decomposition is initialized by the identity matrix multiplied by 0.0001.
Optimization on UCI datasets
All parameters were optimized using the Adam optimizer with a learning rate of 0.001.
We used a batch size of 32 and trained for 50, iterations.
Additional tables and ﬁgures
In this subsection we provide a table showcasing the diﬀerences between the models considered in this paper, respectively additional ﬁgures on toy regression and classiﬁcation task to highlight diﬀerences in function space and inducing points’ locations.
[Figure]
[Figure] [Figure]