Revisiting Active Sets for Gaussian Process Decoders
Abstract
Decoders built on Gaussian processes (GPs) are enticing due to the marginalisationover the non-linear function space.
Such models (also known as GP-LVMs) areoften expensive and notoriously difficult to train in practice, but can be scaled usingvariational inference and inducing points.
In this paper, we revisit active set approximations.
We develop a new stochastic estimate of the log-marginal likelihood basedon recently discovered links to cross-validation, and propose a computationallyefficient approximation thereof.
We demonstrate that the resulting stochastic activesets (SAS) approximation significantly improves the robustness of GP decodertraining while reducing computational cost.
The SAS-GP obtains more structurein the latent space, scales to many datapoints and learns better representations thanvariational autoencoders, which is rarely the case for GP decoders
Introduction
Generative models can be viewed as regression models from unknown inputs.
That is, we assumex =
f
( z ) , where f is an unknown mapping from latent variables z to observations x .
Given theinherent difficulty of this task, it is perhaps sensible to marginalize the unknown mapping f to avoidthe brittleness of point-estimates.
This is the driving idea in the Gaussian process latent variable(GP-LVM)
(Lawrence, 2005), which places a Gaussian process (GP) prior over the unknown mappingf and marginalizes accordingly.
This contrasts contemporary generative models that predominantlyoperate with point-estimates of f (Kingma and Welling, 2013; Sohl-Dickstein et al., 2015).
Whileconceptually elegant, the GP-LVM is, however, notoriously difficult to train and the conceptualbenefits are often not realized in practice.
Exact inference involves computing the marginal likelihood, but (like other GP methods) its cubiccomplexity in the number of observations O ( N 3 ) limits the scalability of the GP-LVM.
The idea ofmarginalizing the decoder is, however, sufficiently attractive to motivate the development of scalableand reliable training techniques: Its Bayesian formulation (Titsias and Lawrence, 2010) variationallyintegrates out the latent variables z to obtain an evidence lower bound.
Using auxiliary inducingvariables, Snelson and Ghahramani (2006) expanded GP regression which is also applicable in theunsupervised learning setting (i.e. the GP-LVM).
However, considering inducing variables here involves dangers.
First, the convergence of inducingpoints is well-studied in the supervised GP scenario, where inputs are fixed , but it differs from theunsupervised case where the inputs are estimated .
Bauer et al.
(2016) notes that even in the supervisedsetting, inducing points are “ not completely trivial to optimise, and often tricks [...] are required ”,and we hypothesize that this is further complicated in the unsupervised setting where we optimizeboth latent and inducing variables while they interact.
In this paper , we revisit active sets for scaling GP decoders, a sparse approximation predominantlyused before the seminal work of Snelson and Ghahramani (2006).
From a practical viewpoint, activesets are fixed inducing variables that belong to the training dataset.
We make links between such active sets and cross-validation allowing us to lean on a recent result from Fong and Holmes (2020)which, in turn, links cross validation and the log-marginal likelihood.
We show how these linksallows for a stochastic estimate of the log-marginal likelihood, and that active sets can be seen as acomputationally efficient approximation of this.
Practically, this amounts to repeatedly randomlysampling active sets rather than trying to find the optimal such set.
We denote this framework asstochastic active sets (SAS) .
We demonstrate that SAS consistently results in significantly better fitted
GP decoders over models trained using inducing points.
Historical remarks.
Methods based on subsets of data diminish the computational demand and wasfirst introduced in a GP context in the foundational work on sparse approximations by Smola and
Bartlett (2001).
Back then, Quiñonero-Candela and Rasmussen (2005) already pointed out the maindifficulties behind the optimal selection of subsets:“ Traditionally, sparse models have very often been built upon a carefully chosensubset of the training inputs.
[...] In sparse Gaussian processes it has also been suggested to select the inducing inputs X u from among the training inputs.
Since thisinvolves a prohibitive combinatorial optimization, greedy optimization approacheshave been suggested [...].
Recently, Snelson and Ghahramani (2006) have proposedto relax the constraint that the inducing variables must be a subset of training/testcases, turning the discrete selection problem into one of continuous optimization. ”
This explains how inducing variables reshaped the Gaussian process community, effectively banishingother subset-based methods.
Our work builds on the advances made in stochastic optimization in thetime since active sets were left behind.
We show a third way over those considered by QuiñoneroCandela and Rasmussen (2005): instead of optimizing the active set, we average with respect to it.
This both simplifies and robustify matters notably.
To justify our approach, we establish a link between active sets and cross validation (CV).
The latterhas a long history for model selection in GPs, dating at least to the seminal work of Wahba (1990).
Forprobabilistic models, Rasmussen and Williams (2006) point out to the utility of CV variants withinnegative log-probabilities.
Building on results from Fong and Holmes (2020) linking CV and logmarginal likelihoods, we argue that for GP-LVMs, active sets combine more gracefully with stochasticoptimization.
The remainder of this paper elaborates on this viewpoint and demonstrates it empirically.
Gaussian Process Decoders
The Gaussian process latent variable model (GP-LVM)
(Lawrence, 2005) defines a decoder which isa non-linear mapping 1 x = f ( z ) from the latent space Z ∈ R Q to observation space X ∈ R D .
Theprior on this map is a Gaussian process (GP) so it is drawn like f ∼ GP (0 , k θ ( · , · )) , where k θ is the 70 covariance function or kernel and K
NN denotes the evaluated kernel function so the i, j th elementof K NN equals k ( z i , z j ) .
For clarity, we omit the dependence on covariance function parameters, θ .
The original version of the GP-LVM start from one-dimensional observations x = { x n } Nn =1 and 73 latent variables z
= { z n } Nn =1 , and factorise the joint distribution of the model as p ( x , f | z ) = 74 p ( x | f, z ) p ( f | z ) .
Here the conditional distributions correspond to the likelihood model and the priorp (
[Equation]
|
[Equation]
When the data dimensionality is D> 1 , the model factorises across dimensions, and we have differentmappings f per d th feature.
One of the principal assumptions of the GP-LVM is that the prior p ( f )regularises the smoothness of all mappings equally, so we only consider one kernel .
This assumptioncan be relaxed if needed, but at increased computational cost and more learnable hyperparameters
Mapping marginalization.
A GP prior over the non-linear decoder f allows for marginalisation off to obtain a closed-form expression of the marginal likelihood of the GP-LVM
[Equation]
On a log -scale, this gives the following objective function (Lawrence, 2004), which can be optimizedw.r.t.
both hyperparameters θ and latent variables z
[Equation]
The difficulties of training the GP-LVM using this objective function are evident above as theevaluation cost grows cubically with the number of observations N .
Furthermore, notice thatobservations x are no longer independent (in contrast with Eq. 1) once f is integrated out.
Thelog-marginal likelihood will be the starting point for our approach in Sec. 3.
Bayesian extension.
In the seminal works of Lawrence (2004, 2005), the GP-LVM is first derived as anon-linear extension of probabilistic principal component analysis ( PPCA ) (Tipping and Bishop, 1999).
Considering the isotropic prior on the latent variables z , such that p ( z n ) =
N ( z n | 0 , I ) ∀
z n ∈ z , 90 the general idea is to optimize them rather than introducing marginalization.
From a full probabilisticperspective, one can be also interested in the posterior distribution over z , which leads to use Bayesianinference for the GP-LVM approach.
This is the driving idea of Titsias and Lawrence (2010), wherevariational methods are introduced.
In particular, latent variables are not easy to marginalize, mainlydue to their presence in the kernel mappings, so a lower-bound on the log-marginal likelihoodlog p ( x ) = log (cid:82) p
x | z ) p ( z )d z of the model is derived.
So far, the Bayesian GP-LVM model (Titsias and Lawrence, 2010) has been considered as thestandard methodology to apply GPs to large unsupervised datasets with 10 4 − 10 6 observations, e.g.in regression (Bui and Turner, 2015), classification (Gal et al., 2015) and representation learning(Märtens et al., 2019) tasks.
Stochastic Active Sets
[Figure]
Our key objective is a computationally efficient estimate of the log-marginal likelihood in Eq.
2, asthis is known to be a good measure of generalization performance (Rasmussen and Ghahramani, 2000;
Germain et al., 2016).
Another popular measure of generalization performance is cross validation(CV) (Geisser and Eddy, 1979; Vehtari and Lampinen, 2002), which is arguably most used outside therealm of Bayesian models.
Recently, Fong and Holmes (2020) linked these two measures, effectivelyshowing that the marginal likelihood is equivalent to the average over exhaustive leave- R -out CVscores.
In particular, the average is w.r.t.
the size of the hold-out set.
More precisely, let
[Equation]
denote leave- R -out CV using log-predictive scoring functions log p ( x n | x A p , z ) .
Here A p denotesthe active set indices of the training data, such that A p ⊂ { 1 , 2 , . . .
, N } and R p
= { 1 , 2 , . .
. , N }\A p 110 are the remaining hold-out samples.
The subindex p ∈ C denotes the permutation and we averageover all C = (cid:0) NR (cid:1) possible hold-out sets.
We use use R to indicate the size of the hold-out set
R p ( R ≡ |R p | ) and let A = |A p | = N − R .
If we average S CV ( x | R ) over all possible sizes of 113 the hold-out set, then Fong and Holmes (2020) has shown that we recover the exact log-marginallikelihood,
[Equation]
Here S PCV ( x | R ) =
E A [log p ( x A | z A )] and S CCV ( x | R ) =
(cid:80)
Rr =1 S CV ( x | r ) is the cumulative CV 116 score, which reduces to a sum of expectations over the predictive factors.
Further details on
S PCV and
S CCV can be found in the Appendix.
Fong and Holmes (2020) used Eq.
4 to argue in favor of usingthe marginal likelihood over cross-validation for model selection.
Stochastic approximation.
We take a slightly different view than Fong and Holmes (2020) andargue that Eq.
4 can be the grounds for an efficient stochastic gradient (Robbins and Monro, 1951)of the log-marginal likelihood suitable for training.
In the context of GPs, we have that conditionalprobabilities for S PCV ( x | R ) and S CCV ( x | R ) in Eq.
4 are
[Equation]
where we used Eq.
2.22 from Rasmussen and Williams (2006) to obtain
[Equation]
and K AA ∈ R
A × A has entries k
( z i , z j ) with z i , z j ∈ z A .
The computational cost of S PCV ( x | R ) is
O ( A 3 ) , while log p ( x n | x A , z ) can reuse the matrix inversion K − 1 AA to only have an additional linearcost.
Clearly, we can obtain an unbiased stochastic estimate of the log-marginal likelihood by firstuniformly sampling R , second making a random split permutation into train and hold-out data R andfinally evaluating
[Equation]
where we remark that the summation can be mini-batched.
This approach is equivalent to thedecomposition log p ( x | z )
= log p ( x R | x A , z ) + log p ( x A | z A ) and also assuming conditional inde- 131 pendence among observations x n for n ∈R .
This is similar to the standard active set approximation(Seeger et al., 2003) previously discussed, and we may think of A as a stochastic active set (SAS) .
However, the estimate of log p ( x | z ) still has the same computational complexity O ( N 3 ) as theusual deterministic approach, since we may sample A
= N in the worst case.
Instead, we proposeto make an stochastic approximation where we choose the size of the active set deterministicallythrough a user-specified parameter, such that the computational cost reduces to O
( A 3 ) , as in sparseapproximations based on inducing points.
This does not need to be unbiased; empirically we find thatmost often the approximation behaves as a lower bound to the true marginal likelihood, and that itin all instances is a rather close approximation.
We include a longer discussion on this point in the
Appendix and the training methodology using SAS is in Alg. 1.
3.1 Extension for Bayesian GP decoders
We next seek to extend the previous SAS approach to the Bayesian GP-LVM, where we aim to obtainthe posterior distribution p ( z | x ) .
From this perspective, we are interested on marginalising latentvariables z to obtain the marginal likelihood p ( x ) of the model.
However, this integration is notpossible as latent variables appear non-linearly in the kernel function (Titsias and Lawrence, 2010).
Alternatively, we consider the variational inference scheme, where an auxiliary distribution q ( z ) isintroduced into the formulation.
Therefore, we are able to build the evidence lower bound (ELBO) ofthe model using Jensen’s inequality as
[Equation]
[Equation]
) which is equivalent to the one obtained by Titsias and Lawrence (2010, Eq.
At this point, thecomputational cost is O ( N 3 ) , since the ELBO requires evaluating log p ( x | z ) where we invert K NN .
The expectation in Eq.
7 can also benefit from a stochastic SAS approximation, just as with inducingpoints (Hensman et al., 2013).
Thus, the lower bound can be approximated as
[Equation]
where we consider mean-field VI to factorize the distribution q ( z ) .
The Bayesian GP-LVM shareshigh-level similarities with other generative models that marginalize the latent variable according to asimple prior (Rezende and Mohamed, 2015).
The proposed SAS approximation (8) scales similarlyto mini-batched inducing point approximations, but we will later see that SAS behaves notably betterin practice.
Algorithmically, the approach is simple, and summary code is provided in Alg. 2.
1: Input: Observed data x 2: Parameters: Initialize θ , ϕ //
θ , z if NA 3: for e in epochs do 4: for b in batches do 5:
Sample x batch ∼ x 6: x R , x A ← random_split ( x batch ) 7: if amortized then 8:
Get { z R ,
z A } ← g ( x R , x A | ϕ ) 9: end if 10:
Compute K − 1 AA // via Cholesky 11: Evaluate log p ( x A | z A ) 12:
Evaluate log p ( x n | x A , z ) , ∀ x n ∈ x R 13:
Evaluate Eq.14: do Adam( θ , ϕ ) step for L 15: end for 16: end for
NA : Non-amortized.
: Input: do in batches do 5:
Sample x batch ∼ x 6: x R , x A ← random_split ( x batch ) 7: if amortized then 8:
Get µ z ← g µ ( x R , x A | ϕ ) 9:
Get σ z ← g σ ( x R , x A | ϕ ) 10: end if 11:
Sample { z R , z A } ∼ q ( µ z , σ z ) //
RT 12:
Compute K − 1 AA // via Cholesky 13:
Evaluate L in Eq.
8 14: do Adam( θ , ϕ ) step for L ELBO 15: end for 16: end for
NA : Non-amortized, RT : Reparametrization trick.
3.2 The Role of Amortization
Early after the emergence of the seminal GP-LVM (Lawrence, 2005), the lengthy optimizationrequired for obtaining all latent representations z n became noticeable.
An additional considerationis that while most approaches on non-linear low-dimensionality reduction focus on preservingsimilarities, the GPLVM does the opposite.
This property was initially discussed by Lawrence and
Quiñonero-Candela (2006), since in some sense the GP-LVM is dissimilarity preserving , such thatdifferent observations will generally be represented far away from each other.
In practice, we areoften more interested in embeddings that reflect the true distance between the observed objects intheir representations, particularly those ones that are close together.
This observation inspired backconstraints for locality preservation (Lawrence and Quiñonero-Candela, 2006), which enforces latentvariables z to be an explicit function of observations
z = g ( x | ϕ ) parameterized by ϕ .
This is similarto the stochastic encoder applied in variational autoencoders (Kingma and Welling, 2013; Rezende and
Mohamed, 2015).
This idea was also extended to the VI framework in GP-LVMs by Bui and Turner(2015) using a recognition model, e.g. q ( z n )=
N ( z n | g µ ( x n | ϕ ) , g σ ( x n | ϕ )) and more recently, to 173 accelerate hyperparameter learning in GPs with hierarchical attention networks (Liu et al., 2020).
In our context, we assume the mappings to be neural networks (NNs) like Bui and Turner (2015),referring to them as amortization networks.
We find such networks accelerate learning very nicelywhen used in conjunction with SAS.
It is also worth noting that amortization has been empiricallyshown to improve generalization performance (Shu et al., 2018).
Related Work
Marginal likelihood approximations were used in GPs (Smola and Bartlett, 2001; Csató and Opper,2002) before the apparition of pseudo-inputs (Snelson and Ghahramani, 2006) and the associated variational inference framework (Titsias, 2009).
In the former case, stochastic approximations to the
ELBO were first presented by Hensman et al.
(2013, 2015), in line with the Bayesian counterpartof SAS.
In terms of active sets, Seeger et al.
(2003) empirically observed that the approximationwas generally stable for optimisation even if the size of A was a small fraction of the training sizeonly.
However, they also observed that random selection of active sets led to non-smooth fluctuations,making it difficult to converge through exact gradient ascent.
Particularly, the issue with reselectingof A motivated Snelson and Ghahramani (2006) as a smoother optimization alternative and we findthat SAS also circumvents this problem via stochastic estimates as shown in Sec. 5.
The connection between cross-validation and GPs was already described in Rasmussen and Williams(2006) as an alternative for model selection.
However, the equivalence between exahustive CV andthe log-marginal likelihood provided in Fong and Holmes (2020) provides a novel perspective thatwe exploit.
Additionally, the notion of back constraints has been recently considered in Lalchandet al.
(2022) for GP-LVMs with inducing points, where a doubly stochastic stochastic formulationis used.
More recently, amortization networks has been used to drastically reduce the number ofinducing points in supervised GPs.
Experiments
In this section, we evaluate the performance of the SAS approach for stochastic learning of GPdecoders, both the deterministic GP-LVM (Lawrence, 2004) and its Bayesian counterpart (Titsias and Lawrence, 2010).
For this purpose, we consider three different datasets: MNIST (LeCun et al., 1998),
FASHION MNIST (Xiao et al., 2017) and CIFAR -10 (Krizhevsky et al., 2009).
For a fair comparisonof our model with baseline methods, we use the same amortization across models, namely a neuralnetwork (three linear layers ReLU activation functions 3 ) for all models in all experiments and all
GPs use an quadratic exponentiated kernel. In all experiments, the learning rates are set in the range[10 − 4 , 10 − 2 ] , the maximum number of epochs considered is 300 and we used the ADAM optimizer(Kingma and Ba, 2015). For SAS experiments, we only consider batch sizes greater than the activeset size as this is a requirement for SAS.
Performance metrics of the SAS-GP decoders are given in terms of the negative log-predictive density( NLPD ), root mean-square error ( RMSE ) and mean absolute error ( MAE ).
In all cases, we optimizew.r.t.
an approximation to the log-marginal likelihood log p ( x | z ) in the deterministic scenario or w.r.t.a lower bound on the log p ( x ) of the model in the Bayesian cases.
We also provide P YTORCH codethat allows for reproducing all experiments and models.
We monitor the run-time of convergenceas we suspect that rotating active sets (see Fig. 1) across the dataset is a fast way to capture thecorrelation structure of the regression problem.
5.1 Representation performance
First, we analysed our SAS approach and the approximate optimization procedure on an unsupervisedversion of MNIST , FMNIST and CIFAR -10, where we took the full training corpus for learning twodimensional latent representations of images.
The approximation curves are included in Fig.
2, wherewe observe convergence in less that 2h runtime in CPU for most cases.
For all experiments, we canobserve that for larger active set sizes A , the SAS approaches takes longer time to complete the 300epochs as the computational cost of inversion is higher.
Evaluation with SOTA methods.
We also tested the performance o representation learning in thestate-of-the-art methods with and without GPs for unsupervised scenarios.
Results are shown in Fig.4.
We include a short description of the models considered:
• SAS - GP DECODER — It uses stochastic active sets to approximate the log-marginal likeli-
hood log p ( x | z ) .
The training methodology is described in Alg. 1.
• B AYESIAN SAS - GP DECODER — It uses stochastic active sets to approximate the ELBO in
Eq.
(8) on log p ( x ) .
See Alg. 2.
for Bayesian SAS.
We fix the batch size in Alg.
1 to be B = 1024 and study the convergence for different active set sizes A . All values in the curves displayed are per-epoch . [Figure] z –
[Figure]
ec o ndd i
[Figure]
e n s i o n
[Figure]
[Table] =</latexit> [Figure] <latexit sha1_base64="q7lwc3TYTwlb1dOXIsCN45KzWl4=">AAAB+XicbVDLTsMwEHTKq5RXAIkLF4sKiVOVIAQcK7hwLBJ9SG1UOe6mseo8ZG8QVejHwAkBN36DH+BvcEsO0DKn8cxYuzt+KoVGx/mySkvLK6tr5fXKxubW9o69u9fSSaY4NHkiE9XxmQYpYmiiQAmdVAGLfAltf3Q99dv3oLRI4jscp+BFbBiLQHCGRurbBz2EB9Q8D5gOjUKj2Ayd9O2qU3NmoIvELUiVFGj07c/eIOFZBDFyybTuuk6KXs4UCi5hUullGlLGR2wIXUNjFoH28tn+E3ocJIpiCHT2/p3NWaT1OPJNJmIY6nlvKv7ndTMMLr1cxGmGEHMTMV6QSYoJndZAB0IBRzk2hHElzJaUh0wxjqasijnfnT92kbROa+55zb09q9aviiLK5JAckRPikgtSJzekQZqEk0fyTN7Iu5VbT9aL9foTLVnFn33yB9bHN6uUlEQ
[Figure] [Figure]
and
FMNIST . • B AYESIAN GP - LVM — It is based on the model in Titsias and Lawrence (2010). Means andvariance parameters are generated by an amortization NN as in Bui and Turner (2015).
Themodel is trained using stochastic variational inference (Hensman et al., 2013).• GP - LVM — We used the model proposed by Lawrence (2005) enhanced with a NN encodingto the latent space.
The model is trained using maximum likelihood (ML).• V ARIATIONAL AUTOENCODER ( VAE ) — (Kingma and Welling, 2013)
The NN encoder hasthe same architecture as the amortization function used in the SAS-GP models.
For the SAS-GP decoder and GP-LVM [Table] a neural network encodes the latent variables.
Likewise,for the Bayesian SAS-GP decoder and the Bayesian GP-LVM, two neural networks encode eachthe latent means and latent variances.
We refer to this encoding as amortization .
All models uses aGaussian likelihood 5 .
[Table]
5.2 Evaluation metrics
In this section, we are interested in the evaluation of the GP decoder with the standard error metricsused for GPs.
In Tab. 1 we provide RMSE , MAE and NLPD for the three dataset considered in thisexperiments.
Interestingly, the performance usually improves with larger active sets sizes A , as the
SAS model captures captures the underlying correlation of datapoints better and this enhances theapproximation.
This happens for both deterministic and Bayesian cases.
Additionally, we observethat the NLPD metric is generally better for smaller active set sizes A .
This can be also noticed in Fig.2, where loss curves have a better convergence for the lowest A .
Classification accuracy.
We are interested in evaluating the structure of the representation.
Forthis purpose, we trained a (one) nearest neighbour classifier on the encoded, two-dimensional latentvariables and we tested the accuracy using encoded test data.
Tab. 2 shows the mean and standarddeviations of test the accuracy
[Figure]
[Figure]
[Table]
[Figure]
<latexit sha1_base64="V3zRl+208uNrhHF/LOIRPQDCqPI=">AAAB4nicbZDLTgJBEEVr8IX4Ql266UhMXJEZY9Ql6sYlJCIkMCE9TQ106Hmku8aEEH5AV0bd+Un+gH9jg7NQ8K5O172d1K0gVdKQ6345hZXVtfWN4mZpa3tnd6+8f/BgkkwLbIpEJbodcINKxtgkSQrbqUYeBQpbweh25rceURuZxPc0TtGP+CCWoRSc7Khx3StX3Ko7F1sGL4cK5Kr3yp/dfiKyCGMSihvT8dyU/AnXJIXCaambGUy5GPEBdizGPELjT+aLTtlJmGhGQ2Tz9+/shEfGjKPAZiJOQ7PozYb/eZ2Mwit/IuM0I4yFjVgvzBSjhM36sr7UKEiNLXChpd2SiSHXXJC9SsnW9xbLLsPDWdW7qHqN80rtJj9EEY7gGE7Bg0uowR3UoQkCEJ7hDd6dvvPkvDivP9GCk/85hD9yPr4Bix+KYA==</latexit>
A
[Table]
Runtime and convergence of SAS approximation.
The loss curves obtained for the (sparse)
Bayesian GP-LVM model and the Bayesian SAS-GP decoder are shown in Fig.
We observethat the two methods scale similarly, but that SAS is faster by a notable constant within stochasticoptimization.
Additionally, we observe the performance of the SAS approximation to closely fit theexact log-marginal likelihood log p ( x | z ) .
For the plot in Fig.
5, we computed the exact probabilityfor a subset of MNIST N =40 , 000 samples.
Conclusion
State-of-the-art representation learning is generally based on neural networks, as this allows forscaling to large datasets.
However, often we want reliable uncertainty estimates from the model and wecan achieve that with Gaussian process decoders if we can scale them sufficiently.
We have reviewedthe main difficulties to obtain decent performances with GP-LVM approaches when applied to largescale learning, even with inducing variables.
Revisiting active set approximations, we considered astochastic viewpoint to approximate the marginal likelihood while simultaneously keeping the modelmarginalized.
We formulate our stochastic active sets (SAS) approach for both deterministic and
Bayesian versions of GP decoders.
We find that our approach works well with amortization, suchthat a neural network encoder approximately inverts the GP decoder.
While amortization also helpswhen using inducing points, we find the combination with SAS to be particularly efficient and robust.
Empirically, we illustrated the advantage of our method first on image-based observations, where ourapproach learns better representations using fewer computational resources compared to inducingpoint methods.
We further demonstrate that our approach easily scales to nearly 10 6 observations.
Inthis experiment, we find that the learnt representations are qualitatively on par with those attainedby a comparable autoencoder.
This is an important finding as, beyond small datasets, GP decodersgenerally recovers less useful representations compared with models based on neural networks.
From this result, we speculate that improvements in training might be enough to get state-of-the-artrepresentations with GP decoders.
Additional benefits.
Besides the empirical benefits demonstrated by SAS in the previous section,we have also observed other practical benefits worth reporting.
First , we have observed that SASeasily runs in 32-bit numerical precision, unlike inducing point methods that generally requires64-bits of precision (when reporting running times we consistently used 64 bits).
Similarly, the jitterusually added to the Cholesky factorization is of less importance in SAS.
Second , we note that ourimplementation is surprisingly free of ‘tricks’ and no numerical heuristics were needed to realize areliable implementation.
Limitations and future work.
Stochastic active sets rely on the Gaussian likelihood and this isperhaps the strongest limitation.
This works well for continuous data, but many data sources areinherently discrete and this requires a suitable likelihood, e.g. the discretized mixture of logistics(Salimans et al., 2017).
Having more powerful likelihoods would surely improve the GP decoders,but this requires further developments to realize using SAS.
Future work will focus on applying the SAS approach the supervised setting as well and building
SAS-like methods for discrete data.
Other possible directions include extending the decoder withdeep kernels (Wilson et al., 2016) to capture more features in the data and applying convolutional
GPs (Van der
Wilk et al., 2017) which are more suited to high-dimensional images.