Revisiting Active Sets for Gaussian Process Decoders
Anonymous Author(s)
Abstract
Decoders built on Gaussian processes (GPs) are enticing due to the marginalisationover the non-linear function space.
Such models (also known as GP-LVMs) areoften expensive and notoriously difﬁcult to train in practice, but can be scaled usingvariational inference and inducing points.
In this paper, we revisit active set approximations.
We develop a new stochastic estimate of the log-marginal likelihood basedon recently discovered links to cross-validation, and propose a computationallyefﬁcient approximation thereof.
We demonstrate that the resulting stochastic activesets (SAS) approximation signiﬁcantly improves the robustness of GP decodertraining while reducing computational cost.
The SAS-GP obtains more structurein the latent space, scales to many datapoints and learns better representations thanvariational autoencoders, which is rarely the case for GP decoders.
Introduction
Generative models can be viewed as regression models from unknown inputs.
That is, we assumex =
f
( z ) , where f is an unknown mapping from latent variables z to observations x .
Given theinherent difﬁculty of this task, it is perhaps sensible to marginalize the unknown mapping f to avoidthe brittleness of point-estimates.
This is the driving idea in the Gaussian process latent variable(GP-LVM)
(Lawrence, 2005), which places a Gaussian process (GP) prior over the unknown mappingf and marginalizes accordingly.
This contrasts contemporary generative models that predominantlyoperate with point-estimates of f (Kingma and Welling, 2013; Sohl-Dickstein et al., 2015).
Whileconceptually elegant, the GP-LVM is, however, notoriously difﬁcult to train and the conceptualbeneﬁts are often not realized in practice.
Exact inference involves computing the marginal likelihood, but (like other GP methods) its cubiccomplexity in the number of observations O ( N 3 ) limits the scalability of the GP-LVM.
The idea ofmarginalizing the decoder is, however, sufﬁciently attractive to motivate the development of scalableand reliable training techniques: Its Bayesian formulation (Titsias and Lawrence, 2010) variationallyintegrates out the latent variables z to obtain an evidence lower bound.
Using auxiliary inducingvariables, Snelson and Ghahramani (2006) expanded GP regression which is also applicable in theunsupervised learning setting (i.e. the GP-LVM).
However, considering inducing variables here involves dangers.
First, the convergence of inducingpoints is well-studied in the supervised GP scenario, where inputs are ﬁxed , but it differs from theunsupervised case where the inputs are estimated .
Bauer et al.
(2016) notes that even in the supervisedsetting, inducing points are “ not completely trivial to optimise, and often tricks [...] are required ”,and we hypothesize that this is further complicated in the unsupervised setting where we optimizeboth latent and inducing variables while they interact.
In this paper , we revisit active sets for scaling GP decoders, a sparse approximation predominantlyused before the seminal work of Snelson and Ghahramani (2006).
From a practical viewpoint, activesets are ﬁxed inducing variables that belong to the training dataset.
We make links between such active sets and cross-validation allowing us to lean on a recent result from Fong and Holmes (2020)which, in turn, links cross validation and the log-marginal likelihood.
We show how these linksallows for a stochastic estimate of the log-marginal likelihood, and that active sets can be seen as acomputationally efﬁcient approximation of this.
Practically, this amounts to repeatedly randomlysampling active sets rather than trying to ﬁnd the optimal such set.
We denote this framework asstochastic active sets (SAS) .
We demonstrate that SAS consistently results in signiﬁcantly better ﬁtted
GP decoders over models trained using inducing points.
Historical remarks.
Methods based on subsets of data diminish the computational demand and wasﬁrst introduced in a GP context in the foundational work on sparse approximations by Smola and
Bartlett (2001).
Back then, Quiñonero-Candela and Rasmussen (2005) already pointed out the maindifﬁculties behind the optimal selection of subsets:“ Traditionally, sparse models have very often been built upon a carefully chosensubset of the training inputs.
[...] In sparse Gaussian processes it has also been suggested to select the inducing inputs X u from among the training inputs.
Since thisinvolves a prohibitive combinatorial optimization, greedy optimization approacheshave been suggested [...].
Recently, Snelson and Ghahramani (2006) have proposedto relax the constraint that the inducing variables must be a subset of training/testcases, turning the discrete selection problem into one of continuous optimization. ”
This explains how inducing variables reshaped the Gaussian process community, effectively banishingother subset-based methods.
Our work builds on the advances made in stochastic optimization in thetime since active sets were left behind.
We show a third way over those considered by QuiñoneroCandela and Rasmussen (2005): instead of optimizing the active set, we average with respect to it.
This both simpliﬁes and robustify matters notably.
To justify our approach, we establish a link between active sets and cross validation (CV).
The latterhas a long history for model selection in GPs, dating at least to the seminal work of Wahba (1990).
Forprobabilistic models, Rasmussen and Williams (2006) point out to the utility of CV variants withinnegative log-probabilities.
Building on results from Fong and Holmes (2020) linking CV and logmarginal likelihoods, we argue that for GP-LVMs, active sets combine more gracefully with stochasticoptimization.
The remainder of this paper elaborates on this viewpoint and demonstrates it empirically.
Gaussian Process Decoders
The Gaussian process latent variable model (GP-LVM)
(Lawrence, 2005) deﬁnes a decoder which isa non-linear mapping 1 x = f ( z ) from the latent space Z ∈ R Q to observation space X ∈ R D .
Theprior on this map is a Gaussian process (GP) so it is drawn like f ∼ GP (0 , k θ ( · , · )) , where k θ is the 70 covariance function or kernel and θ its corresponding hyperparameters.
The original version of the GP-LVM start from one-dimensional observations x = { x n } Nn =1 and 72 latent variables z
= { z n } Nn =1 , and factorise the joint distribution of the model as p ( x , f | z ) = 73 p ( x | f, z ) p ( f | z ) .
Here the conditional distributions correspond to the likelihood model and the priorp ( x | f,
[Equation]
= N (cid:89) [Equation]
[Equation]
When the data dimensionality is D> 1 , the model factorises across dimensions, and we have differentmappings f per d th feature.
One of the principal assumptions of the GP-LVM is that the prior p ( f )regularises the smoothness of all mappings equally, so we only consider one kernel .
This assumptioncan be relaxed if needed, but at increased computational cost and more learnable hyperparameters.
Mapping marginalization.
A GP prior over the non-linear decoder f allows for marginalisation off to obtain a closed-form expression of the marginal likelihood of the GP-LVM
[Equation]
On a log -scale, this gives the following objective function (Lawrence, 2004), which can be optimizedw.r.t.
both hyperparameters θ and latent variables z
[Equation]
The difﬁculties of training the GP-LVM using this objective function are evident above as theevaluation cost grows cubically with the number of observations N .
Furthermore, notice thatobservations x are no longer independent (in contrast with Eq. 1) once f is integrated out.
Thelog-marginal likelihood will be the starting point for our approach in Sec. 3.
Bayesian extension.
In the seminal works of Lawrence (2004, 2005), the GP-LVM is ﬁrst derived as anon-linear extension of probabilistic principal component analysis ( PPCA ) (Tipping and Bishop, 1999).
Considering the isotropic prior on the latent variables z , such that p ( z n ) =
N ( z n | 0 , I ) ∀
z n ∈ z , 89 the general idea is to optimize them rather than introducing marginalization.
From a full probabilisticperspective, one can be also interested in the posterior distribution over z , which leads to use Bayesianinference for the GP-LVM approach.
This is the driving idea of Titsias and Lawrence (2010), wherevariational methods are introduced.
In particular, latent variables are not easy to marginalize, mainlydue to their presence in the kernel mappings, so a lower-bound on the log-marginal likelihoodlog p ( x ) = log (cid:82) p
x | z ) p ( z )d z of the model is derived.
So far, the Bayesian GP-LVM model (Titsias and Lawrence, 2010) has been considered as thestandard methodology to apply GPs to large unsupervised datasets with 10 4 − 10 6 observations, e.g.in regression (Bui and Turner, 2015), classiﬁcation (Gal et al., 2015) and representation learning(Märtens et al., 2019) tasks.
Stochastic Active Sets
Our key objective is a computationally efﬁcient estimate of the log-marginal likelihood in Eq.
2, asthis is known to be a good measure of generalization performance (Rasmussen and Ghahramani, 2000;
Germain et al., 2016).
Another popular measure of generalization performance is cross validation(CV) (Geisser and Eddy, 1979; Vehtari and Lampinen, 2002), which is arguably most used outside therealm of Bayesian models.
Recently, Fong and Holmes (2020) linked these two measures, effectivelyshowing that the marginal likelihood is equivalent to the average over exhaustive leave- R -out CVscores.
In particular, the average is w.r.t.
the size of the hold-out set.
More precisely, let
[Equation]
denote leave- R -out CV using log-predictive scoring functions log p ( x n | x A p , z ) .
Here A p denotesthe active set indices of the training data, such that A p ⊂ { 1 , 2 , . . .
, N } and R p
= { 1 , 2 , . .
. , N }\A p 109 are the remaining hold-out samples.
The subindex p ∈ C denotes the permutation and we averageover all C = (cid:0) N R (cid:1) possible hold-out sets.
We use use R to indicate the size of the hold-out set R andlet A = dim ( A ) =
N − R .
If we average S CV ( x | R ) over all possible sizes of the hold-out set, then 112 Fong and Holmes (2020) has shown that we recover the exact log-marginal likelihood,
[Equation]
Here S PCV ( x | R ) =
E A [log p ( x A | z A )] and S CCV ( x | R ) =
(cid:80)
Rr =1 S CV ( x | r ) is the cumulative CV 114 score, which reduces to a sum of expectations over the predictive factors.
Further details on
S PCV and
S CCV can be found in the Appendix.
Fong and Holmes (2020) used Eq.
4 to argue in favor of usingthe marginal likelihood over cross-validation for model selection.
Stochastic approximation.
We take a slightly different view than Fong and Holmes (2020) andargue that Eq.
4 can be the grounds for an efﬁcient stochastic gradient (Robbins and Monro, 1951)of the log-marginal likelihood suitable for training.
In the context of GPs, we have that conditionalprobabilities for S PCV ( x | R ) and S CCV ( x | R ) in Eq.
4 are
[Equation]
where we used Eq.
2.22 from Rasmussen and Williams (2006) to obtain
[Equation]
and K AA ∈ R
A × A has entries k
( z i , z j ) with z i , z j ∈ z A .
The computational cost of S PCV ( x | R ) is
O ( A 3 ) , while log p ( x n | x A , z ) can reuse the matrix inversion K − 1 AA to only have an additional linearcost.
Clearly, we can obtain an unbiased stochastic estimate of the log-marginal likelihood by ﬁrstuniformly sampling R , second making a random split permutation into train and hold-out data R andﬁnally evaluating
[Equation]
where we remark that the summation can be mini-batched.
This approach is equivalent to thedecomposition log p ( x | z )
= log p ( x R | x A , z ) + log p ( x A | z A ) and also assuming conditional inde- 129 pendence among observations x n for n ∈R .
This is similar to the standard active set approximation(Seeger et al., 2003) previously discussed, and we may think of A as a stochastic active set (SAS) .
However, the estimate of log p ( x | z ) still has the same computational complexity O ( N 3 ) as theusual deterministic approach, since we may sample A
= N in the worst case.
Instead, we proposeto make an stochastic approximation where we choose the size of the active set deterministicallythrough a user-speciﬁed parameter, such that the computational cost reduces to O
( A 3 ) , as in sparseapproximations based on inducing points.
This does not need to be unbiased; empirically we ﬁnd thatmost often the approximation behaves as a lower bound to the true marginal likelihood, and that itin all instances is a rather close approximation.
We include a longer discussion on this point in the
Appendix and the training methodology using SAS is in Alg. 1.
3.1 Extension for Bayesian GP decoders
We next seek to extend the previous SAS approach to the Bayesian GP-LVM, where we aim to obtainthe posterior distribution p ( z | x ) .
From this perspective, we are interested on marginalising latentvariables z to obtain the marginal likelihood p ( x ) of the model.
However, this integration is notpossible as latent variables appear non-linearly in the kernel function (Titsias and Lawrence, 2010).
Alternatively, we consider the variational inference scheme, where an auxiliary distribution q ( z ) isintroduced into the formulation.
Therefore, we are able to build the evidence lower bound (ELBO) ofthe model using Jensen’s inequality as
[Equation]
(7) which is equivalent to the one obtained by Titsias and Lawrence (2010, Eq.
At this point, thecomputational cost is O ( N 3 ) , since the ELBO requires evaluating log p ( x | z ) where we invert K NN .
The expectation in Eq.
7 can also beneﬁt from a stochastic SAS approximation, just as with inducingpoints (Hensman et al., 2013).
Thus, the lower bound can be approximated as
[Equation]
where we consider mean-ﬁeld VI to factorize the distribution q ( z ) .
The Bayesian GP-LVM shareshigh-level similarities with other generative models that marginalize the latent variable according to asimple prior (Rezende and Mohamed, 2015).
The proposed SAS approximation (8) scales similarlyto mini-batched inducing point approximations, but we will later see that SAS behaves notably betterin practice.
Algorithmically, the approach is simple, and summary code is provided in Alg. 2.
1: Input 2: Parameters: Initialize θ φ / θ z
NA 3: for do 4: for b batches do 5:
Sample x batch ∼ x 6: x R x A ← random_split ( x batch ) 7: if amortized then 8:
Get {
) 9: end if 10: Compute
AA via
Evaluate p z Evaluate log p ( x x A , z x x
Evaluate 14: do Adam θ , φ ) step for L 15: end for 16: end for
NA :
: for b in batches do 5:
Sample x batch ∼ x 6: x R , x A ← random_split ( x batch ) 7: if amortized then 8:
Get µ z ← g µ ( x R , x A | φ ) 9:
Get σ z ← g σ ( x R , x A | φ ) 10: end if 11:
Sample { z R , z A } ∼ q ( µ z , σ z ) //
RT 12:
Compute K − 1 AA // via Cholesky 13:
Evaluate L in Eq.
8 14: do Adam( θ , φ ) step for L ELBO 15: end for 16: end for
NA : Non-amortized, RT : Reparametrization trick.
3.2 The Role of Amortization
Early after the emergence of the seminal GP-LVM (Lawrence, 2005), the lengthy optimizationrequired for obtaining all latent representations z n became noticeable.
An additional considerationis that while most approaches on non-linear low-dimensionality reduction focus on preservingsimilarities, the GPLVM does the opposite.
This property was initially discussed by Lawrence and
Quiñonero-Candela (2006), since in some sense the GP-LVM is dissimilarity preserving , such thatdifferent observations will generally be represented far away from each other.
In practice, we areoften more interested in embeddings that reﬂect the true distance between the observed objects intheir representations, particularly those ones that are close together.
This observation inspired backconstraints for locality preservation (Lawrence and Quiñonero-Candela, 2006), which enforces latentvariables z to be an explicit function of observations
z = g ( x | φ ) parameterized by φ .
This is similarto the stochastic encoder applied in variational autoencoders (Kingma and Welling, 2013; Rezende and
Mohamed, 2015).
This idea was also extended to the VI framework in GP-LVMs by Bui and Turner(2015) using a recognition model, e.g. q ( z n )=
N ( z n | g µ ( x n | φ ) , g σ ( x n | φ )) and more recently, to 171 accelerate hyperparameter learning in GPs with hierarchical attention networks (Liu et al., 2020).
In our context, we assume the mappings to be neural networks (NNs) like Bui and Turner (2015),referring to them as amortization networks.
We ﬁnd such networks accelerate learning very nicelywhen used in conjunction with SAS.
It is also worth noting that amortization has been empiricallyshown to improve generalization performance (Shu et al., 2018).
Related Work
Marginal likelihood approximations were used in GPs (Smola and Bartlett, 2001; Csató and Opper,2002) before the apparition of pseudo-inputs (Snelson and Ghahramani, 2006) and the associated variational inference framework (Titsias, 2009).
In the former case, stochastic approximations to the
ELBO were ﬁrst presented by Hensman et al.
(2013, 2015), in line with the Bayesian counterpartof SAS.
In terms of active sets, Seeger et al.
(2003) empirically observed that the approximationwas generally stable for optimisation even if the size of A was a small fraction of the training sizeonly.
However, they also observed that random selection of active sets led to non-smooth ﬂuctuations,making it difﬁcult to converge through exact gradient ascent.
Particularly, the issue with reselectingof A motivated Snelson and Ghahramani (2006) as a smoother optimization alternative and we ﬁndthat SAS also circumvents this problem via stochastic estimates as shown in Sec. 5.
The connection between cross-validation and GPs was already described in Rasmussen and Williams(2006) as an alternative for model selection.
However, the equivalence between exahustive CV andthe log-marginal likelihood provided in Fong and Holmes (2020) provides a novel perspective thatwe exploit.
Additionally, the notion of back constraints has been recently considered in Lalchandet al.
(2022) for GP-LVMs with inducing points, where a doubly stochastic stochastic formulationis used.
More recently, amortization networks has been used to drastically reduce the number ofinducing points in supervised GPs.
Experiments
In this section, we evaluate the performance of the SAS approach for stochastic learning of GPdecoders, both the deterministic GP-LVM (Lawrence, 2004) and its Bayesian counterpart (Titsiasand Lawrence, 2010).
For this purpose, we consider three different datasets: MNIST (LeCun et al.
FASHION MNIST (Xiao et al., 2017) and CIFAR -10 (Krizhevsky et al., 2009).
For a faircomparison of our model with baseline methods, we use the same amortization network for allexperiments.
That is, we study such models where every n th latent variable z n is parametrised by anequivalent recognition NN .
Performance metrics of the SAS-GP decoders are given in terms of thenegative log-predictive density ( NLPD ), root mean-square error ( RMSE ) and mean square error ( MAE ).
In all cases, we optimize w.r.t.
an approximation to the log-marginal likelihood log p ( x | z ) in thedeterministic scenario or w.r.t. a lower bound on the log p ( x ) of the model in the Bayesian cases.
Forall experiments, we used the ADAM optimizer (Kingma and Ba, 2015).
We also provide P YTORCHcode that allows for reproducing all experiments and models.
3 Baseline methods as GP-LVM arealso included.
Learning rates are set in the range [10 − 4 , 10 − 2 ] and the maximum number of epochsconsidered is 300 .
An important detail of the SAS method is that the size of stochastic batches mustbe always larger that the active set A .
One additional detail is that we are interested in monitoring therun-time of convergence, as our hypothesis is that rotating active sets (see Fig. 1) across the datasetmight be a fast way to capture the correlation structure of the regression problem. [Figure] [Table]
z – S ec o ndd i m e n s i o n
[Table]
=">AAAB+XicbVDLTsMwEHTKq5RXAIkLF4sKiVOVIAQcK7hwLBJ9SG1UOe6mseo8ZG8QVejHwAkBN36DH+BvcEsO0DKn8cxYuzt+KoVGx/mySkvLK6tr5fXKxubW9o69u9fSSaY4NHkiE9XxmQYpYmiiQAmdVAGLfAltf3Q99dv3oLRI4jscp+BFbBiLQHCGRurbBz2EB9Q8D5gOjUKj2Ayd9O2qU3NmoIvELUiVFGj07c/eIOFZBDFyybTuuk6KXs4UCi5hUullGlLGR2wIXUNjFoH28tn+E3ocJIpiCHT2/p3NWaT1OPJNJmIY6nlvKv7ndTMMLr1cxGmGEHMTMV6QSYoJndZAB0IBRzk2hHElzJaUh0wxjqasijnfnT92kbROa+55zb09q9aviiLK5JAckRPikgtSJzekQZqEk0fyTN7Iu5VbT9aL9foTLVnFn33yB9bHN6uUlEQ
[Figure] 5.1 Representation performance
First, we analysed our SAS approach and the approximate optimization procedure on an unsupervisedversion of MNIST , FMNIST and CIFAR -10, where we took the full training corpus for learning twodimensional latent representations of images.
The approximation curves are included in Fig.
2, wherewe observe convergence in less that 2h runtime in CPU for most cases.
For all experiments, we canobserve that for larger active set sizes A , the SAS approaches takes longer time to complete the 300epochs as the computational cost of inversion is higher.
Evaluation with SOTA methods.
We also tested the performance o representation learning in thestate-of-the-art methods with and without GPs for unsupervised scenarios.
Results are shown in Fig.4.
We include a short description of the models considered:
• SAS - GP DECODER — It uses stochastic active sets to approximate the log-marginal likeli-
hood log p ( x | z ) .
The training methodology is described in Alg. 1.
• B AYESIAN SAS - GP DECODER — It uses stochastic active sets to approximate the ELBO in
Eq.
(8) on log p ( x ) .
See Alg. 2.
• B AYESIAN GP - LVM — It is based on the model in Titsias and Lawrence (2010).
Means and variance parameters are generated by an amortization NN as in Bui and Turner (2015).
The model is trained using stochastic variational inference (Hensman et al., 2013). • GP - LVM — We used the model proposed by Lawrence (2005) enhanced with a NN encodingto the latent space.
The model is trained using maximum likelihood (ML). • V ARIATIONAL AUTOENCODER ( VAE ) — (Kingma and Welling, 2013)
The NN encoder hasthe same architecture as the amortization function used in the SAS-GP models.
For the SAS-GP decoder and GP-LVM, a neural network encodes the latent variables.
Likewise,for the Bayesian SAS-GP decoder and the Bayesian GP-LVM, two neural networks encode eachthe latent means and latent variances.
We refer to this encoding as amortization .
All models uses a Gaussian likelihood 4 .
[Table]
5.2 Evaluation metrics
In this section, we are interested in the evaluation of the GP decoder with the standard error metricsused for GPs.
In Tab. 1 we provide RMSE , MAE and NLPD for the three dataset considered in thisexperiments.
Interestingly, the performance usually improves with larger active sets sizes A , as the
SAS model captures captures the underlying correlation of datapoints better and this enhances theapproximation.
This happens for both deterministic and Bayesian cases.
Additionally, we observethat the NLPD metric is generally better for smaller active set sizes A .
This can be also noticed in Fig.2, where loss curves have a better convergence for the lowest A .
Classiﬁcation accuracy.
We are interested in evaluating the structure of the representation.
Forthis purpose, we trained a (one) nearest neighbour classiﬁer on the encoded, two-dimensional latentvariables and we tested the accuracy using encoded test data.
Tab. 2 shows the mean and standarddeviations of test the accuracy. [Table]
[Figure]
°
[Figure]
B O
[Figure] [Figure] [Figure]
aagZxVz
[Figure] [Figure] Dk [Figure]
Ts7JKGUgzDQmZConpz1xuJ [Table] +208uNrhHF/LOIRPQDCqPI=">AAAB4nicbZDLTgJBEEVr8IX4Ql266UhMXJEZY9Ql6sYlJCIkMCE9TQ106Hmku8aEEH5AV0bd+Un+gH9jg7NQ8K5O172d1K0gVdKQ6345hZXVtfWN4mZpa3tnd6+8f/BgkkwLbIpEJbodcINKxtgkSQrbqUYeBQpbweh25rceURuZxPc0TtGP+CCWoRSc7Khx3StX3Ko7F1sGL4cK5Kr3yp/dfiKyCGMSihvT8dyU/AnXJIXCaambGUy5GPEBdizGPELjT+aLTtlJmGhGQ2Tz9+/shEfGjKPAZiJOQ7PozYb/eZ2Mwit [Figure]
IuM0I4yFjVgvzBSjhM36sr7UKEiNLXChpd2SiSHXXJC9SsnW9xbLLsPDWdW7qHqN80rtJj9EEY7gGE7Bg0uowR3UoQkCEJ7hDd6dvvPkvDivP9GCk
[Figure]
85hD9yPr4Bix
[Figure] < [Figure] Runtime and convergence of SAS approximation.
The loss curves obtained for the (sparse)
Bayesian GP-LVM model and the Bayesian SAS-GP decoder are shown in Fig.
We observethat the two methods scale similarly, but that SAS is faster by a notable constant within stochasticoptimization.
Additionally, we observe the performance of the SAS approximation to closely ﬁt theexact log-marginal likelihood log p ( x | z ) .
For the plot in Fig.
5, we computed the exact probabilityfor a subset of MNIST N =40 , 000 samples.
Conclusion
State-of-the-art representation learning is generally based on neural networks, as this allows forscaling to large datasets.
However, often we want reliable uncertainty estimates from the model and wecan achieve that with Gaussian process decoders if we can scale them sufﬁciently.
We have reviewedthe main difﬁculties to obtain decent performances with GP-LVM approaches when applied to largescale learning, even with inducing variables.
Revisiting active set approximations, we considered astochastic viewpoint to approximate the marginal likelihood while simultaneously keeping the modelmarginalized.
We formulate our stochastic active sets (SAS) approach for both deterministic and
Bayesian versions of GP decoders.
We ﬁnd that our approach works well with amortization, suchthat a neural network encoder approximately inverts the GP decoder.
While amortization also helpswhen using inducing points, we ﬁnd the combination with SAS to be particularly efﬁcient and robust.
Empirically, we illustrated the advantage of our method ﬁrst on image-based observations, where ourapproach learns better representations using fewer computational resources compared to inducingpoint methods.
We further demonstrate that our approach easily scales to nearly 10 6 observations.
Inthis experiment, we ﬁnd that the learnt representations are qualitatively on par with those attainedby a comparable autoencoder.
This is an important ﬁnding as, beyond small datasets, GP decodersgenerally recovers less useful representations compared with models based on neural networks.
From this result, we speculate that improvements in training might be enough to get state-of-the-artrepresentations with GP decoders.
Additional beneﬁts.
Besides the empirical beneﬁts demonstrated by SAS in the previous section,we have also observed other practical beneﬁts worth reporting.
First , we have observed that SASeasily runs in 32-bit numerical precision, unlike inducing point methods that generally requires64-bits of precision (when reporting running times we consistently used 64 bits).
Similarly, the jitterusually added to the Cholesky factorization is of less importance in SAS.
Second , we note that ourimplementation is surprisingly free of ‘tricks’ and no numerical heuristics were needed to realize areliable implementation.
Limitations and future work.
Stochastic active sets rely on the Gaussian likelihood and this isperhaps the strongest limitation.
This works well for continuous data, but many data sources areinherently discrete and this requires a suitable likelihood, e.g. the discretized mixture of logistics(Salimans et al., 2017).
Having more powerful likelihoods would surely improve the GP decoders,but this requires further developments to realize using SAS.
Future work will focus on applying the SAS approach the supervised setting as well and building
SAS-like methods for discrete data.
Other possible directions include extending the decoder withdeep kernels (Wilson et al., 2016) to capture more features in the data and applying convolutional
GPs (Van der
Wilk et al., 2017) which are more suited to high-dimensional images. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] We put particular attention on active sets, scalability and the connectionbetween marginal likelihood of GPs and CV.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
In Section 3 and the
Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] We provide an URL for the
Pytorch code, which will be available in a public repository.
Additionally, experiments ﬁles will
be included and details on the data can be found in the Appendix.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] Set up of GP decoder variables ( A , initial θ , etc) used in the experiments, further details
on initialization, algorithms and optimization are provided in the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple
times)?
[Yes] All our experiments consider multiple initialization and seeds for randomness,
which are reﬂected on the metrics and ﬁgures.
(d) Did you include the total
(a) If your work uses existing assets, did you cite the creators?
[Yes]
In the experiments and in the
[Figure]
scripts.
[Figure]
(b) Did you mention the license of the assets?
[Yes]
In the appendix and scripts.
[Figure]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Figure]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
[Figure]
ing/curating?
[N/A]
[Figure]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
informa-
[Figure]
tion or offensive content?
[N/A]
[Figure]
If you used crowdsourcing or conducted research with human subjects...
[Figure]
you include the full text of instructions given to participants and screenshots,
applicable?
[Figure]
[Figure]
(b
Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
[Figure]
approvals, if applicable?
N/
[Figure]
(c
Did you include the estimated hourly wage paid to participants and the total amount spent on
[Figure]
participant compensation?
[N/A]
[Figure]