Gaussian Process Latent Variable Flows for Massively Missing Data
Abstract
Gaussian process latent variable models (GPLVM) are used to perform nonlinear and probabilistic dimensionality reduction.
They extend Gaussian processes (GP) to the domain of unsupervised learning (Lawrence, 2004).
The Bayesian incarnation of the GPLVM uses a variational framework, where the posterior over all unknown quantities is approximated by a well-behaved variational family, a factorised Gaussian (Titsias and Lawrence, 2010).
This gives not only implicit regularisation but also mathematical convenience.
In this work we narrow our focus on examining the quality of the latent representation learnt under this Gaussian assumption.
We introduce non-Gaussianity in the distribution of the latent space through normalising ﬂows .
The ﬂexibility aﬀorded by ﬂows is critical in modelling massively missing data.
Inference is performed using Stochastic
Variational Inference (SVI) with a structured variational lower bound that factorizes across data points permitting eﬃcient and scalable mini-batching of gradients.
We call this ﬂexible model class Gaussian process latent variable ﬂows (GPLVF).
We compare this framework with traditional models like the Bayesian GPLVM.
Our experiments focus on massively missing data settings.
1. Introduction
Gaussian processes (GPs) represent a powerful non-parametric probabilistic framework for performing regression and classiﬁcation with inductive biases controlled by a kernel function (Rasmussen and
Williams, 2006).
The Gaussian process latent variable model (GPLVM)
(Lawrence, 2004) paved the way for GPs to be used in unsupervised learning tasks like dimensionality reduction and structure discovery for high-dimensional data.
The GPLVM provides a probabilistic mapping from (an unobserved) latent space to data-space where a GP acts as a decoder , with the smoothness of the mapping controlled by a kernel function.
Several traditional dimensionality reduction models learn a projection of high dimensional data to lower dimensional manifolds (the direction of the mapping is reversed in a GPLVM).
An important attribute of the smooth GP decoder mapping in the GPLVM is it ensures that points close in latent space are mapped to points close in data space.
The notion of an encoder for GPLVMs was introduced in (Lawrence and Qui˜nonero Candela, 2006) where an additional mapping (called the back-constraint by the authors) was learnt expressing each latent point in the evidence (marginal likelihood) as a function of its corresponding data point.
This incarnation ensured that data space proximities were preserved in latent encodings.
Hence, GPLVMs can be put on the same footing as autoencoding models with an encoder mapping from data to latent space and a decoder mapping from latent to data space.
The Bayesian formulation of the GPLVM in (Titsias and Lawrence, 2010) variationally integrates out latent variables, providing principled uncertainty around the latent encoding.
The sparse variational formulation relying on inducing variables (Titsias, 2009) serves a dual purpose of making the lower [Table] bound to the marginal likelihood tractable, and providing computational savings.
The Bayesian
GPLVM also allows for the dimensionality of the latent space to be automatically determined by using ARD ( automatic relevance determination ) kernels.
It prunes dimensions which correspond to a small inverse lengthscale.
Techniques to apply Gaussian processes to very large datasets were introduced in Hensman et al.
(2013) which demonstrated how stochastic variational inference (SVI) (Hoﬀman et al., 2013) can be used with inducing variables.
The key idea was re-formulating the evidence lower bound (ELBO) in Titsias and Lawrence (2010) in a way that factorizes across the data enabling mini-batching for gradients.
However, in Hensman et al.
(2013) this was only explored for regression (with a Gaussian likelihood) settings rather than the unsupervised latent variable model setting.
This work looks at the Bayesian GPLVM through the lens of SVI.
The generalised SVI scheme allows for richer non-Gaussian variational families through the use of normalising ﬂows (Rezende and Mohamed, 2015).
The model introduced in this work allows the distribution around the latent encoding to be ﬂexible and expressive by warping the Gaussian variational distribution through a sequence of invertible transformations.
The departure from Gaussian uncertainty means we can model latent variables as being driven by multi-modal/complex distributions which may provide a more faithful approximation to the true unknown posterior in several settings.
Concretely, we summarise the main contributions of this work below:
• We extend the Bayesian GPLVM framework to non-Gaussian variational distributions through normalising ﬂows.
• We demonstrate how this class of models can be used on datasets which are extremely sparse (bulk of the features are missing for every data point) - we call this framework massively missing data which is frequently embodied in real-world datasets.
2. Review of Bayesian GPLVM
In this section we provide an overview of the Bayesian GPLVM (Titsias and Lawrence, 2010).
We have a training set comprising of N D -dimensional real valued observations Y ≡ { y n } Nn =1 ∈ R N × D .
These data are associated with N Q -dimensional latent variables, X ≡ { x n } Nn =1 ∈ R N × Q where
Q < D provides dimensionality reduction (Lawrence, 2004).
The forward mapping ( X −→ Y ) is governed by GPs independently deﬁned across dimensions D .
The probabilistic model describing the data is as follows:
Prior on latents: p ( X
) = N (cid:89) n = N ( x n ; 0 , I Q ) ,
Prior on mapping: p ( f | X, θ ) = D (cid:89) d = N ( f d ; 0 , K ( d ) nn ) ,
Data likelihood: p ( Y | f , X ) =
N (cid:89) n = D (cid:89) d =
N ( y n,d ; f d ( x n ) , σ 2 y ) , where f ≡ { f d } Dd =1 and y d is the d th column of Y .
K ( d ) nn is the covariance matrix corresponding to a user chosen positive-deﬁnite kernel function
k θ ( x, x (cid:48) ) evaluated on latent points { x n } Nn = parameterised by some hyperparameters θ .
Further, f d is a draw from a GP with covariance matrix
K ( d ) nn .
If same kernel function k θ ( x, x (cid:48) ) is chosen across the dimensions, then K ( d ) nn is identical across dimensions d and we can drop the superscript.
Exact Bayesian inference in this set-up entails ﬁnding the marginal likelihood obtained by integrating out the mapping f and latent variables X ; unfortunately this is intractable owing to the integration over the latent variables X 1 .
[Equation]
where p ( y d | X ) =
N ( 0 , K d + σ 2 y I )
Further, the posterior over all the unknown quantities p ( f , X | Y )
∝ p ( Y | f , X ) p ( f | X, θ ) p ( X ) is intractable.
This intractability is side-stepped by introducing a variational distribution over unknowns ( f , X ) augmented with M inducing variables u ≡ { u d } D d =1 , each distributed with a GP prior p ( u d ) ∼ N ( 0 , K mm ).
K mm takes as input inducing input locations Z ∈ R M × Q which live in latent space, are shared across dimensions and have dimensionality Q (matching x n )(Titsias, 2009).
The formulation
[Equation]
where q ( u d ) is the variational distribution over the inducing variables and q ( X )
= (cid:81)
Nn =1 N ( x n ; µ n , s n I Q ) admits a tractable lower bound to the marginal likelihood p
( Y | θ ) for speciﬁc choices of kernel functions.
The variational formulation above gives rise to an evidence lower bound (ELBO) written in rudimentary form (full derivation enclosed in supplementary),
[Equation]
where KL( · ) denotes Kullback-Leibler divergence and can be computed in closed form if the variational distributions q ( X ) and q ( u ) are chosen to be Gaussian 2 .
In fact, the q ( u d )s are unrestricted and an analytic maximisation of the bound w.r.t q ( u d )s turns out to yield Gaussian distributions as the optimal form.
The ﬁnal bound in Titsias and Lawrence (2010) is obtained by collapsing the bound by plugging in the optimal q ∗ ( u d ) = arg [Equation]
q [Equation] u
[Equation]
In this work we take a diﬀerent approach, we use the uncollapsed version of the lower bound (with an explicit representation of q ( u d )) enabling stochastic gradient methods and parallelised mini-batching of gradients for a truly scalable solution.
2.1. SVI for Bayesian GPLVM
Stochastic Variational Inference (SVI) (Hoﬀman et al., 2013) is used to scale variational inference to massive datasets by sub-sampling data to compute noisy gradients.
The main prerequisite is that the ELBO should factorise across the data points.
Another detail is the existence of global variables that make the observations conditionally independent given their local latent variable.
The introduction of inducing variables u satisﬁes this criteria for variational sparse GP regression models, Hensman et al.
(2013) derives the uncollapsed lower bound for the regression case with a 1d output, here we show the ﬁnal form of the uncollapsed lower bound for the latent variable model setting with D -dimensional outputs y d by expanding L 1 . L D =
[Table]
This bound is optimised with respect to variational parameters of q ( { u } Dd =1 ) given by ( { m d , S d } Dd =1 , Z ), the variational parameters concerning q ( X ) given by ( { µ n , s n } Nn =1 ), the kernel hyperparameters θ and the likelihood noise variance σ 2 y .
Please refer to the appendix A.3 for a derivation of this bound and appendix A.4 for the Ψ terms where we show how this formulation preserves factorisability.
3. Gaussian Process Latent Variable Flows
One of the advantages of the generalised SVI framework is that it only pre-requisites a factorisation of the approximating variational family and does not impose any restrictions over its form.
As such, the choice of approximating variational family q ( X ) can be ﬂexible as long as we can sample from it and compute gradients w.r.t its parameters.
In this class of models we make q ( X ) generic by allowing the base Gaussian distribution around each latent point to be transformed by a sequence of normalising ﬂows.
The motivation for this is that the posterior distribution p ( X | Y ) can be arbitrarily complex manifesting in multiple-modes and non-linear correlations in latent space.
Further, the pathologies in the posterior distribution over the latent variables X can be more pronounced in high-dimensional missing data settings.
Normalising ﬂows (Rezende and Mohamed, 2015) leverage the fundamental rule for specifying probability densities of transformed random variables.
When a random variable z (0) ∼ p ( z (0) ) is transformed by a sequence of k invertible and diﬀerentiable mappings composed together, the resulting random variable g k ◦ g k − 1 ◦ . . .
g 1 ( z (0) ) =
z ( k ) has a density given by, p ( z ( k ) ) = p ( z (
[Equation]
The variational distribution of the Bayesian GPLVM with a transformed Gaussian distribution is given by,
[Equation]
(4) where the parameters of the ﬂow mappings g j are collected in λ , a set of additional variational parameters.
We call this model Gaussian Process latent Variable Flows .
It’s main attributes are:
• The base Gaussian distribution is channeled through a ﬁnite sequence of normalising ﬂows yielding an arbitrarily complex marginal distribution over the latent points x n • It is the marginal densities q ( x n ) that are warped by the ﬂow transformations rather than the joint, yielding a joint density that still factorises across data points.
• This formulation preserves the factorisability of the bound (it can be written as a sum of N terms) but with a richer non-Gaussian variational approximation.
The parameters λ of the Q dimensional ﬂow are shared between the data points enabling amortised inference.
We perform Monte Carlo expectations of the terms in the uncollapsed lower bound that involve q ( X ) by sampling from the base Gaussian at each step and pushing them through the ﬂow g k ◦ g k − 1 ◦ . . .
g 1 ( x (0) )
=
x k to yield the ﬁnal latent point.
We do this for all N across each dimension Q .
3.1. Missing Data Framework
In this section we summarise a framework for dealing with massively missing data at test time.
The
SVI objective factorises across data points and dimensions.
The crux of the training procedure relies on the marginalisation principle of Gaussian distributions.
More concretely, we can marginalise out the missing dimensions y a as long as our data point y is modelled as a joint Gaussian.
For a single data point split by its unobserved and observed dimensions,
[Equation]
where a and o denote the indices of missing and observed dimensions respectively with the full set of dimensions given as, D = a ∪ o .
u
d ∈ R M ∀
d
= 1 , . .
, D denote the inducing variables which ensure conditional independence.
This set-up reﬂects real-world data very closely which is often sparse with many missing and few overlapping dimensions.
4. Experiments 4.1. Missing MNIST
The focus of this experiment is to gauge how the models capture uncertainty when training with missing pixels.
We use 5000 training samples from the MNIST digits dataset (LeCun et al., 2010) across classes [0-9] with ≈ 40% of the pixels missing in each digit.
Fig.
3 summarises sample generation from the 2d latent distribution for a single training digit.
The key feature of the ﬂow based distribution is that it demonstrates the ability to capture non-linear correlations in the uncertainty structure of the latent posterior for the missing pixels digit.
Notice that the samples generated from the ﬂow based variational approximation are much more diverse than the ones from the latent Gaussian approximation; in some parts of the ﬂow distribution they recover digits resembling the ground truth.
Missing dimensions in the training data are highly likely to give rise to pathological posteriors with high uncertainty, and in these cases a ﬂexible variational distribution provides a better approximation than a Gaussian.
Fig. 2 shows samples generated from the ﬂow based approximation, each point denotes a MAP estimate of the ﬂow and we superimpose the reconstructed digits for some of the samples.
4.2. Movie Lens 100K
[Table] The movie lens 100K data has 1682 movies (columns/dimensions) across 943 users (rows/data points) where each user has rated an average of 20 movies (Harper and Konstan, 2015).
This yields an extremely sparse data grid with 93 .
8% of the entries missing 3 , truly embodying the massively missing data framework.
We learn a 10 d latent distribution for the movie lens data and summarise the test RMSE and test log likelihoods for increasing ﬂow lengths.
We trained on 843 users and made predictions forusers.
5. Further work & Extensions
The ability to model the distribution of a latent encoding ﬂexibly could be invaluable in several settings.
Flows enable capturing correlations in the latent space without paying the price of learning a full covariance matrix.
In the missing training data framework, their beneﬁt is even more pronounced as unsupervised training with massively missing dimensions induce arbitrarily complex posteriors in latent space.
The implementation shown here preserves the factorisability of the lower bound enabling parallelisable inference achieving dual goals of scalability and ﬂexibility.
The highlight of the work is its applicability in massively missing data regimes which very few frameworks can handle.
Further, there are two questions that surround the use of normalising ﬂows, 1) the choice of ﬂow and 2) the ﬂow length.
Diﬀerent transformations yield ﬂows with diﬀerent inductive biases, this can be studied in more detail.
Based on prior knowledge we can introduce transformations which produce a desired eﬀect in the latent space.
Other avenues for theoretical work include carefully comparing the collapsed bound proposed in (Titsias and Lawrence, 2010) with the uncollapsed bound introduced in (Hensman et al., 2013) and derived here.
Appendix A. Derivations A.1. Motivation for inducing variables
As is standard, we wish to minimize the KL divergence between the variational approximation and the true posterior given by, KL( q ( { f d } Dd =1 , X ) || p
( { f } Dd =1 , X | Y )).
For an arbitrary d we have,
KL
[Equation]
The evidence lower bound shown above is mathematically intractable due to the term p
( f d | X, θ ) =
N ( 0 , K ( d ) nn ) involving the variables X which appear non-linearly in the kernel matrix.
It turns out that an augmented bound with inducing variables u d for each dimension is tractable.
A.2. Derivation of the ELBO eq. (3)
In this section we detail the derivation of the rudimentary ELBO as in eq.
(3) of the paper,
We introduce auxiliary inducing variables, u d ∈ R M for each of the latent functions f d , collecting all of the u d ’s in u and f d ’s in f for ease of notation, it turns out that variational inference in ( f , u , X ) space is actually tractable.
The marginal prior p ( u | Z ) is given as,
[Equation]
) where Z ∈ R M × Q denote inducing input locations.
Writing down the augmented variational approximation as,
[Equation]
we derive the form of the ELBO in this augmented space by writing down the KL divergence explicitly,
KL (
[Equation]
( f , u , X ) || p (
[Equation]
u
[Equation]
X | Y )) =
[Equation]
where the terms in red (the diﬃcult terms) cancel out and we suppress the implicit conditioning over the hyperparameters θ for brevity.
The ﬁnal ELBO is given by,
[Equation]
where p ( f | u , X ) is the conditional prior, q ( u ) is a free-form variational distribution, p ( X ) =
(cid:81)
Nn =1 N ( 0 , I Q ) is just a product of standard normals and q ( X ) is chosen to be a product of multivariate
Gaussians with diagonal covariances s n ,
[Equation]
p ( u | Z ) is as deﬁned in eq. 8.
Going back to (10), we ﬁrst apply the integration w.r.t f and rewrite it more conveniently,
L 1: D =
[Equation] [Equation]
[Equation]
A.3. Derivation of the uncollapsed SVI ELBO
Consider the ﬁrst term of the ELBO in eq.
[Equation]
First, performing the integration w.r
[Equation]
t f d ,
L ( n,d ) f
[Equation] [Equation]
Note: y n,d is a scalar ( d th dimension of point y n ), k Tn is a 1 × M matrix - the n th row of K nm , we know that p ( f d | u d , X ) =
N ( K nm K − 1 mm u d , K nn − K nm K − 1 mm K mn ).
Further, f d ( x n ) is a scalar, denoting the value at index x n of the vector f d . q n,n is the n th entry in the diagonal of matrix
Q nn = K nn − K nm K − 1 mm K mn .
Then, performing the integration w.r.t u d (we parameterise q ( u d ) =
N ( m d , S d ) as we know its optimal form is a Gaussian and using similar identities as above),
[Equation]
[Equation]
where Λ n,n is the n th entry of the diagonal of matrix Λ =
S d K − 1 mm K mn K nm K − 1 mm .
Now, what remains is to perform the integration w.r.t q ( X ).
[Equation]
We note that the only terms in L ( n,d ) u involving the latent X are K nm , K nn and K nm K mn ;we show in the next section that they admit a factorisation across data points.
The ﬁnal bound is given by,
L
[Equation]
A.4. Ψ statistics
In this section we detail how ψ 0 , Ψ 1 and Ψ 2 are factorisable across data points.
[Equation]
q (
[Equation]
where K ( ii ) nn are the diagonal entries of matrix K nn (25) = N
[Equation]
where { x i } Ni =1 ≡ X and q ( X ) =
N (cid:89) i = q ( x i ) (26)
Next, we look at Ψ 1 ,
[Equation] [Equation]

[Equation]
where we notice that Ψ 1 is a N × M matrix where each row just depends on a data point x i .
[Equation] [Equation]
which is an M × M matrix decomposable as a sum of N M × M matrices where each component matrix is only dependent on a data point x i .
A.5. KL divergence between factorised Gaussians
In eq.
(9) we re-write the KL term involving q ( X ) as a factorisation across n , we show the proof below:
[Equation]
A.6. Test log-likelihood
The test log-likelihood is given by log p ( y ∗ | Y ), we
report this metric averaged across the number of test points n test and describe how its computed below.
This metric is sometimes called the log predictive density, they both refer to the same quantity.
The test log-likelihood is given by log p ( y ∗ | Y ),
[Equation]
where, Q
[Equation]
= K ∗ m K − 1 mm K m
[Equation]
[Equation] [Equation]
The integral over x ∗ is resolved numerically and the integral over u is analytical as per Titsias (2009).