Paper:
Gaussian Process Prior Variational Autoencoders
Variational autoencoders (VAE) are a powerful and widely-used class of models to learn complex data distributions in an unsupervised fashion. One important limitation of VAEs is the prior assumption that latent sample representations are independent and identically distributed. However, for many important datasets, such as time-series of images, this assumption is too strong: accounting for covariances between samples, such as those in time, can yield to a more appropriate model specification and improve performance in downstream tasks. In this work, we introduce a new model, the Gaussian Process (GP) Prior Variational Autoencoder (GPPVAE), to specifically address this issue. The GPPVAE aims to combine the power of VAEs with the ability to model correlations afforded by GP priors. To achieve efficient inference in this new class of models, we leverage structure in the covariance matrix, and introduce a new stochastic backpropagation strategy that allows for computing stochastic gradients in a distributed and low-memory fashion. We show that our method outperforms conditional VAEs (CVAEs) and an adaptation of standard VAEs in two image data applications.
image sequences from multiple patients. Therefore, the VAE prior should be able to capture multiple levels of correlations at once, including time, object identities, etc. A natural solution to this problem is to replace the VAE iid prior over the latent space with a Gaussian Process (GP) prior (Rasmussen, 2004), which enables the specification of sample correlations through a kernel function (Durrande et al., 2011;Gönen and Alpaydın, 2011;Wilson and Adams, 2013;Wilson et al., 2016;Rakitsch et al., 2013;Bonilla et al., 2007). GPs are often amenable to exact inference, and a large body of work in making computationally challenging GP-based models tractable can be leveraged (GPs naively scale cubically in the number of samples) (Gal et al., 2014;Bauer et al., 2016;Hensman et al., 2013;Csató and Opper, 2002;Quiñonero-Candela and Rasmussen, 2005;Titsias, 2009).
In this work, we introduce the Gaussian Process Prior Variational Autoencoder (GPPVAE), an extension of the VAE latent variable model where correlation between samples is modeled through a GP prior on the latent encodings. The introduction of the GP prior, however, introduces two main computational challenges. First, naive computations with the GP prior have cubic complexity in the number of samples, which is impractical in most applications. To mitigate this problem one can leverage several tactics commonly used in the GP literature, including the use of pseudo-inputs (Csató and Opper, 2002;Gal et al., 2014;Hensman et al., 2013;Quiñonero-Candela and Rasmussen, 2005;Titsias, 2009), Kronecker-factorized covariances (Casale et al., 2017;Stegle et al., 2011;Rakitsch et al., 2013), and low rank structures (Casale et al., 2015;Lawrence, 2005). Specifically, in the instantiations of GPPVAE considered in this paper, we focus on low-rank factorizations of the covariance matrix. A second challenge is that the iid assumption which guarantees unbiasedness of mini-batch gradient estimates (used to train standard VAEs) no longer holds due to the GP prior. Thus mini-batch GD is no longer applicable. However, for the applications we are interested in, comprising sequences of large-scale images, it is critical from a practical standpoint to avoid processing all samples simultaneously; we require a procedure that is both low in memory use and yields fast inference. Thus, we propose a new scheme for gradient descent that enables Monte Carlo gradient estimates in a distributable and memory-efficient fashion. This is achieved by exploiting the fact that sample correlations are only modeled in the latent (low-dimensional) space, whereas high-dimensional representations are independent when conditioning on the latent ones.
In the next sections we (i) discuss our model in the context of related work, (ii) formally develop the model and the associated inference procedure, (iii) compare GPPVAE with alternative models in empirical settings, demonstrating the advantages of our approach.

Review:
In this manuscript the authors propose a method for introducing auxiliary information regarding object 'view' (i.e., information ancillary to the class 'label') into the method of (Bayesian) variational autoencoders for object classification.  The challenge here, of course, is not simply writing down an additional layer in the hierarchical structure but in devising an efficient computational implementation, which the authors do achieve (via a low rank GP approximation, Taylor series expansion and sensible memory management).  Overall I find this to be a strong contribution, although I would accept that with respect to the guidelines for authors this could be argued to be more of an 'incremental' extension of an exisiting method rather than a new paradigm in itself.  A minor concern I have regards the presentation of the method: it seems to me that the strength and applicability of this method is more in terms of allowing the introduction of auxiliary information from well-understood transformations of the base objects in a dataset with relatively complete transformation instances for most objects; rather than for more general regularisation such as for potentially heterogeneous datasets including the autonomous driving and perhaps even the medical imaging applications suggested in the introduction.  The reason I say this is because choosing an appropriate kernel over the auxiliary data + object class space requires a degree of 'supervised' understanding of the problem and controlling/learning the low rank representation will be challenging when the data 'design' is sparse.  Likewise, for the range of applications for which a separable kernel is appropriate.  (Related: I do not understand the kernel description in eqn 9: is the x^T x term correct?  I was expecting something more like a distance than a vector length.)  After author feedback: Yes, I understand now to see that these are feature vectors and hence taking the inner product in the feature space for the kernel. Thanks