Original: Factorized Gaussian Process Variational Autoencoders
Revised: Factorized Gaussian Process Variational Autoencoders

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: Variational autoencoders often assume isotropic Gaussian priors and mean-ﬁeld posteriors, hence do not exploit structure in scenarios where we may expect similarity or consistency across latent variables.
Revised: Variational autoencoders often assume isotropic Gaussian priors and mean-ﬁeld posteriors, hence do not exploit structure in scenarios where we may expect similarity or consistency across latent variables.

Original: Gaussian process variational autoencoders alleviate this problem through the use of a latent Gaussian process, but lead to a cubic inference time complexity.
Revised: Gaussian process variational autoencoders alleviate this problem through the use of a latent Gaussian process, but lead to a cubic inference time complexity.

Original: We propose a more scalable extension of these models by leveraging the independence of the auxiliary features, which is present in many datasets.
Revised: We propose a more scalable extension of these models by leveraging the independence of the auxiliary features, which is present in many datasets.

Original: Our model factorizes the latent kernel across these features in diﬀerent dimensions, leading to a signiﬁcant speed-up (in theory and practice), while empirically performing comparably to existing non-scalable approaches.
Revised: Our model factorizes the latent kernel across these features in diﬀerent dimensions, leading to a signiﬁcant speed-up (in theory and practice), while empirically performing comparably to existing non-scalable approaches.

Original: Moreover, our approach allows for additional modeling of global latent information and for more general extrapolation to unseen input combinations
Revised: Moreover, our approach allows for additional modeling of global latent information and for more general extrapolation to unseen input combinations.

Original: 1. Introduction
Revised: 1. Introduction

Original: Variational autoencoders (VAEs) have achieved great success in many representation learning tasks (Kingma and Welling, 2013; Rezende et al., 2014).
Revised: Variational autoencoders (VAEs) have achieved great success in many representation learning tasks (Kingma and Welling, 2013; Rezende et al., 2014).

Original: However, their isotropic Gaussian prior and variational posterior hinge on the strong assumption that all data points are independent.
Revised: However, their isotropic Gaussian prior and variational posterior hinge on the strong assumption that all data points are independent.

Original: This can often lead to problems in real-world use cases, where the data exhibit signiﬁcant correlations (Fraccaro et al., 2017; Krishnan et al., 2016).
Revised: This can often lead to problems in real-world use cases, where the data exhibit signiﬁcant correlations (Fraccaro et al., 2017; Krishnan et al., 2016).

Original: Many alternative priors and posteriors have been proposed for VAEs (Tomczak and
Revised: Many alternative priors and posteriors have been proposed for VAEs (Tomczak and

Original: Welling, 2018; Fortuin et al., 2018; Kopf et al., 2019; Abiri and Ohlsson, 2020).
Revised: Welling, 2018; Fortuin et al., 2018; Kopf et al., 2019).

Original: Especially when each input comes with meta-data, or auxiliary features , such extra information can be used to construct Gaussian processes (GPs) and use them as priors in the latent space (Casale et al., 2018; Fortuin et al., 2020; Pearce, 2020).
Revised: Especially when each input comes with meta-data, or auxiliary features , such extra information can be used to construct Gaussian processes (GPs) and use them as priors in the latent space (Casale et al., 2018; Fortuin et al., 2020; Pearce, 2020).

Original: By choosing appropriate kernels, these resulting GP-VAE models allow to capture the structured correlations across latent variables of diﬀerent data points (Rasmussen and Williams, 2006).
Revised: By choosing appropriate kernels, these resulting GP-VAE models allow to capture the structured correlations across latent variables of diﬀerent data points (Williams and Rasmussen, 2006).

Original: However, they are also haunted by the computational cost of exact GP inference, which scales as O ( N 3 ) for N elements in the dataset.
Revised: However, they are also haunted by the computational cost of exact GP inference, which scales as O ( N 3 ) for N elements in the dataset.

Original: We consider the setting in which the inputs are images, and each image is associated with several features, some of which are unique to the image and some of which are shared with other images.
Revised: We consider the setting in which the inputs are images, and each image is associated with several features, some of which are unique to the image and some of which are shared with other images.

Original: For example in a set of MNIST digits rotated by multiple angles (Casale et al., 2018), the digit ID is shared by other images of the same digit while the angle is unique.
Revised: For example in a set of MNIST digits rotated by multiple angles (Casale et al., 2018), the digit ID is shared by other images of the same digit while the angle is unique.

Original: Similarly, in a set of faces viewed from multiple perspectives, the person is common to multiple images while each image has a unique position (Casale et al., 2018).
Revised: Similarly, in a set of faces viewed from multiple perspectives, the person is common to multiple images while each image has a unique position (Casale et al., 2018).

Original: Further applications include a set of scenes viewed from multiple positions (Eslami et al., 2018), high dimensional spatio-temporal datasets (Ashman et al., 2020), or speech segments where one speaker’s voice is shared by multiple segments (Li and Mandt, 2018).
Revised: Further applications include a set of scenes viewed from multiple positions (Eslami et al., 2018), high dimensional spatio-temporal datasets (Ashman et al., 2020), or speech segments where one speaker’s voice is shared by multiple segments (Li and Mandt, 2018).

Original: For such settings, we propose a novel factorized GP-VAE model, FGP-VAE, with two desirable properties.
Revised: For such settings, we propose a novel factorized GP-VAE model, FGP-VAE, with two desirable properties.

Original: Firstly, by carefully exploiting factorization, inference is signiﬁcantly reduced from O ( N 3 ).
Revised: Firstly, by carefully exploiting factorization, inference is signiﬁcantly reduced from O ( N 3 ).

Original: Secondly, the representations are encouraged to be disentangled, which is a highly desirable property for VAE models (Locatello et al., 2019; van Steenkiste et al., 2019;
Revised: Secondly, the representations are encouraged to be disentangled, which is a highly desirable property for VAE models (Locatello et al., 2019; van Steenkiste et al., 2019;

Original: Tr¨auble et al., 2020).
Revised: Tr¨auble et al., 2020).

Original: We describe the problem setting in Section 2 and the proposed model in Section 3.
Revised: We describe the problem setting in Section 2 and the proposed model in Section 3.

Original: We present experimental results in Section 4, and conclude in Section 5.
Revised: We present experimental results in Section 4, and conclude in Section 5.

Original: 2. Problem Setting
Revised: 2. Problem Setting

Original: Consider high-dimensional data of N elements Y =
Revised: Consider high-dimensional data of N elements Y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y N ] (cid:62)
Revised: , y N ] (cid:62)

Original: where
Revised: where

Original: y i ∈
Revised: y i ∈

Original: R K and each data point has corresponding low-dimensional auxiliary data X =
Revised: R K and each data point has corresponding low-dimensional auxiliary data X =

Original: [ x 1 , . . .
Revised: [ x 1 , . . .

Original: , x N ] (cid:62) ∈ X N , X ⊆
Revised: , x N ] (cid:62) ∈ X N , X ⊆

Original: R D .
Revised: R D .

Original: For ease of exposition, we will focus on the example of the rotated MNIST dataset (Casale et al., 2018).
Revised: For ease of exposition, we will focus on the example of the rotated MNIST dataset (Casale et al., 2018).

Original: It consists of P digits, each observed at Q diﬀerent angles, amounting to a total of N = P · Q images.
Revised: It consists of P digits, each observed at Q diﬀerent angles, amounting to a total of N = P · Q images.

Original: 1 Each x i =
Revised: 1 Each x i =

Original: ( d i , w i ) ∈
Revised: ( d i , w i ) ∈

Original: X is composed of a categorical digit instance d i (integer index or one-hot encoding) and a continuous angle w i .
Revised: X is composed of a categorical digit instance d i (integer index or one-hot encoding) and a continuous angle w i .

Original: We wish to train a model that can (1) given new x ∗ ∈ X generate y ∗ ∈ R K , and (2) infer an interpretable and disentangled latent representation.
Revised: We wish to train a model that can (1) given new x ∗ ∈ X generate y ∗ ∈ R K , and (2) infer an interpretable and disentangled latent representation.

Original: 3. Method
Revised: 3. Method

Original: Generative model : we follow a latent GP approach, ﬁrst proposed in Casale et al.
Revised: Generative model : we follow a latent GP approach, ﬁrst proposed in Casale et al.

Original: (2018) and later extended in Pearce (2020).
Revised: (2018) and later extended in Pearce (2020).

Original: As in a standard VAE, each image y
Revised: As in a standard VAE, each image y

Original: i is associated with a latent variable z i ∈ R L .
Revised: i is associated with a latent variable z i ∈ R L .

Original: Making use of the auxiliary data, we further expect that two images y i and y j with similar x
Revised: Making use of the auxiliary data, we further expect that two images y i and y j with similar x

Original: i and x j should also have similar z i and z j .
Revised: i and x j should also have similar z i and z j .

Original: To this end, a Gaussian process regression is used to model a joint distribution over all latent variables
Revised: To this end, a Gaussian process regression is used to model a joint distribution over all latent variables

Original: Z =
Revised: Z =

Original: [ z 1 , . . .
Revised: [ z 1 , . . .

Original: , z N ] T ∈ R N × L .
Revised: , z N ] T ∈ R N × L .

Original: Given L latent dimensions, we assume L independent latent functions f l ∼ GP (0 , k lθ ) , l = 1 , . . .
Revised: Given L latent dimensions, we assume L independent latent functions f l ∼ GP (0 , k lθ ) , l = 1 , . . .

Original: , L with kernels k lθ and therefore the latent variable for y i may be written as z i =
Revised: , L with kernels k lθ and therefore the latent variable for y i may be written as z i =

Original: [ f 1 ( x i ) , . . .
Revised: [ f 1 ( x i ) , . . .

Original: , f L ( x i )] T .
Revised: , f L ( x i )] T .

Original: Likewise, all latent variables of the l th channel z l 1: N =
Revised: Likewise, all latent variables of the l th channel z l 1: N =

Original: [ f l ( x 1 ) , . . .
Revised: [ f l ( x 1 ) , . . .

Original: , f l ( x N )] ∈ R N are assumed to come from a single (unknown) function, speciﬁcally z l 1: N has a correlated Gaussian prior with covariance K lNN = k lθ ( X , X ) ∈ R N × N .
Revised: , f l ( x N )] ∈ R N are assumed to come from a single (unknown) function, speciﬁcally z l 1: N has a correlated Gaussian prior with covariance K l NN = k lθ ( X , X ) ∈ R N × N .

Original: The generative model, p ψ ( Y , Z | X ) =
Revised: The generative model, p ψ ( Y , Z | X ) =

Original: p ψ ( Y | Z ) p θ ( Z | X ), is thus
Revised: p ψ ( Y | Z ) p θ ( Z | X ), is thus

Original: [Equation]
Revised: [Equation]

Original: where µ ψ : R L → R K is a (generative) network with parameters ψ .
Revised: where µ ψ : R L → R K is a (generative) network with parameters ψ .

Original: Note that if K NN
Revised: Note that if K NN

Original: = I is the identity matrix, the model recovers a standard VAE; using the X values enables the use of more sophisticated prior.
Revised: = I is the identity matrix, the model recovers a standard VAE; using the X values enables the use of more sophisticated prior.

Original: In prior work (Casale et al., 2018; Jazbec et al., 2020), a single GP prior is used in all L latent channels.
Revised: In prior work (Casale et al., 2018; Jazbec et al., 2020), a single GP prior is used in each of the L latent channels.

Original: Speciﬁcally, for rotated MNIST, a product kernel between a periodic and a linear kernel is considered
Revised: Speciﬁcally, for rotated MNIST, a product kernel between a periodic and a linear kernel is considered

Original: [Equation]
Revised: [Equation]

Original: with parameters θ = { σ, r, Σ } .
Revised: with parameters θ = { σ, r, Σ } .

Original: Σ =
Revised: Σ =

Original: DD T has a low-rank form, and D ∈ R P × m is a (learned) matrix that captures information common to all images of each digit such as written style.
Revised: DD T has a low-rank form, and D ∈ R P × m is a (learned) matrix that captures information common to all images of each digit such as written style.

Original: This approach has multiple drawbacks.
Revised: This approach has multiple drawbacks.

Original: Firstly, given a set of N images, the above kernel gives rise to a dense matrix K NN , where all latent variables are correlated with each other.
Revised: Firstly, given a set of N images, the above kernel gives rise to a dense matrix K NN , where all latent variables are correlated with each other.

Original: This necessitates either O ( N 3 ) cost or non-trivial approximations, such as sparse GPs (Jazbec et al., 2020) or assuming equally spaced x i enabling specialized matrix decompositions (Casale et al., 2018).
Revised: This necessitates either O ( N 3 ) cost or non-trivial approximations, such as sparse GPs (Jazbec et al., 2020) or assuming equally spaced x i enabling specialized matrix decompositions (Casale et al., 2018).

Original: Statistically, the prior does not factorize across any subsets of the data.
Revised: Statistically, the prior does not factorize across any subsets of the data.

Original: Secondly, if new digits are added to the dataset, the Σ matrix needs be augmented with a new row and column, the new hyperparamters of D must be learned from scratch, and it is not “amortized” over digits.
Revised: Secondly, if new digits are added to the dataset, the Σ matrix needs be augmented with a new row and column, the new hyperparamters of D must be learned from scratch, and it is not “amortized” over digits.

Original: In this work, we propose two simple (yet still unexplored) changes that alleviate the aforementioned issues.
Revised: In this work, we propose two simple (yet still unexplored) changes that alleviate the aforementioned issues.

Original: We start by partitioning the dataset into digit speciﬁc subsets { X , Y , Z } = ∪ Pp
Revised: We start by partitioning the dataset into digit speciﬁc subsets { X , Y , Z } = ∪ Pp

Original: =1 { X p , Y p , Z p } and denote partitions by
Revised: =1 { X p , Y p , Z p } and denote partitions by

Original: [Equation]
Revised: [Equation]

Original: Instead of assuming a single correlated prior over all N latent variables, we assume P separate correlated priors over Q latent variables each, that is, one prior for each partition of the data.
Revised: Instead of assuming a single correlated prior over all N latent variables, we assume P separate correlated priors over Q latent variables each, that is, one prior for each partition of the data.

Original: Secondly, we assume that each latent variable z i ∈ R L is composed of two parts.
Revised: Secondly, we assume that each latent variable z i ∈ R L is composed of two parts.

Original: Temporarily dropping i for clarity, we propose to have
Revised: Temporarily dropping i for clarity, we propose to have

Original: [Equation]
Revised: [Equation]

Original: z J (cid:124) (cid:123)(cid:122) (cid:125) local , z J + [Equation] [Equation] z L (cid:124) (cid:123)(cid:122) (cid:125) global (cid:1) , where the local variable z 1: J i is unique to the given image and the global variable z J
Revised: where the local variable z 1: J i is unique to the given image and the global variable z J

Original: +1:
Revised: +1:

Original: L i is shared by all elements in the subset Z p .
Revised: L i is shared by all elements in the subset Z p .

Original: The global variable captures style information or other common features of digit d i that are angle-agnostic.
Revised: The global variable captures style information or other common features of digit d i that are angle-agnostic.

Original: For example, for frames from videos of moving objects, local variables could capture object position and location while global variables could capture object color and shape.
Revised: For example, for frames from videos of moving objects, local variables could capture object position and location while global variables could capture object color and shape.

Original: We must construct a kernel for each latent dimension l ∈ { 1 , ..., L } satisfying the above criteria.
Revised: We must construct a kernel for each latent dimension l ∈ { 1 , ..., L } satisfying the above criteria.

Original: For the local latent channels l ∈ { 1 , ..., J } , we specify the following kernel
Revised: For the local latent channels l ∈ { 1 , ..., J } , we specify the following kernel

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst Kronecker delta term ensures that two images corresponding to diﬀerent digits have zero assumed latent variable similarity, while for two images of the same digit the assumed similarity depends upon the diﬀerence in the rotation angles.
Revised: The ﬁrst Kronecker delta term ensures that two images corresponding to diﬀerent digits have zero assumed latent variable similarity, while for two images of the same digit the assumed similarity depends upon the diﬀerence in the rotation angles.

Original: This enforces that the GP priors for each subset of digits are independent.
Revised: This enforces that the GP priors for each subset of digits are independent.

Original: For the global latent channels l ∈ { J + 1 , ..., L } , we wish to capture rotation-agnostic characteristics of each image, that is, its style.
Revised: For the global latent channels l ∈ { J + 1 , ..., L } , we wish to capture rotation-agnostic characteristics of each image, that is, its style.

Original: To this end, a simple binary kernel is used
Revised: To this end, a simple binary kernel is used

Original: [Equation]
Revised: [Equation]

Original: Thus, among rotated images of the same digit, the global latents have perfect correlation; for each global channel there is a single univariate distribution shared by all images of a single digit.
Revised: Thus, among rotated images of the same digit, the global latents have perfect correlation; for each global channel there is a single univariate distribution shared by all images of a single digit.

Original: Simultaneously, the global latent variable of images in another subset is treated as independent (this may also be viewed as the local kernel with length scale r → ∞ ).
Revised: Simultaneously, the global latent variable of images in another subset is treated as independent (this may also be viewed as the local kernel with length scale r → ∞ ).

Original: Due to the kernel structure, there is now a separate generative model for every digit instance p ψ,θ ( Y , Z | X ) =
Revised: Due to the kernel structure, there is now a separate generative model for every digit instance p ψ,θ ( Y , Z | X ) =

Original: (cid:81)
Revised: (cid:81)

Original: Pp =1 p ψ ( Y p | Z p ) p θ ( Z p | X p ).
Revised: Pp =1 p ψ ( Y p | Z p ) p θ ( Z p | X p ).

Original: Within each of the digit-speciﬁc generative models, working with the GP prior is much less prohibitive as Q (cid:28) N .
Revised: Within each of the digit-speciﬁc generative models, working with the GP prior is much less prohibitive as Q (cid:28) N .

Original: Secondly, digit style is captured in the global latent variables z j +1: L i which can be estimated from images via amortization.
Revised: Secondly, digit style is captured in the global latent variables z j +1: L i which can be estimated from images via amortization.

Original: Such global information is no longer encoded in the generative model hyperparameters Σ.
Revised: Such global information is no longer encoded in the generative model hyperparameters Σ.

Original: Approximate Posterior : exploiting the factorized structure of the generative model, we may consider the posterior of each subset Z p independently.
Revised: Approximate Posterior : exploiting the factorized structure of the generative model, we may consider the posterior of each subset Z p independently.

Original: Since the true posterior for latent variables p ψ,θ ( Z p | Y p , X p ) is intractable, approximate inference is required.
Revised: Since the true posterior for latent variables p ψ,θ ( Z p | Y p , X p ) is intractable, approximate inference is required.

Original: In VAEs, an inference network (with parameters φ ) takes y q ∈ Y p as input to predict the mean and variance of a mean-ﬁeld approximate posterior of each latent encoding which we denote as 
Revised: In VAEs, an inference network (with parameters φ ) takes y q ∈ Y p as input to predict the mean and variance of a mean-ﬁeld approximate posterior of each latent encoding which we denote as ˜ q [Equation] [Equation] y

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: L (cid:89) l =

Original:  and one possible approximate posterior is to use the product of the above factors over all
Revised: [Equation] and one possible approximate posterior is to use the product of the above factors over all

Original: N latent variables.
Revised: N latent variables.

Original: Instead, closely following Pearce (2020), we use ˜ q φ ( · ) to replace only the intractable likelihood p ψ ( y i |
Revised: Instead, closely following Pearce (2020), we use ˜ q φ ( · ) to replace only the intractable likelihood p ψ ( y i |

Original: z i ) in the exact posterior.
Revised: z i ) in the exact posterior.

Original: This gives rise to the following approximate posterior
Revised: This gives rise to the following approximate posterior

Original: [Equation]
Revised: [Equation]

Original: The conjugacy of the Gaussian prior and (approximate) Gaussian likelihoods yields a closedform solution for the normalizing constant Z φ,θ ( Y p , X p ).
Revised: The conjugacy of the Gaussian prior and (approximate) Gaussian likelihoods yields a closedform solution for the normalizing constant Z φ,θ ( Y p , X p ).

Original: Moreover, the approximate posterior q ( Z p |· ) is mathematically equivalent to a product of J exact GP posteriors of Q points (one GP for each angle latent channel) and L − J univariate Gaussian distributions that are common to all Q elements in the subset.
Revised: Moreover, the approximate posterior q ( Z p |· ) is mathematically equivalent to a product of J exact GP posteriors of Q points (one GP for each angle latent channel) and L − J univariate Gaussian distributions that are common to all Q elements in the subset.

Original: Exact derivations are given in Appendix B.
Revised: Exact derivations are given in Appendix B.

Original: Finally, the FGP-VAE ELBO has the form:
Revised: Finally, the FGP-VAE ELBO has the form:

Original: [Equation]
Revised: [Equation]

Original:  Due to its factorization across digit subsets, and the assumption that Q (cid:28) N, exact GP inference (with approximate likelihoods) is feasible, resulting in O ( P Q 3 ) complexity for one
Revised: [Table] Due to its factorization across digit subsets [Table] and the assumption that Q (cid:28) N

Original: [Table]
Revised: [Table]

Original:  epoch.
Revised: exact GP inference (with approximate likelihoods) is feasible [Table] resulting in O ( P Q 3 ) complexity for one epoch.

Original: Additionally, training can be done in mini-batches of digit subsets { Y p , X p } , hence the ELBO does not require the whole dataset in memory.
Revised: Additionally, training can be done in mini-batches of digit subsets { Y p , X p } , hence the ELBO does not require the whole dataset in memory.

Original: In cases where Q is large or Y p do not ﬁt into memory, the factorized kernel we propose may be combined with a sparse
Revised: In cases where Q is large or Y p do not ﬁt into memory [Table] the factorized kernel we propose may be combined with a sparse

Original: GP-VAE method (Ashman et al., 2020; Jazbec et al., 2020) to further reduce computational complexity.
Revised: GP-VAE method (Ashman et al., 2020; Jazbec et al., 2020) to further reduce computational complexity.

Original: 4. Experiments
Revised: 4. Experiments

Original: We follow the experimental setup from Casale et al.
Revised: We follow the experimental setup from Casale et al.

Original: (2018) in conditionally generating rotated images of MNIST handwritten digits (LeCun et al., 1998).
Revised: (2018) in conditionally generating rotated images of MNIST handwritten digits (LeCun et al., 1998).

Original: The dataset consists of P = 400 diﬀerent instances of the digit 3 at Q = 16 diﬀerent angles each, resulting in a total of N = 6400 possible combinations.
Revised: The dataset consists of P = 400 diﬀerent instances of the digit 3 at Q = 16 diﬀerent angles each, resulting in a total of N = 6400 possible combinations.

Original: From these combinations, N train = 4050 images are used for training and N test = 270 for testing.
Revised: From these combinations, N train =images are used for training and N test = 270 for testing.

Revised: The dimension of the latent space is set to L = 16 in all models.

Revised: In our FGP-VAE, we use 8 latent channels for local variables and 8 latent channels for global variables ( J = 18).

Original: We choose the same network architecture as the one in Casale et al.
Revised: We choose the same network architecture as the one in Casale et al.

Original: (2018) for all models (see Appendix A for implementation details).
Revised: (2018) for all models (see Appendix A for implementation details).

Original: For running the baselines, we used the code from Jazbec et al.
Revised: For running the baselines, we used the code from Jazbec et al.

Original: Moreover, we make use of the GECO algorithm (Rezende and Viola, 2018) to train our FGP-VAE model, as it improves training stability.
Revised: Moreover, we make use of the GECO algorithm (Rezende and Viola, 2018) to train our FGP-VAE model, as it improves training stability.

Original: Our code is made available at ANONYMOUS LINK .
Revised: Our code is made available at https://github.com/metodj/FGP-VAE .

Original: Reconstruction performance.
Revised: Reconstruction performance.

Original: We see qualitatively in Figure 1( a ) and quanitatively in
Revised: We see qualitatively in Figure 1( a ) and quanitatively in

Original: Table 1 that our proposed FGP-VAE clearly outperforms the non-correlated CVAE model (Sohn et al., 2015) and performs comparably to the non-factorized GP-VAE (Casale et al., 2018).
Revised: Table 1 that our proposed FGP-VAE clearly outperforms the non-correlated CVAE model (Sohn et al., 2015) and performs comparably to the non-factorized GP-VAE (Casale et al., 2018).

Original: However, our proposed model is an order of magnitude faster than the non-factorized
Revised: However, our proposed model is an order of magnitude faster than the non-factorized

Original: GP-VAE and reaches runtimes that are almost as fast as the CVAE.
Revised: GP-VAE and reaches runtimes that are almost as fast as the CVAE.

Original: Scaling behavior.
Revised: Scaling behavior.

Original: We additionally studied the scaling of our proposed model to differently sized subsets, including the full dataset.
Revised: We additionally studied the scaling of our proposed model to differently sized subsets, including the full dataset.

Original: (We do not compare against the other
Revised: (We do not compare against the other

Original: GP-VAE model here, since we did not manage to scale it to this larger dataset).
Revised: GP-VAE model here, since we did not manage to scale it to this larger dataset).

Original: We can seein Figure 1( b ) that our proposed model does not deteriorate in performance when scaling the dataset, while the runtime scales as gracefully as theoretically predicted (see Sec. 3). Extrapolation in the digit space. An additional attractive feature of the factorized model is that it can (unlike past works) extrapolate beyond digits observed in the training
Revised: We can see

Original: [Figure]
Revised: [Figure]

Original: t
Revised: [Figure] [Figure]

Original: i
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: M S E 0.

Original:  e / e p
Revised: t i m e / e p

Original: o c h [ s ]
Revised: o c h [ s ]

Original: [Figure]
Revised: [Figure]

Original:  data.
Revised: in Figure 1( b ) that our proposed model does not deteriorate in performance when scaling the dataset, while the runtime scales as gracefully as theoretically predicted (see Sec. 3). Extrapolation in the digit space. An additional attractive feature of the factorized model is that it can (unlike past works) extrapolate beyond digits observed in the training data.

Original: Past works only considered images seen during training and generated them at new angles.
Revised: Past works only considered images seen during training and generated them at new angles.

Original: The FGP-VAE can also generate arbitrary rotations for previously unseen digits given either a single example (sample the posterior of global latents), or even generated randomly (sample the prior of global latents).
Revised: The FGP-VAE can also generate arbitrary rotations for previously unseen digits given either a single example (sample the posterior of global latents), or even generated randomly (sample the prior of global latents).

Original: This is a consequence of our disentangled (and arguably much simpler) GP kernel.
Revised: This is a consequence of our disentangled (and arguably much simpler) GP kernel.

Original: The FGP-VAE achieved an MSE of 0 .
Revised: The FGP-VAE achieved an MSE of 0 .

Original: 0316 ± 0 .for
Revised: 0316 ± 0 .for

Original: the extrapolation experiment, which is only slightly worse than the 0 . 0284 ± 0 .for
Revised: the extrapolation experiment, which is only slightly worse than the 0 . 0284 ± 0 .for

Original: the non-extrapolation version of the experiment.
Revised: the non-extrapolation version of the experiment.

Original: This demonstrates the FGP-VAE’s strong extrapolation ability in the digit space.
Revised: This demonstrates the FGP-VAE’s strong extrapolation ability in the digit space.

Original: Generated images from the extrapolation experiment are shown in Appendix D. We further elaborate on the extrapolation properties of GP-VAE models in Appendix C.
Revised: Generated images from the extrapolation experiment are shown in Appendix D. We further elaborate on the extrapolation properties of GP-VAE models in Appendix C.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We have proposed a novel method for improving the scalability of GP-VAE models in settings where the auxiliary data consists of several independent features.
Revised: We have proposed a novel method for improving the scalability of GP-VAE models in settings where the auxiliary data consists of several independent features.

Original: Our method factorizes the latent GP kernel across the diﬀerent data features in diﬀerent latent dimensions, leading to a large reduction in inference time complexity.
Revised: Our method factorizes the latent GP kernel across the diﬀerent data features in diﬀerent latent dimensions, leading to a large reduction in inference time complexity.

Original: We have shown that our model is faster than existing non-factorized approaches in practice, while yielding a comparable predictive performance and oﬀering more general extrapolation properties.
Revised: We have shown that our model is faster than existing non-factorized approaches in practice, while yielding a comparable predictive performance and oﬀering more general extrapolation properties.

Original: In future work, it would be interesting to study the combination of our method with the recently proposed sparse GP-VAE approaches (Jazbec et al., 2020; Ashman et al., 2020), to reduce the inference time even further.
Revised: In future work, it would be interesting to study the combination of our method with the recently proposed sparse GP-VAE approaches (Jazbec et al., 2020; Ashman et al., 2020), to reduce the inference time even further.

Original: Appendix A. Implementation details
Revised: Appendix A. Implementation details

Original: For the rotated MNIST experiment described, we used the same neural networks architectures as in Casale et al.
Revised: For the rotated MNIST experiment described, we used the same neural networks architectures as in Casale et al.

Original: (2018): three convolutional layers followed by a fully connected layer in the inference network and vice-versa in the generative network.
Revised: (2018): three convolutional layers followed by a fully connected layer in the inference network and vice-versa in the generative network.

Original: For more details, see Table A.1.
Revised: For more details, see Table A.1.

Original: [Table] The FGP-VAE model is trained for 1000 epochs with a batch size of 220 images ( digits subsets, each with 11 rotations).
Revised: The FGP-VAE model is trained for 1000 epochs with a batch size of 220 images ( digits subsets, each with 11 rotations).

Original: The Adam optimizer (Kingma and Ba, 2014) is used with its default parameters and a learning rate of 0.001.
Revised: The Adam optimizer (Kingma and Ba, 2014) is used with its default parameters and a learning rate of 0.001.

Original: Moreover, the GECO algorithm  (Rezende and Viola, 2018) is used for training our FGP-VAE model in this experiment.
Revised: Moreover, the GECO algorithm [Table] (Rezende and Viola, 2018) is used for training our FGP-VAE model in this experiment.

Original: The reconstruction parameter in GECO was set to κ = 0 .
Revised: The reconstruction parameter in GECO was set to κ = 0 .

Original: 020 in all reported experiments.
Revised: 020 in all reported experiments.

Original: GP parameters are kept ﬁxed throughout training for FGP-VAE.
Revised: GP parameters are kept ﬁxed throughout training for FGP-VAE.

Original: The amplitude is set to σ = 1 and the length scale to r = 1.
Revised: The amplitude is set to σ = 1 and the length scale to r = 1.

Original: For the baseline GP-VAE model (Casale et al., 2018), GP parameters are optimized during training as proposed in Casale et al.
Revised: For the baseline GP-VAE model (Casale et al., 2018), GP parameters are optimized during training as proposed in Casale et al.

Original: Appendix B. Derivations
Revised: Appendix B. Derivations

Original: To simplify the notation, we use here one latent channel per feature set, i.e. J = 1 and
Revised: To simplify the notation, we use here one latent channel per feature set, i.e. J = 1 and

Original: L = 2.
Revised: L = 2.

Original: For z q ∈ Z p , we thus have that z q =
Revised: For z q ∈ Z p , we thus have that z q =

Original: [ z q z p ] T , where we drop latent channel superscripts for clarity.
Revised: [ z q z p ] T , where we drop latent channel superscripts for clarity.

Original: With z q we denote a local latent variable that is speciﬁc to the q -th rotation, while z p represents a global latent variable that is shared among all rotations of the p -th digit.
Revised: With z q we denote a local latent variable that is speciﬁc to the q -th rotation, while z p represents a global latent variable that is shared among all rotations of the p -th digit.

Original: Further, let z 1 p =
Revised: Further, let z 1 p =

Original: [ z 1 . . .
Revised: [ z 1 . . .

Original: z Q ] T ∈ R Q contain all local latent variables in Z p , and K p =
Revised: z Q ] T ∈ R Q contain all local latent variables in Z p , and K p =

Original: k 1 θ ( X p , X p ).
Revised: k 1 θ ( X p , X p ).

Original: For notational convenience, let µ q,l := µ lφ ( y q ) and σ q,l :=
Revised: For notational convenience, let µ q,l := µ lφ ( y q ) and σ q,l :=

Original: σ lφ ( y q ).
Revised: σ lφ ( y q ).

Original: We proceed as
Revised: We proceed as

Original: [Equation]
Revised: [Equation]

Original: By exploiting the symmetry of Gaussian distribution, N ( z | µ, σ ) =
Revised: By exploiting the symmetry of Gaussian distribution, N ( z | µ, σ ) =

Original: N ( µ | z, σ ), the ﬁrst integral equals a marginal GP likelihood in the standard GP regression with inputs X p and outputs
Revised: N ( µ | z, σ ), the ﬁrst integral equals a marginal GP likelihood in the standard GP regression with inputs X p and outputs

Original: µ p :=
Revised: µ p :=

Original: [ µ 1 , 1 . . .
Revised: [ µ 1 , 1 . . .

Original: µ Q, 1 ] T ∈ R Q with (heteroscedastic) noise σ p :=
Revised: µ Q, 1 ] T ∈ R Q with (heteroscedastic) noise σ p :=

Original: [ σ 1 , 1 . . .
Revised: [ σ 1 , 1 . . .

Original: σ Q, 1 ] T ∈ R Q .
Revised: σ Q, 1 ] T ∈ R Q .

Original: Combining the same symmetry property with a formula for conjugate posterior parameters for Gaussian likelihood with known heteroscedastic variance yields the following expression for the second integral
Revised: Combining the same symmetry property with a formula for conjugate posterior parameters for Gaussian likelihood with known heteroscedastic variance yields the following expression for the second integral

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: a closed form for the approximate posterior can be obtained as
Revised: a closed form for the approximate posterior can be obtained as

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] an FGP-VAE ELBO can be derived as follows using the standard steps:
Revised: Finally, an FGP-VAE ELBO can be derived as follows using the standard steps:

Original: [Equation]
Revised: [Equation]

Original: Appendix C. Amortization of auxiliary data in GP-VAE models
Revised: Appendix C. Amortization of auxiliary data in GP-VAE models

Original: Auxiliary data X is crucial in applications of GP-VAE models as it represents the data over which a GP prior is placed.
Revised: Auxiliary data X is crucial in applications of GP-VAE models as it represents the data over which a GP prior is placed.

Original: 3 While it is often fully observed, there are cases where auxiliary data is not given (or is only partially observed).
Revised: 3 While it is often fully observed, there are cases where auxiliary data is not given (or is only partially observed).

Original: In such instances, the authors in
Revised: In such instances, the authors in

Original: Casale et al.
Revised: Casale et al.

Original: (2018) rely on the GP-LVM
Revised: (2018) rely on the GP-LVM

Original: (Lawrence, 2004) to learn the missing parts of the auxiliary information.
Revised: (Lawrence, 2004) to learn the missing parts of the auxiliary information.

Original: Such an approach solves the issue of (partly) unobserved X in an elegant way, however by doing so, the extrapolation ability of GP-VAE models is diminished.
Revised: Such an approach solves the issue of (partly) unobserved X in an elegant way, however by doing so, the extrapolation ability of GP-VAE models is diminished.

Original: Suppose we want to generate new views or angles for previously unseen digits or objects.
Revised: Suppose we want to generate new views or angles for previously unseen digits or objects.

Original: In that case, we need to re-run the training optimization so that the respective GP-LVM vectors are obtained.
Revised: In that case, we need to re-run the training optimization so that the respective GP-LVM vectors are obtained.

Original: Note that GP-LVM vectors correspond to rows in the low-rank matrix Σ ∈ R P ×
Revised: Note that GP-LVM vectors correspond to rows in the low-rank matrix Σ ∈ R P ×

Original: m that is part of the GP kernel proposed in Casale et al.
Revised: m that is part of the GP kernel proposed in Casale et al.

Original:  Another way of endowing GP-VAE models with the extrapolation ability, besides considering factorized (and simpler) GP priors as done in our FGP-VAE, would be to amortize the GP-LVM information using a representation network r ζ : R K → R m , similar to what is done in Eslami et al.
Revised: a GP regression. Another way of endowing GP-VAE models with the extrapolation ability, besides considering factorized (and simpler) GP priors as done in our FGP-VAE, would be to amortize the GP-LVM information using a representation network r ζ : R K → R m , similar to what is done in Eslami et al.

Original: The representation for the p -th digit instance is then
Revised: The representation for the p -th digit instance is then

Original: [Equation]
Revised: [Equation]

Original: where Y p =
Revised: where Y p =

Original: [ y 1 . . .
Revised: [ y 1 . . .

Original: y Q ] T , and f is a chosen aggregation function, for instance, a sum or a mean.
Revised: y Q ] T , and f is a chosen aggregation function, for instance, a sum or a mean.

Original: Instead of GP-LVM vectors, the parameters of the representation network ζ would be learned jointly with the rest of GP-VAE parameters.
Revised: Instead of GP-LVM vectors, the parameters of the representation network ζ would be learned jointly with the rest of GP-VAE parameters.

Original: Appendix D. Extrapolation in the digit space
Revised: Appendix D. Extrapolation in the digit space