Original: C AN K ERNEL T RANSFER O PERATORS H ELP F LOW BASED G ENERATIVE M ODELS ?
Revised: C AN K ERNEL T RANSFER O PERATORS H ELP F LOW BASED G ENERATIVE M ODELS ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Flow-based generative models refer to deep generative models with tractable likelihoods, and offer several attractive properties including efﬁcient density estimation and sampling.
Revised: Flow-based generative models refer to deep generative models with tractable likelihoods, and offer several attractive properties including efﬁcient density estimation and sampling.

Original: Despite many advantages, current formulations (e.g., normalizing ﬂow) often have an expensive memory/runtime footprint, which hinders their use in a number of applications.
Revised: Despite many advantages, current formulations (e.g., normalizing ﬂow) often have an expensive memory/runtime footprint, which hinders their use in a number of applications.

Original: In this paper, we consider the setting where we have access to an autoencoder, which is suitably effective for the dataset of interest.
Revised: In this paper, we consider the setting where we have access to an autoencoder, which is suitably effective for the dataset of interest.

Original: Under some mild conditions, we show that we can calculate a mapping to a RKHS which subsequently enables deploying mature ideas from the kernel methods literature for ﬂow-based generative models.
Revised: Under some mild conditions, we show that we can calculate a mapping to a RKHS which subsequently enables deploying mature ideas from the kernel methods literature for ﬂow-based generative models.

Original: Speciﬁcally, we can explicitly map the RKHS distribution (i.e., approximate the ﬂow) to match or align with a template/well-characterized distribution, via kernel transfer operators.
Revised: Speciﬁcally, we can explicitly map the RKHS distribution (i.e., approximate the ﬂow) to match or align with a template/well-characterized distribution, via kernel transfer operators.

Original: This leads to a direct and resource efﬁcient approximation avoiding iterative optimization.
Revised: This leads to a direct and resource efﬁcient approximation avoiding iterative optimization.

Original: We empirically show that this simple idea yields competitive results on popular datasets such as CelebA, as well as promising results on a public 3D brain imaging dataset where the sample sizes are much smaller.
Revised: We empirically show that this simple idea yields competitive results on popular datasets such as CelebA, as well as promising results on a public 3D brain imaging dataset where the sample sizes are much smaller.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A ﬂow-based generative model refers to a deep generative model composed using a set of invertible transformations.
Revised: A ﬂow-based generative model refers to a deep generative model composed using a set of invertible transformations.

Original: While GANs and VAEs remain the two dominant generative models in the community, ﬂow based formulations have continually evolved and now offer competitive performance in applications including audio/speech synthesis Kim et al.
Revised: While GANs and VAEs remain the two dominant generative models in the community, ﬂow based formulations have continually evolved and now offer competitive performance in applications including audio/speech synthesis Kim et al.

Original: (2019; 2020), text to speech Miao et al.
Revised: (2019; 2020), text to speech Miao et al.

Original: (2020), photo-realistic image generation Kingma & Dhariwal (2018), and learning cross-domain mappings Mahajan et al.
Revised: (2020), photo-realistic image generation Kingma & Dhariwal (2018), and learning cross-domain mappings Mahajan et al.

Original: An important property of such models is the explicit use of a tractable likelihood function, which enables leveraging maximum likelihood principles during training as well as efﬁcient/exact density estimation and sampling.
Revised: An important property of such models is the explicit use of a tractable likelihood function, which enables leveraging maximum likelihood principles during training as well as efﬁcient/exact density estimation and sampling.

Original: The formulation is invertible by design but this involves higher memory requirements.
Revised: The formulation is invertible by design but this involves higher memory requirements.

Original: For example, permitting the bijective mapping to be expressive enough involves increases in the memory footprint Lee et al.
Revised: For example, permitting the bijective mapping to be expressive enough involves increases in the memory footprint Lee et al.

Original: Kim et al.
Revised: Kim et al.

Original: an issue that is a focus of several recent results Jacobsen et al.
Revised: an issue that is a focus of several recent results Jacobsen et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Moreover, in these models we need to calculate the inverse and backpropagate through all invertible transformations during training.
Revised: Moreover, in these models we need to calculate the inverse and backpropagate through all invertible transformations during training.

Original: Calculating the inverse incurs a multiplicative increase in cost, usually as a function of the feature dimension, relative to the calculation of the likelihood, an issue addressed to some extent in Dinh et al.
Revised: Calculating the inverse incurs a multiplicative increase in cost, usually as a function of the feature dimension, relative to the calculation of the likelihood, an issue addressed to some extent in Dinh et al.

Original: ; Kingma & Dhariwal (2018).
Revised: ; Kingma & Dhariwal (2018).

Original: At a high level, a ﬂow-based generative model bijectively pushes the data density from a source to a target, i.e., from a known simple distribution to an unknown (may be intractable) data distribution.
Revised: At a high level, a ﬂow-based generative model bijectively pushes the data density from a source to a target, i.e., from a known simple distribution to an unknown (may be intractable) data distribution.

Original: During training, we seek to learn this bijective mapping by maximizing the likelihood of the mapped training samples.
Revised: During training, we seek to learn this bijective mapping by maximizing the likelihood of the mapped training samples.

Original: In the generation step, we need the inverse of this mapping (given such an inverse exists) to map from a sample drawn from the known distribution back to the input (data) space.
Revised: In the generation step, we need the inverse of this mapping (given such an inverse exists) to map from a sample drawn from the known distribution back to the input (data) space.

Original: When the Jacobian of the transformation mapping can be efﬁciently computed or estimated (e.g., having a lower triangular form), directly optimizing the likelihood of the training samples is possible.
Revised: When the Jacobian of the transformation mapping can be efﬁciently computed or estimated (e.g., having a lower triangular form), directly optimizing the likelihood of the training samples is possible.

Original: However, in training ﬂow-based generative either we must restrict the expressiveness at each layer or fall back on more numerically heavy solutions, see (Chen et al., 2018).
Revised: However, in training ﬂow-based generative either we must restrict the expressiveness at each layer or fall back on more numerically heavy solutions, see (Chen et al., 2018).

Original: Next, we discuss how several existing results may provide a simpliﬁcation strategy.
Revised: Next, we discuss how several existing results may provide a simpliﬁcation strategy.

Original: 1. R ELATED WORKS AND R ATIONALE
Revised: 1. R ELATED WORKS AND R ATIONALE

Original: Our starting point is the existing literature on Koopman and Perron-Frobenius operators Song et al.
Revised: Our starting point is the existing literature on Koopman and Perron-Frobenius operators Song et al.

Original: ; Fukumizu et al.
Revised: ; Fukumizu et al.

Original: ; Klus et al.
Revised: ; Klus et al.

Original: (2020), that offers an arguably easier, optionally linear , procedure that can be used to analyze non-linear dynamics of measurements that evolve temporally.
Revised: (2020), that offers an arguably easier, optionally linear , procedure that can be used to analyze non-linear dynamics of measurements that evolve temporally.

Original: For instance, as described in Arbabi (2018); Lusch et al.
Revised: For instance, as described in Arbabi (2018); Lusch et al.

Original: (2018), if we view the data/measurements as evaluations of functions of the state (of a dynamical system) – where the functions are also called observables – then the entire set of such functions forms a linear vector space.
Revised: (2018), if we view the data/measurements as evaluations of functions of the state (of a dynamical system) – where the functions are also called observables – then the entire set of such functions forms a linear vector space.

Original: Transfer operators on this space describe a linear evolution of the dynamics, i.e., ﬁnite-dimensional nonlinear dynamics are replaced by inﬁnite-dimensional linear dynamics Brunton et al.
Revised: Transfer operators on this space describe a linear evolution of the dynamics, i.e., ﬁnite-dimensional nonlinear dynamics are replaced by inﬁnite-dimensional linear dynamics Brunton et al.

Original: (2017), perfectly evolving one set of measurements to another over time if the space can be well characterized.
Revised: (2017), perfectly evolving one set of measurements to another over time if the space can be well characterized.

Original: Of course, this is not practically beneﬁcial because constructing such inﬁnite-dimensional spaces could be intractable.
Revised: Of course, this is not practically beneﬁcial because constructing such inﬁnite-dimensional spaces could be intractable.

Original: Nonetheless, results in optimal control demonstrate that the idea can still be effective in speciﬁc cases, using approximations with either spectral analysis of large but ﬁnite number of functions Williams et al.
Revised: Nonetheless, results in optimal control demonstrate that the idea can still be effective in speciﬁc cases, using approximations with either spectral analysis of large but ﬁnite number of functions Williams et al.

Original: (2015) or via a search for potential eigenfunctions of the operators using neural networks Li et al.
Revised: (2015) or via a search for potential eigenfunctions of the operators using neural networks Li et al.

Original: ; Lusch et al.
Revised: ; Lusch et al.

Original: Within the last year, several results describe its potential beneﬁts of such operators in machine learning problems as well Li et al.
Revised: Within the last year, several results describe its potential beneﬁts of such operators in machine learning problems as well Li et al.

Original: Azencot et al.
Revised: Azencot et al.

Original: If we consider the transformations that ﬂow-based generative models learn as a non-linear dynamics, also used in (Chen et al., 2018), a data-driven approximation strategy one can consider is to map the given data (or distribution) into inﬁnite dimension space of functions through the kernel trick , which may allow the use of well known results based on kernel methods, including old and new results on powerful neural kernels Neal (1994);
Revised: If we consider the transformations that ﬂow-based generative models learn as a non-linear dynamics, also used in (Chen et al., 2018), a data-driven approximation strategy one can consider is to map the given data (or distribution) into inﬁnite dimension space of functions through the kernel trick , which may allow the use of well known results based on kernel methods, including old and new results on powerful neural kernels Neal (1994);

Original: Jacot et al.
Revised: Jacot et al.

Original: Arora et al.
Revised: Arora et al.

Original: Utilizing these results, a mean embedding on the corresponding Reproducing Kernel Hilbert Space (RKHS) would correspond to the distribution in the input space (the distribution from which input samples are drawn).
Revised: Utilizing these results, a mean embedding on the corresponding Reproducing Kernel Hilbert Space (RKHS) would correspond to the distribution in the input space (the distribution from which input samples are drawn).

Original: Therefore, the problem of identifying a nonlinear mapping (or dynamics) in the input space (going from an intractable distribution to a known distribution or vice-versa) reduces to estimating a linear mapping operator between two empirical mean kernel embeddings where recent results on kernel transfer operators Klus et al.
Revised: Therefore, the problem of identifying a nonlinear mapping (or dynamics) in the input space (going from an intractable distribution to a known distribution or vice-versa) reduces to estimating a linear mapping operator between two empirical mean kernel embeddings where recent results on kernel transfer operators Klus et al.

Original: (2020) could be relevant or applicable.
Revised: (2020) could be relevant or applicable.

Original: However, due to the high variability of the data, estimation of the distribution directly in the input space, as we will see shortly, can be difﬁcult.
Revised: However, due to the high variability of the data, estimation of the distribution directly in the input space, as we will see shortly, can be difﬁcult.

Original: But, if the input space is low-dimensional or otherwise structured, this problem could be mitigated.
Revised: But, if the input space is low-dimensional or otherwise structured, this problem could be mitigated.

Original: Fortunately, for many image datasets, one can identify a low-dimensional latent space such that, in theory, the above pipeline could be instantiated, enabling us to learn a transfer operator.
Revised: Fortunately, for many image datasets, one can identify a low-dimensional latent space such that, in theory, the above pipeline could be instantiated, enabling us to learn a transfer operator.

Original: Conceptually, it is not difﬁcult to see how the foregoing idea could potentially help (or simplify) ﬂow-based generative models.
Revised: Conceptually, it is not difﬁcult to see how the foregoing idea could potentially help (or simplify) ﬂow-based generative models.

Original: In principle, using a transfer operator, one could push-forward the input data distribution to a target distribution of our choice, if both have already been mapped to a sufﬁciently high dimensional space.
Revised: In principle, using a transfer operator, one could push-forward the input data distribution to a target distribution of our choice, if both have already been mapped to a sufﬁciently high dimensional space.

Original: If – additionally – the operator could also be inverted, this strategy may, at least, be viable.
Revised: If – additionally – the operator could also be inverted, this strategy may, at least, be viable.

Original: Of course, several key components are missing.
Revised: Of course, several key components are missing.

Original: We need to (a) assess if setting up a suitable inﬁnite dimensional space is possible, (b) identify if we can estimate the transfer operator and then ﬁnally, (c) check if the procedure works at all.
Revised: We need to (a) assess if setting up a suitable inﬁnite dimensional space is possible, (b) identify if we can estimate the transfer operator and then ﬁnally, (c) check if the procedure works at all.

Original: In the following sections of the paper, we will verify these key components and show that using only a linear operator yield surprisingly competitive results on several image generation tasks.
Revised: In the following sections of the paper, we will verify these key components and show that using only a linear operator yield surprisingly competitive results on several image generation tasks.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Auto-encoders.
Revised: Auto-encoders.

Original: Images often lie close to an (unknown) lower dimensional manifold M ⊂ R m such that dim ( M ) (cid:28) m , and operating with densities in a lower dimensional setting is often much easier.
Revised: Images often lie close to an (unknown) lower dimensional manifold M ⊂ R m such that dim ( M ) (cid:28) m , and operating with densities in a lower dimensional setting is often much easier.

Original: VAEs leverage this explicitly via auto-encoders.
Revised: VAEs leverage this explicitly via auto-encoders.

Original: If the parameterized empirical density is p τ ( x ) , in
Revised: If the parameterized empirical density is p τ ( x ) , in

Original: VAEs, we write it as (cid:82) p τ ( x | z ) p ( z ) dz where z is the low dimensional representation with a suitable prior.
Revised: VAEs, we write it as (cid:82) p τ ( x | z ) p ( z ) dz where z is the low dimensional representation with a suitable prior.

Original: We then use a decoder distribution q τ (cid:48)
Revised: We then use a decoder distribution q τ (cid:48)

Original: ( z | x ) and an encoder distribution p τ ( x | z ) and train the parameters τ and τ (cid:48) .
Revised: ( z | x ) and an encoder distribution p τ ( x | z ) and train the parameters τ and τ (cid:48) .

Original: In practice, p τ ( · ) and q τ (cid:48) ( · ) are assumed to be Gaussian, but it is known that jointly ﬁtting the manifold as well as regressing to a particular prior distribution can be challenging in VAEs Kingma et al.
Revised: In practice, p τ ( · ) and q τ (cid:48) ( · ) are assumed to be Gaussian, but it is known that jointly ﬁtting the manifold as well as regressing to a particular prior distribution can be challenging in VAEs Kingma et al.

Original: ; Dai & Wipf (2019).
Revised: ; Dai & Wipf (2019).

Original: Now, consider the following approach, where we do not impose a distributional assumption on z .
Revised: Now, consider the following approach, where we do not impose a distributional assumption on z .

Original: If a well-regularized auto-encoder is able to capture information about the data generating density Alain & Bengio (2014), we can think of z as the input measurements (likely, meaningful representations from the input data) which is subsequently mapped to an inﬁnite dimensional space (of observables of these measurements).
Revised: If a well-regularized auto-encoder is able to capture information about the data generating density Alain & Bengio (2014), we can think of z as the input measurements (likely, meaningful representations from the input data) which is subsequently mapped to an inﬁnite dimensional space (of observables of these measurements).

Original: Now, if we could push-forward the embedded RKHS distribution to the RKHS mapping of a simpler distribution, similar to ﬂow-based generation, one could easily sample from the simple (e.g., standard normal) distribution and transform it via the learned mapping to samples on the latent space.
Revised: Now, if we could push-forward the embedded RKHS distribution to the RKHS mapping of a simpler distribution, similar to ﬂow-based generation, one could easily sample from the simple (e.g., standard normal) distribution and transform it via the learned mapping to samples on the latent space.

Original: In summary, instead of explicitly searching for the eigenfunctions, we propose to Step 1: embed the density from an auto-encoder into a RKHS, Step 2: learn a kernel transfer operator in RKHS in one step.
Revised: In summary, instead of explicitly searching for the eigenfunctions, we propose to Step 1: embed the density from an auto-encoder into a RKHS, Step 2: learn a kernel transfer operator in RKHS in one step.

Original: In the remainder of this paper, we will provide the details to operationalize this idea and show that this simple approach, in fact, performs surprisingly well with highly favorable computational properties.
Revised: In the remainder of this paper, we will provide the details to operationalize this idea and show that this simple approach, in fact, performs surprisingly well with highly favorable computational properties.

Original: We now introduce the deﬁnition of reproducing kernel Hilbert space (RKHS) and the kernel embedding of probability distributions which are the building blocks of this paper.
Revised: We now introduce the deﬁnition of reproducing kernel Hilbert space (RKHS) and the kernel embedding of probability distributions which are the building blocks of this paper.

Original: Deﬁnition 1 (RKHS Aronszajn (1950)) .
Revised: Deﬁnition 1 (RKHS Aronszajn (1950)) .

Original: Given a set X and H to be a set of functions φ : X → R ,
Revised: Given a set X and H to be a set of functions φ : X → R ,

Original: H is called a reproducing kernel Hilbert space (RKHS) with corresponding product (cid:104) .
Revised: H is called a reproducing kernel Hilbert space (RKHS) with corresponding product (cid:104) .

Original: (cid:105) H if there exists a function k : X × X
Revised: (cid:105) H if there exists a function k : X × X

Original: → R (called a reproducing kernel) such that (i) ∀ X ∈ X , φ ∈
Revised: → R (called a reproducing kernel) such that (i) ∀ X ∈ X , φ ∈

Original: H , φ ( X ) = (cid:104) φ, k ( X, . )
Revised: H , φ ( X ) = (cid:104) φ, k ( X, . )

Original: (cid:105) H (ii) H = span ( { k ( X, . ) , X ∈ X} ) , where .
Revised: (cid:105) H (ii) H = span ( { k ( X, . ) , X ∈ X} ) , where .

Original: is the completion .
Revised: is the completion .

Original: The kernel mean embedding can be used to embed a probability measure in a RKHS.
Revised: The kernel mean embedding can be used to embed a probability measure in a RKHS.

Original: Deﬁnition 2 (Kernel Mean Embedding Smola et al.
Revised: Deﬁnition 2 (Kernel Mean Embedding Smola et al.

Original: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup X ∈X k ( X, X ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ H ∈ H , is deﬁned as µ H =
Revised: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup X ∈X k ( X, X ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ H ∈ H , is deﬁned as µ H =

Original: (cid:82) k
Revised: (cid:82) k

Original: ( X, . )
Revised: ( X, . )

Original: p ( X ) , and the mean embedding operator E : L 1 ( X ) → H is deﬁned as µ H = E p .
Revised: p ( X ) , and the mean embedding operator E : L 1 ( X ) → H is deﬁned as µ H = E p .

Original: For a characteristic kernel, the mapping from an input space distribution to its corresponding kernel mean embedding is one-to-one.
Revised: For a characteristic kernel, the mapping from an input space distribution to its corresponding kernel mean embedding is one-to-one.

Original: Thus, two distributions in the input space are identical if and only if their kernel mean embedding matches exactly.
Revised: Thus, two distributions in the input space are identical if and only if their kernel mean embedding matches exactly.

Original: This property enables Maximum Mean Discrepancy (MMD) for distribution matching.
Revised: This property enables Maximum Mean Discrepancy (MMD) for distribution matching.

Original: For a ﬁnite number of samples { X i } Ni =1 , drawn from the probability measure µ , an empirical estimation of µ H is ˆ µ H = 1 N (cid:80) Ni =1 k
Revised: For a ﬁnite number of samples { X i } Ni =1 , drawn from the probability measure µ , an empirical estimation of µ H is ˆ µ H = 1 N (cid:80) Ni =1 k

Original: ( X i , . ) .
Revised: ( X i , . ) .

Original: Deﬁnition 3 (Flow-based generative model
Revised: Deﬁnition 3 (Flow-based generative model

Original: Dinh et al.
Revised: Dinh et al.

Original: A ﬂow-based generative model explicitly learns the data distribution by trying to bijectively map it to a tractable density via invertible transformations.
Revised: A ﬂow-based generative model explicitly learns the data distribution by trying to bijectively map it to a tractable density via invertible transformations.

Original: Formally, given a random variable z following a tractable density, i.e., z ∼ p θ ( z ) , a ﬂow-based model learns an invertible mapping f φ such that the data sample x = f φ ( z ) and the corresponding data distribution, p (cid:101) θ ( x ) =
Revised: Formally, given a random variable z following a tractable density, i.e., z ∼ p θ ( z ) , a ﬂow-based model learns an invertible mapping f φ such that the data sample x = f φ ( z ) and the corresponding data distribution, p (cid:101) θ ( x ) =

Original: f ∗ p θ ( z ) , where, f ∗ = | dzdx | is the push-forward operator.
Revised: f ∗ p θ ( z ) , where, f ∗ = | dzdx | is the push-forward operator.

Original:  A S IMPLIFICATION STRATEGY FOR FLOW - BASED GENERATIVE MODELS
Revised:  A S IMPLIFICATION STRATEGY FOR FLOW - BASED GENERATIVE MODELS

Original: Flow-based generative model directly learn the mapping from a known distribution to the target data distribution in the input space through likelihood maximization.
Revised: Flow-based generative model directly learn the mapping from a known distribution to the target data distribution in the input space through likelihood maximization.

Original: During image generation, the fact that data may often lie on or near a low-dimensional manifold is usually not explicitly leveraged.
Revised: During image generation, the fact that data may often lie on or near a low-dimensional manifold is usually not explicitly leveraged.

Original: If one could estimate this manifold perfectly, we could uniquely identify a distribution in the input space.
Revised: If one could estimate this manifold perfectly, we could uniquely identify a distribution in the input space.

Original: But often, the manifold can only be empirically estimated.
Revised: But often, the manifold can only be empirically estimated.

Original: Our simplifying assumption here is that with some structural or smoothness constraints on the distribution on the latent space, we should still be able to approximately identify the true distribution.
Revised: Our simplifying assumption here is that with some structural or smoothness constraints on the distribution on the latent space, we should still be able to approximately identify the true distribution.

Original: VAEs use this intuition to make the latent distribution consistent with a standard normal distribution, and several contemporary works also propose estimating ﬂow on latent spaces Kingma et al.
Revised: VAEs use this intuition to make the latent distribution consistent with a standard normal distribution, and several contemporary works also propose estimating ﬂow on latent spaces Kingma et al.

Original: ; Mahajan et al.
Revised: ; Mahajan et al.

Original: We will follow this rationale and use an autoencoder – here, beneﬁting from the reduced variance in the latent space, our approach will effectively learn the mapping from a known simple distribution to the empirical latent distribution.
Revised: We will follow this rationale and use an autoencoder – here, beneﬁting from the reduced variance in the latent space, our approach will effectively learn the mapping from a known simple distribution to the empirical latent distribution.

Original: Structure of the latent space.
Revised: Structure of the latent space.

Original: We will assume that a suitable auto-encoder is either provided or can be successfully trained from scratch for the data at hand.
Revised: We will assume that a suitable auto-encoder is either provided or can be successfully trained from scratch for the data at hand.

Original: Given measurements from an input data space, M , we seek to learn a mapping E : M → L ⊂ R d , with d < m , where M ⊂ R m and
Revised: Given measurements from an input data space, M , we seek to learn a mapping E : M → L ⊂ R d , with d < m , where M ⊂ R m and

Original: L is the latent space.
Revised: L is the latent space.

Original: Additionally, we also learn the inverse of the mapping E which is denoted by
Revised: Additionally, we also learn the inverse of the mapping E which is denoted by

Original: D : L → M , i.e., we require D ◦ E
Revised: D : L → M , i.e., we require D ◦ E

Original: = id (identity).
Revised: = id (identity).

Original: Further, we require L to be a bounded subset of R d .
Revised: Further, we require L to be a bounded subset of R d .

Original: For simplicity, we model it as a unit hypersphere S d − 1 .
Revised: For simplicity, we model it as a unit hypersphere S d − 1 .

Original: This essentially ensures that L is (a) bounded (b) geodesically complete (c) geodesically convex space.
Revised: This essentially ensures that L is (a) bounded (b) geodesically complete (c) geodesically convex space.

Original: [Figure]
Revised: [Figure]

Original: Remark 1.
Revised: Remark 1.

Original: A bounded latent space with speciﬁed structure provides several beneﬁts over one without any such constraints.
Revised: A bounded latent space with speciﬁed structure provides several beneﬁts over one without any such constraints.

Original: First, it guarantees that the Gram matrix constructed with a (positive-deﬁnite) kernel function is bounded, which is required for a valid empirical estimation of the mean embedding in Def.
Revised: First, it guarantees that the Gram matrix constructed with a (positive-deﬁnite) kernel function is bounded, which is required for a valid empirical estimation of the mean embedding in Def.

Original: 2 Song et al.
Revised: 2 Song et al.

Original: Moreover, the projection onto the hypersphere also helps ensure the positive-deﬁniteness of kernels constructed using the latent representations, which is discussed in 3.2.
Revised: Moreover, the projection onto the hypersphere also helps ensure the positive-deﬁniteness of kernels constructed using the latent representations, which is discussed in 3.2.

Original: At this point, the requirement of geodesic completeness and convexity may not be apparent but these properties will be useful for sample generation.
Revised: At this point, the requirement of geodesic completeness and convexity may not be apparent but these properties will be useful for sample generation.

Original: 3. U SING T RANSFER O PERATORS TO ESTIMATE F LOW
Revised: 3. U SING T RANSFER O PERATORS TO ESTIMATE F LOW

Original: If training the auto-encoder provides us a suitably well-structured latent space, as desired, we can now study how the density transfer operators can help transfer a tractable density to the density on the latent space, i.e., mimicking the mechanics within a ﬂow-based generative model.
Revised: If training the auto-encoder provides us a suitably well-structured latent space, as desired, we can now study how the density transfer operators can help transfer a tractable density to the density on the latent space, i.e., mimicking the mechanics within a ﬂow-based generative model.

Original: Transfer operators.
Revised: Transfer operators.

Original: Let { X t } t ≥ 0 be a stationary and ergodic Markov process deﬁned on X .
Revised: Let { X t } t ≥ 0 be a stationary and ergodic Markov process deﬁned on X .

Original: Then, we can deﬁne the transition density function, p τ (with time lag τ ), by
Revised: Then, we can deﬁne the transition density function, p τ (with time lag τ ), by

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: a measurable set
Revised: a measurable set

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: Let L 1 ( X ) be the space of probability densities.
Revised: Let L 1 ( X ) be the space of probability densities.

Original: We wish to emulate the dynamical system of interest by instead utilizing a well-known transfer operator, namely the (Ruelle) Perron-Frobenius operator
Revised: We wish to emulate the dynamical system of interest by instead utilizing a well-known transfer operator, namely the (Ruelle) Perron-Frobenius operator

Original: Mayer (1980), where the “transfer” terminology originates from statistical mechanics.
Revised: Mayer (1980), where the “transfer” terminology originates from statistical mechanics.

Original: Notice that the notion of t in our work is not literal - it merely serves as a convenient way to describe the transfer operators in the context of evolution of states, and is also analogous to how ﬂow is considered explicitly as continuous dynamical systems in Neural ODE (Chen et al.
Revised: Notice that the notion of t in our work is not literal - it merely serves as a convenient way to describe the transfer operators in the context of evolution of states, and is also analogous to how ﬂow is considered explicitly as continuous dynamical systems in Neural ODE (Chen et al.

Original: Deﬁnition 4 (Perron-Frobenius operator Mayer (1980)) .
Revised: Deﬁnition 4 (Perron-Frobenius operator Mayer (1980)) .

Original: The Perron-Frobenius (PF) operator P :
Revised: The Perron-Frobenius (PF) operator P :

Original: L 1 ( X ) →
Revised: L 1 ( X ) →

Original: L 1 ( X ) push-forwards or transfers a probability density
Revised: L 1 ( X ) push-forwards or transfers a probability density

Original: p t ∈ L 1 ( X ) given the lag τ as ( P p t ) ( y )
Revised: p t ∈ L 1 ( X ) given the lag τ as ( P p t ) ( y )

Original: =
Revised: =

Original: (cid:82) p τ ( y | x ) p t ( x ) dx .
Revised: (cid:82) p τ ( y | x ) p t ( x ) dx .

Original: Remark 2.
Revised: Remark 2.

Original: If such an operator can be efﬁciently estimated, we can use it to transfer the tractable probability density from a known distribution p t to the target distribution p t + τ whose density is generally unknown.
Revised: If such an operator can be efﬁciently estimated, we can use it to transfer the tractable probability density from a known distribution p t to the target distribution p t + τ whose density is generally unknown.

Original: Since we do not make any distributional assumptions on the target distribution, a linear solution to such dynamics in the input space, in general, may not exist.
Revised: Since we do not make any distributional assumptions on the target distribution, a linear solution to such dynamics in the input space, in general, may not exist.

Original: Nevertheless, in spaces spanned by a sufﬁciently large set of non-linear functions, or speciﬁcally, an RKHS, one can potentially identify a linear operator that is equivalent to a highly non-linear operator in the input space.
Revised: Nevertheless, in spaces spanned by a sufﬁciently large set of non-linear functions, or speciﬁcally, an RKHS, one can potentially identify a linear operator that is equivalent to a highly non-linear operator in the input space.

Original: In Klus et al.
Revised: In Klus et al.

Original: , the authors showed two important results that deﬁne such operators in RKHS in terms of covariance and cross-covariance operators, namely kernel Perron-Frobenius operator (kPF) P k and embedded Perron-Frebenius operator (ePF) P E .
Revised: , the authors showed two important results that deﬁne such operators in RKHS in terms of covariance and cross-covariance operators, namely kernel Perron-Frobenius operator (kPF) P k and embedded Perron-Frebenius operator (ePF) P E .

Original: Let X ∼ p t and Y ∼ p t
Revised: Let X ∼ p t and Y ∼ p t

Original: + τ be observations at time t and t + τ .
Revised: + τ be observations at time t and t + τ .

Original: Let H and G be the RKHSs associated with a certain kernel k , φ H and φ G be their respective feature mapping.
Revised: Let H and G be the RKHSs associated with a certain kernel k , φ H and φ G be their respective feature mapping.

Original: The cross-covariance operator C Y X : H → G is deﬁned as C Y X
Revised: The cross-covariance operator C Y X : H → G is deﬁned as C Y X

Original: =
Revised: =

Original: E Y X
Revised: E Y X

Original: [ φ G ( Y ) ⊗ φ H ( X )] , and the covariance operator
Revised: [ φ G ( Y ) ⊗ φ H ( X )] , and the covariance operator

Original: C XX : H → H is deﬁned analogously.
Revised: C XX : H → H is deﬁned analogously.

Original: If both densities of interest live in the same RKHS, that is, p t ∈ H and p t + τ ∈ H , P k can be used to push forward the density such that p t + τ = P k p t .
Revised: If both densities of interest live in the same RKHS, that is, p t ∈ H and p t + τ ∈ H , P k can be used to push forward the density such that p t + τ = P k p t .

Original: However, in our setting, P k is not directly applicable since the densities we are interested are elements of L 1 ( X ) .
Revised: However, in our setting, P k is not directly applicable since the densities we are interested are elements of L 1 ( X ) .

Original: Therefore, the densities need to be ﬁrst embedded into the RKHS and can then be transferred using the P E in the embedded form, namely E p t + τ = ( P E ◦ E ) p t .
Revised: Therefore, the densities need to be ﬁrst embedded into the RKHS and can then be transferred using the P E in the embedded form, namely E p t + τ = ( P E ◦ E ) p t .

Original: Deﬁnition 5 (Embedded Perron-Frobenius operator Klus et al.
Revised: Deﬁnition 5 (Embedded Perron-Frobenius operator Klus et al.

Original: Given p t ∈ L 1 ( X ) and p t + τ ∈ L 1 ( X ) .
Revised: Given p t ∈ L 1 ( X ) and p t + τ ∈ L 1 ( X ) .

Original: Let µ t =
Revised: Let µ t =

Original: E p t and µ t
Revised: E p t and µ t

Original: + τ = E p t + τ be their respective kernel mean embedding.
Revised: + τ = E p t + τ be their respective kernel mean embedding.

Original: The kernel Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned by µ t
Revised: The kernel Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned by µ t

Original: + τ = P E µ t =
Revised: + τ = P E µ t =

Original: C Y X ( C XX + (cid:15)I ) − 1 µ t (2) , under the condition that (i) C XX is injective (ii) µ t
Revised: C Y X ( C XX + (cid:15)I ) − 1 µ t (2) , under the condition that (i) C XX is injective (ii) µ t

Original: ∈ Range ( C XX ) (iii) E
Revised: ∈ Range ( C XX ) (iii) E

Original: [ Y | X = · ] ∈ H .
Revised: [ Y | X = · ] ∈ H .

Original: Notice that the above deﬁned P E essentially has the same form as the kernel conditional embedding in Song et al.
Revised: Notice that the above deﬁned P E essentially has the same form as the kernel conditional embedding in Song et al.

Original: While the ﬁrst two conditions in the above deﬁnition can be satisﬁed (see
Revised: While the ﬁrst two conditions in the above deﬁnition can be satisﬁed (see

Original: Theorem 2 of Fukumizu et al.
Revised: Theorem 2 of Fukumizu et al.

Original: , the last condition requires ∃ f ∈ H s.t.
Revised: , the last condition requires ∃ f ∈ H s.t.

Original: ∀ x , f ( x ) =
Revised: ∀ x , f ( x ) =

Original: E [ Y | X = x ] , which highlights the importance of the kernel choice.
Revised: E [ Y | X = x ] , which highlights the importance of the kernel choice.

Original: With this operator, we can transfer p t to p t + τ in their embedded forms.
Revised: With this operator, we can transfer p t to p t + τ in their embedded forms.

Original: The following proposition demonstrates the implication of using the embedded Perron-Frobenius operator.
Revised: The following proposition demonstrates the implication of using the embedded Perron-Frobenius operator.

Original: Proposition 1 (Klus et al.
Revised: Proposition 1 (Klus et al.

Original: With the above notations, E ◦ P = P E ◦ E .
Revised: With the above notations, E ◦ P = P E ◦ E .

Original: Remark 3.
Revised: Remark 3.

Original: The commutativity in the proposition shows that the transferred kernel mean embedding of p t by the linear operator we constructed in RKHS is equivalent to the kernel mean embedding of transferred p t by a highly nonlinear operator in the input space.
Revised: The commutativity in the proposition shows that the transferred kernel mean embedding of p t by the linear operator we constructed in RKHS is equivalent to the kernel mean embedding of transferred p t by a highly nonlinear operator in the input space.

Original: Since p t + τ is generally intractable, the operator can only be empirically estimated.
Revised: Since p t + τ is generally intractable, the operator can only be empirically estimated.

Original: Given samples
Revised: Given samples

Original: X =
Revised: X =

Original: { x i } Ni =1 ∼ p Nt and Y =
Revised: { x i } Ni =1 ∼ p Nt and Y =

Original: { y i } Ni =1 ∼ p
Revised: { y i } Ni =1 ∼ p

Original: Nt + τ , let Φ H and Φ G denote the feature maps of X and Y , respectively.
Revised: Nt + τ , let Φ H and Φ G denote the feature maps of X and Y , respectively.

Original: The sample estimate of the embedded Perron-Frobenius operator is given by (cid:98) P E = (cid:98) C Y X ( (cid:98) C XX
Revised: The sample estimate of the embedded Perron-Frobenius operator is given by (cid:98) P E = (cid:98) C Y X ( (cid:98) C XX

Original: + (cid:15)I ) − 1 =
Revised: + (cid:15)I ) − 1 =

Original: Φ G ( G XX + N(cid:15)I ) − 1 Φ TH (3) , where G XX =
Revised: Φ G ( G XX + N(cid:15)I ) − 1 Φ TH (3) , where G XX =

Original: Φ (cid:62) H Φ H is the Gram matrix of Φ H .
Revised: Φ (cid:62) H Φ H is the Gram matrix of Φ H .

Original: To generate a sample of the transferred density which is approximately p t + τ using a sample x ∗ of p t , we can construct φ G ( y ∗ )
Revised: To generate a sample of the transferred density which is approximately p t + τ using a sample x ∗ of p t , we can construct φ G ( y ∗ )

Original: =
Revised: =

Original: ˆ P E φ H ( x ∗ i ) =
Revised: ˆ P E φ H ( x ∗ i ) =

Original: Φ G ( G XX + (cid:15)I ) − 1 k
Revised: Φ G ( G XX + (cid:15)I ) − 1 k

Original: ( X, x ∗ ) .
Revised: ( X, x ∗ ) .

Original: The distribution of the resulting samples has the following property,
Revised: The distribution of the resulting samples has the following property,

Original: Theorem 1 (Proof in A.1) .
Revised: Theorem 1 (Proof in A.1) .

Original: Let µ t + τ be the kernel mean embedding of the true distribution p t + τ .
Revised: Let µ t + τ be the kernel mean embedding of the true distribution p t + τ .

Original: The resulting empirical mean embedding ˆ µ ∗ t + τ = 1 n (cid:80) n i =1 φ G ( y ∗ i ) satisﬁes E [ˆ µ ∗ t + τ ] = µ t
Revised: The resulting empirical mean embedding ˆ µ ∗ t + τ = 1 n (cid:80) n i =1 φ G ( y ∗ i ) satisﬁes E [ˆ µ ∗ t + τ ] = µ t

Original: + τ
Revised: + τ

Original: Theorem 1 simply implies that ˆ P E ˆ µ ∗ t + τ is an unbiased estimator of the kernel mean embedding of the true distribution on the latent space.
Revised: Theorem 1 simply implies that ˆ P E ˆ µ ∗ t + τ is an unbiased estimator of the kernel mean embedding of the true distribution on the latent space.

Original: If the kernel is characteristic and the exact preimage exists, then φ − 1 G ( ˆ P E φ H ( x ∗ )) ∼ p t + τ asymptotically, which concludes our main result.
Revised: If the kernel is characteristic and the exact preimage exists, then φ − 1 G ( ˆ P E φ H ( x ∗ )) ∼ p t + τ asymptotically, which concludes our main result.

Original: 3. K ERNEL TRANSFER OPERATOR FOR SAMPLE GENERATION
Revised: 3. K ERNEL TRANSFER OPERATOR FOR SAMPLE GENERATION

Original: We now present an algorithm for sample generation assuming that a pre-trained auto-encoder is available.
Revised: We now present an algorithm for sample generation assuming that a pre-trained auto-encoder is available.

Original: The detailed algorithm is described in Figure 3.
Revised: The detailed algorithm is described in Figure 3.

Original: The idea is simple yet powerful: we ﬁrst generate n samples from a simple distribution restricted to a hypersphere (which we model as a uniform distribution on S n − 1 ).
Revised: The idea is simple yet powerful: we ﬁrst generate n samples from a simple distribution restricted to a hypersphere (which we model as a uniform distribution on S n − 1 ).

Original: Then, we will construct the operator on RKHS described in the previous section using the sampled points and the latent representations of training samples.
Revised: Then, we will construct the operator on RKHS described in the previous section using the sampled points and the latent representations of training samples.

Original: At inference time, we will use the operator to transfer new points sampled from the simple distribution to the target feature map.
Revised: At inference time, we will use the operator to transfer new points sampled from the simple distribution to the target feature map.

Original: Since it is not practically possible to compute the preimage directly from the inﬁnite-dimensional feature map, we left multiply with Φ G and use the geodesic interpolation (gI) module to construct an approximate preimage.
Revised: Since it is not practically possible to compute the preimage directly from the inﬁnite-dimensional feature map, we left multiply with Φ G and use the geodesic interpolation (gI) module to construct an approximate preimage.

Original: Finally, we decodes the interpolated latent representation to the image space.
Revised: Finally, we decodes the interpolated latent representation to the image space.

Original: A visual description is shown in Figure 2.
Revised: A visual description is shown in Figure 2.

Original: [Figure]
Revised: [Figure]

Original: Properties.
Revised: Properties.

Original: If K Z is indeed invertible, the empirical mean embedding of samples generated using the proposed algorithm (assuming the preimage is exact) is equal to the empirical mean embedding of the latent representations, indicating a match in distribution.
Revised: If K Z is indeed invertible, the empirical mean embedding of samples generated using the proposed algorithm (assuming the preimage is exact) is equal to the empirical mean embedding of the latent representations, indicating a match in distribution.

Original: Further, we use locallyweighted Fr´echet mean on sphere to construct approximate preimage of sample in RKHS.
Revised: Further, we use locallyweighted Fr´echet mean on sphere to construct approximate preimage of sample in RKHS.

Original: Since a closed-form solution for the weighted Fr´echet mean does not exist on the sphere, we propose to use a simple and efﬁcient algorithm, namely the geodesic interpolation (gI) Salehian et al.
Revised: Since a closed-form solution for the weighted Fr´echet mean does not exist on the sphere, we propose to use a simple and efﬁcient algorithm, namely the geodesic interpolation (gI) Salehian et al.

Original: (2015), that uses the geodesic on a hypersphere to iteratively computes the weighted Fr´echet mean Fr´echet (1948) of top γ latent representations (see
Revised: (2015), that uses the geodesic on a hypersphere to iteratively computes the weighted Fr´echet mean Fr´echet (1948) of top γ latent representations (see

Original: Fig. 4).
Revised: Fig. 4).

Original: The algorithm has the following properties: (i) the geodesically completeness of the latent space guarantees that the geodesic interpolation is well-deﬁned (ii) the geodesic convexity of the latent space guarantees that the output of “gI” algorithm lies on the latent space (iii) the “gI” algorithm converges asymptotically to the Fr´echet mean Salehian et al.
Revised: The algorithm has the following properties: (i) the geodesically completeness of the latent space guarantees that the geodesic interpolation is well-deﬁned (ii) the geodesic convexity of the latent space guarantees that the output of “gI” algorithm lies on the latent space (iii) the “gI” algorithm converges asymptotically to the Fr´echet mean Salehian et al.

Original: Choice of Kernels.
Revised: Choice of Kernels.

Original: In order to be able to linearize the dynamics between the prior and the target distribution, one must ﬁrst identify a set of nonlinear basis functions such that the corresponding dynamics lies in its span.
Revised: In order to be able to linearize the dynamics between the prior and the target distribution, one must ﬁrst identify a set of nonlinear basis functions such that the corresponding dynamics lies in its span.

Original: Known results in dynamical systems guarantee the existence of such a linear operator, such as the Koopman operator Koopman (1931), given an inﬁnite set of basis functions.
Revised: Known results in dynamical systems guarantee the existence of such a linear operator, such as the Koopman operator Koopman (1931), given an inﬁnite set of basis functions.

Original: But rather than identifying certain modes that best characterize the dynamics Williams et al.
Revised: But rather than identifying certain modes that best characterize the dynamics Williams et al.

Original: ; Brunton et al.
Revised: ; Brunton et al.

Original: we care most about minimizing the error of the transferred density, and whether the span of functions is rich/expressive enough and can be efﬁciently computed.
Revised: we care most about minimizing the error of the transferred density, and whether the span of functions is rich/expressive enough and can be efﬁciently computed.

Original: Therefore, the choice of kernel is important since it directly determines the family of functions spanned by the operator.
Revised: Therefore, the choice of kernel is important since it directly determines the family of functions spanned by the operator.

Original: We empirically evaluate the effect of using several different kernels by a simple experiment on MNIST.
Revised: We empirically evaluate the effect of using several different kernels by a simple experiment on MNIST.

Original: The MNIST digits are ﬁrst trained for 100 epochs using an autoencoder with latent space restricted to S 2 , then samples are generated using procedure described in Figure 3 using the respective kernel function.
Revised: The MNIST digits are ﬁrst trained for 100 epochs using an autoencoder with latent space restricted to S 2 , then samples are generated using procedure described in Figure 3 using the respective kernel function.

Original: Subplot (b) and (c) show the generated samples when using Radial
Revised: Subplot (b) and (c) show the generated samples when using Radial

Original: Basis Function (RBF) kernel and arc-cosine kernel, respectively.
Revised: Basis Function (RBF) kernel and arc-cosine kernel, respectively.

Original: Observe that the choice of kernel has a clear inﬂuence on the posterior, but and a kernel with superior empirical behavior would be desirable.
Revised: Observe that the choice of kernel has a clear inﬂuence on the posterior, but and a kernel with superior empirical behavior would be desirable.

Original: NTK.
Revised: NTK.

Original: We use Neural Tangent Kernel (NTK)
Revised: We use Neural Tangent Kernel (NTK)

Original: Jacot et al.
Revised: Jacot et al.

Original: as our embedding kernel due to the following properties: (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, and can be non-asymptotically approximated Arora et al.
Revised: as our embedding kernel due to the following properties: (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, and can be non-asymptotically approximated Arora et al.

Original: (b) For well-conditioned inputs (i.e., no duplicates) on the sphere, the positive-deﬁniteness of NTK is proved in Jacot et al.
Revised: (b) For well-conditioned inputs (i.e., no duplicates) on the sphere, the positive-deﬁniteness of NTK is proved in Jacot et al.

Original: Therefore, invertibility of G XX is guaranteed if the sampling distribution is restricted on a
Revised: Therefore, invertibility of G XX is guaranteed if the sampling distribution is restricted on a

Original: Generator
Revised: Generator

Original: (a) Let z ∗ be a sample drawn from u S d − 1 .
Revised: (a) Let z ∗ be a sample drawn from u S d − 1 .

Original: We ﬁrst post-multiply with k ( z ∗ ,. )
Revised: We ﬁrst post-multiply with k ( z ∗ ,. )

Original: to get
Revised: to get

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: the geodesic interpolation algorithm (as presented in Fig. 4).
Revised: the geodesic interpolation algorithm (as presented in Fig. 4).

Original: hypersphere, which is true by our modeling choice.
Revised: hypersphere, which is true by our modeling choice.

Original: (c) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough
Revised: (c) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough

Original: Arora et al.
Revised: Arora et al.

Original: The embedding of generated samples for MNIST using NTK is also shown in
Revised: The embedding of generated samples for MNIST using NTK is also shown in

Original: Fig.
Revised: Fig.

Original: Observe that the sample distribution more closely resembles the original distribution.
Revised: Observe that the sample distribution more closely resembles the original distribution.

Original: Nystr¨om’s approximation.
Revised: Nystr¨om’s approximation.

Original: A known bottleneck with kernel based methods is instantiating a kernel matrix whose size depends on the dataset size.
Revised: A known bottleneck with kernel based methods is instantiating a kernel matrix whose size depends on the dataset size.

Original: Here, we use use the Nystr¨om’s method Nystr¨om (1930) to obtain approximate kernel matrices.
Revised: Here, we use use the Nystr¨om’s method Nystr¨om (1930) to obtain approximate kernel matrices.

Original: More speciﬁcally, a kernel matrix K ∈ R N × N can be approximated by (cid:101) K = KS (cid:0) S T KS (cid:1) † S T K , where S ∈ R N × s is the sampling matrix.
Revised: More speciﬁcally, a kernel matrix K ∈ R N × N can be approximated by (cid:101) K = KS (cid:0) S T KS (cid:1) † S T K , where S ∈ R N × s is the sampling matrix.

Original: To minimize information loss, we use the RecursiveRLS-Nystr¨om algorithm proposed in Musco &
Revised: To minimize information loss, we use the RecursiveRLS-Nystr¨om algorithm proposed in Musco &

Original: Musco (2017) to sample S based on the approximate ridge leverage scores.
Revised: Musco (2017) to sample S based on the approximate ridge leverage scores.

Original: We refer the readers to the work by Musco & Musco (2017) for detailed algorithm.
Revised: We refer the readers to the work by Musco & Musco (2017) for detailed algorithm.

Original: Notice that after the approximation, we only need to store KS ∈
Revised: Notice that after the approximation, we only need to store KS ∈

Original: R N × s and S T
Revised: R N × s and S T

Original: KS ∈ R s × s matrices to obtain the Nystr¨om approximation of K , bringing down the memory complexity from O ( N 2 ) to O
Revised: KS ∈ R s × s matrices to obtain the Nystr¨om approximation of K , bringing down the memory complexity from O ( N 2 ) to O

Original: ( Ns + s 2 ) .
Revised: ( Ns + s 2 ) .

Original: When s (cid:28) N , the savings are signiﬁcant.
Revised: When s (cid:28) N , the savings are signiﬁcant.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: [Table]
Revised: [Table]

Original: We qualitatively evaluate the proposed generative model on two types of applications, (a) datasets with a sufﬁcient number of samples to model the data distribution (b) datasets with fewer number of samples so that learning the data distribution is harder .
Revised: We qualitatively evaluate the proposed generative model on two types of applications, (a) datasets with a sufﬁcient number of samples to model the data distribution (b) datasets with fewer number of samples so that learning the data distribution is harder .

Original: In the ﬁrst case, we use standard vision datasets, including
Revised: In the ﬁrst case, we use standard vision datasets, including

Original: MNIST, CIFAR and CelebA, where the number of data samples is larger than the dimension.
Revised: MNIST, CIFAR and CelebA, where the number of data samples is larger than the dimension.

Original: In order to handle the more challenging second case we use a dataset of T1 Magnetic Resonance (MR) images from a public brain imaging project called Alzheimer’s Disease NeuroImaging Initiative (ADNI).
Revised: In order to handle the more challenging second case we use a dataset of T1 Magnetic Resonance (MR) images from a public brain imaging project called Alzheimer’s Disease NeuroImaging Initiative (ADNI).

Original: The purpose of the ﬁrst set of experiments is to show that the proposed method yields competitive measures compared to other non-adversarial generative methods while enjoying the beneﬁt of one step density estimation.
Revised: The purpose of the ﬁrst set of experiments is to show that the proposed method yields competitive measures compared to other non-adversarial generative methods while enjoying the beneﬁt of one step density estimation.

Original: In the second setting, our goal is to demonstrate that unlike traditional non-adversarial models, our proposed method produces reasonable sample generation both quantitative and qualitatively.
Revised: In the second setting, our goal is to demonstrate that unlike traditional non-adversarial models, our proposed method produces reasonable sample generation both quantitative and qualitatively.

Original: Computer vision datasets: We evaluate the quality by calculating the Fr´echet Inception Distance (FID)
Revised: Computer vision datasets: We evaluate the quality by calculating the Fr´echet Inception Distance (FID)

Original: Heusel et al.
Revised: Heusel et al.

Original: All models share the same encoder-decoder architecture and trained for 100 epochs.
Revised: All models share the same encoder-decoder architecture and trained for 100 epochs.

Original: Subscript 10k indicates the kPF is estimated using 10000 latent points, whereas Nystr¨om indicates approximation using 4000 landmark points from all latent points.
Revised: Subscript 10k indicates the kPF is estimated using 10000 latent points, whereas Nystr¨om indicates approximation using 4000 landmark points from all latent points.

Original: Comparative results are shown in Table 1.
Revised: Comparative results are shown in Table 1.

Original: We compare our results to vanilla VAE Kingma & Welling (2013), 2-Stage VAE
Revised: We compare our results to vanilla VAE Kingma & Welling (2013), 2-Stage VAE

Original: Dai & Wipf (2019), and Glow + AE (denoted by SRAE Glow in the table).
Revised: Dai & Wipf (2019), and Glow + AE (denoted by SRAE Glow in the table).

Original: FIDs of two best GAN methods are also reported from Dai & Wipf (2019) for comparison (note that the reported scores have been optimized by large-scale neural architecture search).
Revised: FIDs of two best GAN methods are also reported from Dai & Wipf (2019) for comparison (note that the reported scores have been optimized by large-scale neural architecture search).

Original: We observe that for images with structured feature spaces, e.g., MNIST and CelebA, the proposed method matches or outperforms other non-adversarial generative models.
Revised: We observe that for images with structured feature spaces, e.g., MNIST and CelebA, the proposed method matches or outperforms other non-adversarial generative models.

Original: On the contrary, on images with less structured features such as CIFAR10, our model performs worse.
Revised: On the contrary, on images with less structured features such as CIFAR10, our model performs worse.

Original: We suspect that this is due to a non-smooth latent space, and we evaluate this by showing that visually similar images picked by our operators decodes to out-of-distribution (blurred-out) images (see Fig. 6).
Revised: We suspect that this is due to a non-smooth latent space, and we evaluate this by showing that visually similar images picked by our operators decodes to out-of-distribution (blurred-out) images (see Fig. 6).

Original: Here, we use a regularized AE Ghosh et al.
Revised: Here, we use a regularized AE Ghosh et al.

Original: (2020) with a latent space restricted to the hypersphere (denoted by SRAE).
Revised: (2020) with a latent space restricted to the hypersphere (denoted by SRAE).

Original: Further, we present qualitative results on CelebA (in Fig. 7), where we compare our kPF based model with other density estimation techniques on the latent space.
Revised: Further, we present qualitative results on CelebA (in Fig. 7), where we compare our kPF based model with other density estimation techniques on the latent space.

Original: Observe that our model generates comparative visual results with SRAE Glow .
Revised: Observe that our model generates comparative visual results with SRAE Glow .

Original: Hence, when a sufﬁcient number of samples are available, our method performs as good as the alternatives, which is beneﬁcial given the efﬁcient training stage.
Revised: Hence, when a sufﬁcient number of samples are available, our method performs as good as the alternatives, which is beneﬁcial given the efﬁcient training stage.

Original: We present training time comparison and show that our proposed method is about 40 × faster as in Fig.
Revised: We present training time comparison and show that our proposed method is about 40 × faster as in Fig.

Original: [Figure]
Revised: [Figure]

Original: : In this section, we present results on generating high-resolution (160 × 196 × 160) 3D brain images from ADNI consists of 183 samples from group AD (diagnosed as
Revised: : In this section, we present results on generating high-resolution (160 × 196 × 160) 3D brain images from ADNI consists of 183 samples from group AD (diagnosed as

Original: Alzheimer’s Disease) and 291 samples from group CN (control normals).
Revised: Alzheimer’s Disease) and 291 samples from group CN (control normals).

Original: In this setting (where n = 474 (cid:28) d = 5017600 ), it is often hard to model the data distribution using either variational methods or ﬂow-based methods due to the high variance of the data or the memory-inefﬁciency of the operations.
Revised: In this setting (where n = 474 (cid:28) d = 5017600 ), it is often hard to model the data distribution using either variational methods or ﬂow-based methods due to the high variance of the data or the memory-inefﬁciency of the operations.

Original: However, beneﬁting from the regularized and linear nature of our kernel operator
Revised: However, beneﬁting from the regularized and linear nature of our kernel operator

Original: Willoughby (1979); Arora et al.
Revised: Willoughby (1979); Arora et al.

Original: (2020), we can still generate high-quality samples in such resolution that are in-distribution.
Revised: (2020), we can still generate high-quality samples in such resolution that are in-distribution.

Original: As before, we present the comparative results with respect to the VAE model.
Revised: As before, we present the comparative results with respect to the VAE model.

Original: The generated samples presented in Fig.
Revised: The generated samples presented in Fig.

Original: 8 clearly demonstrate that proposed method generates sharper images.
Revised: 8 clearly demonstrate that proposed method generates sharper images.

Original: To evaluate whether these results are also scientiﬁcally meaningful (and not merely visually pleasing), we tested consistency between statistical group difference testing on the real images (groups were AD and CN) and the same testing performed on the generated samples.
Revised: To evaluate whether these results are also scientiﬁcally meaningful (and not merely visually pleasing), we tested consistency between statistical group difference testing on the real images (groups were AD and CN) and the same testing performed on the generated samples.

Original: We performed a FWER corrected two-sample t-test Ashburner & Friston (2000) in a manner consistent with standard practice Ashburner & Friston (2000);
Revised: We performed a FWER corrected two-sample t-test Ashburner & Friston (2000) in a manner consistent with standard practice Ashburner & Friston (2000);

Original: Winkler et al.
Revised: Winkler et al.

Original: The results (see Fig 8) show that while there is a deterioration in regions identiﬁed to be affected by disease (and different across groups), most of the statistically-signiﬁcant regions from the test on the original images are preserved in tests performed on the generated images.
Revised: The results (see Fig 8) show that while there is a deterioration in regions identiﬁed to be affected by disease (and different across groups), most of the statistically-signiﬁcant regions from the test on the original images are preserved in tests performed on the generated images.

Original: Take-home message: We conclude that due to the kernel approach and one-step formulation of the proposed model, it offers better memory and sample efﬁciency than other non-adversarial generative methods.
Revised: Take-home message: We conclude that due to the kernel approach and one-step formulation of the proposed model, it offers better memory and sample efﬁciency than other non-adversarial generative methods.

Original: This is useful in many tasks, e.g., medical image analysis, where the number of samples is usually much less than the dimension of data.
Revised: This is useful in many tasks, e.g., medical image analysis, where the number of samples is usually much less than the dimension of data.

Original: Moreover, in case of datasets with larger number of samples, the proposed method still performs well (and sometimes better) compared to alternatives, with a sizably smaller resource footprint.
Revised: Moreover, in case of datasets with larger number of samples, the proposed method still performs well (and sometimes better) compared to alternatives, with a sizably smaller resource footprint.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have shown that when a dataset allows a lowdimensional structured latent space representation, a kernel
Revised: We have shown that when a dataset allows a lowdimensional structured latent space representation, a kernel

Original: Perron-Frobenius operator can provide an efﬁcient strategy to simplify ﬂow-based generative models to transfer the density from a known simple distribution to the empirical data distribution.
Revised: Perron-Frobenius operator can provide an efﬁcient strategy to simplify ﬂow-based generative models to transfer the density from a known simple distribution to the empirical data distribution.

Original: While the algorithmic simpliﬁcations proposed here can be variously useful, deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS remain unresolved at this time.
Revised: While the algorithmic simpliﬁcations proposed here can be variously useful, deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS remain unresolved at this time.

Original: While the problem of deriving the density has been addressed in Schuster et al.
Revised: While the problem of deriving the density has been addressed in Schuster et al.

Original: (2020), identifying a pre-image can be hard and often ill-posed.
Revised: (2020), identifying a pre-image can be hard and often ill-posed.

Original: We also note that our geodesic interpolation only converges asymptotically to the weighted Fr´echet mean which we use as the approximate preimage, and additional improvements on this front are possible.
Revised: We also note that our geodesic interpolation only converges asymptotically to the weighted Fr´echet mean which we use as the approximate preimage, and additional improvements on this front are possible.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we show that with the assistance of recent developments in regularized autoencoder and neural kernels, a linear kernel transfer operator can potentially be an efﬁcient substitute to ﬂow-based generative models.
Revised: In this paper, we show that with the assistance of recent developments in regularized autoencoder and neural kernels, a linear kernel transfer operator can potentially be an efﬁcient substitute to ﬂow-based generative models.

Original: Our proposed method shows comparable empirical results to other state-of-theart generative models on several computer vision datasets, while enjoying higher computational efﬁciency.
Revised: Our proposed method shows comparable empirical results to other state-of-theart generative models on several computer vision datasets, while enjoying higher computational efﬁciency.

Original: The results on brain imaging data also shed light on the potential application on highvolume data generation, which are typically hard to model using existing methods.
Revised: The results on brain imaging data also shed light on the potential application on highvolume data generation, which are typically hard to model using existing methods.

Original: A A PPENDIX A. P ROOF FOR T HEOREM
Revised: A A PPENDIX A. P ROOF FOR T HEOREM

Original: The proof follows easily from the fact that ˆ P E is linear and independent to { x i } ni =1 ,
Revised: The proof follows easily from the fact that ˆ P E is linear and independent to { x i } ni =1 ,

Original: E [ˆ µ ∗ t + τ ] = E [
Revised: E [ˆ µ ∗ t + τ ] = E [

Original: 1 n n (cid:88) i =
Revised: 1 n n (cid:88) i =

Original: ˆ P E φ ( x i )]
Revised: ˆ P E φ ( x i )]

Original: =
Revised: =

Original: E [ ˆ P E ] E [
Revised: E [ ˆ P E ] E [

Original: 1 n n (cid:88) i = φ ( x i )]
Revised: 1 n n (cid:88) i = φ ( x i )]

Original: = P E µ t = µ t
Revised: = P E µ t = µ t

Original: + τ
Revised: + τ

Original: A. A BLATION STUDIES
Revised: A. A BLATION STUDIES

Original: We present ablation studies on MNIST (Fig. 10), showing importance and different components of our proposed method.
Revised: We present ablation studies on MNIST (Fig. 10), showing importance and different components of our proposed method.

Original: Our proposed framework in general consists of (a) learning latent space (b) estimating transfer operator using kPF.
Revised: Our proposed framework in general consists of (a) learning latent space (b) estimating transfer operator using kPF.

Original: We replace the ﬁrst part in two ways (i) remove AE and directly transfer the distribution from image space to the known distribution (ii) replace AE with known invertible basis representation, e.g., mapping onto Fourier basis .
Revised: We replace the ﬁrst part in two ways (i) remove AE and directly transfer the distribution from image space to the known distribution (ii) replace AE with known invertible basis representation, e.g., mapping onto Fourier basis .

Original: In both these cases, we observe that the generation quality deteriorates.
Revised: In both these cases, we observe that the generation quality deteriorates.

Original: Notice that such degradation can be justiﬁed due to the lack of smoothness on the input space.
Revised: Notice that such degradation can be justiﬁed due to the lack of smoothness on the input space.

Original: Next, we ﬁxed the AE and replace the kPF operator with a normalizing ﬂow model Kingma & Dhariwal (2018) as transfer operator.
Revised: Next, we ﬁxed the AE and replace the kPF operator with a normalizing ﬂow model Kingma & Dhariwal (2018) as transfer operator.

Original: Observe that the generation quality are comparable between the two methods.
Revised: Observe that the generation quality are comparable between the two methods.

Original: [Table]
Revised: [Table]

Original: We further show the advantage of using a regularized
Revised: We further show the advantage of using a regularized

Original: AE over a regular one.
Revised: AE over a regular one.

Original: The regularization scheme we applied is derived from Ghosh et al.
Revised: The regularization scheme we applied is derived from Ghosh et al.

Original: (2020), where we apply spectral normalization on all encoder-decoder transformations.
Revised: (2020), where we apply spectral normalization on all encoder-decoder transformations.

Original: Regularization has been shown to lead to smoother latent spaces Alain & Bengio (2014), and therefore is critical in our interpolation-based sampling approach.
Revised: Regularization has been shown to lead to smoother latent spaces Alain & Bengio (2014), and therefore is critical in our interpolation-based sampling approach.

Original: The effect of such regularization can be seen in Table 2, where it leads to higher FIDs in all three datasets using smoother latent space.
Revised: The effect of such regularization can be seen in Table 2, where it leads to higher FIDs in all three datasets using smoother latent space.

Original: Take-home message: To summarize, the presented ablation study demonstrates the importance of the latent space as well as the richness of our one-step estimated operator.
Revised: Take-home message: To summarize, the presented ablation study demonstrates the importance of the latent space as well as the richness of our one-step estimated operator.

Original: Moreover, the boundedness and smoothness of the latent space is crucial as can be justiﬁed by replacing AE with a Fourier basis or by using a regularized AE.
Revised: Moreover, the boundedness and smoothness of the latent space is crucial as can be justiﬁed by replacing AE with a Fourier basis or by using a regularized AE.

Original: A. ADNI DATASET AND BRAIN IMAGE PROCESSING
Revised: A. ADNI DATASET AND BRAIN IMAGE PROCESSING

Original: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging
Revised: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging

Original: Initiative (ADNI) database (adni.loni.usc.edu).
Revised: Initiative (ADNI) database (adni.loni.usc.edu).

Original: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.
Revised: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.

Original: A complete listing of ADNI investigators can be found at: .
Revised: A complete listing of ADNI investigators can be found at: .

Original: The T1 MR brain data we use consists of images from 184 subjects diagnosed with Alzheimers’s disease and 292 control normal subjects.
Revised: The T1 MR brain data we use consists of images from 184 subjects diagnosed with Alzheimers’s disease and 292 control normal subjects.

Original: Images were ﬁrst coregisted to a MNI template and segmented to keep only the white matter and grey matter.
Revised: Images were ﬁrst coregisted to a MNI template and segmented to keep only the white matter and grey matter.

Original: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1] .
Revised: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1] .

Original: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images.
Revised: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: I MAGE QUALITY 
Revised: I MAGE QUALITY EVALUATION

Original: [Figure]
Revised: [Figure]

Original: A. C HOICE OF γ
Revised: A. C HOICE OF γ

Original: In the inference stage our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by interpolating among the top γ latent representations of the training samples weighted by their kernel values.
Revised: In the inference stage our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by interpolating among the top γ latent representations of the training samples weighted by their kernel values.

Original: The choice of γ therefore has implications on the generation quality.
Revised: The choice of γ therefore has implications on the generation quality.

Original: From
Revised: From

Original: Figure we can observe that, in general, FID worsens as γ increases.
Revised: Figure we can observe that, in general, FID worsens as γ increases.

Original: This observation aligns with our intuition of preserving only the local similarities in kernel embeddings and similar idea has been used in the literature Kwok & Tsang (2004).
Revised: This observation aligns with our intuition of preserving only the local similarities in kernel embeddings and similar idea has been used in the literature Kwok & Tsang (2004).

Original: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1 , generated samples will just be reconstructions of training samples).
Revised: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1 , generated samples will just be reconstructions of training samples).

Original: Therefore, in our experiments, we choose γ = 10 to achieve a balance between generation quality and the distance to training samples.
Revised: Therefore, in our experiments, we choose γ = 10 to achieve a balance between generation quality and the distance to training samples.

Original: A. E XPERIMENTAL SPECIFICATIONS
Revised: A. E XPERIMENTAL SPECIFICATIONS

Original: For all experiments in Table 1 except the GAN experiments, we adapted a custom ResNet implementation where each block has residual connection
Revised: For all experiments in Table 1 except the GAN experiments, we adapted a custom ResNet implementation where each block has residual connection

Original: [Equation]
Revised: [Equation]

Original: The detailed architecture is given by the following table
Revised: The detailed architecture is given by the following table

Original: MNIST
Revised: MNIST

Original: CIFARCelebA
Revised: CIFARCelebA

Original: Encoder 5x5 conv, stride
Revised: Encoder 5x5 conv, stride

Original: ResBlock 32 ×
Revised: ResBlock 32 ×

Original: ResBlock 32 × ResBlock 32 ×
Revised: ResBlock 32 × ResBlock 32 ×

Original: ResBlock 64 × ResBlock 64 ×
Revised: ResBlock 64 × ResBlock 64 ×

Original: ResBlock 64 × ResBlock 128 ×
Revised: ResBlock 64 × ResBlock 128 ×

Original: ResBlock 128 × ResBlock 128 ×
Revised: ResBlock 128 × ResBlock 128 ×

Original: ResBlock 256 ×
Revised: ResBlock 256 ×

Original: ResBlock 256 × ResBlock 256 ×
Revised: ResBlock 256 × ResBlock 256 ×

Original: ResBlock 512 ×
Revised: ResBlock 512 ×

Original: Decoder
Revised: Decoder

Original: ResBlock 256 × ResBlock 256 ×
Revised: ResBlock 256 × ResBlock 256 ×

Original: ResBlock 512 ×
Revised: ResBlock 512 ×

Original: ResBlock 128 × ResBlock 128 ×
Revised: ResBlock 128 × ResBlock 128 ×

Original: ResBlock 256 ×
Revised: ResBlock 256 ×

Original: ResBlock 64 × ResBlock 64 ×
Revised: ResBlock 64 × ResBlock 64 ×

Original: ResBlock 128 ×
Revised: ResBlock 128 ×

Original: ResBlock 32 × ResBlock 32 ×
Revised: ResBlock 32 × ResBlock 32 ×

Original: ResBlock 64 ×
Revised: ResBlock 64 ×

Original: ResBlock 32 ×5x5 conv, stride
Revised: ResBlock 32 ×5x5 conv, stride

Original: We train  for 100 epochs on optimizer with β = (0 .
Revised: We train each model for 100 epochs on every dataset

Original:  and batch size of 512.
Revised: Adam optimizer Kingma & Ba 2014) with β = (0 . 9 , 0 . 999) and batch size of 512.

Original:  learning rate starts from 1e-3 and is halved every 30 epochs
Revised: The learning rate starts from 1e-3 and is halved every 30 epochs.

Revised: We used implemetation of Novak et al.

Revised: (2020) to estimate the NTK with four fullyconnected layers, each with 10000 units, and Erf activation.

Original: VAE and 2-Stage VAE are trained using learnable- γ as in Dai & Wipf (2019).
Revised: VAE and 2-Stage VAE are trained using learnable- γ as in Dai & Wipf (2019).

Original: For experiments in A.4 and A.5, we adopted the training procedure and architecture in Ghosh et al.
Revised: For experiments in A.4 and A.5, we adopted the training procedure and architecture in Ghosh et al.