Original: C AN K ERNEL T RANSFER O PERATORS H ELP F LOW BASED G ENERATIVE M ODELS ?
Revised: C AN K ERNEL T RANSFER O PERATORS H ELP F LOW BASED G ENERATIVE M ODELS ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Flow-based generative models refer to deep generative models with tractable likelihoods, and offer several attractive properties including efﬁcient density estimation and sampling.
Revised: Flow-based generative models refer to deep generative models with tractable likelihoods, and offer several attractive properties including efﬁcient density estimation and sampling.

Original: Despite many advantages, current formulations (e.g., normalizing ﬂow) often have an expensive memory/runtime footprint, which hinders their use in a number of applications.
Revised: Despite many advantages, current formulations (e.g., normalizing ﬂow) often have an expensive memory/runtime footprint, which hinders their use in a number of applications.

Original: In this paper, we consider the setting where we have access to an autoencoder, which is suitably effective for the dataset of interest.
Revised: In this paper, we consider the setting where we have access to an autoencoder, which is suitably effective for the dataset of interest.

Original: Under some mild conditions, we show that we can calculate a mapping to a RKHS which subsequently enables deploying mature ideas from the kernel methods literature for ﬂow-based generative models.
Revised: Under some mild conditions, we show that we can calculate a mapping to a RKHS which subsequently enables deploying mature ideas from the kernel methods literature for ﬂow-based generative models.

Original: Speciﬁcally, we can explicitly map the RKHS distribution (i.e., approximate the ﬂow) to match or align with a template/well-characterized distribution, via kernel transfer operators.
Revised: Speciﬁcally, we can explicitly map the RKHS distribution (i.e., approximate the ﬂow) to match or align with a template/well-characterized distribution, via kernel transfer operators.

Original: This leads to a direct and resource efﬁcient approximation avoiding iterative optimization.
Revised: This leads to a direct and resource efﬁcient approximation avoiding iterative optimization.

Original: We empirically show that this simple idea yields competitive results on popular datasets such as CelebA, as well as promising results on a public 3D brain imaging dataset where the sample sizes are much smaller.
Revised: We empirically show that this simple idea yields competitive results on popular datasets such as CelebA, as well as promising results on a public 3D brain imaging dataset where the sample sizes are much smaller.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A ﬂow-based generative model refers to a deep generative model composed using a set of invertible transformations.
Revised: A ﬂow-based generative model refers to a deep generative model composed using a set of invertible transformations.

Original: While GANs and VAEs remain the two dominant generative models in the community, ﬂow based formulations have continually evolved and now offer competitive performance in applications including audio/speech synthesis Kim et al.
Revised: While GANs and VAEs remain the two dominant generative models in the community, ﬂow based formulations have continually evolved and now offer competitive performance in applications including audio/speech synthesis Kim et al.

Original: (2019; 2020), text to speech Miao et al.
Revised: (2019; 2020), text to speech Miao et al.

Original: (2020), photo-realistic image generation Kingma & Dhariwal (2018), and learning cross-domain mappings Mahajan et al.
Revised: (2020), photo-realistic image generation Kingma & Dhariwal (2018), and learning cross-domain mappings Mahajan et al.

Original: An important property of such models is the explicit use of a tractable likelihood function, which enables leveraging maximum likelihood principles during training as well as efﬁcient/exact density estimation and sampling.
Revised: An important property of such models is the explicit use of a tractable likelihood function, which enables leveraging maximum likelihood principles during training as well as efﬁcient/exact density estimation and sampling.

Original: The formulation is invertible by design but this involves higher memory requirements.
Revised: The formulation is invertible by design but this involves higher memory requirements.

Original: For example, permitting the bijective mapping to be expressive enough involves increases in the memory footprint Lee et al.
Revised: For example, permitting the bijective mapping to be expressive enough involves increases in the memory footprint Lee et al.

Original: Kim et al.
Revised: Kim et al.

Original: an issue that is a focus of several recent results Jacobsen et al.
Revised: an issue that is a focus of several recent results Jacobsen et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Moreover, in these models we need to calculate the inverse and backpropagate through all invertible transformations during training.
Revised: Moreover, in these models we need to calculate the inverse and backpropagate through all invertible transformations during training.

Original: Calculating the inverse incurs a multiplicative increase in cost, usually as a function of the feature dimension, relative to the calculation of the likelihood, an issue addressed to some extent in Dinh et al.
Revised: Calculating the inverse incurs a multiplicative increase in cost, usually as a function of the feature dimension, relative to the calculation of the likelihood, an issue addressed to some extent in Dinh et al.

Original: ; Kingma & Dhariwal (2018).
Revised: ; Kingma & Dhariwal (2018).

Original: At a high level, a ﬂow-based generative model bijectively pushes the data density from a source to a target, i.e., from a known simple distribution to an unknown (may be intractable) data distribution.
Revised: At a high level, a ﬂow-based generative model bijectively pushes the data density from a source to a target, i.e., from a known simple distribution to an unknown (may be intractable) data distribution.

Original: During training, we seek to learn this bijective mapping by maximizing the likelihood of the mapped training samples.
Revised: During training, we seek to learn this bijective mapping by maximizing the likelihood of the mapped training samples.

Original: In the generation step, we need the inverse of this mapping (given such an inverse exists) to map from a sample drawn from the known distribution back to the input (data) space.
Revised: In the generation step, we need the inverse of this mapping (given such an inverse exists) to map from a sample drawn from the known distribution back to the input (data) space.

Original: When theknown distribution is appropriately chosen (e.g., Gaussian), the Jacobian of the transformation mapping has additional structure (e.g.,  lower triangular) and immensely facilitates training.
Revised: When the Jacobian of the transformation mapping can be efﬁciently computed or estimated (e.g., having a lower triangular form), directly optimizing the likelihood of the training samples is possible.

Original: Rationale.
Revised: However, in training ﬂow-based generative either we must restrict the expressiveness at each layer or fall back on more numerically heavy solutions, see (Chen et al., 2018). Next, we discuss how several existing results may provide a simpliﬁcation strategy.

Original: The starting pointof our work is the existing literature on Koopman and PerronFrobenius operators Song et al.
Revised: Our starting point is the existing literature on Koopman and Perron-Frobenius operators Song et al.

Original: ; Fukumizu et al.
Revised: ; Fukumizu et al.

Original: ; Klus et al.
Revised: ; Klus et al.

Original: (2020), that offers an arguably easier, optionally linear , procedure  to push-forward densities.
Revised: (2020), that offers an arguably easier, optionally linear , procedure that can be used to analyze non-linear dynamics of measurements that evolve temporally.

Original: For instance, it is known that given inﬁnite-dimensional observables of the input measurements , there exists a linear transfer operator that perfectly pushes one probability density to another.
Revised: For instance, as described in Arbabi (2018); Lusch et al. (2018), if we view the data/measurements as evaluations of functions of the state (of a dynamical system) – where the functions are also called observables – then the entire set of such functions forms a linear vector space. Transfer operators on this space describe a linear evolution of the dynamics, i.e., ﬁnite-dimensional nonlinear dynamics are replaced by inﬁnite-dimensional linear dynamics Brunton et al. (2017), perfectly evolving one set of measurements to another over time if the space can be well characterized.

Original: Of course, this is not practically beneﬁcial becauseexplicitly constructing such inﬁnite-dimensional observables for the data could be intractable.
Revised: Of course, this is not practically beneﬁcial because constructing such inﬁnite-dimensional spaces could be intractable.

Original: Nonetheless, results in  controltheory demonstrate that the idea can still be effective in speciﬁc cases, using approximations with either spectral analysis of large but ﬁnite number ofobservable functions Williams et al.
Revised: Nonetheless, results in optimal control demonstrate that the idea can still be effective in speciﬁc cases, using approximations with either spectral analysis of large but ﬁnite number of functions Williams et al.

Original: (2015) or via a search for potential eigenfunctions  using neural networks Li et al.
Revised: (2015) or via a search for potential eigenfunctions of the operators using neural networks Li et al.

Original: ; Lusch et al.
Revised: ; Lusch et al.

Original: Within the last year, several results describe its potential beneﬁts  in machine learning problems as well Li et al.
Revised: Within the last year, several results describe its potential beneﬁts of such operators in machine learning problems as well Li et al.

Original: Azencot et al.
Revised: Azencot et al.

Original: Can Transfer Operators help? Conceptually, it is not difﬁcult to see how the foregoing idea could potentially help (or simplify) ﬂow-based generative models. In principle, using a transfer operator, one could push-forward the input data distribution to a target distribution of our choice, if both have already been mapped to a sufﬁciently high dimensional space. If – additionally – the operator could also be inverted, this strategy may, at least, be viable.
Revised: If we consider the transformations that ﬂow-based generative models learn as a non-linear dynamics, also used in (Chen et al., 2018), a data-driven approximation strategy one can consider is to map the given data (or distribution) into inﬁnite dimension space of functions through the kernel trick , which may allow the use of well known results based on kernel methods, including old and new results on powerful neural kernels Neal (1994);

Original: Of course, several key components are missing.

Original: We need to (a) assess if setting up a suitable inﬁnite dimensional space is possible, (b) identify if we can estimate the transfer operator and then ﬁnally, (c) check if the procedure works at all. Inﬁnite dimensional mapping. Mapping input data to inﬁnite dimension spaces (which in this case could serve as the space of observables) is well studied and textbook kernel method ideas may sufﬁce. Further, known results concretely links kernels to inﬁnite width deep neural networks Neal (1994) and exciting results on neural kernels have appeared in the last few years Jacot et al.
Revised: Jacot et al.

Original: Arora et al.
Revised: Arora et al.

Original: Utilizing these results, a mean embedding on the corresponding Reproducing Kernel Hilbert Space (RKHS) would uniquely identify the distribution in the input space.
Revised: Utilizing these results, a mean embedding on the corresponding Reproducing Kernel Hilbert Space (RKHS) would correspond to the distribution in the input space (the distribution from which input samples are drawn).

Original: Therefore, the problem of identifying a nonlinear mapping  in the input space  reduces to estimating a linear mapping operator between two empirical mean kernel embeddings where recent results on kernel transfer operators Klus et al.
Revised: Therefore, the problem of identifying a nonlinear mapping (or dynamics) in the input space (going from an intractable distribution to a known distribution or vice-versa) reduces to estimating a linear mapping operator between two empirical mean kernel embeddings where recent results on kernel transfer operators Klus et al.

Original: (2020) could be relevant or applicable.
Revised: (2020) could be relevant or applicable.

Original: However, due to the high variability of the data, estimation of the distribution directly in the input space can be difﬁcult.
Revised: However, due to the high variability of the data, estimation of the distribution directly in the input space, as we will see shortly, can be difﬁcult.

Original: But, if the input space is low-dimensional or otherwise structured, this problem could be mitigated.
Revised: But, if the input space is low-dimensional or otherwise structured, this problem could be mitigated.

Original: Fortunately, for many image datasets, one can identify a low-dimensional latent space such that, in theory, the above pipeline could be instantiated, enabling us to learn an operator.
Revised: Fortunately, for many image datasets, one can identify a low-dimensional latent space such that, in theory, the above pipeline could be instantiated, enabling us to learn a transfer operator.

Revised: Conceptually, it is not difﬁcult to see how the foregoing idea could potentially help (or simplify) ﬂow-based generative models.

Revised: In principle, using a transfer operator, one could push-forward the input data distribution to a target distribution of our choice, if both have already been mapped to a sufﬁciently high dimensional space.

Original: Auto-encoders.
Revised: If – additionally – the operator could also be inverted, this strategy may, at least, be viable. Of course, several key components are missing. We need to (a) assess if setting up a suitable inﬁnite dimensional space is possible, (b) identify if we can estimate the transfer operator and then ﬁnally, (c) check if the procedure works at all. In the following sections of the paper, we will verify these key components and show that using only a linear operator yield surprisingly competitive results on several image generation tasks.

Original:  Images often lie close to an (unknown) lower dimensional manifold M ⊂ R m such that dim ( M ) (cid:28) m , and operating with densities in a lower dimensional setting is often much easier.
Revised: Auto-encoders. Images often lie close to an (unknown) lower dimensional manifold M ⊂ R m such that dim ( M ) (cid:28) m , and operating with densities in a lower dimensional setting is often much easier.

Original: VAEs leverage this explicitly via auto-encoders.
Revised: VAEs leverage this explicitly via auto-encoders.

Original: If the parameterized empirical density is p τ ( x ) , in
Revised: If the parameterized empirical density is p τ ( x ) , in

Original: VAEs, we write it as (cid:82) p τ ( x | z ) p ( z ) dz where z is the low dimensional representation with a suitable prior.
Revised: VAEs, we write it as (cid:82) p τ ( x | z ) p ( z ) dz where z is the low dimensional representation with a suitable prior.

Original: We then use a decoder distribution q τ (cid:48)
Revised: We then use a decoder distribution q τ (cid:48)

Original: ( z | x ) and an encoder distribution p τ ( x | z ) and train the parameters τ and τ (cid:48) .
Revised: ( z | x ) and an encoder distribution p τ ( x | z ) and train the parameters τ and τ (cid:48) .

Original: In practice, p τ ( · ) and q τ (cid:48) ( · ) are assumed to be Gaussian, but it is known that jointly ﬁtting the manifold as well as regressing to a particular prior distribution can be challenging in VAEs Kingma et al.
Revised: In practice, p τ ( · ) and q τ (cid:48) ( · ) are assumed to be Gaussian, but it is known that jointly ﬁtting the manifold as well as regressing to a particular prior distribution can be challenging in VAEs Kingma et al.

Original: ; Dai & Wipf (2019).
Revised: ; Dai & Wipf (2019).

Original: Now, consider the following approach, where we do not impose a distributional assumption on z .
Revised: Now, consider the following approach, where we do not impose a distributional assumption on z .

Original: If a well-regularized auto-encoder is able to capture information about the data generating density Alain & Bengio (2014), we can think of z as the input measurements (likely, meaningful representations from the input data) which is subsequently mapped to an inﬁnite dimensional space (of observables of these measurements).
Revised: If a well-regularized auto-encoder is able to capture information about the data generating density Alain & Bengio (2014), we can think of z as the input measurements (likely, meaningful representations from the input data) which is subsequently mapped to an inﬁnite dimensional space (of observables of these measurements).

Original: Now, if we could push-forward the embedded RKHS distribution to the RKHS mapping of a simpler distribution, similar to ﬂow-based generation, one could easily sample from the simple (e.g., standard normal) distribution and transform it via the learned mapping to samples on the latent space.
Revised: Now, if we could push-forward the embedded RKHS distribution to the RKHS mapping of a simpler distribution, similar to ﬂow-based generation, one could easily sample from the simple (e.g., standard normal) distribution and transform it via the learned mapping to samples on the latent space.

Original: In summary, instead of explicitly searching for the eigenfunctions, we propose to Step 1: embed the density from an auto-encoder into a RKHS, Step 2: learn a kernel transfer operator in RKHS in one step.
Revised: In summary, instead of explicitly searching for the eigenfunctions, we propose to Step 1: embed the density from an auto-encoder into a RKHS, Step 2: learn a kernel transfer operator in RKHS in one step.

Original: In the remainder of this paper, we will provide the details to operationalize this idea and show that this simple approach, in fact, performs surprisingly well with highly favorable computational properties.
Revised: In the remainder of this paper, we will provide the details to operationalize this idea and show that this simple approach, in fact, performs surprisingly well with highly favorable computational properties.

Original: P RELIMINARIES

Original: We start with the deﬁnition of reproducing kernel Hilbert space (RKHS) and the kernel embedding of probability distributions which are the building blocks of this paper.
Revised: We now introduce the deﬁnition of reproducing kernel Hilbert space (RKHS) and the kernel embedding of probability distributions which are the building blocks of this paper.

Original: Deﬁnition 1 (RKHS Aronszajn (1950)) .
Revised: Deﬁnition 1 (RKHS Aronszajn (1950)) .

Original: Given a set X and H to be a set of functions Φ : X → R ,
Revised: Given a set X and H to be a set of functions φ : X → R ,

Original: H is called a reproducing kernel Hilbert space (RKHS) with corresponding product (cid:104) .
Revised: H is called a reproducing kernel Hilbert space (RKHS) with corresponding product (cid:104) .

Original: (cid:105) H if there exists a function k : X × X
Revised: (cid:105) H if there exists a function k : X × X

Original: → R (called a reproducing kernel) such that (i) ∀ X ∈ X , Φ ∈
Revised: → R (called a reproducing kernel) such that (i) ∀ X ∈ X , φ ∈

Original: H , Φ( X ) = (cid:104) Φ , k 
Revised: H , φ ( X ) = (cid:104) φ, k ( X, . )

Original: ( X, . ) (cid:105) H (ii) H = span  { k  X, . ) , X ∈ X} 
Revised: (cid:105) H (ii) H = span ( { k ( X, . ) , X ∈ X} ) , where .

Original:  The kernel mean embedding can be used to embed a probability measure in a RKHS.
Revised: is the completion . The kernel mean embedding can be used to embed a probability measure in a RKHS.

Original: Deﬁnition 2 (Mean Embedding Smola et al.
Revised: Deﬁnition 2 (Kernel Mean Embedding Smola et al.

Original: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup X ∈X k ( X, X ) < ∞ , the  mean embedding of µ in RKHS H , denoted by µ H : X → R , is deﬁned as µ H = (cid:82) k
Revised: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup X ∈X k ( X, X ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ H ∈ H , is deﬁned as µ H = (cid:82) k

Original: ( X, . )
Revised: ( X, . )

Original: p ( X )  .
Revised: p ( X ) , and the mean embedding operator E : L 1 ( X ) → H is deﬁned as µ H = E p .

Original: For a characteristic kernel, the mapping from an input space distribution to its corresponding kernel mean embedding is one-to-one.
Revised: For a characteristic kernel, the mapping from an input space distribution to its corresponding kernel mean embedding is one-to-one.

Original: Thus, two distributions in the input space are identical if and only if their kernel mean embedding matches exactly.
Revised: Thus, two distributions in the input space are identical if and only if their kernel mean embedding matches exactly.

Original: This property enables Maximum Mean Discrepancy (MMD) for distribution matching.
Revised: This property enables Maximum Mean Discrepancy (MMD) for distribution matching.

Original: For a ﬁnite number of samples { X i } N i =1 , drawn from the probability measure µ , the mean embedding µ H can be estimated using ˆ µ H = 1/ N (cid:80) Ni =1 k
Revised: For a ﬁnite number of samples { X i } Ni =1 , drawn from the probability measure µ , an empirical estimation of µ H is ˆ µ H = 1 N (cid:80) Ni =1 k

Original: ( X i , . ) .
Revised: ( X i , . ) .

Original: Our developments are focused on simplifying ﬂow-based generative models,

Original: Deﬁnition 3 (Flow-based generative model
Revised: Deﬁnition 3 (Flow-based generative model

Original: Dinh et al.
Revised: Dinh et al.

Original: A ﬂow-based generative model explicitly learns the data distribution by trying to bijectively map it to a tractable density via invertible transformations.
Revised: A ﬂow-based generative model explicitly learns the data distribution by trying to bijectively map it to a tractable density via invertible transformations.

Original: Formally, given a random variable z following a tractable density, i.e., z ∼ p θ ( z ) , a ﬂow-based model learns an invertible mapping f φ such that the data sample x = f φ ( z ) and the corresponding data distribution, p (cid:101) θ ( x ) =
Revised: Formally, given a random variable z following a tractable density, i.e., z ∼ p θ ( z ) , a ﬂow-based model learns an invertible mapping f φ such that the data sample x = f φ ( z ) and the corresponding data distribution, p (cid:101) θ ( x ) =

Original: f ∗ p θ ( z ) , where, f ∗ = | dzdx | is the push-forward operator.
Revised: f ∗ p θ ( z ) , where, f ∗ = | dzdx | is the push-forward operator.

Original:  A S IMPLIFICATION STRATEGY FOR FLOW - BASED GENERATIVE MODELS
Revised:  A S IMPLIFICATION STRATEGY FOR FLOW - BASED GENERATIVE MODELS

Original: Flow-based generative model directly learn the mapping from a known distribution to the target data distribution in the input space through likelihood maximization.
Revised: Flow-based generative model directly learn the mapping from a known distribution to the target data distribution in the input space through likelihood maximization.

Original: During image generation, the fact that data may often lie on or near a low-dimensional manifold is usually not explicitly leveraged.
Revised: During image generation, the fact that data may often lie on or near a low-dimensional manifold is usually not explicitly leveraged.

Original: If one could estimate this manifold perfectly, we could uniquely identify a distribution in the input space.
Revised: If one could estimate this manifold perfectly, we could uniquely identify a distribution in the input space.

Original: But often, the manifold can only be empirically estimated.
Revised: But often, the manifold can only be empirically estimated.

Original: Our simplifying assumption here is that with some structural or smoothness constraints on the distribution on the latent space, we should still be able to approximately identify the true distribution.
Revised: Our simplifying assumption here is that with some structural or smoothness constraints on the distribution on the latent space, we should still be able to approximately identify the true distribution.

Original: VAEs use this intuition to make the latent distribution consistent with a standard normal distribution, and several contemporary works also propose estimating ﬂow on latent spaces Kingma et al.
Revised: VAEs use this intuition to make the latent distribution consistent with a standard normal distribution, and several contemporary works also propose estimating ﬂow on latent spaces Kingma et al.

Original: ; Mahajan et al.
Revised: ; Mahajan et al.

Original: We will follow this rationale and use an autoencoder – here, beneﬁting from the reduced variance in the latent space, our approach will  learn the mapping from a known  distribution to the empirical latent distribution with the expectation that the optimal posterior for likelihood maximization converges to the empirical distribution as the variance is decreased.
Revised: We will follow this rationale and use an autoencoder – here, beneﬁting from the reduced variance in the latent space, our approach will effectively learn the mapping from a known simple distribution to the empirical latent distribution.

Original: Structure of the latent space.
Revised: Structure of the latent space.

Original: We will assume that a suitable auto-encoder is either provided or can be successfully trained from scratch for the data at hand.
Revised: We will assume that a suitable auto-encoder is either provided or can be successfully trained from scratch for the data at hand.

Original: Given measurements from an input data space, M , we seek to learn a mapping E : M → L ⊂ R d , with d < m , where M ⊂ R m .
Revised: Given measurements from an input data space, M , we seek to learn a mapping E : M → L ⊂ R d , with d < m , where M ⊂ R m and L is the latent space.

Original: Additionally, we also learn the inverse of the mapping E which is denoted by D : L → M , i.e., we require D ◦ E
Revised: Additionally, we also learn the inverse of the mapping E which is denoted by D : L → M , i.e., we require D ◦ E

Original: = id (identity).
Revised: = id (identity).

Original: Further, we require L to be a bounded subset of R d .
Revised: Further, we require L to be a bounded subset of R d .

Original: For simplicity, we model it as a unit hypersphere S d − 1 .
Revised: For simplicity, we model it as a unit hypersphere S d − 1 .

Original: This essentially ensures that L is (a) bounded (b) geodesically complete (c) geodesically convex space. 
Revised: This essentially ensures that L is (a) bounded (b) geodesically complete (c) geodesically convex space. [Figure]

Original: Remark 1.
Revised: Remark 1.

Original: A bounded latent space with speciﬁed structure provides several beneﬁts over one without any such constraints.
Revised: A bounded latent space with speciﬁed structure provides several beneﬁts over one without any such constraints.

Original: First, it guarantees that the Grammian matrix constructed with a (positivedeﬁnite) kernel function is bounded, which is required for a valid empirical estimation of the mean embedding in Def.
Revised: First, it guarantees that the Gram matrix constructed with a (positive-deﬁnite) kernel function is bounded, which is required for a valid empirical estimation of the mean embedding in Def.

Original: 2 Song et al.
Revised: 2 Song et al.

Original: Moreover, the projection onto the hypersphere also helps ensure the positive-deﬁniteness of kernels constructed using the latent representations, that we will use later.
Revised: Moreover, the projection onto the hypersphere also helps ensure the positive-deﬁniteness of kernels constructed using the latent representations, which is discussed in 3.2.

Original: At this point, the requirement of geodesic completeness and convexity may not be apparent but these properties will be useful for sample generation.
Revised: At this point, the requirement of geodesic completeness and convexity may not be apparent but these properties will be useful for sample generation.

Original: b × m b × d b × m sampleencoder decoder latentlatent kernel prior kernel prior sample new prior sample

Original: N × N

Original: N × N = K ( ) N × d

Original: N × d , = K ( ) N × d

Original: N × d ,

Original: Operator learning:

Original: K L K − 1 Z

Original: =

Original: ( + n ǫ I ) −

Original: Generation: Ψ T Ψ ∗ =

Original: K L K − 1 Z K ( , ) N × d b × d Ψ T Ψ ∗ ( ) T b

Original: × N b × d b × m (a) (b) (c) (d)

Original: 3. U SING T RANSFER O PERATORS TO ESTIMATE F LOW
Revised: 3. U SING T RANSFER O PERATORS TO ESTIMATE F LOW

Original: If training the auto-encoder provides us a suitably well-structured latent space, as desired, we can now study how the density transfer operators can help transfer a tractable density to the density on the latent space, i.e., mimicking the mechanics within a ﬂow-based generative model.
Revised: If training the auto-encoder provides us a suitably well-structured latent space, as desired, we can now study how the density transfer operators can help transfer a tractable density to the density on the latent space, i.e., mimicking the mechanics within a ﬂow-based generative model.

Original: Transfer operators.
Revised: Transfer operators.

Original: Let { X t } t ≥ 0 be a stationary and ergodic Markov process deﬁned on X .
Revised: Let { X t } t ≥ 0 be a stationary and ergodic Markov process deﬁned on X .

Original: Then, we can deﬁne the transition density function, p τ (with time lag τ ), by 
Revised: Then, we can deﬁne the transition density function, p τ (with time lag τ ), by [Equation] A

Original: [Equation]
Revised: [Equation]

Original: , where A is a measurable set on X.
Revised: a measurable set [Equation] X

Original: [Equation]
Revised: [Equation]

Original: Let L 1 ( X ) be the space of probability densities.
Revised: Let L 1 ( X ) be the space of probability densities.

Original: We wish to emulate the dynamical system of interest by instead utilizing a well-known transfer operator, namely the (Ruelle) Perron-Frobenius operator
Revised: We wish to emulate the dynamical system of interest by instead utilizing a well-known transfer operator, namely the (Ruelle) Perron-Frobenius operator

Original: Mayer (1980), where the “transfer” terminology originates from statistical mechanics.
Revised: Mayer (1980), where the “transfer” terminology originates from statistical mechanics.

Revised: Notice that the notion of t in our work is not literal - it merely serves as a convenient way to describe the transfer operators in the context of evolution of states, and is also analogous to how ﬂow is considered explicitly as continuous dynamical systems in Neural ODE (Chen et al.

Original: Deﬁnition 4 (Perron-Frobenius operator Mayer (1980)) .
Revised: Deﬁnition 4 (Perron-Frobenius operator Mayer (1980)) .

Original: The Perron-Frobenius (PF) operator P :
Revised: The Perron-Frobenius (PF) operator P :

Original: L 1 ( X ) →
Revised: L 1 ( X ) →

Original: L 1 ( X ) push-forwards or transfers a probability density
Revised: L 1 ( X ) push-forwards or transfers a probability density

Original: p t ∈ L 1 ( X ) given the lag τ as ( P p t ) ( y )
Revised: p t ∈ L 1 ( X ) given the lag τ as ( P p t ) ( y )

Original: =
Revised: =

Original: (cid:82) p τ ( y | x ) p t ( x ) dx .
Revised: (cid:82) p τ ( y | x ) p t ( x ) dx .

Original: Remark 2.
Revised: Remark 2.

Original: If such an operator can be efﬁciently estimated, we can use it to transfer the tractable probability density  to the target distribution  whose density is generally unknown.
Revised: If such an operator can be efﬁciently estimated, we can use it to transfer the tractable probability density from a known distribution p t to the target distribution p t + τ whose density is generally unknown.

Original: However, since we do not make any distributional assumptions on the target distribution, a linear solution to such dynamics in the input space, in general, may not exist.
Revised: Since we do not make any distributional assumptions on the target distribution, a linear solution to such dynamics in the input space, in general, may not exist.

Original: But notice that performing these operations after a mapping to an higher dimensional (linear) space is more sensible. If the dynamical system we wish toemulate involves an action Γ (say, a matrix with positive entries), inother words, the iterates Γ t , the behavior of the Perron-Frobenius operator must emulate Γ .
Revised: Nevertheless, in spaces spanned by a sufﬁciently large set of non-linear functions, or speciﬁcally, an RKHS, one can potentially identify a linear operator that is equivalent to a highly non-linear operator in the input space.

Original: This statement is exact for certain Markov chains.
Revised: In Klus et al.

Original: Further, since we wish to utilize a mapping to a RKHS, we will need to utilize analogous results for kernelized Perron-Frobenius operators Fukumizu et al. Deﬁnition 5 (Kernel Perron-Frobenius operator Fukumizu et al.
Revised: , the authors showed two important results that deﬁne such operators in RKHS in terms of covariance and cross-covariance operators, namely kernel Perron-Frobenius operator (kPF) P k and embedded Perron-Frebenius operator (ePF) P E .

Original: Let X = X t and Y = X t
Revised: Let X ∼ p t and Y ∼ p t

Original: + τ  .
Revised: + τ be observations at time t and t + τ .

Original: Let H(where an element is denoted by Φ H ) and G(where an element is denoted by Φ G ) be the RKHSs on X and Y respectively.
Revised: Let H and G be the RKHSs associated with a certain kernel k , φ H and φ G be their respective feature mapping.

Original: Let the cross-covariance operator C Y X :G → H be deﬁned as C Y X
Revised: The cross-covariance operator C Y X : H → G is deﬁned as C Y X

Original: =
Revised: =

Original: E Y X
Revised: E Y X

Original: [Φ G ( y ) ⊗ Φ H ( x )] . Let us also assume that the covariance operatorC XX : H → H is deﬁned analogously. Assume that g ∈ H , the kernel Perron-Frobenius (kPF) operator, denoted by
Revised: [ φ G ( Y ) ⊗ φ H ( X )] , and the covariance operator

Original: P (cid:15) : G → H, is deﬁned by C XX P k g =
Revised: C XX : H → H is deﬁned analogously.

Original: C Y X g (2) . The above deﬁned kPF operator can be used to transfer g ∈ H ,denoted by (as given in equation 2), P k g = ( C XX + (cid:15)I ) − 1 C Y X g .
Revised: If both densities of interest live in the same RKHS, that is, p t ∈ H and p t + τ ∈ H , P k can be used to push forward the density such that p t + τ = P k p t .

Original: Notice that this equation is well-deﬁned only if G = H . But in our setting,  we are interested in transferring a probability density, p t ∈ L 1 ( X ) . So, kPF cannot be used directly since g lies on a RKHS and may not be a probability density.
Revised: However, in our setting, P k is not directly applicable since the densities we are interested are elements of L 1 ( X ) . Therefore, the densities need to be ﬁrst embedded into the RKHS and can then be transferred using the P E in the embedded form, namely E p t + τ = ( P E ◦ E ) p t .

Original: As a solution, it turns out that we can utilize mean embedding as deﬁned in Defn.
Revised: Deﬁnition 5 (Embedded Perron-Frobenius operator Klus et al.

Original: Thus, given a p t ∈ L 1 ( X ) , we may ﬁrst use mean embedding to get the corresponding µ H ∈ H . 
Revised: Given p t ∈ L 1 ( X ) and p t + τ ∈ L 1 ( X ) . Let µ t = E p t and µ t

Original: Then, the Perron-Frobenius operator for embedded densities, denoted byembedded Perron-Frobenius operator , P (cid:15) , can be expressed as P (cid:15) =
Revised: + τ = E p t + τ be their respective kernel mean embedding. The kernel Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned by µ t + τ = P E µ t =

Original: C Y X ( C XX + (cid:15)I ) − 1 .
Revised: C Y X ( C XX + (cid:15)I ) − 1 µ t (2) , under the condition that (i) C XX is injective (ii) µ t ∈ Range ( C XX ) (iii) E

Original:  Notice thatcompared to kPF operator, here we do not need the assumption of G = H and this is due to thereproducing property of kernel (see Klus et al.
Revised: [ Y | X = · ] ∈ H . Notice that the above deﬁned P E essentially has the same form as the kernel conditional embedding in Song et al.

Original: (2020) section 3.3 for details).
Revised: While the ﬁrst two conditions in the above deﬁnition can be satisﬁed (see

Original: The sample estimate of the embedded Perron-Frobenius operator is given by (cid:98) P (cid:15)
Revised: Theorem 2 of Fukumizu et al.

Original:  = (cid:98) C Y  X(cid:98) C − 1 XX = Φ G ( G XX + N(cid:15)I ) − 1 Φ TH (3) , with the terms of the RKHS deﬁned as before.
Revised: , the last condition requires ∃ f ∈ H s.t. ∀ x , f ( x ) = E [ Y | X = x ] , which highlights the importance of the kernel choice.

Original: Let E be the mean embedding operator, the following proposition suggests that mapping p twith a (nonlinear) transfer operator P in the input space followed by a mean embedding is equivalent tomapping the mean embeded p t , or namely µ t , with P (cid:15) .
Revised: With this operator, we can transfer p t to p t + τ in their embedded forms. The following proposition demonstrates the implication of using the embedded Perron-Frobenius operator.

Original: Proposition 1 (Klus et al.
Revised: Proposition 1 (Klus et al.

Original: With the above notations, E ◦ P = P (cid:15) ◦ E .
Revised: With the above notations, E ◦ P = P E ◦ E .

Original: Remark 3.
Revised: Remark 3.

Original: Prop. 1 will help transfer the embedded density using the closed form expression in equation 3. In particular, the commutativity in the proposition justiﬁes the use of mean-embedding followed by the kPF operator instead of estimating a non-linear PF operator in the input space. 
Revised: The commutativity in the proposition shows that the transferred kernel mean embedding of p t by the linear operator we constructed in RKHS is equivalent to the kernel mean embedding of transferred p t by a highly nonlinear operator in the input space. Since p t + τ is generally intractable, the operator can only be empirically estimated.

Original: With these results, we have all the tools in hand to have almost a single step ﬂow based generative model if we start from a well-structured encoding latent space. 3. K ERNEL TRANSFER OPERATOR FOR SAMPLE GENERATION We now present an algorithm for sample generation assuming that a pre-trained auto-encoder is available.
Revised: Given samples

Original: For the moment, it also sufﬁces to assume that the choice of a suitable kernel has already been made (we will discuss this shortly). We can now walk through the sample generation process.
Revised: X =

Original: Notations. We assume we have N data samples on M with their respective latent space representation { l i } where the samples will be indexed by i . Let k L : L × L → R be the (bounded) kernel (and the corresponding RKHS G ).
Revised: { x i } Ni =1 ∼ p Nt and Y =

Original: Let { z i } ⊂
Revised: { y i } Ni =1 ∼ p

Original: Z be samples from the prior distribution, which we model as a uniform distribution on sphere, u S d − 1 , with k Z : Z × Z → R as the (bounded) kernel (and the corresponding RKHS H ). We will construct
Revised: Nt + τ , let Φ H and Φ G denote the feature maps of X and Y , respectively.

Original: [Equation] We can generate a data sample using the following simple steps: Draw a sample z ∗ ∼ u S d
Revised: The sample estimate of the embedded Perron-Frobenius operator is given by (cid:98) P E = (cid:98) C Y X ( (cid:98) C XX

Original:  − 1. Construct s =Φ (cid:62) G P (cid:15)
Revised: + (cid:15)I ) − 1 =

Original: Φ ∗ H = K L K − 1 Z v t , where Φ ∗ H =k ( z ∗ , . ) .
Revised: Φ G ( G XX + N(cid:15)I ) − 1 Φ TH (3) , where G XX =

Original: Compute the approximate preimage of s given by X = D (cid:0) gI ( { l j } γj =1 , { w j } γj =1 ) (cid:1) , where { l j } γj =1 are the top λ (a hyper-parameter) latent representations and { w j } γj =1 are the corresponding weighting factors from s . gI ( ., .) denotes the geodesic interpolation algorithm described in Fig Generator
Revised: Φ (cid:62) H Φ H is the Gram matrix of Φ H .

Original: (a) Let z ∗ be a sample drawn from u S d − 1 . In (3), we post-multiply with k Z ( z ∗,. )to get (cid:98) P (cid:15)
Revised: To generate a sample of the transferred density which is approximately p t + τ using a sample x ∗ of p t , we can construct φ G ( y ∗ )

Original: k Z ( z ∗ ,. )

Original: =
Revised: =

Original:  Φ G ( G XX + N(cid:15)I ) − 1[ kZ ( z i , z ∗ )] t (5) , where, (a) G XX =
Revised: ˆ P E φ H ( x ∗ i ) = Φ G ( G XX + (cid:15)I ) − 1 k

Original: [ k Z ( z i , z j )] .
Revised: ( X, x ∗ ) .

Original: (b) Φ G = [ k L ( l i ,. )] (cid:62) (b) We take inner product, (cid:104) . (cid:105) G of Φ G with (cid:98) P (cid:15) k Z ( z ∗ ,. ) to get (cid:104)
Revised: The distribution of the resulting samples has the following property,

Original: Φ G , (cid:98) P (cid:15)
Revised: Theorem 1 (Proof in A.1) .

Original: k Z ( z ∗ ,. )
Revised: Let µ t + τ be the kernel mean embedding of the true distribution p t + τ .

Original: (cid:105) = Φ (cid:62) G Φ G ( G XX + N(cid:15)I ) − 1 [ k Z ( z i , z ∗ )] (cid:62) = [ k L ( l n , l o )] · ([ k Z ( z i, z j )] + N(cid:15)I ) − 1 [ k Z ( z k , z ∗ )] (cid:62) = 
Revised: The resulting empirical mean embedding ˆ µ ∗ t + τ = 1 n (cid:80) n i =1 φ G ( y ∗ i ) satisﬁes E [ˆ µ ∗ t + τ ] = µ t

Original: K L ( K Z + N(cid:15)I ) − 1 v (cid:62) , where K L = [ k L ( l n , l o )] ∈ R N × N and K Z = [ k Z ( z i , z j )]
Revised: + τ

Original: ∈ R N × N ; v = [ k Z ( z k , z ∗ )] ∈ R N .
Revised: Theorem 1 simply implies that ˆ P E ˆ µ ∗ t + τ is an unbiased estimator of the kernel mean embedding of the true distribution on the latent space.

Original: Notice that both K Z and ( K Z + N(cid:15)I ) − 1can be pre-computed. (d) Let s = (cid:104) Φ G , (cid:98) P (cid:15) k Z ( z ∗ ,. )
Revised: If the kernel is characteristic and the exact preimage exists, then φ − 1 G ( ˆ P E φ H ( x ∗ )) ∼ p t + τ asymptotically, which concludes our main result.

Original: (cid:105) .
Revised: We now present an algorithm for sample generation assuming that a pre-trained auto-encoder is available. The detailed algorithm is described in Figure 3.

Original: Let { l j } γj =1 be the latent space representations of γ data samples with largest corresponding entries in s (in descending order).
Revised: The idea is simple yet powerful: we ﬁrst generate n samples from a simple distribution restricted to a hypersphere (which we model as a uniform distribution on S n − 1 ).

Original: (e) Return the generated sample X = D (cid:16) gI (cid:16) { l j } γj =1 , { w j } γj =1 (cid:17)(cid:17) , here, w j = s j / (cid:80) i s i and gI ( .,. )
Revised: Then, we will construct the operator on RKHS described in the previous section using the sampled points and the latent representations of training samples. At inference time, we will use the operator to transfer new points sampled from the simple distribution to the target feature map. Since it is not practically possible to compute the preimage directly from the inﬁnite-dimensional feature map, we left multiply with Φ G and use the geodesic interpolation (gI) module to construct an approximate preimage. Finally, we decodes the interpolated latent representation to the image space.

Original:  is the geodesic interpolation algorithm (as presented in Fig. 4). [Figure]
Revised: A visual description is shown in Figure 2. [Figure]

Original: Properties.
Revised: Properties.

Original: If K Z is indeed invertible, the empirical mean embedding of samples generated using the proposed algorithm (assuming the preimage is exact) is equal to the empirical mean embedding of the latent representations, indicating a match in distribution.
Revised: If K Z is indeed invertible, the empirical mean embedding of samples generated using the proposed algorithm (assuming the preimage is exact) is equal to the empirical mean embedding of the latent representations, indicating a match in distribution.

Original: Further, we use weighted Fr´echet mean on sphere as approximate preimage of sample in RKHS.
Revised: Further, we use locallyweighted Fr´echet mean on sphere to construct approximate preimage of sample in RKHS.

Original: Recall that we model L as a unit hypersphere, S d − 1 .
Revised: Since a closed-form solution for the weighted Fr´echet mean does not exist on the sphere, we propose to use a simple and efﬁcient algorithm, namely the geodesic interpolation (gI) Salehian et al.

Original: Therefore, the geodesic interpolation (gI) module uses the geodesic on a hypersphere and iteratively computes the weighted Fr´echet mean Fr´echet (1948) of top γ latent representations (see Fig. 4).
Revised: (2015), that uses the geodesic on a hypersphere to iteratively computes the weighted Fr´echet mean Fr´echet (1948) of top γ latent representations (see Fig. 4).

Original: It has the following properties: (i) the geodesically completeness of the latent space guarantees that the geodesic interpolation is well-deﬁned (ii) the geodesic convexity of the latent space guarantees that the output of “gI” algorithm lies on the latent space (iii) the “gI” algorithm converges asymptotically to the true solution Salehian et al.
Revised: The algorithm has the following properties: (i) the geodesically completeness of the latent space guarantees that the geodesic interpolation is well-deﬁned (ii) the geodesic convexity of the latent space guarantees that the output of “gI” algorithm lies on the latent space (iii) the “gI” algorithm converges asymptotically to the Fr´echet mean Salehian et al.

Original: Choice of Kernels.
Revised: Choice of Kernels.

Original: In order to be able to linearize the dynamics between the prior and the target distribution, one must ﬁrst identify a set of nonlinear basis functions such that the corresponding dynamics lies in its span.
Revised: In order to be able to linearize the dynamics between the prior and the target distribution, one must ﬁrst identify a set of nonlinear basis functions such that the corresponding dynamics lies in its span.

Original: Known results in dynamical systems guarantee the existence of such a linear operator, such as the Koopman operator Koopman (1931), given an inﬁnite set of basis functions.
Revised: Known results in dynamical systems guarantee the existence of such a linear operator, such as the Koopman operator Koopman (1931), given an inﬁnite set of basis functions.

Original: But rather than identifying certain modes that best characterize the dynamics Williams et al.
Revised: But rather than identifying certain modes that best characterize the dynamics Williams et al.

Original: ; Brunton et al.
Revised: ; Brunton et al.

Original: we care most about minimizing the error of the transferred density, and whether the span of functions is rich/expressive enough and can be efﬁciently computed.
Revised: we care most about minimizing the error of the transferred density, and whether the span of functions is rich/expressive enough and can be efﬁciently computed.

Original: The behavior of Radial Basis Function (RBF) and arc-cosine, two standard options in the literature, in our setting can be observed in Fig.
Revised: Therefore, the choice of kernel is important since it directly determines the family of functions spanned by the operator.

Original: 5, where an operator is learned in RKHS to push a uniform distribution on S 2 to the empirical distribution of 10,000 random samples from MNIST wrapped on  S 2 .
Revised: We empirically evaluate the effect of using several different kernels by a simple experiment on MNIST. The MNIST digits are ﬁrst trained for 100 epochs using an autoencoder with latent space restricted to S 2 , then samples are generated using procedure described in Figure 3 using the respective kernel function.

Original: Subplot (b) and (c) show the generated samples.
Revised: Subplot (b) and (c) show the generated samples when using Radial

Revised: Basis Function (RBF) kernel and arc-cosine kernel, respectively.

Original: Observe that the choice of kernel has a clear inﬂuence on the posterior,  and a kernel with superior empirical behavior would be desirable.
Revised: Observe that the choice of kernel has a clear inﬂuence on the posterior, but and a kernel with superior empirical behavior would be desirable.

Original: NTK.
Revised: NTK.

Original: We use Neural Tangent Kernel (NTK)
Revised: We use Neural Tangent Kernel (NTK)

Original: Jacot et al.
Revised: Jacot et al.

Original: as our embedding kernel due to the following properties: (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, and can be non-asymptotically approximated Arora et al.
Revised: as our embedding kernel due to the following properties: (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, and can be non-asymptotically approximated Arora et al.

Original: (b) For well-conditioned inputs (i.e., no duplicate points) on the sphere, the positive-deﬁniteness of NTK is guaranteed Jacot et al. 
Revised: (b) For well-conditioned inputs (i.e., no duplicates) on the sphere, the positive-deﬁniteness of NTK is proved in Jacot et al. Therefore, invertibility of G XX is guaranteed if the sampling distribution is restricted on a Generator (a) Let z ∗ be a sample drawn from u S d − 1 .

Revised: We ﬁrst post-multiply with k ( z ∗ ,. )

Revised: to get

Original: (2018), and therefore it can be inverted without the need to regularize (although in practice, due to numerical issues, a weak regularization is sometimes needed).
Revised: [Equation] z [Equation] the geodesic interpolation algorithm (as presented in Fig. 4).

Revised: hypersphere, which is true by our modeling choice.

Original: (c) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough.
Revised: (c) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough Arora et al.

Original: The embedding of generated samples for MNIST using NTK is also shown in Fig.
Revised: The embedding of generated samples for MNIST using NTK is also shown in Fig.

Original: Observe that the sample distribution is similar to the original distribution.
Revised: Observe that the sample distribution more closely resembles the original distribution.

Original: Nystr¨om’s approximation.
Revised: Nystr¨om’s approximation.

Original: A known bottleneck with kernel based methods is instantiating a kernel matrix whose size depends on the dataset size.
Revised: A known bottleneck with kernel based methods is instantiating a kernel matrix whose size depends on the dataset size.

Original: Here, we use use the Nystr¨om’s method Nystr¨om (1930) to obtain approximate kernel matrices.
Revised: Here, we use use the Nystr¨om’s method Nystr¨om (1930) to obtain approximate kernel matrices.

Original: More speciﬁcally, a kernel matrix K ∈ R N × N can be approximated by (cid:101) K = KS (cid:0) S T KS (cid:1) † S T K , where S ∈ R N × s is the sampling matrix.
Revised: More speciﬁcally, a kernel matrix K ∈ R N × N can be approximated by (cid:101) K = KS (cid:0) S T KS (cid:1) † S T K , where S ∈ R N × s is the sampling matrix.

Original: To minimize information loss, we use the RecursiveRLS-Nystr¨om algorithm proposed in Musco &
Revised: To minimize information loss, we use the RecursiveRLS-Nystr¨om algorithm proposed in Musco &

Original: Musco (2017) to sample S .
Revised: Musco (2017) to sample S based on the approximate ridge leverage scores.

Revised: We refer the readers to the work by Musco & Musco (2017) for detailed algorithm.

Original: Notice that after the approximation, we only need to store KS ∈
Revised: Notice that after the approximation, we only need to store KS ∈

Original: R N × s and S T
Revised: R N × s and S T

Original: KS ∈ R s × s matrices to obtain the Nystr¨om approximation of K , bringing down the memory complexity from O ( N 2 ) to O
Revised: KS ∈ R s × s matrices to obtain the Nystr¨om approximation of K , bringing down the memory complexity from O ( N 2 ) to O

Original: ( Ns + s 2 ) .
Revised: ( Ns + s 2 ) .

Original: When s (cid:28) N , the savings are signiﬁcant.
Revised: When s (cid:28) N , the savings are signiﬁcant.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original:  We qualitatively evaluate the proposed generative model on two types of applications, (a) datasets with a sufﬁcient number of samples to model the data distribution (b) datasets with fewer number of samples so that learning the data distribution is harder .
Revised: [Table] We qualitatively evaluate the proposed generative model on two types of applications, (a) datasets with a sufﬁcient number of samples to model the data distribution (b) datasets with fewer number of samples so that learning the data distribution is harder .

Original: In the ﬁrst case, we use standard vision datasets, includingMNIST, CIFAR [Table]
Revised: In the ﬁrst case, we use standard vision datasets, including

Original:  and CelebA, where the number of data samples is larger than the dimension.
Revised: MNIST, CIFAR and CelebA, where the number of data samples is larger than the dimension.

Original: In order to handle the more challenging second case, we use a dataset of T Magnetic Resonance (MR) images from a public brain imaging project called Alzheimer’s Disease Neuro-Imaging Initiative (ADNI).[Table]
Revised: In order to handle the more challenging second case we use a dataset of T1 Magnetic Resonance (MR) images from a public brain imaging project called Alzheimer’s Disease NeuroImaging Initiative (ADNI).

Original: The purpose of the ﬁrst set of experiments is to show that the proposed method yields competitive measures compared to other non-adversarial generative methods while enjoying the beneﬁt of one step density estimation.
Revised: The purpose of the ﬁrst set of experiments is to show that the proposed method yields competitive measures compared to other non-adversarial generative methods while enjoying the beneﬁt of one step density estimation.

Original: In the second setting, our goal is to demonstrate that unlike traditional non-adversarial models, our proposed method produces reasonable sample generation both quantitative and qualitatively.
Revised: In the second setting, our goal is to demonstrate that unlike traditional non-adversarial models, our proposed method produces reasonable sample generation both quantitative and qualitatively.

Original: Computer vision datasets: We observe that for images with structured feature spaces, e.g., MNIST and CelebA, the proposed method matches or outperforms other non-adversarial generative models.

Original:  We evaluate the quality by calculating the Fr´echet Inception Distance (FID)
Revised: Computer vision datasets: We evaluate the quality by calculating the Fr´echet Inception Distance (FID)

Original: Heusel et al.
Revised: Heusel et al.

Original: All models share the same encoder-decoder  latent points, where Nystr¨om indicates approximation using 4000 landmark points from all latent points.Scores for GAN methods are reported from Dai & Wipf (2019).
Revised: All models share the same encoder-decoder architecture and trained for 100 epochs. Subscript 10k indicates the kPF is estimated using 10000 latent points, whereas Nystr¨om indicates approximation using 4000 landmark points from all latent points.

Original: Comparative results are shown in Table 1.
Revised: Comparative results are shown in Table 1.

Revised: We compare our results to vanilla VAE Kingma & Welling (2013), 2-Stage VAE

Revised: Dai & Wipf (2019), and Glow + AE (denoted by SRAE Glow in the table).

Revised: FIDs of two best GAN methods are also reported from Dai & Wipf (2019) for comparison (note that the reported scores have been optimized by large-scale neural architecture search).

Revised: We observe that for images with structured feature spaces, e.g., MNIST and CelebA, the proposed method matches or outperforms other non-adversarial generative models.

Original: On the contrary, on images with less structured features such as CIFAR10, our model performs worse.
Revised: On the contrary, on images with less structured features such as CIFAR10, our model performs worse.

Original: We suspect that this is due to a non-smooth latent space, and we evaluate this by showing that visually similar images picked by our operators decodes to out-of-distribution (blurred-out) images (see Fig. 7).
Revised: We suspect that this is due to a non-smooth latent space, and we evaluate this by showing that visually similar images picked by our operators decodes to out-of-distribution (blurred-out) images (see Fig. 6).

Original: Here, we use a regularized AE Ghosh et al.
Revised: Here, we use a regularized AE Ghosh et al.

Original: (2020) with a latent space restricted to the hypersphere (denoted by SRAE).
Revised: (2020) with a latent space restricted to the hypersphere (denoted by SRAE).

Original: Further, we present qualitative results on CelebA (in Fig. 6), where we compare our kPF based model with other sampling techniques on the latent space.
Revised: Further, we present qualitative results on CelebA (in Fig. 7), where we compare our kPF based model with other density estimation techniques on the latent space.

Original: Observe that our model generates comparative visual results with SRAE Glow .
Revised: Observe that our model generates comparative visual results with SRAE Glow .

Original: Hence, when a sufﬁcient number of samples are available, our method performs as good as the alternatives, which is beneﬁt, given the efﬁcient.
Revised: Hence, when a sufﬁcient number of samples are available, our method performs as good as the alternatives, which is beneﬁcial given the efﬁcient training stage.

Original: We present training time comparison and show that our proposed method is about 40 × faster as in Fig. 
Revised: We present training time comparison and show that our proposed method is about 40 × faster as in Fig. [Figure]

Original: Brain Imaging dataset: In this section, we present results on generating high-resolution (160 × 196 × 160) 3D brain images from ADNI consists of 183 samples from group AD (diagnosed as
Revised: : In this section, we present results on generating high-resolution (160 × 196 × 160) 3D brain images from ADNI consists of 183 samples from group AD (diagnosed as

Original: Alzheimer’s Disease) and 291 samples from group CN (control normals).
Revised: Alzheimer’s Disease) and 291 samples from group CN (control normals).

Original: In this setting (where n = 474 (cid:28) d = 5017600 ), it is often hard to model the data distribution using either variational methods or ﬂow-based methods due to the high variance of the data or the memory-inefﬁciency of the operations.
Revised: In this setting (where n = 474 (cid:28) d = 5017600 ), it is often hard to model the data distribution using either variational methods or ﬂow-based methods due to the high variance of the data or the memory-inefﬁciency of the operations.

Original: However, beneﬁting from the regularized and linear nature of our kernel operator
Revised: However, beneﬁting from the regularized and linear nature of our kernel operator

Original: Willoughby (1979); Arora et al.
Revised: Willoughby (1979); Arora et al.

Original: (2020), we can still generate high-quality samples in such resolu- tion that are in-distribution.
Revised: (2020), we can still generate high-quality samples in such resolution that are in-distribution.

Original: As before, we present the comparative results with respect to the VAE model.
Revised: As before, we present the comparative results with respect to the VAE model.

Original: The generated samples presented in Fig.
Revised: The generated samples presented in Fig.

Original: 8 clearly demonstrate that proposed method generates sharper images.
Revised: 8 clearly demonstrate that proposed method generates sharper images.

Original: To evaluate whether these results are also scientiﬁcally meaningful (and not merely visually pleasing), we tested consistency between statistical group difference testing on the real images (groups were AD and CN) and the same testing performed on the generated samples.
Revised: To evaluate whether these results are also scientiﬁcally meaningful (and not merely visually pleasing), we tested consistency between statistical group difference testing on the real images (groups were AD and CN) and the same testing performed on the generated samples.

Original: We performed a FWER corrected two-sample t-test Ashburner & Friston (2000) in a manner consistent with standard practice Ashburner & Friston (2000);
Revised: We performed a FWER corrected two-sample t-test Ashburner & Friston (2000) in a manner consistent with standard practice Ashburner & Friston (2000);

Original: Winkler et al.
Revised: Winkler et al.

Original: The results (see Fig 8) show that while there is a deterioration in regions identiﬁed to be affected by disease (and different across groups), most of the statistically-signiﬁcant regions from the test on the original images are preserved in tests performed on the generated images.
Revised: The results (see Fig 8) show that while there is a deterioration in regions identiﬁed to be affected by disease (and different across groups), most of the statistically-signiﬁcant regions from the test on the original images are preserved in tests performed on the generated images.

Original: Take-home message: We conclude that due to the kernel approach and one-step formulation of the proposed model, it offers better memory and sample efﬁciency than other non-adversarial generative methods.
Revised: Take-home message: We conclude that due to the kernel approach and one-step formulation of the proposed model, it offers better memory and sample efﬁciency than other non-adversarial generative methods.

Original: This is useful in many tasks, e.g., medical image analysis, where the number of samples is usually much less than the dimension of data.
Revised: This is useful in many tasks, e.g., medical image analysis, where the number of samples is usually much less than the dimension of data.

Original: Moreover, in case of datasets with larger number of samples, the proposed method still performs well (and sometimes better) compared to alternatives, with a sizably smaller resource footprint.
Revised: Moreover, in case of datasets with larger number of samples, the proposed method still performs well (and sometimes better) compared to alternatives, with a sizably smaller resource footprint.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have shown that when a dataset allows a low-dimensional structured latent space representation, a kernel PerronFrobenius operator can provide an efﬁcient strategy to simplify ﬂow-based generative models to transfer the density from a known simple distribution to the empirical data distribution.
Revised: We have shown that when a dataset allows a lowdimensional structured latent space representation, a kernel Perron-Frobenius operator can provide an efﬁcient strategy to simplify ﬂow-based generative models to transfer the density from a known simple distribution to the empirical data distribution.

Original: While the algorithmic simpliﬁcations proposed here can be variously useful, deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS remain unresolved at this time.
Revised: While the algorithmic simpliﬁcations proposed here can be variously useful, deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS remain unresolved at this time.

Original: While the problem of deriving the density has been addressed in Schuster et al.
Revised: While the problem of deriving the density has been addressed in Schuster et al.

Original: (2020), identifying a pre-image can be hard and often ill-posed.
Revised: (2020), identifying a pre-image can be hard and often ill-posed.

Original: We also note that our geodesic interpolation only converges asymptotically to the weighted Fr´echet mean which we use as the approximate preimage, and additional improvements on this front are possible.  
Revised: We also note that our geodesic interpolation only converges asymptotically to the weighted Fr´echet mean which we use as the approximate preimage, and additional improvements on this front are possible.  C ONCLUSION

Revised: In this paper, we show that with the assistance of recent developments in regularized autoencoder and neural kernels, a linear kernel transfer operator can potentially be an efﬁcient substitute to ﬂow-based generative models.

Revised: Our proposed method shows comparable empirical results to other state-of-theart generative models on several computer vision datasets, while enjoying higher computational efﬁciency.

Revised: The results on brain imaging data also shed light on the potential application on highvolume data generation, which are typically hard to model using existing methods.

Original: A A PPENDIX A. A BLATION STUDIES
Revised: A A PPENDIX A. P ROOF FOR T HEOREM

Revised: The proof follows easily from the fact that ˆ P E is linear and independent to { x i } ni =1 ,

Revised: E [ˆ µ ∗ t + τ ] = E [

Revised: 1 n n (cid:88) i =

Revised: ˆ P E φ ( x i )]

Revised: =

Revised: E [ ˆ P E ] E [

Revised: 1 n n (cid:88) i = φ ( x i )]

Original:  We present ablation studies on MNIST (Fig. 10), showing importance and different components of our proposed method.
Revised: = P E µ t = µ t + τ A. A BLATION STUDIES We present ablation studies on MNIST (Fig. 10), showing importance and different components of our proposed method.

Original: Our proposed framework in general consists of (a) learning latent space (b) estimating transfer operator using kPF.
Revised: Our proposed framework in general consists of (a) learning latent space (b) estimating transfer operator using kPF.

Original: We replace the ﬁrst part in two ways (i) remove AE and directly transfer the distribution from image space to the known distribution (ii) replace AE with known invertible basis representation, e.g., mapping onto Fourier basis .
Revised: We replace the ﬁrst part in two ways (i) remove AE and directly transfer the distribution from image space to the known distribution (ii) replace AE with known invertible basis representation, e.g., mapping onto Fourier basis .

Original: In both these cases, we observe that the generation quality deteriorates.
Revised: In both these cases, we observe that the generation quality deteriorates.

Original: Notice that such degradation can be justiﬁed due to the lack of smoothness on the input space.
Revised: Notice that such degradation can be justiﬁed due to the lack of smoothness on the input space.

Original: Next, we ﬁxed the AE and replace the kPF operator with a normalizing ﬂow model Kingma & Dhariwal (2018) as transfer operator.
Revised: Next, we ﬁxed the AE and replace the kPF operator with a normalizing ﬂow model Kingma & Dhariwal (2018) as transfer operator.

Original: Observe that the generation quality are comparable between the two methods.
Revised: Observe that the generation quality are comparable between the two methods.

Original: [Table]
Revised: [Table]

Original: We further show the advantage of using a regularized
Revised: We further show the advantage of using a regularized

Original: AE over a regular one.
Revised: AE over a regular one.

Original: The regularization scheme we applied is derived from Ghosh et al.
Revised: The regularization scheme we applied is derived from Ghosh et al.

Original: (2020), where we apply spectral normalization on all encoder-decoder transformations.
Revised: (2020), where we apply spectral normalization on all encoder-decoder transformations.

Original: Regularization has been shown to lead to smoother latent spaces Alain & Bengio (2014), and therefore is critical in our interpolation-based sampling approach.
Revised: Regularization has been shown to lead to smoother latent spaces Alain & Bengio (2014), and therefore is critical in our interpolation-based sampling approach.

Original: The effect of such regularization can be seen in Table 2, where it leads to higher FIDs in all three datasets using smoother latent space.
Revised: The effect of such regularization can be seen in Table 2, where it leads to higher FIDs in all three datasets using smoother latent space.

Original: Take-home message: To summarize, the presented ablation study demonstrates the importance of the latent space as well as the richness of our one-step estimated operator.
Revised: Take-home message: To summarize, the presented ablation study demonstrates the importance of the latent space as well as the richness of our one-step estimated operator.

Original: Moreover, the boundedness and smoothness of the latent space is crucial as can be justiﬁed by replacing AE with a Fourier basis or by using a regularized AE.
Revised: Moreover, the boundedness and smoothness of the latent space is crucial as can be justiﬁed by replacing AE with a Fourier basis or by using a regularized AE.

Original: A. ADNI DATASET AND BRAIN IMAGE PROCESSING
Revised: A. ADNI DATASET AND BRAIN IMAGE PROCESSING

Original: Data used in preparation of this article were obtained from the Alzheimers Disease Neuroimaging
Revised: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging

Original: Initiative (ADNI) database (adni.loni.usc.edu).
Revised: Initiative (ADNI) database (adni.loni.usc.edu).

Original: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.
Revised: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.

Original: A complete listing of ADNI investigators can be found at: .
Revised: A complete listing of ADNI investigators can be found at: .

Original: The T1 MR brain data we use consists of images from 184 subjects diagnosed with Alzheimers’s disease and 292 control normal subjects.
Revised: The T1 MR brain data we use consists of images from 184 subjects diagnosed with Alzheimers’s disease and 292 control normal subjects.

Original: Images were ﬁrst coregisted to a MNI template and segmented to keep only the white matter and grey matter.
Revised: Images were ﬁrst coregisted to a MNI template and segmented to keep only the white matter and grey matter.

Original: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1] .
Revised: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1] .

Original: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images. 
Revised: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images. [Figure] [Table]

Revised: I MAGE QUALITY

Revised: [Figure]

Revised: A. C HOICE OF γ

Revised: In the inference stage our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by interpolating among the top γ latent representations of the training samples weighted by their kernel values.

Revised: The choice of γ therefore has implications on the generation quality.

Revised: From

Revised: Figure we can observe that, in general, FID worsens as γ increases.

Revised: This observation aligns with our intuition of preserving only the local similarities in kernel embeddings and similar idea has been used in the literature Kwok & Tsang (2004).

Revised: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1 , generated samples will just be reconstructions of training samples).

Revised: Therefore, in our experiments, we choose γ = 10 to achieve a balance between generation quality and the distance to training samples.

Revised: A. E XPERIMENTAL SPECIFICATIONS

Revised: For all experiments in Table 1 except the GAN experiments, we adapted a custom ResNet implementation where each block has residual connection

Revised: [Equation]

Revised: The detailed architecture is given by the following table

Revised: MNIST

Revised: CIFARCelebA

Revised: Encoder 5x5 conv, stride

Revised: ResBlock 32 ×

Revised: ResBlock 32 ×

Revised: ResBlock 32 ×

Revised: ResBlock 64 ×

Revised: ResBlock 64 ×

Revised: ResBlock 64 ×

Revised: ResBlock 128 ×

Revised: ResBlock 128 ×

Revised: ResBlock 128 ×

Revised: ResBlock 256 ×

Revised: ResBlock 256 ×

Revised: ResBlock 256 ×

Revised: ResBlock 512 ×

Revised: Decoder

Revised: ResBlock 256 ×

Revised: ResBlock 256 ×

Revised: ResBlock 512 ×

Revised: ResBlock 128 ×

Revised: ResBlock 128 ×

Revised: ResBlock 256 ×

Revised: ResBlock 64 ×

Revised: ResBlock 64 ×

Revised: ResBlock 128 ×

Revised: ResBlock 32 ×

Revised: ResBlock 32 ×

Revised: ResBlock 64 ×

Revised: ResBlock 32 ×5x5 conv, stride

Revised: We train for 100 epochs on optimizer with β = (0 .

Revised: and batch size of 512.

Revised: learning rate starts from 1e-3 and is halved every 30 epochs

Revised: VAE and 2-Stage VAE are trained using learnable- γ as in Dai & Wipf (2019).

Revised: For experiments in A.4 and A.5, we adopted the training procedure and architecture in Ghosh et al.