Original: Distribution Matching inDeep Generative Models with Kernel Transfer Operators
Revised: Forward Operator Estimation in Generative Models with Kernel Transfer Operators

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Generative models which use explicit density modeling (e.g., variational autoencoders, ﬂow-based generative models) involve ﬁnding a mapping from a known distribution, e.g. Gaussian, to the unknown input distribution.
Revised: Generative models which use explicit density modeling (e.g., variational autoencoders, ﬂow-based generative models) involve ﬁnding a mapping from a known distribution, e.g. Gaussian, to the unknown input distribution.

Original: This often requires searching over a class of non-linear functions (e.g., representable by a deep neural network).
Revised: This often requires searching over a class of non-linear functions (e.g., representable by a deep neural network).

Original: While eﬀective in practice, the associated runtime/memory costs can increase rapidly, usually as a function of the performance desired in an application.
Revised: While eﬀective in practice, the associated runtime/memory costs can increase rapidly, usually as a function of the performance desired in an application.

Original: We propose a substantially cheaper (and simpler)distribution matching strategy  based on adapting known results on kernel transfer operators.
Revised: We propose a much cheaper (and simpler) strategy to estimate this mapping based on adapting known results in kernel transfer operators.

Original: We show that our formulation enables highly eﬃcient distribution approximation and sampling, and oﬀers surprisingly good empirical performance that compares favorably with powerful baselines, but with signiﬁcant runtime savings.
Revised: We show that our formulation enables highly eﬃcient distribution approximation and sampling, and oﬀers surprisingly good empirical performance that compares favorably with powerful baselines, but with signiﬁcant runtime savings.

Original: We show that the algorithm also performs well in small sample size settings (in brain imaging).
Revised: We show that the algorithm also performs well in small sample size settings (in brain imaging).

Original:  Introduction
Revised:  Introduction

Original: Generative modeling, in its unconditional form, refers to the problem of estimating the data generating distribution: given i i d. samples X with an unknown distribution P X , a generative model seeks to ﬁnd a parametric distribution that closely resembles P X .
Revised: Generative modeling, in its unconditional form, refers to the problem of estimating the data generating distribution: given i i d. samples X with an unknown distribution P X , a generative model seeks to ﬁnd a parametric distribution that closely resembles P X .

Original: In modern deep generative models, we often approaches this problemby via a latent variable – i e., we assume that there is some variable Z ∈ Z associated with the observed data X ∈ X that follows a known distribution P Z (also referred to as the prior in generative models).
Revised: In modern deep generative models, we often approach this problem via a latent variable – i e., we assume that there is some variable Z ∈ Z associated with the observed data X ∈ X that follows a known distribution P Z (also referred to as the prior in generative models).

Original: Thus, we can learn a mapping f : Z → X such that the distribution after transformation, denoted by P f ( Z ) , aligns well with the data generating distribution P X .
Revised: Thus, we can learn a mapping f : Z → X such that the distribution after transformation, denoted by P f ( Z ) , aligns well with the data generating distribution P X .

Original: Therefore, sampling from
Revised: Therefore, sampling from

Original: P X becomes convenient since P Z can be eﬃciently sampled.
Revised: P X becomes convenient since P Z can be eﬃciently sampled.

Original: Frequently, f is parameterized by deep neural networks and optimized with SGD.
Revised: Frequently, f is parameterized by deep neural networks and optimized with stochastic gradient descent (SGD).

Original: Existing generative modeling methods variously optimize the transformation f , most commonly modeling it as a Maximum Likelihood (ML) or distribution matching problem.
Revised: Existing generative modeling methods variously optimize the transformation f , most commonly modeling it as a Maximum Likelihood Estimation (MLE) or distribution matching problem.

Original: For instance, given data X = { x 1 ,x 2 , . . .
Revised: For instance, given data X = { x 1 , . . .

Original: , x n } , a variational autoencoder (VAE) Kingma & 
Revised: , x n } , a variational autoencoder (VAE) (Kingma & Welling, 2013)

Original: Welling (2013) ﬁrst constructs Z through the approximate posterior q Z | X and maximizes the lower bound of likelihood p f ( Z ) ( X ).
Revised: ﬁrst constructs Z through the approximate posterior q Z | X and maximizes a lower bound of likelihood p f ( Z ) ( X ).

Original: Generative adversarial networks (GANs) Goodfellow et al. (2014) relies on a simultaneously learned discriminator such that samples of P f ( Z ) are indistinguishable from X .
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) relies on a simultaneously learned discriminator such that samples of P f ( Z ) are indistinguishable from X .

Original: Results in Arjovsky et al. ; Li et al. (2017) suggest that GANs minimize the distributional discrepancies between P f ( Z ) and P X .
Revised: Results in (Arjovsky et al., 2017; Li et al., 2017) suggest that GANs minimize the distributional discrepancies between P f ( Z ) and P X .

Original: Flow-based generative models optimize p f ( Z ) ( X ) explicitly through the change of variable rule and eﬃciently calculating the Jacobian determinant of the inverse mapping f − 1 .
Revised: Flow-based generative models optimize p f ( Z ) ( X ) explicitly through the change of variable rule and eﬃciently calculating the Jacobian determinant of the inverse mapping f − 1 .

Original: In all examples above, the architecture or objective notwithstanding, the common goal is to ﬁnd a suitable function f that reduces the diﬀerence between P f ( Z ) and P X .
Revised: In all examples above, the architecture or objective notwithstanding, the common goal is to ﬁnd a suitable function f that reduces the diﬀerence between P f ( Z ) and P X .

Original: Thus, a key component in many deep generative models is to learn a forward operator as deﬁned below.
Revised: Thus, a key component in many deep generative models is to learn a forward operator as deﬁned below.

Original: Deﬁnition 1.1 (Forward operator) .
Revised: Deﬁnition 1.1 (Forward operator) .

Original: A forward operator f (cid:63) ∈ C :
Revised: A forward operator f (cid:63) ∈ C :

Original: Z → X is deﬁned to be a mapping associated with some latent variable Z ∼ P Z such that f (cid:63) =
Revised: Z → X is deﬁned to be a mapping associated with some latent variable Z ∼ P Z such that f (cid:63) =

Original: arg min f ∈C d ( P f ( Z ) , P X ) for some function class C and a distance measure d ( · , · ) .
Revised: arg min f ∈C d ( P f ( Z ) , P X ) for some function class C and a distance measure d ( · , · ) .

Original: Motivation: The speciﬁcs of the forward operator may diﬀer from case to case.
Revised: Motivation: The speciﬁcs of the forward operator may diﬀer from case to case.

Original: But its properties and how it is estimated numerically greatly inﬂuences the empirical performance of the model.
Revised: But its properties and how it is estimated numerically greatly inﬂuences the empirical performance of the model.

Original: For instance, mode collapse issues in GANs are well known andinteresting solutions continue to emerge Srivastava et al.
Revised: For instance, mode collapse issues in GANs are well known and solutions continue to emerge (Srivastava et al., 2017).

Original: To learn the forward operator, VAEs use an approximate posterior q Z | X that may sometimes fail to align with the prior Kingma et al. ; Dai & Wipf (2019).
Revised: To learn the forward operator, VAEs use an approximate posterior q Z | X that may sometimes fail to align with the prior (Kingma et al., 2016; Dai & Wipf, 2019).

Original: Flow-based generative models enable direct access to the posterior likelihood, yet in order to tractably evaluate the Jacobian of the transformation during training, one must either restrict the expressiveness at each layer Dinh et al. Kingma & Dhariwal (2018) ormake useof more involved solutions Chen et al.
Revised: Flow-based generative models enable direct access to the posterior likelihood, yet in order to tractably evaluate the Jacobian of the transformation during training, one must either restrict the expressiveness at each layer (Dinh et al., 2017; Kingma & Dhariwal, 2018) or use more involved solutions (Chen et al., 2018).

Original: Of course, solutions to mitigate these weaknesses Ho et al. (2019) is an active area of research.
Revised: Of course, solutions to mitigate these weaknesses (Ho et al., 2019) remains an active area of research.

Original: The starting point of our work is to evaluate the extent to which we can radically simplify the forward operator in deep generative models.
Revised: The starting point of our work is to evaluate the extent to which we can radically simplify the forward operator in deep generative models.

Original: Consider some desirable properties of a hypothetical forward operator (in Def.
Revised: Consider some desirable properties of a hypothetical forward operator (in Def.

Original: (1.1)): (a) Upon convergence, the learned operator f (cid:63) minimizes the distance between P X and P f ( Z ) over all possible operators of a certain class.
Revised: (1.1)): (a) Upon convergence, the learned operator f (cid:63) minimizes the distance between P X and P f ( Z ) over all possible operators of a certain class.

Original: (b) The trainingscheme directly learns the mapping from the prior distribution P Z , rather than an approximation.
Revised: (b) The training directly learns the mapping from the prior distribution P Z , rather than a variational approximation.

Original: (c) The forward operator f (cid:63) can be eﬃciently learned and sample generation is also eﬃcient.
Revised: (c) The forward operator f (cid:63) can be eﬃciently learned and sample generation is also eﬃcient.

Original: It would appear that these criteria violate the “no free lunch rule”, and some compromise must be involved.
Revised: It would appear that these criteria violate the “no free lunch rule”, and some compromise must be involved.

Original: Our goal is to investigate this trade-oﬀ: which design choices can make this approach work?
Revised: Our goal is to investigate this trade-oﬀ: which design choices can make this approach work?

Original: Speciﬁcally, a well studied result in dynamical systems, namely the Perron-Frobenius operator Lemmens & Nussbaum (2012), suggests an alternative linear route to model the forward operator.
Revised: Speciﬁcally, a well studied construct in dynamical systems, namely the Perron-Frobenius operator (Lemmens & Nussbaum, 2012), suggests an alternative linear route to model the forward operator.

Original: Here, we show that  the forward operator in generative models can be eﬃciently approximated as the estimation of a closed-form linear operator in the reproducing kernel Hilbert space (RKHS). 
Revised: Here, we show that if we are willing to give up on a few features in existing models – this may be acceptable depending on the downstream use case – then, the forward operator in generative models can be eﬃciently approximated as the estimation of a closed-form linear operator in the reproducing kernel Hilbert space (RKHS). With simple adjustments of existing results, we identify a novel way to replace the expensive training for generative tasks with a simple principled kernel approach.

Original: Contributions.
Revised: Contributions.

Original: Our results are largely based on results in kernel methods and dynamical systems, but we demonstrate their relevance in generative modeling and complement recent ideas that emphasizeinteresting links between deep generative models and dynamical systems..
Revised: Our results are largely based on results in kernel methods and dynamical systems, but we demonstrate their relevance in generative modeling and complement recent ideas that emphasize links between deep generative models and dynamical systems.

Original: Our contributions are (a) We propose a simple framework for transferring a known prior density linearly in RKHS to an unknown data density, which is equivalent to learning a nonlinear forward operator in the input space; (b) We evaluate this idea in multiple scenarios and show competitive generation performance  with pre-trained autoencoders on popular image datasets including MNIST, CIFAR-10, CelebA and FFHQ; (c) As a special use case, we demonstrates the advantage of using the proposed framework in generative modeling tasks over otherdeep generative methods in limited data settings.
Revised: Our contributions are (a) We propose a non-parametric method for transferring a known prior density linearly in RKHS to an unknown data density – equivalent to learning a nonlinear forward operator in the input space. When compared to its functionally-analogous module used in other deep generative methods, our method avoids multiple expensive training steps yielding signiﬁcant eﬃciency gains; (b) We evaluate this idea in multiple scenarios and show competitive generation performance and eﬃciency beneﬁts with pre-trained autoencoders on popular image datasets including MNIST, CIFAR-10, CelebA and FFHQ; (c) As a special use case, we demonstrate the advantages over other methods in limited data settings.

Original:  Preliminaries
Revised:  Preliminaries

Original:  Weﬁrst brieﬂy introduce reproducing kernel Hilbert space (RKHS) and kernel embedding of probability distributions, conceptsthat we will use frequently.
Revised: [Table] We brieﬂy introduce reproducing kernel Hilbert space (RKHS) and kernel embedding of probability distributions, concepts we will use frequently

Original: Deﬁnition 2.1 (RKHS Aronszajn (1950)) .
Revised: Deﬁnition 2.1 (RKHS (Aronszajn, 1950)) .

Original: For a set X , let H be a set of functions
Revised: For a set X , let H be a set of functions

Original: g : X → R .
Revised: g : X → R .

Original: Then, H is a reproducing kernel Hilbert space (RKHS) with a product (cid:104)· , ·(cid:105) H if there exists a function k : X × X
Revised: Then, H is a reproducing kernel Hilbert space (RKHS) with a product (cid:104)· , ·(cid:105) H if there exists a function k : X × X

Original: → R (called a reproducing kernel) such that (i) ∀ x ∈ X , g ∈ H , g ( x ) = (cid:104) g, k ( x, · ) (cid:105) H ; (ii) H = cl ( span ( { k ( x, · ) , x ∈ X} )) , where cl ( · ) is the set closure.
Revised: → R (called a reproducing kernel) such that (i) ∀ x ∈ X , g ∈ H , g ( x ) = (cid:104) g, k ( x, · ) (cid:105) H ; (ii) H = cl ( span ( { k ( x, · ) , x ∈ X} )) , where cl ( · ) is the set closure.

Original: The function φ ( x ) =
Revised: The function φ ( x ) =

Original: k ( x, · ) : X → H is referred to as the feature mapping of the induced
Revised: k ( x, · ) : X → H is referred to as the feature mapping of the induced

Original: RKHS H .
Revised: RKHS H .

Original: A useful identity derived from feature mappings is the kernel mean
Revised: A useful identity derived from feature mappings is the kernel mean

Original: embedding : it deﬁnes a mapping from a probablity measure in X to an element in the RKHS.
Revised: embedding : it deﬁnes a mapping from a probablity measure in X to an element in the RKHS.

Original: Deﬁnition 2.2 (Kernel Mean Embedding Smola et al.
Revised: Deﬁnition 2.2 (Kernel Mean Embedding (Smola et al., 2007)) .

Original: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup x ∈X k ( x, x ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ p ∈ H , is deﬁned as µ p =
Revised: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup x ∈X k ( x, x ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ p ∈ H , is deﬁned as µ p =

Original: E p
Revised: E p

Original: [ φ ( x )]
Revised: [ φ ( x )]

Original: = (cid:82) k
Revised: = (cid:82) k

Original: ( x, · ) p ( x ) dx , and the mean embedding operator E :
Revised: ( x, · ) p ( x ) dx , and the mean embedding operator E :

Original: L 1 ( X ) → H is deﬁned as µ p = E p .
Revised: L 1 ( X ) → H is deﬁned as µ p = E p .

Original: Remark 1.
Revised: Remark 1.

Original: For characteristic kernels, the operator E is injective.
Revised: For characteristic kernels, the operator E is injective.

Original: Thus, two distributions ( p, q ) in X are identical iﬀ E p =
Revised: Thus, two distributions ( p, q ) in X are identical iﬀ E p =

Original: E q .
Revised: E q .

Original: This property allows using of Maximum Mean Discrepancy (MMD) for distribution matching Gretton et al. ; Li et al. and is common, see Muandet et al.
Revised: This property allows using of Maximum Mean Discrepancy (MMD) for distribution matching (Gretton et al., 2012; Li et al., 2017) and is common, see (Muandet et al., 2017;

Original: Zhou et al.
Revised: Zhou et al., 2018).

Original: For a ﬁnite number of samples { x i } ni =1 drawn from the probabil- ity measure p , an unbiased empirical estimate of µ H is ˆ µ H = 1 n (cid:80) ni
Revised: For a ﬁnite number of samples { x i } ni =1 drawn from the probability measure p , an unbiased empirical estimate of µ H is ˆ µ H = 1 n (cid:80) ni

Original: =1 k ( x i , · ) such thatlim n →∞ 1 n (cid:80) ni =
Revised: =1 k ( x i , · ) such that

Original: [Equation]
Revised: [Equation]

Original: k ( x i , · ) = µ H .

Original: Next, we review the covariance/cross covariance operators, two widely-used identities in kernel methods Fukumizu et al. ; Song et al. and building blocks of our approach.
Revised: Next, we review the covariance/cross covariance operators, two widely-used identities in kernel methods (Fukumizu et al., 2013; Song et al., 2013) and building blocks of our approach.

Original: Deﬁnition 2.3 (Covariance/Cross-covariance Operator) .
Revised: Deﬁnition 2.3 (Covariance/Cross-covariance Operator) .

Original: Let X, Z be random variables deﬁned on X × Z with joint distribution P X,Z and marginal distributions P X , P Z .
Revised: Let X, Z be random variables deﬁned on X × Z with joint distribution P X,Z and marginal distributions P X , P Z .

Original: Let ( l, φ, H ) and ( k, ψ, G ) be two sets of (a) bounded kernel, (b) their corresponding feature map, and (c) their induced RKHS, respectively.
Revised: Let ( l, φ, H ) and ( k, ψ, G ) be two sets of (a) bounded kernel, (b) their corresponding feature map, and (c) their induced RKHS, respectively.

Original: The (uncentered) covariance operator C ZZ : H → H and cross-covariance operator C XZ : H → G are deﬁned as
Revised: The (uncentered) covariance operator C ZZ : H → H and cross-covariance operator C XZ : H → G are deﬁned as

Original: [Equation]
Revised: [Equation]

Revised: where ⊗ is the outer product operator.

Original:  Simplifying  estimation of the forward operator
Revised:  Simplifying the estimation of the forward operator

Original: [Table] Let us consider the forward operatorin generative models as a dynamical system, a viewtaken by a number of results Chen et al. Grathwohl et al. ; Behrmann et al.
Revised: Forward operator as a dynamical system: The dynamical system view of generative models has been described by others (Chen et al., 2018; Grathwohl et al., 2019; Behrmann et al., 2019).

Original: Let us temporarily assume that Z = X – this is not true for VAEs and GANs, where the  latent variable Z lives in a lower dimensional space relative to the input space (i.e. |Z| (cid:28) |X| ). Nevertheless, since Z is often modeled as a subspace of X , we can just embed Z into X to satisfy the equivalence.
Revised: These strategies model the evolution of latent variables in a residual neural network in terms of its dynamics over continuous or discrete time t , and consider the output function f as the evaluation function at a predetermined boundary condition t

Original: Forward operator as a dynamical system: In Def. 1.1, we deﬁne the forward operator as the minimizer of the distance between the data generating distribution p X and the post -transformation prior p f ( Z ) .While the following perspective does not inform our formulation, readers familiar with recent work on hybrid models may easily view the forward operator f (cid:63) as the solution function of a (deterministic) continuous dynamical system, similar to the dynamical-system view of ﬂow-based generative in Neural ODE Chen et al. Other generative models with explicit layered structure can be viewed as an Euler discretization of the underlying dynamics Behrmann et al.
Revised: = t 1 .

Original: As an illustration, the dynamics on an element z ( t 0 ) ∈ X by applying f(cid:63) is governed by,
Revised: Speciﬁcally, given an input (i.e., initial condition) z ( t 0 ), f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: [-We can integrate the RHS and write g (cid:63) ( z ( t 0 )) = (cid:82) t 1 t 
-] ∆ t ( z ( t )). The forward operator f (cid:63) =
Revised: [-We can integrate the RHS and write g (cid:63) ( z ( t 0 )) = (cid:82) t 1 t 
-]where ∆ t is a time-dependent neural network function and z ( t ) is the intermediate solution at t .

Original: ( I + g (cid:63) ) : X → X , therefore, controls the discrete evolution of z ( t 0 ) at time t 1 as x = f (cid:63) ( z ( t 0 )).
Revised: This view of generative models is not limited to speciﬁc methods or model archetypes, but generally useful, for example, viewing the outputs of each hidden layer as evaluations in discrete-time dynamics.

Original: So, the marginal densityover a subset of the state space Λ ⊆
Revised: After applying f on a random variable Z , the marginal density of the output over any subspace Λ ⊆

Original: X can be expressed as
Revised: X can be expressed as

Original: [Equation]
Revised: [Equation]

Original: Towards a one-step estimation of forward operator: Both f (cid:63) and f (cid:63) − 1 in (2) will be highly nonlinear in practice.
Revised: If there exists some neural network instance ∆ (cid:63)t such that the corresponding output function f (cid:63) satisﬁes P X = P f (cid:63) ( Z ) , by Def. 1.1, f (cid:63) is a forward operator.

Original: Therefore, learning f requires searching over a large space of nonlinear functions (e.g., functions represented by a given network architecture).
Revised: Let X be a set of i.i.d. samples drawn from P X . In typical generative learning, either maximizing the likelihood 1 | X | (cid:80) x ∈

Original: Sometimes, evaluations of the RHS integral or its lower bound (i.e., ELBO in VAE and exact likelihood in ﬂow based models) can be expensive.
Revised: X p f ( Z ) ( x ) or minimizing the distributional divergence d ( P f ( Z ) , P X ) requires evaluating and diﬀerentiating through f or f − 1 many times. Towards a one-step estimation of forward operator: Since f and f − 1 in (3) will be highly nonlinear in practice, evaluating and computing the gradients can be expensive.

Original: Nevertheless, the dynamical systems literature suggests asimpler, linear extension of the transformation of p Z , namely the Perron-Frobenius operator or transfer operator, that conveniently transfers p Z to p X .
Revised: Nevertheless, the dynamical systems literature suggests a linear extension of f ∗ , namely the Perron-Frobenius operator or transfer operator, that conveniently transfers p Z to p X .

Original: Deﬁnition 3.1 (Perron-Frobenius operator Mayer (1980)) .
Revised: Deﬁnition 3.1 (Perron-Frobenius operator (Mayer, 1980)) .

Original: Given a dynamical system f : X → X , the Perron-Frobenius (PF) operator P : L 1 ( X ) → L 1 ( X ) is an inﬁnitedimensional linear operator deﬁned as (cid:82) Λ ( P p Z )( x ) dx =
Revised: Given a dynamical system f : X → X , the Perron-Frobenius (PF) operator P : L 1 ( X ) → L 1 ( X ) is an inﬁnitedimensional linear operator deﬁned as (cid:82) Λ ( P p Z )( x ) dx =

Original: (cid:82) z ∈ f − 1 (Λ)
Revised: (cid:82) z ∈ f − 1 (Λ)

Original: p Z
Revised: p Z

Original: ( z ) dz for all Λ ⊆ X .
Revised: ( z ) dz for all Λ ⊆ X .

Revised: involves embedding the prior samples z ∗ ∼ P Z , applying the operator Ψ ∗ =

Revised: ˆ P E φ ( z ∗ ) and ﬁnding preimages ψ † (Ψ ∗ ).

Revised: The pre-trained autoencoder projects the data a smooth latent space and is only required when generating high-dimensional data such as images

Revised: Although in Def. 3.1, the PF operator P is deﬁned for self-maps, it is trivial to extend P to mappings f : Z → X by restricting the RHS integral (cid:82) z ∈ f − 1 (Λ)

Revised: p Z ( z ) dz to Z .

Original: With this deﬁnition of the  PF operator, P , we have
Revised: It can be seen that, for the forward operator f ∗ , the corresponding PF operator P satisﬁes

Original: [Equation]
Revised: [Equation]

Original: If P can be eﬃciently estimated, we can use it to transfer the tractableprobability density p Z to the target density p X  .
Revised: If P can be eﬃciently computed, transferring the tractable density p Z to the target density p X can be accomplished simply by applying P .

Original: However, since P is an inﬁnite-dimensional operator on L 1 ( X ), it is impractical to instantiate it explicitly and exactly.
Revised: However, since P is an inﬁnite-dimensional operator on L 1 ( X ), it is impractical to instantiate it explicitly and exactly.

Original: Nonetheless, common methods for estimating the Perron-Frobenius operator include Ulam’s method Ulam (1960) and the extended dynamical mode decomposition (EDMD) Williams et al. (2015a).
Revised: Nonetheless, there exist several methods for estimating the Perron-Frobenius operator, including Ulam’s method (Ulam, 1960) and the Extended Dynamical Mode Decomposition (EDMD) (Williams et al., 2015a).

Original: Both strategies project P onto a ﬁnite number of hand-crafted basis functions – this may suﬃce in many settings but may fall short in modeling highly complex dynamics.
Revised: Both strategies project P onto a ﬁnite number of hand-crafted basis functions – this may suﬃce in many settings but may fall short in modeling highly complex dynamics.

Original: “Kernel embedded” form of PF operator: A natural extension Klus et al. (2020) is to represent P by an inﬁnite set of functions, e.g., projecting it onto the bases of an RKHS via the kernel trick .
Revised: Kernel-embedded form of PF operator: A natural extension of PF operator is to represent P by an inﬁnite set of functions (Klus et al., 2020), e.g., projecting it onto the bases of an RKHS via the kernel trick .

Original: There, for a characteristic kernel l , the kernel mean embedding uniquely identiﬁes an element µ X =
Revised: There, for a characteristic kernel l , the kernel mean embedding uniquely identiﬁes an element µ X =

Original: E l p X ∈ G for  p X ∈
Revised: E l p X ∈ G for any p X ∈

Original: L 1 ( X ).
Revised: L 1 ( X ).

Original: Thus, to approximate P , we may alternatively solve for the dynamics from p Z to p X in their embedded form.
Revised: Thus, to approximate P , we may alternatively solve for the dynamics from p Z to p X in their embedded form.

Original: Using Tab.notations, we have the following linear operator that deﬁnes the dynamics between two embedded densities.
Revised: Using Tab. 1 notations, we have the following linear operator that deﬁnes the dynamics between two embedded densities.

Original: Deﬁnition 3.2 (Kernel-embedded Perron-Frobenius operator Klus et al.
Revised: Deﬁnition 3.2 (Kernel-embedded Perron-Frobenius operator (Klus et al., 2020)) .

Original: Given p Z ∈ L 1 ( X ) and p X ∈ L 1 ( X ) .
Revised: Given p Z ∈ L 1 ( X ) and p X ∈ L 1 ( X ) .

Original: Denote k as the input kernel and l as the output kernel .
Revised: Denote k as the input kernel and l as the output kernel .

Original: Let µ X =
Revised: Let µ X =

Original: E l p X and µ Z =
Revised: E l p X and µ Z =

Original: E k p
Revised: E k p

Original: Z be their corresponding mean kernel embeddings.
Revised: Z be their corresponding mean kernel embeddings.

Original: The kernel-embedded Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned as
Revised: The kernel-embedded Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.1 (Song et al.
Revised: Proposition 3.1 (Song et al.

Original: With the above deﬁnition, P E satisﬁes
Revised: With the above deﬁnition, P E satisﬁes

Original: [Equation]
Revised: [Equation]

Original: under the conditions: (i) C ZZ is injective (ii) µ t ∈ range ( C ZZ )
Revised: under the conditions: (i) C ZZ is injective (ii) µ t ∈ range ( C ZZ )

Original: (iii) E [ g ( X )
Revised: (iii) E [ g ( X )

Original: | Z = · ] ∈ H for any g ∈ G .
Revised: | Z = · ] ∈ H for any g ∈ G .

Original: The last two assumptions can sometimes be diﬃcult to satisfy for certain RKHS (see Theorem 2 of Fukumizu et al.
Revised: The last two assumptions can sometimes be diﬃcult to satisfy for certain RKHS (see Theorem 2 of Fukumizu et al.

Original: In such cases, a relaxed solution can be constructed by replacing
Revised: In such cases, a relaxed solution can be constructed by replacing

Original: C − 1 ZZ by a regularized inverse ( C ZZ + λI ) − 1 or a Moore-Penrose inverse C † ZZ .
Revised: C − 1 ZZ by a regularized inverse ( C ZZ + λI ) − 1 or a Moore-Penrose pseudoinverse C † ZZ .

Original: The following proposition shows commutativity between the (kernel-embedded) PF operator and the mean embedding operator, showing its equivalence to P when l is characteristic.
Revised: The following proposition shows commutativity between the (kernel-embedded) PF operator and the mean embedding operator, showing its equivalence to P when l is characteristic.

Original: Proposition 3.2 (Klus et al.
Revised: Proposition 3.2 ((Klus et al., 2020)) .

Original: With the above notations, E l ◦ P =
Revised: With the above notations, E l ◦ P =

Original: P
Revised: P

Original: E ◦ E k .
Revised: E ◦ E k .

Original: Transferring embedded densities with the kPF operator: The kPF operator is a powerful tool that allows transferring embedded densities in RKHS.
Revised: Transferring embedded densities with the kPF operator: The kPF operator is a powerful tool that allows transferring embedded densities in RKHS.

Original: The main steps are:
Revised: The main steps are:

Original: (1) Use mean embedded operator E l on p Z .
Revised: (1) Use mean embedding operator E l on p Z .

Original: Let us denote it by µ Z .
Revised: Let us denote it by µ Z .

Original: (2) Transfer µ Z using kPF operator P E to get  mean embedded p X , given by µ X .
Revised: (2) Transfer µ Z using kPF operator P E to get the mean embedded p X , given by µ X .

Original: Of course, in practice with ﬁnite data, { x i } i ∈ [ n ] ∼ P X and { z i } i ∈ [ n ] ∼ P X , P E must be estimated empirically (See Klus et al.
Revised: Of course, in practice with ﬁnite data, { x i } i ∈ [ n ] ∼ P X and { z i } i ∈ [ n ] ∼ P X , P E must be estimated empirically (see Klus et al.

Original: (2020) for an error analysis).
Revised: (2020) for an error analysis).

Original: [Equation]
Revised: [Equation]

Original: where Φ =
Revised: where Φ =

Original: [ k ( z 1 , · ) , · · · , k ( z n , · )] , Ψ =
Revised: [ k ( z 1 , · ) , · · · , k ( z n , · )] , Ψ =

Original: [ l ( x 1 , · ) , · · · , l ( x n , · )] are simply the corresponding feature matrices for samples of P X and P Z , and λ is a small penalty term.
Revised: [ l ( x 1 , · ) , · · · , l ( x n , · )] are simply the corresponding feature matrices for samples of P X and P Z , and λ is a small penalty term.

Original: Learning kPF for unconditional generative modeling: Some generative modeling methods such as VAEs and ﬂow-based formulations explicitly model the latent variable Z as conditionally dependent on the data variable X .
Revised: Learning kPF for unconditional generative modeling: Some generative modeling methods such as VAEs and ﬂow-based formulations explicitly model the latent variable Z as conditionally dependent on the data variable X .

Original: This allows deriving/optimizing the likelihood p f ( Z ) ( X ).
Revised: This allows deriving/optimizing the likelihood p f ( Z ) ( X ).

Original: This is desirable but may not be critical in all applications.
Revised: This is desirable but may not be essential in all applications.

Original: To learn a kPF, however, X and Z can be independent RVs.
Revised: To learn a kPF, however, X and Z can be independent RVs.

Original: While it may not be immediately obvious why we could assume this independence, we can observe the following property for the empirical kPF operator, assuming that the empirical covariance operator ˆ C ZZ is non-singular:
Revised: While it may not be immediately obvious why we could assume this independence, we can observe the following property for the empirical kPF operator, assuming that the empirical covariance operator ˆ C ZZ is non-singular:

Original: [Equation]
Revised: [Equation]

Original: Suppose that { x i } i ∈ [ n ] and { z j } i ∈ [ n ] are independently sampled from the marginals P X and P Z .
Revised: Suppose that { x i } i ∈ [ n ] and { z j } i ∈ [ n ] are independently sampled from the marginals P X and P Z .

Original: It is easy to verify that that (6) holds for any pairing { ( x i , z j ) }
Revised: It is easy to verify that (7) holds for any pairing { ( x i , z j ) }

Original: ( i,j ) ∈ [ n ] × [ n ] .
Revised: ( i,j ) ∈ [ n ] × [ n ] .

Original: Learning the kPF operator is similar in spirit to other implicit density models such as GAN, where the search criteria is based on sample discrepancy rather than explicit likelihoods.
Revised: However, instantiating the RVs in this way rules out the use of kPF for certain downstream tasks such as controlled generation or mode detection, since Z does not contain information regarding X . Nevertheless, if sampling is our only goal, then this instantiation of kPF will suﬃce.

Original: Mapping Z to G : Now, since P E is a deterministic linear operator, we can easily set up a scheme to map samples of Z to elements of G where the expectation of the mapped samples equals µ X
Revised: Mapping Z to G : Now, since P E is a deterministic linear operator, we can easily set up a scheme to map samples of Z to elements of G where the expectation of the mapped samples equals µ X

Original: Deﬁne φ ( z ) =
Revised: Deﬁne φ ( z ) =

Original: k ( z, · ) and ψ ( x ) =
Revised: k ( z, · ) and ψ ( x ) =

Original: l ( x, · ) as feature maps of kernels k and l .
Revised: l ( x, · ) as feature maps of kernels k and l .

Original: We can rewrite µ X as
Revised: We can rewrite µ X as

Original: [Equation]
Revised: [Equation]

Original: Here ψ − 1 is the inverse or the preimage map of ψ .
Revised: Here ψ − 1 is the inverse or the preimage map of ψ .

Original: Such an inverse, in general, may not exist Kwok & Tsang (2004); Honeine & Richard (2011).
Revised: Such an inverse, in general, may not exist (Kwok & Tsang, 2004; Honeine & Richard, 2011).

Original: We will discuss a procedure to approximate ψ − 1 in § 4.1.
Revised: We will discuss a procedure to approximate ψ − 1 in § 4.1.

Original: In what follows, we will temporarily assume that an exact preimage map exists and is tractable to compute.
Revised: In what follows, we will temporarily assume that an exact preimage map exists and is tractable to compute.

Original: Deﬁne Ψ ∗ =
Revised: Deﬁne Ψ ∗ =

Original: ˆ P E φ ( Z ) as the transferred sample in G using the empirical embedded PF operator ˆ P E .
Revised: ˆ P E φ ( Z ) as the transferred sample in G using the empirical embedded PF operator ˆ P E .

Original: Then the following result shows that asymptotically the transferred samples converge in distribution to the target distribution.
Revised: Then the next result shows that asymptotically the transferred samples converge in distribution to the target distribution.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: As n → ∞ , ψ − 1 (Ψ ∗ ) D → P X .
Revised: As n → ∞ , ψ − 1 (Ψ ∗ ) D → P X .

Original: In other words, the preimage of the transferred sample approximately conforms to P X underthe previous assumptions when n is large.
Revised: That is, the preimage of the transferred sample approximately conforms to P X under previous assumptions when n is large.

Original: Proof.
Revised: Proof.

Original:  E [ ψ ( ψ − 1 (Ψ ∗ ))] = [Equation]
Revised: Since ˆ P E asymp. → P , the proof immediately follows from (8).

Original:  Sample generation using the Kernel transfer operator
Revised:  Sample generation using the Kernel transfer operator

Original: At this point, the transferred sample Ψ ∗ , obtained by the kPF operator, remains an element of RKHS G .
Revised: At this point, the transferred sample Ψ ∗ , obtained by the kPF operator, remains an element of RKHS G .

Original: To translate the samples back to the input space, we must ﬁnd the preimage x ∗ such that ψ ( x ∗ ) =
Revised: To translate the samples back to the input space, we must ﬁnd the preimage x ∗ such that ψ ( x ∗ ) =

Original: Ψ ∗ .
Revised: Ψ ∗ .

Original: 4. Solving for an approximate preimage
Revised: 4. Solving for an approximate preimage

Original: Solving the preimage in kernel-based methods is known to be ill-posed Mika et al. (1999) because the mapping ψ ( · ) is not necessarily surjective, i.e., a unique preimage x ∗ = ψ − 1 (Ψ ∗ ) , Ψ ∗ ∈ H may not exist.
Revised: Solving the preimage in kernel-based methods is known to be ill-posed (Mika et al., 1999) because the mapping ψ ( · ) is not necessarily surjective, i.e., a unique preimage x ∗ = ψ − 1 (Ψ ∗ ) , Ψ ∗ ∈ H may not exist.

Original: Often, an approximate preimage ψ † ( X , Ψ ∗ )
Revised: Often, an approximate preimage ψ † ( X , Ψ ∗ )

Original: ≈ ψ − 1 (Ψ ∗ ) is constructed instead based on relational properties among the training data in the RKHS.
Revised: ≈ ψ − 1 (Ψ ∗ ) is constructed instead based on relational properties among the training data in the RKHS.

Original: We consider two options in our framework (1) MDS-basd method Kwok & Tsang (2004);
Revised: We consider two options in our framework (1) MDS-based method (Kwok & Tsang, 2004;

Original: Honeine & Richard (2011), ψ †
Revised: Honeine & Richard, 2011),

Original: [Equation]
Revised: [Equation]

Original: ( X , Ψ

Original: [Equation]
Revised: [Equation]

Original: = 12 ( X(cid:48) X (cid:48)(cid:62) )
Revised: ( X (cid:48)(cid:62) X (cid:48)

Original: − 1 X (cid:48)
Revised: [Equation]

Original: (diag( X (cid:48)(cid:62) X (cid:48) ) − d (cid:62) ) , where ∀ i ∈ [ γ ] , d i = (cid:107) l ( x (cid:48) i , · ) − Ψ ∗ (cid:107) G (8) which optimally preserves the distances in RKHS to the preimages in the input space, and (2) weighted Fr´echet mean Friedman et al.
Revised: − d (cid:62) ) , where ∀ i ∈ [ γ ] , d i = (cid:107) l ( x (cid:48) i , · ) − Ψ ∗ (cid:107) G (9) which optimally preserves the distances in RKHS to the preimages in the input space, and (2) weighted Fr´echet mean (Friedman et al., 2001), which in Euclidean space takes the form ψ

Original: (2001), which in Euclidean space takes the form ψ † wFM ( X ,Ψ
Revised: [Equation] wFM ( X ,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: ψ
Revised: ψ

Original: [Equation]
Revised: [Equation]

Original: wFM ( X (cid:48) ; s )=
Revised: wFM ( X [Equation] s )

Original:  X (cid:48) s / (cid:107) s (cid:107) 1 , where ∀ i ∈ [ γ ] , s i =
Revised: = X (cid:48) s / (cid:107) s (cid:107) 1 , where ∀ i ∈ [ γ ] , s i =

Original: (cid:104) l
Revised: (cid:104) l

Original: ( x (cid:48)
Revised: ( x [Equation]

Original: i , · ) , Ψ ∗ (cid:105) (9)
Revised: (cid:105) (10)

Original: [Table]
Revised: [Table]

Original: Sample { z i } i ∈ [ n ] ∼ P
Revised: Sample { z i } i ∈ [ n ] ∼ P

Original: Z
Revised: Z

Original: independently 5:
Revised: independently 5:

Original: Construct L, K ∈ R n × n s.t.
Revised: Construct L, K ∈ R n × n s.t.

Original: L ij = l ( x i , x j ) , K ij = k
Revised: L ij = l ( x i , x j ) , K ij = k

Original: ( z i , z j ) 6:
Revised: ( z i , z j ) 6:

Original: K inv =
Revised: K inv =

Original: ( K + λnI ) − 1 or K † 7: Inference 8:
Revised: ( K + λnI ) − 1 or K † 7: Inference 8:

Original: Generate new prior sample z ∗ ∼ P Z 9: s = L · K inv [ k ( z 1 , z ∗ ) . . .
Revised: Generate new prior sample z ∗ ∼ P Z 9: s = L · K inv [ k ( z 1 , z ∗ ) . . .

Original: k ( z n , z ∗ )] (cid:62) 10: ind = argsort( s )[ − γ :] 11: x ∗ = ψ † wFM ( X [ ind ] ; s [ ind ]).
Revised: k ( z n , z ∗ )] (cid:62) 10: ind = argsort( s )[ − γ :] 11: x ∗ = ψ † wFM ( X [ ind ] ; s [ ind ]).

Original: 12: Output D ( x ∗ ) if D is provided else x ∗ 
Revised: 12: Output D ( x ∗ ) if D is provided else x ∗ where X (cid:48) a neighborhood of γ training samples based on pairwise distance or similarity in RKHS, following (Kwok & Tsang, 2004).

Original: The weighted Fr´echet mean preimage uses the inner product weights (cid:104)
Revised: The weighted Fr´echet mean preimage uses the inner product weights (cid:104)

Original: Ψ ∗ , ψ ( x i ) (cid:105) as measures of similarities to interpolate training samples.
Revised: Ψ ∗ , ψ ( x i ) (cid:105) as measures of similarities to interpolate training samples.

Original: Similar to Kwok & Tsang (2004), we only consider a neighborhood X (cid:48) of γ training samples based on pairwise distance or similarity in RKHS.

Original: On the toy data (as in Fig. 2), weighted Fr´echet mean produces fewer samples that deviate from the true distribution and is easier to compute.
Revised: On the toy data (as in Fig. 2), weighted Fr´echet mean produces fewer samples that deviate from the true distribution and is easier to compute.

Original: Based on this observation, we use the weighted Fr´echet mean as the preimage module for all experiments that requires samples, while acknowledging that other preimage methods can also be substituted in.
Revised: Based on this observation, we use the weighted Fr´echet mean as the preimage module for all experiments that requires samples, while acknowledging that other preimage methods can also be substituted in.

Original: With all the ingredients in hand, we now present an algorithm for sample generation using the kPF operator in Alg. 1.
Revised: With all the ingredients in hand, we now present an algorithm for sample generation using the kPF operator in Alg. 1.

Original: The idea is simple yet powerful: at training time, we construct the empirical kPF operator ˆ P E using the training data { x i } i ∈ [ s ] and samples of the known prior { z i } ı ∈ [ n ] .
Revised: The idea is simple yet powerful: at training time, we construct the empirical kPF operator ˆ P E using the training data { x i } i ∈ [ s ] and samples of the known prior { z i } ı ∈ [ n ] .

Original: At test time, we will transfer new points sampled from P Z to feature maps in H , and construct their preimages as the generated output samples.
Revised: At test time, we will transfer new points sampled from P Z to feature maps in H , and construct their preimages as the generated output samples.

Original: 4. Image generation
Revised: 4. Image generation

Original: Image generation is a common application for generative models Goodfellow et al.
Revised: Image generation is a common application for generative models (Goodfellow et al., 2014;

Original: Dinh et al.
Revised: Dinh et al., 2017).

Original: While our framework is not speciﬁc to images, constructing sample preimages in a high dimensional space with limited training samples can be challenging, since the space of images is usually not dense in a reasonably sized neighborhood.
Revised: While our proposal is not image speciﬁc, constructing sample preimages in a high dimensional space with limited training samples can be challenging, since the space of images is usually not dense in a reasonably sized neighborhood.

Original: However, empirically images often lie near a low dimensional manifold in the ambient space Seung & Lee (2000), and one may utilize an autoencoder (AE) ( E, D ) to embed the images onto a latent space that represents coordinates on a learned manifold.
Revised: However, empirically images often lie near a low dimensional manifold in the ambient space (Seung & Lee, 2000), and one may utilize an autoencoder (AE) ( E, D ) to embed the images onto a latent space that represents coordinates on a learned manifold.

Original: If the learned manifold lies close to the true manifold, we can learn densities on the manifold directly Dai & Wipf (2019).
Revised: If the learned manifold lies close to the true manifold, we can learn densities on the manifold directly (Dai & Wipf, 2019).

Original: Therefore, for image generation tasks, the training data is ﬁrst projected onto the latent space of a pretrained AE.
Revised: Therefore, for image generation tasks, the training data is ﬁrst projected onto the latent space of a pretrained AE.

Original: Then, the operator will be constructed using the projected latent representations, and samples will be mapped back to image space with the decoder of AE.
Revised: Then, the operator will be constructed using the projected latent representations, and samples will be mapped back to image space with the decoder of AE.

Original: Our setup can be viewed analogously to other generative methods based on so called “ ex-post ” density estimation of latent variables Ghosh et al.
Revised: Our setup can be viewed analogously to other generative methods based on so called “ ex-post ” density estimation of latent variables (Ghosh et al., 2020).

Original: We also restrict the AE latent space to a hypersphere S n − 1 to ensure that (a) k ( · , · ) and l ( · , · ) are bounded and (b) the space is geodesically convex and complete, which is required by the preimage computation.
Revised: We also restrict the AE latent space to a hypersphere S n − 1 to ensure that (a) k ( · , · ) and l ( · , · ) are bounded and (b) the space is geodesically convex and complete, which is required by the preimage computation.

Original: To compute the weighted Fr´echet mean on a hypersphere, we adopt the recursive algorithm in Chakraborty & Vemuri (2015) (see appendix D for details).
Revised: To compute the weighted Fr´echet mean on a hypersphere, we adopt the recursive algorithm in Chakraborty & Vemuri (2015) (see appendix D for details).

Original: [Figure]
Revised: [Figure]

Original:  Experimental Results
Revised:  Experimental Results

Original: Goals.
Revised: Goals.

Original: In our experiments, we seek to answer three questions: (a) With suﬃcient data, can our method generate new data with comparable performance with other state-of-the-art generative models?
Revised: In our experiments, we seek to answer three questions: (a) With suﬃcient data, can our method generate new data with comparable performance with other state-of-the-art generative models?

Original: (b) If only limited data samples were given, can our method still estimate the density with reasonable accuracy?
Revised: (b) If only limited data samples were given, can our method still estimate the density with reasonable accuracy?

Original: (c) What are the runtime beneﬁts, if any?
Revised: (c) What are the runtime beneﬁts, if any?

Original: Datasets/setup.
Revised: Datasets/setup.

Original: To answer the ﬁrst question, we evaluate our method on standard vision datasets, including MNIST, CIFAR10, and CelebA, where the number of data samples is much larger than the latent dimension.
Revised: To answer the ﬁrst question, we evaluate our method on standard vision datasets, including MNIST, CIFAR10, and CelebA, where the number of data samples is much larger than the latent dimension.

Original: We compare our results with other VAE variants (Two-stage VAE Dai & Wipf (2019), WAE Arjovsky et al.
Revised: We compare our results with other VAE variants (Two-stage VAE Dai & Wipf (2019), WAE Arjovsky et al.

Original: CV-VAE Ghosh et al.
Revised: CV-VAE Ghosh et al.

Original: and ﬂow-based generative models (Glow Kingma & Dhariwal (2018), CAGlow Liu et al.
Revised: and ﬂow-based generative models (Glow Kingma & Dhariwal (2018), CAGlow Liu et al.

Original: The second question is due to the broad use of kernel methods in small sample size settings.
Revised: The second question is due to the broad use of kernel methods in small sample size settings.

Original: For this more challenging case, we randomly choose 100 training samples ( < 1% of the full dataset) from CelebA and evaluate the quality of generation compared to other density approximation schemes.
Revised: For this more challenging case, we randomly choose 100 training samples ( < 1% of the full dataset) from CelebA and evaluate the quality of generation compared to other density approximation schemes.

Original: We also use a dataset of T1 Magnetic Resonance (MR) images from the Alzheimer’s Disease Neuromaging Initiative (ADNI) study.
Revised: We also use a dataset of T1 Magnetic Resonance (MR) images from the Alzheimer’s Disease Neuromaging Initiative (ADNI) study.

Original: Distribution matching with many data samples.
Revised: Distribution transfer with many data samples.

Original: We evaluate the quality by calculating the Fr´echet Inception Distance (FID) Heusel et al. (2017) with 10K generated images from each model.
Revised: We evaluate the quality by calculating the Fr´echet Inception Distance (FID) (Heusel et al., 2017) with 10K generated images from each model.

Original: Here, we use a pretained regularized autoencoder Ghosh et al. (2020) with a latent space restricted to the hypersphere (denoted by SRAE) to encourage smooth latent representations.
Revised: Here, we use a pretained regularized autoencoder (Ghosh et al., 2020) with a latent space restricted to the hypersphere (denoted by SRAE) to obtain smooth latent representations.

Original: All implemented models share the same encoder/decoder structure used in Ghosh et al. The subscript indicates diﬀerent types of density estimators learned on the latent variables, including Glow Kingma & Dhariwal (2018), Gaussian mixture model (GMM), and two proposed kPF operators with Gaussian kernel as the input kernel (RBF-kPF) and with NTK as  the input kernel (NTK-kPF). The use of NTK is motivated by promising results at the interface of kernel methods and neural networks Jacot et al.
Revised: We compare our kPF to competitive end-to-end deep generative baselines (i.e. ﬂow and VAE variants) as well as other density estimation models over the same

Original: Arora et al. Implementation details are included in the appendix.
Revised: SRAE latent space.

Original: We also provide a code implementation along with the submission.
Revised: For the latent space models, we experimented with Glow (Kingma &

Original:  Comparative results are shown in Table 2.
Revised: [Figure] Comparative results are shown in Table 2.

Original: We see that for images with structured feature spaces, e.g., MNIST and CelebA, our method matchesor outperforms other non-adversarial generative models, which provides evidence in support of the premise that the forward[Figure] operator can be simpliﬁed.
Revised: We see that for images with structured feature spaces, e.g., MNIST and CelebA, our method matches other non-adversarial generative models, which provides evidence in support of the premise that the forward operator can be simpliﬁed.

Original: Further, we present qualitative results on all datasets (in Fig. 4), where we compare our kPF operator based model with other density estimation techniques on the latent space.
Revised: Further, we present qualitative results on all datasets (in Fig. 3), where we compare our kPF operator based model with other density estimation techniques on the latent space.

Original: Observe that our model generates comparable visual results as SRAE Glow .
Revised: Observe that our model generates comparable visual results as SRAE Glow .

Original: Since kPF learns the distribution on a pre-trained AE latent space for image generation, using a more powerful AE can oﬀer improvements in generation quality.
Revised: Since kPF learns the distribution on a pre-trained AE latent space for image generation, using a more powerful AE can oﬀer improvements in generation quality.

Original: In Fig.
Revised: In Fig.

Original: 3, we present representative images by learning our kPF on NVAE Vahdat & Kautz (2020) latent space, pre-trained on the FFHQ dataset.
Revised: 4, we present representative images by learning our kPF on NVAE Vahdat & Kautz (2020) latent space, pre-trained on the FFHQ dataset.

Original: NVAE builds a hierarchical prior and achieves state-of-the-art generation quality among VAEs.
Revised: NVAE builds a hierarchical prior and achieves state-of-the-art generation quality among VAEs.

Original: We see that kPF can indeed generate high-quality and diverse samples with the help of NVAE encoder/decoder.
Revised: We see that kPF can indeed generate high-quality and diverse samples with the help of NVAE encoder/decoder.

Original: In fact, any
Revised: In fact, any

Original: AE/VAE may be substituted in, assuming that the latent space is smooth .
Revised: AE/VAE may be substituted in, assuming that the latent space is smooth .

Original: Summary: When a suﬃcient number of samples are available, our algorithm performs as well as the alternatives, which is attractive given the eﬃcient training.
Revised: Summary: When a suﬃcient number of samples are available, our algorithm performs as well as the alternatives, which is attractive given the eﬃcient training.

Original: In Fig.
Revised: In Fig.

Original: 5, we present training time comparison until convergence for all latent space models (excluding the time topre-train AE) and show that our method is much faster than other deep generative methods.
Revised: 5, we present comparative result of FIDs with respect to the training time. Since kPF can be computed in closed-form, it achieves signiﬁcant training eﬃciency gain compared to other deep generative methods while delivering competitive generative quality.

Original: Distribution matching with limited data samples.
Revised: Distribution transfer with limited data samples.

Original: Next, we present our evaluations when only a limited number of samples are available.
Revised: Next, we present our evaluations when only a limited number of samples are available.

Original: Here, each of the density estimators was trained on latent representations of the same set of 100 randomly sampled CelebA images, and 10K images were generated to evaluate FID (see Table 3).
Revised: Here, each of the density estimators was trained on latent representations of the same set of 100 randomly sampled CelebA images, and 10K images were generated to evaluate FID (see Table 3).

Original: Our method outperforms
Revised: Our method outperforms

Original: Glow and VAE, while oﬀering competitive performance with GMM.
Revised: Glow and VAE, while oﬀering competitive performance with GMM.

Original: Surprisingly, GMM remains a strong baseline for both tasks, which agrees with results in Ghosh et al. 
Revised: Surprisingly, GMM remains a strong baseline for both tasks, which agrees with results in Ghosh et al. [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: However, note that GMM is restricted by its parametric form and is less ﬂexible than our method (as shown in Fig 2)
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Learning from few samples is common in biomedical applications where acquisition is costly.
Revised: [Figure] [Table] [Figure] [Table]

Original: We present results on generating high-resolution (160 × 196 × 160) brain images: 183 samples from group AD (diagnosed as Alzheimer’s disease) and 291 samples from group CN (control normals).
Revised: [Table] [Figure] space models All models are learned on the latent representations encoded by the same pre-trained

Revised: SRAE

Original: For n = 474 (cid:28) d = 5017600, it is often diﬃcult to model the data distribution due to the high variance of the data or the memory-ineﬃciency of the operations.
Revised: However, note that GMM is restricted by its parametric form and is less ﬂexible than our method (as shown in Fig 2). [Table] Learning from few samples is common in biomedical applications where acquisition is costly. Motivated by interest in making synthetic but statistic preserving data (rather than the real patient records) publicly available to researchers (see NIH N3C

Revised: Data Overview), we present results on generating high-resolution (160 × 196 × 160) brain images: 183 samples from group AD (diagnosed as Alzheimer’s disease) and 291 samples from group CN (control normals).

Original: Using properties of our kernel operator, we can generate high-quality samples that are in-distribution.
Revised: For n = 474 (cid:28) d = 5017600, using our kernel operator, we can generate high-quality samples that are in-distribution.

Original: We present comparative results with VAEs.
Revised: We present comparative results with VAEs.

Original: The generated samples in Fig. 6 clearly show that our method generates sharper images.
Revised: The generated samples in Fig. 6 clearly show that our method generates sharper images.

Original: To check if the results are also scientiﬁcally meaningful, we test consistency between group diﬀerence testing (i.e., cases versus controls diﬀerential tests on each voxel) on the real images (groups were AD and CN) and the same test was performed on the generated samples (AD and CN groups), using a FWER corrected two-sample t -test 
Revised: To check if the results are also scientiﬁcally meaningful, we test consistency between group diﬀerence testing (i.e., cases versus controls diﬀerential tests on each voxel) on the real images (groups were AD and CN) and the same test was performed on the generated samples (AD and CN groups), using a FWER corrected two-sample t -test Ashburner &

Original: Ashburner & Friston (2000) consistent with standard practice.
Revised: Friston (2000).

Original: The results (see Fig 6) show that while there is a deterioration in regions identiﬁed to be aﬀected by disease (diﬀerent across groups), many statistically-signiﬁcant regions from tests on the original images are preserved in voxel-wise tests on the generated images.
Revised: The results (see Fig 6) show that while there is a deterioration in regions identiﬁed to be aﬀected by disease (diﬀerent across groups), many statistically-signiﬁcant regions from tests on the real images are preserved in voxel-wise tests on the generated images.

Original: Summary: We achieve improvements in the small sample size setting compared to other generative methods.
Revised: Summary: We achieve improvements in the small sample size setting compared to other generative methods.

Original: This is useful in many data-poor settings.
Revised: This is useful in many data-poor settings.

Original: For larger datasets, our method still compares competitively with alternatives, but with a smaller resource footprint.
Revised: For larger datasets, our method still compares competitively with alternatives, but with a smaller resource footprint.

Original:  Limitations
Revised:  Limitations

Original: Our proposed simpliﬁcations can be variously useful, but deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS is unresolved at this time.
Revised: Our proposed simpliﬁcations can be variously useful, but deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS is unresolved at this time.

Original: While density estimation from kPF has been partially addressed in Schuster et al.
Revised: While density estimation from kPF has been partially addressed in Schuster et al.

Original: (2020b),
Revised: (2020b),

Original: ﬁnding the pre-image is often ill-posed.
Revised: ﬁnding the pre-image is often ill-posed.

Original: The weighted Fr´echet mean preimage only provides an approximate solution and evaluated empirically.
Revised: The weighted Fr´echet mean preimage only provides an approximate solution and evaluated empirically, and due to the interpolation-based sampling strategy, samples cannot be obtained beyond the convex hull of training examples.

Revised: Making Z and X independent RVs also limits its use for certain downstream task such as representation learning or semantic clustering.

Original: Finally, like other kernel-based methods, the  memory/compute cost can be a bottleneck onvery large datasets where kernel approximation methods (Rahimi & Recht, 2009) may have to be applied.
Revised: Finally, like other kernel-based methods, the sup-quadratic memory/compute cost can be a bottleneck on large datasets and kernel approximation (e.g. (Rahimi & Recht, 2009)) may have to be applied; we discuss this in appendix F.

Original:  Conclusions
Revised:  Conclusions

Original: We show that using recent developments in regularized autoencoders, a linear kernel transfer operator can potentially be an eﬃcient substitute for the forward operator in some generative models.
Revised: We show that using recent developments in regularized autoencoders, a linear kernel transfer operator can potentially be an eﬃcient substitute for the forward operator in some generative models, if some compromise in capabilities/performance is acceptable.

Original: Our proposal, despite its simplicity, shows comparable empirical results to otherstate-of-the-art generative models, while enjoying much better computational eﬃciency.
Revised: Our proposal, despite its simplicity, shows comparable empirical results to other generative models, while oﬀering eﬃciency beneﬁts.

Original: Results on brain images also show promise for applications to high-resolution 3D imaging data generation, which is being pursued actively in the community.
Revised: Results on brain images also show promise for applications to highresolution 3D imaging data generation, which is being pursued actively in the community.

Original: A Choice of Kernel is relevant yet flexible
Revised: A Choice of Kernel is relevant yet flexible

Original: In some cases, one would focus on identifying (ﬁnite) eigenfuntions and modes of the underlying operator Williams et al. (2015b) ; Brunton et al.
Revised: In some cases, one would focus on identifying (ﬁnite) eigenfuntions and modes of the underlying operator (Williams et al., 2015b; Brunton et al., 2016).

Original: But rather than ﬁnding certain modes that best characterize the dynamics, we care most about minimizing the error of the transferred density and therefore whether the span of functions is rich/expressive enough.
Revised: But rather than ﬁnding certain modes that best characterize the dynamics, we care most about minimizing the error of the transferred density and therefore whether the span of functions is rich/expressive enough.

Original: In particular, condition (iii) in Proposition 3.2 requires an input RKHS spanned by suﬃciently rich bases (Fukumizu et al., 2013).
Revised: In particular, condition (iii) in Proposition 3.2 requires an input RKHS spanned by suﬃciently rich bases (Fukumizu et al., 2013).

Original: For this reason, the choice of kernel here cannot be completely ignored since it determines the family of functions contained in the induced RKHS.
Revised: For this reason, the choice of kernel here cannot be completely ignored since it determines the family of functions contained in the induced RKHS.

Original: To explore the appropriate kernel setup for our application, we empirically evaluate the eﬀect of using several diﬀerent kernels via a simple experiment on MNIST.
Revised: To explore the appropriate kernel setup for our application, we empirically evaluate the eﬀect of using several diﬀerent kernels via a simple experiment on MNIST.

Original: We ﬁrst train an autoencoder to embed MNIST digits on to a hypersphere S 2 , then generate samples from kPF by the procedure described by Alg.
Revised: We ﬁrst train an autoencoder to embed MNIST digits on to a hypersphere S 2 , then generate samples from kPF by the procedure described by Alg.

Original: 1 using the respective kernel function as the input kernel k .
Revised: 1 using the respective kernel function as the input kernel k .

Original: Subplot (b) and (c) in Fig.
Revised: Subplot (b) and (c) in Fig.

Original: 7 show the generated samples using Radial Basis
Revised: 7 show the generated samples using Radial Basis

Original: Function (RBF) kernel and arc-cosine kernel, respectively.
Revised: Function (RBF) kernel and arc-cosine kernel, respectively.

Original: Observe that the choice of kernel has an inﬂuence on the sample population, and a kernel function with superior empirical behavior is desirable.
Revised: Observe that the choice of kernel has an inﬂuence on the sample population, and a kernel function with superior empirical behavior is desirable.

Original: Motivated by this observation, we evaluated the Neural Tangent Kernel (NTK)
Revised: Motivated by this observation, we evaluated the Neural Tangent Kernel (NTK)

Original: Jacot et al. (2018), a well-studied neural kernel in recent works.
Revised: (Jacot et al., 2018), a well-studied neural kernel in recent works.

Original: We use it for a few reasons, (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, which spans a rich set of functions that satisﬁes the assumption.
Revised: We use it for a few reasons, (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, which spans a rich set of functions that satisﬁes the assumption.

Original: (b) For well-conditioned inputs (i.e., no duplicates) on hypersphere, the positive-deﬁniteness of NTK is proved in Jacot et al.
Revised: (b) For well-conditioned inputs (i.e., no duplicates) on hypersphere, the positive-deﬁniteness of NTK is proved in (Jacot et al., 2018).

Original: Therefore, invertibility of the Gram matrix K ZZ =
Revised: Therefore, invertibility of the Gram matrix K ZZ =

Original: Φ T Φ is almost guaranteed if the prior distribution p Z is restricted on a hypersphere (c) NTK can be non-asymptotically approximated Arora et al.
Revised: Φ T Φ is almost guaranteed if the prior distribution p Z is restricted on a hypersphere (c) NTK can be non-asymptotically approximated (Arora et al., 2019).

Original: (d) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough Arora et al.
Revised: (d) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough (Arora et al., 2019).

Original: Subplot (d) of Fig.
Revised: Subplot (d) of Fig.

Original: 7 shows that kPF learned with NTK as input kernel is able to generate samples that are more consistent with the data distribution.
Revised: 7 shows that kPF learned with NTK as input kernel is able to generate samples that are more consistent with the data distribution.

Original: However, we should note that
Revised: However, we should note that

Original: NTK is merely a convenient choice of kernel that requires less tuning, and is not otherwise central to our work.
Revised: NTK is merely a convenient choice of kernel that requires less tuning, and is not otherwise central to our work.

Original: In fact, as shown in our experiment in Tab. 2, a well-tuned RBF kernel may also achieve a similar performance.
Revised: In fact, as shown in our experiment in Tab. 2, a well-tuned RBF kernel may also achieve a similar performance.

Original: Indeed, in practice, any suitable choice of kernel may be conveniently adopted into the proposed framework without major modiﬁcations.
Revised: Indeed, in practice, any suitable choice of kernel may be conveniently adopted into the proposed framework without major modiﬁcations.

Original: B Density Estimation with kPF Operator
Revised: B Density Estimation with kPF Operator

Original: The displayed transferred density with the kPF operator on toy data in Fig.
Revised: The displayed transferred density with the kPF operator on toy data in Fig.

Original: 1of the main text is approximated using the empirical kernel conditional density operator (CDO) Schuster et al. (2020a), since there is currently no known methods that can exactly reconstruct density from the transferred mean embeddings.
Revised: 1 is approximated using the empirical kernel conditional density operator (CDO) (Schuster et al., 2020a), since there is currently no known methods that can exactly reconstruct density from the transferred mean embeddings.

Original: The marginalized transferred density p P E has the following form
Revised: The marginalized transferred density p P E has the following form

Original: [Equation]
Revised: [Equation]

Original:  where C ρ = E y ∼ ρ [ l ( y, · ) ⊗ l ( y, · )] is the covariance operator of a independent reference density ρ in G .
Revised: ) where C ρ = E y ∼ ρ [ l ( y, · ) ⊗ l ( y, · )] is the covariance operator of a independent reference density ρ in G .

Original: The above density function is also an element of RKHS G , and therefore we can evaluate the density at x ∗ by using the reproducing property p P E ( x ∗ ) =
Revised: The above density function is also an element of RKHS G , and therefore we can evaluate the density at x ∗ by using the reproducing property p P E ( x ∗ ) =

Original: (cid:104) p P E , l ( x ∗ , · ) (cid:105) .
Revised: (cid:104) p P E , l ( x ∗ , · ) (cid:105) .

Original: The results in Schuster et al.
Revised: The results in Schuster et al.

Original: (2020a) show that the empirical estimate of p P E may be constructed from m samples of the reference density { y i } i ∈ [ m ] and n training samples { x i } i ∈ [ n ] and { z i } i ∈ [ n ] as
Revised: (2020a) show that the empirical estimate of p P E may be constructed from m samples of the reference density { y i } i ∈ [ m ] and n training samples { x i } i ∈ [ n ] and { z i } i ∈ [ n ] as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In Fig. 8, we use a uniform density in the square ( ± 4 , ± 4) as the reference density ρ and constructed ˆ p P E using m = 10000 samples { y i } i ∈
Revised: In Fig. 8, we use a uniform density in the square ( ± 4 , ± 4) as the reference density ρ and constructed ˆ p P E using m = 10000 samples { y i } i ∈

Original: [ m ] from ρ .
Revised: [ m ] from ρ .

Original: Due to the form of the empirical kernel CDO, where the estimated density function ˆ p P E is a linear combination of { l ( y i , · ) } i ∈
Revised: Due to the form of the empirical kernel CDO, where the estimated density function ˆ p P E is a linear combination of { l ( y i , · ) } i ∈

Original: [ m ] (as in Eq. 11), the approximation can be inaccurate if reference samples are relatively sparse and the densities are ‘sharp’.
Revised: [ m ] (as in Eq. 12), the approximation can be inaccurate if reference samples are relatively sparse and the densities are ‘sharp’.

Original: In those cases, to obtain a better density estimate, we may either increase the number of reference samples used to construct the empirical CDO (which can be computationally diﬃcult due to the need to compute ( L Y + α (cid:48) I ) − 2 ), or, with some prior knowledge to the true distribution, choose a reference density which is localized around the ground truth density.
Revised: In those cases, to obtain a better density estimate, we may either increase the number of reference samples used to construct the empirical CDO (which can be computationally diﬃcult due to the need to compute ( L Y + α (cid:48) I ) − 2 ), or, with some prior knowledge to the true distribution, choose a reference density which is localized around the ground truth density.

Original: Therefore, to show a more faithful density estimate of the transferred distribution for visualization purpose, we use a composite of the uniform density and the true density with weight 4 : 1 as the reference density ρ in Fig. 2.
Revised: Therefore, to show a more faithful density estimate of the transferred distribution for visualization purpose, we use a composite of the uniform density and the true density with weight 4 : 1 as the reference density ρ in Fig. 2.

Original: In this case, approximately 20% of the reference samples concentrates around the high-density areas of the true density, which helps to form a better basis for ˆ p P E .
Revised: In this case, approximately 20% of the reference samples concentrates around the high-density areas of the true density, which helps to form a better basis for ˆ p P E .

Original: Note that the choice of ρ does not aﬀect the transferred density embedding ˆ P E ˆ µ Z since it is independent of p X and p Z .
Revised: Note that the choice of ρ does not aﬀect the transferred density embedding ˆ P E ˆ µ Z since it is independent of p X and p Z .

Original: After this modiﬁcation, the reconstructed density more accurately reﬂects the true density compared to GMM and
Revised: After this modiﬁcation, the reconstructed density more accurately reﬂects the true density compared to GMM and

Original: GLOW, indicating the transferred distribution by kPF in RKHS matches better to the ground truth distribution.
Revised: GLOW, indicating the transferred distribution by kPF in RKHS matches better to the ground truth distribution.

Original: This is also reﬂected in the generated samples in Fig.
Revised: This is also reﬂected in the generated samples in Fig.

Original: 9, where samples generated by the kPF operator are clearly more aligned with the ground truth distribution. 
Revised: 9, where samples generated by the kPF operator are clearly more aligned with the ground truth distribution. [Figure]

Original: C Effect of γ on Sample Quality
Revised: C Effect of γ on Sample Quality

Original: In the sampling stage, our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by taking the weighted Fr´echet mean of the top γ neighbors among the training samples.
Revised: In the sampling stage, our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by taking the weighted Fr´echet mean of the top γ neighbors among the training samples.

Original: The choice of γ therefore inﬂuences the quality of generation.
Revised: The choice of γ therefore inﬂuences the quality of generation.

Original: From Figure 10, we can observe that, in general, FID worsens as γ increases.
Revised: From Figure 10, we can observe that, in general, FID worsens as γ increases.

Original: This observation aligns with our intuition of preserving only the local similarities represented by the kernel, and similar ideas have been previously used in the literature Hastie et al. ; Kwok &[Figure]
Revised: This observation aligns with our intuition of preserving only the local similarities represented by the kernel, and similar ideas have been previously used in the literature (Hastie et al., 2001; Kwok &

Original: Tsang (2004).
Revised: Tsang, 2004).

Original: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1, generated samples will just be reconstructions of training samples).
Revised: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1, generated samples will just be reconstructions of training samples).

Original: Therefore, in our experiments, we choose γ = 5 to achieve a balance between generation quality and the distance to training samples.
Revised: Therefore, in our experiments, we choose γ = 5 to achieve a balance between generation quality and the distance to training samples.

Original: D Weighted Fr´echet Mean on  Hypersphere
Revised: D Weighted Fr´echet Mean on the Hypersphere

Original: While the weighted Fr´echet Mean in Euclidean space can be computed in closed-form as a weighted arithmetic mean (as in Eq. 9), on  hypersphere there is no known closed-form solution.
Revised: While the weighted Fr´echet Mean in Euclidean space can be computed in closed-form as a weighted arithmetic mean (as in Eq. 10), on the hypersphere there is no known closed-form solution.

Original: Thus, we adopt the iterative algorithm in Chakraborty & Vemuri (2015) for an approximate solution given data points
Revised: Thus, we adopt the iterative algorithm in (Chakraborty & Vemuri, 2015) for an approximate solution given data points

Original: X = { x 1 . . .
Revised: X = { x 1 . . .

Original: x γ } and weight vector s :
Revised: x γ } and weight vector s :

Original: [Equation]
Revised: [Equation]

Revised: where, v =

Original: where, v = θ sin( θ ) ( X i +1 − M i cos( θ )) , θ = arccos( M ti X i +1 ).
Revised: [Equation] ( X [Equation] − M i cos( θ )) , θ = arccos

Revised: [Equation]

Original:  This algorithm iterates through the data points once, yielding a complexity of only O ( γd ), where d is the dimension of X .
Revised: M ti X i +1 ). This algorithm iterates through the data points once, yielding a complexity of only O ( γd ), where d is the dimension of X .

Original: Under the prescribed iteration, M n converges asymptotically to the true Fr´echet mean for ﬁnite data points.
Revised: Under the prescribed iteration, M n converges asymptotically to the true Fr´echet mean for ﬁnite data points.

Original: We refer the readers to Chakraborty & Vemuri (2015) for further details.
Revised: We refer the readers to (Chakraborty & Vemuri, 2015) for further details.

Original: E Fast approximation of Moore-Penrose inverse
Revised: E Fast approximation of Moore-Penrose inverse

Original: When computing the inverted kernel matrix K inv in Algo. 1, conventional approaches typically performs SVD or Cholesky decomposition.
Revised: When computing the inverted kernel matrix K inv in Algo. 1, conventional approaches typically performs SVD or Cholesky decomposition.

Original: Both procedures are hard to parallelize, and therefore maybe slow when K is large.
Revised: Both procedures are hard to parallelize, and therefore, can be slow when K is large.

Original: Alternatively, we can utilize an iterative procedure proposed in Razavi et al.
Revised: Alternatively, we can utilize an iterative procedure proposed in Razavi et al.

Original: (2014) to approximate the Moore-Penrose inverse.
Revised: (2014) to approximate the Moore-Penrose inverse.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since this iterative procedure mostly involves matrix multiplications, it can be eﬃciently parallelized and implemented on GPU.
Revised: Since this iterative procedure mostly involves matrix multiplications, it can be eﬃciently parallelized and implemented on GPU.

Original: The same procedure has also seen success in approximating large self-attention matrices in language modeling Xiong et al.
Revised: The same procedure has also seen success in approximating large self-attention matrices in language modeling (Xiong et al., 2021).

Original: For the NVAE experiment, we run this iteration for 10 steps and use K inv =
Revised: For the NVAE experiment, we run this iteration for 10 steps and use K inv =

Original: Z 10 .
Revised: Z 10 .

Original: F Experimental Details
Revised: F Nystrom Approximation of kPF

Revised: Due to the need to store and compute a kernel matrix inverse ( K + λnI ) − 1 or K † , the memory and computational complexity of kPF is at least O ( n 2 ) and O ( n 3 ), respectively. The sup-quadratic complexity hinders the use of kPF on extremely large datasets. In our experiments, we already adopted a simple subsampling strategy which randomly select 10k training samples from each dataset ( ≤ 50k samples) to ﬁt our hardware conﬁguration which works well.

Revised: But for larger datasets with potentially more modes, a larger set of subsamples must be considered, and in those cases kPF may not be suitable for commodity/aﬀordable hardware.

Revised: In order to overcome this problem, we can combine kPF with conventional kernel approximation methods such as the Nystr¨om method (Williams & Seeger, 2001).

Revised: Let ( X (cid:63) , Z (cid:63) ) be a size v subset of the training set (which we refer to as the landmark points ) and (Ψ (cid:63) , Φ (cid:63) ) be their corresponding kernel feature maps.

Revised: The weighting coeﬃcients s for each prior sample z ∗ ∼ Z derived in Alg.

Revised: 1 can be approximated by

Revised: [Equation]

Revised: , · ) (18) where L Ψ (cid:63) =

Revised: Ψ (cid:62) Ψ (cid:63) ∈ R n × v , W Ψ (cid:63)

Revised: =

Revised: Ψ (cid:62) (cid:63) Ψ (

Revised: cid:63) ∈ R v × v , K Φ (cid:63) =

Revised: Φ (cid:62) Φ (cid:63) ∈ R n × v , W Φ (cid:63) =

Revised: Φ (cid:62) (cid:63) Φ (cid:63) ∈

Revised: R v × v , and the last identity is due to applying the Woodbury formula on ( K Φ (cid:63) W † Φ (cid:63) K (cid:62) Φ (cid:63) + λnI ) † .

Revised: Assuming v (cid:28) n , the memory complexity is reduced to O ( nv ) and the computation complexity to O ( nv 2 ).

Revised: We empirically evaluated the Nystr¨om-approximated kPF on the CelebA experiment and present the result in Tab. 4.

Revised: It can be observed that when v is suﬃciently large, the performance of Nystr¨om approximated kPFs is as good as the ones using the full kernel matrices.

Revised: [Table]

Revised: of performance

Revised: while delivering signiﬁcant eﬃciency gain.

Revised: G Does kPF Memorize Training Data?

Revised: Since in kPF, samples are generated by linearly interpolating between training samples, it is natural to wonder whether it ‘fools’ the metrics by simply replicating training samples.

Revised: For comparison, we consider an alternative scheme that generates data through direct manipulation of the training data, namely Kernel Density Estimation (KDE).

Revised: We ﬁt KDEs by varying noise levels σ and compare their FIDs and nearest samples in the latent space to kPF in Fig.

Revised: We observe that, although KDE can reach very low FIDs when σ is small, almost all new samples closely resemble some instance in the training set, which is a clear indication of memorization.

Revised: In contrast, kPF can generate diverse samples that do not simply replicate the observed data.

Revised: [Figure]

Revised: H Assessing Sample Diversity

Revised: Although FID is one of the most frequently used measures for assessing sample quality of generative models, certain diversity considerations, such as mode collapse, may not be conveniently deduced from it (Sajjadi et al., 2018).

Revised: To enable explicit examination of generative models with respect to both accuracy (i

Revised: [Figure]

Revised: e., generating samples within the support of the data distribution) and diversity (i

Revised: [Figure]

Revised: e., covering the support of the data distribution as much as possible)

Revised: [Figure]

Revised: Sajjadi et al.

Revised: (2018) proposed an approach to evaluate generative models with generalized deﬁnitions of precision and recall between distributions.

Revised: Quality of generation can then be assessed by evaluating the PRD curve

Revised: [Figure]

Revised: which depicts the trade-oﬀs between accuracy (precision) and diversity (recall)

Revised: We present the PRD curves in Fig.

Revised: [Figure]

Revised: The observations align with our results in Tab. 2 and kPF performs competitively in both accuracy and sample diversity

Revised: [Figure]

Revised: CIFARCelebA

Revised: [Figure]

Revised: I Exploring Kernel Configurations

Revised: To investigate the implication of kernel choices on generation quality

Revised: [Table]

Revised: we tested 25 diﬀerent kernel conﬁgurations for CelebA generation (results are presented in Tab. 5).

Revised: For RBF kernels used in the CelebA experiments of the main text, we use a bandwidth of σ in = (cid:112) 2 |Z| / 8 ≈ 0 .when used as input kernel and σ out ≈ 0 . 34, and we adopt the same notation here.

Revised: [Table]

Revised: It can be seen that kernel conﬁgurations and parameters indeed has a non-trivial impact on the generation quality, with NTK-kPF being the most robust to the choice of parameters.

Revised: This aligns with our previous observations and oﬀers some support for using NTK as an input kernel despite the additional compute cost.

Revised: J Experimental Details

Original: In this section, we provide the detailed speciﬁcations for all of our experiments.
Revised: In this section, we provide the detailed speciﬁcations for all of our experiments.

Original: We have also provided our code in the supplemental material.
Revised: We have also provided our code in the supplemental material.

Original: F. Density Estimation on Toy Densities
Revised: J. Density Estimation on Toy Densities

Original: We generated 10000 samples from each of the toy densities to learn the kPF operator.
Revised: We generated 10000 samples from each of the toy densities to learn the kPF operator.

Original: The input kernel k is a ReLU-activated NTK corresponding to a fully-connected network with depth L = 4 and width w = 10000 at each layer, and the output kernel l is a Gaussian kernel.
Revised: The input kernel k is a ReLU-activated NTK corresponding to a fully-connected network with depth L = 4 and width w = 10000 at each layer, and the output kernel l is a Gaussian kernel.

Original: Unless speciﬁed otherwise, we always uses a Gaussian kernel as the output kernel for the remainder of this appendix.
Revised: Unless speciﬁed otherwise, we always uses a Gaussian kernel as the output kernel for the remainder of this appendix.

Original: The bandwidth of the output kernel was adjusted separately for density estimation and sampling for the purpose of demonstration.
Revised: The bandwidth of the output kernel was adjusted separately for density estimation and sampling for the purpose of demonstration.

Original: For comparison, we also ﬁt/estimate a 10-component GMM and a Glow model with 50 coupling layers, where each of them were trained until convergence.
Revised: For comparison, we also ﬁt/estimate a 10-component GMM and a Glow model with 50 coupling layers, where each of them were trained until convergence.

Original: F. Image Generation with Computer Vision Datasets
Revised: J. Image Generation with Computer Vision Datasets

Original: To generate results in Tab. 2 and Tab. 3, we ﬁrst trained an autoencoder for each dataset following the model setup in Ghosh et al. (2020), which uses a modiﬁed Wasserstein autoencoder Tolstikhin et al. (2018) architecture with batch normalization.
Revised: To generate results in Tab. 2 and Tab. 3, we ﬁrst trained an autoencoder for each dataset following the model setup in (Ghosh et al., 2020), which uses a modiﬁed Wasserstein autoencoder (Tolstikhin et al., 2018) architecture with batch normalization.

Original: Additionally, we applied spectral normalization on both the encoder and the decoder, following Ghosh et al. (2020), to obtain a regularized autoencoder.
Revised: Additionally, we applied spectral normalization on both the encoder and the decoder, following (Ghosh et al., 2020), to obtain a regularized autoencoder.

Original: The latent representations were projected onto a hypersphere before decoding to image space.
Revised: The latent representations were projected onto a hypersphere before decoding to image space.

Original: We trained the models on two NVIDIA GTX 1080TI GPUs.
Revised: We trained the models on two NVIDIA GTX 1080TI GPUs.

Original: A detailed model speciﬁcation is provided below in Table 4.
Revised: A detailed model speciﬁcation is provided below in Table 6.

Original: We used an NTK with L = 8 and w = 10000 as the input kernel k (i.e. the embedding kernel of p Z ) for NTK-kPF, and a Gaussian kernel with bandwidth σ  = (cid:112) 2 |Z| / 8 for RBF-kPF.
Revised: We used an NTK with L = 8 and w = 10000 as the input kernel k (i.e. the embedding kernel of p Z ) for NTK-kPF, and a Gaussian kernel with bandwidth σ in = (cid:112) 2 |Z| / 8 for RBF-kPF.

Original: The bandwidth for the output Gaussian kernel is selected  based on cross-validation of a degree 3 polynomial kernel MMD between the sampled and the ground-truth latent points.
Revised: The bandwidth for the output Gaussian kernel is selected by grid search over { 2 − i ∗ σ data | i ∈ [8] } , where σ data is the empirical data standard deviation, based on crossvalidation of a degree 3 polynomial kernel MMD between the sampled and the ground-truth latent points.

Original: Further, to mitigate the deterioration of performance of kernel methods in a high-dimensional setting due to the curse of dimensionality Evangelista et al. , in practice, we model Z as a space with fewer dimensions than the input space X .
Revised: Further, to mitigate the deterioration of performance of kernel methods in a high-dimensional setting due to the curse of dimensionality (Evangelista et al., 2006), in practice, we model Z as a space with fewer dimensions than the input space X .

Original: As a rule of the thumb, we choose Z such that |Z| =
Revised: As a rule of the thumb, we choose Z such that |Z| =

Original: |X| / 4.
Revised: |X| / 4.

Original: To generate images from kPF learned on NVAE latent space [Table] we used the pre-trained checkpoints provided in Vahdat & Kautz (2020) to obtain the latent embeddings for
Revised: To generate images from kPF learned on NVAE latent space, we used the pre-trained checkpoints provided in (Vahdat & Kautz, 2020) to obtain the latent embeddings for

Original: FFHQ images.
Revised: FFHQ images.

Original: We then construct the kPF from the concatenated latent space of the lowest resolution (8 × 8).
Revised: We then construct the kPF from the concatenated latent space of the lowest resolution (8 × 8).

Original: During sampling, prior samples at those resolutions are replace by the kPF samples, while for other resolutions samples remain generated from inferred Gaussian distributions.
Revised: During sampling, prior samples at those resolutions are replace by the kPF samples, while for other resolutions samples remain generated from inferred Gaussian distributions.

Original: The batchnorm statistics were readjusted for 500 iterations following Vahdat & Kautz (2020).
Revised: The batchnorm statistics were readjusted for 500 iterations following (Vahdat & Kautz, 2020).

Original: We use rbf kernels as input and output kernels, with bandwidths σ k , σ l chosen by the median heuristic ( ∼ 100 for input and ∼ 70 for output in our experiments) [Table] [Table]
Revised: We use rbf kernels as input and output kernels, with bandwidths σ k , σ l chosen by the median heuristic ( ∼ 100 for input and ∼ 70 for output in our experiments).

Original: F. Image Generation for Brain Images
Revised: J. Image Generation for Brain Images

Original: For the high-resolution brain imaging dataset, we used a custom version of ResNet He et al. (2016) with 3D convolutions.
Revised: For the high-resolution brain imaging dataset, we used a custom version of ResNet (He et al., 2016) with 3D convolutions.

Original: The detailed architecture is shown in Fig.
Revised: The detailed architecture is shown in Fig.

Original: Due to the large size of the data, we trained the model on 4 NVIDIA Tesla V100 GPUs.
Revised: Due to the large size of the data, we trained the model on 4 NVIDIA Tesla V100 GPUs.

Original: Mandatory ADNI statement regarding data use.
Revised: Mandatory ADNI statement regarding data use.

Original: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu).
Revised: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu).

Original: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.
Revised: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.

Original: A complete listing of ADNI investigators can be found in the
Revised: A complete listing of ADNI investigators can be found in the

Original: ADNI Acknowledgement List.
Revised: ADNI Acknowledgement List.

Original: The T1 MR brain dataset we utilize consists of images from 184 subjects diagnosed with
Revised: The T1 MR brain dataset we utilize consists of images from 184 subjects diagnosed with

Original: Alzheimers’s disease and 292 healthy controls/ normal subjects.
Revised: Alzheimers’s disease and 292 healthy controls/ normal subjects.

Original: Images were ﬁrst coregistered  to a MNI template and segmented to preserve only the white matter and grey matter.
Revised: Images were ﬁrst coregistered [Table] [Table] to a MNI template and segmented to preserve only the white matter and grey matter.

Original: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1].
Revised: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1].

Original: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images.
Revised: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images.

Original: G More Samples
Revised: K More Samples

Original: In this section we present additional uncurated set of samples on MNIST, CIFAR-10, CelebA based on pre-trained SRAE and FFHQ based on NVAE.
Revised: In this section we present additional uncurated set of samples on MNIST, CIFAR-10, CelebA based on pre-trained SRAE and FFHQ based on NVAE.

Original: From the ﬁgures, it can be seen that kPF produces consistent and diverse samples, often better in quality than the alternatives.
Revised: From the ﬁgures, it can be seen that kPF produces consistent and diverse samples, often better in quality than the alternatives.