Original: Forward Operator Estimation in Generative Models with Kernel Transfer Operators
Revised: Forward Operator Estimation in Generative Models with Kernel Transfer Operators

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Generative models which use explicit density modeling (e.g., variational autoencoders, ﬂow-based generative models) involve ﬁnding a mapping from a known distribution, e.g. Gaussian, to the unknown input distribution.
Revised: Generative models which use explicit density modeling (e.g., variational autoencoders, ﬂow-based generative models) involve ﬁnding a mapping from a known distribution, e.g. Gaussian, to the unknown input distribution.

Original: This often requires searching over a class of non-linear functions (e.g., representable by a deep neural network).
Revised: This often requires searching over a class of non-linear functions (e.g., representable by a deep neural network).

Original: While eﬀective in practice, the associated runtime/memory costs can increase rapidly, usually as a function of the performance desired in an application.
Revised: While eﬀective in practice, the associated runtime/memory costs can increase rapidly, usually as a function of the performance desired in an application.

Original: We propose a much cheaper (and simpler) strategy to estimate this mapping based on adapting known results in kernel transfer operators.
Revised: We propose a much cheaper (and simpler) strategy to estimate this mapping based on adapting known results in kernel transfer operators.

Original: We show that our formulation enables highly eﬃcient distribution approximation and sampling, and oﬀers surprisingly good empirical performance that compares favorably with powerful baselines, but with signiﬁcant runtime savings.
Revised: We show that our formulation enables highly eﬃcient distribution approximation and sampling, and oﬀers surprisingly good empirical performance that compares favorably with powerful baselines, but with signiﬁcant runtime savings.

Original: We show that the algorithm also performs well in small sample size settings (in brain imaging).
Revised: We show that the algorithm also performs well in small sample size settings (in brain imaging).

Original:  Introduction
Revised:  Introduction

Original: Generative modeling, in its unconditional form, refers to the problem of estimating the data generating distribution: given i i d. samples X with an unknown distribution P X , a generative model seeks to ﬁnd a parametric distribution that closely resembles P X .
Revised: Generative modeling, in its unconditional form, refers to the problem of estimating the data generating distribution: given i i d. samples X with an unknown distribution P X , a generative model seeks to ﬁnd a parametric distribution that closely resembles P X .

Original: In modern deep generative models, we often approach this problem via a latent variable – i e., we assume that there is some variable Z ∈ Z associated with the observed data X ∈ X that follows a known distribution P Z (also referred to as the prior in generative models).
Revised: In modern deep generative models, we often approach this problem via a latent variable – i e., we assume that there is some variable Z ∈ Z associated with the observed data X ∈ X that follows a known distribution P Z (also referred to as the prior in generative models).

Original: Thus, we can learn a mapping f : Z → X such that the distribution after transformation, denoted by P f ( Z ) , aligns well with the data generating distribution P X .
Revised: Thus, we can learn a mapping f : Z → X such that the distribution after transformation, denoted by P f ( Z ) , aligns well with the data generating distribution P X .

Original: Therefore, sampling from
Revised: Therefore, sampling from

Original: P X becomes convenient since P Z can be eﬃciently sampled.
Revised: P X becomes convenient since P Z can be eﬃciently sampled.

Original: Frequently, f is parameterized by deep neural networks and optimized with stochastic gradient descent (SGD).
Revised: Frequently, f is parameterized by deep neural networks and optimized with stochastic gradient descent (SGD).

Original: Existing generative modeling methods variously optimize the transformation f , most commonly modeling it as a Maximum Likelihood Estimation (MLE) or distribution matching problem.
Revised: Existing generative modeling methods variously optimize the transformation f , most commonly modeling it as a Maximum Likelihood Estimation (MLE) or distribution matching problem.

Original: For instance, given data X = { x 1 , . . .
Revised: For instance, given data X = { x 1 , . . .

Original: , x n } , a variational autoencoder (VAE) (Kingma & Welling, 2013)
Revised: , x n } , a variational autoencoder (VAE) (Kingma & Welling, 2013)

Original: ﬁrst constructs Z through the approximate posterior q Z | X and maximizes a lower bound of likelihood p f
Revised: ﬁrst constructs Z through the approximate posterior q Z | X and maximizes a lower bound of likelihood p f

Original: ( Z ) ( X ).
Revised: ( Z ) ( X ).

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) relies on a simultaneously learned discriminator such that samples of
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) relies on a simultaneously learned discriminator such that samples of

Original: P f ( Z ) are indistinguishable from X .
Revised: P f ( Z ) are indistinguishable from X .

Original: Results in (Arjovsky et al., 2017; Li et al., 2017) suggest that GANs minimize the distributional discrepancies between P f ( Z ) and P X .
Revised: Results in (Arjovsky et al., 2017; Li et al., 2017) suggest that GANs minimize the distributional discrepancies between P f ( Z ) and P X .

Original: Flow-based generative models optimize p f ( Z ) ( X ) explicitly through the change of variable rule and eﬃciently calculating the Jacobian determinant of the inverse mapping f − 1 .
Revised: Flow-based generative models optimize p f ( Z ) ( X ) explicitly through the change of variable rule and eﬃciently calculating the Jacobian determinant of the inverse mapping f − 1 .

Original: In all examples above, the architecture or objective notwithstanding, the common goal is to ﬁnd a suitable function f that reduces the diﬀerence between P f ( Z ) and P X .
Revised: In all examples above, the architecture or objective notwithstanding, the common goal is to ﬁnd a suitable function f that reduces the diﬀerence between P f ( Z ) and P X .

Original: Thus, a key component in many deep generative models is to learn a forward operator as deﬁned below.
Revised: Thus, a key component in many deep generative models is to learn a forward operator as deﬁned below.

Original: Deﬁnition 1.1 (Forward operator) .
Revised: Deﬁnition 1.1 (Forward operator) .

Original: A forward operator f (cid:63) ∈ C :
Revised: A forward operator f (cid:63) ∈ C :

Original: Z → X is deﬁned to be a mapping associated with some latent variable Z ∼ P Z such that f (cid:63) =
Revised: Z → X is deﬁned to be a mapping associated with some latent variable Z ∼ P Z such that f (cid:63) =

Original: arg min f ∈C d ( P f ( Z ) , P X ) for some function class C and a distance measure d ( · , · ) .
Revised: arg min f ∈C d ( P f ( Z ) , P X ) for some function class C and a distance measure d ( · , · ) .

Original: Motivation: The speciﬁcs of the forward operator may diﬀer from case to case.
Revised: Motivation: The speciﬁcs of the forward operator may diﬀer from case to case.

Original: But its properties and how it is estimated numerically greatly inﬂuences the empirical performance of the model.
Revised: But its properties and how it is estimated numerically greatly inﬂuences the empirical performance of the model.

Original: For instance, mode collapse issues in GANs are well known and solutions continue to emerge (Srivastava et al., 2017).
Revised: For instance, mode collapse issues in GANs are well known and solutions continue to emerge (Srivastava et al., 2017).

Original: To learn the forward operator, VAEs use an approximate posterior q Z | X that may sometimes fail to align with the prior (Kingma et al., 2016; Dai & Wipf, 2019).
Revised: To learn the forward operator, VAEs use an approximate posterior q Z | X that may sometimes fail to align with the prior (Kingma et al., 2016; Dai & Wipf, 2019).

Original: Flow-based generative models enable direct access to the posterior likelihood, yet in order to tractably evaluate the Jacobian of the transformation during training, one must either restrict the expressiveness at each layer (Dinh et al., 2017; Kingma & Dhariwal, 2018) or use more involved solutions (Chen et al., 2018).
Revised: Flow-based generative models enable direct access to the posterior likelihood, yet in order to tractably evaluate the Jacobian of the transformation during training, one must either restrict the expressiveness at each layer (Dinh et al., 2017; Kingma & Dhariwal, 2018) or use more involved solutions (Chen et al., 2018).

Original: Of course, solutions to mitigate these weaknesses (Ho et al., 2019) remains an active area of research.
Revised: Of course, solutions to mitigate these weaknesses (Ho et al., 2019) remains an active area of research.

Original: The starting point of our work is to evaluate the extent to which we can radically simplify the forward operator in deep generative models.
Revised: The starting point of our work is to evaluate the extent to which we can radically simplify the forward operator in deep generative models.

Original: Consider some desirable properties of a hypothetical forward operator (in Def.
Revised: Consider some desirable properties of a hypothetical forward operator (in Def.

Original: (1.1)): (a) Upon convergence, the learned operator f (cid:63) minimizes the distance between P X and P f ( Z ) over all possible operators of a certain class.
Revised: (1.1)): (a) Upon convergence, the learned operator f (cid:63) minimizes the distance between P X and P f ( Z ) over all possible operators of a certain class.

Original: (b) The training directly learns the mapping from the prior distribution P Z , rather than a variational approximation.
Revised: (b) The training directly learns the mapping from the prior distribution P Z , rather than a variational approximation.

Original: (c) The forward operator f (cid:63) can be eﬃciently learned and sample generation is also eﬃcient.
Revised: (c) The forward operator f (cid:63) can be eﬃciently learned and sample generation is also eﬃcient.

Original: It would appear that these criteria violate the “no free lunch rule”, and some compromise must be involved.
Revised: It would appear that these criteria violate the “no free lunch rule”, and some compromise must be involved.

Original: Our goal is to investigate this trade-oﬀ: which design choices can make this approach work?
Revised: Our goal is to investigate this trade-oﬀ: which design choices can make this approach work?

Original: Speciﬁcally, a well studied construct in dynamical systems, namely the Perron-Frobenius operator (Lemmens & Nussbaum, 2012), suggests an alternative linear route to model the forward operator.
Revised: Speciﬁcally, a well studied construct in dynamical systems, namely the Perron-Frobenius operator (Lemmens & Nussbaum, 2012), suggests an alternative linear route to model the forward operator.

Original: Here, we show that if we are willing to give up on a few features in existing models – this may be acceptable depending on the downstream use case – then, the forward operator in generative models can be eﬃciently approximated as the estimation of a closed-form linear operator in the reproducing kernel Hilbert space (RKHS).
Revised: Here, we show that if we are willing to give up on a few features in existing models – this may be acceptable depending on the downstream use case – then, the forward operator in generative models can be eﬃciently approximated as the estimation of a closed-form linear operator in the reproducing kernel Hilbert space (RKHS).

Original: With simple adjustments of existing results, we identify a novel way to replace the expensive training for generative tasks with a simple principled kernel approach.
Revised: With simple adjustments of existing results, we identify a novel way to replace the expensive training for generative tasks with a simple principled kernel approach.

Original: Contributions.
Revised: Contributions.

Original: Our results are largely based on results in kernel methods and dynamical systems, but we demonstrate their relevance in generative modeling and complement recent ideas that emphasize links between deep generative models and dynamical systems.
Revised: Our results are largely based on results in kernel methods and dynamical systems, but we demonstrate their relevance in generative modeling and complement recent ideas that emphasize links between deep generative models and dynamical systems.

Original: Our contributions are (a) We propose a non-parametric method for transferring a known prior density linearly in RKHS to an unknown data density – equivalent to learning a nonlinear forward operator in the input space.
Revised: Our contributions are (a) We propose a non-parametric method for transferring a known prior density linearly in RKHS to an unknown data density – equivalent to learning a nonlinear forward operator in the input space.

Original: When compared to its functionally-analogous module used in other deep generative methods, our method avoids multiple expensive training steps yielding signiﬁcant eﬃciency gains; (b) We evaluate this idea in multiple scenarios and show competitive generation performance and eﬃciency beneﬁts with pre-trained autoencoders on popular image datasets including MNIST, CIFAR-10, CelebA and FFHQ; (c) As a special use case, we demonstrate the advantages over other methods in limited data settings.
Revised: When compared to its functionally-analogous module used in other deep generative methods, our method avoids multiple expensive training steps yielding signiﬁcant eﬃciency gains; (b) We evaluate this idea in multiple scenarios and show competitive generation performance and eﬃciency beneﬁts with pre-trained autoencoders on popular image datasets including MNIST, CIFAR-10, CelebA and FFHQ; (c) As a special use case, we demonstrate the advantages over other methods in limited data settings.

Original:  Preliminaries
Revised:  Preliminaries

Original: [Table]
Revised: [Table]

Original: We brieﬂy introduce reproducing kernel Hilbert space (RKHS) and kernel embedding of probability distributions, concepts we will use frequently
Revised: We brieﬂy introduce reproducing kernel Hilbert space (RKHS) and kernel embedding of probability distributions, concepts we will use frequently

Original: Deﬁnition 2.1 (RKHS (Aronszajn, 1950)) .
Revised: Deﬁnition 2.1 (RKHS (Aronszajn, 1950)) .

Original: For a set X , let H be a set of functions
Revised: For a set X , let H be a set of functions

Original: g : X → R .
Revised: g : X → R .

Original: Then, H is a reproducing kernel Hilbert space (RKHS) with a product (cid:104)· , ·(cid:105) H if there exists a function k : X × X
Revised: Then, H is a reproducing kernel Hilbert space (RKHS) with a product (cid:104)· , ·(cid:105) H if there exists a function k : X × X

Original: → R (called a reproducing kernel) such that (i) ∀ x ∈ X , g ∈ H , g ( x ) = (cid:104) g, k ( x, · ) (cid:105) H ; (ii) H = cl ( span ( { k ( x, · ) , x ∈ X} )) , where cl ( · ) is the set closure.
Revised: → R (called a reproducing kernel) such that (i) ∀ x ∈ X , g ∈ H , g ( x ) = (cid:104) g, k ( x, · ) (cid:105) H ; (ii) H = cl ( span ( { k ( x, · ) , x ∈ X} )) , where cl ( · ) is the set closure.

Original: The function φ ( x ) =
Revised: The function φ ( x ) =

Original: k ( x, · ) : X → H is referred to as the feature mapping of the induced
Revised: k ( x, · ) : X → H is referred to as the feature mapping of the induced

Original: RKHS H .
Revised: RKHS H .

Original: A useful identity derived from feature mappings is the kernel mean
Revised: A useful identity derived from feature mappings is the kernel mean

Original: embedding : it deﬁnes a mapping from a probablity measure in X to an element in the RKHS.
Revised: embedding : it deﬁnes a mapping from a probablity measure in X to an element in the RKHS.

Original: Deﬁnition 2.2 (Kernel Mean Embedding (Smola et al., 2007)) .
Revised: Deﬁnition 2.2 (Kernel Mean Embedding (Smola et al., 2007)) .

Original: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup x ∈X k ( x, x ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ p ∈ H , is deﬁned as µ p =
Revised: Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel k such that sup x ∈X k ( x, x ) < ∞ , the kernel mean embedding of p in RKHS H , denoted by µ p ∈ H , is deﬁned as µ p =

Original: E p
Revised: E p

Original: [ φ ( x )]
Revised: [ φ ( x )]

Original: = (cid:82) k
Revised: = (cid:82) k

Original: ( x, · ) p ( x ) dx , and the mean embedding operator E :
Revised: ( x, · ) p ( x ) dx , and the mean embedding operator E :

Original: L 1 ( X ) → H is deﬁned as µ p = E p .
Revised: L 1 ( X ) → H is deﬁned as µ p = E p .

Original: Remark 1.
Revised: Remark 1.

Original: For characteristic kernels, the operator E is injective.
Revised: For characteristic kernels, the operator E is injective.

Original: Thus, two distributions ( p, q ) in X are identical iﬀ E p =
Revised: Thus, two distributions ( p, q ) in X are identical iﬀ E p =

Original: E q .
Revised: E q .

Original: This property allows using of Maximum Mean Discrepancy (MMD) for distribution matching (Gretton et al., 2012; Li et al., 2017) and is common, see (Muandet et al., 2017;
Revised: This property allows using of Maximum Mean Discrepancy (MMD) for distribution matching (Gretton et al., 2012; Li et al., 2017) and is common, see (Muandet et al., 2017;

Original: Zhou et al., 2018).
Revised: Zhou et al., 2018).

Original: For a ﬁnite number of samples { x i } ni =1 drawn from the probability measure p , an unbiased empirical estimate of µ H is ˆ µ H = 1 n (cid:80) ni
Revised: For a ﬁnite number of samples { x i } ni =1 drawn from the probability measure p , an unbiased empirical estimate of µ H is ˆ µ H = 1 n (cid:80) ni

Original: =1 k ( x i , · ) such that
Revised: =1 k ( x i , · ) such that

Original: [Equation]
Revised: [Equation]

Original: Next, we review the covariance/cross covariance operators, two widely-used identities in kernel methods (Fukumizu et al., 2013; Song et al., 2013) and building blocks of our approach.
Revised: Next, we review the covariance/cross covariance operators, two widely-used identities in kernel methods (Fukumizu et al., 2013; Song et al., 2013) and building blocks of our approach.

Original: Deﬁnition 2.3 (Covariance/Cross-covariance Operator) .
Revised: Deﬁnition 2.3 (Covariance/Cross-covariance Operator) .

Original: Let X, Z be random variables deﬁned on X × Z with joint distribution P X,Z and marginal distributions P X , P Z .
Revised: Let X, Z be random variables deﬁned on X × Z with joint distribution P X,Z and marginal distributions P X , P Z .

Original: Let ( l, φ, H ) and ( k, ψ, G ) be two sets of (a) bounded kernel, (b) their corresponding feature map, and (c) their induced RKHS, respectively.
Revised: Let ( l, φ, H ) and ( k, ψ, G ) be two sets of (a) bounded kernel, (b) their corresponding feature map, and (c) their induced RKHS, respectively.

Original: The (uncentered) covariance operator C ZZ : H → H and cross-covariance operator C XZ : H → G are deﬁned as
Revised: The (uncentered) covariance operator C ZZ : H → H and cross-covariance operator C XZ : H → G are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ⊗ is the outer product operator.
Revised: where ⊗ is the outer product operator.

Original:  Simplifying the estimation of the forward operator
Revised:  Simplifying the estimation of the forward operator

Original: Forward operator as a dynamical system: The dynamical system view of generative models has been described by others (Chen et al., 2018; Grathwohl et al., 2019; Behrmann et al., 2019).
Revised: Forward operator as a dynamical system: The dynamical system view of generative models has been described by others (Chen et al., 2018; Grathwohl et al., 2019; Behrmann et al., 2019).

Original: These strategies model the evolution of latent variables in a residual neural network in terms of its dynamics over continuous or discrete time t , and consider the output function f as the evaluation function at a predetermined boundary condition t
Revised: These strategies model the evolution of latent variables in a residual neural network in terms of its dynamics over continuous or discrete time t , and consider the output function f as the evaluation function at a predetermined boundary condition t

Original: = t 1 .
Revised: = t 1 .

Original: Speciﬁcally, given an input (i.e., initial condition) z
Revised: Speciﬁcally, given an input (i.e., initial condition) z

Original: ( t 0 ), f is deﬁned as
Revised: ( t 0 ), f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ∆ t is a time-dependent neural network function and z ( t ) is the intermediate solution at t .
Revised: where ∆ t is a time-dependent neural network function and z ( t ) is the intermediate solution at t .

Original: This view of generative models is not limited to speciﬁc methods or model archetypes, but generally useful, for example, viewing the outputs of each hidden layer as evaluations in discrete-time dynamics.
Revised: This view of generative models is not limited to speciﬁc methods or model archetypes, but generally useful, for example, viewing the outputs of each hidden layer as evaluations in discrete-time dynamics.

Original: After applying f on a random variable Z , the marginal density of the output over any subspace Λ ⊆
Revised: After applying f on a random variable Z , the marginal density of the output over any subspace Λ ⊆

Original: X can be expressed as
Revised: X can be expressed as

Original: [Equation]
Revised: [Equation]

Original: If there exists some neural network instance ∆ (cid:63)t such that the corresponding output function f (cid:63) satisﬁes P X
Revised: If there exists some neural network instance ∆ (cid:63)t such that the corresponding output function f (cid:63) satisﬁes P X

Original: =
Revised: =

Original: P f (cid:63) ( Z ) , by Def. 1.1, f (cid:63) is a forward operator.
Revised: P f (cid:63) ( Z ) , by Def. 1.1, f (cid:63) is a forward operator.

Original: Let X be a set of i.i.d.
Revised: Let X be a set of i.i.d.

Original: samples drawn from P X .
Revised: samples drawn from P X .

Original: In typical generative learning, either maximizing the likelihood 1 | X | (cid:80) x
Revised: In typical generative learning, either maximizing the likelihood 1 | X | (cid:80) x

Original: ∈
Revised: ∈

Original: X p f
Revised: X p f

Original: ( Z ) ( x ) or minimizing the distributional divergence d ( P f ( Z ) , P X ) requires evaluating and diﬀerentiating through f or f − 1 many times.
Revised: ( Z ) ( x ) or minimizing the distributional divergence d ( P f ( Z ) , P X ) requires evaluating and diﬀerentiating through f or f − 1 many times.

Original: Towards a one-step estimation of forward operator: Since f and f − 1 in (3) will be highly nonlinear in practice, evaluating and computing the gradients can be expensive.
Revised: Towards a one-step estimation of forward operator: Since f and f − 1 in (3) will be highly nonlinear in practice, evaluating and computing the gradients can be expensive.

Original: Nevertheless, the dynamical systems literature suggests a linear extension of f ∗ , namely the
Revised: Nevertheless, the dynamical systems literature suggests a linear extension of f ∗ , namely the

Original: Perron-Frobenius operator or transfer operator, that conveniently transfers p Z to p X .
Revised: Perron-Frobenius operator or transfer operator, that conveniently transfers p Z to p X .

Original: Deﬁnition 3.1 (Perron-Frobenius operator (Mayer, 1980)) .
Revised: Deﬁnition 3.1 (Perron-Frobenius operator (Mayer, 1980)) .

Original: Given a dynamical system f : X → X , the Perron-Frobenius (PF) operator P : L 1 ( X ) → L 1 ( X ) is an inﬁnitedimensional linear operator deﬁned as (cid:82) Λ ( P p Z )( x ) dx =
Revised: Given a dynamical system f : X → X , the Perron-Frobenius (PF) operator P : L 1 ( X ) → L 1 ( X ) is an inﬁnitedimensional linear operator deﬁned as (cid:82) Λ ( P p Z )( x ) dx =

Original: (cid:82) z ∈ f − 1 (Λ)
Revised: (cid:82) z ∈ f − 1 (Λ)

Original: p Z
Revised: p Z

Original: ( z ) dz for all Λ ⊆ X .
Revised: ( z ) dz for all Λ ⊆ X .

Original: involves embedding the prior samples z ∗ ∼ P Z , applying the operator Ψ ∗ =
Revised: involves embedding the prior samples z ∗ ∼ P Z , applying the operator Ψ ∗ =

Original: ˆ P E φ ( z ∗ ) and ﬁnding preimages ψ † (Ψ ∗ ).
Revised: ˆ P E φ ( z ∗ ) and ﬁnding preimages ψ † (Ψ ∗ ).

Original: The pre-trained autoencoder projects the data a smooth latent space and is only required when generating high-dimensional data such as images
Revised: The pre-trained autoencoder projects the data a smooth latent space and is only required when generating high-dimensional data such as images

Original: Although in Def. 3.1, the PF operator P is deﬁned for self-maps, it is trivial to extend P to mappings f : Z → X by restricting the RHS integral (cid:82) z ∈ f − 1 (Λ)
Revised: Although in Def. 3.1, the PF operator P is deﬁned for self-maps, it is trivial to extend P to mappings f : Z → X by restricting the RHS integral (cid:82) z ∈ f − 1 (Λ)

Original: p Z ( z ) dz to Z .
Revised: p Z ( z ) dz to Z .

Original: It can be seen that, for the forward operator f ∗ , the corresponding PF operator P satisﬁes
Revised: It can be seen that, for the forward operator f ∗ , the corresponding PF operator P satisﬁes

Original: [Equation]
Revised: [Equation]

Original: If P can be eﬃciently computed, transferring the tractable density p Z to the target density p X can be accomplished simply by applying P .
Revised: If P can be eﬃciently computed, transferring the tractable density p Z to the target density p X can be accomplished simply by applying P .

Original: However, since P is an inﬁnite-dimensional operator on L 1 ( X ), it is impractical to instantiate it explicitly and exactly.
Revised: However, since P is an inﬁnite-dimensional operator on L 1 ( X ), it is impractical to instantiate it explicitly and exactly.

Original: Nonetheless, there exist several methods for estimating the Perron-Frobenius operator, including Ulam’s method (Ulam, 1960) and the Extended Dynamical Mode Decomposition (EDMD) (Williams et al., 2015a).
Revised: Nonetheless, there exist several methods for estimating the Perron-Frobenius operator, including Ulam’s method (Ulam, 1960) and the Extended Dynamical Mode Decomposition (EDMD) (Williams et al., 2015a).

Original: Both strategies project P onto a ﬁnite number of hand-crafted basis functions – this may suﬃce in many settings but may fall short in modeling highly complex dynamics.
Revised: Both strategies project P onto a ﬁnite number of hand-crafted basis functions – this may suﬃce in many settings but may fall short in modeling highly complex dynamics.

Original: Kernel-embedded form of PF operator:
Revised: Kernel-embedded form of PF operator:

Original: A natural extension of PF operator is to represent P by an inﬁnite set of functions (Klus et al., 2020), e.g., projecting it onto the bases of an RKHS via the kernel trick .
Revised: A natural extension of PF operator is to represent P by an inﬁnite set of functions (Klus et al., 2020), e.g., projecting it onto the bases of an RKHS via the kernel trick .

Original: There, for a characteristic kernel l , the kernel mean embedding uniquely identiﬁes an element µ X =
Revised: There, for a characteristic kernel l , the kernel mean embedding uniquely identiﬁes an element µ X =

Original: E l p X ∈ G for any p X ∈
Revised: E l p X ∈ G for any p X ∈

Original: L 1 ( X ).
Revised: L 1 ( X ).

Original: Thus, to approximate P , we may alternatively solve for the dynamics from p Z to p X in their embedded form.
Revised: Thus, to approximate P , we may alternatively solve for the dynamics from p Z to p X in their embedded form.

Original: Using
Revised: Using

Original: Tab. 1 notations, we have the following linear operator that deﬁnes the dynamics between two embedded densities.
Revised: Tab. 1 notations, we have the following linear operator that deﬁnes the dynamics between two embedded densities.

Original: Deﬁnition 3.2 (Kernel-embedded Perron-Frobenius operator (Klus et al., 2020)) .
Revised: Deﬁnition 3.2 (Kernel-embedded Perron-Frobenius operator (Klus et al., 2020)) .

Original: Given p Z ∈ L 1 ( X ) and p X ∈ L 1 ( X ) .
Revised: Given p Z ∈ L 1 ( X ) and p X ∈ L 1 ( X ) .

Original: Denote k as the input kernel and l as the output kernel .
Revised: Denote k as the input kernel and l as the output kernel .

Original: Let µ X =
Revised: Let µ X =

Original: E l p X and µ Z =
Revised: E l p X and µ Z =

Original: E k p
Revised: E k p

Original: Z be their corresponding mean kernel embeddings.
Revised: Z be their corresponding mean kernel embeddings.

Original: The kernel-embedded Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned as
Revised: The kernel-embedded Perron-Frobenius (kPF) operator, denoted by P E : H → G , is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.1 (Song et al.
Revised: Proposition 3.1 (Song et al.

Original: With the above deﬁnition, P E satisﬁes
Revised: With the above deﬁnition, P E satisﬁes

Original: [Equation]
Revised: [Equation]

Original: under the conditions: (i) C ZZ is injective (ii) µ t ∈ range ( C ZZ )
Revised: under the conditions: (i) C ZZ is injective (ii) µ t ∈ range ( C ZZ )

Original: (iii) E [ g ( X )
Revised: (iii) E [ g ( X )

Original: | Z = · ] ∈ H for any g ∈ G .
Revised: | Z = · ] ∈ H for any g ∈ G .

Original: The last two assumptions can sometimes be diﬃcult to satisfy for certain RKHS (see Theorem 2 of Fukumizu et al.
Revised: The last two assumptions can sometimes be diﬃcult to satisfy for certain RKHS (see Theorem 2 of Fukumizu et al.

Original: In such cases, a relaxed solution can be constructed by replacing
Revised: In such cases, a relaxed solution can be constructed by replacing

Original: C − 1 ZZ by a regularized inverse ( C ZZ + λI ) − 1 or a Moore-Penrose pseudoinverse C † ZZ .
Revised: C − 1 ZZ by a regularized inverse ( C ZZ + λI ) − 1 or a Moore-Penrose pseudoinverse C † ZZ .

Original: The following proposition shows commutativity between the (kernel-embedded) PF operator and the mean embedding operator, showing its equivalence to P when l is characteristic.
Revised: The following proposition shows commutativity between the (kernel-embedded) PF operator and the mean embedding operator, showing its equivalence to P when l is characteristic.

Original: Proposition 3.2 ((Klus et al., 2020)) .
Revised: Proposition 3.2 ((Klus et al., 2020)) .

Original: With the above notations, E l ◦ P =
Revised: With the above notations, E l ◦ P =

Original: P
Revised: P

Original: E ◦ E k .
Revised: E ◦ E k .

Original: Transferring embedded densities with the kPF operator: The kPF operator is a powerful tool that allows transferring embedded densities in RKHS.
Revised: Transferring embedded densities with the kPF operator: The kPF operator is a powerful tool that allows transferring embedded densities in RKHS.

Original: The main steps are:
Revised: The main steps are:

Original: (1) Use mean embedding operator E l on p Z .
Revised: (1) Use mean embedding operator E l on p Z .

Original: Let us denote it by µ Z .
Revised: Let us denote it by µ Z .

Original: (2) Transfer µ Z using kPF operator P E to get the mean embedded p X , given by µ X .
Revised: (2) Transfer µ Z using kPF operator P E to get the mean embedded p X , given by µ X .

Original: Of course, in practice with ﬁnite data, { x i } i ∈ [ n ] ∼ P X and { z i } i ∈ [ n ] ∼ P X , P E must be estimated empirically (see Klus et al.
Revised: Of course, in practice with ﬁnite data, { x i } i ∈ [ n ] ∼ P X and { z i } i ∈ [ n ] ∼ P X , P E must be estimated empirically (see Klus et al.

Original: (2020) for an error analysis).
Revised: (2020) for an error analysis).

Original: [Equation]
Revised: [Equation]

Original: where Φ =
Revised: where Φ =

Original: [ k ( z 1 , · ) , · · · , k ( z n , · )] , Ψ =
Revised: [ k ( z 1 , · ) , · · · , k ( z n , · )] , Ψ =

Original: [ l ( x 1 , · ) , · · · , l ( x n , · )] are simply the corresponding feature matrices for samples of P X and P Z , and λ is a small penalty term.
Revised: [ l ( x 1 , · ) , · · · , l ( x n , · )] are simply the corresponding feature matrices for samples of P X and P Z , and λ is a small penalty term.

Original: Learning kPF for unconditional generative modeling: Some generative modeling methods such as VAEs and ﬂow-based formulations explicitly model the latent variable Z as conditionally dependent on the data variable X .
Revised: Learning kPF for unconditional generative modeling: Some generative modeling methods such as VAEs and ﬂow-based formulations explicitly model the latent variable Z as conditionally dependent on the data variable X .

Original: This allows deriving/optimizing the likelihood p f ( Z ) ( X ).
Revised: This allows deriving/optimizing the likelihood p f ( Z ) ( X ).

Original: This is desirable but may not be essential in all applications.
Revised: This is desirable but may not be essential in all applications.

Original: To learn a kPF, however, X and Z can be independent RVs.
Revised: To learn a kPF, however, X and Z can be independent RVs.

Original: While it may not be immediately obvious why we could assume this independence, we can observe the following property for the empirical kPF operator, assuming that the empirical covariance operator ˆ C ZZ is non-singular:
Revised: While it may not be immediately obvious why we could assume this independence, we can observe the following property for the empirical kPF operator, assuming that the empirical covariance operator ˆ C ZZ is non-singular:

Original: [Equation]
Revised: [Equation]

Original: Suppose that { x i } i ∈ [ n ] and { z j } i ∈ [ n ] are independently sampled from the marginals P X and P Z .
Revised: Suppose that { x i } i ∈ [ n ] and { z j } i ∈ [ n ] are independently sampled from the marginals P X and P Z .

Original: It is easy to verify that (7) holds for any pairing { ( x i , z j ) }
Revised: It is easy to verify that (7) holds for any pairing { ( x i , z j ) }

Original: ( i,j ) ∈ [ n ] × [ n ] .
Revised: ( i,j ) ∈ [ n ] × [ n ] .

Original: However, instantiating the RVs in this way rules out the use of kPF for certain downstream tasks such as controlled generation or mode detection, since Z does not contain information regarding
Revised: However, instantiating the RVs in this way rules out the use of kPF for certain downstream tasks such as controlled generation or mode detection, since Z does not contain information regarding

Original: X .
Revised: X .

Original: Nevertheless, if sampling is our only goal, then this instantiation of kPF will suﬃce.
Revised: Nevertheless, if sampling is our only goal, then this instantiation of kPF will suﬃce.

Original: Mapping Z to G : Now, since P E is a deterministic linear operator, we can easily set up a scheme to map samples of Z to elements of G where the expectation of the mapped samples equals µ X
Revised: Mapping Z to G : Now, since P E is a deterministic linear operator, we can easily set up a scheme to map samples of Z to elements of G where the expectation of the mapped samples equals µ X

Original: Deﬁne φ ( z ) =
Revised: Deﬁne φ ( z ) =

Original: k ( z, · ) and ψ ( x ) =
Revised: k ( z, · ) and ψ ( x ) =

Original: l ( x, · ) as feature maps of kernels k and l .
Revised: l ( x, · ) as feature maps of kernels k and l .

Original: We can rewrite µ X as
Revised: We can rewrite µ X as

Original: [Equation]
Revised: [Equation]

Original: Here ψ − 1 is the inverse or the preimage map of ψ .
Revised: Here ψ − 1 is the inverse or the preimage map of ψ .

Original: Such an inverse, in general, may not exist (Kwok & Tsang, 2004; Honeine & Richard, 2011).
Revised: Such an inverse, in general, may not exist (Kwok & Tsang, 2004; Honeine & Richard, 2011).

Original: We will discuss a procedure to approximate ψ − 1 in § 4.1.
Revised: We will discuss a procedure to approximate ψ − 1 in § 4.1.

Original: In what follows, we will temporarily assume that an exact preimage map exists and is tractable to compute.
Revised: In what follows, we will temporarily assume that an exact preimage map exists and is tractable to compute.

Original: Deﬁne Ψ ∗ =
Revised: Deﬁne Ψ ∗ =

Original: ˆ P E φ ( Z ) as the transferred sample in G using the empirical embedded PF operator ˆ P E .
Revised: ˆ P E φ ( Z ) as the transferred sample in G using the empirical embedded PF operator ˆ P E .

Original: Then the next result shows that asymptotically the transferred samples converge in distribution to the target distribution.
Revised: Then the next result shows that asymptotically the transferred samples converge in distribution to the target distribution.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: As n → ∞ , ψ − 1 (Ψ ∗ ) D → P X .
Revised: As n → ∞ , ψ − 1 (Ψ ∗ ) D → P X .

Original: That is, the preimage of the transferred sample approximately conforms to P X under previous assumptions when n is large.
Revised: That is, the preimage of the transferred sample approximately conforms to P X under previous assumptions when n is large.

Original: Proof.
Revised: Proof.

Original: Since ˆ P E asymp.
Revised: Since ˆ P E asymp.

Original: →
Revised: →

Original: P , the proof immediately follows from (8).
Revised: P , the proof immediately follows from (8).

Original:  Sample generation using the Kernel transfer operator
Revised:  Sample generation using the Kernel transfer operator

Original: At this point, the transferred sample Ψ ∗ , obtained by the kPF operator, remains an element of RKHS G .
Revised: At this point, the transferred sample Ψ ∗ , obtained by the kPF operator, remains an element of RKHS G .

Original: To translate the samples back to the input space, we must ﬁnd the preimage x ∗ such that ψ ( x ∗ ) =
Revised: To translate the samples back to the input space, we must ﬁnd the preimage x ∗ such that ψ ( x ∗ ) =

Original: Ψ ∗ .
Revised: Ψ ∗ .

Original: 4. Solving for an approximate preimage
Revised: 4. Solving for an approximate preimage

Original: Solving the preimage in kernel-based methods is known to be ill-posed (Mika et al., 1999) because the mapping ψ ( · ) is not necessarily surjective, i.e., a unique preimage x ∗ = ψ − 1 (Ψ ∗ ) , Ψ ∗ ∈ H may not exist.
Revised: Solving the preimage in kernel-based methods is known to be ill-posed (Mika et al., 1999) because the mapping ψ ( · ) is not necessarily surjective, i.e., a unique preimage x ∗ = ψ − 1 (Ψ ∗ ) , Ψ ∗ ∈ H may not exist.

Original: Often, an approximate preimage ψ † ( X , Ψ ∗ )
Revised: Often, an approximate preimage ψ † ( X , Ψ ∗ )

Original: ≈ ψ − 1 (Ψ ∗ ) is constructed instead based on relational properties among the training data in the RKHS.
Revised: ≈ ψ − 1 (Ψ ∗ ) is constructed instead based on relational properties among the training data in the RKHS.

Original: We consider two options in our framework (1) MDS-based method (Kwok & Tsang, 2004;
Revised: We consider two options in our framework (1) MDS-based method (Kwok & Tsang, 2004;

Original: Honeine & Richard, 2011),
Revised: Honeine & Richard, 2011),

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( X (cid:48)(cid:62) X (cid:48)
Revised: ( X (cid:48)(cid:62) X (cid:48)

Original: [Equation]
Revised: [Equation]

Original: − d (cid:62) ) , where ∀ i ∈ [ γ ] , d i = (cid:107) l ( x (cid:48) i , · ) − Ψ ∗ (cid:107) G (9) which optimally preserves the distances in RKHS to the preimages in the input space, and (2) weighted Fr´echet mean (Friedman et al., 2001), which in Euclidean space takes the form ψ
Revised: − d (cid:62) ) , where ∀ i ∈ [ γ ] , d i = (cid:107) l ( x (cid:48) i , · ) − Ψ ∗ (cid:107) G (9) which optimally preserves the distances in RKHS to the preimages in the input space, and (2) weighted Fr´echet mean (Friedman et al., 2001), which in Euclidean space takes the form ψ

Original: [Equation]
Revised: [Equation]

Original: wFM ( X ,
Revised: wFM ( X ,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: ψ
Revised: ψ

Original: [Equation]
Revised: [Equation]

Original: wFM ( X
Revised: wFM ( X

Original: [Equation]
Revised: [Equation]

Original: s )
Revised: s )

Original: =
Revised: =

Original: X (cid:48) s / (cid:107) s (cid:107) 1 , where ∀ i ∈ [ γ ] , s i =
Revised: X (cid:48) s / (cid:107) s (cid:107) 1 , where ∀ i ∈ [ γ ] , s i =

Original: (cid:104) l
Revised: (cid:104) l

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: (cid:105) (10)
Revised: (cid:105) (10)

Original: [Table]
Revised: [Table]

Original: Sample { z i } i ∈ [ n ] ∼ P
Revised: Sample { z i } i ∈ [ n ] ∼ P

Original: Z
Revised: Z

Original: independently 5:
Revised: independently 5:

Original: Construct L, K ∈ R n × n s.t.
Revised: Construct L, K ∈ R n × n s.t.

Original: L ij = l ( x i , x j ) , K ij = k
Revised: L ij = l ( x i , x j ) , K ij = k

Original: ( z i , z j ) 6:
Revised: ( z i , z j ) 6:

Original: K inv =
Revised: K inv =

Original: ( K + λnI ) − 1 or K † 7: Inference 8:
Revised: ( K + λnI ) − 1 or K † 7: Inference 8:

Original: Generate new prior sample z ∗ ∼ P Z 9: s = L · K inv [ k ( z 1 , z ∗ ) . . .
Revised: Generate new prior sample z ∗ ∼ P Z 9: s = L · K inv [ k ( z 1 , z ∗ ) . . .

Original: k ( z n , z ∗ )] (cid:62) 10: ind = argsort( s )[ − γ :] 11: x ∗ = ψ † wFM ( X [ ind ] ; s [ ind ]).
Revised: k ( z n , z ∗ )] (cid:62) 10: ind = argsort( s )[ − γ :] 11: x ∗ = ψ † wFM ( X [ ind ] ; s [ ind ]).

Original: 12: Output D ( x ∗ ) if D is provided else x ∗ where X (cid:48) a neighborhood of γ training samples based on pairwise distance or similarity in RKHS, following (Kwok & Tsang, 2004).
Revised: 12: Output D ( x ∗ ) if D is provided else x ∗ where X (cid:48) a neighborhood of γ training samples based on pairwise distance or similarity in RKHS, following (Kwok & Tsang, 2004).

Original: The weighted Fr´echet mean preimage uses the inner product weights (cid:104)
Revised: The weighted Fr´echet mean preimage uses the inner product weights (cid:104)

Original: Ψ ∗ , ψ ( x i ) (cid:105) as measures of similarities to interpolate training samples.
Revised: Ψ ∗ , ψ ( x i ) (cid:105) as measures of similarities to interpolate training samples.

Original: On the toy data (as in Fig. 2), weighted Fr´echet mean produces fewer samples that deviate from the true distribution and is easier to compute.
Revised: On the toy data (as in Fig. 2), weighted Fr´echet mean produces fewer samples that deviate from the true distribution and is easier to compute.

Original: Based on this observation, we use the weighted Fr´echet mean as the preimage module for all experiments that requires samples, while acknowledging that other preimage methods can also be substituted in.
Revised: Based on this observation, we use the weighted Fr´echet mean as the preimage module for all experiments that requires samples, while acknowledging that other preimage methods can also be substituted in.

Original: With all the ingredients in hand, we now present an algorithm for sample generation using the kPF operator in Alg. 1.
Revised: With all the ingredients in hand, we now present an algorithm for sample generation using the kPF operator in Alg. 1.

Original: The idea is simple yet powerful: at training time, we construct the empirical kPF operator ˆ P E using the training data { x i } i ∈ [ s ] and samples of the known prior { z i } ı ∈ [ n ] .
Revised: The idea is simple yet powerful: at training time, we construct the empirical kPF operator ˆ P E using the training data { x i } i ∈ [ s ] and samples of the known prior { z i } ı ∈ [ n ] .

Original: At test time, we will transfer new points sampled from P Z to feature maps in H , and construct their preimages as the generated output samples.
Revised: At test time, we will transfer new points sampled from P Z to feature maps in H , and construct their preimages as the generated output samples.

Original: 4. Image generation
Revised: 4. Image generation

Original: Image generation is a common application for generative models (Goodfellow et al., 2014;
Revised: Image generation is a common application for generative models (Goodfellow et al., 2014;

Original: Dinh et al., 2017).
Revised: Dinh et al., 2017).

Original: While our proposal is not image speciﬁc, constructing sample preimages in a high dimensional space with limited training samples can be challenging, since the space of images is usually not dense in a reasonably sized neighborhood.
Revised: While our proposal is not image speciﬁc, constructing sample preimages in a high dimensional space with limited training samples can be challenging, since the space of images is usually not dense in a reasonably sized neighborhood.

Original: However, empirically images often lie near a low dimensional manifold in the ambient space (Seung & Lee, 2000), and one may utilize an autoencoder (AE) ( E, D ) to embed the images onto a latent space that represents coordinates on a learned manifold.
Revised: However, empirically images often lie near a low dimensional manifold in the ambient space (Seung & Lee, 2000), and one may utilize an autoencoder (AE) ( E, D ) to embed the images onto a latent space that represents coordinates on a learned manifold.

Original: If the learned manifold lies close to the true manifold, we can learn densities on the manifold directly (Dai & Wipf, 2019).
Revised: If the learned manifold lies close to the true manifold, we can learn densities on the manifold directly (Dai & Wipf, 2019).

Original: Therefore, for image generation tasks, the training data is ﬁrst projected onto the latent space of a pretrained AE.
Revised: Therefore, for image generation tasks, the training data is ﬁrst projected onto the latent space of a pretrained AE.

Original: Then, the operator will be constructed using the projected latent representations, and samples will be mapped back to image space with the decoder of AE.
Revised: Then, the operator will be constructed using the projected latent representations, and samples will be mapped back to image space with the decoder of AE.

Original: Our setup can be viewed analogously to other generative methods based on so called “ ex-post ” density estimation of latent variables (Ghosh et al., 2020).
Revised: Our setup can be viewed analogously to other generative methods based on so called “ ex-post ” density estimation of latent variables (Ghosh et al., 2020).

Original: We also restrict the AE latent space to a hypersphere S n − 1 to ensure that (a) k ( · , · ) and l ( · , · ) are bounded and (b) the space is geodesically convex and complete, which is required by the preimage computation.
Revised: We also restrict the AE latent space to a hypersphere S n − 1 to ensure that (a) k ( · , · ) and l ( · , · ) are bounded and (b) the space is geodesically convex and complete, which is required by the preimage computation.

Original: To compute the weighted Fr´echet mean on a hypersphere, we adopt the recursive algorithm in Chakraborty & Vemuri (2015) (see appendix D for details).
Revised: To compute the weighted Fr´echet mean on a hypersphere, we adopt the recursive algorithm in Chakraborty & Vemuri (2015) (see appendix D for details).

Original: [Figure]
Revised: [Figure]

Original:  Experimental Results
Revised:  Experimental Results

Original: Goals.
Revised: Goals.

Original: In our experiments, we seek to answer three questions: (a) With suﬃcient data, can our method generate new data with comparable performance with other state-of-the-art generative models?
Revised: In our experiments, we seek to answer three questions: (a) With suﬃcient data, can our method generate new data with comparable performance with other state-of-the-art generative models?

Original: (b) If only limited data samples were given, can our method still estimate the density with reasonable accuracy?
Revised: (b) If only limited data samples were given, can our method still estimate the density with reasonable accuracy?

Original: (c) What are the runtime beneﬁts, if any?
Revised: (c) What are the runtime beneﬁts, if any?

Original: Datasets/setup.
Revised: Datasets/setup.

Original: To answer the ﬁrst question, we evaluate our method on standard vision datasets, including MNIST, CIFAR10, and CelebA, where the number of data samples is much larger than the latent dimension.
Revised: To answer the ﬁrst question, we evaluate our method on standard vision datasets, including MNIST, CIFAR10, and CelebA, where the number of data samples is much larger than the latent dimension.

Original: We compare our results with other VAE variants (Two-stage VAE Dai & Wipf (2019), WAE Arjovsky et al.
Revised: We compare our results with other VAE variants (Two-stage VAE Dai & Wipf (2019), WAE Arjovsky et al.

Original: CV-VAE Ghosh et al.
Revised: CV-VAE Ghosh et al.

Original: and ﬂow-based generative models (Glow Kingma & Dhariwal (2018), CAGlow Liu et al.
Revised: and ﬂow-based generative models (Glow Kingma & Dhariwal (2018), CAGlow Liu et al.

Original: The second question is due to the broad use of kernel methods in small sample size settings.
Revised: The second question is due to the broad use of kernel methods in small sample size settings.

Original: For this more challenging case, we randomly choose 100 training samples ( < 1% of the full dataset) from CelebA and evaluate the quality of generation compared to other density approximation schemes.
Revised: For this more challenging case, we randomly choose 100 training samples ( < 1% of the full dataset) from CelebA and evaluate the quality of generation compared to other density approximation schemes.

Original: We also use a dataset of T1 Magnetic Resonance (MR) images from the Alzheimer’s Disease Neuromaging Initiative (ADNI) study.
Revised: We also use a dataset of T1 Magnetic Resonance (MR) images from the Alzheimer’s Disease Neuromaging Initiative (ADNI) study.

Original: Distribution transfer with many data samples.
Revised: Distribution transfer with many data samples.

Original: We evaluate the quality by calculating the Fr´echet Inception Distance (FID) (Heusel et al., 2017) with 10K generated images from each model.
Revised: We evaluate the quality by calculating the Fr´echet Inception Distance (FID) (Heusel et al., 2017) with 10K generated images from each model.

Original: Here, we use a pretained regularized autoencoder (Ghosh et al., 2020) with a latent space restricted to the hypersphere (denoted by SRAE) to obtain smooth latent representations.
Revised: Here, we use a pretained regularized autoencoder (Ghosh et al., 2020) with a latent space restricted to the hypersphere (denoted by SRAE) to obtain smooth latent representations.

Original: We compare our kPF to competitive end-to-end deep generative baselines (i.e. ﬂow and VAE variants) as well as other density estimation models over the same
Revised: We compare our kPF to competitive end-to-end deep generative baselines (i.e. ﬂow and VAE variants) as well as other density estimation models over the same

Original: SRAE latent space.
Revised: SRAE latent space.

Original: For the latent space models, we experimented with Glow (Kingma &
Revised: For the latent space models, we experimented with Glow (Kingma &

Original: [Figure]
Revised: [Figure]

Original: Comparative results are shown in Table 2.
Revised: Comparative results are shown in Table 2.

Original: We see that for images with structured feature spaces, e.g., MNIST and CelebA, our method matches other non-adversarial generative models, which provides evidence in support of the premise that the forward operator can be simpliﬁed.
Revised: We see that for images with structured feature spaces, e.g., MNIST and CelebA, our method matches other non-adversarial generative models, which provides evidence in support of the premise that the forward operator can be simpliﬁed.

Original: Further, we present qualitative results on all datasets (in Fig. 3), where we compare our kPF operator based model with other density estimation techniques on the latent space.
Revised: Further, we present qualitative results on all datasets (in Fig. 3), where we compare our kPF operator based model with other density estimation techniques on the latent space.

Original: Observe that our model generates comparable visual results as SRAE Glow .
Revised: Observe that our model generates comparable visual results as SRAE Glow .

Original: Since kPF learns the distribution on a pre-trained AE latent space for image generation, using a more powerful AE can oﬀer improvements in generation quality.
Revised: Since kPF learns the distribution on a pre-trained AE latent space for image generation, using a more powerful AE can oﬀer improvements in generation quality.

Original: In Fig.
Revised: In Fig.

Original: 4, we present representative images by learning our kPF on NVAE Vahdat & Kautz (2020) latent space, pre-trained on the FFHQ dataset.
Revised: 4, we present representative images by learning our kPF on NVAE Vahdat & Kautz (2020) latent space, pre-trained on the FFHQ dataset.

Original: NVAE builds a hierarchical prior and achieves state-of-the-art generation quality among VAEs.
Revised: NVAE builds a hierarchical prior and achieves state-of-the-art generation quality among VAEs.

Original: We see that kPF can indeed generate high-quality and diverse samples with the help of NVAE encoder/decoder.
Revised: We see that kPF can indeed generate high-quality and diverse samples with the help of NVAE encoder/decoder.

Original: In fact, any
Revised: In fact, any

Original: AE/VAE may be substituted in, assuming that the latent space is smooth .
Revised: AE/VAE may be substituted in, assuming that the latent space is smooth .

Original: Summary: When a suﬃcient number of samples are available, our algorithm performs as well as the alternatives, which is attractive given the eﬃcient training.
Revised: Summary: When a suﬃcient number of samples are available, our algorithm performs as well as the alternatives, which is attractive given the eﬃcient training.

Original: In Fig.
Revised: In Fig.

Original: 5, we present comparative result of FIDs with respect to the training time.
Revised: 5, we present comparative result of FIDs with respect to the training time.

Original: Since kPF can be computed in closed-form, it achieves signiﬁcant training eﬃciency gain compared to other deep generative methods while delivering competitive generative quality.
Revised: Since kPF can be computed in closed-form, it achieves signiﬁcant training eﬃciency gain compared to other deep generative methods while delivering competitive generative quality.

Original: Distribution transfer with limited data samples.
Revised: Distribution transfer with limited data samples.

Original: Next, we present our evaluations when only a limited number of samples are available.
Revised: Next, we present our evaluations when only a limited number of samples are available.

Original: Here, each of the density estimators was trained on latent representations of the same set of 100 randomly sampled CelebA images, and 10K images were generated to evaluate FID (see Table 3).
Revised: Here, each of the density estimators was trained on latent representations of the same set of 100 randomly sampled CelebA images, and 10K images were generated to evaluate FID (see Table 3).

Original: Our method outperforms
Revised: Our method outperforms

Original: Glow and VAE, while oﬀering competitive performance with GMM.
Revised: Glow and VAE, while oﬀering competitive performance with GMM.

Original: Surprisingly, GMM remains a strong baseline for both tasks, which agrees with results in Ghosh et al.[Figure] [Table] [Figure]
Revised: Surprisingly, GMM remains a strong baseline for both tasks, which agrees with results in Ghosh et al.

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: However, note that GMM is restricted by its parametric form and is less ﬂexible than our method (as shown in Fig 2).

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  space models
Revised: for latent space models.

Original: All models are learned on the latent representations encoded by the same pre-trained
Revised: All models are learned on the latent representations encoded by the same pre-trained

Original: SRAE
Revised: SRAE.

Original: However, note that GMM is restricted by its parametric form and is less ﬂexible than our method (as shown in Fig 2).
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Learning from few samples is common in biomedical applications where acquisition is costly.
Revised: Learning from few samples is common in biomedical applications where acquisition is costly.

Original: Motivated by interest in making synthetic but statistic preserving data (rather than the real patient records) publicly available to researchers (see NIH N3C Data Overview), we present results on generating high-resolution (160 × 196 × 160) brain images: 183 samples from group AD (diagnosed as Alzheimer’s disease) and 291 samples from group CN (control normals).
Revised: Motivated by interest in making synthetic but statistic preserving data (rather than the real patient records) publicly available to researchers (see NIH N3C Data Overview), we present results on generating high-resolution (160 × 196 × 160) brain images: 183 samples from group AD (diagnosed as Alzheimer’s disease) and 291 samples from group CN (control normals).

Original: For n = 474 (cid:28) d = 5017600, using our kernel operator, we can generate high-quality samples that are in-distribution.
Revised: For n = 474 (cid:28) d = 5017600, using our kernel operator, we can generate high-quality samples that are in-distribution.

Original: We present comparative results with VAEs.
Revised: We present comparative results with VAEs.

Original: The generated samples in Fig. 6 clearly show that our method generates sharper images.
Revised: The generated samples in Fig. 6 clearly show that our method generates sharper images.

Original: To check if the results are also scientiﬁcally meaningful, we test consistency between group diﬀerence testing (i.e., cases versus controls diﬀerential tests on each voxel) on the real images (groups were AD and CN) and the same test was performed on the generated samples (AD and CN groups), using a FWER corrected two-sample t -test Ashburner & 
Revised: To check if the results are also scientiﬁcally meaningful, we test consistency between group diﬀerence testing (i.e., cases versus controls diﬀerential tests on each voxel) on the real images (groups were AD and CN) and the same test was performed on the generated samples (AD and CN groups), using a FWER corrected two-sample t -test Ashburner & Friston (2000).

Original: Friston (2000).
Revised: The results (see

Original: The results (see Fig 6) show that while there is a deterioration in regions identiﬁed to be aﬀected by disease (diﬀerent across groups), many statistically-signiﬁcant regions from tests on the real images are preserved in voxel-wise tests on the generated images.
Revised: Fig 6) show that while there is a deterioration in regions identiﬁed to be aﬀected by disease (diﬀerent across groups), many statistically-signiﬁcant regions from tests on the real images are preserved in voxel-wise tests on the generated images.

Original: Summary: We achieve improvements in the small sample size setting compared to other generative methods.
Revised: Summary: We achieve improvements in the small sample size setting compared to other generative methods.

Original: This is useful in many data-poor settings.
Revised: This is useful in many data-poor settings.

Original: For larger datasets, our method still compares competitively with alternatives, but with a smaller resource footprint.
Revised: For larger datasets, our method still compares competitively with alternatives, but with a smaller resource footprint.

Original:  Limitations
Revised:  Limitations

Original: Our proposed simpliﬁcations can be variously useful, but deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS is unresolved at this time.
Revised: Our proposed simpliﬁcations can be variously useful, but deriving the density of the posterior given a mean embedding or providing an exact preimage for the generated sample in RKHS is unresolved at this time.

Original: While density estimation from kPF has been partially addressed in Schuster et al.
Revised: While density estimation from kPF has been partially addressed in Schuster et al.

Original: (2020b),
Revised: (2020b),

Original: ﬁnding the pre-image is often ill-posed.
Revised: ﬁnding the pre-image is often ill-posed.

Original: The weighted Fr´echet mean preimage only provides an approximate solution and evaluated empirically, and due to the interpolation-based sampling strategy, samples cannot be obtained beyond the convex hull of training examples.
Revised: The weighted Fr´echet mean preimage only provides an approximate solution and evaluated empirically, and due to the interpolation-based sampling strategy, samples cannot be obtained beyond the convex hull of training examples.

Original: Making Z and X independent RVs also limits its use for certain downstream task such as representation learning or semantic clustering.
Revised: Making Z and X independent RVs also limits its use for certain downstream task such as representation learning or semantic clustering.

Original: Finally, like other kernel-based methods, the sup-quadratic memory/compute cost can be a bottleneck on large datasets and kernel approximation (e.g. (Rahimi & Recht, 2009)) may have to be applied; we discuss this in appendix F.
Revised: Finally, like other kernel-based methods, the sup-quadratic memory/compute cost can be a bottleneck on large datasets and kernel approximation (e.g. (Rahimi & Recht, 2009)) may have to be applied; we discuss this in appendix F.

Original:  Conclusions
Revised:  Conclusions

Original: We show that using recent developments in regularized autoencoders, a linear kernel transfer operator can potentially be an eﬃcient substitute for the forward operator in some generative models, if some compromise in capabilities/performance is acceptable.
Revised: We show that using recent developments in regularized autoencoders, a linear kernel transfer operator can potentially be an eﬃcient substitute for the forward operator in some generative models, if some compromise in capabilities/performance is acceptable.

Original: Our proposal, despite its simplicity, shows comparable empirical results to other generative models, while oﬀering eﬃciency beneﬁts.
Revised: Our proposal, despite its simplicity, shows comparable empirical results to other generative models, while oﬀering eﬃciency beneﬁts.

Original: Results on brain images also show promise for applications to highresolution 3D imaging data generation, which is being pursued actively in the community.
Revised: Results on brain images also show promise for applications to highresolution 3D imaging data generation, which is being pursued actively in the community.

Original: A Choice of Kernel is relevant yet flexible
Revised: A Choice of Kernel is relevant yet flexible

Original: In some cases, one would focus on identifying (ﬁnite) eigenfuntions and modes of the underlying operator (Williams et al., 2015b; Brunton et al., 2016).
Revised: In some cases, one would focus on identifying (ﬁnite) eigenfuntions and modes of the underlying operator (Williams et al., 2015b; Brunton et al., 2016).

Original: But rather than ﬁnding certain modes that best characterize the dynamics, we care most about minimizing the error of the transferred density and therefore whether the span of functions is rich/expressive enough.
Revised: But rather than ﬁnding certain modes that best characterize the dynamics, we care most about minimizing the error of the transferred density and therefore whether the span of functions is rich/expressive enough.

Original: In particular, condition (iii) in Proposition 3.2 requires an input RKHS spanned by suﬃciently rich bases (Fukumizu et al., 2013).
Revised: In particular, condition (iii) in Proposition 3.2 requires an input RKHS spanned by suﬃciently rich bases (Fukumizu et al., 2013).

Original: For this reason, the choice of kernel here cannot be completely ignored since it determines the family of functions contained in the induced RKHS.
Revised: For this reason, the choice of kernel here cannot be completely ignored since it determines the family of functions contained in the induced RKHS.

Original: To explore the appropriate kernel setup for our application, we empirically evaluate the eﬀect of using several diﬀerent kernels via a simple experiment on MNIST.
Revised: To explore the appropriate kernel setup for our application, we empirically evaluate the eﬀect of using several diﬀerent kernels via a simple experiment on MNIST.

Original: We ﬁrst train an autoencoder to embed MNIST digits on to a hypersphere S 2 , then generate samples from kPF by the procedure described by Alg.
Revised: We ﬁrst train an autoencoder to embed MNIST digits on to a hypersphere S 2 , then generate samples from kPF by the procedure described by Alg.

Original: 1 using the respective kernel function as the input kernel k .
Revised: 1 using the respective kernel function as the input kernel k .

Original: Subplot (b) and (c) in Fig.
Revised: Subplot (b) and (c) in Fig.

Original: 7 show the generated samples using Radial Basis
Revised: 7 show the generated samples using Radial Basis

Original: Function (RBF) kernel and arc-cosine kernel, respectively.
Revised: Function (RBF) kernel and arc-cosine kernel, respectively.

Original: Observe that the choice of kernel has an inﬂuence on the sample population, and a kernel function with superior empirical behavior is desirable.
Revised: Observe that the choice of kernel has an inﬂuence on the sample population, and a kernel function with superior empirical behavior is desirable.

Original: Motivated by this observation, we evaluated the Neural Tangent Kernel (NTK)
Revised: Motivated by this observation, we evaluated the Neural Tangent Kernel (NTK)

Original: (Jacot et al., 2018), a well-studied neural kernel in recent works.
Revised: (Jacot et al., 2018), a well-studied neural kernel in recent works.

Original: We use it for a few reasons, (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, which spans a rich set of functions that satisﬁes the assumption.
Revised: We use it for a few reasons, (a) NTK, in theory, corresponds to a trained inﬁnitely-wide neural network, which spans a rich set of functions that satisﬁes the assumption.

Original: (b) For well-conditioned inputs (i.e., no duplicates) on hypersphere, the positive-deﬁniteness of NTK is proved in (Jacot et al., 2018).
Revised: (b) For well-conditioned inputs (i.e., no duplicates) on hypersphere, the positive-deﬁniteness of NTK is proved in (Jacot et al., 2018).

Original: Therefore, invertibility of the Gram matrix K ZZ =
Revised: Therefore, invertibility of the Gram matrix K ZZ =

Original: Φ T Φ is almost guaranteed if the prior distribution p Z is restricted on a hypersphere (c) NTK can be non-asymptotically approximated (Arora et al., 2019).
Revised: Φ T Φ is almost guaranteed if the prior distribution p Z is restricted on a hypersphere (c) NTK can be non-asymptotically approximated (Arora et al., 2019).

Original: (d) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough (Arora et al., 2019).
Revised: (d) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparameters, as long as the number of units used is large enough (Arora et al., 2019).

Original: Subplot (d) of Fig.
Revised: Subplot (d) of Fig.

Original: 7 shows that kPF learned with NTK as input kernel is able to generate samples that are more consistent with the data distribution.
Revised: 7 shows that kPF learned with NTK as input kernel is able to generate samples that are more consistent with the data distribution.

Original: However, we should note that
Revised: However, we should note that

Original: NTK is merely a convenient choice of kernel that requires less tuning, and is not otherwise central to our work.
Revised: NTK is merely a convenient choice of kernel that requires less tuning, and is not otherwise central to our work.

Original: In fact, as shown in our experiment in Tab. 2, a well-tuned RBF kernel may also achieve a similar performance.
Revised: In fact, as shown in our experiment in Tab. 2, a well-tuned RBF kernel may also achieve a similar performance.

Original: Indeed, in practice, any suitable choice of kernel may be conveniently adopted into the proposed framework without major modiﬁcations.
Revised: Indeed, in practice, any suitable choice of kernel may be conveniently adopted into the proposed framework without major modiﬁcations.

Original: B Density Estimation with kPF Operator
Revised: B Density Estimation with kPF Operator

Original: The displayed transferred density with the kPF operator on toy data in Fig.
Revised: The displayed transferred density with the kPF operator on toy data in Fig.

Original: 1 is approximated using the empirical kernel conditional density operator (CDO) (Schuster et al., 2020a), since there is currently no known methods that can exactly reconstruct density from the transferred mean embeddings.
Revised: 1 is approximated using the empirical kernel conditional density operator (CDO) (Schuster et al., 2020a), since there is currently no known methods that can exactly reconstruct density from the transferred mean embeddings.

Original: The marginalized transferred density p P E has the following form
Revised: The marginalized transferred density p P E has the following form

Original: [Equation]
Revised: [Equation]

Original: ) where C ρ = E y ∼ ρ [ l ( y, · ) ⊗ l ( y, · )] is the covariance operator of a independent reference density ρ in G .
Revised: ) where C ρ = E y ∼ ρ [ l ( y, · ) ⊗ l ( y, · )] is the covariance operator of a independent reference density ρ in G .

Original: The above density function is also an element of RKHS G , and therefore we can evaluate the density at x ∗ by using the reproducing property p P E ( x ∗ ) =
Revised: The above density function is also an element of RKHS G , and therefore we can evaluate the density at x ∗ by using the reproducing property p P E ( x ∗ ) =

Original: (cid:104) p P E , l ( x ∗ , · ) (cid:105) .
Revised: (cid:104) p P E , l ( x ∗ , · ) (cid:105) .

Original: The results in Schuster et al.
Revised: The results in Schuster et al.

Original: (2020a) show that the empirical estimate of p P E may be constructed from m samples of the reference density { y i } i ∈ [ m ] and n training samples { x i } i ∈ [ n ] and { z i } i ∈ [ n ] as
Revised: (2020a) show that the empirical estimate of p P E may be constructed from m samples of the reference density { y i } i ∈ [ m ] and n training samples { x i } i ∈ [ n ] and { z i } i ∈ [ n ] as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In Fig. 8, we use a uniform density in the square ( ± 4 , ± 4) as the reference density ρ and constructed ˆ p P E using m = 10000 samples { y i } i ∈
Revised: In Fig. 8, we use a uniform density in the square ( ± 4 , ± 4) as the reference density ρ and constructed ˆ p P E using m = 10000 samples { y i } i ∈

Original: [ m ] from ρ .
Revised: [ m ] from ρ .

Original: Due to the form of the empirical kernel CDO, where the estimated density function ˆ p P E is a linear combination of { l ( y i , · ) } i ∈
Revised: Due to the form of the empirical kernel CDO, where the estimated density function ˆ p P E is a linear combination of { l ( y i , · ) } i ∈

Original: [ m ] (as in Eq. 12), the approximation can be inaccurate if reference samples are relatively sparse and the densities are ‘sharp’.
Revised: [ m ] (as in Eq. 12), the approximation can be inaccurate if reference samples are relatively sparse and the densities are ‘sharp’.

Original: In those cases, to obtain a better density estimate, we may either increase the number of reference samples used to construct the empirical CDO (which can be computationally diﬃcult due to the need to compute ( L Y + α (cid:48) I ) − 2 ), or, with some prior knowledge to the true distribution, choose a reference density which is localized around the ground truth density.
Revised: In those cases, to obtain a better density estimate, we may either increase the number of reference samples used to construct the empirical CDO (which can be computationally diﬃcult due to the need to compute ( L Y + α (cid:48) I ) − 2 ), or, with some prior knowledge to the true distribution, choose a reference density which is localized around the ground truth density.

Original: Therefore, to show a more faithful density estimate of the transferred distribution for visualization purpose, we use a composite of the uniform density and the true density with weight 4 : 1 as the reference density ρ in Fig. 2.
Revised: Therefore, to show a more faithful density estimate of the transferred distribution for visualization purpose, we use a composite of the uniform density and the true density with weight 4 : 1 as the reference density ρ in Fig. 2.

Original: In this case, approximately 20% of the reference samples concentrates around the high-density areas of the true density, which helps to form a better basis for ˆ p P E .
Revised: In this case, approximately 20% of the reference samples concentrates around the high-density areas of the true density, which helps to form a better basis for ˆ p P E .

Original: Note that the choice of ρ does not aﬀect the transferred density embedding ˆ P E ˆ µ Z since it is independent of p X and p Z .
Revised: Note that the choice of ρ does not aﬀect the transferred density embedding ˆ P E ˆ µ Z since it is independent of p X and p Z .

Original: After this modiﬁcation, the reconstructed density more accurately reﬂects the true density compared to GMM and
Revised: After this modiﬁcation, the reconstructed density more accurately reﬂects the true density compared to GMM and

Original: GLOW, indicating the transferred distribution by kPF in RKHS matches better to the ground truth distribution.
Revised: GLOW, indicating the transferred distribution by kPF in RKHS matches better to the ground truth distribution.

Original: This is also reﬂected in the generated samples in Fig.
Revised: This is also reﬂected in the generated samples in Fig.

Original: 9, where samples generated by the kPF operator are clearly more aligned with the ground truth distribution.
Revised: 9, where samples generated by the kPF operator are clearly more aligned with the ground truth distribution.

Original: [Figure]
Revised: [Figure]

Original: C Effect of γ on Sample Quality
Revised: C Effect of γ on Sample Quality

Original: In the sampling stage, our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by taking the weighted Fr´echet mean of the top γ neighbors among the training samples.
Revised: In the sampling stage, our proposed method ﬁnds the approximate preimage of the transferred kernel embeddings by taking the weighted Fr´echet mean of the top γ neighbors among the training samples.

Original: The choice of γ therefore inﬂuences the quality of generation.
Revised: The choice of γ therefore inﬂuences the quality of generation.

Original: From Figure 10, we can observe that, in general, FID worsens as γ increases.
Revised: From Figure 10, we can observe that, in general, FID worsens as γ increases.

Original: This observation aligns with our intuition of preserving only the local similarities represented by the kernel, and similar ideas have been previously used in the literature (Hastie et al., 2001; Kwok &
Revised: This observation aligns with our intuition of preserving only the local similarities represented by the kernel, and similar ideas have been previously used in the literature (Hastie et al., 2001; Kwok &

Original: Tsang, 2004).
Revised: Tsang, 2004).

Original: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1, generated samples will just be reconstructions of training samples).
Revised: However, signiﬁcantly decreasing γ leads to the undesirable result where the generator merely generates the training samples (in the extreme case where γ = 1, generated samples will just be reconstructions of training samples).

Original: Therefore, in our experiments, we choose γ = 5 to achieve a balance between generation quality and the distance to training samples.
Revised: Therefore, in our experiments, we choose γ = 5 to achieve a balance between generation quality and the distance to training samples.

Original: D Weighted Fr´echet Mean on the Hypersphere
Revised: D Weighted Fr´echet Mean on the Hypersphere

Original: While the weighted Fr´echet Mean in Euclidean space can be computed in closed-form as a weighted arithmetic mean (as in Eq. 10), on the hypersphere there is no known closed-form solution.
Revised: While the weighted Fr´echet Mean in Euclidean space can be computed in closed-form as a weighted arithmetic mean (as in Eq. 10), on the hypersphere there is no known closed-form solution.

Original: Thus, we adopt the iterative algorithm in (Chakraborty & Vemuri, 2015) for an approximate solution given data points
Revised: Thus, we adopt the iterative algorithm in (Chakraborty & Vemuri, 2015) for an approximate solution given data points

Original: X = { x 1 . . .
Revised: X = { x 1 . . .

Original: x γ } and weight vector s :
Revised: x γ } and weight vector s :

Original: [Equation]
Revised: [Equation]

Original: where, v =
Revised: where, v =

Original: [Equation]
Revised: [Equation]

Original: ( X
Revised: ( X

Original: [Equation]
Revised: [Equation]

Original: − M i cos( θ )) , θ = arccos
Revised: − M i cos( θ )) , θ = arccos

Original: [Equation]
Revised: [Equation]

Original: M ti X i +1 ).
Revised: M ti X i +1 ).

Original: This algorithm iterates through the data points once, yielding a complexity of only O ( γd ), where d is the dimension of X .
Revised: This algorithm iterates through the data points once, yielding a complexity of only O ( γd ), where d is the dimension of X .

Original: Under the prescribed iteration, M n converges asymptotically to the true Fr´echet mean for ﬁnite data points.
Revised: Under the prescribed iteration, M n converges asymptotically to the true Fr´echet mean for ﬁnite data points.

Original: We refer the readers to (Chakraborty & Vemuri, 2015) for further details.
Revised: We refer the readers to (Chakraborty & Vemuri, 2015) for further details.

Original: E Fast approximation of Moore-Penrose inverse
Revised: E Fast approximation of Moore-Penrose inverse

Original: When computing the inverted kernel matrix K inv in Algo. 1, conventional approaches typically performs SVD or Cholesky decomposition.
Revised: When computing the inverted kernel matrix K inv in Algo. 1, conventional approaches typically performs SVD or Cholesky decomposition.

Original: Both procedures are hard to parallelize, and therefore, can be slow when K is large.
Revised: Both procedures are hard to parallelize, and therefore, can be slow when K is large.

Original: Alternatively, we can utilize an iterative procedure proposed in Razavi et al.
Revised: Alternatively, we can utilize an iterative procedure proposed in Razavi et al.

Original: (2014) to approximate the Moore-Penrose inverse.
Revised: (2014) to approximate the Moore-Penrose inverse.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since this iterative procedure mostly involves matrix multiplications, it can be eﬃciently parallelized and implemented on GPU.
Revised: Since this iterative procedure mostly involves matrix multiplications, it can be eﬃciently parallelized and implemented on GPU.

Original: The same procedure has also seen success in approximating large self-attention matrices in language modeling (Xiong et al., 2021).
Revised: The same procedure has also seen success in approximating large self-attention matrices in language modeling (Xiong et al., 2021).

Original: For the
Revised: For the

Original: NVAE experiment, we run this iteration for 10 steps and use K inv =
Revised: NVAE experiment, we run this iteration for 10 steps and use K inv =

Original: Z 10 .
Revised: Z 10 .

Original: F Nystrom Approximation of kPF
Revised: F Nystrom Approximation of kPF

Original: Due to the need to store and compute a kernel matrix inverse ( K + λnI ) − 1 or K † , the memory and computational complexity of kPF is at least O ( n 2 ) and O ( n 3 ), respectively.
Revised: Due to the need to store and compute a kernel matrix inverse ( K + λnI ) − 1 or K † , the memory and computational complexity of kPF is at least O ( n 2 ) and O ( n 3 ), respectively.

Original: The sup-quadratic complexity hinders the use of kPF on extremely large datasets.
Revised: The sup-quadratic complexity hinders the use of kPF on extremely large datasets.

Original: In our experiments, we already adopted a simple subsampling strategy which randomly select 10k training samples from each dataset ( ≤ 50k samples) to ﬁt our hardware conﬁguration which works well.
Revised: In our experiments, we already adopted a simple subsampling strategy which randomly select 10k training samples from each dataset ( ≤ 50k samples) to ﬁt our hardware conﬁguration which works well.

Original: But for larger datasets with potentially more modes, a larger set of subsamples must be considered, and in those cases kPF may not be suitable for commodity/aﬀordable hardware.
Revised: But for larger datasets with potentially more modes, a larger set of subsamples must be considered, and in those cases kPF may not be suitable for commodity/aﬀordable hardware.

Original: In order to overcome this problem, we can combine kPF with conventional kernel approximation methods such as the Nystr¨om method (Williams & Seeger, 2001).
Revised: In order to overcome this problem, we can combine kPF with conventional kernel approximation methods such as the Nystr¨om method (Williams & Seeger, 2001).

Original: Let ( X (cid:63) , Z (cid:63) ) be a size v subset of the training set (which we refer to as the landmark points ) and (Ψ (cid:63) , Φ (cid:63) ) be their corresponding kernel feature maps.
Revised: Let ( X (cid:63) , Z (cid:63) ) be a size v subset of the training set (which we refer to as the landmark points ) and (Ψ (cid:63) , Φ (cid:63) ) be their corresponding kernel feature maps.

Original: The weighting coeﬃcients s for each prior sample z ∗ ∼ Z derived in Alg.
Revised: The weighting coeﬃcients s for each prior sample z ∗ ∼ Z derived in Alg.

Original: 1 can be approximated by
Revised: 1 can be approximated by

Original: [Equation]
Revised: [Equation]

Original: , · ) (18) where L Ψ (cid:63) =
Revised: , · ) (18) where L Ψ (cid:63) =

Original: Ψ (cid:62) Ψ (cid:63) ∈ R n × v , W Ψ (cid:63)
Revised: Ψ (cid:62) Ψ (cid:63) ∈ R n × v , W Ψ (cid:63)

Original: =
Revised: =

Original: Ψ (cid:62) (cid:63) Ψ (
Revised: Ψ (cid:62) (cid:63) Ψ (

Original: cid:63) ∈ R v × v , K Φ (cid:63) =
Revised: cid:63) ∈ R v × v , K Φ (cid:63) =

Original: Φ (cid:62) Φ (cid:63) ∈ R n × v , W Φ (cid:63) =
Revised: Φ (cid:62) Φ (cid:63) ∈ R n × v , W Φ (cid:63) =

Original: Φ (cid:62) (cid:63) Φ (cid:63) ∈
Revised: Φ (cid:62) (cid:63) Φ (cid:63) ∈

Original: R v × v , and the last identity is due to applying the Woodbury formula on ( K Φ (cid:63) W † Φ (cid:63) K (cid:62) Φ (cid:63) + λnI ) † .
Revised: R v × v , and the last identity is due to applying the Woodbury formula on ( K Φ (cid:63) W † Φ (cid:63) K (cid:62) Φ (cid:63) + λnI ) † .

Original: Assuming v (cid:28) n , the memory complexity is reduced to O ( nv ) and the computation complexity to O ( nv 2 ).
Revised: Assuming v (cid:28) n , the memory complexity is reduced to O ( nv ) and the computation complexity to O ( nv 2 ).

Original: We empirically evaluated the Nystr¨om-approximated kPF on the CelebA experiment and present the result in Tab. 4.
Revised: We empirically evaluated the Nystr¨om-approximated kPF on the CelebA experiment and present the result in Tab. 4.

Original: It can be observed that when v is suﬃciently large, the performance of Nystr¨om approximated kPFs is as good as the ones using the full kernel matrices.
Revised: It can be observed that when v is suﬃciently large, the performance of Nystr¨om approximated kPFs is as good as the ones using the full kernel matrices.

Original: [Table]
Revised: [Table]

Original: of performance
Revised: of performance

Original: while delivering signiﬁcant eﬃciency gain.
Revised: while delivering signiﬁcant eﬃciency gain.

Original: G Does kPF Memorize Training Data?
Revised: G Does kPF Memorize Training Data?

Original: Since in kPF, samples are generated by linearly interpolating between training samples, it is natural to wonder whether it ‘fools’ the metrics by simply replicating training samples.
Revised: Since in kPF, samples are generated by linearly interpolating between training samples, it is natural to wonder whether it ‘fools’ the metrics by simply replicating training samples.

Original: For comparison, we consider an alternative scheme that generates data through direct manipulation of the training data, namely Kernel Density Estimation (KDE).
Revised: For comparison, we consider an alternative scheme that generates data through direct manipulation of the training data, namely Kernel Density Estimation (KDE).

Original: We ﬁt KDEs by varying noise levels σ and compare their FIDs and nearest samples in the latent space to kPF in Fig.
Revised: We ﬁt KDEs by varying noise levels σ and compare their FIDs and nearest samples in the latent space to kPF in Fig.

Original: We observe that, although KDE can reach very low FIDs when σ is small, almost all new samples closely resemble some instance in the training set, which is a clear indication of memorization.
Revised: We observe that, although KDE can reach very low FIDs when σ is small, almost all new samples closely resemble some instance in the training set, which is a clear indication of memorization.

Original: In contrast, kPF can generate diverse samples that do not simply replicate the observed data.
Revised: In contrast, kPF can generate diverse samples that do not simply replicate the observed data.

Original: [Figure]
Revised: [Figure]

Original: H Assessing Sample Diversity
Revised: H Assessing Sample Diversity

Original: Although FID is one of the most frequently used measures for assessing sample quality of generative models, certain diversity considerations, such as mode collapse, may not be conveniently deduced from it (Sajjadi et al., 2018).
Revised: Although FID is one of the most frequently used measures for assessing sample quality of generative models, certain diversity considerations, such as mode collapse, may not be conveniently deduced from it (Sajjadi et al., 2018).

Original: To enable explicit examination of generative models with respect to both accuracy (i
Revised: To enable explicit examination of generative models with respect to both accuracy (i

Original: [Figure]
Revised: [Figure]

Original: e., generating samples within the support of the data distribution) and diversity (i
Revised: e., generating samples within the support of the data distribution) and diversity (i

Original: [Figure]
Revised: [Figure]

Original: e., covering the support of the data distribution as much as possible)
Revised: e., covering the support of the data distribution as much as possible)

Original: [Figure]
Revised: [Figure]

Original: Sajjadi et al.
Revised: Sajjadi et al.

Original: (2018) proposed an approach to evaluate generative models with generalized deﬁnitions of precision and recall between distributions.
Revised: (2018) proposed an approach to evaluate generative models with generalized deﬁnitions of precision and recall between distributions.

Original: Quality of generation can then be assessed by evaluating the PRD curve
Revised: Quality of generation can then be assessed by evaluating the PRD curve

Original: [Figure]
Revised: [Figure]

Original: which depicts the trade-oﬀs between accuracy (precision) and diversity (recall)
Revised: which depicts the trade-oﬀs between accuracy (precision) and diversity (recall)

Original: We present the PRD curves in Fig.
Revised: We present the PRD curves in Fig.

Original: [Figure]
Revised: [Figure]

Original: The observations align with our results in Tab. 2 and kPF performs competitively in both accuracy and sample diversity
Revised: The observations align with our results in Tab. 2 and kPF performs competitively in both accuracy and sample diversity

Original: [Figure]
Revised: [Figure]

Original: CIFARCelebA
Revised: CIFARCelebA

Original: [Figure]
Revised: [Figure]

Original: I Exploring Kernel Configurations
Revised: I Exploring Kernel Configurations

Original: To investigate the implication of kernel choices on generation quality
Revised: To investigate the implication of kernel choices on generation quality

Original: [Table]
Revised: [Table]

Original: we tested 25 diﬀerent kernel conﬁgurations for CelebA generation (results are presented in Tab. 5).
Revised: we tested 25 diﬀerent kernel conﬁgurations for CelebA generation (results are presented in Tab. 5).

Original: For RBF kernels used in the CelebA experiments of the main text, we use a bandwidth of σ in = (cid:112) 2 |Z| / 8 ≈ 0 .when used as input kernel and σ out ≈ 0 . 34, and we adopt the same notation here.
Revised: For RBF kernels used in the CelebA experiments of the main text, we use a bandwidth of σ in = (cid:112) 2 |Z| / 8 ≈ 0 .when used as input kernel and σ out ≈ 0 . 34, and we adopt the same notation here.

Original: [Table]
Revised: [Table]

Original: It can be seen that kernel conﬁgurations and parameters indeed has a non-trivial impact on the generation quality, with NTK-kPF being the most robust to the choice of parameters.
Revised: It can be seen that kernel conﬁgurations and parameters indeed has a non-trivial impact on the generation quality, with NTK-kPF being the most robust to the choice of parameters.

Original: This aligns with our previous observations and oﬀers some support for using NTK as an input kernel despite the additional compute cost.
Revised: This aligns with our previous observations and oﬀers some support for using NTK as an input kernel despite the additional compute cost.

Original: J Experimental Details
Revised: J Experimental Details

Original: In this section, we provide the detailed speciﬁcations for all of our experiments.
Revised: In this section, we provide the detailed speciﬁcations for all of our experiments.

Original: We have also provided our code in the supplemental material.
Revised: We have also provided our code in the supplemental material.

Original: J. Density Estimation on Toy Densities
Revised: J. Density Estimation on Toy Densities

Original: We generated 10000 samples from each of the toy densities to learn the kPF operator.
Revised: We generated 10000 samples from each of the toy densities to learn the kPF operator.

Original: The input kernel k is a ReLU-activated NTK corresponding to a fully-connected network with depth L = 4 and width w = 10000 at each layer, and the output kernel l is a Gaussian kernel.
Revised: The input kernel k is a ReLU-activated NTK corresponding to a fully-connected network with depth L = 4 and width w = 10000 at each layer, and the output kernel l is a Gaussian kernel.

Original: Unless speciﬁed otherwise, we always uses a Gaussian kernel as the output kernel for the remainder of this appendix.
Revised: Unless speciﬁed otherwise, we always uses a Gaussian kernel as the output kernel for the remainder of this appendix.

Original: The bandwidth of the output kernel was adjusted separately for density estimation and sampling for the purpose of demonstration.
Revised: The bandwidth of the output kernel was adjusted separately for density estimation and sampling for the purpose of demonstration.

Original: For comparison, we also ﬁt/estimate a 10-component GMM and a Glow model with 50 coupling layers, where each of them were trained until convergence.
Revised: For comparison, we also ﬁt/estimate a 10-component GMM and a Glow model with 50 coupling layers, where each of them were trained until convergence.

Original: J. Image Generation with Computer Vision Datasets
Revised: J. Image Generation with Computer Vision Datasets

Original: To generate results in Tab. 2 and Tab. 3, we ﬁrst trained an autoencoder for each dataset following the model setup in (Ghosh et al., 2020), which uses a modiﬁed Wasserstein autoencoder (Tolstikhin et al., 2018) architecture with batch normalization.
Revised: To generate results in Tab. 2 and Tab. 3, we ﬁrst trained an autoencoder for each dataset following the model setup in (Ghosh et al., 2020), which uses a modiﬁed Wasserstein autoencoder (Tolstikhin et al., 2018) architecture with batch normalization.

Original: Additionally, we applied spectral normalization on both the encoder and the decoder, following (Ghosh et al., 2020), to obtain a regularized autoencoder.
Revised: Additionally, we applied spectral normalization on both the encoder and the decoder, following (Ghosh et al., 2020), to obtain a regularized autoencoder.

Original: The latent representations were projected onto a hypersphere before decoding to image space.
Revised: The latent representations were projected onto a hypersphere before decoding to image space.

Original: We trained the models on two NVIDIA
Revised: We trained the models on two NVIDIA

Original: GTX 1080TI GPUs.
Revised: GTX 1080TI GPUs.

Original: A detailed model speciﬁcation is provided below in Table 6.
Revised: A detailed model speciﬁcation is provided below in Table 6.

Original: We used an NTK with L = 8 and w = 10000 as the input kernel k (i.e. the embedding kernel of p Z ) for NTK-kPF, and a Gaussian kernel with bandwidth σ in = (cid:112) 2 |Z| / 8 for
Revised: We used an NTK with L = 8 and w = 10000 as the input kernel k (i.e. the embedding kernel of p Z ) for NTK-kPF, and a Gaussian kernel with bandwidth σ in = (cid:112) 2 |Z| / 8 for

Original: RBF-kPF.
Revised: RBF-kPF.

Original: The bandwidth for the output Gaussian kernel is selected by grid search over { 2 − i ∗ σ data | i ∈
Revised: The bandwidth for the output Gaussian kernel is selected by grid search over { 2 − i ∗ σ data | i ∈

Original: [8] } , where σ data is the empirical data standard deviation, based on crossvalidation of a degree 3 polynomial kernel MMD between the sampled and the ground-truth latent points.
Revised: [8] } , where σ data is the empirical data standard deviation, based on crossvalidation of a degree 3 polynomial kernel MMD between the sampled and the ground-truth latent points.

Original: Further, to mitigate the deterioration of performance of kernel methods in a high-dimensional setting due to the curse of dimensionality (Evangelista et al., 2006), in practice, we model Z as a space with fewer dimensions than the input space X .
Revised: Further, to mitigate the deterioration of performance of kernel methods in a high-dimensional setting due to the curse of dimensionality (Evangelista et al., 2006), in practice, we model Z as a space with fewer dimensions than the input space X .

Original: As a rule of the thumb, we choose Z such that |Z| =
Revised: As a rule of the thumb, we choose Z such that |Z| =

Original: |X| / 4.
Revised: |X| / 4.

Original: To generate images from kPF learned on NVAE latent space, we used the pre-trained checkpoints provided in (Vahdat & Kautz, 2020) to obtain the latent embeddings for
Revised: To generate images from kPF learned on NVAE latent space, we used the pre-trained checkpoints provided in (Vahdat & Kautz, 2020) to obtain the latent embeddings for

Original: FFHQ images.
Revised: FFHQ images.

Original: We then construct the kPF from the concatenated latent space of the lowest resolution (8 × 8).
Revised: We then construct the kPF from the concatenated latent space of the lowest resolution (8 × 8).

Original: During sampling, prior samples at those resolutions are replace by the kPF samples, while for other resolutions samples remain generated from inferred Gaussian distributions.
Revised: During sampling, prior samples at those resolutions are replace by the kPF samples, while for other resolutions samples remain generated from inferred Gaussian distributions.

Original: The batchnorm statistics were readjusted for 500 iterations following (Vahdat & Kautz, 2020).
Revised: The batchnorm statistics were readjusted for 500 iterations following (Vahdat & Kautz, 2020).

Original: We use rbf kernels as input and output kernels, with bandwidths σ k , σ l chosen by the median heuristic ( ∼ 100 for input and ∼ 70 for output in our experiments).
Revised: We use rbf kernels as input and output kernels, with bandwidths σ k , σ l chosen by the median heuristic ( ∼ 100 for input and ∼ 70 for output in our experiments).

Original: J. Image Generation for Brain Images
Revised: J. Image Generation for Brain Images

Original: For the high-resolution brain imaging dataset, we used a custom version of ResNet (He et al., 2016) with 3D convolutions.
Revised: For the high-resolution brain imaging dataset, we used a custom version of ResNet (He et al., 2016) with 3D convolutions.

Original: The detailed architecture is shown in Fig.
Revised: The detailed architecture is shown in Fig.

Original: Due to the large size of the data, we trained the model on 4 NVIDIA Tesla V100 GPUs.
Revised: Due to the large size of the data, we trained the model on 4 NVIDIA Tesla V100 GPUs.

Original: Mandatory ADNI statement regarding data use.
Revised: Mandatory ADNI statement regarding data use.

Original: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu).
Revised: Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu).

Original: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.
Revised: As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report.

Original: A complete listing of ADNI investigators can be found in the
Revised: A complete listing of ADNI investigators can be found in the

Original: ADNI Acknowledgement List.
Revised: ADNI Acknowledgement List.

Original: The T1 MR brain dataset we utilize consists of images from 184 subjects diagnosed with
Revised: The T1 MR brain dataset we utilize consists of images from 184 subjects diagnosed with

Original: Alzheimers’s disease and 292 healthy controls/ normal subjects.
Revised: Alzheimers’s disease and 292 healthy controls/ normal subjects.

Original: Images were ﬁrst coregistered
Revised: Images were ﬁrst coregistered

Original: [Table] [Table]
Revised: [Table] [Table]

Original: to a MNI template and segmented to preserve only the white matter and grey matter.
Revised: to a MNI template and segmented to preserve only the white matter and grey matter.

Original: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1].
Revised: Then, all images were resliced and resized to 160 × 196 × 160 and rescaled to the range of [ − 1 , 1].

Original: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images.
Revised: Voxel-based morphometry (VBM) was used to obtain the p -value map of data and generated images.

Original: K More Samples
Revised: K More Samples

Original: In this section we present additional uncurated set of samples on MNIST, CIFAR-10, CelebA based on pre-trained SRAE and FFHQ based on NVAE.
Revised: In this section we present additional uncurated set of samples on MNIST, CIFAR-10, CelebA based on pre-trained SRAE and FFHQ based on NVAE.

Original: From the ﬁgures, it can be seen that kPF produces consistent and diverse samples, often better in quality than the alternatives.
Revised: From the ﬁgures, it can be seen that kPF produces consistent and diverse samples, often better in quality than the alternatives.