Original: D EEP G ENERATIVE M ODELING ON L IMITED D ATA WITH R EGULARIZATION BY N ONTRANSFERABLE P RE - TRAINED M ODELS
Revised: D EEP G ENERATIVE M ODELING ON L IMITED D ATA WITH R EGULARIZATION BY N ONTRANSFERABLE P RE - TRAINED M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep generative models (DGMs) are data-eager because learning a complex model on limited data suffers from a large variance and easily overfits.
Revised: Deep generative models (DGMs) are data-eager because learning a complex model on limited data suffers from a large variance and easily overfits.

Original: Inspired by the classical perspective of the bias-variance tradeoff , we propose regularized deep generative model (Reg-DGM), which leverages a nontransferable pre-trained model to reduce the variance of generative modeling with limited data.
Revised: Inspired by the classical perspective of the bias-variance tradeoff , we propose regularized deep generative model (Reg-DGM), which leverages a nontransferable pre-trained model to reduce the variance of generative modeling with limited data.

Original: Formally,
Revised: Formally,

Original: Reg-DGM optimizes a weighted sum of a certain divergence and the expectation of an energy function, where the divergence is between the data and the model distributions, and the energy function is defined by the pre-trained model w.r.t.
Revised: Reg-DGM optimizes a weighted sum of a certain divergence and the expectation of an energy function, where the divergence is between the data and the model distributions, and the energy function is defined by the pre-trained model w.r.t.

Original: the model distribution.
Revised: the model distribution.

Original: We analyze a simple yet representative Gaussian-fitting case to demonstrate how the weighting hyperparameter trades off the bias and the variance.
Revised: We analyze a simple yet representative Gaussian-fitting case to demonstrate how the weighting hyperparameter trades off the bias and the variance.

Original: Theoretically, we characterize the existence and the uniqueness of the global minimum of Reg-DGM in a non-parametric setting and prove its convergence with neural networks trained by gradient-based methods.
Revised: Theoretically, we characterize the existence and the uniqueness of the global minimum of Reg-DGM in a non-parametric setting and prove its convergence with neural networks trained by gradient-based methods.

Original: Empirically, with various pre-trained feature extractors and a data-dependent energy function, RegDGM consistently improves the generation performance of strong DGMs with limited data and achieves competitive results to the state-of-the-art methods
Revised: Empirically, with various pretrained feature extractors and a data-dependent energy function, Reg-DGM consistently improves the generation performance of strong DGMs with limited data and achieves competitive results to the state-of-the-art methods. Our implementation is available at https://github.com/ML-GSAI /Reg-ADA-APA .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep generative models (DGMs) (Kingma & Welling, 2013; Goodfellow et al., 2014; SohlDickstein et al., 2015; Van den Oord et al., 2016; Dinh et al., 2016; Hinton & Salakhutdinov, 2006) employ neural networks to capture the underlying distribution of high-dimensional data and find applications in various learning tasks (Kingma et al., 2014;
Revised: Deep generative models (DGMs) (Kingma & Welling, 2013; Goodfellow et al., 2014; SohlDickstein et al., 2015; Van den Oord et al., 2016; Dinh et al., 2016; Hinton & Salakhutdinov, 2006) employ neural networks to capture the underlying distribution of high-dimensional data and find applications in various learning tasks (Kingma et al., 2014;

Original: Zhu et al., 2017; Razavi et al., 2019;
Revised: Zhu et al., 2017; Razavi et al., 2019;

Original: Ramesh et al., 2021; 2022; Ho et al., 2022).
Revised: Ramesh et al., 2021; 2022; Ho et al., 2022).

Original: Such models are often data-eager (Li et al., 2021;
Revised: Such models are often data-eager (Li et al., 2021;

Original: Wang et al., 2018) due to the presence of complex function classes.
Revised: Wang et al., 2018) due to the presence of complex function classes.

Original: Recent work (Karras et al., 2020a) found that the classical variants of generative adversarial networks (GANs) (Goodfellow et al., 2014; Karras et al., 2020b) produce poor samples with limited data, which is shared by other
Revised: Recent work (Karras et al., 2020a) found that the classical variants of generative adversarial networks (GANs) (Goodfellow et al., 2014; Karras et al., 2020b) produce poor samples with limited data, which is shared by other

Original: DGMs in principle.
Revised: DGMs in principle.

Original: Thus, improving the sample efficiency is a common challenge for DGMs.
Revised: Thus, improving the sample efficiency is a common challenge for DGMs.

Original: The root cause of the problem is that learning a model in a complex class on limited data suffers from a large variance and easily overfits the training data (Mohri et al., 2018).
Revised: The root cause of the problem is that learning a model in a complex class on limited data suffers from a large variance and easily overfits the training data (Mohri et al., 2018).

Original: To relieve the problem, previous work either employed sophisticated data augmentation strategies (Zhao et al., 2020a;
Revised: To relieve the problem, previous work either employed sophisticated data augmentation strategies (Zhao et al., 2020a;

Original: Karras et al., 2020a; Jiang et al., 2021), or designed new losses for the discriminator in GANs (Cui et al., 2021;
Revised: Karras et al., 2020a; Jiang et al., 2021), or designed new losses for the discriminator in GANs (Cui et al., 2021;

Original: Yang et al., 2021), or transferred a pre-trained DGM (Wang et al., 2018; Noguchi &
Revised: Yang et al., 2021), or transferred a pre-trained DGM (Wang et al., 2018; Noguchi &

Original: Harada, 2019; Mo et al., 2020).
Revised: Harada, 2019; Mo et al., 2020).

Original: Although not pointed out in the literature to our knowledge, prior work can be understood as reducing the variance of the estimate implicitly (Mohri et al., 2018).
Revised: Although not pointed out in the literature to our knowledge, prior work can be understood as reducing the variance of the estimate implicitly (Mohri et al., 2018).

Original: In this perspective, we propose a complementary framework, named regularized deep generative model (Reg-DGM) , which employs a pre-trained model as regularization to achieve a better bias-variance tradeoff when training a DGM with limited data.
Revised: In this perspective, we propose a complementary framework, named regularized deep generative model (Reg-DGM) , which employs a pre-trained model as regularization to achieve a better bias-variance tradeoff when training a DGM with limited data.

Original: In Sec. 2, we formulate the objective function of Reg-DGM as the sum of a certain divergence and a regularization term weighted by a hyperparameter.
Revised: In Sec. 2, we formulate the objective function of Reg-DGM as the sum of a certain divergence and a regularization term weighted by a hyperparameter.

Original: The divergence is between the data distribution and model distribution, and the regularization term can be understood as the negative expected loglikelihood of an energy-based model, whose energy function is defined by a pre-trained model, w.r.t.
Revised: The divergence is between the data distribution and model distribution, and the regularization term can be understood as the negative expected loglikelihood of an energy-based model, whose energy function is defined by a pre-trained model, w.r.t.

Original: the model distribution.
Revised: the model distribution.

Original: Intuitively, with an appropriate weighting hyperparameter, Reg-DGM balances between the data distribution and the pre-trained model to achieve a better bias-variance tradeoff with limited data, as validated by a simple yet prototypical Gaussian-fitting example.
Revised: Intuitively, with an appropriate weighting hyperparameter, Reg-DGM balances between the data distribution and the pre-trained model to achieve a better bias-variance tradeoff with limited data, as validated by a simple yet prototypical Gaussian-fitting example.

Original: In Sec. 3, we characterize the optimization behavior of Reg-DGM in both non-parametric and parametric settings under mild regularity conditions.
Revised: In Sec. 3, we characterize the optimization behavior of Reg-DGM in both non-parametric and parametric settings under mild regularity conditions.

Original: On one hand, we prove the existence and the uniqueness of the global minimum of the regularized optimization problem with the Kullback–Leibler (KL) and Jensen–Shannon (JS) divergence in the non-parametric setting.
Revised: On one hand, we prove the existence and the uniqueness of the global minimum of the regularized optimization problem with the Kullback–Leibler (KL) and Jensen–Shannon (JS) divergence in the non-parametric setting.

Original: On the other hand, we prove that, parameterized by a standard neural network architecture, Reg-DGM converges with a high probability to a global (or local) optimum trained by gradient-based methods.
Revised: On the other hand, we prove that, parameterized by a standard neural network architecture, Reg-DGM converges with a high probability to a global (or local) optimum trained by gradient-based methods.

Original: In Sec. 4, we specify the components in Reg-DGM.
Revised: In Sec. 4, we specify the components in Reg-DGM.

Original: We employ strong variants of GANs (Karras et al., 2020b;a; Jiang et al., 2021) as the base DGM for broader interests.
Revised: We employ strong variants of GANs (Karras et al., 2020b;a; Jiang et al., 2021) as the base DGM for broader interests.

Original: We consider a nontransferable setting where the pre-trained model does not necessarily have to be a generative model.
Revised: We consider a nontransferable setting where the pre-trained model does not necessarily have to be a generative model.

Original: Indeed, we employ several feature extractors, which are trained for non-generative tasks such as image classification, representation learning, and face recognition.
Revised: Indeed, we employ several feature extractors, which are trained for non-generative tasks such as image classification, representation learning, and face recognition.

Original: Notably, such models cannot be directly used in the fine-tuning approaches (Wang et al., 2018; Mo et al., 2020).
Revised: Notably, such models cannot be directly used in the fine-tuning approaches (Wang et al., 2018; Mo et al., 2020).

Original: With a comprehensive ablation study, we define our default energy function as the expected mean squared error between the features of the generated samples and the training data.
Revised: With a comprehensive ablation study, we define our default energy function as the expected mean squared error between the features of the generated samples and the training data.

Original: Such an energy not only fits our theoretical analyses, but also results in consistent and significant improvements over baselines in all settings.
Revised: Such an energy not only fits our theoretical analyses, but also results in consistent and significant improvements over baselines in all settings.

Original: In Sec. 5, we present experiments on several benchmarks, including the FFHQ (Karras et al., 2019),
Revised: In Sec. 5, we present experiments on several benchmarks, including the FFHQ (Karras et al., 2019),

Original: LSUN CAT (Yu et al., 2015), and CIFAR-10 (Krizhevsky et al., 2009) datasets.
Revised: LSUN CAT (Yu et al., 2015), and CIFAR-10 (Krizhevsky et al., 2009) datasets.

Original: We compare RegDGM with a large family of methods, including the base DGMs (Karras et al., 2020b;a; Jiang et al., 2021), the transfer-based approaches (Wang et al., 2018; Mo et al., 2020), the augmentation-based methods (Zhao et al., 2020a) and others (Cui et al., 2021; Yang et al., 2021).
Revised: We compare RegDGM with a large family of methods, including the base DGMs (Karras et al., 2020b;a; Jiang et al., 2021), the transfer-based approaches (Wang et al., 2018; Mo et al., 2020), the augmentation-based methods (Zhao et al., 2020a) and others (Cui et al., 2021; Yang et al., 2021).

Original: With a classifier pretrained on ImageNet (Deng et al., 2009), or an image encoder pre-trained on the CLIP dataset (Radford et al., 2021), or a face recognizer pre-trained on VGGFace2 (Cao et al., 2018), Reg-DGM consistently improves strong DGMs under commonly used performance measures with limited data and achieves competitive results to the state-of-the-art methods.
Revised: With a classifier pretrained on ImageNet (Deng et al., 2009), or an image encoder pre-trained on the CLIP dataset (Radford et al., 2021), or a face recognizer pre-trained on VGGFace2 (Cao et al., 2018), Reg-DGM consistently improves strong DGMs under commonly used performance measures with limited data and achieves competitive results to the state-of-the-art methods.

Original: Our results demonstrate that RegDGM can achieve a good bias-variance tradeoff in practice, which supports our motivation.
Revised: Our results demonstrate that RegDGM can achieve a good bias-variance tradeoff in practice, which supports our motivation.

Original: In Sec. 6, we present the related work.
Revised: In Sec. 6, we present the related work.

Original: In Sec. 7, we conclude the paper and discuss limitations.
Revised: In Sec. 7, we conclude the paper and discuss limitations.

Original:  M ETHOD
Revised:  M ETHOD

Original: The goal of generative modeling is to learn a model distribution p g (implicitly or explicitly) from a training set S
Revised: The goal of generative modeling is to learn a model distribution p g (implicitly or explicitly) from a training set S

Original: = { x i } mi =1 of size m on a sample space X .
Revised: = { x i } mi =1 of size m on a sample space X .

Original: The elements in S are assumed to be drawn i.i.d.
Revised: The elements in S are assumed to be drawn i.i.d.

Original: according to an unknown data distribution p d ∈ P X , where P X is the set of all valid distributions over X .
Revised: according to an unknown data distribution p d ∈ P X , where P X is the set of all valid distributions over X .

Original: A general formulation for learning generative models is minimizing a certain statistical divergence D ( ·||· ) between the two distributions as follows:
Revised: A general formulation for learning generative models is minimizing a certain statistical divergence D ( ·||· ) between the two distributions as follows:

Original: [Equation]
Revised: [Equation]

Original: where H ⊂ P X is the hypothesis class, for instance, a set of distributions defined by neural networks in a deep generative model (DGM).
Revised: where H ⊂ P X is the hypothesis class, for instance, a set of distributions defined by neural networks in a deep generative model (DGM).

Original: Notably, the divergence in Eq.
Revised: Notably, the divergence in Eq.

Original: (1) is estimated by the Monte
Revised: (1) is estimated by the Monte

Original: Carlo method over the training set S and its solution has a small bias if not zero.
Revised: Carlo method over the training set S and its solution has a small bias if not zero.

Original: However, learning a DGM with limited data is challenging because solving the problem in Eq.
Revised: However, learning a DGM with limited data is challenging because solving the problem in Eq.

Original: (1) with a small sample size m essentially suffers from a large variance and probably overfits (Mohri et al., 2018).
Revised: (1) with a small sample size m essentially suffers from a large variance and probably overfits (Mohri et al., 2018).

Original: Inspired by the classical perspective of the bias-variance tradeoff , we propose to leverage an external model f pre-trained on a related and large dataset (e.g., ImageNet) as a data-dependent regularization to reduce the variance of training a DGM on limited data (e.g., CIFAR-10), which is complementary to prior work (see more details in Sec. 6).
Revised: Inspired by the classical perspective of the bias-variance tradeoff , we propose to leverage an external model f pre-trained on a related and large dataset (e.g., ImageNet) as a data-dependent regularization to reduce the variance of training a DGM on limited data (e.g., CIFAR-10), which is [Figure] [Figure] [Figure]

Original:  In particular, given a pre-trained f , we first introduce an energy function (LeCun et al., 2006)
Revised: [Figure] [Figure] complementary to prior work (see more details in Sec. 6). In particular, given a pre-trained f , we first introduce an energy function (LeCun et al., 2006)

Original: E f : X → R that satisfies mild regularity conditions (see Assumption A.1) and then define a probabilistic energy-based model (EBM) p f
Revised: E f : X → R that satisfies mild regularity conditions (see Assumption A.1) and then define a probabilistic energy-based model (EBM) p f

Original: as p f ( x ) ∝ exp( −E f ( x )) .
Revised: as p f ( x ) ∝ exp( −E f ( x )) .

Original: We can treat p f as a special estimate of p d if f is pre-trained on a dataset closely related to p d .
Revised: We can treat p f as a special estimate of p d if f is pre-trained on a dataset closely related to p d .

Original: In this perspective, p f has a potentially large bias yet a variance of zero.
Revised: In this perspective, p f has a potentially large bias yet a variance of zero.

Original: To trade off the bias and variance, we simply optimize a weighted sum of the statistical divergence as in Eq.
Revised: To trade off the bias and variance, we simply optimize a weighted sum of the statistical divergence as in Eq.

Original: (1) and the expected log-likelihood of the EBM as follows:
Revised: (1) and the expected log-likelihood of the EBM as follows:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Figure] [Figure] [Figure] where λ > 0 is the weighting hyperparameter balancing the two terms.
Revised: where λ > 0 is the weighting hyperparameter balancing the two terms.

Original: The first one encourages p g to fit the data as in the original DGM and the second one encourages p g to produce samples with a high likelihood evaluated by the EBM p f .
Revised: The first one encourages p g to fit the data as in the original DGM and the second one encourages p g to produce samples with a high likelihood evaluated by the EBM p f .

Original: Naturally, a properly selected λ can hopefully achieve a better bias-variance tradeoff.
Revised: Naturally, a properly selected λ can hopefully achieve a better bias-variance tradeoff.

Original: We refer to our approach as regularized deep generative model (Reg-DGM) when H consists of distributions defined by neural networks.
Revised: We refer to our approach as regularized deep generative model (Reg-DGM) when H consists of distributions defined by neural networks.

Original: In the following, we analyze a simple yet prototypical Gaussian-fitting example to demonstrate how a regularization term defined by a pre-trained model can relieve the bias-variance dilemma in generative modeling.
Revised: In the following, we analyze a simple yet prototypical Gaussian-fitting example to demonstrate how a regularization term defined by a pre-trained model can relieve the bias-variance dilemma in generative modeling.

Original: Such an example is helpful to illustrate the motivation of Reg-DGM precisely and provide valuable insights on its practical performance.
Revised: Such an example is helpful to illustrate the motivation of Reg-DGM precisely and provide valuable insights on its practical performance.

Original: In the same spirit, representative prior work (Arjovsky et al., 2017; Mescheder et al., 2018) in the literature of DGMs has investigated similar examples with several parameters and closed-form solutions.
Revised: In the same spirit, representative prior work (Arjovsky et al., 2017; Mescheder et al., 2018) in the literature of DGMs has investigated similar examples with several parameters and closed-form solutions.

Original: 2. A P ROTOTYPICAL G AUSSIAN -F ITTING E XAMPLE
Revised: 2. A P ROTOTYPICAL G AUSSIAN -F ITTING E XAMPLE

Original: Example 2.1 (Gaussian-fitting example) .
Revised: Example 2.1 (Gaussian-fitting example) .

Original: The data distribution is a (univariate) Gaussian p d ( x ) =
Revised: The data distribution is a (univariate) Gaussian p d ( x ) =

Original: N ( x | µ ∗ , σ 2 ) , where σ 2 is known and µ ∗ is the parameter to be estimated.
Revised: N ( x | µ ∗ , σ 2 ) , where σ 2 is known and µ ∗ is the parameter to be estimated.

Original: A training sample S =
Revised: A training sample S =

Original: { x i } mi =1 is drawn i.i.d.
Revised: { x i } mi =1 is drawn i.i.d.

Original: according to p d ( x ) .
Revised: according to p d ( x ) .

Original: The hypothesis class for p g is H
Revised: The hypothesis class for p g is H

Original: =
Revised: =

Original: {N ( x | µ, σ 2 ) | µ ∈ R } .
Revised: {N ( x | µ, σ 2 ) | µ ∈ R } .

Original: The regularization term in Eq.
Revised: The regularization term in Eq.

Original: (2) is E f ( x ) :
Revised: (2) is E f ( x ) :

Original: = − log N (ˆ µ PRE , σ 2 ) , i.e., p f ( x ) =
Revised: = − log N (ˆ µ PRE , σ 2 ) , i.e., p f ( x ) =

Original: N ( x | ˆ µ PRE , σ 2 ) .
Revised: N ( x | ˆ µ PRE , σ 2 ) .

Original: Note that we let ˆ µ PRE ̸ = µ ∗ and their gap | ˆ µ PRE − µ ∗ | can be large in general.
Revised: Note that we let ˆ µ PRE ̸ = µ ∗ and their gap | ˆ µ PRE − µ ∗ | can be large in general.

Original: For simplicity, we consider the classical maximum likelihood estimation (MLE) (i.e., using the
Revised: For simplicity, we consider the classical maximum likelihood estimation (MLE) (i.e., using the

Original: KL divergence in Eq.
Revised: KL divergence in Eq.

Original: (1) as the performance measure), and its solution for Example 2.1 is given by the sample mean (Bishop & Nasrabadi, 2006): ˆ µ MLE = 1 m (cid:80) m i =1 x i , ˆ µ MLE ∼ N (cid:0) µ ∗ , 1 m σ 2 (cid:1) .
Revised: (1) as the performance measure), and its solution for Example 2.1 is given by the sample mean (Bishop & Nasrabadi, 2006): ˆ µ MLE = 1 m (cid:80) mi =1 x i , ˆ µ MLE ∼ N (cid:0) µ ∗ , 1 m σ 2 (cid:1) .

Original: The pre-trained model ˆ µ PRE is another meaningful baseline for our approach.
Revised: The pre-trained model ˆ µ PRE is another meaningful baseline for our approach.

Original: Clearly, it has a bias of ˆ µ PRE − µ ∗ and a zero variance.
Revised: Clearly, it has a bias of ˆ µ PRE − µ ∗ and a zero variance.

Original: Formally, the solution of our approach based on MLE is ˆ µ REG = 

Revised: Formally, the solution of our approach based on MLE is ˆ µ REG =

Original: + λ ˆ µ MLE + λ 1+ λ ˆ µ PRE , ˆ µ REG ∼ N (cid:16) 11+ λ µ ∗ + λ 1+ λ ˆ µ PRE , σ 2 m (1+ λ ) 2 (cid:17) .
Revised: + λ ˆ µ MLE + λ 1+ λ ˆ µ PRE , ˆ µ REG ∼ N (cid:16) 11+ λ µ ∗ + λ 1+ λ ˆ µ PRE , σ 2 m (1+ λ ) 2 (cid:17) .

Original: We compare all estimators under the mean squared error (MSE) 1 , which is a common measure in statistics and machine learning.
Revised: We compare all estimators under the mean squared error (MSE) 1 , which is a common measure in statistics and machine learning.

Original: Formally, the MSE of an estimator ˆ θ w.r.t.
Revised: Formally, the MSE of an estimator ˆ θ w.r.t.

Original: an unknown parameter θ is defined as:
Revised: an unknown parameter θ is defined as:

Original:  MSE [ˆ θ ] := E [( θ − ˆ θ ) 2 ] , which can be decomposed as the sum of the variance of the estimator and the squared bias of the estimator (Bishop & Nasrabadi, 2006).
Revised: [Figure] MSE [ˆ θ ] := E [( θ − ˆ θ ) 2 ] , which can be decomposed as the sum of the variance of the estimator and the squared bias of the estimator (Bishop & Nasrabadi, 2006).

Original: As summarized in Proposition 2.2, our method achieves a better bias-variance tradeoff than the baselines if λ is in an appropriate range.
Revised: As summarized in Proposition 2.2, our method achieves a better bias-variance tradeoff than the baselines if λ is in an appropriate range.

Original: Proposition 2.2.
Revised: Proposition 2.2.

Original: Let β = λλ
Revised: Let β = λλ

Original: +

Revised: +

Original: be the normalized weight of the regularization term.
Revised: be the normalized weight of the regularization term.

Original: In the
Revised: In the

Original: Gaussian-fitting example 2.1, if max
Revised: Gaussian-fitting example 2.1, if max

Original: [Equation]
Revised: [Figure]

Original: then the following inequalities holds:
Revised: then the following inequalities holds:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Figure]

Original: Due to space limit, the proof is deferred to Appendix A.1.1.
Revised: Due to space limit, the proof is deferred to Appendix A.1.1.

Original: We plot the MSE curves for all estimators w.r.t.
Revised: We plot the MSE curves for all estimators w.r.t.

Original: the value of λ , m and | ˆ µ PRE − µ ∗ | in Fig.
Revised: the value of λ , m and | ˆ µ PRE − µ ∗ | in Fig.

Original: 1 for a clearer illustration.
Revised: 1 for a clearer illustration.

Original: Fig.
Revised: Fig.

Original: 1 (a) directly validates the results of Proposition 2.2.
Revised: 1 (a) directly validates the results of Proposition 2.2.

Original: Fig.
Revised: Fig.

Original: 1 (b) and (c) show that with the optimal λ , the regularization gains more w.r.t.
Revised: 1 (b) and (c) show that with the optimal λ , the regularization gains more w.r.t.

Original: MLE as the sample size m decreases and | ˆ µ PRE − µ ∗ | increases.
Revised: MLE as the sample size m decreases and | ˆ µ PRE − µ ∗ | increases.

Original: See more details of  plot  Appendix 
Revised: See more details of the plot in Appendix B.1.

Original: Since the generalization analysis in deep learning is still largely open (Belkin, 2021; Bartlett et al., 2021), it is difficult to generalize our Proposition 2.2 to the cases with deep models due to lack of appropriate tools.
Revised: Since the generalization analysis in deep learning is still largely open (Belkin, 2021; Bartlett et al., 2021), it is difficult to generalize our Proposition 2.2 to the cases with deep models due to lack of appropriate tools.

Original: However, the intuition behind the Gaussian example also holds in deep learning.
Revised: However, the intuition behind the Gaussian example also holds in deep learning.

Original: Namely, training a model on limited data suffers from a large variance (overfitting) and using a pretrained model suffers from a large bias (underfitting).
Revised: Namely, training a model on limited data suffers from a large variance (overfitting) and using a pretrained model suffers from a large bias (underfitting).

Original: Thus, Reg-DGM with a proper hyperparameter can balance between them and potentially achieve a better performance.
Revised: Thus, Reg-DGM with a proper hyperparameter can balance between them and potentially achieve a better performance.

Original: Further, according to Fig.(c), Reg-DGM benefits if the EBM defined by the pre-trained model is close to the target distribution, which inspires a data-dependent energy function presented in Sec. 4.
Revised: Further, according to Fig.(c), Reg-DGM benefits if the EBM defined by the pre-trained model is close to the target distribution, which inspires a data-dependent energy function presented in Sec. 4.

Original:  C ONVERGENCE A NALYSES 3. A NALYSES IN THE N ON - PARAMETRIC S ETTING
Revised:  C ONVERGENCE A NALYSES 3. A NALYSES IN THE N ON - PARAMETRIC S ETTING

Original: We assume that our hypothesis class contains all valid distributions (i.e. H = P X ), and the data distribution p d is accessible.
Revised: We assume that our hypothesis class contains all valid distributions (i.e. H = P X ), and the data distribution p d is accessible.

Original: Although the setting is impractical, such analyses characterize the existence and uniqueness of the global minimum in an ideal case and have been widely considered in deep generative models (Goodfellow et al., 2014; Arjovsky et al., 2017).
Revised: Although the setting is impractical, such analyses characterize the existence and uniqueness of the global minimum in an ideal case and have been widely considered in deep generative models (Goodfellow et al., 2014; Arjovsky et al., 2017).

Original: Further, it is insightful to see how the regularization affects the solution of Reg-DGM in the ideal case.
Revised: Further, it is insightful to see how the regularization affects the solution of Reg-DGM in the ideal case.

Original: Built upon the classical recipe of the calculus of variations and properties of the KL divergence in the topology of weak convergence, we establish our theory on the existence and uniqueness of the global minimum of Eq.
Revised: Built upon the classical recipe of the calculus of variations and properties of the KL divergence in the topology of weak convergence, we establish our theory on the existence and uniqueness of the global minimum of Eq.

Original: (2) with the KL and JS divergence 3 .
Revised: (2) with the KL and JS divergence 3 .

Original: The results are formally characterized in Theorem 3.1 and Theorem 3.2 respectively.
Revised: The results are formally characterized in Theorem 3.1 and Theorem 3.2 respectively.

Original: We refer the readers to Appendix A.2.1 for the proofs.
Revised: We refer the readers to Appendix A.2.1 for the [Table]

Revised: We consider two divergences that are employed in the two representative DGMs: variational auto-encoders (VAE) (Kingma & Welling, 2013) and generative adversarial networks (GANs) (Goodfellow et al., 2014).

Original:  To establish our theory, we assume that (1) X is a nonempty compact set; (2) E f : X → R is continuous and bounded; (3) (cid:82) X
Revised: proofs. To establish our theory, we assume that (1) X is a nonempty compact set; (2) E f : X → R is continuous and bounded; (3) (cid:82) X

Original: e −E f
Revised: e −E f

Original: ( x ) dx < ∞ , which are common and mild.
Revised: ( x ) dx < ∞ , which are common and mild.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Under mild regularity conditions in Assumption A.1, for any λ > 0 , there exists a unique global minimum of the problem in Eq.
Revised: Under mild regularity conditions in Assumption A.1, for any λ > 0 , there exists a unique global minimum of the problem in Eq.

Original: (2) with the KL divergence.
Revised: (2) with the KL divergence.

Original: Furthermore, the global minimum is in the form of p ∗ g ( x ) =
Revised: Furthermore, the global minimum is in the form of p ∗ g ( x ) =

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where α

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Under mild regularity conditions in Assumption A.1, for any λ > 0 , there exists a unique global minimum of the problem in Eq.
Revised: Under mild regularity conditions in Assumption A.1, for any λ > 0 , there exists a unique global minimum of the problem in Eq.

Original: (2) with the JS divergence.
Revised: (2) with the JS divergence.

Original: Furthermore, the global minimum is in the form of p ∗ g ( x ) =
Revised: Furthermore, the global minimum is in the form of p ∗ g ( x ) =

Original: [Equation]
Revised: [Equation]

Original: , where [Equation] R . As shown in Theorem 3.1 and Theorem 3.2, the global minimum is in the form of a reweighted data distribution and the weights are negatively correlated to the energy function defined by the pre-trained model.
Revised: As shown in Theorem 3.1 and Theorem 3.2, the global minimum is in the form of a reweighted data distribution and the weights are negatively correlated to the energy function defined by the pre-trained model.

Original: Qualitatively, the global minimum assigns high density for a sample x if it has high density under the data distribution (i.e., p d ( x ) ) and low value of the energy function (i.e.,
Revised: Qualitatively, the global minimum assigns high density for a sample x if it has high density under the data distribution (i.e., p d ( x ) ) and low value of the energy function (i.e.,

Original: E f ( x ) ).
Revised: E f ( x ) ).

Original: Notably, the weights in Theorem 3.1 and Theorem 3.2 are different because of the different divergences.
Revised: Notably, the weights in Theorem 3.1 and Theorem 3.2 are different because of the different divergences.

Original: In particular, the effect of the pre-trained model is enlarged by the exponential term in
Revised: In particular, the effect of the pre-trained model is enlarged by the exponential term in

Original: Theorem 3.2 using JS divergence (JSD).
Revised: Theorem 3.2 using JS divergence (JSD).

Original: Fig.
Revised: Fig.

Original: 2 (c) and (d) show that with the same value of λ , the weighting coefficients of JSD are distributed in a larger range than KL divergence (KLD).
Revised: 2 (c) and (d) show that with the same value of λ , the weighting coefficients of JSD are distributed in a larger range than KL divergence (KLD).

Original: Naturally, in both theorems, as λ → 0 , the denominator of p ∗ g ( x ) tends to a constant and p ∗ g ( x ) → p d ( x ) , which recovers the solution of pure divergence minimization in Eq.
Revised: Naturally, in both theorems, as λ → 0 , the denominator of p ∗ g ( x ) tends to a constant and p ∗ g ( x ) → p d ( x ) , which recovers the solution of pure divergence minimization in Eq.

Original: Therefore, Reg-DGM is consistent if the weighting parameter λ is a function of m , and lim m →∞ λ ( m ) → 0 .
Revised: Therefore, Reg-DGM is consistent if the weighting parameter λ is a function of m , and lim m →∞ λ ( m ) → 0 .

Original: 3. A NALYSES IN THE P ARAMETRIC S ETTINGS
Revised: 3. A NALYSES IN THE P ARAMETRIC S ETTINGS

Original: Although it provides theoretical insights of Reg-DGM, the non-parametric setting is far from practice.
Revised: Although it provides theoretical insights of Reg-DGM, the non-parametric setting is far from practice.

Original: In fact, in our experiments, the hypothesis class is parameterized by neural networks and the training data is finite.
Revised: In fact, in our experiments, the hypothesis class is parameterized by neural networks and the training data is finite.

Original: In such a case, Reg-DGM can be formulated as a non-convex optimization problem, which is solved by gradient-based methods.
Revised: In such a case, Reg-DGM can be formulated as a non-convex optimization problem, which is solved by gradient-based methods.

Original: Therefore, we also analyze the convergence of Reg-DGM trained by (stochastic) gradient descent in the presence of neural networks upon the general convergence framework (Allen-Zhu et al., 2019).
Revised: Therefore, we also analyze the convergence of Reg-DGM trained by (stochastic) gradient descent in the presence of neural networks upon the general convergence framework (Allen-Zhu et al., 2019).

Original: In particular, as summarized in Theorem 3.3, we show that Reg-DGM with a standard neural network architecture converges with a high probability under mild smoothness assumptions.
Revised: In particular, as summarized in Theorem 3.3, we show that Reg-DGM with a standard neural network architecture converges with a high probability under mild smoothness assumptions.

Original: The assumptions, result and proof are formally presented in Appendix A.2.2.
Revised: The assumptions, result and proof are formally presented in Appendix A.2.2.

Original: Theorem 3.3 (Convergence of Reg-DGM (informal)) .
Revised: Theorem 3.3 (Convergence of Reg-DGM (informal)) .

Original: Under standard and verifiable smoothness assumptions, with a high probability, Reg-DGM with a sufficiently wide ReLU CNN converges to a global optimum of Eq.
Revised: Under standard and verifiable smoothness assumptions, with a high probability, Reg-DGM with a sufficiently wide ReLU CNN converges to a global optimum of Eq.

Original: (2) trained by GD and converges to a local minimum trained by SGD.
Revised: (2) trained by GD and converges to a local minimum trained by SGD.

Original:  I MPLEMENTATION
Revised:  I MPLEMENTATION

Original: In this section, we discuss the base DGM, the pre-trained model and the energy function in practice.
Revised: In this section, we discuss the base DGM, the pre-trained model and the energy function in practice.

Original: 4. B ASE M ODEL
Revised: 4. B ASE M ODEL

Original: Although Reg-DGM applies to variational auto-encoders (VAE) (Kingma & Welling, 2013) and many other DGMs, we focus on GANs (Goodfellow et al., 2014), which are most representative and popular in the scenarios with limited data (Karras et al., 2020a; Mo et al., 2020;
Revised: Although Reg-DGM applies to variational auto-encoders (VAE) (Kingma & Welling, 2013) and many other DGMs, we focus on GANs (Goodfellow et al., 2014), which are most representative and popular in the scenarios with limited data (Karras et al., 2020a; Mo et al., 2020;

Original: Cui et al., 2021).
Revised: Cui et al., 2021).

Original: Formally, GANs optimize an estimate of the JS divergence via a minimax formulation as follows:
Revised: Formally, GANs optimize an estimate of the JS divergence via a minimax formulation as follows:

Original: [Equation]
Revised: [Equation]

Original: where G is a generator that defines p g ( x ) and D is a discriminator that estimates the JS divergence by discriminating samples.
Revised: where G is a generator that defines p g ( x ) and D is a discriminator that estimates the JS divergence by discriminating samples.

Original: Both G and D are parameterized by neural networks and Eq.
Revised: Both G and D are parameterized by neural networks and Eq.

Original: (4) is estimated by the Monte Carlo method over mini-batches sampled from the training set.
Revised: (4) is estimated by the Monte Carlo method over mini-batches sampled from the training set.

Original: For a broader interest, we adopt three strong GAN variants, StyleGAN2 (Karras et al., 2020b), adaptive discriminator augmentation (ADA) (Karras et al., 2020a), and adaptive pseudo augmentation (APA)
Revised: For a broader interest, we adopt three strong GAN variants, StyleGAN2 (Karras et al., 2020b), adaptive discriminator augmentation (ADA) (Karras et al., 2020a), and adaptive pseudo augmentation (APA)

Original: (Jiang et al., 2021) as the base DGMs.
Revised: (Jiang et al., 2021) as the base DGMs.

Original: Please refer to Appendix B.2 for more details.
Revised: Please refer to Appendix B.2 for more details.

Original: 4. P RE - TRAINED M ODEL
Revised: 4. P RE - TRAINED M ODEL

Original: As mentioned in Sec. 2, different from the transfer-based methods (Wang et al., 2018; Mo et al., 2020), Reg-DGM applies to a nontransferable setting where the pre-trained model does not necessarily have the same architecture or the same formulation as p g or does not even have to be a generative model, enjoying the flexibility of choosing the pre-trained models.
Revised: As mentioned in Sec. 2, different from the transfer-based methods (Wang et al., 2018; Mo et al., 2020), Reg-DGM applies to a nontransferable setting where the pre-trained model does not necessarily have the same architecture or the same formulation as p g or does not even have to be a generative model, enjoying the flexibility of choosing the pre-trained models.

Original: In our implementation, the pre-trained model is a feature extractor f : X → R d , which is trained for other tasks (e.g., classification or contrastive representation learning) instead of generation.
Revised: In our implementation, the pre-trained model is a feature extractor f : X → R d , which is trained for other tasks (e.g., classification or contrastive representation learning) instead of generation.

Original: We choose such models because they are easily available and achieve excellent performance in supervised learning.
Revised: We choose such models because they are easily available and achieve excellent performance in supervised learning.

Original: In particular, we investigate three prototypical pre-trained models: a ResNet (He et al., 2015) trained in a supervised manner on ImageNet, a CLIP image encoder (Radford et al., 2021) trained in a self-supervised manner on a large-scale image-text dataset, and a FaceNet (Schroff et al., 2015) trained on a face recognition dataset.
Revised: In particular, we investigate three prototypical pre-trained models: a ResNet (He et al., 2015) trained in a supervised manner on ImageNet, a CLIP image encoder (Radford et al., 2021) trained in a self-supervised manner on a large-scale image-text dataset, and a FaceNet (Schroff et al., 2015) trained on a face recognition dataset.

Original: Please refer to Appendix B.2 for more details.
Revised: Please refer to Appendix B.2 for more details.

Original: Note that such models are nontransferable in the fine-tuning manner (Hinton et al., 2006).
Revised: Note that such models are nontransferable in the fine-tuning manner (Hinton et al., 2006).

Original: Nevertheless, with such models and a data-dependent energy function presented later, Reg-DGM is still competitive to the transfer-based approaches (Wang et al., 2018; Mo et al., 2020) as shown in Tab. 1.
Revised: Nevertheless, with such models and a data-dependent energy function presented later, Reg-DGM is still competitive to the transfer-based approaches (Wang et al., 2018; Mo et al., 2020) as shown in Tab. 1.

Original: 4. E NERGY F UNCTION
Revised: E NERGY F UNCTION

Original: According to the results in Fig.
Revised: According to the results in Fig.

Original: 1 (c) and our intuition, we should define E f such that p f is as close to p d as possible.
Revised: 1 (c) and our intuition, we should define E f such that p f is as close to p d as possible.

Original: In most of the cases, f is pre-trained on a dataset with richer semantics than p d .
Revised: In most of the cases, f is pre-trained on a dataset with richer semantics than p d .

Original: Therefore, it is necessary to involve training data (sampled from p d ) in the energy function to reducing the distance between p f and p d .
Revised: Therefore, it is necessary to involve training data (sampled from p d ) in the energy function to reducing the distance between p f and p d .

Original: As presented above, we specify f as a feature extractor and it is natural to match the features of samples from p g and p d as the energy function.
Revised: As presented above, we specify f as a feature extractor and it is natural to match the features of samples from p g and p d as the energy function.

Original: Formally, the energy function is defined by the expected mean squared error between the features of a generated sample and a training sample as follows:
Revised: Formally, the energy function is defined by the expected mean squared error between the features of a generated sample and a training sample as follows:

Original: [Equation]
Revised: [Equation]

Original: Notably, our implementation with the data-dependent energy function in Eq.
Revised: Notably, our implementation with the data-dependent energy function in Eq.

Original: (5) is a valid instance of the general Reg-DGM framework as formulated in Eq.
Revised: (5) is a valid instance of the general Reg-DGM framework as formulated in Eq.

Original: Furthermore, the convergence results in both the non-parametric and parametric settings (see Sec. 3) hold in this case.
Revised: Furthermore, the convergence results in both the non-parametric and parametric settings (see Sec. 3) hold in this case.

Original: The expectation is estimated by the Monte Carlo method of a single sample for efficiency by default and increasing the number of samples will not affect the performance significantly (see results in Appendix C.4).
Revised: The expectation is estimated by the Monte Carlo method of a single sample for efficiency by default and increasing the number of samples will not affect the performance significantly (see results in Appendix C.4).

Original: We emphasize that our main contribution is not designing a specific energy function but the general framework of Reg-DGM.
Revised: We emphasize that our main contribution is not designing a specific energy function but the general framework of Reg-DGM.

Original: Many alternative energy functions can be employed in Reg-DGM.
Revised: Many alternative energy functions can be employed in Reg-DGM.

Original: Indeed, we perform a systematical ablation study of the energy functions in Sec.
Revised: Indeed, we perform a systematical ablation study of the energy functions in Sec.

Original: 5.3 and find that Eq.
Revised: 5.3 and find that Eq.

Original: (5) is the best among them considering the qualitative and quantitative results together.
Revised: (5) is the best among them considering the qualitative and quantitative results together.

Original: Moreover, we evaluate the effectiveness of Reg-DGM implemented by Eq.
Revised: Moreover, we evaluate the effectiveness of Reg-DGM implemented by Eq.

Original: (5), with strong base DGMs (Karras et al., 2020b;a; Jiang et al., 2021), different datasets, different backbones of f and different pretraining datasets in the experiments.
Revised: (5), with strong base DGMs (Karras et al., 2020b;a; Jiang et al., 2021), different datasets, different backbones of f and different pretraining datasets in the experiments.

Original: We observe a consistent and significant improvement over
Revised: We observe a consistent and significant improvement over

Original: SOTA baselines across various settings.
Revised: SOTA baselines across various settings.

Original: Based on such a comprehensive empirical study, we believe that our implementation of Reg-DGM based on Eq.
Revised: Based on such a comprehensive empirical study, we believe that our implementation of Reg-DGM based on Eq.

Original: (5) would be effective in new settings.
Revised: (5) would be effective in new settings.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For a fair comparison to a large family of prior work, we evaluate Reg-DGM on several widely adopted benchmarks with limited data (Karras et al., 2020a), including the FFHQ (Karras et al., 2019), 100 -shot Obama (Zhao et al., 2020a), LSUN CAT (Yu et al., 2015) and CIFAR-
Revised: For a fair comparison to a large family of prior work, we evaluate Reg-DGM on several widely adopted benchmarks with limited data (Karras et al., 2020a), including the FFHQ (Karras et al., 2019), 100 -shot Obama (Zhao et al., 2020a), LSUN CAT (Yu et al., 2015) and CIFAR-

Original: (Krizhevsky et al., 2009) datasets, and the data processing and metric calculating are the same as those of ADA (Karras et al., 2020a).
Revised: (Krizhevsky et al., 2009) datasets, and the data processing and metric calculating are the same as those of ADA (Karras et al., 2020a).

Original: We present the main results and analyses in the section and refer the readers to Appendix B.2 for details and Appendix C for additional results.
Revised: We present the main results and analyses in the section and refer the readers to Appendix B.2 for details and Appendix C for additional results.

Original: Throughout the section, we refer to our approach as the name of the base DGM with the prefix “Reg-”.
Revised: Throughout the section, we refer to our approach as the name of the base DGM with the prefix “Reg-”.

Original: For instance, “Reg-ADA” denotes our approach with ADA (Karras et al., 2020a) as the base DGM.
Revised: For instance, “Reg-ADA” denotes our approach with ADA (Karras et al., 2020a) as the base DGM.

Original: 5. B ENCHMARK R ESULTS WITH L IMITED D ATA
Revised: [Table] [Figure] [Figure]

Original: We employ StyleGAN2 (Karras et al., 2020b), ADA (Karras et al., 2020a) and APA (Jiang et al., 2021) as the base DGMs and a ResNet-

Revised: We employ StyleGAN2 (Karras et al., 2020b), ADA (Karras et al., 2020a) and APA (Jiang et al., 2021) as the base DGMs and a ResNet-

Original: (He et al., 2015) classifier trained on ImageNet (Deng et al., 2009) as the pre-trained model by default.
Revised: (He et al., 2015) classifier trained on ImageNet (Deng et al., 2009) as the pre-trained model by default.

Original: The associated energy function is defined in Eq.
Revised: The associated energy function is defined in Eq.

Original: Quantitatively, we compare Reg-DGM with a large family of existing methods, including the base
Revised: Quantitatively, we compare Reg-DGM with a large family of existing methods, including the base

Original: DGMs, the transfer-based approaches, the augmentation-based methods and many others.
Revised: DGMs, the transfer-based approaches, the augmentation-based methods and many others.

Original: Following the direct and strong competitor (Karras et al., 2020a), we report the median Fr´echet inception distance (FID) (Heusel et al., 2017) on FFHQ and LSUN CAT, and the mean FID on CIFAR-10 out [Table]
Revised: Following the direct and strong competitor (Karras et al., 2020a), we report the median Fr´echet inception distance (FID) (Heusel et al., 2017) on FFHQ and LSUN CAT, and the mean FID on CIFAR-10 out of 3 runs for a fair comparison in Tab. 1.

Original: [Figure]
Revised: For completeness, we also report the mean FID with the standard deviation on FFHQ and LSUN CAT in Appendix C.1.

Original: As shown in Tab. 1, Reg-StyleGAN2, Reg-ADA and Reg-ADA-APA consistently outperform the corresponding base DGM in five settings, demonstrating that Reg-DGM can achieve a good biasvariance tradeoff in practice.
Revised: As shown in Tab. 1, Reg-StyleGAN2, Reg-ADA and Reg-ADA-APA consistently outperform the corresponding base DGM in five settings, demonstrating that Reg-DGM can achieve a good biasvariance tradeoff in practice.

Original: Besides, the superior performance of Reg-ADA over ADA (and RegADA-APA over ADA-APA) shows that our contribution is orthogonal to the augmentation-based approaches.
Revised: Besides, the superior performance of Reg-ADA over ADA (and RegADA-APA over ADA-APA) shows that our contribution is orthogonal to the augmentation-based approaches.

Original: Notably, the improvement of Reg-DGM over the base DGM is larger when the sample size m is smaller.
Revised: Notably, the improvement of Reg-DGM over the base DGM is larger when the sample size m is smaller.

Original: This is as expected because the relative gain of Reg-DGM over the base DGM  increases as m decreases given a fixed pre-trained model and the optimal λ (see Fig.
Revised: This is as expected because the relative gain of Reg-DGM over the base DGM [Table] increases as m decreases given a fixed pre-trained model and the optimal λ (see Fig.

Original: 1 (b)).
Revised: 1 (b)).

Original: Further, we evaluate the base DGMs and Reg-DGMs under the kernel inception distance (KID) (Bi´nkowski et al., 2018) metric in Appendix C.4.
Revised: Further, we evaluate the base DGMs and Reg-DGMs under the kernel inception distance (KID) (Bi ´ nkowski et al., 2018) metric in Appendix C.4.

Original: The conclusion remains the same.
Revised: The conclusion remains the same.

Original: As suggested by Proposition 2.2, the value of the weighting hyperparameter λ is crucial and there is an appropriate range of λ such that Reg-DGM is better than the base DGM.
Revised: As suggested by Proposition 2.2, the value of the weighting hyperparameter λ is crucial and there is an appropriate range of λ such that Reg-DGM is better than the base DGM.

Original: We empirically validate the argument on
Revised: We empirically validate the argument on

Original: FFHQ- 5 k with StyleGAN2 as the base model in Appendix C.5.
Revised: FFHQ- 5 k with StyleGAN2 as the base model in Appendix C.5.

Original: We mention that the methods based on fine-tuning (Wang et al., 2018; Mo et al., 2020) in Tab.employ a GAN pre-trained on CelebA-HQ (Karras et al., 2017), which is also a face dataset of the same resolution as FFHQ.
Revised: We mention that the methods based on fine-tuning (Wang et al., 2018; Mo et al., 2020) in Tab.employ a GAN pre-trained on CelebA-HQ (Karras et al., 2017), which is also a face dataset of the same resolution as FFHQ.

Original: In comparison, our approach is built upon a classifier pertained on
Revised: In comparison, our approach is built upon a classifier pertained on

Original: [Table]

Original: ImageNet.
Revised: ImageNet.

Original: As highlighted in (Wang et al., 2018), the density of the pre-training dataset is more important than the diversity.
Revised: As highlighted in (Wang et al., 2018), the density of the pre-training dataset is more important than the diversity.

Original: Nevertheless, Reg-DGM is competitive with these strong baselines while enjoying the flexibility of choosing the pre-trained model and dataset.
Revised: Nevertheless, Reg-DGM is competitive with these strong baselines while enjoying the flexibility of choosing the pre-trained model and dataset.

Original: Further [Table] we directly adopt the same pre-trained model across all datasets including CIFAR-10, where it takes additional efforts to get a suitable pre-trained generative model to fine-tune.
Revised: Further, we directly adopt the same pre-trained model across all datasets including CIFAR-10, where it takes additional efforts to get a suitable pre-trained generative model to fine-tune.

Original: Based on the results, it is safe to emphasize the complementary role of Reg-DGM to the approaches based on fine-tuning.
Revised: Based on the results, it is safe to emphasize the complementary role of Reg-DGM to the approaches based on fine-tuning.

Original: Qualitatively, we show the samples generated from Reg-DGM on FFHQ- 5 k and 100 -shot Obama in
Revised: Qualitatively, we show the samples generated from Reg-DGM on FFHQ- 5 k and 100 -shot Obama in

Original: Fig.
Revised: Fig.

Original: It can be seen that with the regularization, our approach can produce faces of a normal shape with limited data.
Revised: It can be seen that with the regularization, our approach can produce faces of a normal shape with limited data.

Original: We present the results from the base DGMs and more samples in other settings in Appendix C.2.
Revised: We present the results from the base DGMs and more samples in other settings in Appendix C.2.

Original: For a comprehensive comparison on the visual quality of samples, we perform a human evaluation by the Amazon Mechanical Turk (AMT) as in prior work (Choi et al., 2020).
Revised: For a comprehensive comparison on the visual quality of samples, we perform a human evaluation by the Amazon Mechanical Turk (AMT) as in prior work (Choi et al., 2020).

Original: According with the FID results, Reg-StyleGAN2 trained on FFHQ- 5 k against the baseline StyleGAN is chosen in 63 .
Revised: According with the FID results, Reg-StyleGAN2 trained on FFHQ- 5 k against the baseline StyleGAN is chosen in 63 .

Original: 5% of the 3,000 image quality comparison tasks.
Revised: 5% of the 3,000 image quality comparison tasks.

Original: See details in Appendix C.4.
Revised: See details in Appendix C.4.

Original: 5. A BLATION OF P RE - TRAINED M ODELS AND P RE - TRAINING D ATASETS
Revised: A BLATION OF P RE - TRAINED M ODELS AND P RE - TRAINING D ATASETS

Original: For simplicity, the main results of Reg-DGM presented in Sec.
Revised: For simplicity, the main results of Reg-DGM presented in Sec.

Original: 5.1 are based on a ResNet-18 model pre-trained on ImageNet.
Revised: 5.1 are based on a ResNet-18 model pre-trained on ImageNet.

Original: Although its architecture is significantly different from the Inceptionv3 (Szegedy et al., 2016) used in FID and KID calculation, it is worth performing an ablation on the pre-trained models to eliminate the potential bias caused by the pre-training dataset.
Revised: Although its architecture is significantly different from the Inceptionv3 (Szegedy et al., 2016) used in FID and KID calculation, it is worth performing an ablation on the pre-trained models to eliminate the potential bias caused by the pre-training dataset.

Original: In particular, we test Reg-DGM with the image encoder of the CLIP model (Radford et al., 2021) (an architecture very similar to ResNet-50), which is pre-trained on large-scale noisy text-image pairs instead of ImageNet, and with the face recognizer Inception-ResNet-v1 (Szegedy et al., 2017) of
Revised: In particular, we test Reg-DGM with the image encoder of the CLIP model (Radford et al., 2021) (an architecture very similar to ResNet-50), which is pre-trained on large-scale noisy text-image pairs instead of ImageNet, and with the face recognizer Inception-ResNet-v1 (Szegedy et al., 2017) of

Original: FaceNet (Schroff et al., 2015), which is pre-trained on the large-scale face dataset VGGFace2 (Cao et al., 2018).
Revised: FaceNet (Schroff et al., 2015), which is pre-trained on the large-scale face dataset VGGFace2 (Cao et al., 2018).

Original: See details in Appendix B.2.
Revised: See details in Appendix B.2.

Original: The median FID and corresponding KID results are shown in Tab. 2.
Revised: The median FID and corresponding KID results are shown in Tab. 2.

Original: Without heavily tuning the hyperparameter λ , Reg-DGM shows consistent improvements over the two baselines under both FID and KID metrics, and achieves comparable results to those presented in Table 1.
Revised: Without heavily tuning the hyperparameter λ , Reg-DGM shows consistent improvements over the two baselines under both FID and KID metrics, and achieves comparable results to those presented in Table 1.

Original: The results with the CLIP model and the FaceNet model demonstrate that
Revised: The results with the CLIP model and the FaceNet model demonstrate that

Original: Reg-DGM works well with various backbones and pre-training datasets.
Revised: Reg-DGM works well with various backbones and pre-training datasets.

Original: 5. A BLATION OF E NERGY F UNCTIONS
Revised: 5. A BLATION OF E NERGY F UNCTIONS

Original: As emphasized in Sec. 4, our main contribution is the general framework instead of a specific energy function.
Revised: As emphasized in Sec. 4, our main contribution is the general framework instead of a specific energy function.

Original: Nevertheless, we still investigate two alternative regularization terms for completeness.
Revised: Nevertheless, we still investigate two alternative regularization terms for completeness.

Original: We first consider the famous entropy-minimization regularization (Grandvalet & Bengio, 2004), which is data-independent.
Revised: We first consider the famous entropy-minimization regularization (Grandvalet & Bengio, 2004), which is data-independent.

Original: Intuitively, the regularization forces p g that produces samples with clear semantics.
Revised: Intuitively, the regularization forces p g that produces samples with clear semantics.

Original: We find that the entropy regularization achieves similar FID results of 50 . 87 on FFHQ- 5 k to the baseline 52 .
Revised: We find that the entropy regularization achieves similar FID results of 50 . 87 on FFHQ- 5 k to the baseline 52 .

Original: 71 , showing the importance of the data dependency in the energy function.
Revised: 71 , showing the importance of the data dependency in the energy function.

Original: We then investigate another data-dependent regularization term, i.e., feature matching (Salimans et al., 2016), which matches the averaged features between the model and data distributions.
Revised: We then investigate another data-dependent regularization term, i.e., feature matching (Salimans et al., 2016), which matches the averaged features between the model and data distributions.

Original: We find that feature matching can achieve FID 32 .
Revised: We find that feature matching can achieve FID 32 .

Original: 65 on FFHQ- 5 k greatly reducing the FID of StyleGAN2 while it cannot improve the visual quality of the samples.
Revised: 65 on FFHQ- 5 k greatly reducing the FID of StyleGAN2 while it cannot improve the visual quality of the samples.

Original: Please refer to Appendix C.4 for details of both energy functions.
Revised: Please refer to Appendix C.4 for details of both energy functions.

Original: Therefore, Eq.
Revised: Therefore, Eq.

Original: (5) is the best among them considering the qualitative and quantitative results together and we believe it can be transferred to new settings based on our results in Sec.
Revised: (5) is the best among them considering the qualitative and quantitative results together and we believe it can be transferred to new settings based on our results in Sec.

Original: 5.1 and Sec. 5.2.
Revised: 5.1 and Sec. 5.2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Fine-tuning approaches.
Revised: Fine-tuning approaches.

Original: A milestone of deep learning is that a deep generative model fine-tuned for classification outperforms the classical SVM on recognizing the hand-writing digits (Hinton et al., 2006).
Revised: A milestone of deep learning is that a deep generative model fine-tuned for classification outperforms the classical SVM on recognizing the hand-writing digits (Hinton et al., 2006).

Original: Since then, the idea of fine-tuning has a significant impact (Devlin et al., 2018;
Revised: Since then, the idea of fine-tuning has a significant impact (Devlin et al., 2018;

Original: He et al., 2020) including generative models with limited data (Wang et al., 2018; Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021).
Revised: He et al., 2020) including generative models with limited data (Wang et al., 2018; Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021).

Original: However, an inherent restriction of fine-tuning is that the pre-trained model and target model should partially share a common structure.
Revised: However, an inherent restriction of fine-tuning is that the pre-trained model and target model should partially share a common structure.

Original: Thus, it may take additional efforts to find a suitable pre-trained model to fine-tune.
Revised: Thus, it may take additional efforts to find a suitable pre-trained model to fine-tune.

Original: In comparison, Reg-DGM provides an alternative way to make it possible to exploit a pre-trained classifier to help generative modeling.
Revised: In comparison, Reg-DGM provides an alternative way to make it possible to exploit a pre-trained classifier to help generative modeling.

Original: Notably, the latter is often thought of as much harder than the former.
Revised: Notably, the latter is often thought of as much harder than the former.

Original: Other generative adversarial networks with limited data.
Revised: Other generative adversarial networks with limited data.

Original: To relieve the overfitting problem of the discriminator, DA (Zhao et al., 2020a), ADA (Karras et al., 2020a) and APA (Jiang et al., 2021) design sophisticated data augmentation strategies.
Revised: To relieve the overfitting problem of the discriminator, DA (Zhao et al., 2020a), ADA (Karras et al., 2020a) and APA (Jiang et al., 2021) design sophisticated data augmentation strategies.

Original: GenCo (Cui et al., 2021) designs a cotraining framework that introduces multiple complimentary discriminators.
Revised: GenCo (Cui et al., 2021) designs a cotraining framework that introduces multiple complimentary discriminators.

Original: InsGen (Yang et al., 2021) improves the data efficiency of GANs via an instance discrimination loss (Wu et al., 2018).
Revised: InsGen (Yang et al., 2021) improves the data efficiency of GANs via an instance discrimination loss (Wu et al., 2018).

Original: We believe that Reg-DGM is orthogonal to these methods based on our results in Tab. 1.
Revised: We believe that Reg-DGM is orthogonal to these methods based on our results in Tab. 1.

Original: Regularization in probabilistic models.
Revised: Regularization in probabilistic models.

Original: Extensive regularization approaches have been developed in traditional Bayesian inference (Zhu et al., 2014) and probabilistic modeling (Chang et al., 2007;
Revised: Extensive regularization approaches have been developed in traditional Bayesian inference (Zhu et al., 2014) and probabilistic modeling (Chang et al., 2007;

Original: Liang et al., 2009; Ganchev et al., 2010).
Revised: Liang et al., 2009; Ganchev et al., 2010).

Original: Among them, posterior regularization (PR) (Ganchev et al., 2010) encodes the human knowledge about the task as linear constraints of the latent representations in generative models for better inference performance.
Revised: Among them, posterior regularization (PR) (Ganchev et al., 2010) encodes the human knowledge about the task as linear constraints of the latent representations in generative models for better inference performance.

Original: Such methods have been extended to deep generative models (Hu et al., 2018; Du et al., 2018; Shu et al., 2018; Xu et al., 2019) for a similar reason.
Revised: Such methods have been extended to deep generative models (Hu et al., 2018; Du et al., 2018; Shu et al., 2018; Xu et al., 2019) for a similar reason.

Original: Technically, PR-based methods regularize the latent space via handcrafted or jointly trained constraints.
Revised: Technically, PR-based methods regularize the latent space via handcrafted or jointly trained constraints.

Original: In comparison, our approach regularizes the data space via a pre-trained model.
Revised: In comparison, our approach regularizes the data space via a pre-trained model.

Original: Besides,
Revised: Besides,

Original: PR-based methods are suitable for structured prediction tasks instead of generative modeling with limited data, which is the main focus of this paper.
Revised: PR-based methods are suitable for structured prediction tasks instead of generative modeling with limited data, which is the main focus of this paper.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we propose regularized deep generative model (Reg-DGM), which leverages a pretrained model for regularization to reduce the variance of DGMs with limited data.
Revised: In this paper, we propose regularized deep generative model (Reg-DGM), which leverages a pretrained model for regularization to reduce the variance of DGMs with limited data.

Original: Theoretically, we analyze the convergence properties of Reg-DGM.
Revised: Theoretically, we analyze the convergence properties of Reg-DGM.

Original: Empirically, with various pre-trained feature extractors and a data-dependent energy function, Reg-DGM consistently improves the generation performance of strong DGMs and achieves competitive results to the state-of-the-art methods.
Revised: Empirically, with various pre-trained feature extractors and a data-dependent energy function, Reg-DGM consistently improves the generation performance of strong DGMs and achieves competitive results to the state-of-the-art methods.

Original: An interesting future work is to analyze the generalization behaviour of Reg-DGM in general and inspire new energy functions.
Revised: An interesting future work is to analyze the generalization behaviour of Reg-DGM in general and inspire new energy functions.

Original: Currently, the generalization analysis of deep learning is still largely open (Zhang et al., 2021; Bartlett et al., 2021; Belkin, 2021), and there lacks appropriate tools to formalize our intuition on the bias-variance tradeoff in general.
Revised: Currently, the generalization analysis of deep learning is still largely open (Zhang et al., 2021; Bartlett et al., 2021; Belkin, 2021), and there lacks appropriate tools to formalize our intuition on the bias-variance tradeoff in general.

Original: E THICS S TATEMENT
Revised: A CKNOWLEDGEMENT

Revised: We thank Guoqiang Wu for helpful discussions about the generalization analysis.

Revised: This work was supported by NSF of China (Nos. 62076145, 62106273); Beijing Outstanding Young Scientist

Revised: Program (No. BJJWZYJH012019100020098); Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13, 22XNKJ16).

Revised: Part of the computing resources supporting this work, totaled 720 A

Revised: GPU hours, were provided by High-Flyer AI.

Revised: (Hangzhou High-Flyer AI Fundamental Research Co.,

Revised: Ltd.).

Revised: C. Li was also sponsored by Beijing Nova Program.

Revised: E THICS S TATEMENT

Original: This work presents a framework to train deep generative models on small data.
Revised: This work presents a framework to train deep generative models on small data.

Original: By improving the data efficiency, it can potentially benefit real-world applications like medicine analysis and automatic drive.
Revised: By improving the data efficiency, it can potentially benefit real-world applications like medicine analysis and automatic drive.

Original: However, this work can have negative consequences in the form of “DeepFakes”, as existing
Revised: However, this work can have negative consequences in the form of “DeepFakes”, as existing

Original: GANs.
Revised: GANs.

Original: It is worth noting that this work may exacerbate such issues by improving the data efficiency of GANs.
Revised: It is worth noting that this work may exacerbate such issues by improving the data efficiency of GANs.

Original: How to detect “DeepFakes” is an active research area in machine learning, which aims to relieve the problem.
Revised: How to detect “DeepFakes” is an active research area in machine learning, which aims to relieve the problem.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We submit the source code in the supplementary material and will release it after the blind review.
Revised: We submit the source code in the supplementary material and have released it.

Original: Datasets used in experiments are open and publicly available, and experimental details are provided in the Appendix B.
Revised: Datasets used in experiments are open and publicly available, and experimental details are provided in the Appendix B.

Original: In addition, the complete proof of the propositions and theorems is contained in the Appendix A
Revised: In addition, the complete proof of the propositions and theorems is contained in the Appendix A.

Original: A P ROOFS A. T HE G AUSSIAN -F ITTING E XAMPLE
Revised: A P ROOFS A. T HE G AUSSIAN -F ITTING E XAMPLE

Original: We first derive the solution of Reg-DGM in the main text.
Revised: We first derive the solution of Reg-DGM in the main text.

Original: In the Gaussian fitting example, the regularized optimization problem can be written as
Revised: In the Gaussian fitting example, the regularized optimization problem can be written as

Original: [Equation]
Revised: [Equation]

Original: where the first equality holds by the definition and properties of Gaussian (Bishop & Nasrabadi, 2006) and the second equality holds by omitting a constant irrelevant to the optimization.
Revised: where the first equality holds by the definition and properties of Gaussian (Bishop & Nasrabadi, [Figure] ) and the second equality holds by omitting a constant irrelevant to the optimization.

Original: It is easy to solve the above quadratic programming problem analytically:
Revised: It is easy to solve the above quadratic programming problem analytically:

Original: [Equation]
Revised: [Equation]

Original: where ˆ µ MLE ∼ N [Equation] µ ∗ , σ 2 m [Equation]
Revised: where ˆ µ MLE ∼ N (cid:16) µ ∗ , σ 2 m (cid:17) .

Original: ˆ µ REG is obtained by an affine transformation of a Gaussian random variable and thus is also Gaussian distributed as follows:
Revised: ˆ µ REG is obtained by an affine transformation of a Gaussian random variable and thus is also Gaussian distributed as follows:

Original: [Equation]
Revised: [Equation]

Original: A.1. P ROOF OF P ROPOSITION 2. 
Revised: A.1. P ROOF OF P ROPOSITION 2.

Original: Proof.
Revised: Proof.

Original: In the Gaussian-fitting example, we haveˆ
Revised: In the Gaussian-fitting example, we have

Original: [Equation]
Revised: [Equation]

Original: 11) and ˆ
Revised: and ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: According to the bias-variance decomposition of the MSE, we have
Revised: According to the bias-variance decomposition of the MSE, we have

Original: [Equation]
Revised: [Equation]

Original: 13) andMSE [Equation]
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Let β = λ 1+ λ ∈ (0 , 1) be the normalized weight of the regularization term.
Revised: Let β = λ 1+ λ ∈ (0 , 1) be the normalized weight of the regularization term.

Original: Then, we can rewrite
Revised: Then, we can rewrite

Original: MSE [ˆ µ REG ] as
Revised: MSE [ˆ µ REG ] as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To satisfy MSE [ˆ µ REG ] < MSE [ˆ µ MLE ] , we have
Revised: To satisfy MSE [ˆ µ REG ] < MSE [ˆ µ MLE ] , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The pre-trained model ˆ µ PRE is another meaningful baseline, which has a bias of ˆ µ PRE − µ ∗ and a zero variance.
Revised: The pre-trained model ˆ µ PRE is another meaningful baseline, which has a bias of ˆ µ PRE − µ ∗ and a zero variance.

Original: Its MSE is given by
Revised: Its MSE is given by

Original: [Equation]
Revised: [Equation]

Original: To satisfy MSE [ˆ µ REG ] < MSE [ˆ µ PRE ] , we have
Revised: To satisfy MSE [ˆ µ REG ] < MSE [ˆ µ PRE ] , we have

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: We now computes the optimal MSE [ˆ µ REG ] .
Revised: We now computes the optimal MSE [ˆ µ REG ] .

Original: It is easy to see that the quadratic programming problem in Eq.
Revised: It is easy to see that the quadratic programming problem in Eq.

Original: (15) achieves its minimum at β ∗ =
Revised: (15) achieves its minimum at β ∗ =

Original: σ 2 σ

Original: [Equation]
Revised: [Equation]

Original:  m 
Revised: 2 σ 2 + m (ˆ µ PRE − µ ∗ ) 2 with the corresponding λ ∗ =

Original: [Equation]
Revised: [Equation]

Original: ) 2 with the corresponding λ ∗ = σ 2 m(ˆ
Revised: m

Original: [Equation]
Revised: [Equation]

Original: PRE − µ ∗ ) 2 .
Revised: µ ∗ ) 2 .

Original: The minimum value is
Revised: The minimum value is

Original: [Equation]
Revised: [Equation]

Original: A.1.C OMPARISON UNDER THE E XPECTED R ISK 
Revised: A.1.

Revised: C OMPARISON UNDER THE E XPECTED R ISK

Original: We also evaluate all estimators in terms of the expected risk , which is a commonly used measure in statistical learning theory.
Revised: We also evaluate all estimators in terms of the expected risk , which is a commonly used measure in statistical learning theory.

Original: In a density estimation task for p d , the expected risk of a hypothesis ˆ µ , which depends on the training sample S , is R (ˆ µ ) :=
Revised: In a density estimation task for p d , the expected risk of a hypothesis ˆ µ , which depends on the training sample S , is R (ˆ µ ) :=

Original: E x ∼ p
Revised: E x ∼ p

Original: d [ − log p ( x ; ˆ µ )] .
Revised: d [ − log p ( x ; ˆ µ )] .

Original: We can show that the expectation of the expected risk w.r.t.
Revised: We can show that the expectation of the expected risk w.r.t.

Original: S coincides with the corresponding MSE in the Gaussianfitting example and directly obtain the following Corollary A.0.1 from Proposition 2.2.
Revised: S coincides with the corresponding MSE in the Gaussianfitting example and directly obtain the following Corollary A.0.1 from Proposition 2.2.

Original: Corollary A.0.1.
Revised: Corollary A.0.1.

Original: In the Gaussian-fitting example 2.1, if λ satisfies the same condition as in Proposition 2.2, then the following inequality holds:
Revised: In the Gaussian-fitting example 2.1, if λ satisfies the same condition as in Proposition 2.2, then the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the Gaussian-fitting example, the expected risk for a hypothesis ˆ µ is
Revised: In the Gaussian-fitting example, the expected risk for a hypothesis ˆ µ is

Original: [Equation]
Revised: [Equation]

Original:  which completes the proof together with Proposition 2.2.
Revised: 22) which completes the proof together with Proposition 2.2.

Original: A. O PTIMIZATION A NALYSES A.2. P ROOF OF T HEOREM 3.1 AND T HEOREM 3. 
Revised: A. O PTIMIZATION A NALYSES A.2. P ROOF OF T HEOREM 3.1 AND T HEOREM 3.

Original: Our results rely on the following regularity conditions.
Revised: Our results rely on the following regularity conditions.

Original: Assumption A.1.
Revised: Assumption A.1.

Original: X is a nonempty compact set.
Revised: X is a nonempty compact set.

Original: E f : X → R is continuous and bounded.
Revised: E f : X → R is continuous and bounded.

Original: (cid:82) X
Revised: (cid:82) X

Original: e −E f
Revised: e −E f

Original: ( x ) dx < ∞ .
Revised: ( x ) dx < ∞ .

Original: The regularity conditions in Assumption A.1 are mild in the sense:
Revised: The regularity conditions in Assumption A.1 are mild in the sense:

Original: The sample space X is often a subset of a n -dimensional Euclidean space.
Revised: The sample space X is often a subset of a n -dimensional Euclidean space.

Original: Then, X is compact if and only if X is bounded and closed, which holds for extensive datasets in various types including images, videos, audios, and texts.
Revised: Then, X is compact if and only if X is bounded and closed, which holds for extensive datasets in various types including images, videos, audios, and texts.

Original: In this paper, E f is defined by compositing a neural network and a continuous real-valued function.
Revised: In this paper, E f is defined by compositing a neural network and a continuous real-valued function.

Original: Then E f is continuous and bounded on X if the neural network has bounded weights and uses continuous activation functions including ReLU (Nair & Hinton, 2010), Tanh and Sigmoid.
Revised: Then E f is continuous and bounded on X if the neural network has bounded weights and uses continuous activation functions including ReLU (Nair & Hinton, 2010), Tanh and Sigmoid.

Original: [Equation]
Revised: (cid:82) X

Original: X e −E f
Revised: e −E f

Original: ( x ) dx < ∞ holds following the choice of the sample space and energy function,
Revised: ( x ) dx < ∞ holds following the choice of the sample space and energy function,

Original: We first prove Theorem 3.1 as follows.
Revised: We first prove Theorem 3.1 as follows.

Original: Proof.
Revised: Proof.

Original: Ignoring some constant irrelevant to the optimization, we rewrite the optimization problem of our approach with the KL divergence as follows:
Revised: Ignoring some constant irrelevant to the optimization, we rewrite the optimization problem of our approach with the KL divergence as follows:

Original: [Equation]
Revised: [Equation]

Original: For clarity, we denote the functional in problem Eq.
Revised: For clarity, we denote the functional in problem Eq.

Original: (23) to be optimized as
Revised: (23) to be optimized as

Original: [Equation]
Revised: [Equation]

Original: According to Assumption 3.1 that X is a nonempty compact set, the feasible area characterized by the constraints (i.e., P X ) is compact in the topology of weak convergence by the Prokhorov’s
Revised: According to Assumption 3.1 that X is a nonempty compact set, the feasible area characterized by the constraints (i.e., P X ) is compact in the topology of weak convergence by the Prokhorov’s

Original: Theorem (See Corollary 6.8 in (Van Gaans, 2003)).
Revised: Theorem (See Corollary 6.8 in (Van Gaans, 2003)).

Original: By Theorem 3.6 in (Ajjanagadde et al., 2017),
Revised: By Theorem 3.6 in (Ajjanagadde et al., 2017),

Original: KL divergence is lower semi-continuous in the topology of weak convergence.
Revised: KL divergence is lower semi-continuous in the topology of weak convergence.

Original: According to Assumption 3.2 that E f is continuous and bounded on X , the regularization term is continuous in the topology of weak convergence.
Revised: According to Assumption 3.2 that E f is continuous and bounded on X , the regularization term is continuous in the topology of weak convergence.

Original: Therefore, by the extreme value theorem, the global minimum of
Revised: Therefore, by the extreme value theorem, the global minimum of

Original: J ( p g ) exists in the feasible area.
Revised: J ( p g ) exists in the feasible area.

Original: Note that the optimization problem Eq.
Revised: Note that the optimization problem Eq.

Original: (23) is convex.
Revised: (23) is convex.

Original: To obtain a necessary condition for the global minima, we get the Lagrangian with the equality constraint:
Revised: To obtain a necessary condition for the global minima, we get the Lagrangian with the equality constraint:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where α ∈ R .
Revised: where α ∈ R .

Original: Note that for simplicity, we do not include the inequality constraint, which will be verified shortly.
Revised: Note that for simplicity, we do not include the inequality constraint, which will be verified shortly.

Original: It is easy to check the constraint qualifications for problem Eq.
Revised: It is easy to check the constraint qualifications for problem Eq.

Original: By the calculus of variations, a necessary condition for a global minimum of problem Eq.
Revised: By the calculus of variations, a necessary condition for a global minimum of problem Eq.

Original: (23) is
Revised: (23) is

Original: [Equation]
Revised: [Equation]

Original:  which implies that
Revised: 25) which implies that

Original: [Equation]
Revised: [Equation]

Original: We define A = { α ∈ R | p d ( x ) α + λ E f ( x ) ≥ 0 , ∀ x ∈ X} .
Revised: We define A = { α ∈ R | p d ( x ) α + λ E f ( x ) ≥ 0 , ∀ x ∈ X} .

Original: According to Assumption 3.1 that E f is bounded on X , we have A ̸ = ∅ .
Revised: According to Assumption 3.1 that E f is bounded on X , we have A ̸ = ∅ .

Original: We define a function ϕ : A → R as
Revised: We define a function ϕ : A → R as

Original: [Equation]
Revised: [Equation]

Original: It is easy to see ϕ ( α ) is monotonically decreasing
Revised: It is easy to see ϕ ( α ) is monotonically decreasing

Original: and there is at most one α ∗ ∈
Revised: and there is at most one α ∗ ∈

Original: A such that ϕ ( α ∗ ) = 1 , which finishes the proof together with the existence of the global minimum.
Revised: A such that ϕ ( α ∗ ) = 1 , which finishes the proof together with the existence of the global minimum.

Original: We now prove Theorem 3.2 as follows.
Revised: We now prove Theorem 3.2 as follows.

Original: Proof.
Revised: Proof.

Original: The proof here shares the same spirit of Theorem 3.1.
Revised: The proof here shares the same spirit of Theorem 3.1.

Original: Ignoring some constant irrelevant to the optimization, we rewrite the optimization problem of our approach with the JS divergence as follows:
Revised: Ignoring some constant irrelevant to the optimization, we rewrite the optimization problem of our approach with the JS divergence as follows:

Original: [Equation]
Revised: [Equation]

Original: subject 
Revised: subject to (cid:90) X p g ( x [Equation] dx

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: [Equation]

Original: For clarity, we denote the functional in problem Eq.
Revised: For clarity, we denote the functional in problem Eq.

Original: (23) to be optimized as
Revised: (23) to be optimized as

Original: [Equation]
Revised: [Equation]

Original:  Similarly to the proof of Theorem 3.1, the global minimum of J ( p g ) exists in the feasible area due to Assumption A.1 and the fact that the JS divergence is lower semi-continuous in the topology of weak convergence.
Revised: [Equation] Similarly to the proof of Theorem 3.1, the global minimum of J ( p g ) exists in the feasible area due to Assumption A.1 and the fact that the JS divergence is lower semi-continuous in the topology of weak convergence.

Original: Notably, the optimization problem Eq.
Revised: Notably, the optimization problem Eq.

Original: (28) is convex due to the convexity of the JS divergence (Billingsley, 2013).
Revised: (28) is convex due to the convexity of the JS divergence (Billingsley, 2013).

Original: To obtain a necessary condition for the global minima, we get the Lagrangian with the equality constraint:
Revised: To obtain a necessary condition for the global minima, we get the Lagrangian with the equality constraint:

Original: [Equation]
Revised: [Equation]

Original:  where α ∈ R .
Revised: 31) where α ∈ R .

Original: Similarly to the proof of Theorem 3.1, a necessary condition for a global minimum of problem Eq.
Revised: Similarly to the proof of Theorem 3.1, a necessary condition for a global minimum of problem Eq.

Original: (23) is
Revised: (23) is

Original: [Equation]
Revised: [Equation]

Original:  which implies that
Revised: 32) which implies that

Original: [Equation]
Revised: [Equation]

Original: Similarly to the proof in Theorem 3.1, there is at most one α ∗ ∈ R such that (cid:82) X
Revised: Similarly to the proof in Theorem 3. [Equation] there is at most one α ∗ ∈ R such that (cid:82) X

Original: [Equation]
Revised: [Equation]

Original: ) e α + λ E f ( x ) − 1 = 1 , which finishes the proof together with the existence of the global minimum.
Revised: which finishes the proof together with the existence of the global minimum [Equation]

Original: A.2. C ONVERGENCE WITH N EURAL N ETWORKS T RAINED BY (S TOCHASTIC ) G RADIENT D ESCENT
Revised: A.2. C ONVERGENCE WITH N EURAL N ETWORKS T RAINED BY (S TOCHASTIC ) G RADIENT D ESCENT

Original: We establish the convergence of Reg-DGM with over-parameterized neural networks trained by (stochastic) gradient descent upon the general framework (Allen-Zhu et al., 2019).
Revised: We establish the convergence of Reg-DGM with over-parameterized neural networks trained by (stochastic) gradient descent upon the general framework (Allen-Zhu et al., 2019).

Original: Theorem A.2.
Revised: Theorem A.2.

Original: (General convergence guarantee (Allen-Zhu et al., 2019))
Revised: (General convergence guarantee (Allen-Zhu et al., 2019))

Original: For an arbitrary Lipschitzsmooth loss function L , with probability at least 1 − e − Ω(log m ) , a ReLU convolutional neural network with width m and depth l trained by gradient descent with an appropriate learning rate satisfy the following.
Revised: For an arbitrary Lipschitzsmooth loss function L , with probability at least 1 − e − Ω(log m ) , a ReLU convolutional neural network with width m and depth l trained by gradient descent with an appropriate learning rate satisfy the following.

Original: • If L is non-convex, and σ -gradient dominant, then GD finds ϵ -error minimizer in ˜ O
Revised: • If L is non-convex, and σ -gradient dominant, then GD finds ϵ -error minimizer in ˜ O

Original: ( poly ( n, l, log 1 ϵ , 1 σ ))
Revised: ( poly ( n, l, log 1 ϵ , 1 σ ))

Original: iterations as long as m ≥ ˜Ω( poly ( n, l, 1 σ )) .
Revised: iterations as long as m ≥ ˜Ω( poly ( n, l, 1 σ )) .

Original: • If L is non-convex, then SGD finds a point with ||∇ f || ≤ ϵ in ˜ O
Revised: • If L is non-convex, then SGD finds a point with ||∇ f || ≤ ϵ in ˜ O

Original: ( poly ( m, l, log 1 ϵ ))
Revised: ( poly ( m, l, log 1 ϵ ))

Original: iterations as long as m ≥ ˜Ω
Revised: iterations as long as m ≥ ˜Ω

Original: ( poly ( n, l, 1 ϵ )) .
Revised: ( poly ( n, l, 1 ϵ )) .

Original: We assume the following standard smoothness conditions, which can be verified in practice with bounded data and weights.
Revised: We assume the following standard smoothness conditions, which can be verified in practice with bounded data and weights.

Original: Assumption A.3.
Revised: Assumption A.3.

Original: (Smoothness conditions)
Revised: (Smoothness conditions)

Original: ∃ b > 0 such that ∀ θ ∈ Θ , ∀ x ∈ X , p ( θ ; x ) ≥ b. 2.
Revised: ∃ b > 0 such that ∀ θ ∈ Θ , ∀ x ∈ X , p ( θ ; x ) ≥ b. 2.

Original: ∃ L > 0 such that ∀ x ∈ X , ∀ θ ∈ Θ , ∀ θ ′ ∈ Θ , | p ( θ ; x ) − p ( θ ′ ; x ) | ≤ L || θ − θ ′
Revised: ∃ L > 0 such that ∀ x ∈ X , ∀ θ ∈ Θ , ∀ θ ′ ∈ Θ , | p ( θ ; x ) − p ( θ ′ ; x ) | ≤ L || θ − θ ′

Original: || .
Revised: || .

Original: ∃ K > 0 such that ∀ θ ∈ Θ , ∀ θ ′ ∈ Θ , (cid:82) | p ( θ ; x ) − p ( θ ′ ; x ) | dx ≤ K || θ − θ ′
Revised: ∃ K > 0 such that ∀ θ ∈ Θ , ∀ θ ′ ∈ Θ , (cid:82) | p ( θ ; x ) − p ( θ ′ ; x ) | dx ≤ K || θ − θ ′

Original: || .
Revised: || .

Original: [Equation]
Revised: sup x ∈X sup y ∈X || f ( x ) − f ( y ) || 2 ≤ B .

Original: We consider the general density estimation setting where L MLE ( θ ; x i ) :
Revised: We consider the general density estimation setting where L MLE ( θ ; x i ) :

Original: = − log p θ ( x i ) .
Revised: = − log p θ ( x i ) .

Original: Note that the first two conditions in Assumption A.3 directly imply that L MLE ( · ; x ) is Lb -Lipschitz.
Revised: Note that the first two conditions in Assumption A.3 directly imply that L MLE ( · ; x ) is Lb -Lipschitz.

Original: Formally, given a set of samples S =
Revised: Formally, given a set of samples S =

Original: { x i } ni =1 , Reg-DGM optimizes
Revised: { x i } ni =1 , Reg-DGM optimizes

Original: [Equation]
Revised: [Equation]

Original: If E f is independent from the training data x , then the overall loss function is also Lb -Lipschitz and
Revised: If E f is independent from the training data x , then the overall loss function is also Lb -Lipschitz and

Original: Theorem A.2 directly applies.
Revised: Theorem A.2 directly applies.

Original: Otherwise, we can show that the data-dependent regularization used in our experiments is also Lipschitz-smooth.
Revised: Otherwise, we can show that the data-dependent regularization used in our experiments is also Lipschitz-smooth.

Original: By the linearity of expectation, we have
Revised: By the linearity of expectation, we have

Original: [Equation]
Revised: [Equation]

Original: We define L REG ( θ ; x i ) :
Revised: We define L REG ( θ ; x i ) :

Original: = λd E y ∼ p θ || f ( y ) − f ( x i ) || 22 , which is Lipschitz-smooth:
Revised: = λd E y ∼ p θ || f ( y ) − f ( x i ) || 22 , which is Lipschitz-smooth:

Original: [Equation]
Revised: [Equation]

Original: ) − f ( x ) || 22 (cid:12)(cid:12) ≤ λ d (cid:90) | p θ ( y ) − p θ ′ ( y ) |||

Original: f ( x ) − f ( y ) || 2 dy ≤ λ d (sup y ∈X || f ( x ) − f ( y ) || 2 ) (cid:90) | p θ ( y ) − p θ ′ ( y ) | dy ≤ λBK d || θ − θ ′

Original: || .

Original: Therefore, Theorem A.2 applies to Reg-DGM with the data-dependent energy defined in Sec.
Revised: Therefore, Theorem A.2 applies to Reg-DGM with the data-dependent energy defined in Sec.

Original: 4 of the main text.
Revised: 4 of the main text.

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: Our implementation is built upon some publicly available code.
Revised: Our implementation is built upon some publicly available code.

Original: Below, we include the links and please refer to the licenses therein.
Revised: Below, we include the links and please refer to the licenses therein.

Original: B. T OY DATA
Revised: B. T OY DATA

Original: In the toy example for optimization analyses, the data follows a uniform distribution over [0 , 1] .
Revised: In the toy example for optimization analyses, the data follows a uniform distribution over [0 , 1] .

Original: The energy function is defined as E f ( x ) = 0 .
Revised: The energy function is defined as E f ( x ) = 0 .

Original: 7 x + 0 . 9 .
Revised: 7 x + 0 . 9 .

Original: The optimal β ∗ is estimated by numerical integration.
Revised: The optimal β ∗ is estimated by numerical integration.

Original: In the experiments for the Gaussian-fitting example, we set σ 2 = 1 , m = 150 , and ˆ µ PRE − µ ∗ = 0 .by default.
Revised: In the experiments for the Gaussian-fitting example, we set σ 2 = 1 , m = 150 , and ˆ µ PRE − µ ∗ = 0 .by default.

Original: B. GAN S WITH L IMITED D ATA
Revised: B. GAN S WITH L IMITED D ATA

Original: Datasets.
Revised: Datasets.

Original: In our experiments, we use FFHQ (Karras et al., 2019), which consists of 70 , 000 human face images of resolution 256 × 256 , LSUN CAT (Yu et al., 2015), which consists of 200 ,cat images of resolution 256 × 256 , and CIFAR-10 (Krizhevsky et al., 2009), which consists of 50 , 000 natural images of resolution 32 × 32 .
Revised: In our experiments, we use FFHQ (Karras et al., 2019), which consists of 70 , 000 human face images of resolution 256 × 256 , LSUN CAT (Yu et al., 2015), which consists of 200 ,cat images of resolution 256 × 256 , and CIFAR-10 (Krizhevsky et al., 2009), which consists of 50 , 000 natural images of resolution 32 × 32 .

Original: Specifically, we randomly split training subsets of size 1 k and 5 k from full FFHQ and LSUN CAT in the same way as ADA (Karras et al., 2020a).
Revised: Specifically, we randomly split training subsets of size 1 k and 5 k from full FFHQ and LSUN CAT in the same way as ADA (Karras et al., 2020a).

Original: For reproducibility, we directly use the random seed provided by the official implementation of ADA 5 .
Revised: For reproducibility, we directly use the random seed provided by the official implementation of ADA 5 .

Original: In addition, we also experiment on the smaller dataset 100 -shot Obama (Zhao et al., 2020a) with only 100 face images of 256 × 256 resolution.
Revised: In addition, we also experiment on the smaller dataset 100 -shot Obama (Zhao et al., 2020a) with only 100 face images of 256 × 256 resolution.

Original: In all experiments, we do not use x-flips to amplify training data except for combining with APA (Jiang et al., 2021).
Revised: In all experiments, we do not use x-flips to amplify training data except for combining with APA (Jiang et al., 2021).

Original: Metrics.
Revised: Metrics.

Original: To quantitatively evaluate the experimental results, we choose the Fr´echet inception distance (FID) (Heusel et al., 2017) and the kernel inception distance (KID) (Bi´nkowski et al., 2018) as our metrics.
Revised: To quantitatively evaluate the experimental results, we choose the Fr ´ echet inception distance (FID) (Heusel et al., 2017) and the kernel inception distance (KID) (Bi ´ nkowski et al., 2018) as our metrics.

Original: We compute the FID and KID between 50 , 000 generated images and all real images instead of training subsets (Heusel et al., 2017).
Revised: We compute the FID and KID between 50 , 000 generated images and all real images instead of training subsets (Heusel et al., 2017).

Original: Following ADA (Karras et al., 2020a), we report the medium FID and corresponding KID on FFHQ and LSUN CAT and the mean FID with standard deviation on CIFAR- 10 out of 3 runs.{+
+}
Revised: Following ADA (Karras et al., 2020a), we report the medium FID and corresponding KID on FFHQ and LSUN CAT and the mean FID with standard deviation on CIFAR- {+
+}

Original: We record the best FID during training in each run as in ADA (Karras et al., 2020a).
Revised: out of

Original: Base DGM.
Revised: Base DGM.

Original: In particular, the lighter-weight StyleGAN2 is the backbone for FFHQ and LSUN CAT, and the tuning StyleGAN2 is the backbone for CIFAR-10 following ADA (Karras et al., 2020a).
Revised: In particular, the lighter-weight StyleGAN2 is the backbone for FFHQ and LSUN CAT, and the tuning StyleGAN2 is the backbone for CIFAR-10 following ADA (Karras et al., 2020a).

Original: Compared to the official StyleGAN2, the lighter-weight StyleGAN2 has the same performance and less computing cost on the FFHQ and LSUN CAT and the tuning StyleGAN2 is more suitable for
Revised: Compared to the official StyleGAN2, the lighter-weight StyleGAN2 has the same performance and less computing cost on the FFHQ and LSUN CAT and the tuning StyleGAN2 is more suitable for

Original: CIFAR-10 (Karras et al., 2020a).
Revised: CIFAR-10 (Karras et al., 2020a).

Original: Adaptive discriminator augmentation (ADA) (Karras et al., 2020a) is a representative way of data augmentation for GANs under limited data, and its combination with adaptive pseudo augmentation (APA) (Jiang et al., 2021) is the state-of-the-art method for few-shot image generation
Revised: Adaptive discriminator augmentation (ADA) (Karras et al., 2020a) is a representative way of data augmentation for GANs under limited data, and its combination with adaptive pseudo augmentation (APA) (Jiang et al., 2021) is the state-of-the-art method for few-shot image generation

Original: (Li et al., 2022).
Revised: (Li et al., 2022).

Original: Our implementation is based on the official code of ADA 5 and
Revised: Our implementation is based on the official code of ADA 5 and

Original: APA 6 .
Revised: APA 6 .

Original: Pre-trained model.
Revised: Pre-trained model.

Original: We choose the ResNet-18 7 trained on the ImageNet dataset as the pre-trained model by default due to its excellent performance on ImageNet (Deng et al., 2009) and little computational overhead.
Revised: We choose the ResNet-18 7 trained on the ImageNet dataset as the pre-trained model by default due to its excellent performance on ImageNet (Deng et al., 2009) and little computational overhead.

Original: We normalize both the real and fake images based on the mean and standard deviation of training data and then feed them into the classifier.
Revised: We normalize both the real and fake images based on the mean and standard deviation of training data and then feed them into the classifier.

Original: We extract the features of the last fully connected layer in the pre-trained model, which is frozen during training of generative models.
Revised: We extract the features of the last fully connected layer in the pre-trained model, which is frozen during training of generative models.

Original: On CIFAR-10, we interpolate 8 both the fake and real images to a resolution of 256 × 256 after normalization.
Revised: On CIFAR-10, we interpolate 8 both the fake and real images to a resolution of 256 × 256 after normalization.

Original: Other alternative pre-trained models are the image encoder ResNet-50 of the CLIP model (Radford et al., 2021), which is pre-trained on large-scale noisy text-image pairs instead of ImageNet, and the face recognizer Inception-ResNet-v1 (Szegedy et al., 2017) of FaceNet (Schroff et al., 2015), which is pre-trained on the large-scale face dataset VGGFace2 (Cao et al., 2018).
Revised: Other alternative pre-trained models are the image encoder ResNet-50 of the CLIP model (Radford et al., 2021), which is pre-trained on large-scale noisy text-image pairs instead of ImageNet, and the face recognizer Inception-ResNet-v1 (Szegedy et al., 2017) of FaceNet (Schroff et al., 2015), which is pre-trained on the large-scale face dataset VGGFace2 (Cao et al., 2018).

Original: Notably, we replace the last attention pooling layer in the image encoder of CLIP with the global average pooling layer, and we directly pass the image to the face recognizer without operating the face detector in FaceNet to extract cropped faces.
Revised: Notably, we replace the last attention pooling layer in the image encoder of CLIP with the global average pooling layer, and we directly pass the image to the face recognizer without operating the face detector in FaceNet [Table] λ with pre-trained CLIP and FaceNet. [Table]

Revised: et al.

Original: For pre-trained CLIP 9 and FaceNet 10 , we also employ their last layers as feature extractors.
Revised: [Table] [Table] +APA [Table]

Revised: Jiang et al., 2021)

Revised: [Table]

Revised: to extract cropped faces.

Revised: For pre-trained CLIP 9 and FaceNet 10 , we also employ their last layers as feature extractors.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: Some parameters are shown in Tab. 3.
Revised: Some parameters are shown in Tab. 3.

Original: The weight parameter λ controls the strength of our regularization term.
Revised: The weight parameter λ controls the strength of our regularization term.

Original: We choose the weighting hyperparameter λ by performing grid search over [50 , 20 , 10 , 5 , 4 , 2 , 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 .
Revised: We choose the weighting hyperparameter λ by performing grid search over [50 , 20 , 10 , 5 , 4 , 2 , 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 .

Original: for FFHQ and
Revised: for FFHQ and

Original: LSUN CAT, and [1 , 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 , 0 . 0005 , 0 . 0001 , 0 . 00005 , 0 . 00001 , 0 . 000005] for
Revised: LSUN CAT, and [1 , 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 , 0 . 0005 , 0 . 0001 , 0 . 00005 , 0 . 00001 , 0 . 000005] for

Original: CIFAR-10 according to FID following prior work (Karras et al., 2020a).
Revised: CIFAR-10 according to FID following prior work (Karras et al., 2020a).

Original: Other parameters remain the same settings as ADA (Karras et al., 2020a) and APA (Jiang et al., 2021).
Revised: Other parameters remain the same settings as ADA (Karras et al., 2020a) and APA (Jiang et al., 2021).

Original: For pre-trained CLIP and FaceNet, the adopted parameter λ is shown in Tab. 4.
Revised: For pre-trained CLIP and FaceNet, the adopted parameter λ is shown in Tab. 4.

Original: In addition, we simply set λ as 0.1 for
Revised: In addition, we simply set λ as 0.1 for

Original: Reg-ADA trained on 100-shot Obama.
Revised: Reg-ADA trained on 100-shot Obama.

Original: Computing amount.
Revised: Computing amount.

Original: A single experiment can be completed on 8 NVIDIA 2080Ti GPUs.
Revised: A single experiment can be completed on 8 NVIDIA 2080Ti GPUs.

Original: It takesday 6 hours 19 minutes to run our method with ADA on FFHQ or LSUN CAT and 2 days 17 hours 10 minutes on CIFAR-10 at a time.[Table] hyperparameter λ with pre-trained CLIP and FaceNet. [Table]
Revised: It takesday 6 hours 19 minutes to run our method with ADA on FFHQ or LSUN CAT and 2 days 17 hours 10 minutes on CIFAR-10 at a time.

Original: C A DDITIONAL R ESULTS C. S TANDARD D EVIATION ON FFHQ AND LSUN CAT
Revised: C A DDITIONAL R ESULTS C. S TANDARD D EVIATION ON FFHQ AND LSUN CAT

Original: As shown in Tab. 5, we also provide the mean FID and standard deviation on FFHQ and LSUN
Revised: As shown in Tab. 5, we also provide the mean FID and standard deviation on FFHQ and LSUN

Original: CAT.
Revised: CAT

Original: Reg-DGM can reduce the mean FID significantly (compared to the measurement variance) and achieve a similar if not smaller standard deviation.
Revised: Reg-DGM can reduce the mean FID significantly (compared to the measurement variance ) and achieve a similar if not smaller standard deviation [Table]

Revised: [Figure]

Original: C. M ORE S AMPLES OF GAN S
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Figure]

Original: Fig. 4 and Fig.
Revised: Fig. 4 and Fig.

Original: 5 respectively show the samples randomly generated by models with best FID trained on FFHQ-5k and CIFAR-10, using slight truncation as in ADA (Karras et al., 2020a).
Revised: 5 respectively show the samples randomly generated by models with best FID trained on FFHQ-5k and CIFAR-10, using slight truncation as in ADA (Karras et al., 2020a).

Original: Reg-DGM produces samples of better or comparable quality to the corresponding base DGM.
Revised: Reg-DGM produces samples of better or comparable quality to the corresponding base DGM [Figure] [Figure]

Original: C. L EARNING C URVES
Revised: C. L EARNING C URVES

Original: We show the learning curves of GANs on FFHQ, LSUN CAT and CIFAR-10 in Fig.
Revised: We show the learning curves of GANs on FFHQ, LSUN CAT and CIFAR-10 in Fig.

Original: 6, Fig.
Revised: 6, Fig.

Original: 7 and
Revised: 7 and

Original: Fig. 8 respectively.
Revised: Fig. 8 respectively.

Original: Reg-DGM consistently improves both baselines in all settings and the improvements increase as the number of the training data decreases.
Revised: Reg-DGM consistently improves both baselines in all settings and the improvements increase as the number of the training data decreases.

Original: Moreover, the curves of Reg-DGM generally have a smaller fluctuation, which is consistent with our theory that the regularization reduces the variance of the baselines.
Revised: Moreover, the curves of Reg-DGM generally have a smaller fluctuation, which is consistent with our theory that the regularization reduces the variance of the baselines.

Original: One exception is Fig.
Revised: One exception is Fig.

Original: 8 (a), which shows that Reg-DGM is more unstable than the baseline, which is caused by a bad random initialization.
Revised: 8 (a), which shows that Reg-DGM is more unstable than the baseline, which is caused by a bad random initialization.

Original: We mention that the instability of Reg-ADA in Fig.
Revised: We mention that the instability of Reg-ADA in Fig.

Original: 6 (b) is due to that of ADA.
Revised: 6 (b) is due to that of ADA.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: C. A BLATION OF C LASSIFIER AND R ESULTS UNDER M ORE E VALUATION M ETRICS
Revised: C. A BLATION OF C LASSIFIER AND R ESULTS UNDER M ORE E VALUATION M ETRICS

Original: In this section

Original: [Table] to better understand the influence of classifiers on our method [Table] we try to use pretrained classifiers with different regularization terms, layers, and backbones.
Revised: In this section, to better understand the influence of classifiers on our method, we try to use pretrained classifiers with different regularization terms, layers, and backbones.

Original: We retain the same experimental setting as in Tab. 3.
Revised: We retain the same experimental setting as in Tab. 3.

Original: [Figure]

Original: [Figure]

Original: Regularization form.
Revised: Regularization form.

Original: We first investigate the feature matching objective (Salimans et al., 2016) as an alternative regularization term.
Revised: We first investigate the feature matching objective (Salimans et al., 2016) as an alternative regularization term.

Original: Formally, it computes the square of the l 2 -norm between expected features of real and fake samples from one layer of a feature extractor f , which can be representedas follows:
Revised: Formally, it computes the square of the l 2 -norm between expected features of real and fake samples from one layer of a feature extractor f , which can be represented

Original: [Equation] [Figure]
Revised: [Figure] [Figure]

Revised: as follows:

Original: [Figure]
Revised: [Equation]

Original: Note that the feature matching objective cannot be rewritten as the expectation over p g and thus cannot be understood as an energy function.
Revised: Note that the feature matching objective cannot be rewritten as the expectation over p g and thus cannot be understood as an energy function.

Original: As before, we adopt the last fully-connected layer of ResNet-18, and the results of StyleGAN2 regularized by Eq.
Revised: As before, we adopt the last fully-connected layer of ResNet-18, and the results of StyleGAN2 regularized by Eq.

Original: (36) are shown in Tab. 6.
Revised: (36) are shown in Tab. 6.

Original: Feature matching can greatly reduce FID of StyleGAN2 while it cannot improve the visual quality of the samples, as shown in Fig.
Revised: Feature matching can greatly reduce FID of StyleGAN2 while it cannot improve the visual quality of the samples, as shown in Fig.

Original: Then, we evaluate the entropy-minimization regularization (Grandvalet & Bengio, 2004) as follows:
Revised: Then, we evaluate the entropy-minimization regularization (Grandvalet & Bengio, 2004) as follows:

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) outputs the logits for the prediction distribution.
Revised: where f ( x ) outputs the logits for the prediction distribution.

Original: As shown in Tab. 7, the entropy regularization achieves similar FID results to the baseline within a small search space of λ , showing the importance of the data dependency in the energy function.
Revised: As shown in Tab. 7, the entropy regularization achieves similar FID results to the baseline within a small search space of λ , showing the importance of the data dependency in the energy function.

Original: Layers in f .
Revised: Layers in f .

Original: To explore the different layers of a pre-trained model, we retrain GANs separately using the first convolution layer of ResNet-18 and the last layers of four modules in ResNet-18.
Revised: To explore the different layers of a pre-trained model, we retrain GANs separately using the first convolution layer of ResNet-18 and the last layers of four modules in ResNet-18.

Original: As shown in Tab. 8, our method with features of diverse single layers can all improve the baseline
Revised: As shown in Tab. 8, our method with features of diverse single layers can all improve the baseline

Original: StyleGAN2, and the last layer of ResNet-18 is most beneficial for our regularization strategy.
Revised: StyleGAN2, and the last layer of ResNet-18 is most beneficial for our regularization strategy.

Original: Backbone of f .
Revised: Backbone of f .

Original: We employ ResNet-50 and ResNet-101 as feature extractors to explore the effect of different backbones on Reg-DGM.
Revised: We employ ResNet-50 and ResNet-101 as feature extractors to explore the effect of different backbones on Reg-DGM.

Original: Tab.
Revised: Tab.

Original: 9 shows the results on FFHQ- 5 k. Even using the default λ without tuning, Reg-DGM with ResNet-50 and ResNet-101 can achieve a similar FID to that of our default setting and outperform the baseline.
Revised: 9 shows the results on FFHQ- 5 k. Even using the default λ without tuning, Reg-DGM with ResNet-50 and ResNet-101 can achieve a similar FID to that of our default setting and outperform the baseline.

Original: We believe that Reg-DGM with ResNet-50 and
Revised: We believe that Reg-DGM with ResNet-50 and

Original: ResNet-101 can get better results if we finely search the hyperparameter λ .
Revised: ResNet-101 can get better results if we finely search the hyperparameter λ .

Original: Monte Carlo estimate of f .
Revised: Monte Carlo estimate of f .

Original: We evaluate Reg-StyleGAN2 with 8 , 16 , 32 , and 64 (the default batch size) samples to estimate the energy function in Eq.
Revised: We evaluate Reg-StyleGAN2 with 8 , 16 , 32 , and 64 (the default batch size) samples to estimate the energy function in Eq.

Original: (5) of the main text, as shown in Tab. 10.
Revised: (5) of the main text, as shown in Tab. 10.

Original: We do not observe a significant improvement by increasing the number of samples.
Revised: We do not observe a significant improvement by increasing the number of samples.

Original: For instance, when λ = 1 , the estimate with 64 samples achieves an FID of 38 . 10 , which is similar to 37 .
Revised: For instance, when λ = 1 , the estimate with 64 samples achieves an FID of 38 . 10 , which is similar to 37 .

Original: 77 of the single sample estimate.
Revised: 77 of the single sample estimate.

Original: Intuitively, the features of faces are likely concentrated in a small area of the feature space of f , which is discriminative to other classes of natural images like cars, making the variance negligible to the training process of the generative model. 
Revised: Intuitively, the features of faces are likely concentrated in a small area of the feature space of f , which is discriminative to other classes of natural images like cars, making the variance negligible to the training process of the generative model. [Figure]

Original: KID metric.
Revised: KID metric.

Original: For a comprehensive comparison of Reg-DGM and baselines, we also evaluate the models under KID in Tab. 11.
Revised: For a comprehensive comparison of Reg-DGM and baselines, we also evaluate the models under KID in Tab. 11.

Original: The conclusion remains the same as FID.
Revised: The conclusion remains the same as FID.

Original: Namely, Reg-DGM consistently improves baselines in all settings.
Revised: Namely, Reg-DGM consistently improves baselines in all settings.

Original: [Table]

Original: Human evaluation.
Revised: Human evaluation.

Original: We compare the quality of generated images from the human perspective by
Revised: We compare the quality of generated images from the human perspective by

Original: Amazon Mechanical Turk (AMT) as in prior work (Choi et al., 2020).
Revised: Amazon Mechanical Turk (AMT) as in prior work (Choi et al., 2020).

Original: In particular, we randomly generate 1 , 000 pairs of images by StyleGAN2 and Reg-StyleGAN2 trained on FFHQ- 5 k using the truncation trick ( ψ = 0 . 7 ).
Revised: In particular, we randomly generate 1 , 000 pairs of images by StyleGAN2 and Reg-StyleGAN2 trained on FFHQ- 5 k using the truncation trick ( ψ = 0 . 7 ).

Original: For each image pair, three unique workers in AMT choose the image with the more realistic and high-quality face image.
Revised: For each image pair, three unique workers in AMT choose the image with the more realistic and high-quality face image.

Original: Finally, 3 , 000 selection tasks are completed byworkers within 14 hours.
Revised: Finally, 3 , 000 selection tasks are completed byworkers within 14 hours.

Original: As consistent with the comparison of image quality between Fig. 4 (a) and
Revised: As consistent with the comparison of image quality between Fig. 4 (a) and

Original: Fig.
Revised: Fig.

Original: 4 (b), Reg-StyleGAN2 is chosen in 63 .
Revised: 4 (b), Reg-StyleGAN2 is chosen in 63 .

Original: 5% of the 3 , 000 tasks, and other statistical information is shown in Tab.
Revised: 5% of the 3 , 000 tasks, and other statistical information is shown in Tab.

Original: Results on more training sets with different size.
Revised: Results on more training sets with different size.

Original: To explore the influence of our method on training data with different sizes, we test the Reg-StyleGAN2 with the pre-trained ResNet-18 on different subsets of FFHQ.
Revised: To explore the influence of our method on training data with different sizes, we test the Reg-StyleGAN2 with the pre-trained ResNet-18 on different subsets of FFHQ.

Original: The results are shown in Tab. 13 and the hyperparameter λ is simply fixed as 1 for new data settings 100 , 2 k, 7 k, 10 k, and 15 k. Without heavily tuning the hyperparameter λ , Reg-StyleGAN2 shows consistent improvements over the baseline under the FID metric.
Revised: The results are shown in Tab. 13 and the hyperparameter λ is simply fixed as 1 for new data settings 100 , 2 k, 7 k, 10 k, and 15 k. Without heavily tuning the hyperparameter λ , Reg-StyleGAN2 shows consistent improvements over the baseline under the FID metric.

Original: There seems to be a trend that our method improves more on smaller datasets, which agrees with our
Revised: There seems to be a trend that our method improves more on smaller datasets, which agrees with our

Original: Gaussian case illustrated by the Fig.
Revised: Gaussian case illustrated by the Fig.

Original: 1 (b).
Revised: 1 (b).

Original: C. S ENSITIVITY A NALYSIS OF THE W EIGHTING H YPERPARAMETER
Revised: C. S ENSITIVITY A NALYSIS OF THE W EIGHTING H YPERPARAMETER

Original: We empirically analyze the sensitivity of the weighting hyperparameter λ on FFHQ- 5 k with StyleGAN2 as the base model in Tab. 14.
Revised: We empirically analyze the sensitivity of the weighting hyperparameter λ on FFHQ- 5 k with StyleGAN2 as the base model in Tab. 14.

Original: It can be seen that λ affects the performance significantly.
Revised: It can be seen that λ affects the performance significantly.

Original: Notably, although it is nearly impossible to get the optimal λ via grid search, there is a range of λ (e.g. from 0 .
Revised: Notably, although it is nearly impossible to get the optimal λ via grid search, there is a range of λ (e.g. from 0 .

Original: 01 to 1 in Tab. 14) such that Reg-DGM outperforms the base DGM, which agrees with  our theoretical analyses.
Revised: 01 to 1 in Tab. 14) such that Reg-DGM outperforms the base DGM, which agrees with [Table] [Table] our theoretical analyses.

Original: Besides, Reg-DGM is not too sensitive when the value of λ is around the optimum.
Revised: Besides, Reg-DGM is not too sensitive when the value of λ is around the optimum.

Original: For instance, the gap between the results of λ = 0 .
Revised: For instance, the gap between the results of λ = 0 .

Original: 1 and λ = 1 in Tab.
Revised: 1 and λ = 1 in Tab.

Original: 14 is much smaller than their gain compared to the baseline.
Revised: 14 is much smaller than their gain compared to the baseline.

Original: The performance of Reg-DGM deteriorates with a large λ in Tab.
Revised: The performance of Reg-DGM deteriorates with a large λ in Tab.

Original: 14 as expected.
Revised: 14 as expected.

Original: Proposition 2.2 shows that Reg-DGM is preferable if and only if its value is in a proper interval.
Revised: Proposition 2.2 shows that Reg-DGM is preferable if and only if its value is in a proper interval.

Original: Intuitively, a very large value means that we almost ignore the training data, which should lead to inferior performance.
Revised: Intuitively, a very large value means that we almost ignore the training data, which should lead to inferior performance.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] at least i workers think StyleGAN2 or Reg-StyleGAN2 generates more realistic face images.

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: Values of λ λ =

Original: [Table]
Revised: [Table]

Revised: (base DGM

Original: [Table]
Revised: [Table]

Revised: λ = 0 .λ

Original: [Table]{+= 
 .λ = λ =λ =+}
Revised: {+= 
 .λ = λ =λ =+}

Original: [Table]
Revised: [Table]