Original: S TABLE TARGET FIELD FOR REDUCED VARIANCE SCORE ESTIMATION Anonymous authors Paper under double-blind review
Revised: S TABLE TARGET FIELD FOR REDUCED VARIANCE SCORE ESTIMATION IN DIFFUSION MODELS

Original: A BSTRACT
Revised: A BSTRACT

Original: Score-based generative models(SGMs) generate samples by reversing a fixed forward diffusion process.
Revised: Diffusion models generate samples by reversing a ﬁxed forward diffusion process.

Original: Despite already providing impressive empirical results, these SGM algorithms can be further improved by reducing the variance of the training targets in their  score-matching objective.
Revised: Despite already providing impressive empirical results, these diffusion models algorithms can be further improved by reducing the variance of the training targets in their denoising score-matching objective.

Original: We argue that the source of such variance lies in the handling of intermediate noise-variance scales, where multiple modes in the data affect the direction of reverse paths.
Revised: We argue that the source of such variance lies in the handling of intermediate noise-variance scales, where multiple modes in the data affect the direction of reverse paths.

Original: We propose to remedy the problem by incorporating a reference batch which we use to calculate weighted conditional scores as more stable training targets.
Revised: We propose to remedy the problem by incorporating a reference batch which we use to calculate weighted conditional scores as more stable training targets.

Original: We show that the procedure indeed helps in the challenging intermediate regime by reducing (the trace of) the covariance of training targets.
Revised: We show that the procedure indeed helps in the challenging intermediate regime by reducing (the trace of) the covariance of training targets.

Original: The new stable targets can be seen as trading bias for reduced variance, where the bias vanishes with increasing reference batch size.
Revised: The new stable targets can be seen as trading bias for reduced variance, where the bias vanishes with increasing reference batch size.

Original: Empirically, we show that the new objective improves the image quality, stability, and training speed of various popular SGMs across datasets with both general ODE and SDE solvers.
Revised: Empirically, we show that the new objective improves the image quality, stability, and training speed of various popular diffusion models across datasets with both general ODE and SDE solvers.

Original: When used in combination with state-of-the-art SGMs (EDM (Karras et al., 2022)), our method yields a new SOTA FID of 1. with 35 network evaluations on the unconditional CIFAR-10 generation task. 
Revised: When used in combination with EDM (Karras et al., 2022), our method yields a current SOTA FID of 1.90 with 35 network evaluations on the unconditional CIFAR-10 generation task. The code is available at https://github.com/Newbeeer/stf

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Score-based generative models (SGMs) (Song & Ermon, 2019) and diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) have recently achieved impressive results on a wide spectrum of generative tasks, such as image generation (Nichol et al., 2022; Song et al., 2021b),
Revised: Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020) have recently achieved impressive results on a wide spectrum of generative tasks, such as image generation (Nichol et al., 2022; Song et al., 2021b),

Original: 3D point cloud generation (Luo & Hu, 2021) and molecular conformer generation (Shi et al., 2021; Xu et al., 2022a).
Revised: 3D point cloud generation (Luo & Hu, 2021) and molecular conformer generation (Shi et al., 2021; Xu et al., 2022a).

Original: These models can be subsumed under a unified framework in the form of It ˆ o stochastic differential equations (SDE) (Song et al., 2021b).
Revised: These models can be subsumed under a uniﬁed framework in the form of Itˆo stochastic differential equations (SDE) (Song et al., 2021b).

Original: The models learn time-dependent score fields via score-matching (Hyv¨arinen & Dayan, 2005), which then guides the reverse SDE during generative sampling.
Revised: The models learn time-dependent score ﬁelds via score-matching (Hyv¨arinen & Dayan, 2005), which then guides the reverse SDE during generative sampling.

Original: Popular instances of SGMs include variance-exploding (VE) and variance-preserving (VP) SDE (Song et al., 2021b).
Revised: Popular instances of diffusion models include variance-exploding (VE) and variance-preserving (VP) SDE (Song et al., 2021b).

Original: Building on these formulations, EDM (Karras et al., 2022) provides the best performance to date.
Revised: Building on these formulations, EDM (Karras et al., 2022) provides the best performance to date.

Original: We argue that, despite achieving impressive empirical results, the current training scheme of SGMs can be further improved.
Revised: We argue that, despite achieving impressive empirical results, the current training scheme of diffusion models can be further improved.

Original: In particular, the variance of training targets in the score-matching objective can be large and lead to suboptimal performance.
Revised: In particular, the variance of training targets in the denoising scorematching ( DSM ) objective can be large and lead to suboptimal performance.

Original: To better understand the origin of this instability, we decompose the score field into three regimes.
Revised: To better understand the origin of this instability, we decompose the score ﬁeld into three regimes.

Original: Our analysis shows that the phenomenon arises primarily in the intermediate regime, which is characterized by multiple modes or data points exhibiting comparable influences on the scores.
Revised: Our analysis shows that the phenomenon arises primarily in the intermediate regime, which is characterized by multiple modes or data points exerting comparable inﬂuences on the scores.

Original: In other words, in this regime, the sources of the noisy examples generated in the course of the forward process become ambiguous.
Revised: In other words, in this regime, the sources of the noisy examples generated in the course of the forward process become ambiguous.

Original: We illustrate the problem in Figure 1(a), where each stochastic update of the score model is based on disparate targets.
Revised: We illustrate the problem in Figure 1(a), where each stochastic update of the score model is based on disparate targets.

Original: We propose a generalized version of the vanilla score-matching objective, termed the Stable Target
Revised: We propose a generalized version of the denoising score-matching objective, termed the Stable Target

Original: Field ( STF ) objective.
Revised: Field ( STF ) objective.

Original: The idea is to include an additional reference batch of examples that are used to calculate weighted conditional scores as targets.
Revised: The idea is to include an additional reference batch of examples that are used to calculate weighted conditional scores as targets.

Original: We apply self-normalized importance sampling to aggregate the contribution of each example in the reference batch.
Revised: We apply self-normalized importance sampling to aggregate the contribution of each example in the reference batch.

Original: Although this process can substantially reduce the variance of training targets (Figure 1(b)), especially in the intermediate regime,  it does introduce some bias.
Revised: Although this process can substantially reduce the variance of training targets (Figure 1(b)), especially in the intermediate regime, ⇤ Equal Contribution. [Figure] it does introduce some bias.

Original: Nevertheless, we show that the bias together with the traceof-covariance of the STF training targets shrinks to zero as we increase the size of the reference batch.
Revised: However, we show that the bias together with the trace-of-covariance of the STF training targets shrinks to zero as we increase the size of the reference batch.

Original: Experimentally, we show that our STF objective achieves new state-of-the-art performance on CIFAR10 unconditional generation when incorporated into EDM (Karras et al., 2022).
Revised: Experimentally, we show that our STF objective achieves new state-of-the-art performance on CIFAR10 unconditional generation when incorporated into EDM (Karras et al., 2022).

Original: The resulting FID[Figure] score (Heusel et al., 2017) is 1 .
Revised: The resulting FID score (Heusel et al., 2017) is 1 .

Original: 90 with 35 network evaluations.
Revised: with 35 network evaluations.

Original: STF also improves the FID/Inception scores for other variants of score-based models, i.e . , VE and VP SDEs (Song et al., 2021b), in most cases.
Revised: STF also improves the FID/Inception scores for other variants of score-based models, i.e . , VE and VP SDEs (Song et al., 2021b), in most cases.

Original: In addition, it enhances the stability of converged score-based models on CIFAR-10 and
Revised: In addition, it enhances the stability of converged score-based models on CIFAR-10 and

Original: CelebA 64 2 across random seeds, and helps avoid generating noisy images in VE.
Revised: CelebA 64 2 across random seeds, and helps avoid generating noisy images in VE.

Original: STF accelerates the training of score-based models ( 3 . 6 × speed-up for VE on CIFAR-10) while obtaining comparable or better FID scores.
Revised: STF accelerates the training of score-based models ( 3 . 6 ⇥ speed-up for VE on CIFAR-10) while obtaining comparable or better FID scores.

Original: To the best of our knowledge, STF is the first technique to accelerate the training process of SGMs.
Revised: To the best of our knowledge, STF is the ﬁrst technique to accelerate the training process of diffusion models.

Original: We further demonstrate the performance gain with increasing reference batch size, highlighting the negative effect of large variance.
Revised: We further demonstrate the performance gain with increasing reference batch size, highlighting the negative effect of large variance.

Original: Our contributions are summarized as follows: (1) We detail the instability of the current SGMs training objective in a principled and quantitative manner, characterizing a region in the forward process, termed the intermediate phase , where the score-learning targets are most variable (Section 3).
Revised: Our contributions are summarized as follows: (1) We detail the instability of the current diffusion models training objective in a principled and quantitative manner, characterizing a region in the forward process, termed the intermediate phase , where the score-learning targets are most variable (Section 3).

Original: (2) We propose a generalized score-matching objective, stable target field , which provides more stable training targets (Section 4).
Revised: (2) We propose a generalized score-matching objective, stable target ﬁeld , which provides more stable training targets (Section 4).

Original: (3) We analyze the behavior of the new objective and prove that it is asymptotically unbiased and reduces the trace-of-covariance of the training targets by a factor pertaining to the reference batch size in the intermediate phase under mild conditions (Section 5).
Revised: (3) We analyze the behavior of the new objective and prove that it is asymptotically unbiased and reduces the trace-of-covariance of the training targets by a factor pertaining to the reference batch size in the intermediate phase under mild conditions (Section 5).

Original: (4) We illustrate the theoretical arguments empirically and show that the proposed STF objective improves the performance, stability, and training speed of score-based methods.
Revised: (4) We illustrate the theoretical arguments empirically and show that the proposed STF objective improves the performance, stability, and training speed of score-based methods.

Original: In particular, it achieves the current state-of-the-art FID score on the CIFAR-10 benchmark when combined with
Revised: In particular, it achieves the current state-of-the-art FID score on the CIFAR-10 benchmark when combined with

Original: EDM (Section 6).
Revised: EDM (Section 6).

Original:  B ACKGROUND ON SCORE - BASED GENERATIVE MODELS
Revised:  B ACKGROUND ON DIFFUSION MODELS

Original: In SGMs, the forward process 1 is an SDE with no learned parameter, in the form of:
Revised: In diffusion models, the forward process 1 is an SDE with no learned parameter, in the form of:

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R d with x (0) ∼ p 0 being the data distribution, t ∈ [0 , 1] , f : R d × [0 , 1] → R d , g : [0 , 1] → R , and w ∈ R
Revised: where x 2 R d with x (0) ⇠ p 0 being the data distribution, t 2 [0 , 1] , f : R d ⇥ [0 , 1] ! R d , g : [0 , 1] !

Original:  d is the standard Wiener process.
Revised: R , and w 2 R d is the standard Wiener process.

Original: It gradually transforms the data distribution to a known prior as time goes from 0 to 1.
Revised: It gradually transforms the data distribution to a known prior as time goes from 0 to 1.

Original: Sampling of SGMs is done via a corresponding reverse-time SDE (Anderson, 1982):
Revised: Sampling of diffusion models is done via a corresponding reverse-time SDE (Anderson, 1982):

Original: [Equation]
Revised: [Equation]

Original: where ¯ · denotes time traveling backward from 1 to 0.
Revised: where ¯ · denotes time traveling backward from 1 to 0.

Original: Song et al.
Revised: Song et al.

Original: (2021b) propose a probability flow ODE that induces the same marginal distribution p t ( x ) as the SDE: 
Revised: (2021b) proposes a probability ﬂow ODE that induces the same marginal distribution p t ( x ) as the SDE: d x =

Original: [Equation]
Revised: [Equation]

Revised: ⇤ d¯ t .

Original: Both formulations progressively recover p 0 from the prior p 1 .
Revised: Both formulations progressively recover p 0 from the prior p 1 .

Original: We estimate the score of the transformed data distribution at time t , ∇ x log p t ( x ) , via a neural network, s θ ( x , t ) .
Revised: We estimate the score of the transformed data distribution at time t , r x log p t ( x ) , via a neural network, s ✓ ( x , t ) .

Original: Specifically, the training objective is a weighted sum of the denoising score-matching (Vincent, 2011):
Revised: Speciﬁcally, the training objective is a weighted sum of the denoising score-matching (Vincent, 2011):

Original: [Equation]
Revised: [Equation]

Original: where q t is the distribution for time variable, e.g . , U [0 , 1] for VE/VP (Song et al., 2021b), and λ ( t )
Revised: where q t is the distribution for time variable, e.g . , U [0 , 1] for VE/VP (Song et al., 2021b) and a log-normal distribution for EDM Karras et al. (2022), and  ( t )

Original: =
Revised: =

Original: σ 2 t is the positive weighting function to keep the time-dependent loss at the same magnitude (Song et al., 2021b), and p t | 0 ( x ( t ) | x ) is the transition kernel denoting the conditional distribution of x ( t ) given x 2 .
Revised:  2 t is the positive weighting function to keep the time-dependent loss at the same magnitude (Song et al., 2021b), and p t | 0 ( x ( t ) | x ) is the transition kernel denoting the conditional distribution of x ( t ) given x 2 .

Original: Specifically, SGMs “destroy” data according to a diffusion process utilizing Gaussian transition kernels, which result in p t | 0 ( x ( t ) | x ) =
Revised: Speciﬁcally, diffusion models “destroy” data according to a diffusion process utilizing Gaussian transition kernels, which result in p t | 0 ( x ( t ) | x ) =

Original: N ( µ t , σ 2 t I ) .
Revised: N ( µ t ,  2 t I ) .

Original: Recent works (Xu et al., 2022b; Rissanen et al., 2022) have also extended the underlying principle from the diffusion process to more general physical processes where the training objective is not necessarily score-related.
Revised: Recent works (Xu et al., 2022b; Rissanen et al., 2022) have also extended the underlying principle from the diffusion process to more general physical processes where the training objective is not necessarily score-related.

Original:  U NDERSTANDING THE TRAINING TARGET IN SCORE - MATCHING OBJECTIVE
Revised:  U NDERSTANDING THE TRAINING TARGET IN SCORE - MATCHING OBJECTIVE

Original: The vanilla denoising score-matching objective at time t is:
Revised: The vanilla denoising score-matching objective at time t is:

Original: [Equation]
Revised: [Equation]

Original: where the network is trained to fit the individual targets ∇ x ( [Equation] ) log p [Equation]
Revised: where the network is trained to ﬁt the individual targets r x ( t ) log p t | 0 ( x ( t ) |

Original: t

Original: [Equation]

Original: | x ) at

Original: [Equation]

Original: x  ( t [Equation] [Equation] the “influence” exerted by clean data x on x ( t ) .
Revised: x ) at ( x ( t ) , t ) – the “inﬂuence” exerted by clean data x on x ( t ) .

Original: We can swap the order of the sampling process by first sampling x ( t ) from p t and then x from p 0 | t ( ·| x ( t )) .
Revised: We can swap the order of the sampling process by ﬁrst sampling x ( t ) from p t and then x from p 0 | t ( ·| x ( t )) .

Original: Thus, s θ has a closed form minimizer:
Revised: Thus, s ✓ has a closed form minimizer:

Original: [Equation]
Revised: [Equation]

Original: The score field is a conditional expectation of ∇ x ( t ) log p t | 0 ( x ( t )
Revised: The score ﬁeld is a conditional expectation of r x ( t ) log p t | 0 ( x ( t )

Original: | x ) with respect to the posterior distribution p 0 | t .
Revised: | x ) with respect to the posterior distribution p 0 | t .

Original: In practice, a Monte Carlo estimate of this target can have high variance (Owen, 2013; Elvira & Martino, 2021).
Revised: In practice, a Monte Carlo estimate of this target can have high variance (Owen, 2013; Elvira & Martino, 2021).

Original: In particular, when multiple modes of the data distribution have comparable influences on x ( t ) , p 0 | t ( ·| x ( t )) is a multi-mode distribution, as also observed in Xiao et al.
Revised: In particular, when multiple modes of the data distribution have comparable inﬂuences on x ( t ) , p 0 | t ( ·| x ( t )) is a multi-mode distribution, as also observed in Xiao et al.

Original: Thus the targets ∇ x ( t ) log p t | 0 ( x ( t ) | x ) vary considerably across different x
Revised: Thus the targets r x ( t ) log p t | 0 ( x ( t ) | x ) vary considerably across different x

Original: and this can strongly affect the estimated score at ( x ( t ) , t ) , resulting in slower convergence and worse performance in practical stochastic gradient optimization (Wang et al., 2013).
Revised: and this can strongly affect the estimated score at ( x ( t ) , t ) , resulting in slower convergence and worse performance in practical stochastic gradient optimization (Wang et al., 2013).

Original: To quantitatively characterize the variations of individual targets at different time, we propose a metric – the average trace-of-covariance of training targets at time t :
Revised: To quantitatively characterize the variations of individual targets at different time, we propose a metric – the average trace-of-covariance of training targets at time t :

Original: [Equation]
Revised: [Equation]

Revised: ⇤ .

Original: [Equation]
Revised: [Figure]

Original: We use V v ( t ) to define three successive phases relating to the behavior of training targets.
Revised: We use V DSM ( t ) to deﬁne three successive phases relating to the behavior of training targets.

Original: As shown in Figure 2(a), the three phases partition the score field into near, intermediate, and far regimes (Phase 1 ∼ 3 respectively).
Revised: As shown in Figure 2(a), the three phases partition the score ﬁeld into near, intermediate, and far regimes (Phase 1 ⇠ 3 respectively).

Original: Intuitively, V v ( t ) peaks in the intermediate phase (Phase 2), where multiple distant modes in the data distribution have comparable influences on the same noisy perturbations, resulting in unstable targets.
Revised: Intu- itively, V DSM ( t ) peaks in the intermediate phase (Phase 2), where multiple distant modes in the data distribution have comparable inﬂuences on the same noisy perturbations, resulting in unstable targets.

Original: In Phase 1, the posterior p 0 | t concentrates around one single mode, thus low variation.
Revised: In Phase 1, the posterior p 0 | t concentrates around one single mode, thus low variation.

Original: In Phase 3, the targets remain similar across modes since lim t → 1 p t | 0 ( x ( t ) | x ) ≈ p 1 for commonly used transition kernels.
Revised: In Phase 3, the targets remain similar across modes since lim t ! 1 p t | 0 ( x ( t ) | x ) ⇡ p 1 for commonly used transition kernels.

Original: [Figure] We validate this argument empirically in Figure 2(b), which shows the estimated V v ( t ) for a mixture of two Gaussians as well as a subset of CIFAR-10 dataset (Krizhevsky et al., 2009) for a more realistic setting.
Revised: We validate this argument empirically in Figure 2(b), which shows the estimated V DSM ( t ) for a mixture of two Gaussians as well as a subset of CIFAR-10 dataset (Krizhevsky et al., 2009) for a more realistic setting.

Original: Here we use VE SDE, i.e . , p t | 0 ( x ( t ) | x ) = N (cid:16) x , σ 2 m ( σ M σ m ) 2 t I (cid:17) for some σ m and σ M
Revised: Here we use VE SDE, i.e . , p t | 0 ( x ( t ) | x ) = N ⇣ x ,  2 m (  M  m ) 2 t I ⌘ for some  m and  M

Original: (Song et al., 2021b).
Revised: (Song et al., 2021b).

Original: V v ( t ) exhibits similar phase behavior across t in both toy and realistic cases.
Revised: V DSM ( t ) exhibits similar phase behavior across t in both toy and realistic cases.

Original: Moreover, V v ( t ) reaches its maximum value in the intermediate phase, demonstrating the large variations of individual targets.
Revised: Moreover, V DSM ( t ) reaches its maximum value in the intermediate phase, demonstrating the large variations of individual targets.

Original: We defer more details to Appendix C.
Revised: We defer more details to Appendix C.

Original:  T REATING SCORE AS A FIELD
Revised:  T REATING SCORE AS A FIELD

Original: The vanilla denoising score-matching approach (Equation 3) can be viewed as a Monte Carlo estimator, i [Equation] e . , 
Revised: The vanilla denoising score-matching approach (Equation 3) can be viewed as a Monte Carlo estimator, i.e . , r

Original: [Equation]
Revised: [Equation]

Original: where x i is sampled from p 0 | t ( ·| x ( t )) and n = 1 .
Revised: where x i is sampled from p 0 | t ( ·| x ( t )) and n = 1 .

Original: The variance of a Monte Carlo estimator is proportional to 1 n , so we propose to use a larger batch ( n ) to counter the high variance problem described in Section 3.
Revised: The variance of a Monte Carlo estimator is proportional to 1 n , so we propose to use a larger batch ( n ) to counter the high variance problem described in Section 3.

Original: Since sampling directly from the posterior p 

Revised: Since sampling directly from the posterior p

Original: | t is not practical [Equation] we first apply importance sampling with the proposal distribution p 0 .
Revised: | t is not practical, we ﬁrst apply importance sampling with the proposal distribution p 0 .

Original: Specifically, we sample a large reference batch B L = { x i } ni =1 ∼{+=
 ⇠+} p n 0 and get the following approximation:
Revised: Speciﬁcally, we sample a large reference batch B L = { x i } ni {+=
 ⇠+} p n 0 and get the following approximation:

Original: [Equation]
Revised: [Equation]

Original: The importance weights can be rewritten as p 0 | t ( x | x ( t ))
Revised: The importance weights can be rewritten as p 0 | t ( x | x ( t ))

Original: /p 

Revised: /p

Original: ( x ) =p
Revised: ( x ) =

Original: [Equation]

Original: | 0 ( x

Original: [Equation] ) | x ) [Equation] p t ( x ( t )) .
Revised: p t | 0 ( x ( t ) | x ) /p t ( x ( t )) .

Original: However, this basic importance sampling estimator has two issues.
Revised: However, this basic importance sampling estimator has two issues.

Original: The weights now involve an unknown normalization factor p t ( x ( t )) and the ratio between the prior and posterior distribution can be large in high dimensional spaces.
Revised: The weights now involve an unknown normalization factor p t ( x ( t )) and the ratio between the prior and posterior distribution can be large in high dimensional spaces.

Original: To remedy these problems, we appeal to self-normalization techniques (Hesterberg, 1995) to further stabilize the training targets:
Revised: To remedy these problems, we appeal to self-normalization techniques (Hesterberg, 1995) to further stabilize the training targets:

Original: [Equation]
Revised: [Equation]

Original: We term this new training target in Equation 5 as Stable Target Field (STF) .
Revised: We term this new training target in Equation 5 as Stable Target Field (STF) .

Original: In practice, we sample the reference batch B L = { x i } ni =1 from p n 0 and obtain x ( t ) by applying the transition kernel to the “first” training data x 1 .
Revised: In practice, we sample the reference batch B L = { x i } ni =1 from p n 0 and obtain x ( t ) by applying the transition kernel to the “ﬁrst” training data x 1 .

Original: Taken together, the new STF objective becomes:
Revised: Taken together, the new STF objective becomes:

Original: [Equation]
Revised: [Equation]

Original: When n = 1 , STF reduces to the vanilla  score-matching (Equation 2).
Revised: When n = 1 , STF reduces to the vanilla denoising score-matching (Equation 2).

Original: When n > 1 , STF incorporates a reference batch to stabilize training targets.
Revised: When n > 1 , STF incorporates a reference batch to stabilize training targets.

Original: Intuitively, the new weighted target assigns larger weights to clean data with higher influence on x ( t ) , i.e . , higher transition probability p t | 0 ( x ( t ) | x ) .
Revised: Intuitively, the new weighted target assigns larger weights to clean data with higher inﬂuence on x ( t ) , i.e . , higher transition probability p t | 0 ( x ( t ) | x ) .

Original: Similar to our analysis in Section 3, we can again swap the sampling process in Equation 6 so that, for a perturbation x ( t ) , we sample the reference batch B L = { x i } ni =1 from p 0 | t ( ·| x ( t ))
Revised: Similar to our analysis in Section 3, we can again swap the sampling process in Equation 6 so that, for a perturbation x ( t ) , we sample the reference batch B L = { x i } n i =1 from p 0 | t ( ·| x ( t ))

Original: p n − 1, where the first element involves the posterior, and the rest follow the data distribution.
Revised: p n  1, where the ﬁrst element involves the posterior, and the rest follow the data distribution.

Original: Thus, the minimizer of the new objective (Equation 6) is (derivation can be found in Appendix B.1)
Revised: Thus, the minimizer of the new objective (Equation 6) is (derivation can be found in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: Note that although STF significantly reduces the variance, it introduces bias: the minimizer is no longer the true score.
Revised: Note that although STF signiﬁcantly reduces the variance, it introduces bias: the minimizer is no longer the true score.

Original: Nevertheless, in Section 5, we show that the bias converges to 0 as n → ∞ , while reducing the trace-of-covariance of the training targets by a factor of n when p 0 | t ≈ p 0 .
Revised: Nevertheless, in Section 5, we show that the bias converges to 0 as n ! 1 , while reducing the trace-of-covariance of the training targets by a factor of n when p 0 | t ⇡ p 0 .

Original: We further instantiate the STF objective (Equation 6) with transition kernels in the form of p t | 0 ( x ( t ) | x ) =
Revised: We further instantiate the STF objective (Equation 6) with transition kernels in the form of p t | 0 ( x ( t ) | x ) =

Original: N ( x , σ 2 t I ) , which includes EDM (Karras et al., 2022), VP (through reparameterization) and VE (Song et al., 2021b):
Revised: N ( x ,  2 t I ) , which includes EDM (Karras et al., 2022), VP (through reparameterization) and VE (Song et al., 2021b):

Original: [Equation]
Revised: [Equation]

Original: To aggregate the time-dependent STF objective over t , weuniformly sample the time variable t from [0 , 1] during training  and apply the weighting function λ ( t ) .
Revised: To aggregate the time-dependent STF objective over t , we sample the time variable t from the training distribution q t and apply the weighting function  ( t ) .

Original: Together, the final training objective for 
Revised: Together, the ﬁnal training objective for STF is

Original: STF is E t ∼ q t ( t ) [ λ ( t ) ℓ STF ( θ, t )] .
Revised: E t ⇠ q t ( t ) [  ( t ) ` STF ( ✓ , t )] .

Original: We summarize the training process in Algorithm 1.
Revised: We summarize the training process in Algorithm 1.

Original: The small batch size |B| is the same as the normal batch size in the vanilla training process.
Revised: The small batch size |B| is the same as the normal batch size in the vanilla training process.

Original: We defer specific use cases of STF objectives combined with various popular SGMs to Appendix A.
Revised: We defer speciﬁc use cases of STF objectives combined with various popular diffusion models to Appendix A.

Revised: Input: Training iteration T , Initial model s ✓ , dataset D , learning rate ⌘ .

Revised: for t = 1 . .

Revised: T do

Revised: Sample a large reference

Original: [Table]
Revised: [Table]

Revised: and subsample

Revised: [Table]

Revised: Uniformly

Original: [Table]
Revised: [Table]

Original: A NALYSIS
Revised: Obtain the batch of perturbed samples { x i ( t i ) } |B| i =1 by applying the transition kernel p t | 0 on B Calculate the stable target ﬁeld of B L for all x i ( t i ) : [Table] [Equation]

Revised: [Table]

Revised: Calculate the loss

Revised: [Table]

Revised: Update the

Revised: [Table]

Revised: parameter

Revised: [Table]

Revised: ) end for return s ✓

Revised:  A NALYSIS

Original: In this section, we analyze the theoretical properties of our approach.
Revised: In this section, we analyze the theoretical properties of our approach.

Original: In particular, we show that the new minimizer s ∗ STF ( x ( t ) , t ) (Equation 7) converges to the true score asymptotically (Section 5.1).
Revised: In particular, we show that the new minimizer s ⇤ STF ( x ( t ) , t ) (Equation 7) converges to the true score asymptotically (Section 5.1).

Original: Then, we show that the proposed STF reduces the trace-of-covariance of training targets propositional to the reference batch size in the intermediate phase, with mild conditions (Section 5.2).
Revised: Then, we show that the proposed STF reduces the trace-of-covariance of training targets propositional to the reference batch size in the intermediate phase, with mild conditions (Section 5.2).

Original: 5. A SYMPTOTIC BEHAVIOR
Revised: 5. A SYMPTOTIC BEHAVIOR

Original: Although in general s ∗ STF ( x ( t ) , t ) ̸ = ∇ x ( t ) log p t ( x ( t )) , the bias shrinks toward 0 with a increasing n .
Revised: Although in general s ⇤ STF ( x ( t ) , t ) 6 = r x ( t ) log p t ( x ( t )) , the bias shrinks toward 0 with a increasing n .

Original: In the following theorem we show that the minimizer of STF objective at ( x ( t ) , t ) , i.e . , s ∗ STF ( x ( t ) , t ) , is asymptotically normal when n → ∞ .
Revised: In the following theorem we show that the minimizer of STF objective at ( x ( t ) , t ) , i.e . , s ⇤ STF ( x ( t ) , t ) , is asymptotically normal when n ! 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then
Revised: Suppose 8 t 2 [0 , 1] , 0 <  t < 1 , then

Original: [Equation]
Revised: [Equation]

Original: We defer the proof to Appendix B.2.
Revised: We defer the proof to Appendix B.2.

Original: The theorem states that, for commonly used transition kernels, s ∗ STF ( x ( t ) , t ) − ∇ x ( t ) log p t ( x ( t )) converges to a zero mean normal, and larger reference batch size ( n ) will lead to smaller asymptotic variance.
Revised: The theorem states that, for commonly used transition kernels, s ⇤ STF ( x ( t ) , t )  r x ( t ) log p t ( x ( t )) converges to a zero mean normal, and larger reference batch size ( n ) will lead to smaller asymptotic variance.

Original: As can be seen in Equation 8, when n → ∞ , s ∗ STF ( x ( t ) , t ) highly concentrates around the true score ∇ x ( t ) log p t ( x ( t )) .
Revised: As can be seen in Equation 8, when n ! 1 , s ⇤ STF ( x ( t ) , t ) highly concentrates around the true score r x ( t ) log p t ( x ( t )) .

Original: 5. T RACE OF C OVARIANCE
Revised: 5. T RACE OF C OVARIANCE

Original: We now highlight the small variations of the training targets in the STF objective compared to thevanilla one.
Revised: We now highlight the small variations of the training targets in the STF objective compared to the

Revised: DSM.

Original: As done in Section 3, we study the trace-of-covariance of training targets in STF:
Revised: As done in Section 3, we study the trace-of-covariance of training targets in STF:

Original: [Equation]
Revised: [Equation]

Original: In the following theorem we compare V STF with V v .
Revised: In the following theorem we compare V STF with V DSM .

Original: In particular, we can upper bound V STF ( t ) by
Revised: In particular, we can upper bound V STF ( t ) by

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , thenV STF
Revised: Suppose 8 t 2 [0 , 1] , 0 <  t < 1 , then

Original: [Equation]
Revised: [Equation]

Original: t [Equation] where D f is an f-divergence with f ( y )
Revised: where D f is an f-divergence with f ( y )

Original: = (cid:26) (1 /y − 1) ( y < 1 . 5) 8 y/ 27 − 1 / [-
-] y ≥ 1 . 5) .
Revised: = ⇢ (1 /y  1)( y < 1 . 5) 8 y/ 27  1 / [-
-]3 ( y  1 . 5) .

Original: Further, when n ≫ d and p 0 | t ( x | x ( t )) ≈ p 0 ( x ) for all x ( t ) , V STF ( t ) ⪅ V v ( t ) n − 1 .
Revised: Further, when n  d and p 0 | t ( x | x ( t )) ⇡ p 0 ( x ) for all x ( t ) , V STF ( t ) / V DSM ( t ) n  1 .

Original: We defer the proof to Appendix B.3.
Revised: We defer the proof to Appendix B.3.

Original: The second term that involves f -divergence D f is necessary to capture how the coefficients, i.e . , p t | 0 ( x ( t )
Revised: The second term that involves f -divergence D f is necessary to capture how the coefﬁcients, i.e . , p t | 0 ( x ( t )

Original: | x k ) /
Revised: | x k ) /

Original: (cid:80) j p t | 0 ( x ( t ) | x j ) used to calculate the weighted score target, vary across different samples x ( t ) .
Revised: P j p t | 0 ( x ( t ) | x j ) used to calculate the weighted score target, vary across different samples x ( t ) .

Original: This term decreases monotonically as a function of t .
Revised: This term decreases monotonically as a function of t .

Original: In Phase 1, p 0 | t ( x | x ( t )) differs substantially from p 0 ( x ) and the divergence term D f dominates.
Revised: In Phase 1, p 0 | t ( x | x ( t )) differs substantially from p 0 ( x ) and the divergence term D f dominates.

Original: In contrast to the upper bound, both V STF ( t ) and V v ( t ) have minimal variance at small values of t since the training target is always dominated by one x .
Revised: In contrast to the upper bound, both V STF ( t ) and V DSM ( t ) have minimal variance at small values of t since the training target is always dominated by one x .

Original: The theorem has more relevance in Phase 2, where the divergence term decreases to a value comparable to V v ( t ) .
Revised: The theorem has more relevance in Phase 2, where the divergence term decreases to a value comparable to V DSM ( t ) .

Original: In this phase, we empirically observe that the ratio of the two terms in the upper bound ranges from 10 to 100.
Revised: In this phase, we empirically observe that the ratio of the two terms in the upper bound ranges from 10 to 100.

Original: Thus, when we use a large reference batch size (in thousands), the theorem implies that STF offers a considerably lower variance (by a factor of 10 or more) relative to the vanilla objective.
Revised: Thus, when we use a large reference batch size (in thousands), the theorem implies that STF offers a considerably lower variance (by a factor of 10 or more) relative to the DSM objective.

Original: In Phase 3, the second term vanishes to 0, as p t ≈ p t | 0 with large σ t for commonly used transition kernels.
Revised: In Phase 3, the second term vanishes to 0, as p t ⇡ p t | 0 with large  t for commonly used transition kernels.

Original: As a result, STF reduces the average trace-of-covariance of the training targets by at least n − 1 times in the far field.
Revised: As a result, STF reduces the average trace-of-covariance of the training targets by at least n  1 times in the far ﬁeld.

Original: Together, we demonstrate that the STF targets have diminishing bias (Theorem 1) and are much more stable during training (Theorem 2).
Revised: Together, we demonstrate that the STF targets have diminishing bias (Theorem 1) and are much more stable during training (Theorem 2).

Original: These properties make the STF objective more favorable for SGMs training with stochastic gradient optimization.
Revised: These properties make the STF objective more favorable for diffusion models training with stochastic gradient optimization.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we first empirically validate our theoretical analysis in Section 5, especially for variance reduction in the intermediate phase (Section 6.1).
Revised: In this section, we ﬁrst empirically validate our theoretical analysis in Section 5, especially for variance reduction in the intermediate phase (Section 6.1).

Original: Next, we show that the STF objective improves various score-based models on image generation tasks in terms of image quality (Section 6.2).
Revised: Next, we show that the STF objective improves various diffusion models on image generation tasks in terms of image quality (Section 6.2).

Original: In particular, STF achieves state-of-the-art performance on top of EDM.
Revised: In particular, STF achieves state-of-the-art performance on top of EDM.

Original: In addition, we demonstrate that STF accelerates the training of score-based models (Section 6.3), and improves the convergence speed and final performance with an increasing reference batch size (Section 6.3).
Revised: In addition, we demonstrate that STF accelerates the training of diffusion models (Section 6.3), and improves the convergence speed and ﬁnal performance with an increasing reference batch size (Section 6.3).

Original: 6. V ARIANCE REDUCTION IN THE INTERMEDIATE PHASE
Revised: 6. V ARIANCE REDUCTION IN THE INTERMEDIATE PHASE

Original: [Figure]
Revised: [Figure]

Original: The proposed Algorithm 1 utilizes a large reference batch to calculate the stable target field instead of the individual target.
Revised: The proposed Algorithm 1 utilizes a large reference batch to calculate the stable target ﬁeld instead of the individual target.

Original: In addition to the theoretical analysis in Section 5, we provide further empirical study to characterize the intermediate phase and verify the variance reduction effects by STF.
Revised: In addition to the theoretical analysis in Section 5, we provide further empirical study to characterize the intermediate phase and verify the variance reduction effects by STF.

Original: Apart from V ( t ) , we also quantify the average divergence between the posterior p 0 | t ( ·| x ( t )) and the data distribution p 0 at time t (introduced in Theorem 2): D ( t ) = E p t ( x ( t )) (cid:2) D f (cid:0) p 0 | t ( x | x ( t ))
Revised: Apart from V ( t ) , we also quantify the average divergence between the posterior p 0 | t ( ·| x ( t )) and the data distribution p 0 at time t (introduced in Theorem 2): D ( t ) = E p t ( x ( t )) [Equation]

Revised: ⇤

Original: ∥ p 0 ( x ) (cid:1)(cid:3) .
Revised: [Equation]

Original: Intuitively, the number of high-density modes in p 0 | t ( ·| x ( t )) grows as D ( t ) decreases.
Revised: Intuitively, the number of high-density modes in p 0 | t ( ·| x ( t )) grows as D ( t ) decreases.

Original: To investigate their behaviors, we construct two synthetic datasets: (1) a 64-dimensional mixture of two Gaussian components ( Two Gaussians ), and (2) a subset of 4096 images of CIFAR-10 ( CIFAR-10-4096 ).
Revised: To investigate their behaviors, we construct two synthetic datasets: (1) a 64-dimensional mixture of two Gaussian components ( Two Gaussians ), and (2) a subset of 1024 images of CIFAR-10 ( CIFAR-10-4096 ).

Original: Figure 3(a) and Figure 3(b) show the behaviors of V v ( t ) and D ( t ) on Two Gaussian and CIFAR10-4096.
Revised: Figure 3(a) and Figure 3(b) show the behaviors of V DSM ( t ) and D ( t ) on Two Gaussian and CIFAR10-4096.

Original: In both settings, V v ( t ) reaches its peak in the intermediate phase (Phase 2), while D ( t ) gradually decreases over time.
Revised: In both settings, V DSM ( t ) reaches its peak in the intermediate phase (Phase 2), while D ( t ) gradually decreases over time.

Original: These results agree with our theoretical understanding from Section 3.
Revised: These results agree with our theoretical understanding from Section 3.

Original: In Phase 2 and 3, several modes of the data distribution have noticeable influences on the scores, but only in Phase 2 are the influences much more distinct, leading to high variations of the individual target 
Revised: In Phase 2 and 3, several modes of the data distribution have noticeable inﬂuences on the scores, but only in Phase 2 are the inﬂuences much more distinct, leading to high variations of the individual target r x ( t ) log p t | 0 ( x ( t )

Original: ∇ x ( t ) log p t | 0 ( x ( t ) | x ) , x ∼ p 0 | t ( ·| x ( t )) .
Revised: | x ) , x ⇠ p 0 | t ( ·| x ( t )) .

Original: Figure 3(c) and Figure 3(d) further show the relationship between V STF ( t ) and the reference batch size n .
Revised: Figure 3(c) and Figure 3(d) further show the relationship between V STF ( t ) and the reference batch size n .

Original: Recall that when n = 1 , STF degenerates to individual target and V STF ( t ) =
Revised: Recall that when n = 1 , STF degenerates to individual target and V STF ( t ) =

Original: V v ( t ) .
Revised: V DSM ( t ) .

Original: We observe that V STF ( t ) decreases when enlarging n .
Revised: We observe that V STF ( t ) decreases when enlarging n .

Original: In particular, the predicted relation V STF ( t ) ⪅ V v ( t ) / ( n − 1) in Theorem 2 holds for the two Gaussian datasets where D f is small.
Revised: In particular, the predicted relation V STF ( t ) / V DSM ( t ) / ( n  1) in Theorem 2 holds for the two Gaussian datasets where D f is small.

Original: On the high dimensional dataset CIFAR-10-4096, the stable target field can still greatly reduce the training target variance with large reference batch sizes n .
Revised: On the high dimensional dataset CIFAR-10-4096, the stable target ﬁeld can still greatly reduce the training target variance with large reference batch sizes n .

Original: 6. I MAGE GENERATION
Revised: 6. I MAGE GENERATION

Original: [Table]
Revised: [Table]

Original: We demonstrate the effectiveness of the new objective on image generation tasks.
Revised: We demonstrate the effectiveness of the new objective on image generation tasks.

Original: We consider
Revised: We consider

Original: CIFAR-10 (Krizhevsky et al., 2009), CelebA 64 × 64{+⇥ 
+} (Yang et al., 2015)and ImageNet 64 × 64 (Deng et al., 2009) datasets.
Revised: CIFAR-10 (Krizhevsky et al., 2009) and CelebA 64 {+⇥ 
+} (Yang et al., 2015) datasets.

Original: We set the reference batch size n to 4096 (CIFAR10) and 1024 (CelebA 64 2, ImageNet 64 2 ).
Revised: We set the reference batch size n to 4096 (CIFAR-10) and 1024 (CelebA 64 2 ).

Original: We choose the current state-of-the-art score-based method EDM (Karras et al., 2022) as the baseline, and replace the vanilla objective with our STF objective during training.
Revised: We choose the current state-ofthe-art score-based method EDM (Karras et al., 2022) as the baseline, and replace the DSM objective with our STF objective during training.

Original: We also apply STF to two other popular score-based models, VE/VP SDEs (Song et al., 2021b).
Revised: We also apply STF to two other popular diffusion models, VE/VP SDEs (Song et al., 2021b).

Original: For a fair comparison, we directly adopt the architectures and the hyper-parameters in Karras et al.
Revised: For a fair comparison, we directly adopt the architectures and the hyper-parameters in Karras et al.

Original: and Song et al.
Revised: and Song et al.

Original: (2021b) for EDM and VE/VP respectively.
Revised: (2021b) for EDM and VE/VP respectively.

Original: In particular, we use the improved NCSN++/DDPM++ models (Karras et al., 2022) in the EDM scheme.
Revised: In particular, we use the improved NCSN++/DDPM++ models (Karras et al., 2022) in the EDM scheme.

Original: To highlight the stability issue, we train three models with different seeds for VE on CIFAR-10.
Revised: To highlight the stability issue, we train three models with different seeds for VE on CIFAR-10.

Original: We provide more experimental details in Appendix D.1.
Revised: We provide more experimental details in Appendix D.1.

Original: Numerical Solver.
Revised: Numerical Solver.

Original: The reverse-time ODE and SDE in scored-based models are compatible with any general-purpose solvers.
Revised: The reverse-time ODE and SDE in scored-based models are compatible with any general-purpose solvers.

Original: We use the adaptive solver RK45 method (Dormand & Prince, 1980;
Revised: We use the adaptive solver RK45 method (Dormand & Prince, 1980;

Original: Song et al., 2021b)
Revised: Song et al., 2021b)

Original: (RK45) for VE/VP and the popular DDIM solver (Song et al., 2021a) for VP.
Revised: (RK45) for VE/VP and the popular DDIM solver (Song et al., 2021a) for VP.

Original: We adopt Heun’s 2nd order method (Heun) and the time discretization proposed by Karras et al.
Revised: We adopt Heun’s 2nd order method (Heun) and the time discretization proposed by Karras et al.

Original: (2022) for EDM.
Revised: (2022) for EDM.

Original: For SDEs, we apply the predictor-corrector (PC) sampler used in (Song et al., 2021b).
Revised: For SDEs, we apply the predictor-corrector (PC) sampler used in (Song et al., 2021b).

Original: We denote the methods in a objective-sampler format, i.e . , A - B , where A ∈ { vanilla, STF } and B ∈ { RK45, PC, DDIM, Heun } .
Revised: We denote the methods in a objective-sampler format, i.e . , A - B , where A 2 { DSM, STF } and B{ RK45, PC, DDIM, Heun } .

Original: We defer more details to Appendix D.2.
Revised: We defer more details to Appendix D.2.

Original: Results.
Revised: Results.

Original: For quantitative evaluation of the generated samples, we report the FID scores (Heusel et al., 2017) (lower is better) and Inception (Salimans et al., 2016) (higher is better).
Revised: For quantitative evaluation of the generated samples, we report the FID scores (Heusel et al., 2017) (lower is better) and Inception (Salimans et al., 2016) (higher is better).

Original: We measure the sampling speed by the average NFE (number of function evaluations).
Revised: We measure the sampling speed by the average NFE (number of function evaluations).

Original: We also include the results of several popular generative models (Karras et al., 2020; Ho et al., 2020; Song & Ermon, 2019; Xu et al., 2022b) for reference.
Revised: We also include the results of several popular generative models (Karras et al., 2020; Ho et al., 2020; Song & Ermon, 2019; Xu et al., 2022b) for reference.

Original: Table 1 and Table 2 report the sample quality and the sampling speed on unconditional generation of CIFAR-10, CelebA 64 2 and ImageNet 64 2 .
Revised: Table 1 and Table 2 report the sample quality and the sampling speed on unconditional generation of CIFAR-10 and CelebA 64 2 .

Original: Our main findings are: (1) STF achieves new state-of-the-art FID scores for unconditional generation on CIFAR-10 benchmark.
Revised: Our main ﬁndings are: (1) STF achieves new state-ofthe-art FID scores for unconditional generation on CIFAR-10 benchmark.

Original: As shown in Table 1, The STF objective obtains a FID of 1 . 90 when incorporated with the EDM scheme.[Table]
Revised: As shown in Ta- ble 1, The STF objective obtains a FID of 1 . 90 when incorporated with the EDM scheme.

Original: To the best of our knowledge, this is the lowest FID score on the unconditional CIFAR-10 generation task.
Revised: To the best of our knowledge, this is the lowest FID score on the unconditional CIFAR-10 generation task.

Original: In addition, the STF objective consistently improves the EDM across the two architectures.
Revised: In addition, the STF objective consistently improves the EDM across the two architectures.

Original: The STF objective improves the performance of different score-based models.
Revised: The STF objective improves the performance of different diffusion models.

Original: We observe that the STF objective improves the FID/Inception scores of VE/VP/EDM on CIFAR-10, for most ODE and SDE samplers.
Revised: We observe that the STF objective improves the FID/Inception scores of VE/VP/EDM on CIFAR-10, for most ODE and SDE samplers.

Original: STF consistently provides performance gains for VE across datasets.
Revised: STF consistently provides performance gains for VE across datasets.

Original: Remarkably, our objective achieves much better sample quality using ODE samplers for VE, with an FID score gain of 3 . 39 on CIFAR-10, and 2 . 22 on Celeba 64 2 .
Revised: Remarkably, our objective achieves much better sample quality using ODE samplers for VE, with an FID score gain of 3 . 39 on CIFAR-10, and 2 . 22 on Celeba 64 2 .

Original:  For VP, STF provides better results on the popular DDIM sampler, while suffering from a slight performance drop when using the RK45 sampler.
Revised: [Table] For VP, STF provides better results on the popular DDIM sampler, while suffering from a slight performance drop when using the RK45 sampler.

Original: The STF objective stabilizes the converged VE model with the RK45 sampler.
Revised: The STF objective stabilizes the converged VE model with the RK45 sampler.

Original: In Appendix E.1, we report the standard deviations of performance metrics for converged models with different seeds on CIFAR-10 with VE.
Revised: In Appendix E.1, we report the standard deviations of performance metrics for converged models with different seeds on CIFAR-10 with VE.

Original: We observe that models trained with the STF objective give more consistent results, with a smaller standard deviation of used metrics.
Revised: We observe that models trained with the STF objective give more consistent results, with a smaller standard deviation of used metrics.

Original: We further provide generated samples in Appendix F. One interesting observation is that when using the RK45 sampler for VE on CIFAR-10, the generated samples from the STF objective do not contain noisy images, unlike the vanilla objective on CIFAR-10.
Revised: We further provide generated samples in Appendix F. One interesting observation is that when using the RK45 sampler for VE on CIFAR-10, the generated samples from the STF objective do not contain noisy images, unlike the vanilla DSM objective.

Original: 6. A CCELERATING TRAINING OF SGM S
Revised: 6. A CCELERATING TRAINING OF DIFFUSION MODELS

Original: [Figure]
Revised: [Figure]

Revised: FID and generated samples throughout training on (a) CIFAR-10 and (b) CelebA 64 2 .

Original: The variance-reduction techniques in neural network training can help to find better optima and achieve faster convergence rate (Wang et al., 2013; Defazio et al., 2014; Johnson & Zhang, 2013).
Revised: The variance-reduction techniques in neural network training can help to ﬁnd better optima and achieve faster convergence rate (Wang et al., 2013; Defazio et al., 2014; Johnson & Zhang, 2013).

Original: In
Revised: In

Original: Figure 4, we demonstrate the FID scores every 50k iterations during the course of training.
Revised: Figure 4, we demonstrate the FID scores every 50k iterations during the course of training.

Original: Since our goal is to investigate relative performance during the training process, and because the FID scores computed on 1k samples are strongly correlated with the full FID scores on 50k sample (Song &
Revised: Since our goal is to investigate relative performance during the training process, and because the FID scores computed on 1k samples are strongly correlated with the full FID scores on 50k sample (Song &

Original: Ermon, 2020), we report FID scores on 1k samples for faster evaluations.
Revised: Ermon, 2020), we report FID scores on 1k samples for faster evaluations.

Original: We apply ODE samplers for
Revised: We apply ODE samplers for

Original: FID evaluation, and measure the training time on two NVIDIA A100 GPUs.
Revised: FID evaluation, and measure the training time on two NVIDIA A100 GPUs.

Original: For a fair comparison, we report the average FID scores of models trained by the vanilla and STF objective on VE versus the wall-clock training time (h).[Figure]
Revised: For a fair comparison, we report the average FID scores of models trained by the DSM and STF objective on VE versus the wall-clock training time (h).

Original: The STF objective achieves better FID scores with the same training time, although the calculation of the target field by a large reference batch introduces slight overhead (Algorithm 1).
Revised: The STF objective achieves better FID scores with the same training time, although the calculation of the target ﬁeld by the reference batch introduces slight overhead ( gorithm 1).

Original: In Figure 4(a), we show that the STF objective drastically accelerates the training of score-based models on CIFAR-10.
Revised: In Figure 4(a), we show that the STF objective drastically accelerates the training of diffusion models on CIFAR-10.

Original: The STF objective achieves comparableFID scores with 3 .
Revised: The STF objective achieves comparable

Original:  6 × less training time (25h versus 90h).
Revised: FID scores with 3 . 6 ⇥ less training time (25h versus 90h).

Original: For CelebA 64 2 datasets, the training time improvement is less significant than on CIFAR-10.
Revised: For CelebA 64 2 datasets, the training time improvement is less signiﬁcant than on CIFAR-10.

Original: Our hypothesis is that the STF objective is more effective when there are multiple well-separated modes in data distribution, e.g . , the ten classes in CIFAR-10, making the vanilla objective suffer from relatively larger variations in the intermediate phase.
Revised: Our hypothesis is that the STF objective is more effective when there are multiple well-separated modes in data distribution, e.g . , the ten classes in CIFAR-10, where the DSM objective suffer from relatively larger variations in the intermediate phase.

Original: In addition,we observe that the converged models have better final performance when pairing with the STFobjective on both datasets.
Revised: In addition, the converged models have better ﬁnal performance when pairing with the STF on both datasets.

Original: 6. E FFECTS OF THE REFERENCE BATCH SIZE
Revised: 6. E FFECTS OF THE REFERENCE BATCH SIZE

Original: According to our theory (Theorem 2), the upper bound of the trace-of-covariance of the STF target decreases proportionally to the reference batch size.
Revised: According to our theory (Theorem 2), the upper bound of the trace-of-covariance of the STF target decreases proportionally to the reference batch size.

Original: Here we study the effects of the reference batch size ( n ) on model performances during training.
Revised: Here we study the effects of the reference batch size ( n ) on model performances during training.

Original: The FID scores are evaluated on 1 k samples using the RK45 sampler.
Revised: The FID scores are evaluated on 1 k samples using the RK45 sampler.

Original: As shown in Figure 5, models converge faster and produce better samples when increasing n .
Revised: As shown in Figure 5, models converge faster and produce better samples when increasing n .

Original: It suggests that smaller variations of the training targets can indeed speed up training and improve the final performances of score-based models.
Revised: It suggests that smaller variations of the training targets can indeed speed up training and improve the ﬁnal performances of diffusion models.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Different phases of SGMs.
Revised: Different phases of diffusion models.

Original: The idea of SGMs having different phases has been explored in prior works though the motivations and definitions vary (Karras et al., 2022; Choi et al., 2022).
Revised: The idea of diffusion models having different phases has been explored in prior works though the motivations and deﬁnitions vary (Karras et al., 2022; Choi et al., 2022).

Original: Karras et al.
Revised: Karras et al.

Original: (2022) argues that the training targets are difficult and unnecessary to learn in the very near field (small t in our Phase 1), whereas the training targets are always dissimilar to the true targets in the intermediate and far field (our Phase 2 and Phase 3).
Revised: (2022) argues that the training targets are difﬁcult and unnecessary to learn in the very near ﬁeld (small t in our Phase 1), whereas the training targets are always dissimilar to the true targets in the intermediate and far ﬁeld (our Phase 2 and Phase 3).

Original: As a result, their solution is sampling t with a log-normal distribution to emphasize the relevant region (relatively large t in our Phase 1).
Revised: As a result, their solution is sampling t with a log-normal distribution to emphasize the relevant region (relatively large t in our Phase 1).

Original: In contrast, we focus on reducing large training target variance in the intermediate and far field, and propose STF to better estimate the true target (cf.
Revised: In contrast, we focus on reducing large training target variance in the intermediate and far ﬁeld, and propose STF to better estimate the true target (cf.

Original: Karras et al.
Revised: Karras et al.

Original: Choi et al.
Revised: Choi et al.

Original: (2022) identifies a key region where the model learns perceptually rich contents, and determines the training weights λ ( t ) based on the signal-to-noise ratio (SNR) at different t .
Revised: (2022) identiﬁes a key region where the model learns perceptually rich contents, and determines the training weights  ( t ) based on the signal-to-noise ratio (SNR) at different t .

Original: As SNR is monotonically decreasing over time, the resulting up-weighted region does not match our Phase 2 characterization.
Revised: As SNR is monotonically decreasing over time, the resulting up-weighted region does not match our Phase 2 characterization.

Original: In general, our proposed STF method reduces the training target variance in the intermediate field and is complementary to previous improvements of SGMs.
Revised: In general, our proposed STF method reduces the training target variance in the intermediate ﬁeld and is complementary to previous improvements of diffusion models.

Original: Importance sampling.
Revised: Importance sampling.

Original: The technique of importance sampling has been widely adopted in machine learning community, such as debiasing generative models (Grover et al., 2019), counterfactual learning (Swaminathan & Joachims, 2015) and reinforcement learning (Metelli et al., 2018).
Revised: The technique of importance sampling has been widely adopted in machine learning community, such as debiasing generative models (Grover et al., 2019), counterfactual learning (Swaminathan & Joachims, 2015) and reinforcement learning (Metelli et al., 2018).

Original: Prior works using importance sampling to improve generative model training include reweighted wakesleep (RWS) (Bornschein & Bengio, 2014) and importance weighted autoencoders (IWAE) (Burda et al., 2015).
Revised: Prior works using importance sampling to improve generative model training include reweighted wakesleep (RWS) (Bornschein & Bengio, 2014) and importance weighted autoencoders (IWAE) (Burda et al., 2015).

Original: RWS views the original wake-sleep algorithm (Hinton et al., 1995) as importance sampling with one latent variable, and proposes to sample multiple latents to obtain gradient estimates with lower bias and variance.
Revised: RWS views the original wake-sleep algorithm (Hinton et al., 1995) as importance sampling with one latent variable, and proposes to sample multiple latents to obtain gradient estimates with lower bias and variance.

Original: IWAE utilizes importance sampling with multiple latents to achieve greater flexibility of encoder training and tighter log-likelihood lower bound compared to the standard variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014).
Revised: IWAE utilizes importance sampling with multiple latents to achieve greater ﬂexibility of encoder training and tighter log-likelihood lower bound compared to the standard variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014).

Original: Variance reduction for Fisher divergence.
Revised: Variance reduction for Fisher divergence.

Original: One popular approach to score-matching is to minimize the Fisher divergence between true and predicted scores (Hyv ¨ arinen & Dayan, 2005).
Revised: One popular approach to score-matching is to minimize the Fisher divergence between true and predicted scores (Hyv¨arinen & Dayan, 2005).

Original: Wang et al.
Revised: Wang et al.

Original: (2020) links the Fisher divergence to denoising score-matching (Vincent, 2011) and studies the large variance problem (in O (1 /σ 4 t ) ) of the Fisher divergence when t → 0 .{+! 
+}
Revised: (2020) links the Fisher divergence to denoising score-matching (Vincent, 2011) and studies the large variance problem (in O (1 /  4 t ) ) of the Fisher divergence when t {+! 
+}

Original: They utilize a control variate to reduce the variance.
Revised: They utilize a control variate to reduce the variance.

Original: However, this is typically not a concern for current SGMs as the time-dependent objective can be viewed as multiplying the Fisher divergence by λ ( t ) =
Revised: However, this is typically not a concern for current diffusion models as the time-dependent objective can be viewed as multiplying the Fisher divergence by  ( t ) =

Original: σ 2 t , resulting in a finite-variance objective even when t → 0 .
Revised:  2 t , resulting in a ﬁnite-variance objective even when t ! 0 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We identify large target variance as a significant training issue affecting SGMs.
Revised: We identify large target variance as a signiﬁcant training issue affecting diffusion models.

Original: We define three phases with distinct behaviors, and show that the high-variance targets appear in the intermediate phase.
Revised: We deﬁne three phases with distinct behaviors, and show that the high-variance targets appear in the intermediate phase.

Original: As a remedy, we present a generalized score-matching objective, Stable Target Field (STF), whose formulation is analogous to the self-normalized importance sampling via a large reference batch.
Revised: As a remedy, we present a generalized score-matching objective, Stable Target Field (STF), whose formulation is analogous to the self-normalized importance sampling via a large reference batch.

Original: Albeit no longer an unbiased estimator, our proposed objective is asymptotically unbiased and reduces the trace-of-covariance of the training targets, which we demonstrate theoretically and empirically.
Revised: Albeit no longer an unbiased estimator, our proposed objective is asymptotically unbiased and reduces the trace-of-covariance of the training targets, which we demonstrate theoretically and empirically.

Original: We show the effectiveness of our method on image generation tasks, and show that STF improves the performance, stability, and training speed over various state-of-the-art SGMs.
Revised: We show the effectiveness of our method on image generation tasks, and show that STF improves the performance, stability, and training speed over various state-of-the-art diffusion models.

Original: Future directions include a principled study on the effect of different reference batch sampling procedures.
Revised: Future directions include a principled study on the effect of different reference batch sampling procedures.

Original: Our presented approach is uniformly sampling from the whole dataset { x i } ni{+n i =
+}
Revised: Our presented approach is uniformly sampling from the whole dataset { x i } {+n i =
+}

Original: =2 ∼ p n − 1, so we expect that training SGMs with a reference batch of more samples in the neighborhood of x 1 (the sample from which x ( t ) is perturbed) would lead to an even better estimation of the score field. Appendix A STF SPECIFIED WITH POPULAR SGM S
Revised: ⇠ p n  1, so we expect that training diffusion models with a reference batch of more samples in the neighborhood of x(the sample from which x ( t ) is perturbed) would lead to an even better estimation of the score ﬁeld.

Original: Here, we detail the practically used STF objectives in Section 6, which are built on the popular instances of SGMs, e.g . , VE, VP (Song et al., 2021b), and EDM (Karras et al., 2022). VE and EDM For VE and EDM, the transition kernel is in the form of [Equation] VE has σ t = σ m (cid:16) σ M σ m (cid:17) t for some fixed σ m and σ M .
Revised: Moreover, the three-phase analysis can effectively capture the behaviors of other physics-inspired generative models, such as PFGM (Xu et al., 2022b) or the more advanced PFGM++ (Xu et al., 2023).

Original: EDM has σ t = ( tσ 1 ρ M + (1 − t ) σ 1 ρ m ) ρ for some σ m and σ M , with ρ set to 7 in practice.

Original: The STF objective for both VE and EDM at x ( t ) is then in the following form:

Original: [Equation]

Original: VP VP in its original formulation has the transition kernel as

Original: [Equation]

Original: for some β m and β M .

Original: The STF objective for VP at x ( t ) is

Original: E

Original: [Equation]

Original: 

Original: [Equation]

Original: where β t = − 14 t 2 ( β M − β m ) − 12 tβ m , and σ t = √ 1 − e 2 β t .

Original: Note that as shown in Karras et al.

Original: VP’s transition kernel can be reparameterized in the form of N ( x , σ 2 t I ) with a correspondingly revised sampling process.

Original: Adopting this formulation, we would have the STF objective for VP the same as the one for VE and EDM with a different σ t .

Original: B P ROOFS B. D ERIVATION OF E QUATION

Original: Recall that the STF objective (Equation 6) at time t is

Original: [Equation]

Original: Swapping the sampling order and we get ℓ STF

Original: [Equation]

Original: , t

Original: [Equation]

Original: =

Original: [Equation]

Original: p

Original: [Equation]

Original: t

Original: [Equation]

Original: x

Original: [Equation]

Original: t )

Original: [Equation]

Original: n − 1(cid:34)(cid:13)(cid:13) (cid:13) s θ ( x ( t ) , t ) − n (cid:88)

Original: [Equation]

Original: = p t | 0 ( x ( t ) | x k ) (cid:80) n

Original: [Equation]

Original: =1 p t |

Original: [Equation]

Original: t

Original: [Equation]

Original: x ( t ) log p t | 0 ( x ( t

Original: [Equation]

Original: x k ) (cid:13)(cid:13) (cid:13)  (cid:35) .

Original: This means that with input x ( t ) and t , the model is optimized with

Original: E

Original: [Equation]

Original: (cid:34)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: t

Original: [Equation]

Original: t

Original: [Equation]

Original: − n (cid:88) k

Original: = p t | 0 ( x ( t ) | x k ) (cid:80) n

Original: [Equation]

Original: p t | 0 ( x ( t ) | x j )

Original: [Equation]

Original: x ( t ) log p t | 0 ( x ( t ) | x k ) (cid:13)(cid:13)(cid:13)  (cid:35) .

Original: Taking its derivative w.r.t.

Original: s θ ( x ( t ) , t ) results in

Original: E

Original: [Equation]

Original: t

Original: [Equation]

Original: Setting it to 0, we have

Original: [Equation] [Equation] Thus, we arrive at the minimizer of the STF objective (Equation 7): [Equation]
Revised: Therefore, we anticipate that STF can enhance the performance and stability of these models further.

Original: B. P ROOF FOR T HEOREM

Original: Theorem 1.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then √ n (cid:0) s ∗ STF ( x ( t ) , t ) − ∇ x ( t ) log p t ( x ( t )) (cid:1) d −→

Original: [Equation]

Original: Proof.

Original: Recall that s ∗ STF ( x ( t ) , t ) is calculated via Equation 7.

Original: The mean of the denominator in the expectation 1 n (cid:80) nj =1 p t | 0 ( x ( t ) | x j ) , for large n , approximate 1 n − 1 (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) , which in turn, 1 n − 1 (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) p −→ p t ( x ( t )) by WLLN.

Original: Similarly, for the mean of the remaining terms in the expectation, by CLT, we have

Original: [Equation]

Original: Putting them together via Slutsky’s theorem, we conclude the proof.

Original: B. P ROOF FOR T HEOREM

Original: Theorem 2.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then

Original: V STF ( t

Original: [Equation]

Original: where D f is an f-divergence with f ( y )

Original: = (cid:26) (1 /y − 1)

Original: [-( y < 1 . 5) 8 y/ 27 − 1 / 
-]
Revised: [-( y < 1 . 5) 8 y/ 27 − 1 / 
-]

Original: y ≥ 1 . 5) .

Original: Further, when n ≫ d and p 0 | t ( x | x ( t ))

Original: ≈ p 0 ( x ) for all x ( t ) , V STF ( t )

Original: ⪅ V v ( t ) n − 1 .

Original: Proof.

Original: Step 1: Make the likelihood weighting coefficients “independent”

Original: We first apply Hoeffding’s inequality for the set { x i } ni

Original: =2 ∼ p n − 1to make the summation (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) concentrate to its expectation ( n − 1) p t ( x ( t )) .

Original: Since p t | 0 ( x ( t ) | x j ) ∈ (0 , 1 ( √ 2 πσ t ) d ) , we have

Original: Pr [Equation] ∀ γ ∈
Revised: A CKNOWLEDGEMENTS

Original: Thus the summation can be re-expressed as:

Original: [Equation]

Original: The coefficient for x 1 is then

Original: [Equation]

Original: The coefficient for x k , k ∈ { 2 , . . . , n } is:

Original: [Equation]

Original: Step 2: Re-express the trace-of-covariance by the “independent” weights

Original: Plugging in the above formulation of coefficients, we can rewrite the trace-of-covariance for the new target as:

Original: [Equation] [Table] Next, we examine the new trace-of-covariance term: [Equation]
Revised: We are grateful to Benson Chen for reviewing an early draft of this paper.

Original: We can further upper bound the trace-of-covariance term in Equation 9:

Original: [Equation]

Original: E

Original: [Equation]

Original: Taking the expectation w.r.t p t ( x ( t )) for both sides

Original: [Equation]

Original: we get

Original: [Equation]

Original: (Concavity of x 12 , Cauchy’s inequality)

Original: [Equation]

Original: where D f is an f -divergence with f ( y )

Original: = (cid:26) (1 /y − 1)

Original: [-( y < 1 . 5) 8 y/ 27 − 1 / 
-]
Revised: [-( y < 1 . 5) 8 y/ 27 − 1 / 
-]

Original: y ≥ 1 . 5) .

Original: Note that we choose this particular form of f ( y ) since it is the convex function with the tightest upper bound on ( 1 y − 1) 2 .

Original: C D ETAILS FOR THE BEHAVIOR OF V V ( t )

Original: In Section 3, we demonstrate the behavior of V v ( t ) in the three phases on Two Gaussians and a subset of CIFAR-10.

Original: Here we provide more details about the two datasets.

Original: The distribution of the two Gaussian is 12 N ( µ , ˆ σ 2 I 64 × 64 )+ 12 N ( − µ , ˆ σ 2 I 64 × 64 ) , where µ = 0 .

Original: 1 · 1 ∈

Original: R 64 , and ˆ σ = 1 e − 4 .

Original: We estimate all the integrals in Equation 4 by sampling 1k points from the corresponding distributions.

Original: For the subset of CIFAR-10, we uniformly sample 4096 images from

Original: CIFAR-10 dataset, and assign uniform distribution on the discrete set.

Original: We also approximate V v ( t ) by

Original: Monte Carlo estimation and sample 200 perturbations for each t .

Original: We use VE SDE for all simulations, and set σ m = 1 e − 2 , σ M = 50 .

Original: Interestingly, V v ( t ) is relatively large for the Two Gaussians distribution compared to CIFAR-10 (see

Original: Figure 2(b)) when t → 0 .

Original: This can be explained by their continuous and discrete natures.

Original: For the two

Original: Gaussian distribution, we can rewrite V v ( t ) as

Original: [Equation]

Original: where α ( x ) =

Original: 1+exp − 4 x T µ can be regarded as the probability that x comes from the Gaussian component N ( µ , ˆ σ 2 I 64 × 64 ) .

Original: When t → 0 , obviously α ( x ( t ))

Original: (1 − α ( x ( t )))

Original: → 0 , and the term [Equation] t +ˆ [Equation]
Revised: We would like to thank Hao

Original: vanishes.

Original: Hence lim t → 0 V v ( t ) ≈ lim

Original: [Equation] m [Equation] 2 ) , which can not be neglected when σ m is small.
Revised: He and the anonymous reviewers for their valuable feedback.

Original: On the other hand, we can effectively view the 4069 discrete samples as a mixture of 4096 0-variance Gaussians, i.e . , ˆ σ = 0 .

Original: Thus by similar reasoning we could see lim t

Original: → 0 V v ( t ) ≈ lim t

Original: →

Original: [Equation]

Original: D E XPERIMENTAL D ETAILS

Original: In this section, we include more details about the training and sampling of score-based models by the

Original: STF and vanilla objectives.

Original: All the experiments are run on two NVIDIA A100 GPUs.

Original: D. T RAINING

Original: We consider the CIFAR-10, CelebA 64 2 , and ImageNet 64 2 in image generation tasks.

Original: Following

Original: Song & Ermon (2020), we first center-crop the CelebA images and then resize them to 64 × 64 .

Original: For

Original: VE/VP, we use the same set of hyper-parameters and the NCSN++/DDPM++ backbones and the continuous-time training objectives for forward SDEs in Song et al.

Original: (2021b).

Original: For EDM, we adopt the improved hyper-parameters and architectures for NCSN++ in Karras et al.

Original: We set the reference batch size n to 4096 (VE,VP), 1024 (EDM) on CIFAR-10, 1024 on CelebA 64 2 andon ImageNet 64 2 .

Original: The training iteration is 1.3M on CIFAR-10 and 1M on CelebA 64 2 for VE/VP, and 200M images for EDM (Karras et al., 2022).

Original: The small batch size B in Algorithm 1 is the same as the batch size in the baseline score-based methods.

Original: For model selection, we pick the checkpoint with the lowest FID per 50k iterations on 10k samples for computing all the scores, as in Song et al.

Original: (2021b) for VE/VP, and per 2.5M images on 50k samples as in Karras et al.

Original: (2022) for EDM.

Original: To measure the stability of converged VE models, we repeat the experiment 3 times on CIFAR-10 for vanilla and STF objectives, using different random seeds.

Original: We quantitatively study the training overhead of STF.

Original: All the numbers are measured on two NVIDIA

Original: A100 GPUs.

Original: In Table 3 and Table 4, we report the wall-clock training time (s) per 50 iterations/50k images on VE/EDM.

Original: We can see that the STF introduces additional overhead after incorporating the large reference batch.

Original: Since the calculation of the mini-batch target does not involve neural networks, the STF does not take significantly longer training time.

Original: Indeed, in Section 6.3 we show that STF achieves comparable or better performance within a shorter training time.

Original: [Table]

Original: [Table]

Original: D. S AMPLING

Original: We adopt the RK45 method for the backward ODE sampling of VE, and the DDIM sampler (Song et al., 2021a) for VP.

Original: For RK45 sampler of VE, we use the function implemented in scipy.integrate.solve ivp with the tolerances atol = 1 e − 5 / 1 e − 4 , rtol = 1 e − 5 / 1 e −for CIFAR-10/CelebA 64 2 .

Original: As in Song et al.

Original: (2021b), we set the terminal time to 1 e − 5 / 1 e − 3 for

Original: VE/VP.

Original: For EDM, we adopt Heun’s 2 nd order method and the discretization scheme in (Karras et al., 2022), with 35 NFE.

Original: We use the predictor-corrector (PC) sampler for reverse-time SDE.

Original: We follow Song et al.

Original: (2021b) to set the Euler-Maruyama method as the predictor and the Langevin dynamics (MCMC) as the corrector.

Original: D. E VALUATIONS

Original: For the evaluation, we compute the Fr´echet distance between 50000 samples and the pre-computed statistics of CIFAR-10 and ImageNet 64 2 .

Original: For CelebA 64 2 , we adopt the setting in Song & Ermon (2020) where the distance is computed between 10000 samples and the test set.

Original: E E XTRA E XPERIMENTS E. S TABILITY OF CONVERGED MODELS

Original: In Table

Original: [Table]

Original: we report the sample quality measured by FID/Inception score, and their standard deviations across random seeds on CIFAR-10.

Original: We can see that models trained with STF objective have lower variations of their final performances, in most cases.

Original: In particular, the standard deviation decreases from 4 .

Original: 41 to 0 .

Original: 06 for RK45 sampler on VE.

Original: It suggests that the STF objective can stabilize the performance of converged models.

Original: [Table] E FFECTS OF STEP SIZE In Figure [Table]
Revised: YX and TJ acknowledge support from

Original: we show the FID scores with the number of function evaluations of ODE samplers on

Original: CIFAR-10 and CelebA 64 2 .

Original: To vary the NFE, we adjust the error tolerance in the RK45 method.

Original: The sample quality of the STF objective degrades gracefully when decreasing the NFE.

Original: The STF objective consistently outperforms the vanilla one for all NFEs on CIFAR-10, and largely improves over the baseline when setting the tolerance to 5 e − 3 on CelebA 64 2 .

Original: It suggests that the STF has greater robustness to different step sizes

Original: [Figure]

Original: F E XTENDED S AMPLES

Original: We provide extended samples from score-based models trained by vanilla/STF objective on CIFARand CelebA 64 2 by ODE samplers. For systematic comparison, we visualize samples from models trained on different seeds. We also provide samples generated by the state-of-the-art model — STF with EDM framework [Figure]
Revised: MIT-DSTA Singapore collaboration, from NSF Expeditions grant (award 1918839) “Understanding the World Through Code”, and from MIT-IBM Grand Challenge project.

Original: F. CIFAR-

Original: In Figure 7, we visualize the samples produced by different methods across random seeds for VE.

Original: We use the RK45 sampler for sampling.

Original: We observe that the model trained by the vanilla objective can produce noisy images (in red boxes), and the image quality has great variability across different random seeds.

Original: In contrast, models trained by STF objective generate clean and consistent samples with varying random seeds.

Original: In Figure 8, we further provide samples from a model trained by STF under the EDM framework (Karras et al., 2022).

Original: The model is the current state-of-the-art on the unconditional CIFAR-10 generation task.

Original: In Figure 9, we provide samples from models trained on vanilla and STF objectives with VE. [Figure] [Figure]
Revised: ST and TJ also acknowledge support from the ML for Pharmaceutical Discovery and Synthesis Consortium (MLPDS).