Original: S TABLE TARGET FIELD FOR REDUCED VARIANCE SCORE ESTIMATION
Revised: S TABLE TARGET FIELD FOR REDUCED VARIANCE SCORE ESTIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Score-based generative models (SGMs) generate samples by reversing a fixed forward diffusion process.
Revised: Score-based generative models (SGMs) generate samples by reversing a fixed forward diffusion process.

Original: Despite  impressive empirical results, we observe that the training process leads to unstable outcomes, especially when the reverse-time solvers adopt a large step size. The performance of converged models varies significantly with different random seeds, and they produce noticeable artifacts in generated samples.
Revised: Despite already providing impressive empirical results, these SGM algorithms can be further improved by reducing the variance of the training targets in their score-matching objective.

Original: We suggest that the source of such instability lies in the handling of intermediate noise-variance scales, where multiple modes in the data affect the direction of reverse paths.
Revised: We argue that the source of such variance lies in the handling of intermediate noise-variance scales, where multiple modes in the data affect the direction of reverse paths.

Original: Thus, the score-matching objective has a large sample variance in this regime, explaining lesser quality score estimates.

Original: We propose to remedy the problem by incorporating a reference batch for minibatch updates where the reference batch is used to calculate weighted conditional scores asthe more stable training targets.
Revised: We propose to remedy the problem by incorporating a reference batch which we use to calculate weighted conditional scores as more stable training targets.

Original: We show that the procedure indeed helps in the challenging intermediate regime by reducing (the trace of) the covariance of training targets.
Revised: We show that the procedure indeed helps in the challenging intermediate regime by reducing (the trace of) the covariance of training targets.

Original: The new stable targets can be seen as trading bias for reduced variance where the bias vanishes with increasing reference batch size.
Revised: The new stable targets can be seen as trading bias for reduced variance, where the bias vanishes with increasing reference batch size.

Original: Empirically, we show that the new objective improves the image quality of state-of-the-art SGMs across datasets with both general ODE and SDE solvers.
Revised: Empirically, we show that the new objective improves the image quality, stability, and training speed of various popular SGMs across datasets with both general ODE and SDE solvers.

Revised: When used in combination with state-of-the-art

Original: In particular, our method improves and stabilizes the final performance of SGMs, as well as speeding up the training process.
Revised: SGMs (EDM (Karras et al., 2022)), our method yields a new SOTA FID of 1. with 35 network evaluations on the unconditional CIFAR-10 generation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Score-based generative models (SGMs) (Song & Ermon, 2019; 2020) and diffusion models (Ho et al., 2020) have recently achieved impressive results on a wide spectrum of generative tasks, such as image generation (Nichol et al., 2022; Song et al., 2021),
Revised: Score-based generative models (SGMs) (Song & Ermon, 2019) and diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) have recently achieved impressive results on a wide spectrum of generative tasks, such as image generation (Nichol et al., 2022; Song et al., 2021b),

Original: 3D point cloud generation (Luo & Hu, 2021) and molecular conformer generation (Shi et al., 2021; Xu et al., 2022a).
Revised: 3D point cloud generation (Luo & Hu, 2021) and molecular conformer generation (Shi et al., 2021; Xu et al., 2022a).

Original: These models can be subsumed under a unified framework in the form of It ˆ o stochastic differential equations (SDE) (Song et al., 2021).
Revised: These models can be subsumed under a unified framework in the form of It ˆ o stochastic differential equations (SDE) (Song et al., 2021b).

Original: The models are learned as time-dependent score fields viathe score-matching objective (Vincent, 2011). The resulting score field then guides the reverse SDE during generative sampling.
Revised: The models learn time-dependent score fields via score-matching (Hyv¨arinen & Dayan, 2005), which then guides the reverse SDE during generative sampling.

Original: Among the SDE variants, the so-called variance-exploding (VE)  SDE  provides the best performance to date (Song et al., 2021).
Revised: Popular instances of SGMs include variance-exploding (VE) and variance-preserving (VP) SDE (Song et al., 2021b). Building on these formulations, EDM (Karras et al., 2022) provides the best performance to date.

Original: Despite impressive empirical results, we observe that SGMs can be unstable, especially with samples that adopt larger step sizes. This variability can result in noticeable artifacts in generated samples.
Revised: We argue that, despite achieving impressive empirical results, the current training scheme of SGMs can be further improved. In particular, the variance of training targets in the score-matching objective can be large and lead to suboptimal performance.

Original: To better understand the origin of  instability, we decompose the score-field into three regimes, and show that the phenomenon arises primarilydue to variability of training targets in the intermediate regime. This regime is characterized by multiple modes or data points exhibiting comparable influences on the scores.
Revised: To better understand the origin of this instability, we decompose the score field into three regimes. Our analysis shows that the phenomenon arises primarily in the intermediate regime, which is characterized by multiple modes or data points exhibiting comparable influences on the scores.

Original: In other words, in this regime, the sources of the noisy examples generated in the course of the forward process become ambiguous.
Revised: In other words, in this regime, the sources of the noisy examples generated in the course of the forward process become ambiguous.

Original: As we illustrate  in Figure 1(a), owing to this issue each stochastic update of the score model is based on disparate targets.
Revised: We illustrate the problem in Figure 1(a), where each stochastic update of the score model is based on disparate targets.

Original: We propose a generalized version of the  score-matching objective, termed the Stable Target
Revised: We propose a generalized version of the vanilla score-matching objective, termed the Stable Target

Original: Field ( STF ) objective.
Revised: Field ( STF ) objective.

Original: The idea is to include an additional reference batch of examples that are used to calculate weighted conditional scores as targets.
Revised: The idea is to include an additional reference batch of examples that are used to calculate weighted conditional scores as targets.

Original: We apply self-normalized importance sampling to aggregate the contribution of each example in the reference batch.
Revised: We apply self-normalized importance sampling to aggregate the contribution of each example in the reference batch.

Original: Although this process can substantially reduce the variance of training targets (Figure 1(b)), especially in the intermediate[Figure] regime, it does introduce some bias.
Revised: Although this process can substantially reduce the variance of training targets (Figure 1(b)), especially in the intermediate regime, it does introduce some bias.

Original: We show that the bias together with the trace-of-covariance of the estimated training targets shrink to zero as we increase the size of the reference batch.
Revised: Nevertheless, we show that the bias together with the traceof-covariance of the STF training targets shrinks to zero as we increase the size of the reference batch.

Original: Experimentally, we show that our STF objective greatly improves the stability of converged scorebased models on CIFAR-10 and CelebA 64 2 , while providing better average performance across random seeds with both ODE and SDE samplers.
Revised: Experimentally, we show that our STF objective achieves new state-of-the-art performance on CIFAR10 unconditional generation when incorporated into EDM (Karras et al., 2022).

Original: In particular, STF improves the FID scores (Heusel et al., 2017) from 8 .
Revised: The resulting FID [Figure] score (Heusel et al., 2017) is 1 .

Original: 90 ± 4 . 41 to 5 .
Revised: 90 with 35 network evaluations.

Original: 51 ± 0 . 
Revised: STF also improves the FID/Inception scores for other variants of score-based models, i.e . , VE and VP SDEs (Song et al., 2021b), in most cases.

Original: 06 with the ODE samplers for VE SDE on CIFAR-10 andfrom 7 . 56 to 5 .
Revised: In addition, it enhances the stability of converged score-based models on CIFAR-10 and

Original: 34 on CelebA. In contrast to the vanilla objective, the generated samples of models trained by STF do not contain visually noisy ones.
Revised: CelebA 64 2 across random seeds, and helps avoid generating noisy images in VE.

Original: STF accelerates the training of score-based models ( 3 . 6 × speed-up  on CIFAR-10), while obtaining comparable or better FID scores.
Revised: STF accelerates the training of score-based models ( 3 . 6 × speed-up for VE on CIFAR-10) while obtaining comparable or better FID scores.

Original: To the best of our knowledge, STF is the first technique to accelerate the training process of SGMs.
Revised: To the best of our knowledge, STF is the first technique to accelerate the training process of SGMs.

Original: We further demonstrate the performance gain when enlarging the reference batch size, validating the negative effect of large variance.
Revised: We further demonstrate the performance gain with increasing reference batch size, highlighting the negative effect of large variance.

Original: Our contributions are summarized as follows: (1) We detail the instability of the current SGMs training objective in a principled and quantitative manner, characterizing a region in the forward process, termed the intermediate phase , where the score-learning targets are most variable (Section 3).
Revised: Our contributions are summarized as follows: (1) We detail the instability of the current SGMs training objective in a principled and quantitative manner, characterizing a region in the forward process, termed the intermediate phase , where the score-learning targets are most variable (Section 3).

Original: (2) We propose a generalized score-matching objective, stable target field , which provides more stable training targets (Section 4).
Revised: (2) We propose a generalized score-matching objective, stable target field , which provides more stable training targets (Section 4).

Original: (3) We analyze the behavior of the new objective and prove that it is asymptotically unbiased and reduces the trace-of-covariance of the training targets by a factor pertaining to the reference batch size in the intermediate phase under mild conditions (Section 5).
Revised: (3) We analyze the behavior of the new objective and prove that it is asymptotically unbiased and reduces the trace-of-covariance of the training targets by a factor pertaining to the reference batch size in the intermediate phase under mild conditions (Section 5).

Original: (4) We illustrate the theoretical arguments empirically and show that the proposed STF objective improves  performance, stability, and training speed of  the currently state-of-the-art score-based method (Section 6).
Revised: (4) We illustrate the theoretical arguments empirically and show that the proposed STF objective improves the performance, stability, and training speed of score-based methods. In particular, it achieves the current state-of-the-art FID score on the CIFAR-10 benchmark when combined with EDM (Section 6).

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND ON SCORE - BASED GENERATIVE MODELS

Original: Score-based generative models In SGMs, the forward process 1 is a SDE with no learned parameter, in the form of:
Revised: In SGMs, the forward process 1 is an SDE with no learned parameter, in the form of:

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R d with x (0) ∼ p 0 being the data distribution, t ∈ [0 , 1] , f : R d × [0 , 1] → R d , g : [0 , 1] → R , and w ∈ R
Revised: where x ∈ R d with x (0) ∼ p 0 being the data distribution, t ∈ [0 , 1] , f : R d × [0 , 1] → R d , g : [0 , 1] → R , and w ∈ R

Original: d is the standard Wiener process.
Revised: d is the standard Wiener process.

Original: It gradually transforms the data distribution to a known prior as time goes from 0 to 1.
Revised: It gradually transforms the data distribution to a known prior as time goes from 0 to 1.

Original: Sampling of SGMs is done via a corresponding reverse-time SDE (Anderson, 1982):
Revised: Sampling of SGMs is done via a corresponding reverse-time SDE (Anderson, 1982):

Original: [Equation]
Revised: [Equation]

Original: where ¯ · denotes time traveling backwards from 1 to 0.
Revised: where ¯ · denotes time traveling backward from 1 to 0.

Original: Song et al.
Revised: Song et al.

Original: (2021) propose a probability flow
Revised: (2021b) propose a probability flow

Original: ODE that induces the same marginal distribution p t ( x ) as the SDE:
Revised: ODE that induces the same marginal distribution p t ( x ) as the SDE:

Original: [Equation]
Revised: [Equation]

Original: Both formulations progressively recover p 0 from the prior p 1 .
Revised: Both formulations progressively recover p 0 from the prior p 1 .

Original: We estimate the score of the transformed data distribution at time t , ∇ x log p t ( x ) , via a neural network, s θ ( x , t ) .
Revised: We estimate the score of the transformed data distribution at time t , ∇ x log p t ( x ) , via a neural network, s θ ( x , t ) .

Original: Specifically, the training objective is a weighted sum of the denoising score-matching (Vincent, 2011):
Revised: Specifically, the training objective is a weighted sum of the denoising score-matching (Vincent, 2011):

Original: [Equation]
Revised: [Equation]

Original: where  λ ( t )  is some positive weighting function to keep the time-dependent loss at the same magnitude (Song et al., 2021), and p t | 0 ( x ( t ) | x ) is the transition kernel denoting the conditional distribution of x ( t ) given x  .Note that we omit “ (0) ” from x (0) when there is no ambiguity.
Revised: where q t is the distribution for time variable, e.g . , U [0 , 1] for VE/VP (Song et al., 2021b), and λ ( t ) = σ 2 t is the positive weighting function to keep the time-dependent loss at the same magnitude (Song et al., 2021b), and p t | 0 ( x ( t ) | x ) is the transition kernel denoting the conditional distribution of x ( t ) given x 2 .

Original: Popular instances of this framework “destroy” data according to a diffusion process utilizing Gaussian transition kernels, which result in p t | 0 ( x ( t ) | x ) =
Revised: Specifically, SGMs “destroy” data according to a diffusion process utilizing Gaussian transition kernels, which result in p t | 0 ( x ( t ) | x ) =

Original: N ( µ t , σ 2 t I ) .
Revised: N ( µ t , σ 2 t I ) .

Original: Recent works (Xu et al., 2022b; Rissanen et al., 2022) have also extended the underlying principle from the diffusion process to more general physical processes where the training objective is not necessarily score-related.

Original: Self-normalized importance sampling

Original: We revisit the basics of importance sampling and selfnormalized importance sampling (Owen, 2013), upon which we draw inspirations for our method.

Original: Importance sampling provides a Monte Carlo estimator of the expectation E x ∼ p

Original: [ f ( x )] by sampling from the proposal distribution q , i.e . , E x ∼ p [ f ( x )] ≈ 1 n (cid:80) ni =1 p ( x i ) /q
Revised: Recent works (Xu et al., 2022b;

Original: ( x i ) f

Original: ( x i ) , x i ∼ q . The estimator is effective when sampling fromp is infeasible, or the vanilla Monte Carlo estimator incurs high variance (Owen, 2013; Elvira & Martino, 2021). In practice though, the importance weights w i = p ( x i ) /q
Revised: Rissanen et al., 2022) have also extended the underlying principle from the diffusion process to more general physical processes where the training objective is not necessarily score-related.

Original: ( x i ) may have a large variance across the Monte Carlo batch.

Original: Self-normalized importance sampling (Hesterberg, 1995) remedies the issue by normalizing the importance weights in the batch: (cid:80) mi

Original: =1 ( w i / (cid:80) j w j ) f ( x i ) .

Original: The technique is widely adopted in machine learning community, such as debiasing generative models (Grover et al., 2019), counterfactual learning (Swaminathan &

Original: Joachims, 2015) and reinforcement learning (Metelli et al., 2018).

Original:  U NDERSTANDING THE TRAINING TARGET IN SCORE - MATCHING OBJECTIVE
Revised:  U NDERSTANDING THE TRAINING TARGET IN SCORE - MATCHING OBJECTIVE

Original: The vanilla denoising score-matching objective at time t is:
Revised: The vanilla denoising score-matching objective at time t is:

Original: [Equation]
Revised: [Equation]

Original: where the network is trained to fit the individual targets ∇ x 
Revised: where the network is trained to fit the individual targets ∇ x (

Original: [Equation]
Revised: [Equation]

Original: t ) log pt
Revised: ) log p

Original: [Equation]
Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original:  x 
Revised: | x ) at

Original: [Equation]
Revised: [Equation]

Original:  t)
Revised: x ( t

Original: | x ) at ( x ( t ) , t ) – the “influence” exerted by clean data x on x ( t ) .
Revised: [Equation] [Equation] the “influence” exerted by clean data x on x ( t ) .

Original: We can swap the order of the sampling process by first sampling x ( t ) from p t and then x from p 0 | t ( ·| x ( t )) .
Revised: We can swap the order of the sampling process by first sampling x ( t ) from p t and then x from p 0 | t ( ·| x ( t )) .

Original: Thus, s θ has a closed form minimizer:
Revised: Thus, s θ has a closed form minimizer:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The score field is a conditional expectation of ∇ x ( t ) log p t | 0 ( x ( t )
Revised: The score field is a conditional expectation of ∇ x ( t ) log p t | 0 ( x ( t )

Original: | x ) with respect to the posterior distribution p 0 | t .
Revised: | x ) with respect to the posterior distribution p 0 | t .

Original: In practice, a Monte Carlo estimate of this target could have high variance.
Revised: In practice, a Monte Carlo estimate of this target can have high variance (Owen, 2013; Elvira & Martino, 2021).

Original: In particular, when multiple modes of the data distribution have comparable influences on x ( t ) , p 0 | t ( ·| x ( t )) is a multi-mode distribution.
Revised: In particular, when multiple modes of the data distribution have comparable influences on x ( t ) , p 0 | t ( ·| x ( t )) is a multi-mode distribution, as also observed in Xiao et al.

Original: Thus the targets ∇ x ( t ) log p t | 0 ( x ( t ) | x ) vary considerably across different x
Revised: Thus the targets ∇ x ( t ) log p t | 0 ( x ( t ) | x ) vary considerably across different x

Original: and this can strongly affect the estimated score at ( x ( t ) , t ) , resulting in slower convergence and worse performance in practical stochastic gradient optimization (Wang et al., 2013).
Revised: and this can strongly affect the estimated score at ( x ( t ) , t ) , resulting in slower convergence and worse performance in practical stochastic gradient optimization (Wang et al., 2013).

Original: To quantitatively characterize the variations of individual targets at different time, we propose a metric – the average trace-of-covariance of training targets at time t :
Revised: To quantitatively characterize the variations of individual targets at different time, we propose a metric – the average trace-of-covariance of training targets at time t :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We use V v ( t ) to define three successive phases relating to the behavior of training targets.
Revised: We use V v ( t ) to define three successive phases relating to the behavior of training targets.

Original: As shown in Figure 2(a), the three phases partition the score field into near, intermediate and far regimes (Phase 1 ∼ 3 respectively).
Revised: As shown in Figure 2(a), the three phases partition the score field into near, intermediate, and far regimes (Phase 1 ∼ 3 respectively).

Original: Intuitively, V v ( t ) peaks in the intermediate phase (Phase 2), where multiple distant modes in the data distribution have comparable influences on the same noisy perturbations, resulting in unstable targets.
Revised: Intuitively, V v ( t ) peaks in the intermediate phase (Phase 2), where multiple distant modes in the data distribution have comparable influences on the same noisy perturbations, resulting in unstable targets.

Original: In Phase 1, the posterior p 0 | t concentrates around one single mode, thus reducing the variation.
Revised: In Phase 1, the posterior p 0 | t concentrates around one single mode, thus low variation.

Original: In Phase 3, the targets remain similar across modes since lim t → 1 p t | 0 ( x ( t ) | x ) ≈ p 1 for commonly used transition kernels.
Revised: In Phase 3, the targets remain similar across modes since lim t → 1 p t | 0 ( x ( t ) | x ) ≈ p 1 for commonly used transition kernels.

Original: [Figure]
Revised: [Figure]

Original: We validate this argument empirically in Figure 2(b). The figure shows the estimated V v ( t ) for atoy example involving a mixture of two Gaussians as well asfor a more realistic scenario based on a subset of CIFAR10 dataset (Krizhevsky et al.).
Revised: We validate this argument empirically in Figure 2(b), which shows the estimated V v ( t ) for a mixture of two Gaussians as well as a subset of CIFAR-10 dataset (Krizhevsky et al., 2009) for a more realistic setting.

Original: We use Variance Exploding (VE) SDE, i.e . , p t | 0 ( x ( t ) | x ) = N (cid:16) x( t ); x , σ 2 m ( σ M σ m ) 2 t I (cid:17) for some σ m and σ M
Revised: Here we use VE SDE, i.e . , p t | 0 ( x ( t ) | x ) = N (cid:16) x , σ 2 m ( σ M σ m ) 2 t I (cid:17) for some σ m and σ M

Original: (Song et al., 2021).
Revised: (Song et al., 2021b).

Original: V v ( t ) exhibits similar phase behavior across t in both toy and realistic cases.
Revised: V v ( t ) exhibits similar phase behavior across t in both toy and realistic cases.

Original:  V v ( t ) reaches its maximum value in the intermediate phase, demonstrating the large variation of individual targets.
Revised: Moreover, V v ( t ) reaches its maximum value in the intermediate phase, demonstrating the large variations of individual targets.

Original: We defer more details to Appendix B.
Revised: We defer more details to Appendix C.

Original:  T REATING SCORE AS A FIELD
Revised:  T REATING SCORE AS A FIELD

Original: The vanilla denoising score-matching approach (Equation 3) can  viewed as a Monte Carlo estimator, i.e . , ∇ x ( t ) log p t ( x ( t )) =
Revised: The vanilla denoising score-matching approach (Equation 3) can be viewed as a Monte Carlo estimator, i [Equation]

Original: E p 0 | t ( x | x
Revised: e . ,

Original: [Equation]
Revised: [Equation]

Original: where x i are sampled from p 0 | t ( ·| x ( t )) and n = 1 .
Revised: where x i is sampled from p 0 | t ( ·| x ( t )) and n = 1 .

Original: The variance of a Monte Carlo estimator is proportional to 1 n , so we propose to use a larger batch ( n ) to counter the high variance problem described in Section 3.
Revised: The variance of a Monte Carlo estimator is proportional to 1 n , so we propose to use a larger batch ( n ) to counter the high variance problem described in Section 3.

Original: Since sampling directly from the posterior p 

Revised: Since sampling directly from the posterior p

Revised: | t is not practical

Revised: [Equation]

Original: | t is not practical, we first apply importance samplingtogether with  a large reference batch B L = { x i } ni =1 ∼ p n 0 . As a result, we get the following approximation:
Revised: we first apply importance sampling with the proposal distribution p 0 . Specifically, we sample a large reference batch B L = { x i } ni =1 ∼ p n 0 and get the following approximation:

Original: [Equation]
Revised: [Equation]

Original: The importance weights can be rewritten as p 0 | t ( x | x ( t ))
Revised: The importance weights can be rewritten as p 0 | t ( x | x ( t ))

Original: /p 

Revised: /p

Original: ( x ) = 
Revised: ( x ) = p

Revised: [Equation]

Revised: | 0 ( x

Original: p t | 0 ( x ( t ) | x ) /p t ( x ( t )) .
Revised: [Equation] ) | x ) [Equation] p t ( x ( t )) .

Original: This basic importance sampling estimator has two issues.
Revised: However, this basic importance sampling estimator has two issues.

Original: The weights now involve an unknown normalization factor p t ( x ( t )) and the ratio between the prior and posterior distribution can be large in high dimensional spaces.
Revised: The weights now involve an unknown normalization factor p t ( x ( t )) and the ratio between the prior and posterior distribution can be large in high dimensional spaces.

Original: To remedy these problems, we appeal to self-normalization techniques (Owen, 2013) to further stabilize the training targets:[Equation]
Revised: To remedy these problems, we appeal to self-normalization techniques (Hesterberg, 1995) to further stabilize the training targets:

Original: [Equation]
Revised: [Equation]

Original: We term this new training target in Equation 5 as Stable Target Field (STF) .
Revised: We term this new training target in Equation 5 as Stable Target Field (STF) .

Original: In practice, we sample the reference batch B L = { x i } ni =1 from p n 0 and obtain x ( t ) by applying the transition kernel to the “first” training data x 1 .
Revised: In practice, we sample the reference batch B L = { x i } ni =1 from p n 0 and obtain x ( t ) by applying the transition kernel to the “first” training data x 1 .

Original: Taken together, the new STF objective becomes:
Revised: Taken together, the new STF objective becomes:

Original: [Equation]
Revised: [Equation]

Original: When n = 1 , the weighted objective reduces to the vanilla score-matchingobjective (Equation 2).
Revised: When n = 1 , STF reduces to the vanilla score-matching (Equation 2).

Original: In contrast to the vanilla target, the stable target field incorporates alarge reference batch to stabilize training targets.
Revised: When n > 1 , STF incorporates a reference batch to stabilize training targets.

Original: Intuitively, the stable targetfield assigns larger weights to clean data with higher influence on x ( t ) , i.e . , higher transition probability p t | 0 ( x ( t ) | x ) .
Revised: Intuitively, the new weighted target assigns larger weights to clean data with higher influence on x ( t ) , i.e . , higher transition probability p t | 0 ( x ( t ) | x ) .

Original: Similar to our analysis in Section 3, we can again swap the sampling process in Equation 6 so that, for a perturbation x ( t ) , we sample the reference batch B L = { x i } n i =1 from p 0 | t ( ·| x ( t ))
Revised: Similar to our analysis in Section 3, we can again swap the sampling process in Equation 6 so that, for a perturbation x ( t ) , we sample the reference batch B L = { x i } ni =1 from p 0 | t ( ·| x ( t ))

Original: p n − 1, where the first element involves the posterior, and the rest follow the data distribution.
Revised: p n − 1, where the first element involves the posterior, and the rest follow the data distribution.

Original: Thus, the minimizer of the new objective (Equation 6) is 
Revised: Thus, the minimizer of the new objective (Equation 6) is (derivation can be found in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: Note that although the stable target field significantly reduces the variance, it introduces bias: the minimizer is no longer the true score.
Revised: Note that although STF significantly reduces the variance, it introduces bias: the minimizer is no longer the true score.

Original: Nevertheless, in Section 5, we show that the bias converges to 0 as n → ∞ , while reducing the trace of covariance of the training targets by a factor of n when p 0 | t ≈ p 0 .
Revised: Nevertheless, in Section 5, we show that the bias converges to 0 as n → ∞ , while reducing the trace-of-covariance of the training targets by a factor of n when p 0 | t ≈ p 0 .

Original: To aggregate thetime-dependent STF objective over t , we uniformly sample the time variable t from [0 , 1] during training and apply the weighting function λ ( t ) . Together, the final training objective for STF is E t ∈U [0 , 1] [ λ ( t ) ℓ STF ( θ, t )] . We summarize the training process in
Revised: We further instantiate the STF objective (Equation 6) with transition kernels in the form of p t | 0 ( x ( t ) | x ) = N ( x , σ 2 t I ) , which includes EDM (Karras et al., 2022), VP (through reparameterization) and VE (Song et al., 2021b):

Original: Algorithm 1.
Revised: [Equation]

Original: The small batch size |B| is the same as the normal batch size in vanilla training process. We further specify the  STF objective (Equation 6) with VE SDE, whose transition kernel is p t | 0 ( x ( t ) | x ) ∝ exp( −∥ x ( t )− x ∥ 2 2 σ 2 t ) with σ t = σ m ( σ M σ m ) t .The STF objective for VE SDE at x ( t ) is in the following form: [Equation]
Revised: To aggregate the time-dependent STF objective over t , we uniformly sample the time variable t from [0 , 1] during training and apply the weighting function λ ( t ) .

Original: k = [Equation] [Equation] [Equation]
Revised: Together, the final training objective for

Original:  t ) (cid:80) [Equation] [Equation] [Equation]
Revised: STF is E t ∼ q t ( t ) [ λ ( t ) ℓ STF ( θ, t )] .

Original: j

Original: [Equation]

Original: [Equation]

Original: ( t )) (cid:13)(cid:13)(cid:13)(cid:13)    . Input: Training iteration T , Initial model s θ , dataset D , learning rate η . for t = 1 . . .
Revised: We summarize the training process in Algorithm 1.

Original: T do Sample a large reference batch B L from D , and subsample a small batch B = { x i } |B| i =1 from B L Uniformly sample the time { t i } |B| i =1 ∼ U [0 , 1] |B|
Revised: The small batch size |B| is the same as the normal batch size in the vanilla training process.

Original: Obtain the batch of perturbed samples { x i ( t i ) } |B| i =1 by applying the transition kernel p t | 0 on B Calculate the stable target field of B L for all x i ( t i ) : [Equation] t
Revised: We defer specific use cases of STF objectives combined with various popular SGMs to Appendix A.

Original: [Equation]
Revised: [Table]

Original: ) | x )

Original: Calculate the loss: L ( θ ) = 1 |B| (cid:80) |B| i =1 λ ( t i ) ∥ s θ ( x i ( t i ) , t i ) − v B L ( x i ( t i )) ∥ Update the model parameter: θ = θ − η ∇L ( θ ) end for return s θ
Revised: [Table]

Original:  A NALYSIS
Revised:  A NALYSIS

Original: In this section, we are interested in the theoretical properties of our approach.
Revised: In this section, we analyze the theoretical properties of our approach.

Original: In particular, we show that the new minimizer s ∗ STF ( x ( t ) , t ) (Equation 7) converges to the true score  (Section 5.1).
Revised: In particular, we show that the new minimizer s ∗ STF ( x ( t ) , t ) (Equation 7) converges to the true score asymptotically (Section 5.1).

Original: Then, we show that the new STF targets reduce the trace of covariance by a factor of  the reference batch size in the intermediate phase, with mild conditions (Section 5.2).
Revised: Then, we show that the proposed STF reduces the trace-of-covariance of training targets propositional to the reference batch size in the intermediate phase, with mild conditions (Section 5.2).

Original: 5. A SYMPTOTIC BEHAVIOR
Revised: 5. A SYMPTOTIC BEHAVIOR

Original: Although in general s ∗ STF ( x ( t ) , t ) ̸ = ∇ x ( t ) log p t ( x ( t )) , the bias shrinks toward 0 with a increasing n .
Revised: Although in general s ∗ STF ( x ( t ) , t ) ̸ = ∇ x ( t ) log p t ( x ( t )) , the bias shrinks toward 0 with a increasing n .

Original: In the following theorem we show that the minimizer of STF objective at ( x ( t ) , t ) , i.e . , s ∗ STF ( x ( t ) , t ) , is asymptotically normal when n → ∞ .
Revised: In the following theorem we show that the minimizer of STF objective at ( x ( t ) , t ) , i.e . , s ∗ STF ( x ( t ) , t ) , is asymptotically normal when n → ∞ .

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then
Revised: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then

Original: [Equation]
Revised: [Equation]

Original: We defer the proof to Appendix A.1.
Revised: We defer the proof to Appendix B.2.

Original: The theorem states that, for commonly used transition kernels, s

Original: [Equation] log p t ( x [Equation] t )) converges to a zero mean normal, and larger reference batch size ( n ) will lead to smaller asymptotic variance.
Revised: The theorem states that, for commonly used transition kernels, s ∗ STF ( x ( t ) , t ) − ∇ x ( t ) log p t ( x ( t )) converges to a zero mean normal, and larger reference batch size ( n ) will lead to smaller asymptotic variance.

Original: As can be seen in Equation 8, when n → ∞ , s ∗ STF ( x ( t ) , t ) highly concentrates around the true score ∇ x ( t ) log p t ( x ( t )) .
Revised: As can be seen in Equation 8, when n → ∞ , s ∗ STF ( x ( t ) , t ) highly concentrates around the true score ∇ x ( t ) log p t ( x ( t )) .

Original: 5. T RACE OF C OVARIANCE
Revised: 5. T RACE OF C OVARIANCE

Original: We now highlight the small variations of the training targets in the STF objective compared to the vanilla one.
Revised: We now highlight the small variations of the training targets in the STF objective compared to the vanilla one.

Original: As done in Section 3, we study the trace-of-covariance of training targets in STF:
Revised: As done in Section 3, we study the trace-of-covariance of training targets in STF:

Original: [Equation]
Revised: [Equation]

Original: In the following theorem we compare V STF with V v .
Revised: In the following theorem we compare V STF with V v .

Original: In particular, we can upper bound V STF ( t ) by
Revised: In particular, we can upper bound V STF ( t ) by

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then
Revised: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then

Original: V STF ( t) ≤
Revised: V STF [Equation] t

Original: [Equation]
Revised: [Equation]

Original: where D f is an f-divergence with f ( y )
Revised: where D f is an f-divergence with f ( y )

Original: = [Equation] [Equation] − 
Revised: = (cid:26) (1 /y − 1)

Original: [Equation] < 1 . [Equation] 8 y/ 27 − 1 / 

Revised: ( y < 1 . 5) 8 y/ 27 − 1 /

Original: ( y ≥ 1 . 5) .
Revised: y ≥ 1 . 5) .

Original: Further, when n ≫ d and p 0 | t ( x | x ( t ))
Revised: Further, when n ≫ d and p 0 | t ( x | x ( t ))

Original: ≈ p 0 ( x ) for all x ( [Equation]
Revised: ≈ p 0 ( x ) for all x ( t ) , V STF ( t )

Revised: ⪅ V v ( t ) n − 1 .

Original: We defer the proof to Appendix A.2.
Revised: We defer the proof to Appendix B.3.

Original: Theorem 2 states that when the posterior is close to  the prior, the average trace-of-covariance of the STF targets is reduced by n − 1 times compared to  the vanilla one.
Revised: The second term that involves f -divergence D f is necessary to capture how the coefficients, i.e . , p t | 0 ( x ( t ) | x k ) / (cid:80) j p t | 0 ( x ( t ) | x j ) used to calculate the weighted score target, vary across different samples x ( t ) . This term decreases monotonically as a function of t .

Original: Since p t ≈ p t | 0 with large σ t for commonly used transition kernels, p 0 | t ( ·| x ( t )) ≈ p 0 , ∀ x  ( t ) with large t  .
Revised: In Phase 1, p 0 | t ( x | x ( t )) differs substantially from p 0 ( x ) and the divergence term D f dominates. In contrast to the upper bound, both V STF ( t ) and V v ( t ) have minimal variance at small values of t since the training target is always dominated by one x .

Original: For example, when the data distribution is N ( µ , ˆ σ 2 I ) , ˆ σ > 0 , the posterior in VE SDE is p 0 | t ( x | x ( t ))
Revised: The theorem has more relevance in Phase 2, where the divergence term decreases to a value comparable to V v ( t ) .

Original: =
Revised: In this phase, we empirically observe that the ratio of the two terms in the upper bound ranges from 10 to 100.

Original: N ( x ( t )ˆ σ 2 + µ σ 2 t ˆ σ 2 + σ 2 t , ˆ σ 2 σ 2 t ˆ σ 2 + σ 2 t I ) . Obviously, when σ t ≫ ˆ σ , D f ( p0 | t( x |x ( t )) ∥ p 0 ( x )) ≈ 0 , and V STF ( t )
Revised: Thus, when we use a large reference batch size (in thousands), the theorem implies that STF offers a considerably lower variance (by a factor of 10 or more) relative to the vanilla objective. In Phase 3, the second term vanishes to 0, as p t ≈ p t | 0 with large σ t for commonly used transition kernels.

Original: ⪅ V v ( t ) /n − 1 .
Revised: As a result, STF reduces the average trace-of-covariance of the training targets by at least n − 1 times in the far field.

Original: Together, we demonstrate that the  targets have diminishing bias (Theorem 1) and are much more stable during training (Theorem 2).
Revised: Together, we demonstrate that the STF targets have diminishing bias (Theorem 1) and are much more stable during training (Theorem 2).

Original: These properties make the STF objective more favorable for learning the score of the data distribution using stochastic gradient optimization.
Revised: These properties make the STF objective more favorable for SGMs training with stochastic gradient optimization.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we first empirically validate our theoretical analysis in Section 5, especially for variance reduction in the intermediate phase (Section 6.1).
Revised: In this section, we first empirically validate our theoretical analysis in Section 5, especially for variance reduction in the intermediate phase (Section 6.1).

Original: Next, we show that the STF objective improves the score-based models on image generation tasks in terms of the average performance, the stability to random seeds and the robustness to large step sizes (Section 6.2). 
Revised: Next, we show that the STF objective improves various score-based models on image generation tasks in terms of image quality (Section 6.2). In particular, STF achieves state-of-the-art performance on top of EDM.

Original: In addition, we demonstrate that STF accelerates the training of score-based models (Section 6.3), and improves the convergence speed and final performance with an increasing reference batch size (Section 6.4).
Revised: In addition, we demonstrate that STF accelerates the training of score-based models (Section 6.3), and improves the convergence speed and final performance with an increasing reference batch size (Section 6.3).

Original: 6. V ARIANCE REDUCTION IN THE INTERMEDIATE PHASE
Revised: 6. V ARIANCE REDUCTION IN THE INTERMEDIATE PHASE

Original: [Figure]
Revised: [Figure]

Original: The proposed algorithm 1 utilizes a large reference batch to calculate the stable target field instead of the individual target.
Revised: The proposed Algorithm 1 utilizes a large reference batch to calculate the stable target field instead of the individual target.

Original: In addition to the theoretical analysis in Section 5, we provide further empirical study to characterize the intermediate phase and verify the variance reduction effects by STF.
Revised: In addition to the theoretical analysis in Section 5, we provide further empirical study to characterize the intermediate phase and verify the variance reduction effects by STF.

Original: Apart from V ( t ) , we also quantify the average divergence between the posterior p 0 | t ( ·| x ( t )) and the data distribution p 0 at time t (introduced in Theorem 2): D ( t ) =
Revised: Apart from V ( t ) , we also quantify the average divergence between the posterior p 0 | t ( ·| x ( t )) and the data distribution p 0 at time t (introduced in Theorem 2): D ( t ) =

Original: E p t ( x ( t )) (cid:2) D f (cid:0) p 0 | t ( x | x ( t ))
Revised: E p t ( x ( t )) (cid:2) D f (cid:0) p 0 | t ( x | x ( t ))

Original: ∥ p 0 ( x ) (cid:1)(cid:3) .
Revised: ∥ p 0 ( x ) (cid:1)(cid:3) .

Original: Intuitively, the number of high-density modes in p 0 | t ( ·| x ( t )) grows as D ( t ) decreases.
Revised: Intuitively, the number of high-density modes in p 0 | t ( ·| x ( t )) grows as D ( t ) decreases.

Original: To investigate their behaviors, we construct two synthetic datasets: (1) A 64-dimensional mixture of two Gaussian components ( Two Gaussians ); (2) A subset of 4096 images of CIFAR-10 ( CIFAR-10-4096 ).
Revised: To investigate their behaviors, we construct two synthetic datasets: (1) a 64-dimensional mixture of two Gaussian components ( Two Gaussians ), and (2) a subset of 4096 images of CIFAR-10 ( CIFAR-10-4096 ).

Original: Figure 3(a) and Figure 3(b) show the behaviors of V v ( t ) and D ( t ) on Two Gaussian and CIFAR10-4096.
Revised: Figure 3(a) and Figure 3(b) show the behaviors of V v ( t ) and D ( t ) on Two Gaussian and CIFAR10-4096.

Original: In both settings, V v ( t ) reaches its peak in the intermediate phase (Phase 2) , while D ( t ) gradually decreases over time.
Revised: In both settings, V v ( t ) reaches its peak in the intermediate phase (Phase 2), while D ( t ) gradually decreases over time.

Original: These results agree with our theoretical understanding from Section 3.
Revised: These results agree with our theoretical understanding from Section 3.

Original: In Phase 2 and 3, several modes of the data distribution have noticeable influences on the scores, but only in Phase 2  the influencesare much more distinct, leading to high variations of the individual target
Revised: In Phase 2 and 3, several modes of the data distribution have noticeable influences on the scores, but only in Phase 2 are the influences much more distinct, leading to high variations of the individual target

Original: ∇ x ( t ) log p t | 0 ( x ( t ) | x ) , x ∼ p 0 | t ( ·| x ( t )) .
Revised: ∇ x ( t ) log p t | 0 ( x ( t ) | x ) , x ∼ p 0 | t ( ·| x ( t )) .

Original: Figure 3(c) and Figure 3(d) further show the relationship between V STF ( t ) and the reference batch size n .
Revised: Figure 3(c) and Figure 3(d) further show the relationship between V STF ( t ) and the reference batch size n .

Original: Recall that when n = 1 , STF degenerates to individual target and V STF ( t ) =
Revised: Recall that when n = 1 , STF degenerates to individual target and V STF ( t ) =

Original: V v ( t ) .
Revised: V v ( t ) .

Original: We observe that V STF ( t ) decreases when enlarging n .
Revised: We observe that V STF ( t ) decreases when enlarging n .

Original: In particular, the predicted relation V STF ( t ) ⪅ V v ( t ) / ( n − 1) in Theorem 2 holds for the two Gaussian datasets where D f is small.
Revised: In particular, the predicted relation V STF ( t ) ⪅ V v ( t ) / ( n − 1) in Theorem 2 holds for the two Gaussian datasets where D f is small.

Original: On  high dimensional dataset
Revised: On the high dimensional dataset

Original: CIFAR-10-4096, the stable target field can still largely reduce the V STF ( t ) by increasing the reference batch size.
Revised: CIFAR-10-4096, the stable target field can still greatly reduce the training target variance with large reference batch sizes n .

Original: 6. I MAGE GENERATION
Revised: 6. I MAGE GENERATION

Original: [Table]
Revised: [Table]

Original: We demonstrate the stability of the new objective on image generation tasks.
Revised: We demonstrate the effectiveness of the new objective on image generation tasks.

Original: We consider CIFAR10 (Krizhevsky et al.) and CelebA 64 × 64 (Yang et al., 2015)  datasets.
Revised: We consider CIFAR-10 (Krizhevsky et al., 2009), CelebA 64 × 64 (Yang et al., 2015) and ImageNet 64 × 64 (Deng et al., 2009) datasets.

Original: We set the reference batch size n to 4096 (CIFAR10) and 1024 (CelebA 64 2 ). For comparison
Revised: We set the reference batch size n to 4096 (CIFAR10) and 1024 (CelebA 64 2 ,

Original: [Table]
Revised: ImageNet 64 2 ).

Original: we choose the current state-of-the-art score-based method VE SDE as the baseline [Table] and replace the vanilla objective with our STF objective during training. 
Revised: We choose the current state-of-the-art score-based method EDM (Karras et al., 2022) as the baseline, and replace the vanilla objective with our STF objective during training. We also apply STF to two other popular score-based models, VE/VP SDEs (Song et al., 2021b).

Revised: For a fair comparison, we directly adopt the architectures and the hyper-parameters in Karras et al.

Original: We directly adopt the NCSN++ architecture and the hyper-parameters such as batch size, learning rate andtraining iterations in Song et al. 
Revised: and Song et al. (2021b) for EDM and VE/VP respectively. In particular, we use the improved NCSN++/DDPM++ models (Karras et al., 2022) in the EDM scheme.

Original: To highlight the stability issue, we train three models with different seeds for all methods on CIFAR-10.
Revised: To highlight the stability issue, we train three models with different seeds for VE on CIFAR-10.

Original: We provide more experimental details in Appendix C.1.
Revised: We provide more experimental details in Appendix D.1.

Original: Numerical Solver The reverse-time ODE and SDE in scored-based models are compatible with any general purpose solvers.
Revised: Numerical Solver. The reverse-time ODE and SDE in scored-based models are compatible with any general-purpose solvers.

Original: We use the adaptivestep size ODE solverin the Scipy library (Virtanen et al., 2020) with the RK45 method (Dormand & Prince, 1980)
Revised: We use the adaptive solver RK45 method (Dormand & Prince, 1980; Song et al., 2021b) (RK45) for VE/VP and the popular DDIM solver (Song et al., 2021a) for VP.

Original: ( RK45 ), unless otherwise specified.
Revised: We adopt Heun’s 2nd order method (Heun) and the time discretization proposed by Karras et al. (2022) for EDM.

Original: For SDEs, we apply the predictor-corrector  sampler  (Song et al., 2021) ( PC ), with Euler-Maruyama method as the predictor and the Langevin dynamics (MCMC) as the corrector.
Revised: For SDEs, we apply the predictor-corrector (PC) sampler used in (Song et al., 2021b). We denote the methods in a objective-sampler format, i.e . , A - B , where A ∈ { vanilla, STF } and B ∈ { RK45, PC, DDIM, Heun } .

Original: We defer more details to Appendix C.2.
Revised: We defer more details to Appendix D.2.

Original: Results For quantitative evaluation of the generated samples, we report the FID scores (Heusel et al., 2017) (lower is better) and Inception (Salimans et al., 2016) (higher is better).We measure the [Figure]
Revised: Results. For quantitative evaluation of the generated samples, we report the FID scores (Heusel et al., 2017) (lower is better) and Inception (Salimans et al., 2016) (higher is better).

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] sampling speed by the average NFE (number of function evaluation).
Revised: We measure the sampling speed by the average NFE (number of function evaluations).

Original: We omit the Inception score on CelebA 64 2 as it is not commonly reported on the dataset.

Original: We also include the results of several popular generative models (Karras et al., 2020; Ho et al., 2020; Song & Ermon, 2019) for reference.
Revised: We also include the results of several popular generative models (Karras et al., 2020; Ho et al., 2020; Song & Ermon, 2019; Xu et al., 2022b) for reference.

Original: Table 1 and Table 2 report the sample quality and the sampling speed on CIFAR-10 and CelebA 64 2  .
Revised: Table 1 and Table 2 report the sample quality and the sampling speed on unconditional generation of CIFAR-10, CelebA 64 2 and ImageNet 64 2 .

Original: Our main findings are: (1)  The STF objective improves the (average) performance of models with different seeds.
Revised: Our main findings are: (1) STF achieves new state-of-the-art FID scores for unconditional generation on CIFAR-10 benchmark. As shown in Table 1, The STF objective obtains a FID of 1 . 90 when incorporated with the EDM scheme. [Table]

Original: We observe that the STF objective  improves theaverage performance ofscore-based models trained on different seeds on CIFAR-10, for both ODE and SDE samplers. Remarkably, our objective obtain much better sample quality using ODE samplers, with an average
Revised: To the best of our knowledge, this is the lowest FID score on the unconditional CIFAR-10 generation task. In addition, the STF objective consistently improves the EDM across the two architectures. The STF objective improves the performance of different score-based models.

Original:  FID/Inception score gain of 3 . 39 / 0 . 25 on CIFAR-10,  and a FID gain of 2 . 22 on Celeba 64 2 .
Revised: We observe that the STF objective improves the FID/Inception scores of VE/VP/EDM on CIFAR-10, for most ODE and SDE samplers.

Original: The STF objective stabilizes the model performance at convergence.
Revised: STF consistently provides performance gains for VE across datasets.

Original: We also report standard deviations of performance for converged models with different seeds on CIFAR-10.
Revised: Remarkably, our objective achieves much better sample quality using ODE samplers for VE, with an FID score gain of 3 . 39 on CIFAR-10, and 2 . 22 on Celeba 64 2 .

Original: We observe that the models trained with the STF objective give more consistent results, with a smaller standard deviation of most metrics. Interestingly, the  STF objective provides more improvement using samplers with large step sizes, e.g . , RK45 method.
Revised: For VP, STF provides better results on the popular DDIM sampler, while suffering from a slight performance drop when using the RK45 sampler. The STF objective stabilizes the converged VE model with the RK45 sampler.

Original: In Figure 5, we show the FID scores with the number of function evaluations of ODE samplers on CIFAR-10 and CelebA 64 2 .
Revised: In Appendix E.1, we report the standard deviations of performance metrics for converged models with different seeds on CIFAR-10 with VE.

Original: To vary the NFE, we adjust the error tolerance in the RK45 method. The sample quality of STF objective degrades gracefully when decreasing the NFE. The STF objective consistently outperforms the vanilla one for all NFEs on CIFAR-10, and largely improves over the baseline when setting thetolerance to 5 e − 3 on CelebA 64 2 .
Revised: We observe that models trained with the

Original: It suggests that the STF has greater robustness to different step sizes.
Revised: STF objective give more consistent results, with a smaller standard deviation of used metrics.

Original: In Figure 4, we visualize the generative samples from the vanilla and STF objective. Notably, the generated samples from the STF objective do not contain noisy images, unlike the vanilla objective(marked by red boxes) on CIFAR-10.We provide more samples in Appendix D.
Revised: We further provide generated samples in Appendix F. One interesting observation is that when using the RK45 sampler for VE on CIFAR-10, the generated samples from the STF objective do not contain noisy images, unlike the vanilla objective on CIFAR-10.

Original: 6. A CCELERATING TRAINING OF SGM S
Revised: 6. A CCELERATING TRAINING OF SGM S

Original:  The variance-reduction techniques in neural network training can help to find better optima and achieve faster convergence rate (Wang et al., 2013; Defazio et al., 2014; Johnson & Zhang, 2013).
Revised: [Figure] The variance-reduction techniques in neural network training can help to find better optima and achieve faster convergence rate (Wang et al., 2013; Defazio et al., 2014; Johnson & Zhang, 2013).

Original: In
Revised: In

Original: Figure 6, we demonstrate the FID scorescomputed on 1k samples every 50k iterations during the course of training. We apply ODE sampler for FID evaluation, and measure the training time on two
Revised: Figure 4, we demonstrate the FID scores every 50k iterations during the course of training. Since our goal is to investigate relative performance during the training process, and because the FID scores computed on 1k samples are strongly correlated with the full FID scores on 50k sample (Song &

Original: NVIDIA A100 GPUs. Here we use the forward Euler method with 50 NFE for CelebA 64 2 for faster evaluations. 
Revised: Ermon, 2020), we report FID scores on 1k samples for faster evaluations. We apply ODE samplers for FID evaluation, and measure the training time on two NVIDIA A100 GPUs.

Original: For  fair comparison, we report the average FID scores of models trained by the vanilla and STF objective  versus the wall-clock training time (h). 
Revised: For a fair comparison, we report the average FID scores of models trained by the vanilla and STF objective on VE versus the wall-clock training time (h). [Figure]

Original: The STF objective achieves better FID scores with the same training time, although the calculation of  target field by a large reference batch introduces slight overhead (Algorithm 1).
Revised: The STF objective achieves better FID scores with the same training time, although the calculation of the target field by a large reference batch introduces slight overhead (Algorithm 1).

Original: In Figure 6(a), we show that the STF objective drastically accelerates the training of score-based models on CIFAR-10.
Revised: In Figure 4(a), we show that the STF objective drastically accelerates the training of score-based models on CIFAR-10.

Original: The STF objective achieves comparable FID scores with 3 .
Revised: The STF objective achieves comparable FID scores with 3 .

Original: 6 × less training time (25h versus 90h).
Revised: 6 × less training time (25h versus 90h).

Original: For CelebA 64 2 datasets, the training time improvement is less significant than on CIFAR-10.
Revised: For CelebA 64 2 datasets, the training time improvement is less significant than on CIFAR-10.

Original: Our hypothesis is that the STF objective is more effective when there are multiple well-separated modes in data distribution, e.g . , the ten classes in CIFAR-10, making the vanilla objective suffer from relatively larger variations in the intermediate phase.
Revised: Our hypothesis is that the STF objective is more effective when there are multiple well-separated modes in data distribution, e.g . , the ten classes in CIFAR-10, making the vanilla objective suffer from relatively larger variations in the intermediate phase.

Original: In addition, we observe that the converged models have better final performance when pairing with the STF objective on both datasets.[Figure]
Revised: In addition, we observe that the converged models have better final performance when pairing with the STF objective on both datasets.

Original:  E FFECTS OF THE REFERENCE BATCH SIZE
Revised: 6. E FFECTS OF THE REFERENCE BATCH SIZE

Original: [Figure] According to our theory (Theorem 2), the trace-ofcovariance of the training target in the intermediate phase decreases by a factor of the reference batch size.
Revised: According to our theory (Theorem 2), the upper bound of the trace-of-covariance of the STF target decreases proportionally to the reference batch size.

Original: Here we study the effects of the reference batch size ( n ) on model performances during training.
Revised: Here we study the effects of the reference batch size ( n ) on model performances during training.

Original: The FID scores are evaluated on 1 k samples using ODE sampler.
Revised: The FID scores are evaluated on 1 k samples using the RK45 sampler.

Original: As shown in Figure 7, models converge faster and produce better samples when increasing n .
Revised: As shown in Figure 5, models converge faster and produce better samples when increasing n .

Original: It suggests that smaller variations of the training targets can indeed speed up training and improve the final performances of score-based models.  
Revised: It suggests that smaller variations of the training targets can indeed speed up training and improve the final performances of score-based models.  R ELATED WORK

Revised: Different phases of SGMs.

Original: C ONCLUSION
Revised: The idea of SGMs having different phases has been explored in prior works though the motivations and definitions vary (Karras et al., 2022; Choi et al., 2022). Karras et al. (2022) argues that the training targets are difficult and unnecessary to learn in the very near field (small t in our Phase 1), whereas the training targets are always dissimilar to the true targets in the intermediate and far field (our Phase 2 and Phase 3). As a result, their solution is sampling t with a log-normal distribution to emphasize the relevant region (relatively large t in our Phase 1).

Revised: In contrast, we focus on reducing large training target variance in the intermediate and far field, and propose STF to better estimate the true target (cf.

Revised: Karras et al.

Revised: Choi et al.

Revised: (2022) identifies a key region where the model learns perceptually rich contents, and determines the training weights λ ( t ) based on the signal-to-noise ratio (SNR) at different t .

Revised: As SNR is monotonically decreasing over time, the resulting up-weighted region does not match our Phase 2 characterization.

Revised: In general, our proposed

Revised: STF method reduces the training target variance in the intermediate field and is complementary to previous improvements of SGMs.

Revised: Importance sampling.

Revised: The technique of importance sampling has been widely adopted in machine learning community, such as debiasing generative models (Grover et al., 2019), counterfactual learning (Swaminathan & Joachims, 2015) and reinforcement learning (Metelli et al., 2018).

Revised: Prior works using importance sampling to improve generative model training include reweighted wakesleep (RWS) (Bornschein & Bengio, 2014) and importance weighted autoencoders (IWAE) (Burda et al., 2015).

Revised: RWS views the original wake-sleep algorithm (Hinton et al., 1995) as importance sampling with one latent variable, and proposes to sample multiple latents to obtain gradient estimates with lower bias and variance.

Original: We highlight the instability of the current SGMs, and argue that the issue is  the result of unstable individual targets during training.
Revised: IWAE utilizes importance sampling with multiple latents to achieve greater flexibility of encoder training and tighter log-likelihood lower bound compared to the standard variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014). Variance reduction for Fisher divergence. One popular approach to score-matching is to minimize the Fisher divergence between true and predicted scores (Hyv ¨ arinen & Dayan, 2005). Wang et al.

Original: We identify the three phases with distinct behaviors.
Revised: (2020) links the Fisher divergence to denoising score-matching (Vincent, 2011) and studies the large variance problem (in O (1 /σ 4 t ) ) of the Fisher divergence when t → 0 . They utilize a control variate to reduce the variance. However, this is typically not a concern for current SGMs as the time-dependent objective can be viewed as multiplying the Fisher divergence by λ ( t ) =

Original: In particular, the high-variance targets appear in the intermediate phrase, negatively affecting the model performance.
Revised: σ 2 t , resulting in a finite-variance objective even when t → 0 .  C ONCLUSION We identify large target variance as a significant training issue affecting SGMs. We define three phases with distinct behaviors, and show that the high-variance targets appear in the intermediate phase.

Original: As a remedy, we present a generalized score-matching objective, Stable Target Field (STF) , whose formulation is analogous to the self-normalized importance sampling via a large reference batch.
Revised: As a remedy, we present a generalized score-matching objective, Stable Target Field (STF), whose formulation is analogous to the self-normalized importance sampling via a large reference batch.

Original: Albeit no longer an unbiased estimator, our proposed objective is asymptotically unbiased and reduces the trace-of-covariance of the training targets proportional to the size of the reference batch.
Revised: Albeit no longer an unbiased estimator, our proposed objective is asymptotically unbiased and reduces the trace-of-covariance of the training targets, which we demonstrate theoretically and empirically.

Original: We demonstrate theempirical effectiveness of our method on image generation tasks, and show that STF improves the performance, stability and training speed over the state-of-the-art SGMs.
Revised: We show the effectiveness of our method on image generation tasks, and show that STF improves the performance, stability, and training speed over various state-of-the-art SGMs.

Original: Future directions include a principled study on the effect of different reference batch sampling procedures.
Revised: Future directions include a principled study on the effect of different reference batch sampling procedures.

Original: The approach presented in this paper is simply uniform sampling from the whole dataset { x i } ni
Revised: Our presented approach is uniformly sampling from the whole dataset { x i } ni

Original: =2 ∼ p n − 1, and we expect  training SGMs with a reference batch of more samples in the neighborhood of x 1 (the sample from which x ( t ) is perturbed) will lead to an even better estimation of the score field.
Revised: =2 ∼ p n − 1, so we expect that training SGMs with a reference batch of more samples in the neighborhood of x 1 (the sample from which x ( t ) is perturbed) would lead to an even better estimation of the score field.

Original: Appendix A P ROOFS A. P ROOF FOR T HEOREM
Revised: Appendix A STF SPECIFIED WITH POPULAR SGM S

Revised: Here, we detail the practically used STF objectives in Section 6, which are built on the popular instances of SGMs, e.g . , VE, VP (Song et al., 2021b), and EDM (Karras et al., 2022).

Revised: VE and EDM For VE and EDM, the transition kernel is in the form of

Revised: [Equation]

Revised: VE has σ t = σ m (cid:16) σ M σ m (cid:17) t for some fixed σ m and σ M .

Revised: EDM has σ t = ( tσ 1 ρ M + (1 − t ) σ 1 ρ m ) ρ for some σ m and σ M , with ρ set to 7 in practice.

Revised: The STF objective for both VE and EDM at x ( t ) is then in the following form:

Revised: [Equation]

Revised: VP VP in its original formulation has the transition kernel as

Revised: [Equation]

Revised: for some β m and β M .

Revised: The STF objective for VP at x ( t ) is

Revised: E

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: where β t = − 14 t 2 ( β M − β m ) − 12 tβ m , and σ t = √ 1 − e 2 β t .

Revised: Note that as shown in Karras et al.

Revised: VP’s transition kernel can be reparameterized in the form of N ( x , σ 2 t I ) with a correspondingly revised sampling process.

Revised: Adopting this formulation, we would have the STF objective for VP the same as the one for VE and EDM with a different σ t .

Revised: B P ROOFS B. D ERIVATION OF E QUATION

Revised: Recall that the STF objective (Equation 6) at time t is

Revised: [Equation]

Revised: Swapping the sampling order and we get ℓ STF

Revised: [Equation]

Revised: , t

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: n − 1(cid:34)(cid:13)(cid:13) (cid:13) s θ ( x ( t ) , t ) − n (cid:88)

Revised: [Equation]

Revised: = p t | 0 ( x ( t ) | x k ) (cid:80) n

Revised: [Equation]

Revised: =1 p t |

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: x ( t ) log p t | 0 ( x ( t

Revised: [Equation]

Revised: x k ) (cid:13)(cid:13) (cid:13)  (cid:35) .

Revised: This means that with input x ( t ) and t , the model is optimized with

Revised: E

Revised: [Equation]

Revised: (cid:34)(cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: − n (cid:88) k

Revised: = p t | 0 ( x ( t ) | x k ) (cid:80) n

Revised: [Equation]

Revised: p t | 0 ( x ( t ) | x j )

Revised: [Equation]

Revised: x ( t ) log p t | 0 ( x ( t ) | x k ) (cid:13)(cid:13)(cid:13)  (cid:35) .

Revised: Taking its derivative w.r.t.

Revised: s θ ( x ( t ) , t ) results in

Revised: E

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: Setting it to 0, we have

Revised: [Equation]

Revised: [Equation]

Revised: Thus, we arrive at the minimizer of the STF objective (Equation 7):

Revised: [Equation]

Revised: B. P ROOF FOR T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then 
Revised: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then √ n (cid:0) s ∗ STF ( x ( t ) , t ) − ∇ x ( t ) log p t ( x ( t )) (cid:1) d −→

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that s ∗ STF ( x ( t ) , t ) is calculated via Equation 7.
Revised: Recall that s ∗ STF ( x ( t ) , t ) is calculated via Equation 7.

Original: The mean of the denominator in the expectation 1 n (cid:80) nj =1 p t | 0 ( x ( t ) | x j ) , for large n , approximate 1 n − 1 (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) , which in turn, 1 n − 1 (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) p −→ p t ( x ( t )) by WLLN.
Revised: The mean of the denominator in the expectation 1 n (cid:80) nj =1 p t | 0 ( x ( t ) | x j ) , for large n , approximate 1 n − 1 (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) , which in turn, 1 n − 1 (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) p −→ p t ( x ( t )) by WLLN.

Original: Similarly, for the mean of the remaining terms in the expectation, by CLT, we have
Revised: Similarly, for the mean of the remaining terms in the expectation, by CLT, we have

Original: [Equation]
Revised: [Equation]

Original: Putting them together via Slutsky’s theorem, we conclude the proof.
Revised: Putting them together via Slutsky’s theorem, we conclude the proof.

Original: A. P ROOF FOR T HEOREM
Revised: B. P ROOF FOR T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then 
Revised: Suppose ∀ t ∈ [0 , 1] , 0 < σ t < ∞ , then V STF ( t

Original: [Equation]
Revised: [Equation]

Original: , where D f is an f-divergence with f ( y )
Revised: where D f is an f-divergence with f ( y )

Original: = (cid:26) (1 /y − 1)
Revised: = (cid:26) (1 /y − 1)

Original: ( y < 1 . 5) 8 y/ 27 − 1 / 

Revised: ( y < 1 . 5) 8 y/ 27 − 1 /

Original: y ≥ 1 . 5) .
Revised: y ≥ 1 . 5) .

Original: Further, when n ≫ d and p 0 | t ( x | x ( t ))
Revised: Further, when n ≫ d and p 0 | t ( x | x ( t ))

Original: ≈ p 0 ( x ) for all x ( t ) , V STF ( t )
Revised: ≈ p 0 ( x ) for all x ( t ) , V STF ( t )

Original: ⪅ V v ( t ) n − 1 .
Revised: ⪅ V v ( t ) n − 1 .

Original: Proof.
Revised: Proof.

Original: Step 1: Make the likelihood weighting coefficients “independent”
Revised: Step 1: Make the likelihood weighting coefficients “independent”

Original: We first apply Hoeffding’s inequality for the set { x i } n i
Revised: We first apply Hoeffding’s inequality for the set { x i } ni

Original: =2 ∼ p n − 1to make the summation (cid:80) n j =2 p t | 0 ( x ( t ) | x j ) concentrate to its expectation ( n − 1) p t ( x ( t )) .
Revised: =2 ∼ p n − 1to make the summation (cid:80) nj =2 p t | 0 ( x ( t ) | x j ) concentrate to its expectation ( n − 1) p t ( x ( t )) .

Original: Since p t | 0 ( x ( t ) | x j ) ∈ (0 , 1 ( √ 2 πσ t ) d ) , we have :
Revised: Since p t | 0 ( x ( t ) | x j ) ∈ (0 , 1 ( √ 2 πσ t ) d ) , we have Pr

Original: [Equation]
Revised: [Equation]

Original: for ∀ γ1 ∈( 12 , 1) .
Revised: ∀ γ ∈

Original: Thus the summation can be re-expressed as:
Revised: Thus the summation can be re-expressed as:

Original: [Equation]
Revised: [Equation]

Original: The coefficient for x 1 is then
Revised: The coefficient for x 1 is then

Original: [Equation]
Revised: [Equation]

Original: The coefficient for x k , k ∈ { 2 , . . . , n } is:
Revised: The coefficient for x k , k ∈ { 2 , . . . , n } is:

Original: [Equation]
Revised: [Equation]

Original: Step 2: Re-express the trace-of-covariance by the “independent” weights
Revised: Step 2: Re-express the trace-of-covariance by the “independent” weights

Original: Plugging in the above formulation of coefficients, we can rewrite the trace-of-covariance for the new target as:
Revised: Plugging in the above formulation of coefficients, we can rewrite the trace-of-covariance for the new target as:

Original: [Equation]
Revised: [Equation]

Original: p t | 0 ( x ( t ) | x k ) − E x 1 ∼ p 0 | t ( x | x ( t ))

Original: E { x i } ni =2 ∼ p n − 1 ( x ) n (cid:88) k = p t | 0 ( x ( t ) | x k ) (cid:80) nj =1 p t | 0 ( x ( t ) | x j ) ∇ x ( t ) log p t | 0 ( x ( t ) | x k ) ∥≤ E x 1 ∼ p 0 | t ( x | x ( t )) E { x i } ni
Revised: [Table]

Original: =2 ∼ p n − 1 ( x ) ∥ n (cid:88) k =  ( n − 1) p t | 0 ( x ( t ) | x k ) p t ( x ( t )) ∇ x ( t ) log p t | 0 ( x ( t ) | x k ) − E x 1 ∼ p 0 | t ( x | x ( t )) E { x i } ni =2 ∼ p n − 1 ( x ) n (cid:88) k
Revised: Next, we examine the new trace-of-covariance term:

Original: =  ( n − 1) p t | 0 ( x ( t ) | x k ) p t ( x ( t )) ∇ x ( t ) log p t | 0 ( x ( t ) | x k ) + n (cid:88) k

Original: =

Original: O ( 1 n 2 ) ∇ x ( t ) log p t | 0 ( x ( t ) | x k ) − O ( 1 n ) E p ( x ) [ ∇ x ( t ) log p t | 0 ( x ( t ) | x )]

Original: ∥ 22 + O ( 1 n 2 )

Original: = ( n − 1)n (cid:88)

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: Tr(

Original: [Equation] ( p [Equation] [Equation]
Revised: We can further upper bound the trace-of-covariance term in Equation 9:

Original: t )

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation] ) p t ( x ( [Equation] )) ∇ x (
Revised: E

Original: [Equation]
Revised: [Equation]

Original: p t | 0 (

Original: [Equation]

Original: ( t

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: ) p

Original: [Equation]

Original: [Table] Next we examine the new trace-of-covariance term [Equation]
Revised: Taking the expectation w.r.t p t ( x ( t )) for both sides [Equation]

Original:  inequality)
Revised: we get [Equation] (Concavity of x 12 , Cauchy’s inequality)

Original: [Equation]
Revised: [Equation]

Original: where D f is an f -divergence with f ( y )
Revised: where D f is an f -divergence with f ( y )

Original: = (cid:26) (1 /y − 
Revised: = (cid:26) (1 /y − 1)

Original: [Equation]

Original: ( y < 1 . 5) 8 y/ 27 − 1 / 

Revised: ( y < 1 . 5) 8 y/ 27 − 1 /

Original: y ≥ 1 . 5) .
Revised: y ≥ 1 . 5) .

Original: Note that we choose this particular form of f ( y ) since it is the convex function with the tightest upper bound on ( 1 y − 1) 2 .
Revised: Note that we choose this particular form of f ( y ) since it is the convex function with the tightest upper bound on ( 1 y − 1) 2 .

Original: We can further upper bound the trace-of-covariance term in Equation 9:

Original: [Equation]

Original: Taking the expectation w.r.t p t ( x ( t )) for both sides, we get

Original: V STF ( t ) ≤

Original: [Equation]

Original: (Concavity of x 12 )

Original: B D ETAILS FOR THE BEHAVIOR OF V V ( t )
Revised: C D ETAILS FOR THE BEHAVIOR OF V V ( t )

Original: In Section 3, we demonstrate the behavior of V v ( t ) in the three phases on Two Gaussians and a subset of CIFAR-10.
Revised: In Section 3, we demonstrate the behavior of V v ( t ) in the three phases on Two Gaussians and a subset of CIFAR-10.

Original: Here we provide more details about the two datasets.
Revised: Here we provide more details about the two datasets.

Original: The distribution of the two Gaussian is 12 N ( µ , ˆ σ 2 I 64 × 64 ) + 12 N ( − µ , ˆ σ 2 I 64 × 64 ) , where µ = 0 .∗ 1 ∈ R 64 , and ˆ σ = 1 e − 4 .
Revised: The distribution of the two Gaussian is 12 N ( µ , ˆ σ 2 I 64 × 64 )+ 12 N ( − µ , ˆ σ 2 I 64 × 64 ) , where µ = 0 . 1 · 1 ∈ R 64 , and ˆ σ = 1 e − 4 .

Original: We estimate all the integrals in Equation 4 by sampling 1k points from the corresponding distributions.
Revised: We estimate all the integrals in Equation 4 by sampling 1k points from the corresponding distributions.

Original: For the subset of CIFAR-10, we uniformly sample 4096 images from
Revised: For the subset of CIFAR-10, we uniformly sample 4096 images from

Original: CIFAR-10 dataset, and assign uniform distribution on the discrete set.
Revised: CIFAR-10 dataset, and assign uniform distribution on the discrete set.

Original: We also approximate V v ( t ) by
Revised: We also approximate V v ( t ) by

Original: Monte Carlo estimation and sample 200 perturbations for each t .
Revised: Monte Carlo estimation and sample 200 perturbations for each t .

Original: We use VE SDE for all simulations, and set σ m = 1 e − 2 , σ M = 50 .
Revised: We use VE SDE for all simulations, and set σ m = 1 e − 2 , σ M = 50 .

Original: Interestingly, V v ( t ) is relatively large for the Two Gaussians distribution compared to CIFAR-10 (see
Revised: Interestingly, V v ( t ) is relatively large for the Two Gaussians distribution compared to CIFAR-10 (see

Original: Figure 2(b)) when t → 0 .
Revised: Figure 2(b)) when t → 0 .

Original: This can be explained by their continuous and discrete natures.
Revised: This can be explained by their continuous and discrete natures.

Original: For the two
Revised: For the two

Original: Gaussian distribution, we can rewrite V v ( t ) as
Revised: Gaussian distribution, we can rewrite V v ( t ) as

Original: [Equation]
Revised: [Equation]

Original: where α ( x ) =
Revised: where α ( x ) =

Original: 1+exp − 4 x T µ can be regarded as the probability that x comes from the Gaussian component N ( µ , ˆ σ 2 I 64 × 64 ) .
Revised: 1+exp − 4 x T µ can be regarded as the probability that x comes from the Gaussian component N ( µ , ˆ σ 2 I 64 × 64 ) .

Original: When t → 0 , obviously α ( x ( t ))
Revised: When t → 0 , obviously α ( x ( t ))

Original: (1 − α ( x ( t )))
Revised: (1 − α ( x ( t )))

Original: → 0 , and the termα ( x (
Revised: → 0 , and the term

Original: [Equation]
Revised: [Equation]

Original: t +ˆ
Revised: t +ˆ

Original: [Equation]
Revised: [Equation]

Original: vanishes.
Revised: vanishes.

Original: Hence lim t → 0 V v ( t ) ≈ lim t →ˆ σ 2 σ 2 t ( σ 2 t +ˆ σ 2 ) =
Revised: Hence lim t → 0 V v ( t ) ≈ lim [Equation]

Original: ˆ σ 2 σ 2 m ( σ 2 m +ˆ σ 2 ) , which can not be neglected when σ m is small.
Revised: m [Equation] 2 ) , which can not be neglected when σ m is small.

Original: On the other hand, we can effectively view the 4069 discrete samples as a mixture of 4096 0-variance Gaussians, i.e . , ˆ σ = 0 .
Revised: On the other hand, we can effectively view the 4069 discrete samples as a mixture of 4096 0-variance Gaussians, i.e . , ˆ σ = 0 .

Original: Thus by similar reasoning we could see lim t
Revised: Thus by similar reasoning we could see lim t

Original: → 0 V v ( t ) ≈ lim 
Revised: → 0 V v ( t ) ≈ lim t →

Original: t →ˆ σ 2 σ 2 t ( σ 2 t +ˆ σ 2 ) = 0 .
Revised: [Equation]

Original: C E XPERIMENTAL D ETAILS
Revised: D E XPERIMENTAL D ETAILS

Original: In this section we include more details about the training and sampling of score-based models by the
Revised: In this section, we include more details about the training and sampling of score-based models by the

Original: STF and vanilla objectives.
Revised: STF and vanilla objectives.

Original: All the experiments are run on two NVIDIA A100 GPUs.
Revised: All the experiments are run on two NVIDIA A100 GPUs.

Original: C. T RAINING
Revised: D. T RAINING

Original: We consider the CIFAR-10 and CelebA 64 2  in image geneation tasks.
Revised: We consider the CIFAR-10, CelebA 64 2 , and ImageNet 64 2 in image generation tasks.

Original: Following Song & Ermon (2020), we first center-crop the CelebA images and then resize them to 64 × 64 . 
Revised: Following Song & Ermon (2020), we first center-crop the CelebA images and then resize them to 64 × 64 . For

Original: We use the same set of hyper-parameters and the NCSN++ backbone and the continuous-time training objectives for forward SDEs in Song et al.
Revised: VE/VP, we use the same set of hyper-parameters and the NCSN++/DDPM++ backbones and the continuous-time training objectives for forward SDEs in Song et al.

Revised: (2021b).

Revised: For EDM, we adopt the improved hyper-parameters and architectures for NCSN++ in Karras et al.

Original: We set the reference batch size n to 4096  onCIFAR-10 andon CelebA 64 2  .
Revised: We set the reference batch size n to 4096 (VE,VP), 1024 (EDM) on CIFAR-10, 1024 on CelebA 64 2 andon ImageNet 64 2 .

Original: The training iteration is 1.3M on CIFAR-10 and 1M on CelebA 64 2 .
Revised: The training iteration is 1.3M on CIFAR-10 and 1M on CelebA 64 2 for VE/VP, and 200M images for EDM (Karras et al., 2022).

Original: The small batch size B in Algorithm 1 is the same as the batch size in the baseline score-based method.
Revised: The small batch size B in Algorithm 1 is the same as the batch size in the baseline score-based methods.

Revised: For model selection, we pick the checkpoint with the lowest FID per 50k iterations on 10k samples for computing all the scores, as in Song et al.

Revised: (2021b) for VE/VP, and per 2.5M images on 50k samples as in Karras et al.

Revised: (2022) for EDM.

Original: We repeat the experiment 3 times on CIFAR-10and 2 times on CelebA 64 2 for vanilla and STF, using different random seeds.
Revised: To measure the stability of converged VE models, we repeat the experiment 3 times on CIFAR-10 for vanilla and STF objectives, using different random seeds.

Original: For model selection, we pick the checkpoint with smallest FID per 50k iterations on 10k samples for computing all the scores, as in Song et al.
Revised: We quantitatively study the training overhead of STF. All the numbers are measured on two NVIDIA

Original:  In Table 3, we report the wall-clock training time (s) per 50 iterations.
Revised: A100 GPUs. In Table 3 and Table 4, we report the wall-clock training time (s) per 50 iterations/50k images on VE/EDM.

Original: We can see that the STF introduces additional overhead after incorporating the large reference batch.
Revised: We can see that the STF introduces additional overhead after incorporating the large reference batch.

Original: Since the calculation of the mini-batch target does not involving neural networks, the STF does not take significantly longer training time.
Revised: Since the calculation of the mini-batch target does not involve neural networks, the STF does not take significantly longer training time.

Original: Indeed, in Section 6.3 we show that STF achieves comparable or better performance within a shorter training time
Revised: Indeed, in Section 6.3 we show that STF achieves comparable or better performance within a shorter training time.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. S AMPLING
Revised: D. S AMPLING

Original: We adopt the RK45 method  and forward Euler method for reverse-time ODE.
Revised: We adopt the RK45 method for the backward ODE sampling of VE, and the DDIM sampler (Song et al., 2021a) for VP.

Original: For RK45 sampler, we use the function implemented in scipy.integrate.solve ivp with the tolerances atol = 1 e − 5 / 1 e − 4 , rtol = 1 e − 5 / 1 e − 4 for CIFAR-10/CelebA 64 2 .
Revised: For RK45 sampler of VE, we use the function implemented in scipy.integrate.solve ivp with the tolerances atol = 1 e − 5 / 1 e − 4 , rtol = 1 e − 5 / 1 e −for CIFAR-10/CelebA 64 2 .

Original: For forward Euler method, we discretize the ODE with constant step size determined by the number of function evaluations.

Original: We further adopt the exponential integrator Zhang & Chen (2022) as it empirically gives better results.

Original: As in Song et al.
Revised: As in Song et al.

Original: (2021), we set the terminal time to 1 e − 5 .
Revised: (2021b), we set the terminal time to 1 e − 5 / 1 e − 3 for

Revised: VE/VP.

Revised: For EDM, we adopt Heun’s 2 nd order method and the discretization scheme in (Karras et al., 2022), with 35 NFE.

Original: We use the predictor-corrector (PC) sampler for reverse-time SDE.
Revised: We use the predictor-corrector (PC) sampler for reverse-time SDE.

Original: We follow Song et al.
Revised: We follow Song et al.

Original: (2021) to set the Euler-Maruyama method as the predictor and the Langevin dynamics (MCMC) as the corrector.
Revised: (2021b) to set the Euler-Maruyama method as the predictor and the Langevin dynamics (MCMC) as the corrector.

Original: C. E VALUATIONS
Revised: D. E VALUATIONS

Original: For the evaluation, we compute the Fr´echet distance between 50000 samples and the pre-computed statistics of CIFAR-10 dataset in Heusel et al.
Revised: For the evaluation, we compute the Fr´echet distance between 50000 samples and the pre-computed statistics of CIFAR-10 and ImageNet 64 2 .

Original: For CelebA 64 × 64 , we adopt the setting in Song & Ermon (2020) where the distance is computed between 10000 samples and the test set. 
Revised: For CelebA 64 2 , we adopt the setting in Song & Ermon (2020) where the distance is computed between 10000 samples and the test set. E E XTRA E XPERIMENTS E. S TABILITY OF CONVERGED MODELS In Table

Original: D E XTENDED S AMPLES
Revised: [Table] we report the sample quality measured by FID/Inception score, and their standard deviations across random seeds on CIFAR-10. We can see that models trained with STF objective have lower variations of their final performances, in most cases. In particular, the standard deviation decreases from 4 .

Revised: 41 to 0 .

Revised: 06 for RK45 sampler on VE.

Revised: It suggests that the STF objective can stabilize the performance of converged models.

Revised: [Table]

Revised: E FFECTS OF STEP SIZE

Revised: In Figure

Revised: [Table]

Revised: we show the FID scores with the number of function evaluations of ODE samplers on

Revised: CIFAR-10 and CelebA 64 2 .

Original: We provide extended more samples from score-based models trained by vanilla/STF objective  on 
Revised: To vary the NFE, we adjust the error tolerance in the RK45 method. The sample quality of the STF objective degrades gracefully when decreasing the NFE. The STF objective consistently outperforms the vanilla one for all NFEs on CIFAR-10, and largely improves over the baseline when setting the tolerance to 5 e − 3 on CelebA 64 2 . It suggests that the STF has greater robustness to different step sizes

Original: CIFAR-10 and CelebA 64 2 by ODE samplers.
Revised: [Figure] F E XTENDED S AMPLES We provide extended samples from score-based models trained by vanilla/STF objective on CIFARand CelebA 64 2 by ODE samplers.

Original: For systematic comparison, we visualize samples from models trained on different seeds.
Revised: For systematic comparison, we visualize samples from models trained on different seeds.

Revised: We also provide samples generated by the state-of-the-art model — STF with EDM framework

Revised: [Figure]

Original: D. CIFAR- 
Revised: F. CIFAR-

Original: In Figure 8, we visualize the samples produced by different methods across random seeds.
Revised: In Figure 7, we visualize the samples produced by different methods across random seeds for VE.

Revised: We use the RK45 sampler for sampling.

Original: We observe that the model trained by the vanilla objective can produce noisy images (in red boxes), and the image quality has great variability across different random seeds.
Revised: We observe that the model trained by the vanilla objective can produce noisy images (in red boxes), and the image quality has great variability across different random seeds.

Original: In contrast, model trained by STF objective generate clean and consistent samples with varying random seeds.
Revised: In contrast, models trained by STF objective generate clean and consistent samples with varying random seeds.

Revised: In Figure 8, we further provide samples from a model trained by STF under the EDM framework (Karras et al., 2022).

Revised: The model is the current state-of-the-art on the unconditional CIFAR-10 generation task.

Original: D. C ELEB A 64 ×
Revised: F. C ELEB A 64 ×

Original: In Figure 9, wefurther provide samples from models trained on vanilla and STF objectives.
Revised: In Figure 9, we provide samples from models trained on vanilla and STF objectives with VE.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]