Original: S AMPLING IS AS EASY AS LEARNING THE SCORE : THEORY FOR DIFFUSION MODELS WITH MINIMAL DATA ASSUMPTIONS
Revised: S AMPLING IS AS EASY AS LEARNING THE SCORE : THEORY FOR DIFFUSION MODELS WITH MINIMAL DATA ASSUMPTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We provide theoretical convergence guarantees for score-based generative models (SGMs) such as denoising diffusion probabilistic models (DDPMs), which constitute the backbone of large-scale real-world generative models such as DALL · E 2.
Revised: We provide theoretical convergence guarantees for score-based generative models (SGMs) such as denoising diffusion probabilistic models (DDPMs), which constitute the backbone of large-scale real-world generative models such as DALL · E 2.

Original: Our main result is that, assuming accurate score estimates, such SGMs can efficiently sample from essentially any realistic data distribution.
Revised: Our main result is that, assuming accurate score estimates, such SGMs can efficiently sample from essentially any realistic data distribution.

Original: In contrast to prior works, our results (1) hold for an L 2 -accurate score estimate (rather than
Revised: In contrast to prior works, our results (1) hold for an L 2 -accurate score estimate (rather than

Original: L ∞ -accurate); (2) do not require restrictive functional inequality conditions that preclude substantial non-log-concavity; (3) scale polynomially in all relevant problem parameters; and (4) match state-of-the-art complexity guarantees for discretization of the Langevin diffusion, provided that the score error is sufficiently small.
Revised: L ∞ -accurate); (2) do not require restrictive functional inequality conditions that preclude substantial non-log-concavity; (3) scale polynomially in all relevant problem parameters; and (4) match state-of-the-art complexity guarantees for discretization of the Langevin diffusion, provided that the score error is sufficiently small.

Original: We view this as strong theoretical justification for the empirical success of
Revised: We view this as strong theoretical justification for the empirical success of

Original: SGMs.
Revised: SGMs.

Original: We also examine SGMs based on the critically damped Langevin diffusion (CLD).
Revised: We also examine SGMs based on the critically damped Langevin diffusion (CLD).

Original: Contrary to conventional wisdom, we provide evidence that the use of the
Revised: Contrary to conventional wisdom, we provide evidence that the use of the

Original: CLD does not reduce the complexity of SGMs.
Revised: CLD does not reduce the complexity of SGMs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Score-based generative models (SGMs) are a family of generative models which achieve state-of-theart performance for generating audio and image data (Sohl-Dickstein et al., 2015;
Revised: Score-based generative models (SGMs) are a family of generative models which achieve state-ofthe-art performance for generating audio and image data (Sohl-Dickstein et al., 2015;

Original: Ho et al., 2020; Dhariwal & Nichol, 2021; Kingma et al., 2021; Song et al., 2021a;b; Vahdat et al., 2021); see, e.g., the recent surveys (Cao et al., 2022; Croitoru et al., 2022; Yang et al., 2022).
Revised: Ho et al., 2020; Dhariwal & Nichol, 2021; Kingma et al., 2021; Song et al., 2021a;b; Vahdat et al., 2021); see, e.g., the recent surveys (Cao et al., 2022; Croitoru et al., 2022; Yang et al., 2022).

Original: One notable example of an SGM are denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020), which are a key component in large-scale generative models such as DALL · E(Ramesh et al., 2022).
Revised: For example, denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a key component in large-scale generative models such as DALL · E 2 (Ramesh et al., 2022).

Original: As the importance of SGMs continues to grow due to newfound applications in commercial domains, it is a pressing question of both practical and theoretical concern to understand the mathematical underpinnings which explain their startling empirical successes.
Revised: As the importance of SGMs continues to grow due to newfound applications in commercial domains, it is a pressing question of both practical and theoretical concern to understand the mathematical underpinnings which explain their startling empirical successes.

Original: As we explain in Section 2, at their mathematical core, SGMs consist of two stochastic processes, the forward process and the reverse process.
Revised: As we explain in Section 2, at their mathematical core, SGMs consist of two stochastic processes, the forward process and the reverse process.

Original: The forward process transforms samples from a data distribution q (e.g., images) into noise, whereas the reverse process transforms noise into samples from q , hence performing generative modeling.
Revised: The forward process transforms samples from a data distribution q (e.g., images) into noise, whereas the reverse process transforms noise into samples from q , hence performing generative modeling.

Original: Running the reverse process requires estimating the score function of the law of the forward process; this is typically done by training neural networks on a score matching objective (Hyv¨arinen, 2005; Vincent, 2011; Song & Ermon, 2019).
Revised: Running the reverse process requires estimating the score function of the law of the forward process; this is typically done by training neural networks on a score matching objective (Hyv¨arinen, 2005; Vincent, 2011; Song & Ermon, 2019).

Original: Providing precise guarantees for estimation of the score function is difficult, as it requires an understanding of the non-convex training dynamics of neural network optimization that is currently out of reach.
Revised: Providing precise guarantees for estimation of the score function is difficult, as it requires an understanding of the non-convex training dynamics of neural network optimization that is currently out of reach.

Original: However, given the empirical success of neural networks on the score estimation task, a natural and important question is whether accurate score estimation implies that SGMs provably converge to the true data distribution in realistic settings.
Revised: However, given the empirical success of neural networks on the score estimation task, a natural and important question is whether accurate score estimation implies that SGMs provably converge to the true data distribution in realistic settings.

Original: This is a surprisingly delicate question, as even with accurate score estimates, as we explain in Section 2.1, there are several other sources of error which could cause the SGM to fail to converge.
Revised: This is a surprisingly delicate question, as even with accurate score estimates, as we explain in Section 2.1, there are several other sources of error which could cause the SGM to fail to converge.

Original: Indeed, despite a flurry of recent work (Block et al., 2020; De Bortoli et al., 2021; De Bortoli, 2022; Lee et al., 2022a; Pidstrigach, 2022; Liu et al., 2022), prior analyses fall short of answering this question, for (at least) one of three main reasons:
Revised: Indeed, despite a flurry of recent work (Block et al., 2020; De Bortoli et al., 2021; De Bortoli, 2022; Lee et al., 2022a; Pidstrigach, 2022; Liu et al., 2022), prior analyses fall short of answering this question, for (at least) one of three main reasons:

Original: Super-polynomial convergence.
Revised: Super-polynomial convergence.

Original: The bounds obtained are not quantitative (e
Revised: The bounds obtained are not quantitative (e

Original: .g
Revised: g.,

Original: (De Bortoli et al.,
Revised: De Bortoli et al

Original: ; Pidstrigach
Revised: Pidstrigach,

Original:  or scale exponentially in the dimension and other problem parameters
Revised: , or scale exponentially in the dimension and other problem parameters

Original: like time and smoothness (Block et al., 2020; De Bortoli, 2022; Liu et al., 2022), and hence are typically vacuous for the high-dimensional settings of interest in practice.
Revised: like time and smoothness (Block et al., 2020; De Bortoli, 2022; Liu et al., 2022), and hence are typically vacuous for the high-dimensional settings of interest in practice.

Original: Strong assumptions on the data distribution.
Revised: Strong assumptions on the data distribution.

Original: The bounds require strong assumptions on the true data distribution, such as a log-Sobelev inequality (LSI) (see, e.g., (Lee et al., 2022a)).
Revised: The bounds require strong assumptions on the true data distribution, such as a log-Sobelev inequality (LSI) (see, e.g., Lee et al., 2022a).

Original: While the LSI is slightly weaker than  log-concavity, it ultimately precludes the presence of substantial non-convexity, which impedes the application of these results to complex and highly multi-modal real-world data distributions.
Revised: While the LSI is slightly weaker than strong log-concavity, it ultimately precludes the presence of substantial non-convexity, which impedes the application of these results to complex and highly multi-modal real-world data distributions.

Original: Indeed, obtaining a polynomial-time convergence analysis for SGMs that holds for multi-modal distributions was posed as an open question in (Lee et al., 2022a).
Revised: Indeed, obtaining a polynomial-time convergence analysis for SGMs that holds for multi-modal distributions was posed as an open question in (Lee et al., 2022a).

Original: Strong assumptions on the score estimation error.
Revised: Strong assumptions on the score estimation error.

Original: The bounds require that the score estimate is L ∞ -accurate (i.e., uniformly accurate), as opposed to L 2 -accurate (see, e.g., (De Bortoli et al., 2021)).
Revised: The bounds require that the score estimate is L ∞ -accurate (i.e., uniformly accurate), as opposed to L 2 -accurate (see, e.g., De Bortoli et al., 2021).

Original: This is problematic because the score matching objective is an L 2 loss (see Section A.1 in the supplement), and there are empirical studies suggesting that in practice, the score estimate is not in fact L ∞ -accurate (e.g., (Zhang & Chen, 2022)).
Revised: This is problematic because the score matching objective is an L 2 loss (see Section A.1 in the supplement), and there are empirical studies suggesting that in practice, the score estimate is not in fact L ∞ -accurate (e.g., Zhang & Chen, 2022).

Original: Intuitively, this is because we cannot expect that the score estimate we obtain will be accurate in regions of space where the true density is very low, simply because we do not expect to see many (or indeed, any) samples from there.
Revised: Intuitively, this is because we cannot expect that the score estimate we obtain will be accurate in regions of space where the true density is very low, simply because we do not expect to see many (or indeed, any) samples from there.

Original: Providing an analysis which goes beyond these limitations is a pressing first step towards theoretically understanding why SGMs actually work in practice.
Revised: Providing an analysis which goes beyond these limitations is a pressing first step towards theoretically understanding why SGMs actually work in practice.

Original: 1. O UR CONTRIBUTIONS
Revised: 1. O UR CONTRIBUTIONS

Original: In this work, we take a step towards bridging theory and practice by providing a convergence guarantee for SGMs, under realistic (in fact, quite minimal) assumptions, which scales polynomially in all relevant problem parameters.
Revised: In this work, we take a step towards bridging theory and practice by providing a convergence guarantee for SGMs, under realistic (in fact, quite minimal) assumptions, which scales polynomially in all relevant problem parameters.

Original: Namely, our main result (Theorem 2) only requires the following assumptions on the data distribution q , which we make more quantitative in Section 3:
Revised: Namely, our main result (Theorem 2) only requires the following assumptions on the data distribution q , which we make more quantitative in Section 3:

Original: A

Revised: A

Original: The score function of the forward process is L -Lipschitz.
Revised: The score function of the forward process is L -Lipschitz.

Original: A2 The (2 + η ) -th moment of q is finite, where η > 0 is an arbitrarily small constant.
Revised: A2 The (2 + η ) -th moment of q is finite, where η > 0 is an arbitrarily small constant.

Original: A3 The data distribution q has finite KL divergence w.r.t.
Revised: A3 The data distribution q has finite KL divergence w.r.t.

Original: the standard Gaussian.
Revised: the standard Gaussian.

Original: We note that all of these assumptions are either standard or, in the case of A2 , far weaker than what is needed in prior work.
Revised: We note that all of these assumptions are either standard or, in the case of A2 , far weaker than what is needed in prior work.

Original: Crucially, unlike prior works, we do not assume log-concavity, an LSI, or dissipativity; hence, our assumptions cover arbitrarily non-log-concave data distributions.
Revised: Crucially, unlike prior works, we do not assume log-concavity, an LSI, or dissipativity; hence, our assumptions cover arbitrarily non-log-concave data distributions.

Original: Our main result is summarized informally as follows.
Revised: Our main result is summarized informally as follows.

Original: Theorem 1 (informal, see Theorem 2) .
Revised: Theorem 1 (informal, see Theorem 2) .

Original: Under assumptions A1-A3 , and if the score estimation error in L 2 is sufficiently polynomially small in 1 /d , 1 /L , and ε  , then with an appropriate choice of step size, the SGM outputs a measure which is ε -close in total variation (TV)  to q in (cid:101) O
Revised: Under assumptions A1-A3 , and if the score estimation error in L 2 is at most (cid:101) O ( ε ) , then with an appropriate choice of step size, the SGM outputs a measure which is ε -close in total variation (TV) distance to q in (cid:101) O

Original: ( L 2 d/ε 2 ) iterations.
Revised: ( L 2 d/ε 2 ) iterations.

Original: Our iteration complexity is quite tight: it matches state-of-the-art discretization guarantees for the
Revised: Our iteration complexity is quite tight: it matches state-of-the-art discretization guarantees for the

Original: Langevin diffusion (Vempala & Wibisono, 2019; Chewi et al., 2021a).
Revised: Langevin diffusion (Vempala & Wibisono, 2019; Chewi et al., 2021a).

Original: We find Theorem 1 surprising, because it shows that SGMs can sample from the data distribution q with polynomial complexity, even when q is highly non-log-concave (a task that is usually intractable), provided that one has access to an accurate score estimator .
Revised: We find Theorem 1 surprising, because it shows that SGMs can sample from the data distribution q with polynomial complexity, even when q is highly non-log-concave (a task that is usually intractable), provided that one has access to an accurate score estimator .

Original: This answers the open question of (Lee et al., 2022a) regarding whether or not SGMs can sample from multimodal distributions, e.g., mixtures of distributions with bounded log-Sobolev constant.
Revised: This answers the open question of (Lee et al., 2022a) regarding whether or not SGMs can sample from multimodal distributions, e.g., mixtures of distributions with bounded log-Sobolev constant.

Original: In the context of neural networks, our result implies that so long as the neural network succeeds at the score estimation task, the remaining part of the SGM algorithm based on the diffusion model is completely principled, in that it admits a strong theoretical justification.
Revised: In the context of neural networks, our result implies that so long as the neural network succeeds at the score estimation task, the remaining part of the SGM algorithm based on the diffusion model is completely principled, in that it admits a strong theoretical justification.

Original: In general, learning the score function is also a difficult task.
Revised: In general, learning the score function is also a difficult task.

Original: Nevertheless, our result opens the door to further investigations, such as: do score functions for real-life data have intrinsic (e.g., low-dimensional) structure which can be exploited by neural networks?
Revised: Nevertheless, our result opens the door to further investigations, such as: do score functions for real-life data have intrinsic (e.g., low-dimensional) structure which can be exploited by neural networks?

Original: A positive answer to this question, combined with our sampling result, would then provide an end-to-end guarantee for SGMs.
Revised: A positive answer to this question, combined with our sampling result, would then provide an end-to-end guarantee for SGMs.

Original: More generally, our result can be viewed as a black-box reduction of the task of sampling to the task of learning the score function of the forward process, at least for distributions satisfying our mild assumptions.
Revised: More generally, our result can be viewed as a black-box reduction of the task of sampling to the task of learning the score function of the forward process, at least for distributions satisfying our mild assumptions.

Original: Existing computational hardness results for learning natural high-dimensional distributions like mixtures of Gaussians (Diakonikolas et al., 2017; Bruna et al., 2021; Gupte et al.,
Revised: Existing computational hardness results for learning natural high-dimensional distributions like mixtures of Gaussians (Diakonikolas et al., 2017; Bruna et al., 2021; Gupte et al.,

Original: and pushforwards of Gaussians by shallow ReLU networks (Daniely & Vardi, 2021; Chen et al., 2022a;b) thus immediately imply hardness of score estimation for these distributions.
Revised: and pushforwards of Gaussians by shallow ReLU networks (Daniely & Vardi, 2021; Chen et al., 2022a;b) thus immediately imply hardness of score estimation for these distributions.

Original: To our knowledge this yields the first known information-computation gaps for this task.
Revised: To our knowledge this yields the first known information-computation gaps for this task.

Original: Arbitrary distributions with bounded support.
Revised: Arbitrary distributions with bounded support.

Original: The assumption that the score function is Lipschitz entails in particular that the data distribution has a density w.r.t.
Revised: The assumption that the score function is Lipschitz entails in particular that the data distribution has a density w.r.t.

Original: Lebesgue measure; in particular, our theorem fails when q satisfies the manifold hypothesis, i.e., is supported on a lower-dimensional submanifold of R d .
Revised: Lebesgue measure; in particular, our theorem fails when q satisfies the manifold hypothesis, i.e., is supported on a lower-dimensional submanifold of R d .

Original: But this is for good reason: it is not possible to obtain non-trivial TV guarantees, because the output distribution of the SGM has full support.
Revised: But this is for good reason: it is not possible to obtain non-trivial TV guarantees, because the output distribution of the SGM has full support.

Original: Instead, we show in Section 3.2 that we can obtain polynomial convergence guarantees in the bounded Lipschitz metric by stopping the SGM algorithm early,  under the sole assumption that the data distribution q has bounded support, without assuming that q has a density.
Revised: Instead, we show in Section 3.2 that we can obtain polynomial convergence guarantees in the bounded Lipschitz metric by stopping the SGM algorithm early, or in the Wasserstein metric by an additional truncation step, under the sole assumption that the data distribution q has bounded support, without assuming that q has a density.

Original: Since data distributions encountered in real life satisfy this assumption, our results yield the following compelling takeaway:
Revised: Since data distributions encountered in real life satisfy this assumption, our results yield the following compelling takeaway:

Original: Given an L 2 -accurate score estimate, SGMs can sample from (essentially) any data distribution .
Revised: Given an L 2 -accurate score estimate, SGMs can sample from (essentially) any data distribution .

Original: Critically damped Langevin diffusion (CLD).
Revised: Critically damped Langevin diffusion (CLD).

Original: Using our techniques, we also investigate the use of the critically damped Langevin diffusion (CLD) for SGMs, which was proposed in (Dockhorn et al., 2022).
Revised: Using our techniques, we also investigate the use of the critically damped Langevin diffusion (CLD) for SGMs, which was proposed in (Dockhorn et al., 2022).

Original: Although numerical experiments and intuition from the log-concave sampling literature suggest that the CLD could potentially speed up sampling via SGMs, we provide theoretical evidence to the contrary.
Revised: Although numerical experiments and intuition from the log-concave sampling literature suggest that the CLD could potentially speed up sampling via SGMs, we provide theoretical evidence to the contrary.

Original: Based on this, in Section 3.3, we conjecture that SGMs based on the CLD do not exhibit improved dimension dependence compared to the original DDPM algorithm.
Revised: Based on this, in Section 3.3, we conjecture that SGMs based on the CLD do not exhibit improved dimension dependence compared to the original DDPM algorithm.

Original: 1. P RIOR WORK
Revised: 1. P RIOR WORK

Original: We now provide a detailed comparison to prior work.
Revised: We now provide a detailed comparison to prior work.

Original: By now, there is a vast literature on providing precise complexity estimates for log-concave sampling; see, e.g., (Chewi, 2022) for an exposition on recent developments.
Revised: By now, there is a vast literature on providing precise complexity estimates for log-concave sampling; see, e.g., Chewi (2022) for an exposition on recent developments.

Original: The proofs in this work build upon the techniques developed in this literature.
Revised: The proofs in this work build upon the techniques developed in this literature.

Original: However, our work addresses the significantly more challenging setting of non-log-concave sampling.
Revised: However, our work addresses the significantly more challenging setting of non-log-concave sampling.

Original: The work of De Bortoli et al.
Revised: The work of De Bortoli et al.

Original: (2021) provides guarantees for the diffusion Schr ¨ odinger bridge (Song et al., 2021b).
Revised: (2021) provides guarantees for the diffusion Schr¨odinger bridge (Song et al., 2021b).

Original: However, as previously mentioned their result is not quantitative, and they require an
Revised: However, as previously mentioned their result is not quantitative, and they require an

Original: L ∞ -accurate score estimate.
Revised: L ∞ -accurate score estimate.

Revised: The works Block et al.

Original: The works (Block et al., 2020; Lee et al., 2022a; Liu et al., 2022) 1 instead analyze SGMs under the more realistic assumption of an L 2 -accurate score estimate.
Revised: ; Lee et al. (2022a); Liu et al. (2022) instead analyze SGMs under the more realistic assumption of an L 2 -accurate score estimate.

Original: However, the bounds of (Block et al., 2020; Liu et al., 2022) suffer from exponential dependencies on parameters like dimension and smoothness, whereas the bounds of (Lee et al., 2022a) require q to satisfy an LSI.
Revised: However, the bounds of Block et al. ; Liu et al. (2022) suffer from exponential dependencies on parameters like dimension and smoothness, whereas the bounds of Lee et al. (2022a) require q to satisfy an LSI.

Original: The recent work of De Bortoli (2022), motivated by the manifold hypothesis , considers a different pointwise assumption on the score estimation error which allows the error to blow up at time 0 and at spatial ∞ .
Revised: The recent work of De Bortoli (2022), motivated by the manifold hypothesis , considers a different pointwise assumption on the score estimation error which allows the error to blow up at time 0 and at spatial ∞ .

Original: We discuss the manifold setting in more detail in Section 3.2.
Revised: We discuss the manifold setting in more detail in Section 3.2.

Original: Unfortunately, the bounds of De Bortoli (2022) also scale exponentially in problem parameters such as the manifold diameter.
Revised: Unfortunately, the bounds of De Bortoli (2022) also scale exponentially in problem parameters such as the manifold diameter.

Original: We also mention that the use of reversed SDEs for sampling is implicit in the interpretation of the proximal sampler (Lee et al., 2021) given by Chen et al.
Revised: We also mention that the use of reversed SDEs for sampling is implicit in the interpretation of the proximal sampler (Lee et al., 2021) given by Chen et al.

Original: (2022c).
Revised: (2022c).

Original: Our work can be viewed as expanding upon the theory of Chen et al.
Revised: Our work can be viewed as expanding upon the theory of Chen et al.

Original: (2022c)
Revised: (2022c)

Original: using a different forward channel (the OU process).
Revised: using a different forward channel (the OU process).

Original: Concurrent work.
Revised: Concurrent work.

Original: Very recently, Lee et al.
Revised: Very recently, Lee et al.

Original: (2022b) independently obtained results similar to our results for DDPM.
Revised: (2022b) independently obtained results similar to our results for DDPM.

Original: While our assumptions are technically somewhat incomparable (they assume the score error can vary with time but assume the data is compactly supported), our quantitative bounds are stronger.
Revised: While our assumptions are technically somewhat incomparable (they assume the score error can vary with time but assume the data is compactly supported), our quantitative bounds are stronger.

Original: Additionally, the upper and lower bounds for CLD are unique to our work.
Revised: Additionally, the upper and lower bounds for CLD are unique to our work.

Original:  B ACKGROUND ON SGM S
Revised:  B ACKGROUND ON SGM S

Original: Throughout this paper, given a probability measure p which admits a density w.r.t.
Revised: Throughout this paper, given a probability measure p which admits a density w.r.t.

Original: Lebesgue measure, we abuse notation and identify it with its density function.
Revised: Lebesgue measure, we abuse notation and identify it with its density function.

Original: Additionally, we will let q denote the data distribution from which we want to generate new samples.
Revised: Additionally, we will let q denote the data distribution from which we want to generate new samples.

Original: We assume that q is a probability measure on R d with full support, and that it admits a smoothdensity. Except in Section 3.2, we assume that q admits a smooth density written q = exp( − U ) , where U : R d → R is the potential .
Revised: We assume that q is a probability measure on R d with full support, and that it admits a smooth density written q = exp( − U ) (we relax this assumption in Section 3.2).

Original: In this section, we provide a brief exposition to SGMs, following Song et al.
Revised: In this section, we provide a brief exposition to SGMs, following Song et al.

Original: (2021b).
Revised: (2021b).

Original: 2. B ACKGROUND ON DENOISING DIFFUSION PROBABILISTIC MODELING (DDPM)
Revised: 2. B ACKGROUND ON DENOISING DIFFUSION PROBABILISTIC MODELING (DDPM)

Original: Forward process.
Revised: Forward process.

Original: In denoising diffusion probabilistic modeling (DDPM), we start with a forward process, which is a stochastic differential equation (SDE).
Revised: In denoising diffusion probabilistic modeling (DDPM), we start with a forward process, which is a stochastic differential equation (SDE).

Original: For clarity, we consider the simplest possible choice, which is the Ornstein–Uhlenbeck (OU) process
Revised: For clarity, we consider the simplest possible choice, which is the Ornstein–Uhlenbeck (OU) process

Original: [Equation]
Revised: [Equation]

Original: where ( B t ) t ≥ 0 is a standard Brownian motion in R d .
Revised: where ( B t ) t ≥ 0 is a standard Brownian motion in R d .

Original: The OU process is the unique time-homogeneous
Revised: The OU process is the unique time-homogeneous

Original: Markov process which is also a Gaussian process, with stationary distribution equal to the standard
Revised: Markov process which is also a Gaussian process, with stationary distribution equal to the standard

Original: Gaussian distribution γ d on R d .
Revised: Gaussian distribution γ d on R d .

Original: In practice, it is also common to introduce a positive smooth function g : R + → R and consider the time-rescaled OU process
Revised: In practice, it is also common to introduce a positive smooth function g : R + → R and consider the time-rescaled OU process

Original: [Equation]
Revised: [Equation]

Original: but in this work we stick with the choice g ≡ 1 .
Revised: Although our analysis could be extended to consider these variants, in this work we stick with the choice g ≡ 1 for simplicity; see Song et al. (2021b) for further discussion.

Original: The forward process has the interpretation of transforming samples from the data distribution q into pure noise.
Revised: The forward process has the interpretation of transforming samples from the data distribution q into pure noise.

Original: From the well-developed theory of Markov diffusions, it is known that if q t := law( X t ) denotes the law of the OU process at time t , then q t → γ d exponentially fast in various divergences and metrics such as the 2 -Wasserstein metric W 2 ; see Bakry et al.
Revised: From the well-developed theory of Markov diffusions, it is known that if q t := law( X t ) denotes the law of the OU process at time t , then q t → γ d exponentially fast in various divergences and metrics such as the 2 -Wasserstein metric W 2 ; see Bakry et al.

Original: Reverse process.
Revised: Reverse process.

Original: If we reverse the forward process (1) in time, then we obtain a process that transforms noise into samples from q , which is the aim of generative modeling.
Revised: If we reverse the forward process (1) in time, then we obtain a process that transforms noise into samples from q , which is the aim of generative modeling.

Original: In general, suppose that we have an SDE of the form
Revised: In general, suppose that we have an SDE of the form

Original: [Equation]
Revised: [Equation]

Original: where ( σ t ) t ≥ 0 is a deterministic matrix-valued process.
Revised: where ( σ t ) t ≥ 0 is a deterministic matrix-valued process.

Original: Then, under mild conditions on the process (e.g., (F ¨ ollmer, 1985; Cattiaux et al., 2022)), which are satisfied for all processes under consideration in this work, the reverse process also admits an SDE description.
Revised: Then, under mild conditions on the process (e.g., F ¨ ollmer, 1985; Cattiaux et al., 2022), which are satisfied for all processes under consideration in this work, the reverse process also admits an SDE description.

Original: Namely, if we fix the terminal time T > 0 and set¯
Revised: Namely, if we fix the terminal time T > 0 and set

Original: [Equation]
Revised: [Equation]

Original: for t ∈ [ [Equation] then the process ( ¯ X ← t ) t ∈ [0 ,T ] satisfies the SDE
Revised: then the process ( ¯ X ← t ) t ∈ [0 ,T ] satisfies the SDE

Original: [Equation]
Revised: [Equation]

Original: where the backwards drift satisfies the relation
Revised: where the backwards drift satisfies the relation

Original: [Equation]
Revised: [Equation]

Original: Applying this to the forward process (1), we obtain the reverse process
Revised: Applying this to the forward process (1), we obtain the reverse process

Original: [Equation]
Revised: [Equation]

Original: where now ( B t ) t ∈ [0 ,T ] is the reversed Brownian motion.
Revised: where now ( B t ) t ∈ [0 ,T ] is the reversed Brownian motion.

Original: Here, ∇ ln q t is called the score function for q t .
Revised: Here, ∇ ln q t is called the score function for q t .

Original: Since q (and hence q t for t ≥ 0 ) is not explicitly known, in order to implement the reverse process the score function must be estimated on the basis of samples.
Revised: Since q (and hence q t for t ≥ 0 ) is not explicitly known, in order to implement the reverse process the score function must be estimated on the basis of samples.

Original: The mechanism behind this is the idea of score matching which goes back to Hyv¨arinen (2005); Vincent (2011): roughly speaking,
Revised: The mechanism behind this is the idea of score matching which goes back to Hyv¨arinen (2005); Vincent (2011): roughly speaking,

Original: Gaussian integration by parts implies that minimizing the L 2 ( q t ) loss achieved by an estimate s t for the score ∇ ln q t is equivalent to minimizing the L 2 ( q t ) loss in predicting, given a sample from the forward process at time t , what noise was applied to the corresponding sample at time 0 to obtain it.
Revised: Gaussian integration by parts implies that minimizing the L 2 ( q t ) loss achieved by an estimate s t for the score ∇ ln q t is equivalent to minimizing the L 2 ( q t ) loss in predicting, given a sample from the forward process at time t , what noise was applied to the corresponding sample at time 0 to obtain it.

Original: We defer an exposition of the details of score matching to Sections A.1 and D of the supplement.
Revised: We defer an exposition of the details of score matching to Sections A.1 and D of the supplement.

Original: In light of this, it is thus most natural to assume an L 2 ( q t ) error bound E q t [ ∥ s t − ∇ ln q t ∥ 2 ] ≤ ε 2score for the score estimator s t .
Revised: In light of this, it is thus most natural to assume an L 2 ( q t ) error bound E q [Equation] t ∥ 2 ] ≤ ε 2score for the score estimator s t .

Original: If s t is taken to be the empirical risk minimizer for a suitable function class, then guarantees for the L 2 ( q t ) error can be obtained via standard statistical analysis, see e.g. Block et al.
Revised: If s t is taken to be the empirical risk minimizer for a suitable function class, then guarantees for the L 2 ( q t ) error can be obtained via standard statistical analysis, see, e.g., Block et al.

Original: Discretization and implementation.
Revised: Discretization and implementation.

Original: We now discuss the final steps required to obtain an implementable algorithm.
Revised: We now discuss the final steps required to obtain an implementable algorithm.

Original: First, in the learning phase, given samples ¯ X (1)0 , . . .
Revised: First, in the learning phase, given samples ¯ X (1)0 , . . .

Original: , ¯ X ( n )from q (e.g., a database of natural images), we train a neural network via score matching, see Song & Ermon (2019).
Revised: , ¯ X ( n )from q (e.g., a database of natural images), we train a neural network via score matching, see Song & Ermon (2019).

Original: Let h > 0 be the step size of the discretization; we assume that we have obtained a score estimate s kh of ∇ ln q kh for each time k = 0 , 1 , . . .
Revised: Let h > 0 be the step size of the discretization; we assume that we have obtained a score estimate s kh of ∇ ln q kh for each time k = 0 , 1 , . . .

Original: , N ,
Revised: , N ,

Original: where T =
Revised: where T =

Original: Nh .
Revised: Nh .

Original: In order to approximately implement the reverse SDE (4), we first replace the score function ∇ ln q T − t with the estimate s T − t .
Revised: In order to approximately implement the reverse SDE (4), we first replace the score function ∇ ln q T − t with the estimate s T − t .

Original: Then, for t ∈ [ kh, ( k + 1) h ] we freeze the value of this coefficient in the
Revised: Then, for t ∈ [ kh, ( k + 1) h ] we freeze the value of this coefficient in the

Original: SDE at time kh .
Revised: SDE at time kh .

Original: It yields the new SDE
Revised: It yields the new SDE

Original: [Equation]
Revised: [Equation]

Original: Since this is a linear SDE, it can be integrated in closed form; in particular, conditionally on X ← kh , the next iterate X ← ( k +1)
Revised: Since this is a linear SDE, it can be integrated in closed form; in particular, conditionally on X ← kh , the next iterate X ← ( k +1)

Original: h has an explicit Gaussian distribution.
Revised: h has an explicit Gaussian distribution.

Original: There is one final detail: although the reverse SDE (4) should be started at q T , we do not have access to q T directly.
Revised: There is one final detail: although the reverse SDE (4) should be started at q T , we do not have access to q T directly.

Original: Instead, taking advantage of the fact that q T ≈ γ d , we instead initialize the algorithm at X ← 0 ∼ γ d , i.e., from pure noise.
Revised: Instead, taking advantage of the fact that q T ≈ γ d , we instead initialize the algorithm at X ← 0 ∼ γ d , i.e., from pure noise.

Original: Let p t := law( X ← t ) denote the law of the algorithm at time t .
Revised: Let p t := law( X ← t ) denote the law of the algorithm at time t .

Original: The goal of this work is to bound
Revised: The goal of this work is to bound

Original: TV ( p T , q ) , taking into account three sources of error: (1) estimation of the score; (2) discretization of the SDE with step size h > 0 ; and (3) initialization of the algorithm at γ d rather than at q T .
Revised: TV ( p T , q ) , taking into account three sources of error: (1) estimation of the score; (2) discretization of the SDE with step size h > 0 ; and (3) initialization of the algorithm at γ d rather than at q T .

Original: B ACKGROUND ON THE CRITICALLY DAMPED L ANGEVIN DIFFUSION (CLD)
Revised: B ACKGROUND ON THE CRITICALLY DAMPED L ANGEVIN DIFFUSION (CLD)

Original: The critically damped Langevin diffusion (CLD) is based on the forward process
Revised: The critically damped Langevin diffusion (CLD) is based on the forward process

Original: [Equation]
Revised: [Equation]

Original: Compared to the OU process (1), this is now a coupled system of SDEs, where we have introduced a new variable ¯ V representing the velocity process.
Revised: Compared to the OU process (1), this is now a coupled system of SDEs, where we have introduced a new variable ¯ V representing the velocity process.

Original: The stationary distribution of the process is γ 2 d , the standard Gaussian measure on phase space R d × R d , and we initialize at ¯ X 0 ∼ q and ¯ V 0 ∼ γ d .
Revised: The stationary distribution of the process is γ 2 d , the standard Gaussian measure on phase space R d × R d , and we initialize at ¯ X 0 ∼ q and ¯ V 0 ∼ γ d .

Original: More generally, the CLD (6) is an instance of what is referred to as the kinetic Langevin or the underdamped Langevin process in the sampling literature.
Revised: More generally, the CLD (6) is an instance of what is referred to as the kinetic Langevin or the underdamped Langevin process in the sampling literature.

Original: In the context of  log-concave sampling, the smoother paths of ¯ X lead to smaller discretization error, thereby furnishing an algorithm with (cid:101) O ( √ d/ε ) gradient complexity (as opposed to sampling based on the overdamped Langevin process, which has complexity (cid:101) O
Revised: In the context of strongly log-concave sampling, the smoother paths of ¯ X lead to smaller discretization error, thereby furnishing an algorithm with (cid:101) O ( √ d/ε ) gradient complexity (as opposed to sampling based on the overdamped Langevin process, which has complexity (cid:101) O

Original: ( d/ε 2 ) ), see Cheng et al.
Revised: ( d/ε 2 ) ), see Cheng et al.

Original: ; Shen & Lee (2019); Dalalyan & RiouDurand (2020); Ma et al. 
Revised: ; Shen & Lee (2019); Dalalyan & Riou-Durand (2020); Ma et al. (2021).

Original: The recent paper (Dockhorn et al., 2022) proposed to use the CLD as the basis for an SGM and they empirically observed improvements over DDPM.
Revised: The recent paper Dockhorn et al. (2022) proposed to use the CLD as the basis for an SGM and they empirically observed improvements over DDPM.

Original: Applying (3), the corresponding reverse process is
Revised: Applying (3), the corresponding reverse process is

Original: [Equation]
Revised: [Equation]

Original: where q t := law( ¯ X t , ¯ V t ) is the law of the forward process at time t .
Revised: where q t := law( ¯ X t , ¯ V t ) is the law of the forward process at time t .

Original: Note that the gradient in the score function is only taken w.r.t.
Revised: Note that the gradient in the score function is only taken w.r.t.

Original: the velocity coordinate.
Revised: the velocity coordinate.

Original: Upon replacing the score function with an estimate s , we arrive at the algorithm
Revised: Upon replacing the score function with an estimate s , we arrive at the algorithm

Original: [Equation]
Revised: [Equation]

Original: X ← t + 2 V ← t + 4 s T − kh
Revised: X ← t + 2 V ← t + 4 s T − kh

Original: ( X ← kh , V ← kh ) (cid:1)
Revised: ( X ← kh , V ← kh ) (cid:1)

Original: d t + 2 d B t , for t ∈
Revised: d t + 2 d B t , for t ∈

Original: [ kh, ( k + 1) h ] .
Revised: [ kh, ( k + 1) h ] .

Original: We provide further background on the CLD in Section C.1.
Revised: We provide further background on the CLD in Section C.1.

Original:  R ESULTS
Revised:  R ESULTS

Original: We now formally state our assumptions and our main results.
Revised: We now formally state our assumptions and our main results.

Original: 3. R ESULTS FOR DDPM
Revised: 3. R ESULTS FOR DDPM

Original: For DDPM, we make the following mild assumptions on the data distribution q .
Revised: For DDPM, we make the following mild assumptions on the data distribution q .

Original: Assumption 1 (Lipschitz score) .
Revised: Assumption 1 (Lipschitz score) .

Original: For all t ≥ 0 , the score ∇ ln q t is L -Lipschitz.
Revised: For all t ≥ 0 , the score ∇ ln q t is L -Lipschitz.

Original: Assumption 2 (second moment bound) .
Revised: Assumption 2 (second moment bound) .

Original: For some η > 0 , E q [ ∥·∥ 2+ η ] is finite.
Revised: For some η > 0 , E q [ ∥·∥ 2+ η ] is finite.

Original: We also write m 22 :=
Revised: We also write m 22 :=

Original: E q [ ∥·∥ 2 ] for the second moment of q .
Revised: E q [ ∥·∥ 2 ] for the second moment of q .

Original: For technical reasons, we need to assume that q has a finite moment of order slightly but strictly bigger than 2 , but our quantitative bounds will only depend on the second moment m 22 .
Revised: For technical reasons, we need to assume that q has a finite moment of order slightly but strictly bigger than 2 , but our quantitative bounds will only depend on the second moment m 22 .

Original: Assumption 1 is standard and has been used in the prior works (Block et al., 2020; Lee et al., 2022a).
Revised: Assumption 1 is standard and has been used in the prior works Block et al. ; Lee et al. (2022a).

Original: However, unlike Lee et al. (2022a); Liu et al.
Revised: However, unlike Lee et al. (2022a); Liu et al.

Original: , we do not assume Lipschitzness of the score estimate.
Revised: , we do not assume Lipschitzness of the score estimate.

Original: Moreover, unlike Block et al.
Revised: Moreover, unlike Block et al.

Original: ; De Bortoli et al.
Revised: ; De Bortoli et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2022), we do not assume any convexity or dissipativity assumptions on the potential U , and unlike Lee et al.
Revised: (2022), we do not assume any convexity or dissipativity assumptions on the potential U , and unlike Lee et al.

Original: (2022a)
Revised: (2022a)

Original: we do not assume that q satisfies a log-Sobolev inequality.
Revised: we do not assume that q satisfies a log-Sobolev inequality.

Original: Hence, our assumptions cover a wide range of highly non-log-concave data distributions.
Revised: Hence, our assumptions cover a wide range of highly non-log-concave data distributions.

Original: Our proof technique is fairly robust and even
Revised: Our proof technique is fairly robust and even

Original: Assumption 1 could be relaxed (as well as other extensions, such as considering the time-changed forward process (2)), although we focus on the simplest setting in order to better illustrate the conceptual significance of our results.
Revised: Assumption 1 could be relaxed (as well as other extensions, such as considering the time-changed forward process (2)), although we focus on the simplest setting in order to better illustrate the conceptual significance of our results.

Original: We also assume a bound on the score estimation error.
Revised: We also assume a bound on the score estimation error.

Original: Assumption 3 (score estimation error) .
Revised: Assumption 3 (score estimation error) .

Original: For allk
Revised: For all

Original: [Equation]
Revised: [Equation]

Original: This is the same assumption as in Lee et al.
Revised: This is the same assumption as in Lee et al.

Original: (2022a), and as discussed in Section 2.1, it is a natural and realistic assumption in light of the derivation of the score matching objective.
Revised: (2022a), and as discussed in Section 2.1, it is a natural and realistic assumption in light of the derivation of the score matching objective.

Original: Our main result for DDPM is the following theorem.
Revised: Our main result for DDPM is the following theorem.

Original: Theorem 2 (DDPM, see Section B of supplement) .
Revised: Theorem 2 (DDPM, see Section B of supplement) .

Original: Suppose that Assumptions 1, 2, and 3 hold.
Revised: Suppose that Assumptions 1, 2, and 3 hold.

Original: Let p T be the output of the DDPM algorithm (Section 2.1) at time T , and suppose that the step size h :
Revised: Let p T be the output of the DDPM algorithm (Section 2.1) at time T , and suppose that the step size h :

Original: = T/N satisfies h ≲ 1 /L , where L ≥ 1 .
Revised: = T/N satisfies h ≲ 1 /L , where L ≥ 1 .

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: To interpret this result, suppose that, e.g.,D KL ( q ∥ γ d ) ≤ poly( d ) and m 2 ≤ d .  Choosing T ≍ log(D KL ( q ∥ γ ) /ε ) and h ≍ ε 2 L 2 d , and hiding logarithmic factors,
Revised: To interpret this result, suppose that, e.g., KL ( q ∥ γ d ) ≤ poly( d ) and m 2 ≤ d . 2 Choosing T ≍ log( KL ( q ∥ γ ) /ε ) and h ≍ ε 2 L 2 d , and hiding logarithmic factors,

Original: [Equation]
Revised: [Equation]

Original: In particular, in order to have TV ( p T , q ) ≤ ε , it suffices to have score error ε score ≤ (cid:101) O(cid:16)
Revised: In particular, in order to have TV ( p T , q ) ≤ ε , it suffices to have score error ε score ≤ (cid:101) O

Original: [Equation]
Revised: ( ε ) .

Original: We remark that the iteration complexity of N = (cid:101)
Revised: We remark that the iteration complexity of N = (cid:101)

Original: Θ( L 2 d ε 2 ) matches state-of-the-art complexity bounds for the Langevin Monte Carlo (LMC) algorithm for sampling under a log-Sobolev inequality (LSI), see Vempala & Wibisono (2019);
Revised: Θ( L 2 d ε 2 ) matches state-of-the-art complexity bounds for the Langevin Monte Carlo (LMC) algorithm for sampling under a log-Sobolev inequality (LSI), see Vempala & Wibisono (2019);

Original: Chewi et al.
Revised: Chewi et al.

Original: (2021a).
Revised: (2021a).

Original: This provides some evidence that our discretization bounds are of the correct order, at least with respect to the dimension and accuracy parameters, and without higher-order smoothness assumptions.
Revised: This provides some evidence that our discretization bounds are of the correct order, at least with respect to the dimension and accuracy parameters, and without higher-order smoothness assumptions.

Original: 3. C ONSEQUENCES FOR ARBITRARY DATA DISTRIBUTIONS WITH BOUNDED SUPPORT
Revised: 3. C ONSEQUENCES FOR ARBITRARY DATA DISTRIBUTIONS WITH BOUNDED SUPPORT

Original: We now elaborate upon the implications of our results under the sole assumption that the data distribution q is compactly supported, supp q ⊆ B (0 , R ) .
Revised: We now elaborate upon the implications of our results under the sole assumption that the data distribution q is compactly supported, supp q ⊆ B (0 , R ) .

Original: In particular, we do not assume that q has a smooth density w.r.t.
Revised: In particular, we do not assume that q has a smooth density w.r.t.

Original: Lebesgue measure, which allows for studying the case when q is supported on a lower-dimensional submanifold of R d as in the manifold hypothesis .
Revised: Lebesgue measure, which allows for studying the case when q is supported on a lower-dimensional submanifold of R d as in the manifold hypothesis .

Original: This setting was investigated recently in De Bortoli (2022).
Revised: This setting was investigated recently in De Bortoli (2022).

Original: For this setting, our results do not apply directly because the score function of q is not well-defined and hence Assumption 1 fails to hold.
Revised: For this setting, our results do not apply directly because the score function of q is not well-defined and hence Assumption 1 fails to hold.

Original: Also, the bound in Theorem 2 has a term involvingD KL ( q ∥ γ d ) which is infinite if q is not absolutely continuous w.r.t.
Revised: Also, the bound in Theorem 2 has a term involving KL ( q ∥ γ d ) which is infinite if q is not absolutely continuous w.r.t.

Original: γ d .
Revised: γ d .

Original: As pointed out by De Bortoli (2022), in general we cannot obtain non-trivial guarantees for TV ( p T , q ) , because p T has full support and therefore TV ( p T , q )
Revised: As pointed out by De Bortoli (2022), in general we cannot obtain non-trivial guarantees for TV ( p T , q ) , because p T has full support and therefore TV ( p T , q )

Original: = 1 under the manifold hypothesis.
Revised: = 1 under the manifold hypothesis.

Original: Nevertheless, we show that we can apply our results using an early stopping technique.
Revised: Nevertheless, we show that we can apply our results using an early stopping technique.

Original: Namely, consider q t the law of the OU process at a time t > 0 , initialized at q . We show:
Revised: Namely, using the following lemma, we obtain a sequence of corollaries.

Original: Lemma 3 (Lemma 18 in supplement) .
Revised: Lemma 3 (Lemma 21 in supplement) .

Original: Suppose that supp q ⊆ B (0 , R ) where R ≥ 1 , and let q t denote the law of the OU process at time t , started at q .
Revised: Suppose that supp q ⊆ B (0 , R ) where R ≥ 1 , and let q t denote the law of the OU process at time t , started at q .

Original: Let ε
Revised: Let ε

Original: W 2 > 0 be such that ε W 2 ≪ √ d and set t ≍ ε 2 W 2 / ( √ d
Revised: W 2 > 0 be such that ε W 2 ≪ √ d and set t ≍ ε 2 W 2 / ( √ d

Original: ( R ∨ √ d )) .
Revised: ( R ∨ √ d )) .

Original: Then, (1) W 2 ( q t , q ) ≤ ε W 2 , (2) q t satisfiesD KL ( q t ∥ γ d ) ≲ √ d ( R ∨√ d ) 3 ε 2 W, and (3) for every t ′ ≥ t , q t ′ satisfies Assumption 1 with L ≲ dR 2 ( R ∨√ d ) [-
-] ε 4 W.
Revised: Then, (1) W 2 ( q t , q ) ≤ ε W 2 , (2) q t satisfies KL ( q t ∥ γ d ) ≲ √ d ( R ∨√ d ) 3 ε 2 W, and (3) for every t ′ ≥ t , q t ′ satisfies Assumption 1 with L ≲ dR 2 ( R ∨√ d ) [-
-]2 ε 4 W.

Original: By substituting q t for this choice of t in place of q in Theorem 2, we obtain Corollary 4 below.
Revised: By substituting q t for this choice of t in place of q in Theorem 2, we obtain Corollary 4 below.

Original: We remark that taking q t as the new target corresponds to stopping the algorithm early: instead of running the algorithm backward for a time T , we run the algorithm backward for a time T − t
Revised: We remark that taking q t as the new target corresponds to stopping the algorithm early: instead of running the algorithm backward for a time T , we run the algorithm backward for a time T − t

Original: (note that T − t should be a multiple of the step size h ).
Revised: (note that T − t should be a multiple of the step size h ).

Original: Corollary 4 (compactly supported data) .
Revised: Corollary 4 (compactly supported data) .

Original: Suppose that q is supported on the ball of radius R ≥ 1 .
Revised: Suppose that q is supported on the ball of radius R ≥ 1 .

Original: Let t ≍ ε 2 W 2 / ( √ d ( R ∨ √ d )) .
Revised: Let t ≍ ε 2 W 2 / ( √ d ( R ∨ √ d )) .

Original: Then, the output p T − t of DDPM is ε TV -close in TV to the distribution q t , which is ε W 2 -close in W 2 to q , provided that the step size h is chosen appropriately according to
Revised: Then, the output p T − t of DDPM is ε TV -close in TV to the distribution q t , which is ε W 2 -close in W 2 to q , provided that the step size h is chosen appropriately according to

Original: Theorem 2 and N = (cid:101) Θ (cid:16) d 3 R 4 ( R ∨√ 
Revised: Theorem 2 and N = (cid:101) Θ (cid:16) d 3 R 4 ( R ∨√ d ) 4 ε 2TV

Original: d ) 4 ε 2TV ε 8 W (cid:17) and ε score ≤ (cid:101) O (cid:16) ε5 / 2 TV [Table] 4 Wd 3 / 2 R 2 ( R ∨√ d )(cid:17) .
Revised: ε 8 W (cid:17) and ε score ≤ (cid:101) O ( ε TV ) .

Original: Note that the dependencies in this corollary are polynomial in all of the relevant problem parameters, which vastly improves upon the exponential dependencies of De Bortoli (2022).

Original: Remark. The statement of Corollary 4 can be simplified by observing that both  TV and W 1 distances are upper bounds for the bounded Lipschitz metric d BL ( µ, ν ) :
Revised: Observing that both the TV and W 1 metrics are upper bounds for the bounded Lipschitz metric d BL ( µ, ν ) :

Original: = sup { (cid:82) f d µ − (cid:82) f d ν (cid:12)(cid:12) f : R d → [ − 1 , 1] is 1 -Lipschitz } .
Revised: = sup { (cid:82) f d µ − (cid:82) f d ν (cid:12)(cid:12) f : R d → [ − 1 , 1] is 1 -Lipschitz } , we immediately obtain the following corollary.

Original: Hence, Corollary 4 can be phrased as a convergence guarantee in the bounded
Revised: Corollary 5 (compactly supported data, BL metric) . Suppose that q is supported on the ball of radius R ≥ 1 .

Original: Lipschitz metric, which is known tometrize weak convergence (see Dudley (2002, Theorem 11.3.3)).
Revised: Let t ≍ ε 2 W 2 / ( √ d ( R ∨ √ d )) . Then, the output p T − t of the DDPM algorithm satisfies d BL ( p T − t , q ) ≤ ε , provided that the step size h is chosen appropriately according to Theorem 2 and N = (cid:101) Θ( d 3 R 4 ( R ∨ √ d ) 4 /ε 10 ) and ε score ≤ (cid:101) O ( ε TV ) .

Original: 3. R ESULTS FOR CLD
Revised: Finally, if the output p T − t of DDPM at time T − t is projected onto B (0 , R 0 ) for an appropriate choice of R 0 , then we can also translate our guarantees to the standard W 2 metric, which we state as the following corollary. Corollary 6 (compactly supported data, W 2 metric; see Section B.5 in supplement) . Suppose that q is supported on the ball of radius R ≥ 1 . Let t ≍ ε 2 W 2 / ( √ d ( R ∨ √ d )) , and let p T − t,R 0 denote the output of DDPM at time T − t projected onto B (0 , R 0 ) for R 0 = (cid:101)

Revised: Θ( R ) .

Revised: Then, it holds that

Revised: W 2 ( p T − t,R 0 , q ) ≤ ε , provided that the step size h is chosen appropriately according to Theorem 2,

Revised: N = (cid:101) Θ( d

Revised: R

Revised: R ∨ √ d ) 4 /ε

Revised: and ε score ≤ (cid:101)

Revised: O

Revised: ( ε

Revised: TV ) .

Revised: Note that the dependencies in the three corollaries above are polynomial in all of the relevant problem parameters.

Revised: In particular, since the last corollary holds in the W 2 metric, it is directly comparable to De Bortoli (2022) and vastly improves upon the exponential dependencies therein.

Revised: 3. R ESULTS FOR CLD

Original: In order to state our results for score-based generative modeling based on the CLD, we must first modify Assumptions 1 and 3 accordingly.
Revised: In order to state our results for score-based generative modeling based on the CLD, we must first modify Assumptions 1 and 3 accordingly.

Original: Assumption 4.
Revised: Assumption 4.

Original: For all t ≥ 0 , the score ∇ v ln q t is L -Lipschitz.
Revised: For all t ≥ 0 , the score ∇ v ln q t is L -Lipschitz.

Original: Assumption 5.
Revised: Assumption 5.

Original: For all k = 1 , . . .
Revised: For all [Equation]

Original: , N , E q kh [ ∥ s kh − ∇ v ln q kh ∥ 2 ] ≤ ε 2score .
Revised: [Equation] N [Equation]

Original: If we ignore the dependence on L and assume that the score estimate is sufficiently accurate, then the iteration complexity guarantee of Theorem 2 is N = (cid:101)
Revised: If we ignore the dependence on L and assume that the score estimate is sufficiently accurate, then the iteration complexity guarantee of Theorem 2 is N = (cid:101)

Original: Θ( d/ε 2 ) .
Revised: Θ( d/ε 2 ) .

Original: On the other hand, recall from
Revised: On the other hand, recall from

Original: Section 2.2 that based on intuition from the literature on log-concave sampling and from empirical findings in Dockhorn et al.
Revised: Section 2.2 that based on intuition from the literature on log-concave sampling and from empirical findings in Dockhorn et al.

Original: (2022), we might expect that SGMs based on the CLD have a smaller iteration complexity than DDPM.
Revised: (2022), we might expect that SGMs based on the CLD have a smaller iteration complexity than DDPM.

Original: We prove the following theorem.
Revised: We prove the following theorem.

Original: Theorem 5 (CLD, see Section C of supplement) .
Revised: Theorem 7 (CLD, see Section C of supplement) .

Original: Suppose that Assumptions 2, 4, and 5 hold.
Revised: Suppose that Assumptions 2, 4, and 5 hold.

Original: Let p T be the output of the SGM algorithm based on the CLD (Section 2.2) at time T , and suppose the step size h :
Revised: Let p T be the output of the SGM algorithm based on the CLD (Section 2.2) at time T , and suppose the step size h :

Original: = T/N satisfies h ≲ 1 /L , where L ≥ 1 .
Revised: = T/N satisfies h ≲ 1 /L , where L ≥ 1 .

Original: Then, there is a universal constant c > 0 such that
Revised: Then, there is a universal constant c > 0 such that

Original: TV ( p T , q ⊗ γ d ) is bounded, up to a constant factor, by
Revised: TV ( p T , q ⊗ γ d ) is bounded, up to a constant factor, by

Original: [Equation]
Revised: [Equation]

Original: where FI ( q ∥ γ d ) is the relative Fisher information FI ( q ∥ γ d ) :
Revised: where FI ( q ∥ γ d ) is the relative Fisher information FI ( q ∥ γ d ) :

Original: =
Revised: =

Original: E q
Revised: E q

Original: [ ∥∇ ln( q/γ d ) ∥ 2 ] .
Revised: [ ∥∇ ln( q/γ d ) ∥ 2 ] .

Original: Note that the result of Theorem 5 is in fact no better than our guarantee for DDPM in Theorem 2.
Revised: Note that the result of Theorem 7 is in fact no better than our guarantee for DDPM in Theorem 2.

Original: Although it is possible that this is an artefact of our analysis, we believe that it is in fact fundamental.
Revised: Although it is possible that this is an artefact of our analysis, we believe that it is in fact fundamental.

Original: As we discuss in Remark C.2, from the form of the reverse process (7), the SGM based on CLD lacks a certain property (that the discretization error should only depend on the size of the increment of the X process, not the increments of both the X and V processes) which is crucial for the improved dimension dependence of the CLD over the Langevin diffusion in log-concave sampling.
Revised: As we discuss in Remark C.2, from the form of the reverse process (7), the SGM based on CLD lacks a certain property (that the discretization error should only depend on the size of the increment of the X process, not the increments of both the X and V processes) which is crucial for the improved dimension dependence of the CLD over the Langevin diffusion in log-concave sampling.

Original: Hence, in general, we conjecture that under our assumptions, SGMs based on the CLD do not achieve a better dimension dependence than DDPM.
Revised: Hence, in general, we conjecture that under our assumptions, SGMs based on the CLD do not achieve a better dimension dependence than DDPM.

Original: We provide evidence for our conjecture via a lower bound.
Revised: We provide evidence for our conjecture via a lower bound.

Original: In our proofs of Theorems 2 and 5, we rely on bounding the KL divergence between certain measures on the path space C ([0 , T ]; R d ) via
Revised: In our proofs of Theorems 2 and 7, we rely on bounding the KL divergence between certain measures on the path space C ([0 , T ]; R d ) via

Original: Girsanov’s theorem.
Revised: Girsanov’s theorem.

Original: The following result lower bounds this KL divergence, even for the setting in which the score estimate is perfect ( ε score = 0 ) and the data distribution q is the standard Gaussian.
Revised: The following result lower bounds this KL divergence, even for the setting in which the score estimate is perfect ( ε score = 0 ) and the data distribution q is the standard Gaussian.

Original: Theorem 6 (See Section C.5 of supplement) .
Revised: Theorem 8 (Section C.5 of supplement) .

Original: Let p T be the output of the SGM algorithm based on the CLD (Section 2.2) at time T , where the data distribution q is the standard Gaussian γ d , and the score estimate is exact ( ε score = 0 ).
Revised: Let p T be the output of the SGM algorithm based on the CLD (Section 2.2) at time T , where the data distribution q is the standard Gaussian γ d , and the score estimate is exact ( ε score = 0 ).

Original: Suppose that the step size h satisfies h ≤ 110 .
Revised: Suppose that the step size h satisfies h ≲ 1 / ( T ∨ 1) .

Original: Then, for the path measures P T and Q ← T of the algorithm and the continuous-time process (7) respectively (see
Revised: Then, for the path measures P T and Q ← T of the algorithm and the continuous-time process (7) respectively (see

Original: Section C for details), it holds thatD KL ( Q ← T ∥ P T ) ≥ dhT .
Revised: Section C for details), it holds that KL ( Q ← T ∥ P T ) ≥ dhT .

Original: Theorem 6 shows that in order to make the KL divergence between the path measures small, we must take h ≲ 1 /d , which leads to an iteration complexity that scales linearly in the dimension d .
Revised: Theorem 8 shows that in order to make the KL divergence between the path measures small, we must take h ≲ 1 /d , which leads to an iteration complexity that scales linearly in the dimension d .

Original: Theorem 6 is not a proof that SGMs based on the CLD cannot achieve better than linear dimension dependence, as it is possible that the output p T of the SGM is close to q ⊗ γ d even if the path measures are not close, but it rules out the possibility of obtaining a better dimension dependence via our Girsanov proof technique.
Revised: Theorem 8 is not a proof that SGMs based on the CLD cannot achieve better than linear dimension dependence, as it is possible that the output p T of the SGM is close to q ⊗ γ d even if the path measures are not close, but it rules out the possibility of obtaining a better dimension dependence via our Girsanov proof technique.

Original: We believe that it provides compelling evidence for our conjecture, i.e., that under our assumptions, the CLD does not improve the complexity of SGMs over DDPM.
Revised: We believe that it provides compelling evidence for our conjecture, i.e., that under our assumptions, the CLD does not improve the complexity of SGMs over DDPM.

Original: We remark that in this section, we have only considered the error arising from discretization of the
Revised: We remark that in this section, we have only considered the error arising from discretization of the

Original: SDE.
Revised: SDE.

Original: It is possible that the score function ∇ v ln q t for the SGM with the CLD is easier to estimate than the score function for DDPM, providing a statistical benefit of using the CLD.
Revised: It is possible that the score function ∇ v ln q t for the SGM with the CLD is easier to estimate than the score function for DDPM, providing a statistical benefit of using the CLD.

Revised: Indeed, under the manifold hypothesis, the score ∇ ln q t for DDPM blows up at t = 0 , but the score ∇ v ln q t for CLD is well-defined at t = 0 , and hence may lead to improvements over DDPM.

Original: We do not investigate this question here and leave it as future work.
Revised: We do not investigate this question here and leave it as future work.

Original:  T ECHNICAL OVERVIEW
Revised:  T ECHNICAL OVERVIEW

Original: We now give a detailed technical overview for the proof for DDPM (Theorem 2).
Revised: We now give a detailed technical overview for the proof for DDPM (Theorem 2).

Original: The proof for CLD (Theorem 5) follows along similar lines.
Revised: The proof for CLD (Theorem 7) follows along similar lines.

Original: Recall that we must deal with three sources of error: (1) the estimation of the score function; (2) the discretization of the SDE; and (3) the initialization of the reverse process at γ d rather than at q T .
Revised: Recall that we must deal with three sources of error: (1) the estimation of the score function; (2) the discretization of the SDE; and (3) the initialization of the reverse process at γ d rather than at q T .

Original: First, we ignore the errors (1) and (2), and focus on the error (3).
Revised: First, we ignore the errors (1) and (2), and focus on the error (3).

Original: Hence, we consider the continuoustime reverse SDE (4), initialized from γ d (resp.
Revised: Hence, we consider the continuous-time reverse SDE (4), initialized from γ d (resp.

Original: q T ) and denote by ˜ p t (resp.
Revised: q T ) and denote by ˜ p t (resp.

Original: q T − t )
Revised: q T − t )

Original: its marginal distributions.
Revised: its marginal distributions.

Original: Note that ˜ p 

Revised: Note that ˜ p

Original: = γ d and that q 0 = q , the data distribution.
Revised: = γ d and that q 0 = q , the data distribution.

Original: First, using the exponential contraction of the KL divergence along the (forward) OU process, we haveD KL ( q T ∥ γ d ) ≤ exp( − 2 T )D KL ( q ∥ γ d ) .
Revised: First, using the exponential contraction of the KL divergence along the (forward) OU process, we have KL ( q T ∥ γ d ) ≤ exp( − 2 T ) KL ( q ∥ γ d ) .

Original: Then, using the data processing inequality along the backward process, we have TV (˜ p T , q ) ≤ TV ( γ d , q T ) .
Revised: Then, using the data processing inequality along the backward process, we have TV (˜ p T , q ) ≤ TV ( γ d , q T ) .

Original: Therefore, using Pinsker inequality, we get
Revised: Therefore, using Pinsker inequality, we get

Original: TV

Original: [Equation]
Revised: [Equation]

Original: , (8) i.e., the output ˜ p T converges to the data distribution q exponentially fast as T → ∞ .
Revised: i.e., the output ˜ p T converges to the data distribution q exponentially fast as T → ∞ .

Original: Next, we consider the score estimation error (1) and the discretization error (2).
Revised: Next, we consider the score estimation error (1) and the discretization error (2).

Original: Similarly to (Lee et al., 2022a)

Original: we first assume that the score estimate is L ∞ -accurate:

Original: [Equation]

Original: Let p ∞ t denote the law of the algorithm at time t with L ∞ -accurate score estimate.

Original:  Girsanov’s theorem lets us bound the discretization error incurred by p ∞ t by
Revised: Using Girsanov’s theorem, these errors can be bounded by

Original: [Equation]
Revised: [Equation]

Original: (seeCorollary 9 in the supplement), where the expectation is with respect to the law of the reverse process. To control the integrand inthis expression we can decompose [Equation] The first term on the right-hand side is bounded by the assumption that p ∞ is given by an L ∞ - accurate score estimate.
Revised: (see the inequality (15) in the supplement).

Original: The second term corresponds to the change inthe score function along the forward process, which we can bound using tools from Lee et al.(2022a) (see also Lemma 15 for a generalization thereof that we prove for the purposes of analyzing CLD). The final term can be bounded using the fact that the score function is Lipschitz by assumption, together with a bound on how much the forward process moves over a small time period. Putting everything together, we can obtain the bound on the discretization error given by Theorem 10 in the supplement.
Revised: Unlike other proof techniques, such as the interpolation method in Lee et al.

Original: Finally, we must replace the assumption (9) of an L ∞ -accurate score estimate with Assumption 3, for which we take inspiration from (Lee et al., 2022a). For k = 0 , 1 , . . . , ( N − 1) h , define the bad set B
Revised: (2022a), the error term (8) in Girsanov’s theorem involves an expectation under the law of the true reverse process, instead of the law of the algorithm (see Lee et al.

Original: [Equation]

Original: score [Equation] [Equation] If the algorithm never enters the bad sets, i.e., X ← kh / ∈ B T − kh for k = 0 , 1 , . .
Revised: (2022a)).

Original: , N − 1 , then it is as if thealgorithm is using an L ∞ -accurate score estimate, and we can apply the Girsanov discretization analysis. Therefore, if we can control the error probability [Equation]
Revised: This difference allows us to bound the score estimation error using Assumption 3 directly, which allows a simpler proof that works under milder assumptions on the data distribution.

Original: ∈ B T − kh for some k = 0 , 1 , . . . , N − 1) ≤ . . , then by a coupling argument, we can bound the overall total variation error TV ( p T , q ) . However, by Chebyshev’s, we have q kh ( B kh ) ≤ ε 2score /ε
Revised: However, the use of

Original: 2score , ∞

Original: =: δ , and by a union bound, P [Equation]
Revised: Girsanov’s theorem typically requires a technical condition known as Novikov’s condition , which fails to hold under under our minimal assumptions.

Original: T

Original: [Equation] kh for some k = 0 , 1 , . . . [Equation] In brief, the difficulty now is that our assumption allows us to control the probability of the bad event for the reverse process ¯ X ← , but we need to control the probability under the algorithm X ← .
Revised: To circumvent this issue, we use an approximation argument relying on abstract results on the convergence of stochastic processes.

Original: To control the difference of these probabilities, we notice that Girsanov allows to control the divergence between the joint distributions of the two processes. So we first control the difference between the two probabilities by the TV distance between the joint distributions of the process ¯ X ← and the process X ← initialized from q T . We then pass to KL divergence using Pinsker’s inequality and finally we use the bound from our discretization analysis with L ∞ -accurate score estimate, which reads [Equation]
Revised: A recent concurrent and independent work Liu et al.

Original: Putting Eqs.

Original: (8), (10), and (11) together, we have [Equation] where the three terms correspond to the components of the bound in Theorem 2: the convergence of the forward process, the discretization error, and the probability of failure. For the third term, the parameter δ is balanced in the final bound to optimize a trade-off (a smaller choice of δ leads to a larger value for ε score , ∞ , which in turn increases the error ε Girsanov ), ultimately leading to a bound in terms of the score estimation error ε score .
Revised: (2022) also uses Girsanov’s theorem, but assumes that Novikov’s condition holds at the outset.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we provided the first convergence guarantees for SGMs which hold under realistic assumptions (namely, L 2 -accurate score estimation and arbitrarily non-log-concave data distributions) and which scale polynomially in the problem parameters.
Revised: In this work, we provided the first convergence guarantees for SGMs which hold under realistic assumptions (namely, L 2 -accurate score estimation and arbitrarily non-log-concave data distributions) and which scale polynomially in the problem parameters.

Original: Our results take a step towards explaining the remarkable empirical success of SGMs, at least assuming the score is learned with small L 2 error.
Revised: Our results take a step towards explaining the remarkable empirical success of SGMs, at least assuming the score is learned with small L 2 error.

Original: The main limitation of this work is that we did not address the question of when the score function can be learned well.
Revised: The main limitation of this work is that we did not address the question of when the score function can be learned well.

Original: In general, studying the non-convex training dynamics of learning the score function via neural networks is challenging, but we believe that the resolution of this problem, even for simple learning tasks, would shed considerable light on SGMs.
Revised: In general, studying the non-convex training dynamics of learning the score function via neural networks is challenging, but we believe that the resolution of this problem, even for simple learning tasks, would shed considerable light on SGMs.

Original: Together with the results in this paper, it would yield the first end-to-end guarantees for SGMs.
Revised: Together with the results in this paper, it would yield the first end-to-end guarantees for SGMs.

Original: In light of the interpretation of our result as a reduction of the task of sampling to the task of score function estimation, we also ask whether there are interesting situations where it is easier to learn the score function (not necessarily via a neural network) than to (directly) sample.
Revised: In light of the interpretation of our result as a reduction of the task of sampling to the task of score function estimation, we also ask whether there are interesting situations where it is easier to learn the score function (not necessarily via a neural network) than to (directly) sample.

Original: A P RELIMINARIES
Revised: A P RELIMINARIES

Original: In this section, we review the notion of score matching, state and prove the elementary lemma that leads to (12) above, and provide a list of notation for the proofs.
Revised: In this section, we review the notion of score matching and provide a list of notation for the proofs.

Original: A. P RIMER ON SCORE MATCHING
Revised: A. P RIMER ON SCORE MATCHING

Original: In order to estimate the score function ∇ ln q t , consider minimizing the L 2 ( q t ) loss over a function class F ,
Revised: In order to estimate the score function ∇ ln q t , consider minimizing the L 2 ( q t ) loss over a function class F ,

Original: [Equation]
Revised: [Equation]

Original: where F could be, e.g., a class of neural networks.
Revised: where F could be, e.g., a class of neural networks.

Original: The idea of score matching, which goes back to Hyv¨arinen (2005); Vincent (2011), is that after applying integration by parts for the Gaussian measure, the problem (13) is equivalent to the following problem:
Revised: The idea of score matching, which goes back to Hyv¨arinen (2005); Vincent (2011), is that after applying integration by parts for the Gaussian measure, the problem (9) is equivalent to the following problem:

Original: [Equation]
Revised: [Equation]

Original: where Z t ∼ normal (0 , I d ) is independent of ¯ X 

Revised: where Z t ∼ normal (0 , I d ) is independent of ¯ X

Original: and ¯ X t = exp( − t )
Revised: and ¯ X t = exp( − t )

Original: ¯ X 

Revised: ¯ X

Original: + (cid:112) 1 − exp( − 2 t )
Revised: + (cid:112) 1 − exp( − 2 t )

Original: Z t , in the sense that (13) and (14) share the same minimizers.
Revised: Z t , in the sense that (9) and (10) share the same minimizers.

Original: We give a self-contained derivation in Appendix D for the sake of completeness.
Revised: We give a self-contained derivation in Appendix D for the sake of completeness.

Original: Unlike (13), however, the objective in (14) can be replaced with an empirical version and estimated on the basis of samples ¯ X (1)0 , . .
Revised: Unlike (9), however, the objective in (10) can be replaced with an empirical version and estimated on the basis of samples ¯ X (1)0 , . .

Original: , ¯ X ( n )from q , leading to the finite-sample problem
Revised: , ¯ X ( n )from q , leading to the finite-sample problem

Original: [Equation]
Revised: [Equation]

Original: 2 , where ( Z ( i ) t ) i ∈ [ n ] are i.i.d.
Revised: where ( Z ( i ) t ) i ∈ [ n ] are i.i.d.

Original: standard Gaussians independent ofthe data ( ¯ X ( i ) 0 ) i ∈ [ n ] .
Revised: standard Gaussians independent of ( ¯ X ( i ) 0 ) i ∈ [ n ] .

Original: Moreover, if we parameterize the scorefunction as s t = −
Revised: Moreover, if we parameterize the score as s t = −

Original: [Equation]
Revised: [Equation]

Original: z t , then the empirical problem is equivalent to minimize
Revised: √

Original: [Equation]
Revised: [Equation]

Original: which has the illuminating interpretation of predicting the added noise Z ( i ) t from the noised data ¯ X ( i ) t .

Original: A. T RUNCATION LEMMA

Original: Lemma 7.

Original: Suppose that ( ¯ X ← kh ) k ∈ N , ( X ← kh ) k ∈ N , and ( X ← , ∞ kh ) k ∈ N are random processes defined on the same probability space.

Original: For each k ∈ N , let q ← kh , p kh , and p ∞ kh denote the laws of ¯ X ← kh , X ← kh , and

Original: X ← , ∞ kh respectively.

Original: Suppose that ( B kh ) k ∈ N are measurable subsets of R d and that the following hold.

Original: If X ← , ∞ kh / ∈ B T − kh for k = 0 , 1 , . .

Original: , N − 1 , then X ← kh =

Original: X ← , ∞ kh for all k = 0 , 1 , . .

Original: . , N . 2.

Original: It holds that TV (cid:0) law( X ← , ∞ kh : k

Original: = 0 , 1 , . .

Original: , N − 1) , law( ¯ X ← kh : k

Original: = 0 , 1 , . .

Original: , N − 1) (cid:1) ≤ ε .

Original: P [ ¯ X ← kh ∈ B T − kh ] ≤ δ for all k = 0 , 1 , . . .

Original: , N − 1 .

Original: Then, the following holds: TV ( p Nh , q ← Nh ) ≤ 2 ε + Nδ . Proof. By triangle inequality,
Revised: z t , then the empirical problem is equivalent to minimize

Original: [Equation]
Revised: [Equation]

Original: where we used the data processing inequality. Then, by the union bound [Equation] This proves the lemma.
Revised: which has the illuminating interpretation of predicting the added noise Z ( i ) t from the noised data ¯ X ( i ) t , i.e., denoising.

Original: A. N OTATION
Revised: N OTATION

Original: L ∞ -accurate score estimate.

Original: The bad set is defined to be

Original: [Equation]

Original: where ε score , ∞ > 0 is a parameter to be chosen later. We define the L ∞ -accurate score estimate to be [Equation] We note that ∥ s ∞ t − ∇ ln q t ∥ ≤ ε score , ∞ .
Revised: For a measurable mapping T : X → X and a measure µ on X , where X is a measurable space, the notation T # µ refers to the pushforward of µ by the mapping T , i.e., if X ∼ µ , then T ( X ) ∼ T # µ .

Original: Stochastic processes and their laws.
Revised: Stochastic processes and their laws.

Original: • The data distribution is q = q 0 .
Revised: • The data distribution is q = q 0 .

Original: • The forward process (1) is denoted ( ¯ X t ) t ∈ [0 ,T ] , and ¯ X t ∼ q t .
Revised: • The forward process (1) is denoted ( ¯ X t ) t ∈ [0 ,T ] , and ¯ X t ∼ q t .

Original: • The reverse process (4) is denoted ( ¯ X ← t ) t ∈ [0 ,T ] , where ¯ X ← t :=
Revised: • The reverse process (4) is denoted ( ¯ X ← t ) t ∈ [0 ,T ] , where ¯ X ← t :=

Original: ¯ X T − t ∼ q T − t .
Revised: ¯ X T − t ∼ q T − t .

Original: • The SGM algorithm (5) is denoted ( X ← t ) t ∈ [0 ,T ] , and X ← t ∼ p t .
Revised: • The SGM algorithm (5) is denoted ( X ← t ) t ∈ [0 ,T ] , and X ← t ∼ p t .

Original: Recall that we initialize at p 

Revised: Recall that we initialize at p

Original: = γ d , the standard Gaussian measure.
Revised: = γ d , the standard Gaussian measure.

Original: • The[-SGM algorithm with L ∞ -accurate score, denoted ( X ← , ∞ t ) t ∈ [0 ,T ] , is the same as (5) except that we use s ∞ T − kh defined in (15) instead of s T − kh for k = 0 , 1 , . . , N − 1 . We write X ← , ∞ t ∼ p ∞ t , and we initialize the process at p ∞ 
 = γ d . • The-] process ( X ←, ∞ ,q T t ) t ∈ [0 ,T ] is the same as ( X ←, ∞ t ) t ∈ [0 ,T ] , except that we initialize this process at q T rather than at γ d .
Revised: • The[-SGM algorithm with L ∞ -accurate score, denoted ( X ← , ∞ t ) t ∈ [0 ,T ] , is the same as (5) except that we use s ∞ T − kh defined in (15) instead of s T − kh for k = 0 , 1 , . . , N − 1 . We write X ← , ∞ t ∼ p ∞ t , and we initialize the process at p ∞ 
 = γ d . • The-] process ( X ← ,q T t ) t ∈ [0 ,T ] is the same as ( X ← t ) t ∈ [0 ,T ] , except that we initialize this process at q T rather than at γ d .

Original: We write X ←, ∞ ,q T t ∼ p ∞ ,q T t .
Revised: We write X ← ,q T t ∼ p q T t .

Original: Conventions for Girsanov’s theorem.
Revised: Conventions for Girsanov’s theorem.

Original: When we apply Girsanov’s theorem, it is convenient to instead think about a single stochastic process, which for ease of notation we denote simply via ( X t ) t ∈ [0 ,T ] , and we consider different measures over the path space C
Revised: When we apply Girsanov’s theorem, it is convenient to instead think about a single stochastic process, which for ease of notation we denote simply via ( X t ) t ∈ [0 ,T ] , and we consider different measures over the path space C

Original: ([0 , T ]; R d ) .
Revised: ([0 , T ]; R d ) .

Original: The two measures we consider over path space are:
Revised: The two measures we consider over path space are:

Original: • Q ← T , under which ( X t ) t ∈ [0 ,T ] has the law of the reverse process (4);and • P ∞ ,q T T , under which ( X t ) t ∈ [0 ,T ] has the law of the SGM algorithmwith L ∞ -accurate score initialized at q T (corresponding to the process ( X ←, ∞ ,q T t ) t ∈ [0 ,T ] defined above). 
Revised: • Q ← T , under which ( X t ) t ∈ [0 ,T ] has the law of the reverse process (4); • P q T T , under which ( X t ) t ∈ [0 ,T ] has the law of the SGM algorithm initialized at q T (corresponding to the process ( X ← ,q T t ) t ∈ [0 ,T ] defined above). We also use the following notion from stochastic calculus (Le Gall, 2016, Definition 4.6): • A local martingale ( L t ) t ∈ [0 ,T ] is a stochastic process s.t. there exists a sequence of nondecreasing stopping times T n → T s.t.

Revised: L n =

Revised: ( L t ∧ T n ) t ∈ [0 ,T ] is a martingale.

Original: Other parameters.
Revised: Other parameters.

Original: We recall that T > 0 denotes the total time for which we run the forward process; h > 0 is the step size of the discretization; L ≥ 1 is the Lipschitz constant of the score function; m 22 :=
Revised: We recall that T > 0 denotes the total time for which we run the forward process; h > 0 is the step size of the discretization; L ≥ 1 is the Lipschitz constant of the score function; m 22 :=

Original: E q [ ∥·∥ 2 ] is the second moment under the data distribution; and ε score is the L 2 score estimation error.
Revised: E q [ ∥·∥ 2 ] is the second moment under the data distribution; and ε score is the L 2 score estimation error.

Original: Notation for CLD.
Revised: Notation for CLD.

Original: The notational conventions for the CLD are similar; however, we must also consider a velocity variable V .
Revised: The notational conventions for the CLD are similar; however, we must also consider a velocity variable V .

Original: When discussing quantities which involve both position and velocity (e.g., the joint distribution q t of ( ¯ X t , ¯ V t ) ), we typically use boldface fonts.
Revised: When discussing quantities which involve both position and velocity (e.g., the joint distribution q t of ( ¯ X t , ¯ V t ) ), we typically use boldface fonts.

Original: B P ROOFS FOR DDPM B. P RELIMINARIES ON G IRSANOV ’ S THEOREM 
Revised: B P ROOFS FOR DDPM B. P RELIMINARIES ON G IRSANOV ’ S THEOREM AND A FIRST ATTEMPT AT APPLYING G IRSANOV ’ S THEOREM

Original: First, we recall the statement of Girsanov’s theorem. Theorem 8 (Girsanov’s theorem, (Le Gall, 2016, Theorem 5.22)) . Let P T and Q T be two probability measures on path space C ([0 , T ]; R d ) .
Revised: First, we recall a consequence of Girsanov’s theorem that can be obtained by combining Pages 136–139, Theorem 5.22, and Theorem 4.13 of Le Gall (2016).

Original: Suppose that under P T , the process ( X t ) t ∈ [0 ,T ] follows
Revised: Theorem 9. For t ∈ [0 , T ] , let L t

Original: [Equation]{+= (cid:82) t 
+}
Revised: {+= (cid:82) t 
+}

Original:  where˜ B is a P T -Brownian motion, and under Q  T , the process ( X t ) t ∈ [0 ,T ] follows{+
 ∥ b s ∥+}
Revised: b s d B s where B is a Q -Brownian motion. Assume that E Q (cid:82) T {+
 ∥ b s ∥+}

Original: [Equation]
Revised: 2 d s < ∞ .

Original: where B is a Q T -Brownian motion.
Revised: Then, L is a Q -martingale in L 2 ( Q ) .

Original: We assume that for each t > 0 , σ t is a d × d symmetric positive definite matrix.
Revised: Moreover, if E Q E ( L ) T = 1 , where E ( L ) t := exp (cid:16)(cid:90) t [Equation]

Original: Then, provided that Novikov’s condition holds,
Revised: (cid:90) [Equation] s (cid:17)

Original: [Equation]
Revised: [Equation]

Original: 16) we have that
Revised: (11) then E ( L ) is also a Q -martingale and the process t [Equation] B [Equation]

Revised: b

Original: [Equation]
Revised: [Equation]

Original: We would like to apply Girsanov’s theorem with P T =P ∞ ,q T T , Q 
Revised: d s is a Brownian motion under P := E ( L ) T Q , the probability distribution with density E ( L ) T w.r.t. Q . If the assumptions of Girsanov’s theorem are satisfied (i.e., the condition (11)), we can apply Girsanov’s theorem to Q = Q ← T and

Original: [Equation]
Revised: [Equation]

Original: (for t ∈ [ kh, ( k + 1) h ] ), b t = 
Revised: where t ∈ [ kh, ( k + 1) h ] . This tells us that under P = E ( L ) T Q ← T , there exists a Brownian motion ( β t ) t ∈ [0 ,T ] s.t. [Equation] Recall that under Q ← T we have a.s.

Original: X t + 2 ∇ ln q T − t ( X t ) , and σ t = √ 2 I d .
Revised: [Equation] [Equation] The equation above still holds P -a.s. since P ≪ Q ← T (even if B is no longer a P -Brownian motion).

Original: Unfortunately, under 
Revised: Plugging (12) into (13) we have P -a.s., [Equation] that In other words, under P , the distribution of X is the SGM algorithm started at q T , i.e., P = P q T T =

Revised: E ( L ) T Q ← T .

Revised: Therefore,

Original: Assumptions 1, 2, and 3 alone, Novikov’s condition need not hold.
Revised: [Equation] where we used E Q ← T L t = 0 because L is a martingale. The equality (14) allows us to bound the discrepancy between the SGM algorithm and the reverse process. B. C HECKING THE ASSUMPTIONS OF G IRSANOV ’ S THEOREM AND THE G IRSANOV DISCRETIZATION ARGUMENT

Original: In Appendix F we give a workaround for this issue by defining a sequence of suitable truncations of the reverse and SGM processes for which Novikov’s condition does hold and showing that these truncations converge in an appropriate sense to the processes we care about.
Revised: In most applications of Girsanov’s theorem in sampling, a sufficient condition for (11) to hold, known as Novikov’s condition , is satisfied. Here, Novikov’s condition writes [Equation] and if Novikov’s condition holds, we can apply Girsanov’s theorem directly.

Original: In this section however, we will work under the assumption that P ∞ ,q T T and Q ← T do satisfy Novikov’s condition as it leads to a considerably simpler proof.
Revised: However, under Assumptions 1, 2, and 3 alone, Novikov’s condition need not hold. Indeed, in order to check Novikov’s condition, we would want X 0 to have sub-Gaussian tails for instance.

Original: Corollary 9. Assuming that P ∞ ,q T T and Q T satisfy Novikov’s condition (16) , it holds that
Revised: Furthermore, we also could not check that the condition (11), which is weaker than Novikov’s condition, holds. Therefore, in the proof of the next Theorem, we use a approximation technique to show that

Original: [Equation]
Revised: [Equation]

Original: We now apply Corollary 9 and bound  the discretization error.
Revised: We then use a discretization argument based on stochastic calculus to further bound this quantity. The result is the following theorem.

Original: Theorem 10 (discretization error for DDPM) .
Revised: Theorem 10 (discretization error for DDPM) .

Original: Suppose that Assumptions 1, 2, and 3 hold.
Revised: Suppose that Assumptions 1, 2, and 3 hold.

Original: Let Q ← T and P ∞ ,q T T denote the measures on path space corresponding to the reverse process (4) and the SGM algorithm with L ∞ -accurate score estimate initialized at q T .
Revised: Let Q ← T and P q T T denote the measures on path space corresponding to the reverse process (4) and the SGM algorithm with L 2 -accurate score estimate initialized at q T .

Original: Assume that L ≥ 1 and h ≲ 1 /L .
Revised: Assume that L ≥ 1 and h ≲ 1 /L .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: We first give a proof under the assumption that P ∞ ,q T T and Q ← T satisfy Novikov’s condition (16). In Appendix F, we show how to lift this assumption and give a fully rigorous proof of this theorem. Proof assuming that Novikov’s condition holds.
Revised: Proof.

Original: For t ∈ [ kh, ( k + 1) h ] , we can decompose ∥ s ∞
Revised: We start by proving N − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ln
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: − [Equation]
Revised: we give the approximation argument to prove the inequality (15). Bound on the discretization error.

Original: X [Equation] ∥ [Equation]
Revised: For t ∈ [ kh, ( k + 1) h ] , we can decompose

Original: [Equation]
Revised: [Equation]

Original: We must bound the change in the score function along the forward process.
Revised: We must bound the change in the score function along the forward process.

Original: If S : R d → R d is the mapping S ( x ) :=
Revised: If S : R d → R d is the mapping S ( x ) :=

Original: exp( − ( t − kh ))
Revised: exp( − ( t − kh ))

Original: x , then q T − kh = S
Revised: x , then q T − kh = S

Original: # q T − t ∗ normal (0 , 1 − exp( − 2 ( t − kh ))) .
Revised: # q T − t ∗ normal (0 , 1 − exp( − 2 ( t − kh ))) .

Original: We can then use (Lee et al., 2022a, Lemma C.12) (or the more general Lemma 15 that we prove in
Revised: We can then use Lee et al. (2022a, Lemma C.12) (or the more general Lemma 17 that we prove in

Original: Section C.4) with α = exp( t − kh )
Revised: Section C.4) with α = exp( t − kh )

Original: = 1 + O ( h ) and σ 2 = 1 − exp( − 2 ( t − kh ))
Revised: = 1 + O ( h ) and σ 2 = 1 − exp( − 2 ( t − kh ))

Original: =
Revised: =

Original: O ( h ) to obtain
Revised: O ( h ) to obtain

Original: [Equation]
Revised: [Equation]

Original: where the last line uses L ≥ 1 .
Revised: where the last line uses L ≥ 1 .

Original: For the last term,
Revised: For the last term,

Original: [Equation]
Revised: [Equation]

Original: where the second term above is absorbed into the third term of the decomposition (17).
Revised: where the second term above is absorbed into the third term of the decomposition (16).

Original: Hence,∥
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Using the fact that under Q ← T , the process ( X t ) t ∈ [0 ,T ] is the time reversal of the forward process ( ¯ X t ) t ∈ [0 ,T ] , we can apply the moment bounds in Lemma 11 and the movement bound in Lemmato obtain
Revised: Using the fact that under Q ← T , the process ( X t ) t ∈ [0 ,T ] is the time reversal of the forward process ( ¯ X t ) t ∈ [0 ,T ] , we can apply the moment bounds in Lemma 11 and the movement bound in Lemmato obtain

Original: [Equation]
Revised: [Equation]

Revised: Approximation argument.

Revised: For t ∈ [0 , T ] , let L t

Revised: =

Original: {+(cid:82) t 
+}
Revised: {+(cid:82) t 
+}

Revised: b s d B s where B is a Q ← T -Brownian motion and we define

Revised: [Equation]

Revised: for t ∈ [ kh, ( k + 1) h ] .

Original: {+We proved that E Q ← T (cid:82) T 
+}
Revised: {+We proved that E Q ← T (cid:82) T 
+}

Revised: ∥ b s ∥

Original: {+2 d s ≲ ( ε 2score + L 2 dh + L 2 m 
+}
Revised: {+2 d s ≲ ( ε 2score + L 2 dh + L 2 m 
+}

Revised: h 2 ) T < ∞ .

Revised: Using Le Gall (2016, Proposition 5.11), ( E ( L ) t ) t ∈ [0 ,T ] is a local martingale.

Revised: Therefore, there exists a non-decreasing sequence of stopping times T n ↗ T s.t.

Revised: ( E ( L ) t ∧ T n ) t ∈ [0 ,t ] is a martingale.

Revised: Note that E ( L ) t ∧ T n = E ( L n ) t where L nt =

Revised: L t ∧ T n .

Revised: Since E ( L n ) is a martingale, we have

Revised: [Equation]

Revised: i.e., E Q ← T E ( L ) T n = 1 .

Original: {+We apply Girsanov’s theorem to L nt = (cid:82) t 
+}
Revised: {+We apply Girsanov’s theorem to L nt = (cid:82) t 
+}

Revised: b s 1 [0 ,T n ] ( s ) d B s , where B is a Q ← T -Brownian motion.

Original: {+Since E Q ← T (cid:82) T 0 ∥ b s 1 [0 ,T n ] ( s ) ∥ 2 d s ≤ E Q ← T (cid:82) T 
+}
Revised: {+Since E Q ← T (cid:82) T 0 ∥ b s 1 [0 ,T n ] ( s ) ∥ 2 d s ≤ E Q ← T (cid:82) T 
+}

Revised: ∥ b s ∥

Revised: 2 d s < ∞ (see the last paragraph) and E

Revised: Q ← T E ( L n ) T = 1 , we obtain that under P n := E ( L n ) T Q ← T there exists a Brownian motion β n s.t.

Revised: for t ∈ [0 , T ] ,

Revised: [Equation]

Revised: Recall that under Q ← T we have a.s.

Revised: [Equation]

Revised: The equation above still holds P n -a.s.

Revised: since P n ≪ Q ← T .

Revised: Combining the last two equations we then obtain P n -a.s.,

Revised: [Equation]

Revised: and X 0 ∼ q T .

Revised: In other words, P n is the law of the solution of the SDE (17).

Revised: At this stage we have the bound

Revised: [Equation]

Revised: where we used that E Q ← T L T n = 0 because L is a Q ← T -martingale and T n is a bounded stopping time (Le Gall, 2016, Corollary 3.23).

Revised: Our goal is now to show that we can obtain the final result by an approximation argument.

Revised: We consider a coupling of ( P n ) n ∈ N , P q T T : a sequence of stochastic processes ( X n ) n ∈ N over the same probability space, a stochastic process X and a single Brownian motion W over that space s.t.

Revised: [Equation]

Revised: and

Revised: [Equation]

Original: The result now follows from Corollary 9 and Pinsker’s inequality.
Revised: with X 0 = X n 0 a.s. and X 0 ∼ q T . Note that the distribution of X n (resp. X ) is P n (resp.

Original: B. P ROOF OF T HEOREM
Revised: P q T T ). Let ε > 0 and consider the map π ε : C ([0 , T ]; R d ) → C ([0 , T ]; R d ) defined by [Equation] ε ) .

Revised: Noting that X nt =

Revised: X t for every t ∈ [0 , T n ] and using Lemma 13, we have π ε ( X n ) → π ε ( X ) a.s., uniformly over [0 , T ] .

Revised: Therefore, π ε # P n → π ε # P q T T weakly.

Revised: Using the lower semicontinuity of the

Revised: KL divergence and the data-processing inequality (Ambrosio et al., 2005, Lemma 9.4.3 and Lemma 9.4.5), we obtain

Revised: [Equation]

Revised: Finally, using Lemma 14, π ε ( ω ) → ω as ε → 0 , uniformly over [0 , T ] .

Revised: Therefore, using Ambrosio et al.

Original: We can now conclude our main result, namely a convergence guarantee when we only have an
Revised: (2005, Corollary 9.4.6), KL (( π ε ) # Q ← T ∥ ( π ε ) # P q T T ) → KL ( Q ← T ∥

Original: L 2 -accurate score estimate.
Revised: P q T T ) as ε ↘ 0 .

Original: To prove this, we use the following truncation lemma.
Revised: Therefore, [Equation] We conclude with Pinsker’s inequality ( TV 2 ≤ KL ). B. P ROOF OF T HEOREM

Revised: We can now conclude our main result.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: We recall the notation from Section 4.
Revised: We recall the notation from Section 4.

Original: By the data processing inequality,
Revised: By the data processing inequality,

Original: Theorem 10 implies

Original: [Equation]
Revised: [Equation]

Original: By the data processing inequality and Pinsker’s inequality, [Equation] where we used the convergence of the OU process in KL divergence (see, e.g., (Bakry et al., 2014, Theorem 5.2.1)).
Revised: Using the convergence of the OU process in KL divergence (see, e.g., Bakry et al., 2014, Theorem 5.2.1) and applying Theorem 10 for the second term,

Original: Hence, we can apply Lemma 7.

Original: Here, we take the bad set B t := {∥ s t − ∇ ln q t ∥ ≥ ε score , ∞ } , so by Chebyshev’s inequality and Assumption 3, we can take δ = ( ε score /ε score , ∞ ) 2 .

Original: It yields

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 follows after optimizing over the choice of ε score , ∞ .
Revised: which proves the result.

Original: B. A UXILIARY LEMMAS
Revised: B. A UXILIARY LEMMAS

Original: In this section, we prove some auxiliary lemmas which are used in the proof of Theorem 2.
Revised: In this section, we prove some auxiliary lemmas which are used in the proof of Theorem 2.

Original: Lemma 11 (moment bounds for DDPM) .
Revised: Lemma 11 (moment bounds for DDPM) .

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: Let ( ¯ X t ) t ∈ [0 ,T ] denote the forward process (1) .
Revised: Let ( ¯ X t ) t ∈ [0 ,T ] denote the forward process (1) .

Original: (moment bound)
Revised: (moment bound)

Original: For

Original: [Equation]
Revised: [Equation]

Original: (score function bound) For
Revised: (score function bound) For

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Along the OU process, we have ¯ X t d = exp( − t ) ¯ X 

Revised: Along the OU process, we have ¯ X t d = exp( − t ) ¯ X

Original: + (cid:112) 1 − exp( − 2 t ) ξ , where ξ ∼ normal (0 , I d ) is independent of ¯ X 0 .
Revised: + (cid:112) 1 − exp( − 2 t ) ξ , where ξ ∼ normal (0 , I d ) is independent of ¯ X 0 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: This follows from the L -smoothness of ln q t (see, e.g., (Vempala & Wibisono, 2019, Lemma 9)).
Revised: This follows from the L -smoothness of ln q t (see, e.g., Vempala & Wibisono, 2019, Lemma 9).

Original: We give a short proof for the sake of completeness.
Revised: We give a short proof for the sake of completeness.

Original: If L t f :=
Revised: If L [Equation] f :=

Original: ∆ f − ⟨∇ U t , ∇ f ⟩ is the generator associated with q t ∝ exp( − U t ) , then
Revised: ∆ f − ⟨∇ U t , ∇ f ⟩ is the generator associated with q t ∝ exp( − U t ) , then

Original: [Equation]
Revised: [Equation]

Original: Lemma 12 (movement bound for DDPM) .
Revised: Lemma 12 (movement bound for DDPM) .

Original: Suppose that Assumption 2 holds.
Revised: Suppose that Assumption 2 holds.

Original: Let ( ¯ X t ) t ∈ [0 ,T ] denote the forward process (1) .
Revised: Let ( ¯ X t ) t ∈ [0 ,T ] denote the forward process (1) .

Original: For 0 ≤ s < t with δ :=
Revised: For 0 ≤ s < t with δ :=

Original: t − s , if δ ≤ 1 , then
Revised: t − s , if δ ≤ 1 , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: where we used Lemma 11.
Revised: where we used Lemma 11.

Revised: We omit the proofs of the two next lemmas as they are straightforward.

Revised: Lemma 13.

Original: C P ROOFS FOR CLD C. B ACKGROUND ON THE CLD PROCESS
Revised: Consider f n , f : [0 , T ] → R d s.t. there exists an increasing sequence ( T n ) n ∈ N ⊆ [0 , T ] satisfying the conditions • T n → T as n → ∞ , • f n ( t ) =

Revised: f ( t ) for every t ≤ T n .

Revised: Then, for every ε > 0 , f n → f uniformly over [0 , T − ε ] .

Revised: In particular, f n ( · ∧ T − ε )

Revised: → f ( · ∧ T − ε ) uniformly over [0 , T ] .

Revised: Lemma 14.

Revised: Consider f : [0 , T ] → R d continuous, and f ε : [0 , T ] → R d s.t.

Revised: f ε ( t ) =

Revised: f ( t ∧ ( T − ε )) for ε > 0 .

Revised: Then f ε → f uniformly over [0 , T ] as ε → 0 .

Revised: B. P ROOF OF C OROLLARY

Revised: Proof of Corollary 6.

Revised: For R 0 >

Revised: [Equation]

Revised: let Π R 0 denote the projection onto B (0 , R 0 ) .

Revised: We want to prove that W 2 ((Π R 0 )

Revised: # p T − t , q ) ≤ ε .

Revised: We use the decomposition

Revised: [Equation]

Revised: Π

Revised: [Equation]

Revised: For the first term, since (Π R 0 )

Revised: # p T − t and (Π R 0 )

Revised: # q t both have support contained in B (0 , R 0 )

Revised: [Equation]

Revised: we can upper bound the Wasserstein distance by the total variation distance.

Revised: Namely, Rolland (2022,

Revised: Lemma 9) implies that

Revised: [Equation]

Revised: By the data-processing inequality,

Revised: [Equation]

Revised: where ε TV is from Corollary

Revised: [Equation]

Revised: yielding

Revised: [Equation]

Revised: Next

Revised: [Equation]

Revised: we take R 0 ≥ R so that (Π R 0 )

Revised: # q = q .

Revised: Since Π R 0 is 1 -Lipschitz, we have

Revised: [Equation]

Revised: where ε W 2 is from Corollary 4.

Revised: Combining these bounds,

Revised: [Equation]

Revised: We now take ε W 2 = ε/ 3 , R 0 = (cid:101)

Revised: Θ( R ) , and ε TV = (cid:101) Θ( ε 2 /R 2 ) to obtain the desired result.

Revised: The iteration complexity follows from Corollary 4.

Revised: C P ROOFS FOR CLD C. B ACKGROUND ON THE CLD PROCESS

Original: More generally, for the forward process we can introduce a friction parameter γ > 0 and consider
Revised: More generally, for the forward process we can introduce a friction parameter γ > 0 and consider

Original: [Equation]
Revised: [Equation]

Original: If we write ¯ θ t := ( ¯ X t , ¯ V t ) , then the forward process satisfies the linear SDEd
Revised: If we write ¯ θ t := ( ¯ X t , ¯ V t ) , then the forward process satisfies the linear SDE

Original: [Equation]
Revised: [Equation]

Original: t , where A γ := 
Revised: where A γ := (cid:20) [Equation]

Original: (cid:20) − 1 − γ (cid:21) and Σ γ := 
Revised: [Equation] γ (cid:21) and Σ γ := (cid:20)

Original: (cid:20) 0 √ 2 γ (cid:21) .
Revised: [Equation] 2 γ (cid:21) .

Original: The solution to the SDE is given by
Revised: The solution to the SDE is given by

Original: [Equation]
Revised: [Equation]

Original:  which means that by the Itˆo isometry,
Revised: 18) which means that by the Itˆo isometry,

Original: [Equation]
Revised: [Equation]

Original: Since det A γ = 1 , A γ is always invertible.
Revised: Since det A γ = 1 , A γ is always invertible.

Original: Moreover, from tr A γ = − γ , one can work out that the spectrum of A γ is
Revised: Moreover, from tr A γ = − γ , one can work out that the spectrum of A γ is

Original: [Equation]
Revised: [Equation]

Original: However, A γ is not diagonalizable.
Revised: However, A γ is not diagonalizable.

Original: The case of γ = 2 is special, as it corresponds to the case when the spectrum is {− 1 } , and it corresponds to the critically damped case .
Revised: The case of γ = 2 is special, as it corresponds to the case when the spectrum is {− 1 } , and it corresponds to the critically damped case .

Original: Following (Dockhorn et al., 2022), which advocated for setting γ = 2 , we will also only consider the critically damped case.
Revised: Following Dockhorn et al. [Equation] 2022), which advocated for setting γ = 2 , we will also only consider the critically damped case.

Original: This also has the advantage of substantially simplifying the calculations.
Revised: This also has the advantage of substantially simplifying the calculations.

Original: C. G IRSANOV DISCRETIZATION ARGUMENT
Revised: C. G IRSANOV DISCRETIZATION ARGUMENT

Original: In order to apply Girsanov’s theorem, we introduce the path measures P∞ , q T T and Q ← T , under which
Revised: In order to apply Girsanov’s theorem, we introduce the path measures P q T T and Q ← T , under which

Original: [Equation]
Revised: [Equation]

Original: for t ∈ [ kh, ( k + 1) h ] , and 
Revised: for t ∈ [ kh, ( k + 1) h ] , and d X

Original: [Equation]
Revised: [Equation]

Original: respectively. We can apply Girsanov’s theorem with
Revised: =

Original: [Equation]
Revised: [Equation]

Original:  t) (cid:21) ,
Revised: d t ,

Original: [Equation]
Revised: [Equation]

Original: = (cid:20) [Equation] [Equation] (cid:21) .
Revised: respectively.

Original: Here, σ t is not invertible, so strictly speaking Theorem 8 does not apply; however, since ˜ b t − b t ∈ range σ t , the expression σ − 1 t (˜ b t − b t ) is still well-defined, and the following corollary can be justified, provided that Novikov’s condition holds.
Revised: Applying Girsanov’s theorem, we have the following theorem.

Original: Corollary 13.
Revised: Corollary 15.

Original: Suppose that Novikov’s condition (16) holds.
Revised: Suppose that Novikov’s condition holds: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: Similarly to Appendix B.2, even if Novikov’s condition does not hold, one can use an approximation to argue that the KL divergence is still upper bounded by the last expression.

Revised: Since the argument follows along the same lines, we omit it for brevity.

Original: Using this, we now aim to prove the following theorem.
Revised: Using this, we now aim to prove the following theorem.

Original: Theorem 14 (discretization error for CLD) .
Revised: Theorem 16 (discretization error for CLD) .

Original: Suppose that Assumptions 2, 4, and 5 hold.
Revised: Suppose that Assumptions 2, 4, and 5 hold.

Original: Let Q ← T and P∞ , q T T denote the measures on path space corresponding to the reverse process (7) and the SGM algorithm with L ∞ -accurate score estimate initialized at q T .
Revised: Let Q ← T and P q T T denote the measures on path space corresponding to the reverse process (7) and the SGM algorithm with L 2 -accurate score estimate initialized at q T .

Original: Assume that L ≥ 1 and h ≲ 1 /L .
Revised: Assume that L ≥ 1 and h ≲ 1 /L .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Similarly to Section B.2, we prove the theorem assuming that Novikov’s condition (16) holds. This assumption can be lifted using similar arguments to the ones in Appendix F, and we omit the details.
Revised: Proof. For t ∈ [ kh

Original: Proof assuming that Novikov’s condition holds.
Revised: [Equation]

Original: For t ∈ [ kh, ( k + 1) h ] , we can decompose
Revised: ( k + 1) h ] [Equation] we can decompose

Original: [Equation]
Revised: [Equation]

Original: The change in the score function is bounded by Lemma 15, which generalizes (Lee et al., 2022a,
Revised: The change in the score function is bounded by Lemma 17, which generalizes Lee et al. (2022a,

Original: Lemma C.12).
Revised: Lemma C.12).

Original: From the representation (20) of the solution to the CLD, we note that
Revised: From the representation (18) of the solution to the CLD, we note that

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: In particular, since ∥ A 2 ∥ op ≲ 1 , ∥ A − 1 2 ∥ op ≲ 1 , and ∥ Σ 2 ∥ op ≲ 1 it follows that ∥ M 0 ∥ op
Revised: In particular, since ∥ A 2 ∥ op ≲ 1 , ∥ A − 1 2 ∥ op ≲ 1 , and ∥ Σ 2 ∥ op ≲ 1 it follows that ∥ M 0 ∥ op

Original: = 1+ O ( h ) and ∥
Revised: = 1+ O ( h ) and ∥

Original: M 1 ∥ op =
Revised: M 1 ∥ op =

Original: O ( h ) .
Revised: O ( h ) .

Original: Substituting this into Lemma 15, we deduce that if h ≲ 1 /L , then
Revised: Substituting this into Lemma 17, we deduce that if h ≲ 1 /L , then

Original: [Equation]
Revised: [Equation]

Original: where in the last step we used L ≥ 1 .
Revised: where in the last step we used L ≥ 1 .

Original: For the last term,
Revised: For the last term,

Original: [Equation]
Revised: [Equation]

Original: where the second term above is absorbed into the third term of the decomposition (21).
Revised: where the second term above is absorbed into the third term of the decomposition (19).

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: By applying the moment bounds in Lemma 16 together with Lemma 17 on the movement of the CLD,
Revised: By applying the moment bounds in Lemma 18 together with Lemma 19 on the movement of the CLD process, we obtain

Original: [Equation]
Revised: [Equation]

Original: Together with Corollary 13 and Pinsker’s inequality [Equation] it completes the proof.
Revised: The proof is concluded via an approximation argument as in Section B.2.

Original: Remark.
Revised: Remark.

Original: We now pause to discuss why the discretization bound above does not improve upon the result for DDPM (Theorem 10).
Revised: We now pause to discuss why the discretization bound above does not improve upon the result for DDPM (Theorem 10).

Original: In the context of log-concave sampling, one instead considers the underdamped Langevin process
Revised: In the context of log-concave sampling, one instead considers the underdamped Langevin process

Original: [Equation]
Revised: [Equation]

Original: which is discretized to yield the algorithm
Revised: which is discretized to yield the algorithm

Original: [Equation]
Revised: [Equation]

Original: for t ∈ [ kh, ( k + 1) h ] .
Revised: for t ∈ [ kh, ( k + 1) h ] .

Original: Let P T denote the path measure for the algorithm [Equation] and let Q T denote the path measure for the continuous-time process.
Revised: Let P T denote the path measure for the algorithm, and let Q T denote the path measure for the continuous-time process.

Original: After applying Girsanov’s theorem, we obtain
Revised: After applying Girsanov’s theorem, we obtain

Original: [Equation]
Revised: [Equation]

Original: In this expression, note that ∇ U depends only on the position coordinate.
Revised: In this expression, note that ∇ U depends only on the position coordinate.

Original: Since the X process is smoother (as we do not add Brownian motion directly to X ), the error ∥∇ U ( X t ) − ∇ U ( X kh )
Revised: Since the X process is smoother (as we do not add Brownian motion directly to X ), the error ∥∇ U ( X t ) − ∇ U ( X kh )

Original: ∥is of size O ( dh 2 ) , which allows us to take step size h ≲ 1 / √ d .
Revised: ∥is of size O ( dh 2 ) , which allows us to take step size h ≲ 1 / √ d .

Original: This explains why the use of the underdamped Langevin diffusion leads to improved dimension dependence for log-concave sampling.
Revised: This explains why the use of the underdamped Langevin diffusion leads to improved dimension dependence for log-concave sampling.

Original: In contrast, consider the reverse process, in which
Revised: In contrast, consider the reverse process, in which

Original: [Equation]
Revised: [Equation]

Original: Since discretization of the reverse process involves the score function, which depends on both X and V , the error now involves controlling ∥ V t − V kh ∥ 2 , which is of size O ( dh ) (the process V is not very smooth because it includes a Brownian motion component).
Revised: Since discretization of the reverse process involves the score function, which depends on both X and V , the error now involves controlling ∥ V t − V kh ∥ 2 , which is of size O ( dh ) (the process V is not very smooth because it includes a Brownian motion component).

Original: Therefore, from the form of the reverse process, we may expect that SGMs based on the CLD do not improve upon the dimension dependence of DDPM.
Revised: Therefore, from the form of the reverse process, we may expect that SGMs based on the CLD do not improve upon the dimension dependence of DDPM.

Original: In Section C.5, we use this observation in order to prove a rigorous lower bound against discretization of SGMs based on the CLD.
Revised: In Section C.5, we use this observation in order to prove a rigorous lower bound against discretization of SGMs based on the CLD.

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: Proof of Theorem 5.
Revised: Proof of Theorem 7.

Original: Similarly to the proof of Theorem 2 in Section B.3, Theorem 14 together with the data processing inequality yield
Revised: By the data processing inequality,

Original: [Equation]
Revised: [Equation]

Original: In (Ma et al., 2021), following the entropic hypocoercivity approach of (Villani, 2009), Ma et al. consider a Lyapunov functional L which is equivalent to the sum of the KL divergence and the Fisher information,
Revised: In Ma et al. (2021), following the entropic hypocoercivity approach of Villani (2009), Ma et al. consider a Lyapunov functional L which is equivalent to the sum of the KL divergence and the Fisher information,

Original: [Equation]
Revised: [Equation]

Original: which decays exponentially fast in time: there exists a universal constant c > 0 such that for all t ≥ 0 ,
Revised: which decays exponentially fast in time: there exists a universal constant c > 0 such that for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Since q 0 = q ⊗ γ d and γ 2 d = γ d ⊗ γ d , then L ( q 0 ∥ γ 2 d ) ≲D KL ( q ∥ γ d ) + FI ( q ∥ γ d ) .Applying
Revised: Since q 0 = q ⊗ γ d and γ 2 d = γ d ⊗ γ d , then L ( q 0 ∥ γ 2 d ) ≲ KL ( q ∥ γ d ) + FI ( q ∥ γ d ) .

Original: Lemma 7 and Pinsker’s inequality, we deduce that
Revised: By Pinsker’s inequality and Theorem 16, we deduce that

Original: [Equation]
Revised: [Equation]

Original: Theorem 5 follows from optimizing over the choice of ε score , ∞ .
Revised: which completes the proof.

Original: C. A UXILIARY LEMMAS
Revised: C. A UXILIARY LEMMAS

Original: We begin with the perturbation lemma for the score function.
Revised: We start with a perturbation lemma for the score function.

Original: Lemma 15 (score perturbation lemma) .
Revised: Lemma 17 (score perturbation lemma) .

Original: Let 0 < ζ < 1 .
Revised: Let 0 < ζ < 1 .

Original: Suppose that M 0 , M 1 ∈ R 2 d × 2 d are two matrices, where M 1 is symmetric.
Revised: Suppose that M 0 , M 1 ∈ R 2 d × 2 d are two matrices, where M 1 is symmetric.

Original: Also, assume that ∥ M 0 − I 2 d ∥ op ≤ ζ , so that M 0 is invertible.
Revised: Also, assume that ∥ M 0 − I 2 d ∥ op ≤ ζ , so that M 0 is invertible.

Original: Let q = exp( − H ) be a probability density on R 2 d such that ∇ H is L -Lipschitz with L ≤
Revised: Let q = exp( − H ) be a probability density on R 2 d such that ∇ H is L -Lipschitz with L ≤

Original: [Equation]
Revised: [Equation]

Original: ∥ M 1 ∥ op .
Revised: M [Equation] op .

Original: Then, it holds that 
Revised: Then, it holds that (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows along the lines of (Lee et al., 2022a, Lemma C.12).
Revised: The proof follows along the lines of Lee et al. (2022a, Lemma C.12).

Original: First, we show that when M 

Revised: First, we show that when M

Original: =
Revised: =

Original: I 2 d , if L ≤ 1 2 ∥ M 1 ∥ op then
Revised: I 2 d , if L ≤ 1 2 ∥ M 1 ∥ op then

Original: [Equation]
Revised: [Equation]

Original: Let S denote the subspace S :
Revised: Let S denote the subspace S :

Original: =
Revised: =

Original: range M 1 .
Revised: range M 1 .

Original: Then, since
Revised: Then, since

Original: [Equation]
Revised: [Equation]

Original: where M − 1is well-defined on S , we have
Revised: where M − 1is well-defined on S [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Here, q θ is the measure on θ + S such that
Revised: Here, q θ is the measure on θ + S such that

Original: [Equation]
Revised: [Equation]

Original: Note that since L ≤1 2 ∥ M 1 ∥ op , then if we write q θ ( θ ′ ) ∝ exp( − H θ ( θ ′ )) , we have
Revised: Note that since L ≤

Original: [Equation]
Revised: [Equation]

Original: on θ + S
Revised: , then if we write q θ ( θ ′ ) ∝ exp( − H θ ( θ ′ )) , we have

Original: [Equation]
Revised: [Equation]

Original: Let θ ⋆ ∈ arg min
Revised: Let θ ⋆ ∈ arg min

Original: H θ denote a mode.
Revised: H θ denote a mode.

Original: We bound∥
Revised: We bound

Original: [Equation]
Revised: [Equation]

Original: L ∥ θ ⋆ − θ ∥ .

Original: For the first term, (Dalalyan et al., 2019, Proposition 2) yields
Revised: For the first term, (Dalalyan et al., 2019, Proposition 2) yields

Original: [Equation]
Revised: [Equation]

Original: For the second term, since the mode satisfies ∇ H ( θ ⋆ )
Revised: For the second term, since the mode satisfies ∇ H ( θ ⋆ )

Original: + M − 1 1 ( θ ⋆ − θ )
Revised: + M − 1 1 ( θ ⋆ − θ )

Original: = 0 , we have
Revised: = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: which is rearranged to yield
Revised: which is rearranged to yield

Original: [Equation]
Revised: [Equation]

Original: After combining the bounds, we obtain the claimed estimate (22).
Revised: After combining the bounds, we obtain the claimed estimate (20).

Original: Next, we consider the case of general M 0 .
Revised: Next, we consider the case of general M 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: We can apply (22) with ( M 0 )
Revised: We can apply (20) with ( M 0 )

Original: # q in place of q , noting that ( M 0 )
Revised: # q in place of q , noting that ( M 0 )

Original: # q ∝ exp( − H ′ ) for H ′ :
Revised: # q ∝ exp( − H ′ ) for H ′ :

Original: =
Revised: =

Original: H ◦ Mwhich is L ′ -smooth for L ′ :
Revised: H ◦ Mwhich is L ′ -smooth for L ′ :

Original: =
Revised: =

Original: L ∥ M 0 ∥ 2op ≲ L , to get 
Revised: L ∥ M 0 ∥ 2op ≲ L , to get (cid:13)(cid:13)(cid:13) ∇

Original: [Equation]
Revised: [Equation]

Original: Note that 
Revised: Note that ∥∇

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: Combining the bounds,
Revised: Combining the bounds,

Original: [Equation]
Revised: [Equation]

Original: so the lemma follows.
Revised: so the lemma follows.

Original: Next, we prove the moment and movement bounds for the CLD.
Revised: Next, we prove the moment and movement bounds for the CLD.

Original: Lemma 16 (moment bounds for CLD) .
Revised: Lemma 18 (moment bounds for CLD) .

Original: Suppose that Assumptions 2 and 4 hold.
Revised: Suppose that Assumptions 2 and 4 hold.

Original: Let ( ¯ X t , ¯ V t ) t ∈ [0 ,T ] denote the forward process (6) .
Revised: Let ( ¯ X t , ¯ V t ) t ∈ [0 ,T ] denote the forward process (6) .

Original: (moment bound)
Revised: (moment bound)

Original: Forall t ≥
Revised: For

Original: [Equation]
Revised: [Equation]

Original: (score function bound)
Revised: (score function bound

Original: For all t [Equation]
Revised: For all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: Next, the coupling argument of (Cheng et al., 2018) shows that the CLD converges exponentially fast in the Wasserstein metric associated to a twisted norm |||·||| which is equivalent (up to universal constants) to the Euclidean norm ∥·∥ .
Revised: Next, the coupling argument of Cheng et al. (2018) shows that the CLD converges exponentially fast in the Wasserstein metric associated to a twisted norm |||·||| which is equivalent (up to universal constants) to the Euclidean norm ∥·∥ .

Original: It implies the following result, see, e.g., (Cheng et al., 2018, Lemma 8):
Revised: It implies the following result, see, e.g., Cheng et al. (2018, Lemma 8):

Original: [Equation]
Revised: [Equation]

Original: The proof is the same as in Lemma 11.
Revised: The proof is the same as in Lemma 11.

Original: Lemma 17 (movement bound for CLD) .
Revised: Lemma 19 (movement bound for CLD) .

Original: Suppose that Assumptions 2 holds.
Revised: Suppose that Assumptions 2 holds.

Original: Let ( ¯ X t , ¯ V t ) t ∈ [0 ,T ] denote the forward process (6) .
Revised: Let ( ¯ X t , ¯ V t ) t ∈ [0 ,T ] denote the forward process (6) .

Original: For 0 < s < t with δ :=
Revised: For 0 < s < t with δ :=

Original: t − s , if δ ≤ 1 ,
Revised: t − s , if δ ≤ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First,
Revised: First,

Original: [Equation]
Revised: [Equation]

Original: where we used the moment bound in Lemma 16.
Revised: where we used the moment bound in Lemma 18.

Original: Next,
Revised: Next,

Original: [Equation]
Revised: [Equation]

Original: where we used Lemma 16 again.
Revised: where we used Lemma 18 again.

Original: C. L OWER BOUND AGAINST CLD
Revised: C. L OWER BOUND AGAINST CLD

Revised: When proving upper bounds on the KL divergence, we can use the approximation argument described in Section B.2 in order to invoke Girsanov’s theorem.

Revised: However, when proving lower bounds on the

Revised: KL divergence, this approach no longer works, so we check Novikov’s condition directly for the setting of Theorem 8.

Revised: Lemma 20 (Novikov’s condition holds for CLD) .

Original: Proof of Theorem 6.
Revised: Consider the setting of Theorem 8. Then, Novikov’s condition 15 holds. We defer the proof of Lemma 20 to the end of this section. Admitting Lemma 20, we now prove

Original:  Since q 0 = γ d ⊗ γ d = γ 2 d is stationary for the forward process (6), we have q t = γ 2 d for all t ≥ 0 .
Revised: Theorem 8. Proof of Theorem 8. Since q 0 = γ d ⊗ γ d = γ 2 d is stationary for the forward process (6), we have q t = γ 2 d for all t ≥ 0 .

Original: In this proof, since the score estimate is perfect and q T = γ 2 d , we simply denote the path measure for the algorithm as P T =
Revised: In this proof, since the score estimate is perfect and q T = γ 2 d , we simply denote the path measure for the algorithm as P T =

Original: P∞ , q T T .
Revised: P q T T .

Original: From Girsanov’s theorem in the form of
Revised: From Girsanov’s theorem in the form of

Original: Corollary 13 and from s T − kh
Revised: Corollary 15 and from s T − kh

Original: ( x, v ) = ∇ v ln q T − kh
Revised: ( x, v ) = ∇ v ln q T − kh

Original: ( x, v )
Revised: ( x, v )

Original: = − v [Equation] we have
Revised: = − v , we have

Original: [Equation]
Revised: [Equation]

Original: To lower bound this quantity [Equation] we use the inequality ∥ x + y ∥ 2 ≥ 12 ∥ x ∥ 2 − ∥ y ∥ 2 to write, for t ∈
Revised: To lower bound this quantity, we use the inequality ∥ x + y ∥ 2 ≥ 12 ∥ x ∥ 2 − ∥ y ∥ 2 to write, for t ∈

Original: [ kh, ( k + 1) h ]
Revised: [ kh, ( k + 1) h ]

Original: [Equation]
Revised: [Equation]

Original: Using the fact that ¯ X s ∼ γ d
Revised: Using the fact that ¯ X s ∼ γ d

Original: and ¯ V s ∼ γ d for all s ∈ [0 , T ] , we can then bound
Revised: and ¯ V s ∼ γ d for all s ∈ [ [Equation] T ] [Equation]

Revised: we can then bound

Original: [Equation]
Revised: [Equation]

Original:  provided that h ≤ 110 .
Revised: , provided that h ≤ 110 .

Original: Substituting this into (23),
Revised: Substituting this into (21),

Revised: [Equation]

Revised: dh

Original: [Equation]
Revised: [Equation]

Original: This proves the result.
Revised: This proves the result.

Original: This lower bound shows that the Girsanov discretization argument of Theorem 14 is essentially tight (except possibly the dependence on L ). 
Revised: This lower bound shows that the Girsanov discretization argument of Theorem 16 is essentially tight (except possibly the dependence on L ). We now prove Lemma 20. Proof of Lemma 20.

Revised: Similarly to the proof of Theorem 8 above, we note that

Revised: [Equation]

Revised: Hence, for a universal constant C > 0 (which may change from line to line)

Original: D D ERIVATION OF THE SCORE MATCHING OBJECTIVE
Revised: E Q ← T exp [Equation] By the Cauchy–Schwarz inequality, to prove that this expectation is finite, it suffices to consider the two terms in the exponential separately. Next, we recall that

Revised: [Equation]

Revised: Define ¯ Y t :=

Revised: ¯ X t

Revised: + ¯ V t .

Revised: Then, d ¯ Y t = − ¯ Y t d t + 2 d B t , which admits the explicit solution

Revised: [Equation]

Revised: Also, d ¯ X t = − ¯ X t d t + ¯ Y t d t , which admits the solution

Revised: [Equation]

Revised: Hence,

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: where ˜ B is another standard Brownian motion and we use the interpretation of stochastic integrals as time changes of Brownian motion (Steele, 2001, Corollary 7.1).

Revised: Since ( ¯ X 0 , ¯ V 0 ) ∼ γ 2 d has independent entries

Revised: [Equation]

Revised: then

Revised: E

Revised: [Equation]

Revised: provided that h ≲ 1 / √ T .

Revised: Also, by the Cauchy–Schwarz inequality

Revised: [Equation]

Revised: we can give a crude bound:

Revised: [Equation]

Revised: where, by standard estimates on the supremum of Brownian motion (see, e.g., Chewi et al., 2021b,

Revised: Lemma 23), the first factor is finite if h ≲ 1 / √ T (again using independence across the dimensions).

Revised: For the second factor, if we split the sum according to exp( − 2 t ) ≍ 2 k and use H ¨ older’s inequality,

Revised: [Equation]

Revised: where K = O ( T ) .

Revised: Then,

Revised: [Equation]

Revised: provided h ≲ 1 /T , where we again use Chewi et al.

Revised: (2021b, Lemma 23) and split across the coordinates.

Revised: The Cauchy–Schwarz inequality then implies

Revised: [Equation]

Revised: For the second term, by independence of the increments,

Revised: [Equation]

Revised: By Chewi et al.

Revised: (2021b, Lemma 23), this quantity is finite if h ≲ 1 , which completes the proof.

Revised: D D ERIVATION OF THE SCORE MATCHING OBJECTIVE

Original: In this section, we present a self-contained derivation of the score matching objective (14) for the reader’s convenience.
Revised: In this section, we present a self-contained derivation of the score matching objective (10) for the reader’s convenience.

Original: See also (Hyv ¨ arinen, 2005; Vincent, 2011; Song & Ermon, 2019).
Revised: See also Hyv¨arinen (2005); Vincent (2011); Song & Ermon (2019).

Original: Recall that the problem is to solve
Revised: Recall that the problem is to solve

Original: [Equation]
Revised: [Equation]

Original: This objective cannot be evaluated, even if we replace the expectation over q t with an empirical average over samples from q t .
Revised: This objective cannot be evaluated, even if we replace the expectation over q t with an empirical average over samples from q t .

Original: The trick is to use an integration by parts identity to reformulate the objective.
Revised: The trick is to use an integration by parts identity to reformulate the objective.

Original: Here, C will denote any constant that does not depend on the optimization variable s t .
Revised: Here, C will denote any constant that does not depend on the optimization variable s t .

Original: Expanding the square,
Revised: Expanding the square,

Original: [Equation]
Revised: [Equation]

Original: We can rewrite the second term using integration by parts:
Revised: We can rewrite the second term using integration by parts:

Original: [Equation]
Revised: [Equation]

Original: where γ d = normal (0 , I d ) and we used the explicit form of the law of the OU process at time t .
Revised: where γ d = normal (0 , I d ) and we used the explicit form of the law of the OU process at time t .

Original: Recall the Gaussian integration by parts identity: for any vector field v : R d → R d ,
Revised: Recall the Gaussian integration by parts identity: for any vector field v : R d → R d ,

Original: [Equation]
Revised: [Equation]

Original: Applying this identity,
Revised: Applying this identity,

Original: [Equation]
Revised: [Equation]

Original: where x t = exp( − t ) x 0 + (cid:112) 1 − exp( − 2 t )
Revised: where x t = exp( − t ) x 0 + (cid:112) 1 − exp( − 2 t )

Original: z t .
Revised: z t .

Original: Substituting this in,
Revised: Substituting this in,

Original: [Equation]
Revised: [Equation]

Original: where X 0 ∼ q and Z t ∼ γ d are independent, and X t : = exp( − t )
Revised: where X 0 ∼ q and Z t ∼ γ d are independent, and X t := exp( − t )

Original: X 

Revised: X

Original: + (cid:112) 1 − exp( − 2 t )
Revised: + (cid:112) 1 − exp( − 2 t )

Original: Z t .
Revised: Z t .

Original: E D EFERRED PROOFS
Revised: E D EFERRED PROOFS

Original: Lemma 18.
Revised: Lemma 21.

Original: Suppose that supp q ⊆ B (0 , R ) where R ≥ 1 , and let q t denote the law of the OU process at time t , started at q .
Revised: Suppose that supp q ⊆ B (0 , R ) where R ≥ 1 , and let q t denote the law of the OU process at time t , started at q .

Original: Let ε > 0 be such that ε ≪ √ d and set t ≍ ε 2 / ( √ d ( R ∨ √ d )) .
Revised: Let ε > 0 be such that ε ≪ √ d and set t ≍ ε 2 / ( √ d ( R ∨ √ d )) .

Original: Then, 
Revised: Then, 1.

Original: W 2 ( q t , q ) ≤ ε .
Revised: W 2 ( q t , q ) ≤ ε .

Original: q t satisfies
Revised: q t satisfies

Original: [Equation]
Revised: [Equation]

Original: For every t ′ ≥ t , q t ′ satisfies Assumption 1 with
Revised: For every t ′ ≥ t , q t ′ satisfies Assumption 1 with

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the OU process (1), we have ¯ X t := exp( − t )
Revised: For the OU process (1), we have ¯ X t := exp( − t )

Original: ¯ X 

Revised: ¯ X

Original: + (cid:112) 1 − exp( − 2 t )
Revised: + (cid:112) 1 − exp( − 2 t )

Original: Z , where
Revised: Z , where

Original: Z ∼ normal (0 , I d ) is independent of ¯ X 0 .
Revised: Z ∼ normal (0 , I d ) is independent of ¯ X 0 .

Original: Hence, for t ≲ 1 ,
Revised: Hence, for t ≲ 1 ,

Original: [Equation]
Revised: [Equation]

Original: We now take t ≲ min { ε/R, ε 2 /d } to ensure that W 22 ( q, q t ) ≤ ε 2 .
Revised: We now take t ≲ min { ε/R, ε 2 /d } to ensure that W 22 ( q, q t ) ≤ ε 2 .

Original: Since ε ≪ √ d , it suffices to take t ≍ ε 2 / ( √ d ( R ∨ √ d )) .
Revised: Since ε ≪ √ d , it suffices to take t ≍ ε 2 / ( √ d ( R ∨ √ d )) .

Original: For this, we use the short-time regularization result in (Otto & Villani, 2001, Corollary 2), which implies that
Revised: For this, we use the short-time regularization result in Otto & Villani (2001, Corollary 2), which implies that

Original: [Equation]
Revised: [Equation]

Original: Using (Mikulincer & Shenfeld, 2022, Lemma 4), along the OU process,
Revised: Using Mikulincer & Shenfeld (2022, Lemma 4), along the OU process,

Original: [Equation]
Revised: [Equation]

Original: With our choice of t , it implies
Revised: With our choice of t , it implies

Original: [Equation]
Revised: [Equation]

Original: F N OVIKOV ’ S CONDITION FOR DDPM

Original: In Section B.2, we proved the Girsanov discretization bound for DDPM (Theorem 10) under the assumption that the path measures P ∞ ,q T T and Q ← T satisfy Novikov’s condition (16).

Original: Unfortunately,

Original: Novikov’s condition fails under Assumptions 1, 2, and 3 alone.

Original: In this section, we remedy this by instead applying the Girsanov discretization argument to a sequence of pairs of auxiliary processes defined in place of P ∞ ,q T T and Q ← T .

Original: F. T RUNCATION ARGUMENT

Original: We first construct a smooth cutoff function for truncating the drift terms in the processes.

Original: Lemma 19.

Original: For any R > 0 , there is a smooth function ϕ

Original: R : R d → [0 , 1] satisfying:

Original: ϕ R ( x ) = 1 for all ∥ x ∥ ≤ R ,

Original: [-ϕ R ( x ) = 
-]
Revised: [-ϕ R ( x ) = 
-]

Original: for all ∥ x ∥ ≥ 2 R , 3.

Original: ϕ R is O

Original: (1 /R )

Original: -Lipschitz.

Original: Proof.

Original: Let ψ : R → [0 , 1] be an O (1) -Lipschitz and smooth function for which ψ ( z ) = 1 for all | z | ≤ 1 , ψ ( z ) = 0 for all | z | ≥ 2 .

Original: This can be constructed, for instance, by taking the piecewise linear function

Original: [Equation]

Original: for any small constant c ∈ (0 , 12 ) and convolving (cid:101) ψ with z (cid:55)→ c − 1 ϕ ( z/c ) for any mollifier ϕ supported on ( − 1 , 1) .

Original: Now define ϕ R ( x ) ≜ ψ ( ∥ x ∥ /R ) .

Original: Parts 1 and 2 of the lemma are immediate.

Original: For Lipschitzness, we can calculate for x ̸ =

Original: [Equation]

Original: which has norm at most | ψ ′ ( ∥ x ∥ /R )

Original: | /R

Original: = O

Original: (1 /R ) as desired, whereas ∇ ϕ R (0) = 0 .

Original: Lemma 20.

Original: Suppose that v : R d → R d is an L -Lipschitz vector field.

Original: Then, x (cid:55)→ ϕ R ( x ) v ( x ) is

Original: L ′ -Lipschitz for L ′ ≲ L + 1 R ∥ v (0) ∥ .

Original: Proof.

Original: Let x, y ∈ R d .

Original: If ∥ y ∥ ≤ 2 R , then

Original: [Equation]

Original: If ∥ y ∥ > 2 R , then because ϕ R ( y ) = 0 ,

Original: [Equation]

Original: as claimed.

Original: Now define the following modified processes: for t ∈

Original: [ kh, ( k + 1) h ] ,

Original: [Equation]

Original: where ¯ X tr , ←= X tr , ← , ∞ ,q Tis obtained by sampling ¯ X T ∼ q T and setting ¯ X tr , ←

Original: [Equation]

Original: R and setting ¯ X tr , ←= 0 otherwise.

Original: For clarity of presentation, we suppress the dependence on R in the notation.

Original: Let q tr , ← t := law( ¯ X tr , ← t ) and p tr , ∞ ,q T t := law( X tr , ← , ∞ ,q T t ) .

Original: Likewise, denote the corresponding measures over path space by Q tr , ← T and P tr , ∞ ,q T T .

Original: As it is convenient to think about a single sochastic process, for ease of notation we denote this by ( X tr t ) t ∈ [0 ,T ] so that under Q tr , ← T (resp.

Original: P tr , ∞ ,q T T ), ( X tr t ) t ∈ [0 ,T ] has the law of the modified reverse process (24) (resp.

Original: the modified SGM process (25)).

Original: We first verify that these modified processes satisfy Novikov’s condition.

Original: Lemma 21.

Original: It holds that

Original: [Equation]

Original: Proof.

Original: For any x ,

Original: [Equation]

Original: So the quantity in the lemma is at most exp(2 T ( A 2 + B 2 )) <

Original: ∞ as claimed.

Original: We can thus apply Girsanov’s theorem (Theorem 8), with the choices P T =

Original: P tr , ∞ ,q T T , Q

Original: [Equation]

Original: = Q

Original: [Equation]

Original: ←

Original: [Equation]

Original: ˜

Original: [Equation]

Original: , and

Original: σ t = √

Original: [Equation]

Original: d .

Original: It yields the following corollary.

Original: Corollary 22.

Original: It holds that

Original: [Equation]

Original: Our analysis for the quantity on the right-hand side of Corollary 22 proceeds along similar lines to the argument in Section B.2, though the moment and movement bounds require some additional work as we must work with ¯ X tr , ← t instead of ¯ X ← t , the former of which is not the reverse of the OU process.

Original: Theorem 23 (discretization error for modified process) .

Original: Suppose that Assumptions 1, 2, and 3 hold.

Original: For R > 0 , let Q tr , ← T and P tr , ∞ ,q T T denote the measures on path space corresponding to the modified processes (24) and (25) respectively.

Original: Assume that L ≥ 1 and h ≲ 1 /L .

Original: Then,

Original: [Equation]

Original: Proof.

Original: First, as before, we can decompose

Original: [Equation]

Original: where in the last step we used the fact that ϕ R ( X tr kh ) ∈ [0 , 1] together with Lemma 20 applied to the vector field v = ∇ ln q T − t ( L ′ is the quantity defined therein).

Original: Because (18) and (19) hold pointwise, we can bound the second term above by

Original: [Equation]

Original: where in the last line we applied Lemma 20 again.

Original: Hence,

Original: [Equation]

Original: By taking R → ∞ above, noting that lim sup R →∞ L ′ ≲ L

Original: [Equation]

Original: and applying Lemmas 25,

Original: [Equation]

Original: and 27, we get

Original: [Equation]

Original: We can bound E Q ← T [ ∥ X t ∥ 2 ] ≤ d ∨ m 22 and E Q ← T [ ∥∇ ln q T − t ( X t ) ∥ 2 ] ≤ Ld via the moment bounds of Lemma 11.

Original: Putting everything together, we get

Original: [Equation]

Original: The result now follows from Corollary 22.

Original: We are now ready to bound the discretization error for DDPM in total variation.

Original: Full proof of Theorem 10.

Original: Consider a coupling of the laws of ( X t ) t ∈ [0 ,T ] and ( X tr t ) t ∈ [0 ,T ] under

Original: Q ← T (resp.

Original: P ∞ ,q T T ) and Q tr , ← T (resp.

Original: P tr , ∞ ,q T T ) such that if X t ∈ B (0 , R ) for all t ∈ [0 , T ] , then

Original: X tr t = X t for t ∈ [0 , T ] .

Original: Denote this event by E R (resp.

Original: E ′ R ).

Original: Then

Original: TV ( Q tr , ← T , Q ← T ) ≤ P [ E R ] and

Original: TV ( P tr , ∞ ,q T T , P ∞ ,q T T ) ≤ P [ E ′ R ] .

Original: [-Clearly lim R →∞ P [ E R ] = 
-]
Revised: [-Clearly lim R →∞ P [ E R ] = 
-]

Original: and lim R →∞ P [ E ′ R ]

Original: = 0 .

Original: By Theorem 23 and Pinsker’s inequality, lim sup R →∞ TV ( Q tr , ← T , P tr , ∞ ,q T T ) 2 ≲ (

Original: [Equation]

Original: 2score , ∞ + L 2 dh + L 2 m 22 h 2 ) T .

Original: Hence, by the triangle inequality,

Original: TV ( Q ← T , P ∞ ,q T T ) ≤ lim sup R →∞ { TV ( Q tr ,

Original: [Equation]

Original: T , Q

Original: [Equation]

Original: + TV ( P

Original: [Equation]

Original: , ∞

Original: [Equation]

Original: T , P ∞ ,q

Original: [Equation]

Original: +

Original: [Equation]

Original: [Equation]

Original: This is the claimed result.

Original: F. A UXILIARY LEMMAS

Original: In this section, we prove some auxiliary lemmas which were used in the proof of Theorem 23 above.

Original: Lemma 24 (escape probability for reverse process) .

Original: Under Assumption 2,

Original: [Equation]

Original: Proof.

Original: As ¯ X ← is the reverse of the OU process, we can equivalently bound P { ¯ X t ∈

Original: B (0 , R ) for all t ∈ [0 , T ] } .

Original: From the explicit solution of the OU process, ¯ X t = exp( − t ) ¯ X 0 + √ 2 (cid:82) t 0 exp( − ( t − s ))

Original: d B s .

Original: Let ˜ T :

Original: = (cid:82) t 0 exp( − 2 ( t − s ))

Original: d s .

Original: By the representation of stochastic integrals as time changes of Brownian motion (see, e.g., (Steele, 2001, Theorem 12.4)), it follows that sup t

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: exp

Original: [Equation]

Original: s )

Original: [Equation]

Original: s

Original: ∥ = sup t ∈ [0 , ˜ T ] ∥ ˜ B t ∥ , where ˜ B is another standard Brownian motion.

Original: Then, we can bound

Original: [Equation]

Original: By Assumption 2, we have P {∥ ¯ X 0 ∥ > R/ 2 } ≤ E [ ∥

Original: [-¯ X 
-]
Revised: [-¯ X 
-]

Original: ∥ 2+ η ] / ( R/ 2) 2+ η = o (1 /R 2 ) .

Original: On the other hand, by (Chewi et al., 2021b, Lemma 23) applied to λ = 1 / (4 ˜ T ) ,

Original: P (cid:8) sup t ∈ [0 , ˜ T ] ∥

Original: [Equation]

Original: The lemma follows from a union bound over the two events.

Original: Lemma 25 (effect of truncation on the moment bound) .

Original: Under Assumption 2, we have

Original: [Equation]

Original: Proof.

Original: Consider a coupling of the laws of ( X t ) t ∈ [0 ,T ] and ( X tr t ) t ∈ [0 ,T ] such that if X t ′ ∈ B (0 , R ) for all 0 ≤ t ′ < t , then X tr t = X t .

Original: Denote the complement of this event by E t .

Original: Then we have

Original: [Equation]

Original: [-Because ϕ R ( X tr t ) = 
-]
Revised: [-Because ϕ R ( X tr t ) = 
-]

Original: if ∥ X tr t ∥ ≥ 2 R , we can bound the latter term via H¨older’s inequality by (2 R

Original: [Equation]

Original: [Equation]

Original: P [

Original: [Equation]

Original: o (1) , from Lemma 24.

Original: As this tends to 0 as R → ∞ , the lemma follows.

Original: Lemma 26 (effect of truncation on the score function bound) .

Original: Under Assumptions 1 and 2,

Original: [Equation]

Original: for any t ∈ [0 , T ] .

Original: Proof.

Original: Consider the same coupling of the laws of ( X t ) t ∈ [0 ,T ] and ( X tr t ) t ∈ [0 ,T ] from the proof of

Original: Lemma 25, and let E t again denote the event

Original: that X t ′ ∈ B (0 , R ) for all 0 ≤ t ′ <

Original: t .

Original: Then we have

Original: E [

Original: [Equation]

Original: X tr

Original: [Equation]

Original: ∥ 2 ] ≤ E [ ∥∇ ln q

Original: [Equation]

Original: −

Original: [Equation]

Original: ( X

Original: [Equation]

Original: ∥

Original: [Equation]

Original: ∥∇ ln q

Original: [Equation]

Original: −

Original: [Equation]

Original: ( X tr

Original: [Equation]

Original: ) ∥ 2 1 E t ] .

Original: [-Because ϕ R ( X tr t ) = 
-]
Revised: [-Because ϕ R ( X tr t ) = 
-]

Original: if ∥ X tr t ∥ ≥ 2 R , we can bound the latter term via H ¨ older’s inequality by

Original: [Equation]

Original: By Assumption 1, if ∥ x ∥ ≤ 2 R , then ∥∇ ln q T − t ( x ) ∥ 2 ≲ ∥∇ ln q T − t (0) ∥ 2 + L 2 R 2 , so the above quantity is bounded via Lemma 24 by o (1) as R → ∞ .

Original: Lemma 27 (movement bound for modified process) .

Original: Suppose that Assumptions 1 and 2 hold.

Original: Then for 0 ≤ s < t satisfying t − s ≤ h , if h ≲ 1 /L , then

Original: [Equation]

Original: Proof.

Original: In the proof

Original: [Equation]

Original: we drop the subscript Q tr , ← T .

Original: We can write

Original: [Equation]

Original: Taking the lim sup on both sides as R → ∞

Original: [Equation]

Original: we can apply Lemmas 25 and 26 to get

Original: [Equation]

Original: The claimed bound then follows from the moment and score function bounds of Lemma 11.