Score-Based Generative Models Detect Manifolds
Abstract
Score-based generative models (SGMs) need to approximate the scores ∇ log p t of the intermediate distributions as well as the final distribution p T of the forward process.
The theoretical underpinnings of the effects of these approximations are still lacking.
We find precise conditions under which SGMs are able to produce samples from an underlying (low-dimensional) data manifold M .
This assures us that SGMs are able to generate the “right kind of samples”.
For example, taking
M to be the subset of images of faces, we find conditions under which the SGM robustly produces an image of a face, even though the relative frequencies of these images might not accurately represent the true data generating distribution.
Moreover, this analysis is a first step towards understanding the generalization properties of SGMs: Taking M to be the set of all training samples, our results provide a precise description of when the SGM memorizes its training data.
Introduction
Score-based generative models, also called diffusion models ([Sohl-Dickstein et al., 2015, Song and Ermon, 2019, Song et al., 2021b, Vahdat et al., 2021]) and the related models ([Bordes et al., 2017, Ho et al., 2020, Kingma et al., 2021]) have shown great empirical success in many areas, such as image generation ([Jolicoeur-Martineau et al., 2021, Nichol and Dhariwal, 2021, Dhariwal and
Nichol, 2021, Ho et al., 2022]), audio generation ([Chen et al., 2021, Kong et al., 2021, Jeong et al., 2021, Popov et al., 2021]) as well as in other applications ([Batzolis et al., 2021, De Bortoli et al., 2021, Zhou et al., 2021, Cai et al., 2020, Luo and Hu, 2021, Meng et al., 2021, Saharia et al., 2021, Li et al., 2022, Sasaki et al., 2021]).
Recently some progress has been made to bridge the gap between the different approaches ([Song et al., 2021b, Huang et al., 2021]) through the framework of SDEs and reverse SDEs.
In generative modelling one is given samples { x i } Ni =1 from a measure µ data .
The task is to learn a measure µ sample which approximates µ data .
The performance of a generative model can then be measured by the distance from µ sample to µ data .
In practice however, the true data generating distribution µ data is unknown.
All that is known are the samples { x i } ni =1 , which can be used to define the empirical measure ˆ µ data ,
[Equation]
Any sample from the empirical measure ˆ µ data will be equal to a training example.
Hence, while µ sample being close to µ data is the final goal, µ sample being close to ˆ µ data implies that the generative model has memorized the training data.
To summarize, a good generative model will output a measure µ sample which is as close to µ data as possible, while keeping some distance from ˆ µ data , even though it only knows µ data through ˆ µ data .
Given a target measure π a score-based generative model (SGM) employs two stochastic differential equations (SDEs).
The first one is called the forward SDE
[Equation]
The marginals of X t are denoted by π t .
The forward SDE is run until some terminal time T .
Furthermore, the reverse SDE is defined by
[Equation]
We refer to the marginals of Y t as q t .
The samples are generated from the final distribution q T , i.e. µ sample :=
q T .
The reverse SDE has the property that if q 0 is chosen to be equal to π T , then q t = π T − t .
In particular, this implies that q T = π 0 .
Therefore, if we have samples from π T , we can run the reverse SDE on them to create new samples from π 0 .
In the following we will denote by p t the marginals of the forward SDE when started in the true data generating distribution π 0 = µ data .
We will denote by ˆ p t the marginals of the forward SDE when started in π 0 = ˆ µ data .
Optimally, we would like to run the algorithm using π 0 = µ data , i.e. with marginals π t = p t .
This is however not possible, since µ data itself is unknown.
To circumvent the problem of not knowing p T , the forward SDE is chosen such that it forgets its initial condition p 0 .
At time T , the marginal p T is then well approximated by a proxy distribution µ prior ≈ p T , independently of p 0 .
Additionally, the marginals p t and therefore the scores ∇ log p t cannot be evaluated for p {+
+} = µ data .
Therefore, the scores are replaced by a neural network s θ ( x, t ) , which is trained via score-matching techniques [Vincent, 2011, Hyvärinen and Dayan, 2005].
As a result, SGMs make two approximations.
The first one is in approximating p T by µ prior .
The second one is the approximation of ∇ log p t by the neural net s θ ( x, t ) .
We illustrate this in Figure 1.
It is important to understand how these approximations translate to the distance of µ sample to µ data or ˆ µ data .
Some early works already deal with these questions.
In [De Bortoli et al., 2021] the total variation distance between µ sample and µ data in bounded, whereas [Song et al., 2021a] derives bounds with
(a) Here µ data is chosen as the uniform distribution on the unit sphere S 1 .
We use the Brownian motion for the forward SDE.
We can compute the exact score ∇ log p t ( x ) .
We perturb it with the vector v = ( x = 0 , y = − 1) and define the approximation s ( x, t ) =
∇ log p t ( x )+ v .
Furthermore, we purposely adopt a poor approximation µ prior ̸≈ p 1 by setting µ prior =
N ( m, I ) , where m = ( x = 1 . 5 , y = 0) .
We then run the reverse SDE (8).
The heat maps intermediate t the reverse
At time t = we reach q µ We not
Furthermore, we can observe conditions where chosen drift
(b) The above experiment is repeated but with µ data chosen to be the uniform distribution on M = { x i } 9 i =1 , where the x i are 9 evenly spaced points on the unit sphere S 1 .
Notice that while the approximation errors cause a non-uniform distribution the training examples, µ sample is still supported solely on M and will not generate novel samples.
respect to the KL-Divergence.
The work [De Bortoli et al., 2021] furthermore derives a result similarly to the second part of Theorem 1, but also treating the errors that are introduced by discretizing the
SDE.
However, both of these works assume that the initial distribution µ data is rather well behaved. In particular, it is assumed that µ data ( x ) > 0 for all x . We shortly discuss this assumption now. Assuming that µ data ( x ) > 0 for any x means that one postulates that every x is a possible sample from µ data .
For example, this implies that even if all samples { x 1 , . . .
, x n } of µ data consist of images of human faces, we say that µ data can possibly also generate images of for example furniture, animals or pure white noise.
Even though it might have low probability, any combinations of pixels is a possible sample from µ data .
The assumption that µ data is actually supported on some lower dimensional substructure M is well known under the name manifold hypothesis , see for example [Bengio et al., 2013, Pope et al., 2021].
In practice this means that µ data ( x ) = 0 for many x .
This also leads to exploding scores ∇ log p t as t → 0 (see Section 5), a behaviour which also observed empirically and further underpins the relevance of the manifold assumption.
We will from now on denote the support of µ data as data manifold M .
A fundamental question is then: What is the support of µ sample and how does it compare to M .
This is interesting for multiple reasons.
On the one hand, if we for example assume that M is the set of all images of faces, then the knowledge that µ sample also has support M implies that, regardless of how close µ sample actually is to µ data , it will at least always produce an image of a face.
On the other hand, we can also compare the support of µ sample to that of ˆ µ data .
The measures µ sample and µ data sharing their support translates to µ sample memorizing the training data and not being able to generalize.
Both of these are very valuable insights into the qualities of µ sample on its own.
Furthermore, statistical distances like the KL-Divergence or the total variation distance are only meaningful if the support of the measures overlap to some degree, as otherwise they will equal the maximum distance value.
If two measures have the same support, they are said to be equivalent .
Our main result is the following:
Main result.
We identify conditions under which µ sample is able to learn the data manifold, that is supp ( µ sample ) =
supp ( µ data ) .
Applying these results to different settings, we find precise conditions under which an SGM memorizes its training data or under which it is able to learn the right data manifold M .
In particular we find that for SGMs to be able to generalize, the approximation error made when approximating the training drift ∇ log ˆ p t has to be unbounded.
A first illustration of these results is given in Figure 2.
We use a simple example, where we can perfectly evaluate the true drift ∇ log π t .
We then choose an incorrect initial condition µ prior which is far from p T , and also add a constant error to ∇ log π t ( x ) .
The initial measures π 0 are given as the uniform distribution on the unit sphere and the uniform distribution on 9 samples from the unit sphere in Figure 2a and Figure 2 respectively.
We see that the final distribution of the reverse SDE, µ sample , is not the uniform distribution anymore.
This is due to errors in the initial conditions and the drift.
Nevertheless, µ sample is still supported on the exact same subset as π 0 .
When applying our main result to the empirical measure ˆ µ data one gets the following corollary, supplying precise conditions under which a SGM has memorized its training data.
Corollary 1.
Denote by ˆ X t the forward SDE when started in the empirical measure π 0 = ˆ µ data .
Let (cid:82) T 0 ∥
[Equation] ˆ [Equation] − ∇ log ˆ p t ( ˆ X t )d t be drift approximation error along a path of the forward SDE.
For a given weighting function w ( t ) , the training objective of an SGM can be written as
[Equation]
see Section 3.
Simultaneously, if the exponential integral of the drift approximation error is integrable in the following sense:
[Equation]
the SGM has memorized its training data.
Therefore, while training an SGM one should aim to minimize the mean squared error 3 while ensuring that the mean exponential error stays infinite,
L exp = ∞ .
In particular, if ∥ s θ ( ˆ X t , t ) − ∇ log ˆ p t ( X t ) ∥ is bounded, then L exp is finite.
Therefore, the the generalization capability of a SGM crucially depends on the training error being unbounded.
We now proceed as follows.
In Section 2 we will summarize some of the most popular forward SDEs that are applied in SGMs.
Then, in Section 3 we discuss how the drift approximation s θ ( x, t ) for SGMs is trained in most implementations.
We are ready to state our main results in Section 4.
In Section 5 we show how the empirically observed drift explosion is related to the the manifold hypothesis.
Most of the paper discusses the error in the drift approximation.
But as we have discussed, we also have an error in the initial conditions.
In Section 6 we will discuss how large the error in the initial conditions will be in practice.
Popular SDEs used in SGMs
In this section we introduce some of the most popular SDEs that are used when implementing
SGMs.
The first works on SGMs studied discrete forward and backward processes.
Nevertheless, the transition kernels and algorithms proposed in those works can be seen as discretisations of some well-known SDEs.
More recent works have studied this connection and state the algorithms in terms of SDEs ([Song et al., 2021b, Huang et al., 2021]).
Brownian Motion: The works [Song and Ermon, 2019, 2020] can be seen as a discretization of the
SDE
[Equation]
Denoting h ( t ) =
(cid:82) t 0 σ ( s )d s , the solution to the above process can be explicitly stated as a timechanged Brownian motion, X t = W h ( t ) .
The time-change can help in the implementation but does not alter the qualitative behaviour of the reverse SDE.
In our following analysis we therefore set σ ( t ) = 1 .
Nevertheless, our results still hold for any positive σ ( t ) .
Ornstein-Uhlenbeck Process: The works [Sohl-Dickstein et al., 2015, Ho et al., 2020] can be seen as a discretization of
[Equation]
which is an Ornstein-Uhlenbeck process.
Again, the parameters α t are a time-change and do not influence the properties that we are investigating in this paper.
Therefore, to simplify notation, we again set α ( t ) = 1 .
Critically Damped Langevin Dynamics (CLD): The work [Dockhorn et al., 2021] studies a second order SDE.
Here artificial velocity coordinates V t are introduced and the system under consideration is
[Equation]
where X 0 ∼ p data , V 0 ∼ N (0 , I ) .
For generation one runs the reverse SDE in X t and V t but discards the V coordinate at the end.
The work [Dockhorn et al., 2021] also includes the parameters M and γ .
We set both to 1 as they do in their numerical experiments.
Score approximation with a finite number of samples
We now quickly discuss how the neural network is trained to approximate ∇ log p t and what implications this has.
The score ∇ log p t is approximated by minimizing a variant of
[Equation]
for some weighting function w .
The optimization is done via score matching techniques (see [Hyvärinen and Dayan, 2005, Vincent, 2011, Song et al., 2020]).
However, the above expectation depends on p t , which we cannot evaluate since it depends on p 0 = µ data .
Nevertheless, we can evaluate the approximation ˆ p t ,
[Equation]
The surrogate loss
[Equation]
can be evaluated and is used for training.
The equation (7) is equivalent to (3) since ˆ X t has distribution ˆ p t .
If we would minimize this loss perfectly, then s θ ( x, t ) would be equal to ∇ log ˆ p t .
The reverse SDE started in an appropriate initial condition with drift ∇ log ˆ p t however, will produce samples from ˆ µ data , which are training examples.
This raises the question in which way or to which degree the loss should actually be minimized.
Effects of the approximations
This section contains our main results.
We first state the assumptions we have to make and then the
Theorems.
4.1 Error in the initial condition
The following assumption is needed for the reverse SDE to be defined even in the case when the initial distribution π 0 has a degenerate support.
In Lemma 1 we will show that all the SDEs from
Section 2 satisfy the Assumption.
Assumption 1.
There is a constant C such that
(i) β is globally Lipschitz, i.e. ∥ β ( x ) − β ( y ) ∥ ≤ C ∥ x − y ∥ .
[Figure]
(ii) β grows at most linearly, i.e. ∥ β ( x ) ∥ ≤ C (1 + ∥ x ∥ ) .
(iii) X t has a density π t ∈ C 1 for every t > 0 and (cid:82) 1 t 0 (cid:82) ∥ x
∥ <R | π t ( x ) | 2 + ∥∇ x π t ( x ) ∥
d x d t < ∞ for any R > 0 and 0 < t 0 ≤ T .
Furthermore, for each S ∈ (0 , T ) and all x, y for which ∥ x ∥ , ∥ y ∥ ≤ N there is a constant C S,N such that (iv) ∇ log π t is locally Lipschitz, ∥∇ log π t ( x ) −∇ log π t ( y ) ∥ ≤ C S,N ∥ x − y ∥ for all t ∈ [ S, T ] .
Conditions ( i ) - ( iii ) are technical conditions on the forward SDE.
They ensure that if we run a solution to the forward SDE, X t , backwards in time, then X
Rt :=
X T − t will be a solution to the reverse SDE (2) on [0 , T ) .
The last condition then ensures that the solutions to the reverse SDE are unique, therefore we will be able to transmit the properties of X R to any other solution Y t of (2).
The following result shows that Assumption 1 can be expected to hold in practice.
To simplify the calculations we assume that the data manifold M = supp ( µ data ) is contained in a ball of diameter M .
This is a natural assumption for many data sets.
Nevertheless, we note that this assumption could be weakened by additional technical effort.
Lemma 1.
Assume that the data manifold M is contained in a ball of radius M .
Then all the methods introduced in Section 2 fulfil Assumption 1.
We are now ready to state our first main result,
Theorem 1.
Denote by π t the marginals of the forward SDE started in π 0 .
Assume that Assumption 1 holds and that µ prior is absolutely continuous with respect to π T .
Then the following hold. (i) Let Y t be a solution to (2) on [0 , T ) .
The limit Y T :=
lim t → T Y T exists almost surely.
We refer to its distribution as µ sample .
The distribution µ sample is absolutely continuous with respect to µ data .
If π T and µ prior are equivalent, then so are µ sample and π 0 . (ii)
Furthermore, for any f -divergence D f , D f ( µ sample | π 0 ) ≤ D f ( µ prior | π T ) and D f ( π 0 | µ sample ) ≤ D f
( π T | µ prior ) .
Applying the above theorem with π T =
p T tells us something about the equivalence and distance between µ sample and µ data , since π 0 = p {+
+} = µ data .
Applying the theorem with ˆ p t tells us something about the generalization capabilities of SGMs.
The measures π T and µ prior are normally both supported on all of R d and therefore equivalent.
Since
Assumption 1 also holds in most situations (see Lemma 1), the requirements for Theorem 1 are satisfied in practice.
From item ( i ) with π t = ˆ p t we can conclude that the error we make in the initial conditions is not responsible for the generalization capacities of SGMs.
The second item then shows that the f -divergences between µ sample and µ data are bounded by the f -divergences of µ prior to p T .
The total variation distance and the KL-Divergence are both special cases of f -divergences.
4.2 Error in the drift
Given a forward SDE with marginals π t and an approximation s ( x, t ) to ∇ log π t , we define the reverse SDE for the approximation s as
[Equation]
Assumption 2.
We assume that the reverse SDE ˜ Y t has a solution on [0 , T ) .
For t < T , we define the Girsanov weights
[Equation]
and assume that the Z t are a uniformly integrable martingale.
We shortly discuss this assumption.
The assumption that Z t is a martingale is equivalent to the expectation of Z t being equal to 1 for all t .
The assumption that it is uniformly integrable is more technical (see Appendix A.2), but is fulfilled for example if for some p > 1 , E [ | Z t | p ] < ∞ for all t .
For example, if the Z t have bounded variance, then they are uniformly integrable.
A condition that ensures that Z t is both, a martingale and uniformly integrable, is given by Novikov’s condition Novikov [1980].
It states, that if
[Equation]
then Z t is a uniformly integrable martingale.
Using Assumption 2 we can now state
Theorem 2.
Assume that Assumption 2 holds.
Assume furthermore that Assumption 1 holds with ∇ log π t replaced by s ( x, t ) .
Then ˜ Y T =
lim t → T ˜ Y t is well defined.
Moreover, its distribution is equivalent to the distribution of
Y T .
In particular, if ∥ s ( x, t ) − ∇ log ˆ p t ∥ is bounded, then Assumption 2 holds, and the SGM has memorized its training data.
Putting Theorem 1 and Theorem 2 together, we can conclude that, if both Assumptions hold, µ sample is equivalent to π 0 .
Therefore, if Assumption 2 holds for π t = p t , we have a positive statement and know that µ sample will have the exact same support as µ data , i.e. it has learned the data manifold M .
If the Assumption would hold for π t = ˆ p t , we would however just memorize the training data, see
Figure 2b or Figure 3b for a visualization.
However, empirically it has been shown that SGMs are able to create novel samples (see, Figure 3a or Dhariwal and Nichol [2021]).
Therefore, we can deduce that Assumption 2 is violated in practice.
We evaluated N from (10) on CIFAR-10, once for the difference between the s θ ( x, t ) from Song et al.
[2021b] and ∇ log ˆ p t , and once by just using a perturbed drift with a constant error, s ( x, t ) = ∇ log ˆ p t + 12 (1 , 1 , . .
. , 1) , see Figure 4.
The reverse SDE using the drift ∇ log ˆ p t + 12 (1 , 1 , . .
. , 1) , is equivalent to ˆ µ data , as we know from Theorem 2 and have also already observed in Figure 3b.
Figure 4 confirms that Z t is indeed a uniformly integrable martingale and therefore fulfils Assumption 2.
Corollary 1 can be deduced by exchanging the roles of ˜ Y t and Y t .
We then get an equivalent condition to Assumption 2, which is
[Equation]
where the expectation is now taken over the reverse SDE with the correct drift ∇ log π t instead of the approximate drift s θ .
However, Y is just the time reversal of X , therefore we can also write the above expectation over X t instead of Y t .
If we treat the case where we start X t in the empirical measure ˆ µ data , π t will be equal to π t = ˆ p t by definition and [Equation] (cid:33)(cid:35) < ∞ is a sufficient condition for ˆ µ data and µ sample having the same support. We have here assumed that
Theorem 1 can be applied.
However, this can be assumed in practice, see the discussion after Theorem 1.
In future work we believe that further understanding the characteristics of Z t , how they relate to the minimization of L ( θ ) and generalization is crucial to the understanding of SGMs and their empirical success.
Recent works also study the question on how the neural network architecture and parametrization is related to the boundedness of the output Kim et al.
The relationship between these choices and the properties of Z t is also an interesting research avenue.
Lastly, the distribution of Z t is very heavy-tailed.
Most of the samples are very small with a few extremely large samples in between.
Therefore one needs many samples from Z t to understand its characteristics.
Finding robust estimators for Z t or its expectation could help their usage in the training or evaluation procedure.
Drift Explosion under manifold hypothesis
In practice it is often observed that the drift of the reverse SDE explodes as t → T .
see for example
Kim et al.
[2022, Section 3.1].
We now show how this observed behaviour is related to the manifold hypothesis.
First, we note that all SDEs in Section 2 are linear SDEs.
Therefore, their transition kernels are
Gaussian ([Pavliotis, 2014, Section 3.7]):
[Equation]
The explicit form of m t and Σ t differ for each of the SDEs and can be found in Appendix C.1.
We remark that Σ t does not depend on the initial condition x 0 .
The transition kernel above is the distribution of the SDE started in a single point x 0 .
Since we start the SDE in µ data we need to average over µ data to get the marginal at time t :
[Equation]
We can also compute the additional drift in the reverse SDE (see Appendix C.2),
[Equation]
We now want to evaluate ∇ log p t along a typical path of Y t , i.e. we are interested in E [ ∇ log p t ( Y t )] .
The distribution of Y t however depends on the drift approximation s θ we use.
For this calculation we will assume that we are able to run the reverse SDE with the true drift s θ ( x, t ) =
p t ( x ) .
Then however, since Y t is then just X t run backwards, they have the same distributions and we can calculate
[Equation]
If the manifold M is not to badly behaved, we can expect that for small t and almost all x , E [ X 0 | X t = x ] to be very close to the data manifold M .
Especially, ∥
X t − E
[ X 0 | X t = x ] ∥ will be larger than dist ( X t , M ) in that case.
However, the distribution of X t can be represented as m t ( X 0 ) + √ Σ t ξ , where ξ has a standard normal distribution.
For the SDEs we treated in Section 2, m t ( X t ) is either equal or very close to m t ( X t ) =
X t for small values of t .
Finally, if we assume that M is a subset of relatively low dimension in a high dimensional space, we can expect that with very high probability √ Σ t ξ points away from the data manifold.
Therefore the distance of X t to M can be approximated by √ Σ t ξ .
Putting these approximations together we can calculate
[Equation]
where d is the dimension of the data space in which the samples x i lie.
Therefore we can conclude that
[Equation]
For the Brownian motion for example, Σ t = t and therefore the right hand side scales like 1 √ t .
If Σ t is the covariance of p t , then we can expect ∇ log p t ( Y t ) to be of order[-
-]
[Equation]
Furthermore, ∇ log p t ( Y t ) will point towards the data manifold for small t .
The drift ∇ log p t ( Y t ) then acts like a support matching force , where the force grows to infinity as t → 0 , absorbing all the SDE paths onto the manifold.
Distance from p T to µ prior
We have seen in Theorem 1 that the distance between µ sample and µ data is directly related to the distance between p T and µ prior if we neglect the errors made in the approximation of the drift.
For both, the OU-Process and the CLD, there are plenty of results on the distance of p t to N (0 , I d ) .
In general, expect distance to grown time t .
The Brownian motion however does not converge to a stationary distribution and therefore one has to choose a different µ T prior for each T to approximate p T .
In practice, one normally chooses a normal distribution p T = N ( m t , C t ) (see [Song et al., 2021b, Appendix C]).
The following Lemma is derives the optimal values for m t and C t .
Lemma 2.
Let p T be the T -time marginal of the Brownian motion process X t of Section 2.
The following minimization problem
[Equation]
is minimized by m T
= E [ µ data ] and C T =
Cov[ µ data ] + TI d .
If we restrict the covariance to be a multiple of the identity matrix, the problem is solved by choosing m as above and c as c = E [ ∥ X t − m ∥ 2 ] = trace ( C T ) .
This result is a slight variation on the well known fact that the KL -projection in the second argument matches its moments.
We prove it in Appendix E.2.
The following result shows that the distance between p T and µ T prior decreases with time and also gives a rate.
It justifies using the Brownian motion and a normal prior distribution for SGMs.
Lemma 3.
Let p t be the time t -marginal of a Brownian motion with initial condition µ data .
Denote by c i , i = 1 , . . .
, d, the eigenvalues of Cov( µ data ) .
Let µ T prior be the normal distribution with mean m T
= E [ µ data ] and covariance C T =
Cov[ µ data ] + TI d .
Then
[Equation]
The proof can be found in Appendix E.2.
Broader impact
The results deepen the understanding of score-based generative models.
As such, they can be seen as a step towards improving the quality of generative models.
Therefore the possible negative societal impacts are the same ones that apply to generative modelling in general.
First, generative models can be used to create synthetic data that is hard to distinguish from real data (for example images or videos), see [Mirsky and Lee, 2021].
Second, generative models can learn and reproduce biases that are prevalent in the training data ([Esser et al., 2020]).
Last, depending on the application, generative models might be used to do creative work that was previously done by humans.
Conclusion
We conducted a theoretical study of some properties of SGMs.
We found explicit conditions under which the sample measure µ sample is equivalent to the true data generating distribution µ data .
Under these conditions we can guarantee, that the SGM generates samples that could also be samples from µ data .
Furthermore, each sample that can be generated by µ data also has positive probability under µ sample , meaning that the full support is covered.
Since one can not actually access the full support of µ data , but only a finite number of training examples { x i } Ni =1 , our results can be applied to find conditions under which the SGM memorizes its training data.
We believe that this observation provides a first step towards understanding the generalization capabilities of SGMs.  Funding
The author has been partially supported by Deutsche Forschungsgemeinschaft (DFG) - Project-ID 318763901 - SFB1294.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] The assumptions are followed by a discussion of their strength.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 7. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] There are no error to report, we just have small illustrative numerical experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
We did not use any existing assets.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing or conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Stochastic prerequisites
In this section we give a formal introduction to some of the concepts used in this work.
For a more rigorous treatment, see for example [Klenke, 2013] or [Karatzas and Shreve, 2012].
A.1 Equivalence of measures / Girsanov Theorem
First we define absolute continuity of measures.
Let µ and ν be two measures on (Ω , F ) , where F is a σ -algebra.
Definition 1.
We say that µ is absolutely continuous with respect to ν if µ ( A )
=
for any A ∈ F such that ν ( A ) = 0 .
We also denote this by µ ≪ ν .
Two measures µ and ν are equivalent if µ ≪
ν
and ν ≪ µ .
Loosely speaking, we can say that µ ≪ ν if the support of µ is contained in the support of ν
and they are equivalent if they share the same support.
The Radon-Nikodym theorem tells us that if µ ≪ ν , then under mild conditions there exists a density d µ d ν : Ω → R such that µ ( A ) = (cid:82)
A d µ d ν ( ω )d ν ( ω ) .
Therefore, we can obtain µ through a reweighting of ν .
One specific instance of this is the Girsanov Theorem.
Assume we are given the solutions to two SDEs in R d ,
[Equation]
[Equation]
and
[Equation]
Both of these induce a measure on the space of continuous functions Ω = C
([0 , T ] , R d ) .
We denote them by P and ˜ P respectively.
Then the Girsanov Theorem equips us with conditions under which the measures P and ˜ P are equivalent.
Furthermore, in case of equivalence we get a formula for the density of ˜ P with respect to P .
The relative density is given as
[Equation]
For a full statement of the Girsanov Theorem and under which conditions it holds, see [Karatzas and
Shreve, 2012, Section 3.5].
A.2 Uniform integrability
Since we are treating the case where the drift explodes as t → T we end up with densities
[Equation]
on C ([0 , t ] , R d ) , but not with a density on C ([0 , T ] , R d ) .
Uniform integrability is exactly the condition one needs to extend these local densities.
Definition 2.
A family { X α } of random variables is called uniformly integrable if
[Equation]
as s → ∞ .
In the proof of Theorem 2 we implicitly use the following two results which we here state as a lemma.
The filtration F t is defined as in the proof of Theorem 2.
Lemma 4.
Assume the Z t in (15) form a uniformly integrable martingale on [0 , T ) .
Then,
• the limit lim t → T Z t exists in L 1 .
We denote this limit by Z .
• Furthermore, ˜ P is absolutely continuous with respect to P on F = σ ( ∪ t<T F t ) with density Z .
Proof.
Both of these results are standard.
The first one can for example be found in [Karatzas and
Shreve, 2012, Section 1.3.B].
For the second one we compute that for any A ∈ F s ,
[Equation]
where we used L 1 convergence in the second equality and the martingale property of Z s in the third equality.
Therefore Z is a density of ˜ P with respect to P on each F s for s < T .
Therefore Z is also a density of ˜ P with respect to P on F which concludes the proof.
B Numerics
All numerical experiments can be run on a consumer grade computer within a few minutes.
B.1 Figure
We first discuss the top left figure of Figure 1.
We set p
= µ data to a mixture of two Gaussian
N ( − 2 , 1100 ) and N (2 , 1100 ) with weights w 1 = 13 and w 2 = 23 respectively.
Then, we draw N = 5 000 000 samples from µ data , denoted by Y n 0 , n = 1 , . . .
, N .
An Euler-Maruyama discretization of the Brownian motion propagates these samples from time t = 0 to t = 1 by
[Equation]
where Z in ∼ N (0 , 1) are i.i.d.
random variables, independent of X jm for m ≤ n and j = 1 , . .
. , N .
The time index i runs from 0 to I = 2000 and dt is set to dt =
I .
The initial samples { X n 0 } Nn = are used to create the left line plot of p 0 and the final samples { X nI } Nn =1 are used to create the right line plot of p 1 using kernel density estimation.
The { X ni } Nn =1 are approximate samples from p
i/I .
Therefore, we create histograms using { X ni } Nn =1 to approximate p i/I .
The height of the histogram bars corresponds to the square root of the colour intensity in the heat map.
The horizontal axis in the heat map stands for the time t , whereas the vertical axis stands for the position x .
At location ( t, x ) we plot an estimate of (cid:112) p t ( x ) .
We apply the square root since it improves the contrast in areas where p t ( x ) is close to 0 and makes it more visible where p t ( x ) > 0 to the observer.
For the bottom left figure we show the same plots, just for the reverse SDE (8) instead of the forward
SDE.
Since the initial distribution is a Gaussian mixture we can exactly calculate p t using
[Equation]
where we use N ( x ; m, v ) for the probability density function of a normal distribution with mean m and variance v , evaluated at x .
With the above expression of p t one could compute an analytical representation of ∇ log p t .
We use automatic differentiation instead.
The reverse SDE (8) is simulated with a disturbance e ( x, t ) = 1 and initial condition q 0 = µ prior =
N (0 , 1) .
The Euler-Maruyama method is run with the same step size dt =
I .
More precisely, the one step transition kernel of the discretized reverse SDE is
[Equation]
(17) where ˜ Z in ∼ N (0 , 1) are i.i.d.
random variables, independent of Y jm for m ≤ n and j = 1 , . .
. , N .
The plots are created in the same way as for the upper left plot, except that we reverse the time axis to plot p t and q 1 − t directly underneath each other.
On the right side we plot the same kernel density estimates already plotted on the left side as µ sample and µ data into the same plot for comparison.
B.2 Figure 2b and 2a
Figure 2b is created by setting µ data to be the uniform distribution on M = 9 equally spaced samples { x i } Mi =1 on the unit sphere S 1 .
This can also be viewed as a Gaussian mixture with 9 components where each component having mean x i and variance 0 .
Therefore, we can again explicitly calculate p t for t > 0 as in (16),
[Equation]
m
[Table]
z
[Table]
The score ∇ log p t ( y ) is evaluated using automatic differentiation.
The reverse SDE 8 is simulated with q 0 = N (( x = − 1 .
5 , y = 0) , I 2 ) and e (( x, y ) , t )
= ( x = 0 , y = − 1) .
For the numerical simulation, we again use the Euler-Maruyama scheme.
We use a step width of dt = 0 .
9 1000 ≈ 1for t
=
[0 , 0 . 9] , dt = 0 .
09 1000 ≈ 1 10 000 for t ∈ [0 . 9 , 0 . 99] and of dt = 1 100 000 for t ∈ [0 . 99 , 1] .
For a simulation of N = 50 000 paths of the reverse SDE, we start by drawing Y n 0 =
A n + Z n , where
A n are i.i.d.
uniformly distributed on { x i } Mi =1 and Z n ∼ N (0 , 1) i.i.d..
The { A n } and { Z n } are also independent from each other.
We then propagate the Y n 0 similarly to (17), except that we use a different values for dt depending on t .
This leads to approximate samples Y it from q t .
At the displayed times t we plot the function h t ( x ) =
N (cid:88) i
= k ( x, Y it ) , where k is an unnormalized Gaussian kernel with a very small bandwidth parameter,
[Equation]
Normalizing h t gives us a density estimate of q t .
We plot these estimates as heat maps for different values of t .
For Figure 2a we follow the same steps as for Figure 2b, except that µ data is set to the uniform distribution over M = 256 evenly spaced samples from the unit sphere S 1 .
B.3 Figure
For Figure 3 we used the DDPM++ model from the Github repository for the paper from Song et al. [2021b].
Then we evaluated the true score using (6), where the sum runs through the N =training examples of CIFAR-10, Krizhevsky et al.
A similar experiment using the true marginals ˆ µ data has also been conducted in Peluchetti [2021].
C Studying the forward Densities C.1 Transition kernels
The transition kernels p ( z 0 , · ) for the SDEs from Section 2 are of the from
[Equation]
where m t and Σ t are given in Table 1 for the Brownian Motion and the Ornstein-Uhlenbeck process.
The form of the transition kernels for the CLD are more involved.
They can be found in Dockhorn et al.
[2021, Appendix B.1].
Lemma 5.
The marginal densities p t ( x ) of the SDEs treated in Section 2 depend smoothly on x and t .
Proof.
One can combine the form of m t and Σ t and the explicit representation of p t in (11) to see this.
More generally, for the Brownian motion and the OU-Process this is a result of the Hörmander theorem.
For the CLD it is a result of hypocoercivity.
C.2 Form of the drift
We now prove that we can represent the drift as in (12).
Lemma 6.
Assume that p t has the form (11) , i.e.
[Equation]
Then (12) holds true, i.e.
[Equation]
Proof.
[Equation]
If we now divide everything by p t it cancels in the first summand.
In the second summand we get the formula for the conditional expectation (see, for example [Klenke, 2013, Section 8.2]).
D Reverse SDEs: The general case
One can also treat more general forward SDEs than we did in Section 1.
This leads to a more complicated form of the reverse SDE.
Our Theorems do not use the specific form of the forward
SDEs and therefore also hold in the general case.
We denote the forward SDE by
[Equation]
where µ data is supported on M ⊂ R d and W t is a R r valued Brownian motion.
The drift b maps from R × R d to R d .
The dispersion coefficient σ maps from R × R d to the d × r -matrices.
The time-reversed process Y t :=
X T − t is then a solution to
[Equation]
with
[Equation]
and q 0 = p T , see Haussmann and Pardoux [1986].
This simplifies to the case discussed in Sectionif r = d , σ is is a multiple of the identity matrix and β is independent of the time t .
In Assumptionwe treat the case where the SDEs are of the form written-out in Section 1.
The items ( i ) − ( iii ) need to be replaced by their more general counterpart as found in Haussmann and Pardoux [1986, Section 2].
In the last item ( iv ) , ∇ log p t needs to be replaced by (cid:80) j ∇ j ( a ij ( T − t,y ) p T − t ( y ))
p T − t ( y ) .
E Proofs E.1 Proofs of the theorems
We now give proofs of our main results and briefly summarize the key steps in an intuitive way.
In our study we would like to include the case when µ data is degenerate and supported on a low-dimensional substructure M .
As we have seen in Section ??
, this can lead to an exploding drift in the reverse
SDE as t → T .
Nevertheless, in order to understand the properties of µ data , it is crucial to study the properties of solutions to the reverse SDE at time t = T .
This is where the main mathematical difficulties come from.
The proofs are mostly independent of the specific form of the forward SDE and hold for more general forward/backward SDEs than those stated in Section 1, see Appendix D.
E.1. 1 Theorem
We now proceed with proving Theorem 1.
Proof.
Let P be the measure on Ω = C ([0 , T ] , R n ) induced by the forward SDE (1) started in p 0 = µ data .
P has marginals p t .
Denote by X t the canonical projections X t ( ω ) = ω ( t ) for ω ∈ Ω .
We define Q through
[Equation]
By the data processing inequality we obtain (see [Liese and Vajda, 2006, Theorem 14]),
[Equation]
It remains to prove that by running Q backwards we obtain a solution to (2) started in µ prior .
We denote the generator of the reverse SDE (2) by L .
Denote by Q R and P R the time reversals of Q and
P .
Our assumption are such that P R is a Markov process solving the martingale problem for L (see [Haussmann and Pardoux, 1986, Theorem 2.1]).
A short calculation shows that Q R is still Markov (see, for example [Léonard, 2011, Proposition 4.2]).
Furthermore for f ∈ C ∞ c ( R n ) ,
[Equation]
In the second equality we used the Markov property of P R .
In the last one we used that P R solves the martingale problem for L .
Therefore also Q R solves the martingale problem for L .
Denote by Y t a solution to (2) on [0 , T ) .
Since solutions to (2) are unique in law on [0 , S ] for S < T (see [Karatzas and Shreve, 2012, Section 5.2]) and the solutions are continuous, the law of Y is equal to Q R on [0 , T ) .
But the paths of Q R are continuous on [0 , T ] .
Therefore, Y can be extended to [0 , T ] , i.e. the limit Y T :=
lim t → T Y t exists almost surely and its distribution is equal to the T -time marginal distribution of Q R , which is the 0 -time marginal of Q .
We denote the marginals of Q and P by Q t and P t .
Since Q is absolutely continuous with respect to P , Q 0 = µ sample is absolutely continuous with respect to P 0 = µ data .
Analogously, if µ prior and p T are equivalent, then so are P and Q and therefore P 0 and Q 0 .
This proves ( i
( ii ) is a consequence of the data processing inequality for f -divergences ( Liese and Vajda, 2006, Theorem
, analogous to (
The main idea of this proof is that we look at the forward SDE for X t first.
It induces a distribution P over all continuous paths in Ω = C ([0 , T ] , R d ) .
If we reverse the time direction of this distribution on Ω , we get a solution to the reverse SDE, started in p T .
This reverse solution is well behaved as t → T , since Y T =
X 0 .
The solution for a different initial condition µ prior ̸ = p T is obtained by reweighting P .
This does not change the qualitative behaviour of Y T , which still exists and is well defined.
We then use a uniqueness result to see that any solution of (19) inherits these properties.
E.1.2 Theorem[-
-]
Proof.
Denote the space C
([0 , T ) , R d ) by Ω .
We also define the natural filtration F t = σ ( x ( s ) | s ≤ t ) .
We denote the distribution of Y on Ω by P .
We define ˜ P by reweighting P with Z t on F t .
By Girsanov theorem (see [Karatzas and Shreve, 2012, Section 3.5]) we know that the canonical process under ˜ P is a solution to (8) on [0 , T ) .
Since Z t is uniformly integrable, its limit Z :=
lim t → T Z t exists in L 1 .
Furthermore ˜ P is absolutely continuous with respect to P on F = σ ( ∪ t<T F t )
=
σ ( x ( t ) | t < T ) with density Z .
We define x ( T ) :
=
lim t → T x ( t ) .
Then the event
A = { x ( T ) :=
lim t → T x ( t ) exists and x ( T ) ∈ M} has probability 1 under P (see Theorem 1) and therefore also under ˜ P .
Furthermore, x ( T ) is measurable with respect to F .
Therefore the distributions of x ( T ) under P and ˜ P are equivalent.
The canonical process under ˜ P is therefore a solution of (8), with the property that its time T -marginal is well defined and equivalent to the time T -marginal of (2).
We can use uniqueness in law on [0 , S ) for any S < T and extend it to [0 , T ] as in the proof of Theorem 1.
This shows that every solution to (8) has the desired properties.
Finally we show that if e is bounded, it fulfils Assumptions 2.
We define by H t = (cid:82) t
∥
e ( s, ˆ Y s )
∥
d s .
Then there is a Brownian motion W t such that we can write Z t as
[Equation]
Since e is bounded by M , H t is bounded by TM .
In particular, one can view Z t =
E [ Z TM
|F
H t ] .
Therefore Z t is uniformly integrable since it can be viewed as a family of conditional expectations.
Here we essentially applied the Girsanov Theorem on [0 , T ) .
Using the uniform integrability of the
Girsanov weights Z t , we are able to extend it to [0 , T ] .
Therefore, we can infer that the distribution of Y and ˆ Y are actually equivalent on the whole path space C ([0 , T ] , R d ) .
In particular, their time
T -marginals will be equivalent too, which is the claim of the theorem.
E.2 Proof of the Lemmas
We start by proving Lemma 1.
Proof.
The forward drifts are β ( x ) = 0 , β ( x ) = − α 2 x and β ( x, v )
= ( v, − x − 2 v ) for the Brownian
Motion, the OU-Process and the Critically Damped Langevin Dynamics respectively.
In particular, these are all linear maps and therefore fulfil conditions ( i ) and ( ii ) of Assumption 1.
We show in Appendix C.1 that log p t is C ∞ in t and x for t > 0 .
Therefore we can integrate p t and its derivative over compact sets, implying that condition ( iii ) holds.
Furthermore, the Hessian w.r.t.
( x, t ) is continuous and obtains its maximum and minimum on the compact set [ S, T ] × B N , where B N is the ball of diameter N around the origin.
Therefore the gradient ∇ log p t is Lipschitz on [ S, T ] × B N , which proves ( iv ) .
We now prove Lemma 2.
Proof.
We have that X 0 ∼ µ data .
Denote the mean and covariance of µ data by a and C respectively.
We define
[Equation]
for some functions m t and V t .
If V t would not have full rank, n t would be a degenerate distribution.
Since p t > 0 almost everywhere for t > 0 , the KL divergence from p t to n t would be infinite. We can therefore restrict V t to be an invertible matrix. We denote the entropy of p by H , H ( p ) = − (cid:82) log( p ) p d x .
We can write X t as X t =
X 0 + √ tZ where Z ∼ N (0 , I d ) .
Now for the KL-divergence at time t it holds that
[Equation]
We compute (cid:90)
[Equation]
We added and subtracted a in the second equality.
In the last equality we used that E [ X T AX ] = tr ( Cov ( X ) A ) if X is a centred random variable.
Substituting this into our prior calculation, we arrive at
KL ( p t | n t )
[Equation]
The mean function m t only appears in the last term.
Therefore, we decrease the KL-divergence by setting m t = a .
We now optimize for the covariance matrix V t .
The H ( p t ) term is independent of V t .
Consequently, to optimize the KL ( p t | n t ) with respect to V t , it is enough to optimize
[Equation]
where we used that det ( A − 1 )
= det ( A ) − 1 to rewrite the loss in terms of V − 1 t .
Since − log(det( · )) is convex on the positive semidefinite matrices and the other summands are linear, L is convex in
V − 1 t .
We take the gradient of L with respect to the Frobenius inner product and obtain
[Equation]
Setting the gradient to 0 has the unique solution
[Equation]
which is a local and global minimum due to the convexity of L .
If we restrict V t to be of the form V t = v t I d , the loss becomes
[Equation]
By setting the derivative of this to zero, we arrive at
[Equation]
We now proceed to prove Lemma 3.
For that we first need the following proposition.
Proposition 1.
Let p t be the time t -marginal of a Brownian motion started in µ data .
Assume that the support of µ data is contained in a ball of radius M .
Let m t =
E [ µ data ] and C t =
Cov [ µ data ] + tI d be the optimal mean and covariance operator from Lemma 2.
Denote n t = N ( m t , V t ) .
Then
KL ( p t | n t ) → 0 as t → ∞ .
Proof.
For t > 0 , the KL-divergence is given by
[Equation]
We used that
[Equation] [Equation]
where c i are the eigenvalues of Cov [ µ data ] .
Now,
[Equation]
We bound
[Equation]
We write X t as X t =
X 0 + √ tZ where Z ∼ N (0 , I d ) .
Then,
[Equation]
We use that
[Equation]
where v 0 =
E [ ∥ X 0 − E [ µ data ] ∥ 2 ] = tr ( C 0 ) .
Since Z is centred and independent of X t , the cross-term E [ ⟨ X t , Z ⟩ ] = 0 vanishes.
We now take the expectation over the right hand side of (23) to obtain
[Equation]
Putting it all together we have that
[Equation]
as t → ∞ .
We are now ready to prove Lemma 3.
Proof.
We again use (21).
We now take the derivative of the KL-divergence using De Bruijn’s identity for multivariate random variables (see [Rioul, 2010], [Costa and Cover, 1984]):
[Equation]
where we applied Lemma 6.
Since E [ X 0 | X t ] is the L 2 -orthogonal projection of X 0 to the σ ( X t ) - measurable random variables and X t is σ ( X t ) -measurable [Equation] we have that
[Equation]
For A t =
E [ ∥ X 0 − E
[ X 0 | X t ] ∥ 2 ] it holds that
[Equation]
since the conditional expectation is a contraction in L 2 .
We use (22) again to arrive at
[Equation]
where c i are the eigenvalues of Cov [ µ data ] + tI d .
Integrating the right hand side we obtain
[Equation]
as a possible antiderivative.
Define g ( t ) =
h ( t ) − KL ( p t | n t ) .
Let us now assume that the support of µ data is contained in a ball of radius M .
Then, by Proposition 1, lim t →∞ KL ( p t | n t ) =
lim t →∞ h ( t ) =
and therefore lim g ( t ) → 0 .
Equation (25) implies that g ′ ( t ) ≤ 0 for all t > 0 .
Assume that there is a s and an ϵ > 0 such that g ( s ) ≤ − ϵ < 0 .
Since g ′ ( t ) ≤ 0 for all t > 0 we can then conclude that g ( t ) ≤ − ϵ < 0 for all t ≥ s .
In particular, g ( t ) ̸→ 0 as t → ∞ , which is a contradiction.
Therefore the statement of the Lemma holds for µ data .
Now let µ data be any measure.
Let X be a random variable that is distributed according to µ data and
Z be a standard normal random variable, Z ∼ N (0 , I d ) .
Then X + √ tZ has distribution p t .
We define X M :
= X 1 {∥ X ∥≤ M } and X Mt = X M + √ tZ .
Denote the distribution of X Mt by p Mt .
The distribution of X M is supported on a ball of radius M .
Furthermore, X M + √ tZ → X + √ tZ almost surely and therefore p Mt → p t weakly for any t .
By lower semi-continuity of the KL-divergence,
[Equation]
where c Mi are the eigenvalues of Cov [ µ M data ] + tI d .
Since | X M | ≤ | X | and X has first and second moments one can apply dominated convergence to see that the covariance matrices converge,
Cov ( µ M data ) → Cov ( µ data ) .
Especially
[Equation]
Combining this with (26) concludes the proof.