Score-Based Generative Models Detect Manifolds
Anonymous Author(s)
AffiliationAddress email
Abstract
Score-based generative models (SGMs) need to approximate the scores ∇ log p tof the intermediate distributions as well as the final distribution p T of the forwardprocess.
The theoretical underpinnings of the effects of these approximations arestill lacking.
We find precise conditions under which SGMs are able to producesamples from an underlying (low-dimensional) data manifold M .
This assures usthat SGMs are able to generate the “right kind of samples”.
For example, taking
M to be the subset of images of faces, we find conditions under which the SGMrobustly produces an image of a face, even though the relative frequencies ofthese images might not accurately represent the true data generating distribution.
Moreover, this analysis is a first step towards understanding the generalizationproperties of SGMs: Taking M to be the set of all training samples, our resultsprovide a precise description of when the SGM memorizes its training data
Introduction
Score-based generative models, also called diffusion models ([Sohl-Dickstein et al., 2015, Songand Ermon, 2019, Song et al., 2021b, Vahdat et al., 2021]) and the related models ([Bordes et al.
,2017, Ho et al., 2020, Kingma et al., 2021]) have shown great empirical success in many areas, suchas image generation ([Jolicoeur-Martineau et al., 2021, Nichol and Dhariwal, 2021, Dhariwal and
Nichol, 2021, Ho et al., 2022]), audio generation ([Chen et al., 2021, Kong et al., 2021, Jeong et al.
,2021, Popov et al., 2021]) as well as in other applications ([Batzolis et al., 2021, De Bortoli et al.
,2021, Zhou et al., 2021, Cai et al., 2020, Luo and Hu, 2021, Meng et al., 2021, Saharia et al., 2021, Liet al., 2022, Sasaki et al., 2021]).
Recently some progress has been made to bridge the gap betweenthe different approaches ([Song et al., 2021b, Huang et al., 2021]) through the framework of SDEsand reverse SDEs.
In generative modelling one is given samples { x i } Ni =1 from a measure µ data .
The task is to learna measure µ sample which approximates µ data .
The performance of a generative model can thenbe measured by the distance from µ sample to µ data .
In practice however, the true data generatingdistribution µ data is unknown.
All that is known are the samples { x i } ni =1 , which can be used to definethe empirical measure ˆ µ data ,
[Equation]
Any sample from the empirical measure ˆ µ data will be equal to a training example.
Hence, while µ samplebeing close to µ data is the final goal, µ sample being close to ˆ µ data implies that the generative model hasmemorized the training data.
To summarize, a good generative model will output a measure µ samplewhich is as close to µ data as possible, while keeping some distance from ˆ µ data , even though it onlyknows µ data through ˆ µ data .
Given a target measure π 0 , a score-based generative model (SGM) employs two stochastic differentialequations (SDEs).
The first one is called the forward SDE
[Equation]
The marginals of X t are denoted by π t .
The forward SDE is run until some terminal time T .
Furthermore, the reverse SDE is defined by
[Equation]
We refer to the marginals of Y t as q t .
The samples are generated from the final distribution q T ,i.e. µ sample :=
q T .
The reverse SDE has the property that if q 0 is chosen to be equal to π T , thenq t = π T − t .
In particular, this implies that q T = π 0 .
Therefore, if we have samples from π T , we canrun the reverse SDE on them to create new samples from π 0 .
In the following we will denote by p t the marginals of the forward SDE when started in the true datagenerating distribution π 0 = µ data .
We will denote by ˆ p t the marginals of the forward SDE whenstarted in π 0 = ˆ µ data .
Optimally, we would like to run the algorithm using π 0 = µ data , i.e. withmarginals π t = p t .
This is however not possible, since µ data itself is unknown.
To circumvent the problem of not knowing p T , the forward SDE is chosen such that it forgets itsinitial condition p 0 .
At time T , the marginal p T is then well approximated by a proxy distributionµ prior ≈ p T , independently of p 0 .
Additionally, the marginals p t and therefore the scores ∇ log p tcannot be evaluated for p 0 = µ data .
Therefore, the scores are replaced by a neural network s θ ( x, t ) ,which is trained via score-matching techniques [Vincent, 2011, Hyvärinen and Dayan, 2005].
As a result, SGMs make two approximations.
The first one is in approximating p T by µ prior .
Thesecond one is the approximation of ∇ log p t by the neural net s θ ( x, t ) .
We illustrate this in Figure 1.
It is important to understand how these approximations translate to the distance of µ sample to µ data orˆ µ data .
Some early works already deal with these questions.
In [De Bortoli et al., 2021] the total variationdistance between µ sample and µ data in bounded, whereas [Song et al., 2021a] derives bounds withrespect to the KL-Divergence.
However, both of these works assume that the initial distribution µ datais rather well behaved.
In particular, it is assumed that µ data ( x ) > 0 for all x .
We shortly discuss thisassumption now.
We compute
We
∇
, we purposely adopt m
We
Assuming that µ data ( x ) > 0 for any x means that one postulates that every x is a possible sample fromµ data .
For example, this implies that even if all samples { x 1 , . . .
, x n } of µ data consist of images ofhuman faces, we say that µ data can possibly also generate images of for example furniture, animals orpure white noise.
Even though it might have low probability, any combinations of pixels is a possiblesample from µ data .
The assumption that µ data is actually supported on some lower dimensionalsubstructure M is well known under the name manifold hypothesis , see for example [Bengio et al.
,2013, Pope et al., 2021].
In practice this means that µ data ( x ) = 0 for many x .
This also leads toexploding scores ∇ log p t as t → 0 (see Section 5), a behaviour which also observed empirically andfurther underpins the relevance of the manifold assumption.
We will from now on denote the supportof µ data as data manifold M .
A fundamental question is then: What is the support of µ sample and how does it compare to M .
Thisis interesting for multiple reasons.
On the one hand, if we for example assume that M is the set of allimages of faces, then the knowledge that µ sample also has support M implies that, regardless of howclose µ sample actually is to µ data , it will at least always produce an image of a face.
On the other hand,we can also compare the support of µ sample to that of ˆ µ data .
The measures µ sample and µ data sharingtheir support translates to µ sample memorizing the training data and not being able to generalize.
Bothof these are very valuable insights into the qualities of µ sample on its own.
Furthermore, statisticaldistances like the KL-Divergence or the total variation distance are only meaningful if the support ofthe measures overlap to some degree, as otherwise they will equal the maximum distance value.
If two measures have the same support, they are said to be equivalent .
Our main result is thefollowing:
Main result.
We identify conditions under which µ sample is able to learn the data manifold, that issupp ( µ sample ) =
supp ( µ data ) .
Applying these results to different settings, we find precise conditionsunder which an SGM memorizes its training data or under which it is able to learn the right datamanifold M .
In particular we find that for SGMs to be able to generalize, the approximation error made whenapproximating the training drift ∇ log ˆ p t has to be unbounded.
A first illustration of these results is given in Figure 2.
We use a simple example, where we canperfectly evaluate the true drift ∇ log π t .
We then choose an incorrect initial condition µ prior whichis far from p T , and also add a constant error to ∇ log π t ( x ) .
The initial measures π 0 are given as the uniform distribution on the unit sphere and the uniform distribution on 9 samples from the unitsphere in Figure 2a and Figure 2 respectively.
We see that the final distribution of the reverse SDE,µ sample , is not the uniform distribution anymore.
This is due to errors in the initial conditions and thedrift.
Nevertheless, µ sample is still supported on the exact same subset as π 0 .
We now proceed as follows.
In Section 2 we will summarize some of the most popular forward SDEsthat are applied in SGMs.
Then, in Section 3 we discuss how the drift approximation s θ ( x, t ) for
SGMs is trained in most implementations.
We are ready to state our main results in Section 4.
In
Section 5 we show why and with which speed the true drift ∇ log p t of an SGM explodes under themanifold hypothesis.
Most of the paper discusses the error in the drift approximation.
But as we havediscussed, we also have an error in the initial conditions.
In Section 6 we will discuss how large theerror in the initial conditions will be in practice.
Popular SDEs used in SGMs
In this section we introduce some of the most popular SDEs that are used when implementing
SGMs.
The first works on SGMs studied discrete forward and backward processes.
Nevertheless,the transition kernels and algorithms proposed in those works can be seen as discretisations of somewell-known SDEs.
More recent works have studied this connection and state the algorithms in termsof SDEs ([Song et al., 2021b, Huang et al., 2021]).
Brownian Motion: The works [Song and Ermon, 2019, 2020] can be seen as a discretization of the
SDE
[Equation]
Denoting h ( t ) =
(cid:82) t 0 σ ( s )d s , the solution to the above process can be explicitly stated as a timechanged Brownian motion, X t = W h ( t ) .
The time-change can help in the implementation but doesnot alter the qualitative behaviour of the reverse SDE.
In our following analysis we therefore setσ ( t ) = 1 .
Nevertheless, our results still hold for any positive σ ( t ) .
Ornstein-Uhlenbeck Process: The works [Sohl-Dickstein et al., 2015, Ho et al., 2020] can be seenas a discretization of
[Equation]
which is an Ornstein-Uhlenbeck process.
Again, the parameters α t are a time-change and do notinfluence the properties that we are investigating in this paper.
Therefore, to simplify notation, weagain set α ( t ) = 1 .
Critically Damped Langevin Dynamics (CLD): The work [Dockhorn et al., 2021] studies asecond order SDE.
Here artificial velocity coordinates V t are introduced and the system underconsideration is
[Equation]
where X 0 ∼ p data , V 0 ∼ N (0 , I ) .
For generation one runs the reverse SDE in X t and V t but discardsthe V coordinate at the end.
The work [Dockhorn et al., 2021] also includes the parameters M and γ .
We set both to 1 as they do in their numerical experiments.
Score approximation with a finite number of samples
We now quickly discuss how the neural network is trained to approximate ∇ log p t and what implications this has.
The score ∇ log p t is approximated by minimizing a variant of
[Equation]
(3) for some weighting function w .
This is done via score matching techniques (see [Hyvärinen and
Dayan, 2005, Vincent, 2011, Song et al., 2020]).
However, the above expectation depends on
[Figure]
, whereas used t is a of 50000 Gaussians, one each
We then saved the examples, by calculating the distance to the closest example.
Above we plot the average distance see distance to the
(b): We evaluate ∇ log ˆ p t as in (a).
We do the analogous experiment to Figure 2b on CIFAR-10 and add the 1-vector in R 32 × 32 × 3 to the score, i.e. we use s ( x, t ) = ∇ log ˆ p t ( x ) + (1 , 1 , . . .
The error e ( x, t ) =
, 1) is bounded and therefore fulfils Assumption 1.
All of the plotted images are nearly equal to an CIFAR-10 image.
If we run the SDE with ∇ log ˆ p t , we know that the final distribution should be the uniform distribution on all training examples.
We see, that the error did not change the support, it just changed the distribution on the training examples.
In this case it made very white images much more probable, since the one-vector corresponds to a fully white image.
p t , which we cannot evaluate since it depends on p 0 = µ data .
Nevertheless, we can evaluate theapproximation ˆ p t ,
[Equation]
The surrogate loss
[Equation]
can be evaluated and used for training.
This raises the question in which way or to which degree theloss should actually be minimized.
If we would minimize this loss perfectly, then s θ ( x, t ) wouldbe equal to ∇ log ˆ p t .
The reverse SDE started in an appropriate initial condition with drift ∇ log ˆ p thowever, will produce samples from ˆ µ data , which are training examples.
Effects of the approximations
This section contains our main results.
We first state the assumptions we have to make and then the
Theorems.
4.1 Error in the initial condition
The following assumption is needed for the reverse SDE to be defined even in the case when theinitial distribution π 0 has a degenerate support.
In Section ??
we will show that all the SDEs from
Section 2 satisfy the Assumption.
Assumption 1.
Let C and C S,N be constants that can change from appearance to appearance.
(i) β is globally Lipschitz, i.e. ∥ β ( x ) − β ( y ) ∥ ≤ C ∥ x − y ∥ .
(ii) β grows at most linearly, i.e. ∥ β ( x ) ∥ ≤ C (1 + ∥ x ∥ )
(iii) X t has a density π t ∈ C 1 for every t > 0 and (cid:82) 1 t 0 (cid:82) ∥ x
∥ <R | π t ( x ) | 2 + ∥∇ x π t ( x ) ∥
d x d t < ∞
for any R > 0 and 0 < t 0 ≤ T .
Furthermore, for each S ∈ (0 , T ) ,
(iv) ∇ log π t
is locally Lipschitz, ∥∇ log π t
( x )
−∇ log π t
( y )
∥ ≤ C S
N
∥ x − y ∥
for all t ∈ [
S, T
and
∥
x ∥ , ∥ y ∥ ≤ N
Conditions ( i ) - ( iii ) are technical conditions on the forward SDE.
They ensure that if we run asolution to the forward SDE, X t , backwards in time, then X
Rt :=
X T − t will be a solution to thereverse SDE (2) on [0 , T ) .
The last condition then ensures that the solutions to the reverse SDE areunique, therefore we will be able to transmit the properties of X R to any other solution Y t of (2).
The following result shows that Assumption 1 can be expected to hold in practice.
To simplify thecalculations we assume that the data manifold M = supp ( µ data ) is contained in a ball of diameter M .
This is a natural assumption for many data sets.
Nevertheless, we note that this assumption could beweakened by additional technical effort.
Lemma 1.
Assume that the data manifold M is contained in a ball of radius M .
Then all themethods introduced in Section 2 fulfil Assumption 1.
We are now ready to state our first main result,
Theorem 1.
Denote by π t the marginals of the forward SDE started in π 0 .
Assume that Assumption1 holds and that µ prior is absolutely continuous with respect to π T .
Then the following hold.
i)
Let Y t be a solution to (2) on [0 , T ) .
The limit Y T :=
lim t → T Y T exists almost surely.
Werefer to its distribution as µ sample .
The distribution µ sample is absolutely continuous withrespect to µ data .
If π T and µ prior are equivalent, then so are µ sample and π 0 . ii)
Furthermore for
any f -divergence D f ,
D f ( µ sample | π 0 ) ≤ D f ( µ prior | π T ) and D f ( π 0 | µ sample ) ≤ D f
( π T | µ prior ) .
Applying the above theorem with π T =
p T tells us something about the equivalence and distancebetween µ sample and µ data , since π 0 = p 0 = µ data .
Applying the theorem with ˆ p t tells us somethingabout the generalization capabilities of SGMs.
The measures π T and µ prior are normally both supported on all of R d and therefore equivalent.
Since
Assumption 1 also holds in most situations (see Lemma 1), the requirements for Theorem 1 aresatisfied in practice.
From item ( i ) with π t = ˆ p t we can conclude that the error we make in the initialconditions is not responsible for the generalization capacities of SGMs.
The second item then shows that the f -divergences between µ sample and µ data are bounded by thef -divergences of µ prior to p T .
The total variation distance and the KL-Divergence are both specialcases of f -divergences.
4.2 Error in the drift
Given a forward SDE with marginals π t and an approximation s ( x, t ) to ∇ log π t , we define thereverse SDE for the approximation s as
[Equation]
Assumption 2.
We assume that the reverse SDE ˜ Y t has a solution on [0 , T ) .
For t < T , we definethe Girsanov weights
[Equation]
exp
[Equation]
and assume that are uniformly integrable martingale.
We shortly discuss this assumption.
The assumption that Z t is a martingale is equivalent to theexpectation of Z t being equal to 1 for all t .
The assumption that it is uniformly is more technical (see
Appendix A.2), but is fulfilled for example if for some p > 1 , E [ | Z t | p ] < ∞ for all t .
For example, ifthe Z t have bounded variance, then they are uniformly integrable.
A condition that ensures that Z t is both, a martingale and uniformly integrable, is given by Novikov’scondition Novikov [1980].
It states, that if
[Equation]
then Z t is a uniformly integrable martingale.
Using Assumption 2 we can now state
Theorem 2.
Assume that Assumption 2 holds.
Assume furthermore that Assumption 1 holds with∇ log π t replaced by s ( x, t ) .
Then ˜ Y T =
lim t → T ˜ Y t is well defined.
Moreover, its distribution is equivalent to the distribution of
Y T .
In particular, if ∥ s ( x, t ) − ∇ log ˆ p t ∥ is bounded, then Assumption 2 holds, and the SGM hasmemorized its training data.
Putting Theorem 1 and Theorem 2 together, we can conclude that, if both Assumptions hold, µ sampleis equivalent to π 0 .
Therefore, if Assumption 2 holds for π t = p t , we have a positive statement andknow that µ sample will have the exact same support as µ data , i.e. it has learned the data manifold M .
If the Assumption would hold for π t = ˆ p t , we would however just memorize the training data, see
Figure 2b or Figure 3b for a visualization.
However, empirically it has been shown that SGMs areable to create novel samples (see, Figure 3a or Dhariwal and Nichol [2021]).
Therefore, we candeduce that Assumption 2 is violated in practice.
SGMs are trained by minimizing ˆ L ( θ ) (see (5)).
On the other hand, if L ( θ ) is minimized well enoughfor Z t to be fulfil Assumption 2, then the SGM is collapsed onto its training data.
This is especiallyinteresting since
[Equation]
We evaluated N from (8) on CIFAR-10, once for the difference between the s θ ( x, t ) from Song et
al.[2021b] so and ∇ log ˆ p t , and once by just using a perturbed drift with a constant error, s ( x, t ) =∇ log ˆ p t + 12 (1 , 1 , . .
. , 1) , see Figure 4.
The reverse SDE using the drift ∇ log ˆ p t + 12 (1 , 1 , . .
. , 1) , isequivalent to ˆ µ data , as we know from Theorem 2 and have also already observed in Figure 3b.
Figure4 confirms that Z t is indeed a uniformly integrable martingale and therefore fulfils Assumption 2.
In future work we believe that further understanding the characteristics of Z t , how they relate tothe minimization of L ( θ ) and generalization is crucial to the understanding of SGMs and theirempirical success.
Recent works also study the question on how the neural network architectureand parametrization is related to the boundedness of the output Kim et al.
The relationshipbetween these choices and the properties of Z t is also an interesting research avenue.
Lastly, thedistribution of Z t is very heavy-tailed.
Most of the samples are very small with a few extremely largesamples in between.
Therefore one needs many samples from Z t to understand its characteristics.
Finding robust estimators for Z t or its expectation could help their usage in the training or evaluationprocedure.
Drift Explosion under manifold hypothesis
First, we note that all SDEs in Section 2 are linear SDEs.
Therefore, their transition kernels are
Gaussian ([Pavliotis, 2014, Section 3.7]):
[Equation]
The explicit form of m t and Σ t differ for each of the SDEs and can be found in Appendix C.1.
Weremark that Σ t does not depend on the initial condition z 0 .
The transition kernel above gives us thedistribution of the SDE started in a single point z 0 .
Since we start the SDE in µ data we need to averageover µ data to get the marginal at time t :
[Equation]
We can also compute the additional drift in the reverse SDE (see Appendix C.2),
[Equation]
The drift is intuitive, especially for the Brownian Motion, where m t ( z )
=
z and Σ t = t .
In thatcase, E [ Z 0 | Z t = z ] is the expected position from which the SDE was started on the manifold, if itis at position z at time t .
For a point z which has distance δ > 0 to M , ∥ z − E [ Z 0 | Z t = z ] ∥ > δ ,since Z 0 is in M almost surely.
Therefore, we can infer that ∇ log p t ( z ) explodes like δt as t → 0 .
Asimilar analysis can be conducted for the other two SDEs using the formulas for the transition kernelsin Appendix C.1.
In the Brownian motion case with σ = 1 , the forward SDE can be represented as
X t =
X 0 + B t .
Since B t has distribution N (0 , tI d ) , we can expect the distance of X t − X 0 to be 235 E [ ∥ X t − X 0 ∥ ]
=
E [ ∥ B t ∥ ] ≈ √ t .
Therefore, under the manifold hypothesis ∇ log p t ( Y t ) will behavelike √ tt = 1 √ t .
In particular, it will explode along a typical path of Y t as t → 0 .
This behaviour isalso observed empirically.
This argument can be generalized to arbitrary SDEs.
If σ 2 t is the varianceof p t , then we can expect ∇ log p t ( Y t ) to be of order 1 σ t .
Furthermore, ∇ log p t ( x ) will point towardsthe data manifold for small t .
The drift ∇ log p t then acts like a support matching force , where theforce grows to infinity as t → 0 , absorbing all the SDE paths onto the manifold.
Distance from p T to µ prior
We have seen in Theorem 1 that the distance between µ sample and µ data is directly related to thedistance between p T and µ prior .
The OU-process and the CLD are well-known SDEs in the Markov
Chain Monte Carlo community.
They form building blocks of algorithms like Langevin MCMC([Roberts and Tweedie, 1996, Welling and Teh, 2011]) or Hamiltonian Monte Carlo ([Neal et al.
and the speed at which they converge to their stationary distribution is well studied ([Robertsand Tweedie, 1996, Markowich and Villani, 2000, Bou-Rabee et al., 2010, Eberle et al., 2019, Caoet al., 2019]).
It is known that their intermediate distributions p t will converge to µ prior =
N (0 , I d ) atan exponential rate.
The Brownian motion however does not converge to a stationary distribution and therefore one has tochoose a different µ T prior for each T to approximate p T .
We now study the distance of µ T prior to p T .
Ifwe transfer the suggestions of [Song et al., 2021b, Appendix C] to our setting, they propose to setµ T prior to N (0 , T ) .
However, p t has the same mean as µ data and has variance Var [ µ data ] + T .
Loosely speaking, in contrast to the other two SDEs, the Brownian Motion does not forget its initial mean andcovariance.
As we see in the following lemma, we can decrease the distance between p T and µ T priorby choosing the mean and covariance of µ T prior equal to the mean and covariance of p T .
Lemma 2.
Let p T be the T -time marginal of the Brownian motion process X t of Section 2.
Thefollowing minimization problem
[Equation]
is minimized by m T
= E [ µ data ] and C T =
Cov[ µ data ] + TI d .
If we restrict the covariance to be amultiple of the identity matrix, i.e.
[Equation]
, the problem is solved by choosing m as above and c as c = E [ ∥ X t − m ∥ 2 ] = trace ( C T ) .
This result is a slight variation on the well known fact that the KL -projection in the second argumentmatches its moments.
We prove it in Appendix E.2.
Equivalently to choosing the mean of µ T prior equalto the mean of p T , E [ p T ] =
E [ p 0 ] , one can normalize the training data to have mean 0 and then alsocentre µ T prior .
However, the covariance matrix will not be equal to I d after common normalizationmethods.
The next lemma bounds the distance of p T and µ T prior in the Brownian motion case.
Lemma 3.
Let p t be the time t -marginal of a Brownian motion with initial condition µ data .
Denoteby c i , i = 1 , . . .
, d, the eigenvalues of Cov( µ data ) .
Let µ T prior be any normal distribution.
Then
[Equation]
This lemma is proven by explicitly bounding the distance of p T to the optimal normal distributionfrom Lemma 2, see Appendix E.2.
Broader impact
The results deepen the understanding of score-based generative models.
As such, they can be seen asa step towards improving the quality of generative models.
Therefore the possible negative societalimpacts are the same ones that apply to generative modelling in general.
First, generative modelscan be used to create synthetic data that is hard to distinguish from real data (for example images orvideos), see [Mirsky and Lee, 2021].
Second, generative models can learn and reproduce biases thatare prevalent in the training data ([Esser et al., 2020]).
Last, depending on the application, generativemodels might be used to do creative work that was previously done by humans.
Conclusion
We conducted a theoretical study of some properties of SGMs.
We found explicit conditions underwhich the sample measure µ sample is equivalent to the true data generating distribution µ data .
Underthese conditions we can guarantee, that the SGM generates samples that could also be samples fromµ data .
Furthermore, each sample that can be generated by µ data also has positive probability underµ sample , meaning that the full support is covered.
Since one can not actually access the full support of µ data , but only a finite number of trainingexamples { x i } Ni =1 , our results can be applied to find conditions under which the SGM memorizesits training data.
We believe that this observation provides a first step towards understanding thegeneralization capabilities of SGMs. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] The assumptions are followed by
a discussion of their strength.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix B.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] There are no error to report, we just have small illustrativenumerical experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
We did not use any
existing assets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We did not use crowdsourcing or conduct research with human
subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Stochastic prerequisites
In this section we give a formal introduction to some of the concepts used in this work.
For a morerigorous treatment, see for example [Klenke, 2013] or [Karatzas and Shreve, 2012].
A.1 Equivalence of measures / Girsanov Theorem
First we define absolute continuity of measures.
Let µ and ν be two measures on (Ω , F ) , where F isa σ -algebra.
Definition 1.
We say that µ is absolutely continuous with respect to ν if µ ( A )
=
for any A ∈ Fsuch that ν ( A ) = 0 .
We also denote this by µ ≪ ν .
Two measures µ and ν are equivalent if µ ≪
ν
and ν ≪ µ .
Loosely speaking, we can say that µ ≪ νif the support of µ is contained in the support of ν
and they are equivalent if they share the samesupport.
The Radon-Nikodym theorem tells us that if µ ≪ ν , then under mild conditions there exists a densityd µ d ν : Ω → R such that µ ( A ) = (cid:82)
A d µ d ν ( ω )d ν ( ω ) .
Therefore, we can obtain µ through a reweightingof ν .
One specific instance of this is the Girsanov Theorem.
Assume we are given the solutions totwo SDEs in R d ,
[Equation]
(11) and
[Equation]
Both of these induce a measure on the space of continuous functions Ω = C
([0 , T ] , R d ) .
We denotethem by P and ˜ P respectively.
Then the Girsanov Theorem equips us with conditions under whichthe measures P and ˜ P are equivalent.
Furthermore, in case of equivalence we get a formula for thedensity of ˜ P with respect to P .
The relative density is given as
[Equation]
For a full statement of the Girsanov Theorem and under which conditions it holds, see [Karatzas and
Shreve, 2012, Section 3.5].
A.2 Uniform integrability
Since we are treating the case where the drift explodes as t → T we end up with densities
[Equation]
on C ([0 , t ] , R d ) , but not with a density on C ([0 , T ] , R d ) .
Uniform integrability is exactly the conditionone needs to extend these local densities.
Definition 2.
A family { X α } of random variables is called uniformly integrable if
[Equation]
as s → ∞ .
In the proof of Theorem 2 we implicitly use the following two results which we here state as a lemma.
The filtration F t is defined as in the proof of Theorem 2.
Lemma 4.
Assume the Z t in (13) form a uniformly integrable martingale on [0 , T ) .
Then,
• the limit lim t → T Z t exists in L
We denote this limit by Z
• Furthermore, ˜ P is absolutely continuous with respect to P on F = σ ( ∪ t<T F t ) with density
Z .
Proof.
Both of these results are standard.
The first one can for example be found in [Karatzas and
Shreve, 2012, Section 1.3.B].
For the second one we compute that for any A ∈ F s ,
[Equation]
where we used L 1 convergence in the second equality and the martingale property of Z s in the thirdequality.
Therefore Z is a density of ˜ P with respect to P on each F s for s < T .
Therefore Z is also adensity of ˜ P with respect to P on F which concludes the proof.
B Numerics
All numerical experiments can be run on a consumer grade computer within a few minutes.
B.1 Figure 1
We first discuss the top left figure of Figure 1.
We set p
= µ data to a mixture of two Gaussian
N ( − 2 , 1100 ) and N (2 , 1100 ) with weights w 1 = 13 and w 2 = 23 respectively.
Then, we draw N =5 000 000 samples from µ data , denoted by Y n 0 , n = 1 , . . .
, N .
An Euler-Maruyama discretization ofthe Brownian motion propagates these samples from time t = 0 to t = 1 by
[Equation]
where Z in ∼ N (0 , 1) are i.i.d.
random variables, independent of X jm for m ≤ n and j = 1 , . .
. , N .
The time index i runs from 0 to I = 2000 and dt is set to dt =
I .
The initial samples { X n 0 } Nn =1are used to create the left line plot of p 0 and the final samples { X nI } Nn =1 are used to create the rightline plot of p 1 using kernel density estimation.
The { X ni } Nn =1 are approximate samples from p
i/I .
Therefore, we create histograms using { X ni } Nn =1 to approximate p i/I .
The height of the histogrambars corresponds to the square root of the colour intensity in the heat map.
The horizontal axis inthe heat map stands for the time t , whereas the vertical axis stands for the position x .
At location( t, x ) we plot an estimate of (cid:112) p t ( x ) .
We apply the square root since it improves the contrast in areaswhere p t ( x ) is close to 0 and makes it more visible where p t ( x ) > 0 to the observer.
For the bottom left figure we show the same plots, just for the reverse SDE (6) instead of the forward
SDE.
Since the initial distribution is a Gaussian mixture we can exactly calculate p t using
[Equation]
where we use N ( x ; m, v ) for the probability density function of a normal distribution with meanm and variance v , evaluated at x .
With the above expression of p t one could compute an analyticalrepresentation of ∇ log p t .
We use automatic differentiation instead.
The reverse SDE (6) is simulatedwith a disturbance
e ( x, t ) = 1 and initial condition q 0 = µ prior =
N (0 , 1) .
The Euler-Maruyamamethod is run with the same step size dt =
I .
More precisely, the one step transition kernel of thediscretized reverse SDE is
[Equation]
(15) where ˜ Z in ∼ N (0 , 1) are i.i.d.
random variables, independent of Y jm for m ≤ n and j = 1 , . .
. , N .
The plots are created in the same way as for the upper left plot, except that we reverse the time axis toplot p t and q 1 − t directly underneath each other.
On the right side we plot the same kernel density estimates already plotted on the left side as µ [Table] and µ data into the same plot for comparison.
B.2 Figure 2b and 2a
Figure 2b is created by setting µ data to be the uniform distribution on M = 9 equally spaced samples{ x i } Mi =1 on the unit sphere S 1 .
This can also be viewed as a Gaussian mixture with 9 componentswhere each component having mean x i and variance 0 .
Therefore, we can again explicitly calculatep t for t > 0 as in (14),
[Equation]
m t ( z 0 ) Σ t
Brownian Motion ztI d
OU-Process exp( − t )
z(1 −
[Equation]
) I d
The score ∇ log p t ( y ) is evaluated using automatic differentiation.
The reverse SDE 6 is simulatedwith q 0 =
N (( x = − 1 .
5 , y = 0) , I 2 ) and e (( x, y ) , t )
= ( x = 0 , y = − 1) .
For the numericalsimulation, we again use the Euler-Maruyama scheme.
We use a step width of dt = 0 .
9 1000 ≈ 1 1000for t
=
[0 , 0 . 9] , dt = 0 .
09 1000 ≈ 1 10 000 for t ∈ [0 . 9 , 0 . 99] and of dt = 1 100 000 for t ∈ [0 . 99 , 1] .
For asimulation of N = 50 000 paths of the reverse SDE, we start by drawing Y n 0 =
A n + Z n , where
A n are i.i.d.
uniformly distributed on { x i } Mi =1 and Z n ∼ N (0 , 1) i.i.d..
The { A n } and { Z n } arealso independent from each other.
We then propagate the Y n 0 similarly to (15), except that we usea different values for dt depending on t .
This leads to approximate samples Y it from q t .
At thedisplayed times t we plot the functionh t ( x ) =
N (cid:88) i
= k ( x, Y it ) , where k is an unnormalized Gaussian kernel with a very small bandwidth parameter,
[Equation]
Normalizing h t gives us a density estimate of q t .
We plot these estimates as heat maps for differentvalues of t .
For Figure 2a we follow the same steps as for Figure 2b, except that µ data is set to the uniformdistribution over M = 256 evenly spaced samples from the unit sphere S 1 .
B.3 Figure 3
For Figure 3 we used the DDPMPP model from the Github repository for the paper from Song et
al.[2021b].
Then we evaluated the true score using (4), where the sum runs through the N =
50000training examples of CIFAR-10, Krizhevsky et al.
A similar experiment using the truemarginals ˆ µ data has also been conducted in Peluchetti [2021].
C Studying the forward DensitiesC.1 Transition kernels
The transition kernels p ( z 0 , · ) for the SDEs from Section 2 are of the from
[Equation]
where m t and Σ t are given in Table 1 for the Brownian Motion and the Ornstein-Uhlenbeck process.
The form of the transition kernels for the CLD are more involved.
They can be found in Dockhornet al.
[2021, Appendix B.1].
Lemma 5.
The marginal densities p t ( x ) of the SDEs treated in Section 2 depend smoothly on x andt .
Proof.
One can combine the form of m t and Σ t and the explicit representation of p t in (9) to seethis.
More generally, for the Brownian motion and the OU-Process this is a result of the Hörmandertheorem.
For the CLD it is a result of hypocoercivity.
C.2 Form of the drift
We now prove that we can represent the drift as in (10).
Lemma 6.
Assume that p t has the form (9) , i.e.
[Equation]
Then (10) holds true, i.e.
[Equation]
Proof.
[Equation]
If we now divide everything by p t it cancels in the first summand.
In the second summand we get theformula for the conditional expectation (see, for example [Klenke, 2013, Section 8.2]).
D Reverse SDEs: The general case
One can also treat more general forward SDEs than we did in Section 1.
This leads to a morecomplicated form of the reverse SDE.
Our Theorems do not use the specific form of the forward
SDEs and therefore also hold in the general case.
We denote the forward SDE by
[Equation]
16) where µ data is supported on M ⊂ R d and W t is a R r valued Brownian motion.
The drift b mapsfrom R × R d to R d .
The dispersion coefficient σ maps from R × R d to the d × r -matrices.
Thetime-reversed process Y t :=
X T − t is then a solution to
[Equation]
(17) withb i ( t, y )
= − β
( T − t, y
[Equation]
+ (cid:80) j ∇ j ( a ij ( T − t
[Equation]
y ) p T − t
[Equation]
y ))
p T − t ( y
[Equation]
, a ( t, y )
=
σ ( t, y ) σ ( t, y ) T , and q 0 = p T , see Haussmann and Pardoux [1986].
This simplifies to the case discussed in Section 1if r = d , σ is is a multiple of the identity matrix and β is independent of the time t .
In Assumption 1we treat the case where the SDEs are of the form written-out in Section 1.
The items ( i ) − ( iii ) needto be replaced by their more general counterpart as found in Haussmann and Pardoux [1986, Section2].
In the last item ( iv ) , ∇ log p t needs to be replaced by (cid:80) j ∇ j ( a ij ( T − t,y ) p T − t ( y ))
p T − t ( y ) .
E Proofs
E.1 Proofs of the theorems
We now give proofs of our main results and briefly summarize the key steps in an intuitive way.
In ourstudy we would like to include the case when µ data is degenerate and supported on a low-dimensional substructure M .
As we have seen in Section ??
, this can lead to an exploding drift in the reverse
SDE as t → T .
Nevertheless, in order to understand the properties of µ data , it is crucial to studythe properties of solutions to the reverse SDE at time t = T .
This is where the main mathematicaldifficulties come from.
The proofs are mostly independent of the specific form of the forward SDEand hold for more general forward/backward SDEs than those stated in Section 1, see Appendix D.
E.1.1 Theorem
We now proceed with proving Theorem 1.
Proof.
Let P be the measure on Ω = C ([0 , T ] , R n ) induced by the forward SDE (1) started inp 0 = µ data .
P has marginals p t .
Denote by X t the canonical projections X t ( ω ) = ω ( t ) for ω ∈ Ω .
We define Q through
[Equation]
By the data processing inequality we obtain (see [Liese and Vajda, 2006, Theorem 14]),
[Equation]
It remains to prove that by running Q backwards we obtain a solution to (2) started in µ prior .
Wedenote the generator of the reverse SDE (2) by L .
Denote by Q R and P R the time reversals of Q and
P .
Our assumption are such that P R is a Markov process solving the martingale problem for L (see[Haussmann and Pardoux, 1986, Theorem 2.1]).
A short calculation shows that Q R is still Markov(see, for example [Léonard, 2011, Proposition 4.2]).
Furthermore for f ∈ C ∞ c ( R n ) ,
[Equation]
In the second equality we used the Markov property of P R .
In the last one we used that P R solvesthe martingale problem for L .
Therefore also Q R solves the martingale problem for L .
Denote by Y ta solution to (2) on [0 , T ) .
Since solutions to (2) are unique in law on [0 , S ] for S < T (see [Karatzasand Shreve, 2012, Section 5.2]) and the solutions are continuous, the law of Y is equal to Q R on[0 , T ) .
But the paths of Q R are continuous on [0 , T ] .
Therefore, Y can be extended to [0 , T ] , i.e.the limit Y T :=
lim t → T Y t exists almost surely and its distribution is equal to the T -time marginaldistribution of Q R , which is the 0 -time marginal of Q .
We denote the marginals of Q and P by Q tand P t .
Since Q is absolutely continuous with respect to P , Q 0 = µ sample is absolutely continuouswith respect to P 0 = µ data .
Analogously, if µ prior and p T are equivalent, then so are P and Q andtherefore P 0 and Q 0 .
This proves ( i ) .
( ii ) is a consequence of the data processing inequality for f -divergences ([Liese and Vajda, 2006, Theorem 14]), analogous to (18).
The main idea of this proof is that we look at the forward SDE for X t first.
It induces a distribution Pover all continuous paths in Ω = C
([0 , T ] , R d ) .
If we reverse the time direction of this distributionon Ω , we get a solution to the reverse SDE, started in p T .
This reverse solution is well behaved ast → T , since Y T =
X 0 .
The solution for a different initial condition µ prior ̸ = p T is obtained byreweighting P .
This does not change the qualitative behaviour of Y T , which still exists and is welldefined.
We then use a uniqueness result to see that any solution of (17) inherits these properties.
1.2 Theorem[-
-]
Proof.
Denote the space C
([0 , T ) , R d ) by Ω .
We also define the natural filtration F t = σ ( x ( s ) | s ≤ t ) .
We denote the distribution of Y on Ω by P .
We define ˜ P by reweighting P with Z t on F t .
By Girsanovtheorem (see [Karatzas and Shreve, 2012, Section 3.5]) we know that the canonical process under ˜ Pis a solution to (6) on [0 , T ) .
Since Z t is uniformly integrable, its limit Z :=
lim t → T Z t exists in L 1 .
Furthermore ˜ P is absolutely continuous with respect to P on F = σ ( ∪ t<T F t )
=
σ ( x ( t ) | t < T ) withdensity Z .
We define x ( T ) :
=
lim t → T x ( t ) .
Then the event
A = { x ( T ) :=
lim t → T x ( t ) exists and x ( T ) ∈ M} has probability 1 under P (see Theorem 1) and therefore also under ˜ P .
Furthermore, x ( T ) ismeasurable with respect to F .
Therefore the distributions of x ( T ) under P and ˜ P are equivalent.
Thecanonical process under ˜ P is therefore a solution of (6), with the property that its time T -marginal iswell defined and equivalent to the time T -marginal of (2).
We can use uniqueness in law on [0 , S ) forany S < T and extend it to [0 , T ] as in the proof of Theorem 1.
This shows that every solution to (6)has the desired properties.
Finally we show that if e is bounded, it fulfils Assumptions 2.
We define by H t = (cid:82) t
∥
e ( s, ˆ Y s )
∥
d s .
Then there is a Brownian motion W t such that we can write Z t as
[Equation]
Since e is bounded by M , H t is bounded by TM .
In particular, one can view Z t =
E [ Z TM
|F
H t ] .
Therefore Z t is uniformly integrable since it can be viewed as a family of conditional expectations.
Here we essentially applied the Girsanov Theorem on [0 , T ) .
Using the uniform integrability of the
Girsanov weights Z t , we are able to extend it to [0 , T ] .
Therefore, we can infer that the distributionof Y and ˆ Y are actually equivalent on the whole path space C ([0 , T ] , R d ) .
In particular, their time
T -marginals will be equivalent too, which is the claim of the theorem.
E.2 Proof of the Lemmas
We start by proving Lemma 1.
Proof.
The forward drifts are β ( x ) = 0 , β ( x ) = − α 2 x and β ( x, v )
= ( v, − x − 2 v ) for the Brownian
Motion, the OU-Process and the Critically Damped Langevin Dynamics respectively.
In particular,these are all linear maps and therefore fulfil conditions ( i ) and ( ii ) of Assumption 1.
We show in Appendix C.1 that log p t is C ∞ in t and x for t > 0 .
Therefore we can integrate p tand its derivative over compact sets, implying that condition ( iii ) holds.
Furthermore, the Hessianw.r.t.
( x, t ) is continuous and obtains its maximum and minimum on the compact set [ S, T ] × B N ,where B N is the ball of diameter N around the origin.
Therefore the gradient ∇ log p t is Lipschitzon [ S, T ] × B N , which proves ( iv ) .
We now prove Lemma 2.
Proof.
We have that X 0 ∼ µ data .
Denote the mean and covariance of µ data by a and C respectively.
We define
[Equation]
for some functions m t and V t .
If V t would not have full rank, n t would be a degenerate distribution.
Since p t
We can write X t as X t =
X 0 + √ tZ where Z ∼ N (0 , I d ) .
Now for the KL-divergence at time t itholds that
[Equation]
We compute
[Equation]
We added and subtracted a in the second equality.
In the last equality we used that E [ X T AX ] =tr ( Cov ( X ) A ) if X is a centred random variable.
Substituting this into our prior calculation, we arriveat
[Equation]
p
[Equation]
n t
[Equation]
The mean function m t only appears in the last term.
Therefore, we decrease the KL-divergence bysetting m t = a .
We now optimize for the covariance matrix V t .
The H ( p t ) term is independent of V t .
Consequently, to optimize the KL ( p t | n t ) with respect to V t , it is enough to optimize
[Equation]
where we used that det ( A − 1 )
= det ( A ) − 1 to rewrite the loss in terms of V − 1 t .
Since − log(det( · ))is convex on the positive semidefinite matrices and the other summands are linear, L is convex in
V − 1 t .
We take the gradient of L with respect to the Frobenius inner product and obtain
[Equation]
Setting the gradient to 0 has the unique solution
[Equation]
which is a local and global minimum due to the convexity of L .
If we restrict V t to be of the form V t = v t I d , the loss becomes
[Equation]
By setting the derivative of this to zero, we arrive at
[Equation]
We now proceed to prove Lemma 3.
For that we first need the following proposition.
Proposition 1.
Let p t be the time t -marginal of a Brownian motion started in µ data .
Assume that thesupport of µ data is contained in a ball of radius M .
Let m t =
E [ µ data ] and C t =
Cov [ µ data ] + tI dbe the optimal mean and covariance operator from Lemma 2.
Denote n t = N ( m t , V t ) .
Then
KL ( p t | n t ) → 0 as t → ∞ .
Proof.
For t > 0 , the KL-divergence is given by
[Equation]
We used that
[Equation] [Equation]
where c i are the eigenvalues of Cov [ µ data ] .
Now,
[Equation]
We bound
[Equation]
We write X t as X t =
X 0 + √ tZ where Z ∼ N (0 , I d ) .
Then,
[Equation]
We use that
[Equation]
where v 0 =
E [ ∥ X 0 − E [ µ data ] ∥ 2 ] = tr ( C 0 ) .
Since Z is centred and independent of X t , the cross-term 708 E [ ⟨ X t , Z ⟩ ] = 0 vanishes.
We now take the expectation over the right hand side of (21) to obtain
[Equation]
Putting it all together we have that
[Equation]
as t → ∞ .
We are now ready to prove Lemma 3.
Proof.
We again use (19).
We now take the derivative of the KL-divergence using De Bruijn’s identityfor multivariate random variables (see [Rioul, 2010], [Costa and Cover, 1984]):
[Equation]
where we applied Lemma 6.
Since E [ X 0 | X t ] is the L 2 -orthogonal projection of X 0 to the σ ( X t ) -measurable random variables and X t is σ ( X t ) -measurable, we have that
[Equation]
For A t =
E [ ∥ X 0 − E
[ X 0 | X t ] ∥ 2 ] it holds that
[Equation]
since the conditional expectation is a contraction in L 2 .
We use (20) again to arrive at
[Equation]
where c i are the eigenvalues of Cov [ µ data ] + tI d .
Integrating the right hand side we obtain
[Equation]
as a possible antiderivative.
Define g ( t ) =
h ( t ) − KL ( p t | n t ) .
Let us now assume that the support of µ data is contained in a ball of radius M .
Then, by Proposition 1, lim t →∞ KL ( p t | n t ) =
lim t →∞ h ( t ) =
and therefore lim g ( t ) → 0 .
Equation (23) implies that g ′ ( t ) ≤ 0 for all t > 0 .
Assume that there is a s and an ϵ > 0 such that g ( s ) ≤ − ϵ < 0 .
Since g ′ ( t ) ≤ 0 for all t > 0 wecan then conclude that g ( t ) ≤ − ϵ < 0 for all t ≥ s .
In particular, g ( t ) ̸→ 0 as t → ∞ , which is acontradiction.
Therefore the statement of the Lemma holds for µ data .
Now let µ data be any measure.
Let X be a random variable that is distributed according to µ data and
Z be a standard normal random variable, Z ∼ N (0 , I d ) .
Then X + √ tZ has distribution p t .
Wedefine X M :=
X 1 {∥ X ∥≤ M } and X Mt =
X M + √ tZ .
Denote the distribution of X Mt by p Mt .
Thedistribution of X M is supported on a ball of radius M .
Furthermore, X M + √ tZ → X + √ tZ almostsurely and therefore p Mt → p t weakly for any t .
By lower semi-continuity of the KL-divergence,
[Equation]
where c Mi are the eigenvalues of Cov [ µ M data ] + tI d .
Since | X M | ≤ | X | and X has first and second moments one can apply dominated convergence to see that the covariance matrices converge,
Cov ( µ M data ) → Cov ( µ data ) .
Especially
[Equation]
Combining this with (24) concludes the proof.