Score-Based Generative Models with Lévy Processes
Abstract
Time reversibility of stochastic processes is a primary cornerstone of the scorebased generative models through stochastic differential equations (SDEs).
While a broader class of Markov processes is reversible, previous continuous-time approaches restrict the range of noise processes to Brownian motion (BM) since the closed form of the time reversal formula is only known for diffusion processes.
In this paper, we propose a class of score-based probabilistic generative models,
Lévy-It ¯ o Model (LIM), which utilizes d-dimensional α -stable distribution with independent components for noise injection.
To this end, we derive an exact time reversal formula for the SDEs with Lévy processes that can allow discontinuous pure jump motion.
Consequently, we advance the score-based generative models with a broad range of non-Gaussian Markov processes.
Empirical results on MNIST,
CIFAR-10, CelebA, and CelebA-HQ show that our approach is valid.
Introduction
The recent successes of score-based generative models [26, 28, 11] and their applications [20, 14, 6] draw huge attention from machine learning communities.
Score-based generative models via stochastic differential equations (SDEs) [28] rely on the time reversal theory of diffusion processes,
Anderson theorem [1] which shows that the time reversal of the diffusion process belongs to the class of diffusion processes again.
One can interpret this result as solving a martingale problem which induces a weak solution to the reverse SDEs [9, 4].
Due to the advances in the SDE theory with jump Markov processes [13, 24, 5], one can desire a positive expectation for applying a class of non-Gaussian noise distribution to score-based generative models.
However, since the closed form of the time reversal formula is only known for diffusion processes, whether a score-based method is feasible for a non-Gaussian Markov process other than a Brownian motion has been an open question in this field.
To tackle the challenging problem, we propose an exact formula for the time reversal of
SDEs with Lévy processes and a novel score-based generative method, Lévy-It¯o Model (LIM), which utilizes d-dimensional α -stable Lévy motion with independent components as noise injection.
We apply the proposed method to MNIST, CIFAR-10, CelebA, CelebA-HQ.
Our approach and empirical results establish the bridge between probability theory and generative models.
Score-Based Generative Models with Lévy Processes 2.1 Background
1-Dimensional Symmetric α -stable distribution Let α ∈ (0 , 2] be a characteristic exponent which determines the decay rate at which the tails of the distribution, and γ be a scale parameter.
1-dimensional symmetric α -stable distributions S α S ( γ ) have the heavy-tail properties P ( X > x ) ∼ x − α and densities with unknown closed-form expressions, except for α = 1 or α = 2 . 2 .
∗ Corresponding Author
When α = 2 , it holds S α S ( γ ) =
N (0 , √ 2 γ ) .
[Figure]
Lévy process and α -stable Lévy motion R d -valued stochastic process L t =
( L t ) t ≥ 0 with L 0 =is called Lévy process if (i)
L t has independent increments, (ii) L t has stationary increments, (iii)
L t is stochastically continuous.
If each components of the difference, [( L t − L s )] i and [ L t − s ]
i have the same distribution following S α S (( t − s ) 1 /α ) for s < t , then the Lévy process is called d-dimensional α -stable Lévy motion L αt .
Due to the stochastic continuity (iii), Lévy processes have a countable number of discontinuous points (i.e. jumps) [33].
Notably, L αt is a prototypical pure jump process.
The heavy-tail properties of α -stable distribution imply that the frequency of large jumps of L αt increases as α gets smaller (see Figure 1).
2.2 Lévy-It¯o Model: Time-Reversal of SDEs driven by Lévy Processes
Due to Lévy-Ito decomposition [2, 16], we consider a family of SDEs in R d driven by a Lévy process consisting of continuous Brownian motion part B t and pure jump part L αt as follows:
[Equation]
The following exact time-reversal formula is our main result.
Theorem 2.1 (Time-reversal formula of SDEs with Lévy Processes) .
The reverse SDE of (1) is
[Equation]
where ∂ α − 2 | x k | ( f 1 ( x ) , . .
, f d ( x ))
=
( ∂ α − 2 | x 1 | f 1 ( x ) , . . .
, ∂ α − 2 | x d | f d ( x )) is the partial fractional Riesz potential of order α − 2 with 1 < α < 2 [19] [24] such that F [ ∂ α | x i | f ]( k ) =
|
k
i | α F [ f ]( k ) for each i ∈ { 1 , . . .
, d } , x = ( x 1 , . . .
, x d ) ,
k = ( k 1 , . .
. , k d ) , and F is the Fourier transformation.
¯ B t and ¯ L t is a backward Brownian motion and backward d-dimensional α -stable Lévy motion, respectively.
See Theorem A.10 for more details.
We also remark that (2) recovers the result of [1] if α → 2 .
To shed light on the probabilistic approach with the jump Markov process, we propose L évy-
I t¯o
M odel (LIM), a novel score-based generative model through SDE driven by d-dimensional α -stable Lévy motion only ( σ B ( t ) ≡ 0 in (2)).
Considering a beta-scheduling version of LIM, we obtain d → X t = − β ( t ) α
→ X t + ( β ( t )) 1 /α dL αt .
Then the solution becomes → X t d = a ( t ) → X 0 + γ ( t ) ϵ , where d = means equality in distribution, [ ϵ ] i ∼ S α S (1) for each i ∈ { 1 , . . .
, d } , a ( t ) is exp( − (cid:82) t 0 β ( s ) α ds ) and the scale parameter γ ( t ) is (1 − a ( t ))
1 /α (see Lemma C.3).
Due to the Euler-Maruyama method, we can induce a stochastic sampling of LIM (see Corollary C.4.1).
The Probability ODE We can also derive the probability ODE of (1):
[Equation]
[Figure]
The proof of (3) can be found in Theorem B.4.
Deterministic ODE sampling of LIM can be deduced from (3) by using the Euler-Maruyama method (see Theorem C.5).
2.3 Score function for Lévy-It¯o Model
Let q α ( x ) be the product
q α ( x 1 ) · · · q α ( x n ) of density functions
q α ( x i ) of S α S (1) for x = ( x 1 , . . .
, x d ) .
Recall the solution of beta-scheduling version of (2) is → X t d = a ( t ) → X [-
-]0 + γ ( t ) ϵ with the transition density function p t ( x t | x 0 ) .
The score function of p t ( x t | x 0 ) satisfies ∇ x t log p t ( x t | x 0 ) =
∇ ϵ log q α ( ϵ ) /γ
( t ) (See Lemma D.1).
We denote S α ( x ) = ∇ x log q α ( x ) .
Figure 2.(a) shows that the score function of Brownian motion is linearly decreasing, while the Lévy score functions are not monotonic.
Hence, if we train the score model to target the Lévy score, it is difficult to denoise the divergent large noise generated at the heavy tail (Figure 2.(b)).
These phenomena worsen as α decreases.
Rectified Enhanced Lévy Score (ReELS) To denoise the large noise at the heavy tail without losing the nature of the Lévy score function, we propose Re ctified E nhanced L évy S core (ReELS) as follows:
[Equation]
Here we set the range I α as the interval between two local optimum points of the given Lévy score.
We find parameters
ˆ c, ˆ β in ReELS by fitting − sgn ( x )ˆ c | x | ˆ β to the Lévy score inside I α
(see Figure 2.(b)).
This procedure is equivalent to the fitting score function of a generalized Gaussian distribution to the Lévy score [17].
We remark that utilizing the BM score does not outperform ReELS for
Lévy-driven SDEs because generalized Gaussian distributions have a score function more similar to the Lévy score function (see Table D.1, D.2, and D.3).
The experiments on synthetic data (Mixture of
Gaussian, Two-Moon, Swiss-Roll) demonstrate that LIM trained by ReELS converges to the true data distribution and performs better than using BM score for Lévy-driven SDEs.
2.4 Loss function
We use the U-net architecture [21] as in DDPM [11] and apply L 2 -loss to train the model S θ ( x t , t ) using ReELS α ( ϵ ) as a label through the Denoising Score Matching (DSM) [27] .
For [ ϵ ] i ∼ S α S (1) for each i ∈ { 1 , . . .
, d } and x 0 ∼ p data , we let x t = a ( t ) x 0 + γ ( t ) ϵ where β ( t ) =
β 0 + ( β 1 − β 0 ) t , a ( t ) =
exp( − ( β 1 − β 0 )
α t 2 − β 0 α t ) , and γ ( t ) =
(1 − a ( t ) α ) 1 /α .
Let U (0 , 1) denote a uniform distribution.
Then the loss with the relative weight γ ( t ) is defined as
[Equation]
Experiment
with different NFEs.
ODE shows a faster
We empirically validate the proposed score-based generative model on image data including MNIST (Figure E.1),
CIFAR10 (Figure E.2), CelebA (Figure E.4, E.5), and
CelebA-HQ (Figure E.6).
We adjust the model size of each dataset for training efficiency.
For α ∈ { 1 . 8 , 1 . 5 , 1 . 2 } , we train our model on MNIST for 1000 epochs with β 0 = 0 .
1 , β 1 = 5 .
and use the noise clamping to control the largescale noise to improve the sample quality.
See Section E. to find the other configurations in different datasets.
Figure 3.(a) shows that LIM with ReELS converges faster than
DM (Diffusion Models [28], α = 2 . 0 ).
See Figure 3.(b) to compare FID scores [10] and bits per dimension (bits/dim) on the MNIST dataset for each number of function evaluations (NFE) with different α .
To evaluate bits/dim, we use a uniformly dequantized test dataset with 5 iterations and compute log-likelihood by using an ODE solver (See
Section E.2.2).
LIM achieves competitive sample quality compared to DM at α = 1 . 5 , 1 . 8 , and tends to converge quickly as lower α .
Figure 4 shows ReELS can be adaptive to the probability ODE, which enables fast sampling than stochastic sampling.
Although the large jump of the reverse process can be controlled in the reverse SDE sampling, it is challenging to control the jump size in the deterministic ODE sampling.
Hence, the image quality may degrade due to the effect of uncontrolled large noise, which leads to the higher bits per dimension in LIM 3.17 ( α = 1 . 8 ) >
1.67 ( α = 2 . 0 ).
Conclusion
In this paper, we broaden the range of noise distribution used in score-based generative models by inducing an approximate time reversal formula for SDEs with Lévy processes and by proposing a novel score-based generative model, Lévy-It¯o Model (LIM) with Rectified Enhanced Lévy Score (ReELS).
Empirical results validate that the proposed approach works well with different ranges of d-dimensional α -stable Lévy motions in synthetic datasets and various image data.
Consequently, our study presents a feasible solution and demonstrates the potential for applying a broader class of non-Gaussian Markov processes to score-based generative models.
Acknowledgment
This work was also supported by Research Fund (1.220086.01) of UNIST, Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)(No.
2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior
Intelligence for Embodied AI), and National Research Foundation of Korea(NRF) funded by the
Korea government(MSIT)(2021R1C1C1009256).
Appendix
First, we will explain the core idea of the proof and then describe the necessary theorems.
Detailed definitions are introduced later.
Ω is a probability space and b ( t, x ) , σ B ( t, x ) , σ L ( t, x ) is a scalar function from Ω to R a under some smooth condition.
If a R d -valued stochastic process → X t is a solution of a Stochastic Differential
Equations(SDE) driven by Lévy process, d → X t = b ( t, → X t
[Equation]
, the generator L t satisfies
[Equation]
where ν is a symmetric Lévy measure of L αt .
If for all ( t, x ) , σ L ( t, x ) > 0 , then
[Equation]
If σ L ( t, s ) = 0 , we know the exact time reversal formula [4].
So, our interest is when σ B ( t, s ) =and σ L ( t, s ) =
σ L ( t ) > 0 such that
[Equation]
We know the form of generator L t of the given SDE solution → X t .
Therefore we can get the timereversal formula of the operator L t [5] such that
[Equation]
where p t ( dy ) is a marginal distribution of ( → X t ) t ∈ [0 , 1] and the backward drift ← b ( t, x ) is given by
[Equation]
According to this [29], it can be seen that the reversal of Levy-driven SDE also appears as Levydriven SDE.
Therefore, it can be derived that 10is a generator of a solution of some SDE driven by
Lévy process.
But exactly what SDE does this generator follow?
And ← b ( t, x ) appears as an integral equation, can the exact form be calculated?
We will answer this question in Appendix A.
The proof is divided into two parts.
The first is to find the SDE representation of an operator L t of the formand the second is to derive the exact form of ← b ( t, x ) .
A Time-reversal of SDE
In this chapter, given the generator of the general Markov process with jump kernel, we show that the reverse form can be deduced into an exact formula under certain conditions.
Finally, we introduce the stochastic sampling and the deterministic ODE sampling of LIM.
Let us outline some necessary lemmas before we move on to the proof.
A homogeneous Markov process that corresponds to an inhomogeneous Markov process → X t always exists according to Lemma A.1.
The existence of an
SDE representation of a homogeneous Markov process with a particular generator is given in Lemma
A.5.
Lemma A.6 introduces the general reverse-time formula.
Through the transformation of timeinhomogeneous Markov processes and an SDE representation of given a specific generator, we find the SDE representation that corresponds to the generator of the time-reverse process.
From these lemmas, we deduce a reverse SDE representation, on which we also get a stochastic sampling and a deterministic ODE sampling based on probability ODE if a Markov process is provided as a solution to (1).
A.1 Time-Reversal of General Markov process with jump kernel
Let → X t be an R d -valued continuous time inhomogeneous Markov process on an probability space (Ω , A , P ) .
The evolution system is defined as
[Equation]
and this operator is well-defined on the set of Borel measurable function u on R d , denoted by B ( R d ) .
The operator is linear and positive preserving with T ( s, t )1 = 1 and T ( s, t )
=
T ( s, r ) T ( r, t ) for s ≤ r ≤ t .
This operator is also strongly continuous such that for each v, w ∈ R , v ≤ w and s ≤ t lim ( s,t ) → ( v,w ) ||
U ( s, t ) u − U ( v, w ) u || ∞ = 0 where ||·|| ∞ is the supreme norm.
For all u ∈ C ∞ ( R d ) , the set of a continuous function with vanishing at ∞ , the generators of the evolution system is given by
[Equation]
A family of linear operators T ( s, t ) on C ∞ is a Feller evolution system if it is a strongly continuous, positive, contraction semigroup on C ∞ .
Definition A.1 (Space-time process) .
Let B be a Borel algebra in R d and an a state space ( R + × R d , ˜ B ) with ˜ x ∈ R + × R d and σ -algebra ˜ B = { B ∈ R + ×
R d | B s ∈ B} , and a new sample space (˜Ω , ˜ A ) with ˜ w = ( s,
w ) ∈ R + × Ω = ˜Ω and ˜ A
= { A ⊂ R + × Ω | A s ∈ A , ∀ s ∈ R + } .
A space-time process ( ˜ X t ) is defined by
[Equation]
14) with the probability measure for A ∈ ˜ A and ˜ x ∈ R + × R d such that ˜ P ˜ x ( A ) =
˜ P
( A | ˜ X
= ( s, x )) .
=
P ( A s | → X s = x ) and the transition probabilities are given by ˜ P ( ˜ X t ∈ B | ˜ X 0 = ˜ x ) =
˜ P
( ˜ X t ∈
B | ˜ X
= ( s, x )) =
P ( → X s + t ∈ B
s + t | → X s = x ) where B ∈ ˜ B, ˜ x ∈ R + × R d .
The transition function is defined by ˜ P ( t, ˜ x, B )
=
P ( s, x ; s + t, B s + t ) .
Lemma A.1.
Given a inhomogeneous Markov process ( X t ) , the space-time process ( ˜ X t ) on (˜Ω , ˜ A , ˜ P ) is a homogeneous Markov process.
Proof.
See Transformation 3.1 in [3].
Lemma A.2.
Let ( → X t ) be the stochastic process with Feller evolution system U ( s, t ) and the generator of ( → X t ) be L s .
Let (cid:101)
X t be its space-time process with associated semigroup T ( t ) by
T t u (˜ x ) =
˜ E ( u ( ˜ X t ) | ˜
X
= ˜ x ) for ˜ x ∈ R + × R d and u ∈ B b ( R + × R d ) .
Then the extended generator ˜ L of T t is given for all u ∈ C ∞ (cid:0) [0 , 1] × R d (cid:1) satisfying some conditions,
[Equation]
where (cid:101) x
= ( s, x ) and u s ( x ) =
u ( s, x
[Equation]
Proof.
See Theorem 3.2 in [3].
A Markov process typically has a generator that takes the form
[Equation]
where b ( x ) is a locally bounded R d -valued function and ( a ij ) is a locally bounded and d × m matrix-valued function, B 1 is the ball with a radius of one and a center of zero and η satisfies
[Equation]
[Equation]
Suppose there exist λ : R d × S → [0 , 1] , ˆ γ : R d × S → R d , and a σ -finite measure v on a measurable space ( S, S ) such that
[Equation]
We decompose S into S 1 ∪ S 2 such that 1 S 1 = 1 B 1 (ˆ γ (( s, x ) , y )) and 1 S 2 = 1 B c 1 (ˆ γ (( s, x ) , y )) .
We can rewrite the form of the generator is
[Equation]
Lemma A.3.
Let the generator L be the form of (16).
Let ξ be a Poisson random measure on [0 , 1] × S × [0 , ∞ ) with mean measure m × ν × m , and let (cid:101) ξ ( A ) =
ξ ( A ) − m × ν × m ( A ) .
Let ( S 0 , S 0 ) be a measurable space, µ a σ B -finite measure on ( S 0 , S 0 )
where σ B : R d × S
→ R d satisfies (cid:82)
[Equation]
|
[Equation] [Equation]
<
[Equation]
and
[Equation] [Equation]
Assume that for each compact K ⊂ R d ,
[Equation]
Then → X satisfies a stochastic differential equation of the form
[Equation]
Proof.
See Theorem 2.3 in [15].
Lemma A.4.
Let λ (( s, x ) , y )
=
p s ( x + y ) p s ( x ) σ αL ( s ) for σ L ( s ) ≥ 0 and ˆ γ (( s, x ) , y ) be (0 , y ) and ν ( dy ) be a Lévy measure such that it is a Borel measure on R d and ν ( { 0 } )
=
and (cid:82) R d ( | x | 2 ∧ 1) ν ( dx ) < ∞ .
If ( → X t ) has the corresponding generator L t
[Equation]
25) where u ∈ B b ( R d ) .
Then the corresponding generator ˜ L of the space-time process ˜
X t is ˜
[Equation] [Equation]
(27) where u ∈ C ∞ ([0 , 1] × R d ) .
Proof.
[Equation]
Theorem A.5.
A generator L t has a jump kernel driven by the 1-dimensional symmetric α -Levy process represented by (25).
ξ be a Poisson random measure on R + × R d × [0 , ∞ ) with mean measure m × ν × m such that E [ ξ ( dv × dy × ds )]
= dm × ν
( dy ) × dm and
˜ ξ
( A ) =
ξ ( A ) − m × ν × m ( A ) .
Then the SDE representation of the generator ˜ L satisfies
[Equation]
Proof.
λ (( s, x ) , y ) is
p s ( x + y ) p s ( x ) σ αL ( s ) for σ L ( s ) ≥ 0 and ˆ γ (( s, x ) , y ) is (0 , y ) with S 1 = {| y | < 1 } and S 2 = {| y | > 1 } .
We know λ satisfies
[Equation]
because (cid:82) S 1 | y | 2 ν ( dy ) < ∞ and (cid:82) S 2 ν ( dy ) <
∞ with sup x,y,s p s ( x + y ) p s ( y ) σ αL ( s ) < ∞ .
We set a ( s, x ) = 0 , so that σ B (( s, x ) , y ) is 0.
Therefore, for any compact set K ⊂ R 2 ,
[Equation]
λ (( s, x ) , y ) | r (( s, x ) , y ) | 2 ν ( du ) + (cid:90)
S(cid:90)
λ (( s, x ) , y ) | r (( s, x ) , y )
| ∧ ν ( du ) (cid:17) < ∞ .
since (cid:82) S 1 λ (( s, x ) , y ) | ˆ γ (( s, x ) , y ) | 2 ν ( du ) + (cid:82) S 2 (cid:82) λ (( s, x ) , y ) | ˆ γ (( s, x ) , y ) | ∧ ν ( du )) is well-defined and continuous with respect to ( s, x ) and b ( s, x ) is locally bounded R -valuded function.
We can apply Lemma A. 2 to the transformed homogeneous generator ˜ L of the inhomogeneous generator L t from Lemma A. 4 .
Now, we define Y t = (cid:82) s
= t s =0 (cid:82) |
y | < 1 (cid:82) s
[Equation]
ds )
[Equation]
y
[Equation]
ξ ( dv × dy × ds ) and Z t = (cid:82)
[Equation]
σ L ( s ) dL αs .
If we show
E [exp( i ( u, Y t )) =
E [exp( i ( u, Z t )) , then we can conclude dX t =
b ( t, X ( t ))
dt + σ αL ( t ) dL αt .
[Equation]
Since jumps y occur countably many,
[Equation]
for each t ∈ [0 , 1] .
Thus,
[Equation]
Since the characteristic function uniquely determines the probability distribution, we conclude
Y t =
Z t for almost everywhere (a.e).
If L αt is a d-dimensional α -stable Lévy motion, then we can apply A.5on the each component [ L αt ] i of L αt for i ∈ { 1 , . . .
, d } .
So far, we have proven that SDE representations can be found for inhomogeneous Markov processes that satisfy certain conditions.
Afterward, we will examine how the time reversal of a generator appears when a homogeneous Markov process is given.
Then Lemma A.5 is used to obtain the score-based reverse formula of Lévy-driven SDE.
We use the reversal formula of Theorem 5.7 in [5] to propose a new class of generative model, LIM.
Lemma A.6.
Consider a Markov process ( → X t ) t ∈ [0 , 1] with generator L t defined for any function u in set of continuous functions with compact support C 1 c ( R d ) such that L t u ( x )
=
b ( t, x ) · ∇ u ( x ) + (cid:82) R n [ u ( y ) − u ( x ) − ∇ u ( x ) · [ y − x ] δ ] → J t,x ( dy ) , ( t, x ) ∈ [0 , T ] × R n for some δ > 0 , where b ( t, x ) is a vector field, and the jump kernel is → J t,x ( dy ) .
Let [ x ] δ .
= 1 | x |≤ δ x Then, under some hypotheses, the Markov generator ← L t of the time-reversed process is
[Equation]
(34) where p t ( dy ) is a marginal distribution of ( → X t ) t ∈ [0 , 1] such that it satisfies p t ( dy ) ← J t,x ( dx ) =
p t ( dx ) →
J t,x ( dy ) for almost every t and the backward drift ← b ( t, x ) is given by
[Equation]
Proof.
See Theorem 5.7 in [5].
If we assume the marginal distribution has the density function p t ( x ) such that p t ( dx ) =
p t ( x ) dx and →
J t,x ( dy ) is a symmetric kernel with → J t,x ( dy ) =
v t ( y − x ) dy for some symmetric Lévy measure v t that is a Borel measure such that v t ( { 0 } )
=
and (cid:82) 1 ∧| y | 2 v t ( dy ) <
∞ for each t .
Then ← J t,x ( dy ) =
p t ( y ) p t ( x ) v t ( y − x ) dy .
Therefore (35) satisfies b ( t, x ) + ← b ( t, x )( t, x ) =
(cid:82) | y |≤ δ y · p t ( y + x ) p t ( x ) v t ( y ) dy .
Since v t is symmetric, δ can be ∞ such that
[Equation]
Thus, if the jump kernel has the symmetric Lévy measure v t then
[Equation]
[Equation]
Now, we will deal with 1-dimensional symmetric α -stable Lévy motion for simplicity.
1-dimensional α -stable Lévy motion have the symmetric Lévy measure ν of the symmetric α -stable distribution in
R such that ν ( dy ) =
[Equation]
dy
[Equation]
C =
[Equation]
So, we can use 36 to estimate the reverse drift term ← b ( t, x ) .
Since d-dimensional Lévy motion consists of independent components of 1-dimensional symmetric α -stable Lévy motions, we can easily extend 1-dimensional results for any d -dimensional cases.
Thus we first show our main results for the 1-dimensional cases and then extend the results for d-dimensional cases by applying the results component wisely.
Lemma A.7.
If a R -valued stochastic process ( → X t ) is a solution to d
[Equation]
then the jump kernel of → X t is
[Equation]
Proof.
See Lemma in [22].
By Lemma A.7, the reverse drift term of the R -valued solution ( → X t ) to d → X t = − β ( t ) α
→ X t + ( β ( t )) 1 /α dL αt satisfies
[Equation]
Therefore, the Markov generator ← L t of ( ← X t ) is the form of (25).
So, we can use Theorem A.5 to ← L t such that the reverse SDE of ← X t is d ← X t = − ← b ( t, x ) dt + σ αL ( t ) dL αt .
Now, we will calculate the exact form of ← b ( t, x ) represented by the integral.
For that, we derive a useful equation.
Lemma A.8.
(cid:82) ∞
sin x x α dx =
cos( πα
[Equation]
) · Γ(
[Equation]
−
[Equation]
Lemma A.9.
(cid:82)
[Equation]
Proof.
Let uy = k .
If u > 0 ,
[Equation]
If u < 0 ,
[Equation]
Therefore,
[Equation]
Theorem A
[Equation]
If d → X t = b ( t, → X t ) dt + σ L ( t ) dL
αt then the reverse SDE with respect to backward integral is d ← X t = − ← b ( t, ← X t ) dt 3 + σ ( t )
d ¯ L αt with ← b satisfying
[Equation]
Proof.
[Equation]
( α + 1) π .
since Γ(1 − α )Γ( α ) =
π sin πα and Γ( α +1) Γ( α )
=
α.
Thus, b ( t, x )+ ← b ( t, x )( t, x ) = σ αL ( t ) · α · ∂ α − 2 | x | ∂ x p t ( x ) p t ( x ) .
If a path measure Q on a measure space Ω is given, we denote q t as a marginal distribution of Q .
Then its forward carré du champ is the forward-adapted process defined by
[Equation]
[Equation]
where dom → Γ t = { ( u, v ); u, v, uv ∈ dom L t } .
And the IbP of the Carré du champ is that if u ∈ dom ← L and ← Lu ∈ L 1 (¯ q ) , then for almose every t
[Equation]
By equation 41, the proof of the time reversal formula relies on the integration by parts(IbP) formula for the carré du champ.
Thus the reverse formula depends on the form of the Carré du champ.
If the forward generator L t can be decomposed into L t =
L 1 t + L 2 t , then its Carré de champ also can be decomposed into → Γ t ( u, v ) =
→ Γ 1 t ( u, v ) + → Γ 2 t ( u, v ) such that → Γ 1 t ( u, v ) is the Carré du champ of L 1 t and → Γ t ( u, v ) is the Carré du champ of L 2 t .
Since Carreé du champ → Γ t is only determined by operator L t , and if it satisfies → Γ t ( u, v ) =
→ Γ 1 t ( u, v ) + → Γ 2 t ( u, v ) then
[Equation]
Additional drift term and others are derived from (cid:82) R n → Γ 1 t ( u, v ) dq t and (cid:82) R n → Γ 2 t ( u, v ) dq t
and if we know each term of (cid:82) R n → Γ 1 t ( u, v ) dq t and (cid:82) R n → Γ 2 t ( u, v ) dq t respectively, we can get the time-reversal formula of L t .
From this conclusion, we can induce the time-reversal formula of jump-diffusion processes.
Corollary A.10.
(The general reversal of SDE) .
The reverse SDE of d → X t =
b
( t, x ) dt + σ B ( t ) dB t + σ L ( t ) dL αt is
[Equation]
where ¯ B t , ¯ L αt is a corresponding backward Brownian motion and backward d-dimensional
α -stable
Lévy motion, respectively.
B Probability ODE
This chapter introduces the fractional Fokker-Planck equation, which is an extended version of the
Fokker-Planck equation in diffusion models into a symmetric α -stable distribution and obtains the existence of probability ODEs from the equation.
By deriving the probability ODE with the fractional derivative, the computational formula is obtained by using first-order approximation.
In order to prove the existence of probability ODE, we first define fractional calculus.
B.1 Fractional Calculus
Fractional calculus is a concept that extends the existing differentiation and has the characteristic that it satisfies (46) when Fourier transformation is performed.
Definition B.1 (Partial fractional Riesz potential) .
For α > − 1 and x = ( x 1 , . . .
, x d ) ∈ R d , we define the partial fractional Riesz potential ∂ | x | ( f 1 ( x ) , . . .
, f d ( x )) as follows [19] [24]:
[Equation]
such that
[Equation]
where F denotes the Fourier transform.
Lemma
[Equation] [Equation]
Proof.
[Equation]
B.2 Stochastic Calculus for Lévy-driven Stochastic Differential Equations
Lemma B.2 (Fractional Fokker-Planck equation) .
Given a Lévy-driven SDE, d → X t = b ( t, → X t ) dt + σ ( t ) dL αt for dL
αt = ( dL αt, 1 , . . .
, dL αt,d ) with set of independent symmetric α -stable Lévy motions ( L αt,i ) di =1 .
Then the marginal distribution p t ( x ) satisfies fractional Fokker-Planck equation
[Equation] [Equation]
Proof.
See Proposition 1 in [30]
Corollary B.2.1 (General Fractional Fokker-Planck equation) .
Given a Lévy-driven SDE and, →
X t ∈ R
d which satisfies
[Equation]
where dB t = ( dB t, 1 , . . .
, dB t,d ) with set of independent Brownian motions ( B t,i ) di =1 , and dL αt, 1 , . . .
, dL αt,d ) with set of independent symmetric α -stable Lévy motions ( L αt,i ) di =1 .
Then the marginal distribution p t ( x ) satisfies General fractional-Fokker-Planck equation,
[Equation]
Theorem B.3 (Existence of Probability Ψ DE) .
If p t ( x ) satisfies Fractional Fokker-Planck equation then p t ( x ) satisfies
[Equation]
51) such that F i
[Equation]
, x
[Equation]
So X t satisfies the ODE,
[Equation]
Proof.
[Equation]
Theorem B
[Equation]
4 (The general Probability Ψ DE) .
If p t ( x ) follows Fractional Fokker-Planck equation
[Equation]
then the transition function p t ( x ) satisfies
[Equation] [Equation]
such that F
i
[Equation] [Equation]
Therefore, → X t satisfies the ODE,
[Equation]
Proof.
[Equation]
C General OU process
Given a SDE driven by a d-dimensional α -stable Lévy motions L αt with [ L αt ] i ∼ S α S ( t 1 /α ) for each i ∈ { 1 , . . .
, d } such that
[Equation]
65) the solution of the SDE is
[Equation]
Since the each component of integral [ (cid:82) t 0 e − β ( t − s ) dL αs ] i is also a 1-dimensional symmetric α -stable ∼ S α S ( γ ( t )) for some γ ( t ) as [ L αt ] i is a 1-dimensional symmetric α -stable Lévy motion for each i ∈ { 1 , . . .
, d } .
We want to find the scale parameter γ ( t ) of (cid:82) t 0 e − β ( t − s ) dL αs for each t .
Lemma C.1.
Given α with 0 < α < 2 and f is a measurable function such that f : [0 , T ] → R with (cid:82) T 0 | f ( s ) | α ds < ∞ .
Let R -valued
→
X t = (cid:82) t 0 f ( s ) dL αs then
[Equation]
Proof.
If f ( t ) = (cid:80) N i =1 a i χ ( t i − 1 ,t i ] with t 0 = 0 , t N = t , → X t d = (cid:90) t
[Equation]
Using the above equation,
[Equation]
⇒ → X t ∼ S α S (cid:16)(cid:82) t 0 | f ( s ) |
α ds ) 1 /α (cid:17) .
Let us prove that f is not a simple function.
With the loss of generality, assume f ( t ) ≥ 0 .
If not, we decompose f ( t ) =
f
+ ( t ) − f − ( t ) such that f + , f − are non-negative functions.
Then we can construct an non-decreasing sequence of simple functions f n such that lim n →∞ f n ( t ) =
f ( t ) for all t ∈ [0 , T ] .
So, sup n f n ( t ) ≤ f ( t ) for all t .
Define
X nt ≡ (cid:82) t 0 f n ( s ) dL αs .
As (cid:82) T 0 | f ( s ) | α ds < ∞ , we can use dominated convergence theorem so that lim n →∞ → X n t ( w ) =
→ X t ( w ) for all w ∈ Ω , t ∈ [0 , T ] .
[Equation]
71) ∴
[Equation]
X
[Equation]
(cid:16)(cid:82)
[Equation]
when f is a measurable function.
This theorem can be extended for the solution of the SDE (65) driven by d -dimensional α -stable Lévy motions.
Theorem C.2.
If a ( t ) =
e − βt , γ ( t ) =
(1 − e − αβt ) 1 /α
= (1 − ( a ( t )) α )
1 /α and → X t = a ( t ) x 0 + γ ( t ) ϵ for some ϵ ∼ S α S
then X t is a solution to d → X t = − β
→ X t dt + ( α · β ) 1 /α dL αt and →
[Equation]
d
[Equation] [Equation]
Proof.
Use Lemma C.1.
Lemma C.3.
If → X t is a solution to d → X t = − β ( t ) α
→ X t dt + β ( t ) 1 /α dL αt , then X t can be represented by
[Equation]
If we define a ( t ) =
e − (cid:82) t 0 β ( s ) α ds , then the scale parameter γ ( t ) of (cid:82) t 0 e − (cid:82) tu β ( s ) α ds ( β ( u ) 1 /α dL αt satisfies γ α ( t ) =
(1 − a α ( t )) .
If β ( t ) =
β 0 +( β 1 − β 0 ) t then E
[ X t ] =
e −
[Equation] [Equation]
a
[Equation]
t ) x 0 , with log a ( t ) =
− ( β 1 − β 0 )[-
 α t-] 2 α
[Equation]
Proof.
[Equation]
→ X
[Equation]
= e
[Equation]
s
[Equation]
tu
[Equation]
s )
[Equation]
ds β ( u ) 1 /α dL αt .
If we set a ( t ) =
e − (cid:82) t 0 β ( s ) α ds then ddt log a ( t ) =
− β ( s ) α .
And the scale parameter γ ( t ) satisfies
[Equation]
Theorem C.4.
The partial fractional Riesz potential can be approximated by
[Equation]
If we only approximate this summation on k=0, then ∂ α − 2 | xi | ∂ xi p t ( x ) p t ( x
[Equation]
See Equation (4.1) in [18].
Corollary C.4.1 (Stochastic sampling of LIM) .
When t < s , ∆ t = s − t
[Equation]
where [ ϵ ] i ∼ S α S (1) for each i ∈ { 1 , . . .
, d } .
Proposition C.1.
[Equation]
where λ ( t ) = log a ( t ) γ ( t ) .
Theorem C.5 (Deterministic ODE sampling of LIM) .
[Equation]
Proof.
We apply Euler-Maruyama method to d → X t = ( − β ( t ) α
→ X t − Γ
[Equation]
α − 1) Γ 2 ( α/ 2) β ( t ) h
α − 2 ∇ log p t ( → X ))
dt .
For s > t , we can discretize the ODE such as
[Equation]
D Score Function for Lévy-It¯o Models
Lemma D.1.
Let q α be the density function of S α S and the value of X t satisfies x t = a ( t ) x 0 + γ ( t ) ϵ for given x 0 and [ ϵ ] i ∼ S α S for each i ∈ { 1 , . . .
, d } .
Then the score function of the transition distribution satisfies ∇ log p t ( x t | x 0 ) =
∇ log q α ( ϵ ) /γ
( t ) .
Proof.
Let → X t and Y be defined on a probability space (Ω , A , P ) where the transition density function of → X t is p t ( x t | x 0 ) =
d P ( → X t ≤ x t | → X t = x 0 ) dx t and the density function of Y is q α ( y ) =
d P ( Y ≤ y ) dy .
Let → X t = a ( t ) x 0 + γ ( t ) ϵ .
Then
P ( → X t ≤ x t | → X 0 = x 0 ) =
P ( a ( t ) x 0 + γ ( t ) Y ≤ x t ) since → X t = a ( t ) x 0 + γ ( t ) Y (80) =
P ( Y ≤ x t − a ( t ) xγ ( t ) ) (81) =
P ( Y ≤ ϵ ) (82)
Then p t ( x t | x 0 ) =
d P ( → X t ≤ x t | → X 0 = x 0 ) dx t
=
d P ( Y ≤ ϵ )
dϵ /γ ( t ) =
q α ( ϵ ) /γ
( t ) .
If we take logarithm of both sides then log p ( x t | x 0 )
= log q α ( ϵ ) − log γ ( t ) .
Therefore, we can get ∇ x t log p ( x t | x 0 ) =
∇ ϵ log
q
α
( ϵ ) /γ
( t ) .
Definition D.1 (Generalized Gaussian distribution) .
The generalized Gaussian distribution is two families of parametric probability distributions with a continuous path on R with a shape parameter ˜ β and scale parameter ˜ σ such that
[Equation]
where Γ( · ) is the Gamma function.
The score function of the Generalized Gaussian distribution is
[Equation]
Which is the same form of
ReELS
[Equation]
when ˆ β = ˜ β − 1 and ˆ c = ˜ β ˜ σ ˜ β .
D.1 ReELS
The principle behind the ReELS approaches to approximate the Lévy score function of the α -stable distribution is similar to that provided [17].
Because computing the Lévy score exactly requires higher computation complexity, score functions of generalized Gaussian distribution are employed [17] as an approximation technique.
ReLES is employed with a similar concept to conduct enough denoising at large noise to allow data to converge while maintaining the heavy-tailed features.
We empirically observe that ˆ β becomes an approximation with a value less than 1 when α is less than 2.
This means that a distribution with a score similar to each Lévy score is a generalized Gaussian distribution where ˜ β is less than 2.
D.2 Stochastic sampling for synthetic data
To show that the results predicted by the theory are valid, the performance of the model trained with the BM score and ReLES with synthetic data is compared with FID.
Additionally, it is tested with synthetic data if the synthetic data converges to the original distribution when using ReELS for LIM.
The synthetic data used as the test were two mixtures of Gaussian, Two moons, and swiss-roll.
In the case of two mixtures of Gaussian, the simplest MLP with a model depth of 3 was used, and for
Two Moon and swiss-roll, MLP with a model depth of 6 was used.
Detailed experimental settings are described in (Table D.1, D.2, D.3).
When using different score functions for each α = 1 . 2 , 1 . 5 , 1 . 8 , 2 , the generation ability of LIM was compared with FID.
As a result of the experiments, the mean value and variance of FID were low for the overall α in the case of using ReELS.
The stochastic sampling according to the time step when different α is given can be seen in (Figure E.3, E.8, E.9).
[Table]
a depth of a channel of [
[Table]
to 0.05.
The training data is 5000 pieces and the test data is 5000 pieces.
As a score model, an MLP model with a depth of 6 and a channel [3,32,64,64,32,2] is used.
β 0 is set to 0, β 1 is set to 5, and the clamp is set to 20.
It can be seen that the FID is low when
ReELS is used for all α = 1 . 2 , 1 . 5 , and 1 . 8 .
[Table]
D.3 Deterministic ODE sampling for synthetic data
In this chapter, we demonstrate the validity of the probability ODE (Theorem 3) from the deterministic
ODE sampling of LIM by showing the ability to generate synthetic data.
We train score models by using three synthetic data such as Two mixtures of Gaussian, Two moons, and Swiss roll.
The deterministic ODE sampling according to the time step when different α is given can be seen in (Figure E.10, E.11, E.12).
[Table]
E Dataset Experiment E.1 Implementation Detail
Our diffusion model is U-Net[21] following DDPM[11], which replaces weight normalization[23] with group normalization[32] for simple implementation.
We set the model size suitable for the dataset, such that MNIST ( 28 × 28 ) is [16, 32, 64], CIFAR10 ( 32 × 32 ) is [128, 256, 256, 256],
CelebA ( 64 × 64 ) is [128, 256, 256, 256, 1024]
[Table]
and CelebA-HQ ( 256 × 256 ) is [128, 256, 256, 256, 1024, 1024]
[Table]
but fix the number of residual blocks with 2 in each resolution level, and add self-attention block only in 16 × 16 resolution level.
Continuous diffusion time t ∈ [0 , 1) is injected into the model through Transformer sinusoidal position embedding[31] after adding with 0 . 0001 , and we use swish
function as the activation function
We train our MNIST model used in experiments for 1000 epochs with batch size 128, CIFAR10 model for 250 epochs with batch size 128, CelebA model for 140 epochs with batch size 128, and CelebAHQ model for 160 epochs with batch size 32.
All training and experiments are conducted on NVIDIA
A100 GPU and NVIDIA GeForce RTX 3090, and we tune the batch size for sampling adjusted for computation resources.
Because the target distribution of our model is α -stable distribution, sample quality is very sensitive to hyperparameter setting according to the α scale.
So we improve sample quality by tuning hyperparameters to be optimized for each dataset:
• Though DDPM[11] used linear noise schedule with fixed β 0 = 0 .
1 , β 1 = 20 , we tuned β 0 , β 1 for each α because variance of α -stable distribution depends on α scale.
For MNIST dataset, we fixed the β schedule as β 0 = 0 .
1 , β 1 = 5 in all α values.
In CIFAR10/CelebA/CelebA-HQ, we chose β 1 = 20 for α = 1 . 8 , and β 1 = 15 for α = 1 . 5 to optimize convergence into sample space, and fix β 0 to 0 .
• Different from Gaussian distribution, α -stable distribution can have large-scale noise at lower α values, which leads to sample quality degradation.
To prevent this problem, we used noise clamping as a heuristic in the training and sampling phase.
It consists of 3 clamps, clamp(training, init sample, SDE sample), and we adjusted the scale of clamps suitable for each dataset: – 10, 10, 10 at α = 1 . 8 , and 100, 50, 100 at α = 1 . 5 , 1 . 2 for MNIST.
– 30, 30, 30 at α = 1 . 8 , and 100, 50, 100 at α = 1 . 5 for CIFAR10.
– 10, 10, 10 at α = 1 . 8 , and 50, 50, 50 at α = 1 . 5 for CelebA and CelebA-HQ.
– In the case of deterministic ODE sampling, we only used clamp
(training, init sample), and set them to 10, 5.
E.2 Evaluation Metric E.2.1 FID(Fréchet Inception Distance) score
To evaluate generated sample quality, we choose the widely used FID score metric([10]), where a lower score means better sample quality.
After computing both mean/variance of distributions in the training dataset and generating 50k samples by using the pre-trained Inception-V3 model, we calculate distances between two distributions as FID score.
E.2.2 Likelihood computation
Our ReELS method is adaptive to probability ODE(Figure 4), so we can compute the exact likelihood on any input data in the same way as [28].
By replacing the score ∇ x log p t ( X t ) with score model
S θ ( X t , t ) , we can rewrite (3) as
[Equation]
Then we can compute the log-likelihood of p 0 ( X 0 ) such that
[Equation]
[Equation]
where X T is noise mapping to X 0 which can be obtained by solving the probability ODE in (85) with ODE solver.
Because of the expensive computation of ∇ · ˜ f θ ( → X t , t ) , we estimate it by using the
Skilling-Hutchinson trace estimator([25], [12]), which is following [8].
To solve the integral term in (85), we choose the RK45 ODE solver[7] which can be used as solve-ivp function in scipy.integrate library.
As same [8], we also set parameters atol=1e-5 , rtol=1e-5 .
We use a test dataset applied uniform dequantization, and take the average of the bits/dim values over 5 repeats for exact likelihood computation.
By changing initial time t 0 of integral (cid:82) T t 0 ∇ · ˜ f θ ( → X t , t ) dt after adding 0 .
001 , we compute bits/dim with varied number of function evaluations(NFE) like Figure 3(b).
E.3 Additional Samples
Additional sampling results on MNIST 28 × 28 , CIFAR10 32 × 32 , CelebA 64 × 64 , CelebA-HQ 256 × 256 are reported in below figures.
[Figure]
FID=0. FID=0.
FID=0. FID=0.
[Figure]
FID=1.
FID=0.
FID=0. FID=0.
[Figure]
Stochastic
[Figure] [Figure]
ODE
[Figure]
ODE
[Figure] [Figure]
FID=30.
FID=0.
FID=0. FID=0.
[Figure]