Learning White Noises in Neural Stochastic Differential Equations
Anonymous Author(s)
Abstract
Differential equations play important roles in modeling complex physical systems.
Recent advances present interesting research directions by combining differentialequations with neural networks.
By including noise, stochastic differential equations (SDEs) allows us to model data with uncertainty and measure imprecision.
There are many variants of noises known to exist in many real-world data.
Forexample, previously white noises are idealized and induced by Brownian motions.
Nevertheless, there is a lack of machine learning models that can handle suchnoises.
In this paper, we introduce a generalized white noise to existing models andpropose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian process.
Our experimental results demonstratethat the proposed model can capture noise characteristics such as continuity fromvarious time series data, therefore improving model fittings over existing models.
We examine how we can apply our approach to score-based generative models,showing that there exists a case of our generalized noise resulting in a better imagegeneration measure.
Introduction
Differential equations center around modeling relations between functions and their rate of change,playing important roles in many dynamical systems in physics, biology and finance.
Incorporatingsuch a tool in machine learning not only helps us understand more about machine learning models [41,65, 43] but also expands the range of applications [30, 72].
Recent work on neural ordinarydifferential equations (neural ODEs) open a new direction of connecting differential equations andneural networks [11].
Extensions to stochastic differential equations (SDEs) is proposed to handlingnoise in data or model uncertainty [83, 84, 53, 45].
Noises in data are often encountered in many real-world cases and can directly affect model performance.
The importance of understanding the characteristics of noise in data has been emphasizedin a number of study including: heteroscedastic noise modeling in time series analysis [82, 56],volatility modeling in financial research and market simulation [39, 13]; the significance of colorednoise in neuroscience [19, 7, 4].
Rather than using traditional Brownian noises [5] in neural SDEs,providing a capacity to model more generalized type of noises not only allows this framework toextend to such applications but also retains the expressiveness from representation powers of neuralnetworks.
By learning noises together with the drift functions and diffusion functions of SDEs, themodels become more flexible as the drift functions and diffusion functions, in turn, will be adjustedaccording to the learned noises.
Although mathematical research on analyzing general white noise iswell-developed [37, 32], it is still challenging to implement practically for machine learning modelslike [53, 45].
In this paper, we introduce a novel class of white noises defined under a stochastic integralform [37, 32] to existing neural SDE models [83, 84, 53, 45] which only consider noises generated by Wiener process.
The most prominent examples of such stochastic processes include fractional
Brownian motions (fBm) and multifractional Brownian motions (mBm) which are generalizationsof Brownian motions or Wiener processes.
The importance of such processes is emphasized inmany applications from financial modeling [12, 27], to motion tracking in physical and biophysicalsystems [28, 34, 40, 16].
Existing techniques on sampling these stochastic processes [17, 64] prohibitmodels to scale up as they require to keep track of sample paths during solving SDEs [53].
Toovercome this, we propose an approximation under numerical SDE solvers considering increments ata time.
That is, we first establish a discretized version of stochastic integrals to formulate distributionover noise increments.
The key approach is that we borrow the idea in sparse Gaussian processliterature [74, 36] to decouple the dependency between increments.
Therefore, we can reparameterizethe increments to obtain an equivalent SDE which is easy to solve.
The contributions of this paper are threefold.
First, we introduce a new type of neural SDEs withgeneral white noise.
Second, we propose an approximation of that white noise, which is inspiredby approximation methods in Gaussian process research to obtain equivalent neural SDEs.
Finally,we demonstrate empirically that our approach can capture noise characteristics and provide insightof how white noise affect learning score-based generative models as well as observe a setting thatimproves baseline models.
Related work
Differential equations and neural networks [11] presents neural ODEs using numerical methodin solving differential equations to perform learning parameters in neural network.
There are a numberof follow-up work aiming to improvements in different aspects including numerical solver and/ortraining process [44, 42, 23, 90, 60], implicit termination [10], theoretical understanding [21, 59],and extensions to graph neural networks [9].
Moreover, it has seen some applications in normalizingflows [89, 30, 38] as well as in control [20, 58].
Modeling time series data is one of successfulpractices of such models [72, 91].
Not close but related, work by [77] demonstrates excellent resultson image generation tasks.
Like SDEs are a stochastic extension of differential equations, neural
SDEs are natural extensions of neural ODEs when incorporating uncertainty into models.
However,the formulation is not always straightforward.
[83, 84] established theoretical aspects emphasizinghigh expressiveness of a class of neural SDEs while [53, 45] present how to train neural SDEsefficiently.
Many advanced tools in stochastic analysis like rough path [55] are incorporated intodifferential equations, and therefore empowered machine learning models [48, 47, 63, 62].
SDEs and Gaussian process There are equivalences and connections between Gaussian processand linear time-invariant SDEs that [80] discusses to a large extent.
[73] presents a faster learning
Gaussian process using filtering and smoothing methods [79].
Models of this research directionoften focus on spatial-temporal data and establish state-of-the-art results.
[35, 88] learn differentialequations with Gaussian processes instead of neural networks.
[18] considers a similar noise to oursas filtered white noises, finding an analytic form of corresponding kernels in the same manner as [73].
Unlike our approach focusing on a generic procedure, this work studies a specific class of kernels.
Gaussian processes can be a tool to model posteriors over paths [1] or to match with densities from
Fokker-Planck-Kolmogorov equations [75].
[54] devises a variational inference to approximateposterior inspired by work in sparse GP latent variables.
Background3.1 White Noise
Brownian motion has a long history dated back from the observation of pollen in water in [5], thenwas studied formally by Einstein [22].
In this paper, we focus on a generalization of Brownianmotions under Riemann-Liouville fractional integral defined by:
[Equation]
where h ( t ) : R → (0 , 1) is the Hurst exponent , W s is a Brownian motion and Γ( · ) is the Gammafunction.
When h ( t ) is a constant H , this stochastic process resembles a fractional Brownianmotion (fBm).
However, as noted in [57], it places a great emphasis on the origin ( t = 0 ) and nolonger possesses power-law spectral density like in fBm.
As we are interested in the case that h ( t ) is
[Figure]
approximated by a function t , we emphasize more on the local variation of Hurst exponent than such properties.
Ourformulation in (1) can be understood as a variant of multifractional Brownian motions (mBm) [68].
One of the goals in this paper is to learn h ( t ; θ ) using a function approximator, for example, a neuralnetwork with parameter θ .
Interpretation of Hurst exponent The stochastic process in (1) asymptotically shares someproperties with fBm and mBm [64] including long-range dependencies and self-similarity .
We referour further analysis related to (1) as fBm or mBm.
Here, long-range dependencies are defined for astochastic process if the sum of autocovariance function is unbounded (see Appendix A.4).
Timeseries analysis has a similar notion for fractional ARIMA models [78].
Deep learning approachesalso try to incorporate such long-range dependency to their models [31].
With the Hurst exponent h ( t ) , B t exhibits a long-range dependence when h ( t ) > 1 / 2 .
As h ( t ) = 1 / 2 ,the stochastic process falls back to a Brownian motion which no longer has long-range dependency.
When h ( t ) < 1 / 2 , the stochastic process is “rough”, anti-persistent, having irregularity.
Thisparameter also expresses Hölder continuity of stochastic processes.
If h ( t ) changes over time, thismeans all of such characteristics happen locally.
Brownian motion, B t , or Wiener process has a rich history in modeling physical system [66].
BlackScholes models [39], long-standing models in studying financial markets are based on Brownianmotions.
However, recent work shows fBm is more suitable due to the irregularity of volatility [13].
Correlated noise [7] also can be found in fMRI data.
Applications of fractional Brownian motionscan be found in [87] [14].
3.2 Background of Neural SDEs
Consider a dynamical system { X t } whose dynamic evolution is governed by the following stochasticdifferential equation
[Equation]
where µ : [0 , T ] × R n → R n is the drift and σ : [0 , T ] × R n → R n × m is the diffusion.
Both of thosefunctions are assumed to be well-behaved (i.e., smooth and linear growth) and are parameterized byneural networks.
Readers can refer to [80] for a gentle introduction of SDE in machine learning orto [71] for advanced materials.
[83, 84, 53, 45] are excellent starting points to be familiar with neural
SDEs.
Back-propagation using optimize-then-discretize strategy
This strategy is adopted in neural
ODEs, allowing to perform back-propagation in time [11].
Suppose we are interested in optimizing aloss function L at the terminal state X T .
This approach considers a variable ∂ L /∂X t .
The dynamicof this variable can be modeled by a differential equation leading to computing gradient by usingchain rules.
In neural SDEs, such a strategy requires a more careful treatment as [53] addresses the randomness in∂
L /∂X t .
Also, forward and backward SDEs need to share the same sample paths of Wiener process.
Model learning There are two main parameter estimation methods for Neural SDEs.
The firstone is latent SDE [53] that follows a variational approach approximating posterior distributions by a neural SDE which shares the same diffusion with priors.
In this framework, the evidence lowerbounds are derived by applying the Girsanov theorem.
Recently, [46] proposes SDE-GAN inspiredby generative neural networks [29].
In this paper, we mainly use the former in our experiments.
However, it is worth noting that there is no restriction to training with SDE-GAN as long as chosenmodels satisfy our assumption.
3.3 Sparse Gaussian processes
We also give a brief introduction of sparse Gaussian processes which are known as scalable methodsfor Gaussian processes.
A Gaussian process (GP) [70] is defined by a set of random variables, anyfinite number of which have a joint Gaussian distribution.
Formally, a GP is completely specified by amean function m ( · ) which usually is set to be zero for prior and covariance function k ( · , · ) .
We thencan denote a GP by f ( x ) ∼ GP (0 , k ( x, x ′ )) .
Learning or sampling from a Gaussian process usuallytakes O ( N 3 ) with the number of data points N .
Sparse Gaussian process methods [74, 36] introducea set of pseudo points (inducing points) u at locations Z which jointly follow a Gaussian distributionwith data points.
The density function of f ( x ) , p ( f ( x )) is then approximated by p ( f ( x ) | u ) p ( u ) .
Here the mean and the variance of p ( f ( x ) | u ) can be computed as
[Equation]
where α ( x ) =
k ( Z, Z ) − 1 k
( Z, x ) .
There is a growing interest in this approach because of itsscalability compared to traditional GPs [6, 3, 86, 36].
In this paper, we focus on the samplingprocedure where we first sample u ∼ N (0 , k
( Z, Z )) , then sample f ( x ) ∼ p ( f ( x ) | u ) .
Learning White Noises in Neural SDEs
This section presents our main model.
We first approximate stochastic integrals using Stieltjesapproximations and then use sparse Gaussian processes to leverage dependencies between randomvariables.
We analyze the convergence of SDEs solutions for this approximation.
Finally, we form anequivalent SDE to our model (see Figure 1).
4.1 Neural SDEs with B t
Consider a stochastic differential equationdX t = µ
[Equation]
t, X
[Equation] [Equation]
+
[Equation]
( t, X
[Equation]
dB t , t ∈ [0 , T ] , (4) with µ : [0 , T ] × R n → R n and σ : [0 , T ] × R n → R n × m .
Compared to (2), the noise comingfrom B t is an m -dimensional stochastic process, B t = (cid:82) t 0 f
( t, s ) dW s from (1) as we denotef
[Equation]
We are interested in learning both the parameters of µ ( t, X t ) , σ ( t, X t ) and that of f ( t, s ) .
Solving
SDEs in (4) is challenging since B t is constructed from an integral.
Direct sampling methods of
B t (i.e., require computing Cholesky decomposition) are costly.
Other approaches [64] use anapproximation of stochastic integral as a linear combination of random variables.
However, thenumber of these random variables prevents optimize-then-discretize methods to scale.
4.2 Integral approximation
We use Stieltjes integration to approximate the integral in (1) by discretizing the interval [0 , T ] with0 =
s 1 < s 2 < · · · < s N = T , i.e.,
[Equation]
where ξ i are independently distributed standard normal random variables coming from W s i +1 − W s i ∼
N (0 , √ ∆ s i ) with ∆ s i =
s i
+
− s i , and 1 { s ∈ [0 , t ] } is the indicator function.
Note that this is areasonable approximation because Itô’s integrals, in general, converge in L 2 when N goes to infinity.
Moreover, even if W t has infinite variation, the probability of a sample path leading to the divergenceof approximation infinitely often is zero [71].
The discretization is also inevitable whenever it isrequired to sample B t .
In several numerical approaches, for example, Euler-Maruyama [50], solving stochastic differentialequations involves approximating increments, dB t , by
[Equation]
where ∆ g i ( t )
=
g i ( t + ∆ t ) − g i ( t ) .
Comparing to increments ∆ W t when solving (2), simulating∆ W t is relatively simple as they are independent between each time step.
On the other hand, thereexist dependencies between ∆ B t as
[Equation]
Here, ξ i , ξ j ∼ N (0 , 1) , and E [ ξ i ξ j ] = 0 when i ̸
=
j .
Our aim is to represent (6) in a way that issimple to sample from but preserves the dependencies between ∆ B t .
4.3 Sampling increments with sparse Gaussian process
From the observation in (7), we devise a sampling method inspired by sparse Gaussian processes.
Inparticular, we consider a Gaussian process 1 B ′ t =
∆ B t ∆ t for a fixed ∆ t .
In fact, ∆ t is later used asthe time step of SDE solvers.
Note that directly modeling ∆ B t in (6) is numerically unstable whencomputing (3.3) with covariance matrices having small values.
Indeed, B ′ t is a Gaussian process because at a given time t , B ′ t is Gaussian with a zero mean.
Tocompute the covariance function k
( t, s ) , we simply compute k ( t, s ) =
E [ B ′ t B ′ s ] − E
[ B ′ t ] E [ B ′ s ]
= 181 E [ B ′ t B ′ s ] and follow the same step with (7)
[Equation]
To this end, we denote B ′ t ∼ GP (0 , k ( t, s )) .
Sampling from a Gaussian process based on a fullcovariance matrix is not scalable.
To overcome this difficulty, we propose a sampling scheme basedon sparse-inducing GP [74].
The main idea of this method is to introduce a bottleneck as a summationof the joint distribution of B ′ t .
Define inducing-point locations as Z = { z 1 , . .
. , z M } which spread across [0 , T ] .
The number ofinducing points, M , is much smaller than the number of discretization mesh of numerical solvers.
The inducing points u
=
( B ′ z 1 , . .
B ′ z M ) are distributed as a multivariate Gaussian distribution, i.e.,u ∼ N (0 , k
( Z, Z )) .
Following the description of sparse GPs in §3.3, p ( B ′ t ) is approximated byp ( B ′ t | u ) p ( u ) .
The sampling procedure of B ′ t consists of two steps:
[Equation]
Here L Z is the Cholesky decomposition of k ( Z, Z ) , I M is an M × M identity matrix.
m ( · ) and v ( · )are the conditional mean and variance taken from (3).
The above is a complete sampling procedure for p ( B ′ t | u ) p ( u ) .
Note that in our approximation, B ′ t isindependent conditioned on that u are given.
Therefore, we obtain the increment ∆ B t as
[Equation]
Here, we replace √ ∆ tξ t with increments of a (new) Brownian motion W t since ξ t are independentfor t ∈ [0 , T ] .
Remark One may consider an alternative approach where we use sparse approximation directly for
B t with given kernel function k ( t, s )
∝ | t | 2 h t,s + | s | h t,s − | t − s | 2 h t,s (see (16)), and can obtain asimilar form of increments.
However, this kernel function is not differentiable w.r.t the parametersof Hurst function.
Our approach is to devise a “so-called” differentiation of B t under discretization.
Note that this notion differs from the definition of the derivative of B t in [51] in the sense of Schwartzspace.
However, such a representation is not useful for a practical implementation.
Our approach shares a similarity with [48] on controlled differential equations driven by a signal, i.e.,
X t .
The signal X t is approximated by linear interpolations which allows to compute the derivativeof X t .
We intentionally model the “derivative” of B t by B ′ t first, and then reconstruct the signal B tfrom B ′ t .
4.4 Forming equivalent SDEs
To this end, we show that the increments ∆ B t can be represented with increments of a Wiener process,∆ W t in (11).
Here, the equivalence is under the assumption that the numerical procedure involvesonly ∆ B t , i.e., Euler-Maruyama methods.
Therefore, the original SDE in (4) is approximated by
[Equation]
This is obtained by conversions, dt ≈ ∆ t and dW t ≈ ∆ W t as we expand the diffusion term
[Equation]
where ν ( t ; u, Z ) = m ( t ; u, Z ) , and ς ( t ; Z )
= v ( t ; Z ) √ ∆ t .
The function ς ( t ; Z ) induced from theapproximation of B ′ t is now absorbed into the diffusion function while ν ( t ; u, Z ) together with thediffusion function now is added to the drift function.
Note that ν ( t ; u, Z ) and ς ( t ; Z ) is conditionedon the inducing point u .
Therefore, at the initial time t = 0 , we need to precompute u first.
Although (12) and (4) seem similar in terms of the expressiveness of drift functions and diffusionfunctions, the drift function of (12) contains a randomness stemming from the sampling u in (9) whilethat of (4) is deterministic.
This can be understood that the dependency between ∆ B t is translatedinto the drift functions.
Note that ν ( t ; u, Z ) and ς ( t ; Z ) are associated with two types of discretizations: that of approximating
Stieltjes integral and that of SDE numerical solver.
One of the key advantages of this new SDE is that it can inherit all theory and practices of wellestablished study of stochastic analysis for Brownian motion or Wiener process.
The preliminaryresults of Girsanov’s theorem, i.e. [33] cannot used in latent SDE [53].
Itô’s rules for fBm and mBmcan be found in [51] but hard to use in practice.
On the other hand, our approach can work for any B tdefined in (1).
Another benefit of our formulation is that the sampling procedure is simple, requiring only one ξ tat a time and avoids a multiplication of O ( N ) memory like [64].
This becomes significant as theoptimize-then-discretize method requires constructing the same sample paths of B t in forward SDEsand backward SDEs.
Computational analysis The evaluation of diffusion functions in (12) requires computing an aggregate sum over discretization that scales linearly with the number of discretization points.
It may causesome overhead when computing Itô correction term during solving backward SDE [53].
Therefore,
Stratonovich SDEs are more efficient than Itô SDEs.
One limitation of integral approximation is thatfor a long interval [0 , T ] , one may need more discretized partitions to achieve desired approximationaccuracy.
On the computational cost of sparse GPs, we only need to perform the Cholesky decomposition atthe initial time of every SDE integration.
This takes O ( M 3 ) with M the number of inducing points.
At a time t during solving SDEs, with the pre-computed L Z , sampling B ′ t has O ( N + M 2 ) timecomplexity as we perform matrix multiplications in (3).
Limitation Currently, our model only considers fixed step sizes in SDE numerical solvers.
Mosttheoretical studies in stochastic analysis work for H > 1 / 4 ( H > 1 / 3 in this study) to make sense ofstochastic integrals or SDE solutions [69].
The Euler-Maruyama update in our approach may not besufficiently good when the Hurst exponent is less than 1 / 2 as it is pointed out in [15].
Higher-ordersolvers can be more accurate, requiring handing double integrals, i.e., (cid:82)
(cid:82) dB t dB s which we wish toaddress in future work.
Although this paper presents an initial theoretical justification of convergenceto solutions, the precise order of convergence is not clear yet.
Therefore, for practical use, werecommend avoiding using Hurst exponents with small values in our method.
4.5 Convergence analysis
The error of our approximation is composed of three main factors.
The first is of SDE solvers.
Due to the assumption of fixed step sizes, only a few solvers, i.e., Euler-Maruyama, are applicablewith 1 / 2 strong convergence.
Secondly, the discretization in Stieltjes approximations attributingto the total error leads to the convergence in L 2 .
The convergence of such approximations areknown.
The approximation of noise from sparse GPs is unconventional in solving SDEs becausethe approximation is in distributional sense rather than point-wise or path-wise.
We will show theconvergence using rough path theory [55, 25] and sparse GPs bound from [8] for constant Hurst
H ∈ ( 1 / 3 , 1 / 2 ) .
We follow the notation in [38, 55, 25].
Let us denote the canonical lift of B as B = ( B, B ) .
Let(cid:98)
B be the path corresponding to the sampling procedure in (9) and (10).
Let X be the solution ofd X t = f
( t, X t ) d B t and (cid:98)
X be the solution of d (cid:98) X
= f ( t, (cid:98) X t ) d (cid:98) B where f : [0 , T ] × R n → R nis a bounded function that is four-times continuously differentiable.
We assume that the initialstate follows a distribution p 0 which is is continuous and strictly positive in R n .
Let us denote theprobability density distribution of the solution X at time t by p t ( x ) ; and
that of approximated solution(cid:98) X by (cid:98) p t ( x ) .
Theorem 1 (Convergence of solutions given sparse GP approximation) .
With the Hurst exponentof B , H ∈ ( 1 / 3 , 1 / 2 ) , assume that E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ for any t ∈ [0 , T ] , and ϱ ∈ ( 1 / 2 H , 3 / 2 ] .
Then, for every α ∈ ( 1 / 3 , 1 / 2 ϱ ) , there exists a constant C such that
[Equation]
Here, θ ∈ (0 , 1 / 2 − ϱα ) and KL( ·∥· ) is the Kullback-Leibler divergence.
This theorem says that if the distribution of (cid:98) B converges to that of B in terms of the KL divergence,their corresponding solutions converge in the same sense.
Note that we make an assumption that
E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ , ϱ > 1 / 2 H which means that (cid:98) B is less “rough” than B .
This is areasonable assumption since (cid:98) B suffers information loss from sparse GP approximations.
Our proof is done by (1) assessing the p -variation of (cid:98) B to bound the distance on rough path spacebetween the canonical lifts of B and (cid:98) B via [25, Corollary 10.6]; and (2) using the local Lipschitzproperty of Itô-Lyons maps [25, Theorem 8.5] as we follow the technique in [38].
The completeproof is provided in Appendix B along with necessary backgrounds and discussion.
According to [8], we can quantify the convergence of (cid:98) B to B as its KL divergence can be bounded,
KL( (cid:98) B ∥ B ) ≤ C (cid:80) ∞
i =
M +1 λ i .
Here λ i are eigenvalues in eigenvalue-eigenfunction pairs ( λ i , ϕ i ( · ) )of Mercer’s theorem [61], k ( x, x ′ )
= (cid:80) ∞
i =
λ i ϕ
i ( x ) ϕ i ( x ′ ) .
Recall M is the number of inducingpoints.
Therefore, the convergence of KL( (cid:98) B ∥ B ) depends on the decaying rate of eigenvalues andthe number of inducing points.
We empirically show the fast decaying rate of eigenvalues for finitesamples which also can be found in Appendix B.
4.6 Parameterizing white noise
The previous section concerns how to learn white noises in neural SDEs without providing concrete forms of white noise.
This section will show how it affects SDEs in general and discuss the parameterization form of the Hurst exponent.
[Equation]
When the Hurst exponent h ( t ) is constant, say h ( t )
=
H .
There is a qualitative difference between settings of H .
Figure 2 shows how the Hurst parameter affects the modelfit trained by Latent SDE.
With a smaller value ( H =0 . 3 ), sample paths are rough, producing more irregularity.
When H = 0 .
8 , the model has smoother sample paths,and is more conservative as predictions depend on the past.
This parameter has a great influence on the uncertaintyquantification of the models.
For the case that h ( t ) is function of t , any function approximators can work in general to model h ( t ; θ ) .
In ourimplementation, we use small neural networks.
With thenew representation in (12), we can train the parameters ofh ( t ; θ ) together with the parameters of drift and diffusion.
A more general approach is that f ( t, s ) can accept eitherneural networks or kernel-based functions when we do notconsider normalization.
However, their interpretation isnot clear as that of mBm.
Experimental Results
This section will first justify how close the samples generated by our method are to true ones.
Wethen investigate the ability to learn the Hurst function for synthetic data and test our model for
[Figure]
real-world data.
Finally, we examine the role of Hurst exponents in score-based generative models.
All experiments are implemented using Pytorch [67] with the library torchsde [53, 46].
Our sourcecode is available at [redacted].
How to estimate Hurst from data?
We follow [26] to empirically estimate Hurst exponent fromdata.
The key technique is based on the correlation of increment of B t and estimates a ratio oversliding windows which is an estimation of Hurst exponent.
5.1 Quality of approximate samples
[Figure]
We would like to investigate if our approximation of B ′ t can produce the sample path of B t that is close enough to those of the trueprocess.
The true Hurst function is set as h ( t ) = 0 .
35 + 0 .
5 t .Thesamples from [64, 24] are used as a baseline.
This experimentruns 1000 samples.
The estimations of the Hurst exponent fortwo sampling techniques are in Figure 3.
The Hurst estimationcorresponding to our samples is close to the true Hurst function.
It tends to overestimate the long-range dependency ( h ( t ) > 1 / 2 )and underestimate the irregularity ( h ( t ) < 1 / 2 ) in the samples.
5.2 Recovering Hurst function
[Figure]
Next, we test our model with a synthetic data.
Here the data is generated from a SDE with a known solution:
[Equation]
where B t is a mBm given a fixed h ( t ) = 0 .
3 +0 .
5 sigmoid (7(1 − t )) .
According to Itô integralsfor mBm cases [51], the exact solution is X t =x 0 exp( αt − 12 β 2 t 2 h ( t ) + βB t ) .
We generate 100data points for each sample path and obtain up to 5sample paths.
After training our model, we observethat the obtained Hurst function can capture the dynamic similar to the true Hurst.
As we increase thenumber of sample paths (from 1 to 5 ), our learned
Hurst tends to approach the true Hurst.
Figure 4 shows an example of our learned Hurst functiongiven 5 sample paths.
We also provide the results for different Hurst functions and an ablation studyin Appendix C.3.
5.3 Financial data
In this experiment, we compare the Hurst function found by our model with the time-dependent Hurstestimation computed by [26].
Data set We collect two months (from September 26th, 2021 to October 23rd, 2021) of hourly datafrom four stock indices Amazon (AMZN), Apple (APPL), Google (GOOGL), Microsoft (MSFT).
The data is retrieved from [85].
It is then standardized and split into the first 80% time points to trainand the remaining 20% to test.
Obtaining Hurst function Figure 5 compares our found Hurst functions and estimated Hurstcomputed from [26].
Our obtained Hurst functions have bigger values than that of [26].
Thisresembles the observation in the previous experiment.
A common observation among these stocksis that Hurst functions made some changes (bump up or drop) around the end of October.
Thisagrees with many analyses, for example, Amazon missed revenue expectations for the third quarteror Apple revealed new products.
The posterior plots for our model in each stock index are placed in
Appendix C.4.
Quantitative comparison We compare our model to two main baselines: Latent SDE with standard
Brownian motion [53] and Latent ODE [72].
We report three main comparison criteria: trainingnegative log-likelihood (training NLL), test negative log-likelihood (test NLL) and test root meansquare error (test RMSE).
Overall, our model can fit training data well as our model outperformsothers in terms of training NLLs (see Appendix C.4).
This is because our model can capture correctlynoise part while Latent SDE relies on less flexible Brownian noise.
Still, it is not clear to identifywhich model performs better in predictive tasks.
However, our model produces more consistentresults with less variance.
5.4 Effect of Hurst exponent in score-based generative models
This experiment explores a different aspect by studying the effect of Hurst exponent in trainingscore-based generative models.
Recent work [77] on generative model based on score functionsand SDEs presents impressive results on par with generative adversarial networks (GAN).
Weconsider the case of perturbation in data under a new SDE, dX t = − 12 β ( t )
X t dt + (cid:112) β
( t ) dB t where
B t = (cid:82)
[Equation]
0 f ( t, s ) dW s .
We derive the equivalent SDE (discussed more in Appendix C.5):
[Equation]
Varying Hurst exponents We investigate how different types of noise affect learning and samplingin this model.
We consider four scenarios of noise: (1) more anti-persistent as H = 0 . 3 ; (2) longrange dependent with H = 0 . 8 ; (3) linearly increasing h ( t ) ; (4) linearly decreasing h ( t ) .
We onlyconsider the task of generating MNIST images [52].
Figure 6 shows example of generated images.
We observed that H > 1 / 2 is not favorable, and requires SDEs with long terminal time T .
Toquantify the quality of generated images, we compute the negative log-likelihood (NLL) of a set ofout-of-sample images like in [76]. H = 0 . H = 0 .
[Figure] [Figure]
[Equation]
Discussion Although we do not present a way to find optimal h ( t ) , we show that there is a settingin which the NLL appears to be better (see Figure 7 a).
That is, the NLL of decreasing Hurst setting is3 .
79 (bits/dim) compared to 4 .
97 (bits/dim) of Brownian motion ( H = 0 . 5 ).
There exists work [49]showing that signal-to-noise ratio (SNR) is the key to establishing an optimal strategy how to injectnoises.
There is another work studying the best strategy for reverse variance [2].
In contrast to manydiffusion models, our approach has an additional term (cid:112) β ( t ) ν ( t ; u ) which is unrelated to X t but isincluded in the drift of the backward SDEs when generating data (see Figure 7 b).
Moreover, ourapproach can suggest a way to control the sample path of B t in the way that we condition on inducingpoints u since B ′ t is sampled from the conditional distribution p ( B ′ t | u ) .
For example, we can followthe variational inference of [36] to find the posterior q ( u ) for a designed objective, then sample B ′ tgiven q ( u ) instead of p ( u ) .
Conclusion
In conclusion, this work presents a new class of neural SDEs with a generalization of white noise.
We propose an efficient approximate sampling approach for the noises so that neural SDEs can beconverted into convenient forms.
The techniques used in this paper are relatively simple, and relatedto Gaussian process (GP) research.
Our experiments have verified the ability of our models to learnnoises from synthetic data to financial time-series as well as showcase an image generation task.
As mentioned earlier, this work can improve the current model with higher-order solvers where thedouble integrals require to model precisely.
Prior work [51, 69] is a good starting point for suchan approach.
Another promising direction is to extend neural controlled differential equations withdriving signals which are general GPs instead of Brownian motions.
The closure property of GPsunder differentiation will be useful to develop this model.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing
the
appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See the last paragraph ofSection 4.4(c)
Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Mentioned inthe main text and more detailed in Appendix B(b)
Did you include complete proofs of all theoretical results?
[Yes] Complete proof canbe found in Appendix B3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes]
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes](c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
[Table]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] MNIST dataset for
[Table]
image generation tasks.
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
Notation
Meaning
[Table]
evaluated
[Table]
GP
Gaussian process
[Table] [Table]
for our
[Table]
solving
SDEs Here, we provide our approachin Algorithm 1 in contrast to the SDEs with Brownian motion in Algorithm 2.
[Table]
Output: Solution X t 1 at t/* Compute drift and diffusion in Eq.
(12) */ 1 def f_and_g( t , x ) : /
Python syntax: global */ global inducing points u /* Mean and variance of GP approximation in Eq.
(3) */ m ( t ; u ) , v ( t ; u ) = GP_approx
( t , u, h ( · ) )
ν ( t ; u ) =
m
( t ; u ) ς ( t ; u ) = v ( t ; u )∆ t drift = µ ( t, x ) + σ ( t, x ) ν ( t ; u ) diffusion = σ ( t, x ) ς
( t ; u ) return drift, diffusion 10 def brownian_increment( t ) : return Brownian noise ∆ W t /*
Use Eq.
(9) and (10) */ 13 inducingpoints u
= sample_inducing( Z , h ( · ) ) /* call SDE solver */ 14 X t 1 = sdeint(f_and_g , brownian_increment , X t 0 , t 0 , t 1 ) 15 return X t
Input: Drift function µ ( t, x ) , diffusion function σ ( t, x ) , initial value X t
Output: Solution X t 1 at t/* Compute drift and diffusion directly */ 1 def f_and_g( t , x ) : return µ ( t, x ) , σ ( t, x ) 4 def brownian_increment( t ) : return Brownian noise ∆ W t /* call SDE solver */ 7 X t 1 = sdeint(f_and_g , brownian_increment , X t 0 , t 0 , t 1 ) 8 return X t
Model training with Latent SDE The training method used in experiments of § 5.2 and § 5.3is adopted from [53] via a Bayesian approximation method.
Consider two SDEs sharing the same diffusion function
[Equation]
Applying our approach like in (12), we can obtain new drift and diffusion functions (with conditionedon inducing points) for SDEs with Brownian motion.
Let us write in this form
[Equation]
where ¯ µ θ , ¯ µ ϕ , ¯ σ are derived similar to (12).
The variational bound requires computing the KL divergence between the approximate posterior and prior which is related to the change of measures between prior paths and posterior paths, therefore, is obtained by Girsanov Theorem.
That is, with a well-defined u ( t, x ) satisfying ¯ σ ( t, x ) u ( t, x ) =
¯ µ ϕ ( t, x ) − ¯ µ θ ( t, x ) , the change of measure is written as
[Equation]
Finally, the variational bound consists of the log-likelihood and E [log M t ] .
Details of the learningalgorithm can be found in [53].
A Fractional Brownian Motion and Multifractional Brownian Motion
This section contains additional background of fractional Brownian motions (fBm) and multifractional
Brownian motions (mBm) including their covariance function and a detail description of long-rangedependency in such processes.
A.1 Fractional Brownian Motion
Fractional Brownian motion can be viewed as a Gaussian process with covariance
[Equation]
Here, H is called the Hurst exponent.
It has some properties: (1)long-range dependence ( H > 1 / 2 );(2) anti-persistence or irregularity ( H < 1 / 2 ).
Such properties are interesting to applications inmodeling internet traffic, highly textured images.
A.2 Multifractional Brownian Motion
Multifractional Brownian motion B ( h ) is a Gaussian process with covariance [51]
[Equation]
16) with h : [0 , T ] → (0 , 1) , h t,s :=
h ( t )+ h ( s ), and c x = (cid:16) 2 π Γ(2 x +1)sin( πx (cid:17) 12 .
This extension is to modelcases that behaviors of signals happen locally.
If h ( t ) is close to 0 , we expect irregular local behaviors.
If h ( t ) > 1 / 2 at time t , we expect local long-range dependence.
A.3 Ito Integral for Multifractional Brownian Motion
Traditional Brownian motion case The Itô integral is computed as
[Equation]
Multifractional Brownian motion case The Ito integral has additional derivative of kernel functionf ( T
[Equation]
[Equation] [Equation]
The red term ddt R h ( t, t ) is the derivative of variance function of mBm.
This can be interpreted as thesecond derivative term in Itô’s formula, ∂ 2 f ∂x 2 ( t, B ( h ) t ) , is adjusted according to how fast the variancefunction changes when time goes.
Note that ddt R h ( t, t ) =
2 t 2 h ( t ) − 1 ( h ′ ( t ) t ln( t )
+ h ( t ))
Thisresult is used in the main text to get the analytic solution of dX t = αX t dt + βdB t
A.4 Properties of Fractional Brownian Motion
As discussed in the main text, fBm exhibits distinct properties including self-similarity and long-rangedependency.
We provide a glimpse of these notions to readers.
Self-similarity Simply speaking, self-similarity in general refers to objects which look exact or approximately similar to a part of itself.
This concept is related to fractals.
In statistics, consider a discrete stochastic process { X t } t ∈ N .
We say X is self-similar if a finite sum, i.e., X km + · · · +
X ( k +1) m − 1 and a scaled version a m X k has the same distribution.
In a restricted definition when a m is just a polynomial of m , we say X ( m ) k
= 1 m ( X km + · · · + X ( k +1) m − 1 ) and m 1 − H X ( m ) are distributionally equal or ( X k 1 , . .
, X k d ) distributionally equal to ( m 1 − H X ( m ) k 1 , . . .
, m 1 − H X ( m )
k d ) if X is self-similar.
Now, consider the noise with respect to fBm, defined as
[Equation]
The self-similarity of X is expressed by comparing mX ( m ) and m H X .
Both of them are zero-mean
Gaussian, having the same covariance because
[Equation]
Long-range dependence Long-range dependence in stochastic processes means the sum of the autocovariance function is unbounded.
For fractional Gaussian noise, X k = B Hk +1 − B Hk has the autocovariance function
[Equation]
This can be approximated as γ ( k ) ∼ H (2 H − 1) k 2 H − 2 , leading (cid:80) ∞
k =1 γ ( k ) = ∞
when 1 / 2 <
H < 1 .
Perhaps, fractional ARIMA is a more intuitive model to understand long-range dependency.
While well-known ARIMA model is defined as
[Equation]
where L is the lag operator as LW k = W k − 1 , ϕ
and θ are polynomials.
By introducing
[Equation]
we can construct fractional ARIMA model.
The long-range dependence is expressed by the infinitesum in the right hand side.
Here, Hurst parameter is H =
d + 1 / 2 .
B Convergence analysis
In this section, we provide some backgrounds of rough path theory.
Our proof will investigatethe variation of covariance of Gaussian processes and sparse Gaussian processes which help us tojudge their Hölder continuity.
Then, we use results of rough path theory [55, 25] to bound the KLdivergence between solutions via the KL divergence fractional Brownian motions and their sparse
GP approximations.
Finally, we use [8] to justify the convergence of sparse GPs.
B.1 Rough path theory
Here we give a brief review of rough path theory.
For a complete introduction, readers can referto [25].
Notation We denote X s,t :=
X t − X s and (cid:82) t s X s,r dX r :
= X s,t .
Also, p -variation seminorm is defined as
[Equation]
where P is a set of disjoint intervals of some partitioning scheme over [0 , T ] .
We will write x ≲ y toexpress x ≤ Cy for x, y > 0 .
In this section, we will sometimes write the covariance as R ( s, t ) .
A rectangular increment ofcovariance is denoted by
[Equation]
The p -variation of a rectangle increment is written as ∥ R ∥ p ;[0 ,T 2 .
The space of rough paths The following will provide the specification of the space of rough pathsincluding its norm and its metric.
Definition 2.
Given α ∈ (cid:0) 13 , 12 (cid:3) , the space of α -Hölder rough paths is defined by
• pairs ( X, X
[Equation]
• For any X and Y , (inhomogeneous) α -Hölder rough path metrics is
[Equation]
The space of geometric rough paths Many results in rough path theory require rough paths being geometric where the following holds
[Equation]
For Brownian motions, Stratonovich integration leads to geometric Brownian rough paths.
There is away to convert between Itô and Stratonovich version of Brownian motions (see [38, §3]).
A similartechnique can be achieved for fBm by [69].
Itô-Lyons map One of interesting results in rough path theory is Itô-Lyons maps.
Suppose we have a solution map
[Equation]
Simply speaking, while Ψ is universal as it is just an enhancement of B , the solution map S is acontinuous map on both the initial conditions and driving noise.
This property does not hold for Itômaps from sole Brownian motion paths to solution paths.
B.2 Main proof
Before going to the main proof, we provide the Gaussian process form of sample paths (cid:98) B t corresponding to B ′ t described in (9) and (10).
We can understand (cid:98) B t is the result of applying integrationoperator on B t (with small numerical error).
Since integration operation is linear, (cid:98) B t is also a
Gaussian process:
[Equation]
In fact, one can recover k H ( t, s ) in (15) which is (cid:82) t 0 (cid:82) s 0 k
( u, v ) dudv =
k H ( t, s ) .
For relevant results of rough path theory in the context that driving signals are Gaussian processes,we encourage readers to see [25, Chapter 10] for a complete description.
Proof of Theorem 1.
Since E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ , by [25, Theorem 10.9], the covariance of (cid:98) B t has finite ϱ -variation which is
[Equation]
As B t is a fractional Brownian motion with Hurst exponent H , then the covariance of B t has finite¯ ϱ -variation with ¯ ϱ = 12 H (see [25, Example 10.11]), or,
[Equation]
We consider R ( B, (cid:98) B ) ( u, v ) :
=
E [ B u (cid:98) B v ] .
Using Cauchy-Schwartz inequality, we have E [ B u (cid:98) B v ] ≤(cid:112) E
[ B 2 u ] (cid:113) E [ (cid:98) B 2 v ] .
Since ϱ > ¯ ϱ , we use a similar technique in the proof of [25, Corollary 10.6]
[Equation]
Now applying the result in [25, Corollary 10.6], for every α ∈ ( 1 / 3 , 1 / 2 ϱ ) and every θ ∈ (0 , 1 / 2 − ϱα )and q < ∞
[Equation]
This implies | ϱ α ( B , (cid:98) B ) | L q ≲ sup t ∈ [0 ,T ] W ( B t , (cid:98) B t ) 2 θ .
Using Talagrand inequality [81], we have
[Equation]
We follow closely the proof of [38].
That is, followed [25, Theorem 8.15], the solution map S t ( B , · )is a C 1 -diffeomorphism, i.e., its bijection and its inversion are differentiable.
This means we canobtain the solution X t at time t via canonical lift B and initial X 0 .
Let us define the inverse map as
S − t ( B , · ) which gives us a way to get X 0 given X t .
One can obtain the probability density distribution of solution at time t by a change of variables:
[Equation]
Then continuity of S t or S − t is stated in [25, Theorem 8.5].
That is, given rough paths, X , Y , and initial state x, y , one can have a local Lipschitz, for any γ ∈ [ 1 / 3 , α ]
[Equation]
Combining the continuity of p 0 and S t , one can arrive with| log (cid:98) p t
[Equation]
It is easy to show that from (23), one can have KL( (cid:98) p t ∥ p t ) ≲ | ϱ α ( B , (cid:98) B ) | L 1 .
Combining with 20,
KL( (cid:98) p t ∥ p t ) ≲ sup t ∈ [0 ,T ] KL( (cid:98) B t ∥ B ) θ .
Discussion In Theorem 1, we make an assumption that ϱ is greater than 1 / 2 H because of thefollowing observations.
Compared to the covariance of B t , that of (cid:98) B t in (18) is subtracted with apositive term.
Therefore, it is reasonable to assume that E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] is smaller than E
[( B t −
B t + τ ) 2 ] = | τ | 2 H .
This also means than we can find ϱ such that E
[( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ < | τ | 2 H .
In other words, ϱ is greater than 1 / 2 H .
As (cid:98) B t tries to approximate B t , we believe ϱ is slightly greaterthan 1 / 2 H .
Also, we can understand that (cid:98) B t comes from the covariance with truncated spectralinformation, therefore it is less varying and ϱ tends to bigger.
B.3 Spectral properties of fBm covariance matrix
We empirically compute the eigenvalues of fBm covariance matrices when H = 0 .
Thecovariance matrices are computed for finite data points in range [0 , 2] .
In Figure 8, we can see thatthe eigenvalues rapidly decay.
C Experiment results
C.1 Set up
We run experiment on a server equipped with Nvidia Tesla P40 GPU.
The number of inducing pointsis 70 .
In practice, this parameter is less sensitive in training GP when the number of data points(steps) are not too big.
C.2 Sampling mBm
In the experiment in §5.1, the baseline for sampling mBm is based on [64].
To sample mBm given a Hurst function h ( t ) ,
[64] uses the integral approximation over a discretization of time t 0 < t 1 < · · · < t N ,
[Equation]
Each component in the sum can be expressed analytically, we have the mBm samples over the timemesh as
[Equation]
Here, ξ j are standard Gaussian random variables.
Note that our procedure sampling paths for B tneeds only one Gaussian random variable at a time step (see Eq.
(11)) while using [64] requires all
Gaussian random variables in the sum in Eq.
The number of such variables increases as weincrease the mesh size.
In the experiment, we used the implementation in [24] to get mBm samples.
Estimating Hurst from data Here, we provide a brief description of how to estimate Hurst fromdata according to [26].
Motivating from the following property
[Equation]
the estimation is based on the ratio of two statistics
[Equation]
where W is a window size.
The second statistic has a halved resolution.
From [26], the estimationˆ h ( t ) = 12 log 2 (cid:16) M
[Equation] [Equation]
t ) M 2 ( t ) (cid:17) as the coefficient in Eq.
(26) vanishes, remaining Hurst exponent values.
Thisconverges a.s. to h ( t ) .
In our experiments, we set the windows size, W = 10 .
C.3 Synthetic data
Figure 9 shows the log-likelihood of the model when the number of data points N =100 , 200 , 300 , 400 .
As all the settings have similar average log-likelihoods, the model can fit welldata in such settings.
Additional experiments We consider the true Hurst functions including h ( t ) = 0 .
3 +0 .
5 sigmoid (7(1 − t )) and h ( t ) = 0 .
3 + 0 .
5 sigmoid (7( t − 1)) .
Figure 10 and 11 show the comparisons between our learned Hurst functions and the true one whenwe increase the number sample paths in the training.
The more sample paths are added, the closer ourlearned Hurst functions are to the true one.
This observation is similar to the results of [26] of whichthe plots are placed in the first row of Figure 10 and
We observe that choosing small step sizes helps training better, especially for the case H < 1 / 2 .
[Figure]
C.4 Financial data
[Figure]
Figure 12 contains the negative log-likelihoods of stock data on the training data.
Figure 14 and 13show the negative log-likelihoods and root mean square errors on the test data.
The posterior plot ofthese stocks are in Figure 16(APPL), 15(AMZN), 17(GOOGL), and 18(MSFT).
C.5 Hurst exponent in score-based generative models
Recent work [77] on generative model based on score functions and SDEs presents impressive results on par with generative adversarial networks (GAN).
The main idea is to learn a score function ∇ log p ( x ) of data.
which is the gradient of log likelihood of data via a process of gradually adding perturbation to data up to the point the result is unrecognizable compared to the original data, similar to random noise.
This process models by a SDE
[Equation]
As perturbing data along the time interval, it is possible to compute analytically the distributionof perturbed data X t at time t given input data X 0 , p ( X t | X 0 ) .
During the sampling phrase, suchinformation and the input data X 0 will both not be available.
[77] uses a neural network s ( X t ) toapproximate ∇ log p ( X t | X 0 ) .
Perturbation with noise from B t
We consider the case of perturbation in data under a new SDE,
[Equation]
t = −
[Equation]
β
[Equation]
t
[Equation]
X t dt + (cid:112) β ( t ) dB t where B t = (cid:82) t 0 f ( t, s ) dW s .
Our approach leads to a new form
[Equation]
The analytic solution of p ( X t | X 0 ) according to this SDE is rather complicated.
Therefore, we resort a similar form to
[Equation]
[Figure] [Figure]
We can get a similar p ( X t | X 0 ) but depepend on β ( t ) and ς ( t ; Z ) .
Note that ς ( t ; Z ) is the varianceproduced by sparse GP and ν ( t ; u, Z ) is vanished because its expectation E
[ ν ( t ; u, Z )] = 0 .
Forward vs backward Figure 19 and 20 show how our model adds noise noise during forwardphrases and how images are reconstructed during backward phrases.
These figures are of two settings:decreasing Hurst and increasing Hurst.
The first case having better negative log-likelihood (better thanthe baseline H = 0 . 5 ) demonstrates that noise perturbation with more dependency at the beginningis preferable.
With a similar setup, we can test the performance on the FashionMNIST data set between“decreasing
Hurst” and baseline “H=0.5”.
We included the NLL plot in Figure 21 where “decreasing Hurst"remains having better NLLs.
Based on the empirical evaluation on MNIST and FashionMNIST, we found that it is better to injectcorrelated noise ((long-range dependent H > 1 / 2 ) at the beginning time of perturbation. At the endof perturbation, there is no need for such correlated noise but irregular one ( H < 1 / 2 ). With t close to 0 (close to real images), p ( X t ) is very complex, requiring a careful perturbation inwhich correlated noise ( H > 1 / 2 ) is a good choice because it retains dependency between two timesteps.
On the other hand, with t close to the terminal time T , p ( X t ) becomes close to Gaussiandistribution so that we can accelerate the perturbation by adding even more irregular noise ( H < 1 / 2 ).
APPL
AMZN
GOOGL
MSFT
APPL
AMZN
GOOGL
MSFT
[Figure] [Figure] [Figure]