Learning Fractional White Noises in Neural Stochastic Differential Equations
Abstract
Differential equations play important roles in modeling complex physical systems.
Recent advances present interesting research directions by combining differential equations with neural networks.
By including noise, stochastic differential equations (SDEs) allows us to model data with uncertainty and measure imprecision.
There are many variants of noises known to exist in many real-world data.
For example, previously white noises are idealized and induced by Brownian motions.
Nevertheless, there is a lack of machine learning models that can handle such noises.
In this paper, we introduce a generalized fractional white noise to existing models and propose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian processes.
Our experimental results demonstrate that the proposed model can capture noise characteristics such as continuity from various time series data, therefore improving model fittings over existing models.
We examine how we can apply our approach to score-based generative models, showing that there exists a case of our generalized noise resulting in a better image generation measure.
Introduction
Differential equations center around modeling relations between functions and their rate of change, playing important roles in many dynamical systems in physics, biology and finance.
Incorporating such a tool in machine learning not only helps us understand more about machine learning models [42, 66, 44] but also expands the range of applications [31, 12].
Recent work on neural ordinary differential equations (neural ODEs) opens a new direction of connecting differential equations and neural networks [11].
Extensions to stochastic differential equations (SDEs) is proposed for handling noise in data or model uncertainty [83, 84, 54, 46]
Noises in data are often encountered in many real-world cases and can directly affect model performance.
The importance of understanding the characteristics of noise in data has been emphasized in a number of studies including: heteroscedastic noise modeling in time series analysis [82, 57], volatility modeling in financial research and market simulation [40, 14]; the significance of colored noise in neuroscience [20, 7, 4].
Rather than using traditional Brownian noises [5] in neural SDEs, providing a capacity to model more generalized type of noises not only allows this framework to extend to such applications but also retains the expressiveness from the representation powers of neural networks.
By learning noises together with the drift functions and diffusion functions of SDEs, the models become more flexible as the drift functions and diffusion functions, in turn, will be adjusted according to the learned noises.
Although mathematical research on analyzing general white noise is well-developed [38, 33], it is still challenging to implement practically for machine learning models like [54, 46].
In this paper, we introduce a novel class of white noises defined under a stochastic integral form [38, 33] to existing neural SDE models [83, 84, 54, 46] which only consider noises generated by Wiener process.
The most prominent examples of such stochastic processes include fractional
Brownian motions (fBm) and multifractional Brownian motions (mBm) which are generalizations of Brownian motions or Wiener processes.
The importance of such processes is emphasized in many applications from financial modeling [13, 28], to motion tracking in physical and biophysical systems [29, 35, 41, 17].
Existing techniques on sampling these stochastic processes [18, 65] prohibit models to scale up as they require to keep track of sample paths during SDE solves [54].
To overcome this, we propose an approximation under numerical SDE solvers considering increments at a time.
That is, we first establish a discretized version of stochastic integrals to formulate distribution over noise increments.
The key approach is that we borrow the idea in sparse Gaussian process literature [74, 37] to decouple the dependency between increments.
Therefore, we can reparameterize the increments to obtain an equivalent SDE which is easy to solve.
To show the convergence of our approximations to SDE solutions, we study the approximated covariance functions of underlying stochastic processes to establish error bounds using rough path theory [56, 26] and existing sparse
Gaussian process bounds [8].
The contributions of this paper are threefold.
First, we introduce a new type of neural SDEs with general white noise.
Second, we propose an approximation of that white noise, which is inspired by approximation methods in Gaussian process research to obtain equivalent neural SDEs.
Finally, we demonstrate empirically that our approach can capture noise characteristics and provide insight of how white noise affect learning score-based generative models as well as observe a setting that improves baseline models.
Related work
Differential equations and neural networks [11] presents neural ODEs using numerical method in solving differential equations to learn parameters in neural networks.
There are a number of follow-up works aiming to improvements in different aspects including numerical solver and/or training process [45, 43, 24, 90, 61], implicit termination [10], theoretical understanding [22, 60], and extensions to graph neural networks [9].
Moreover, it has seen some applications in normalizing flows [89, 31, 39] as well as in control [21, 59].
Modeling time series data is one of successful practices of such models [12, 91].
Not close but related, work by [77] demonstrates excellent results on image generation tasks.
Like SDEs are a stochastic extension of differential equations, neural
SDEs are natural extensions of neural ODEs when incorporating uncertainty into models.
However, the formulation is not always straightforward.
[83, 84] established theoretical aspects emphasizing high expressiveness of a class of neural SDEs while [54, 46] present how to train neural SDEs efficiently.
Many advanced tools in stochastic analysis like rough paths [56] are incorporated into differential equations, and therefore empowered machine learning models [49, 48, 64, 63].
SDEs and Gaussian process There are equivalences and connections between Gaussian processes and linear time-invariant SDEs that [80] discusses to a large extent.
[73] presents a faster learning
Gaussian process using filtering and smoothing methods [79].
Models of this research direction often focus on spatial-temporal data and establish state-of-the-art results.
[36, 88] learn differential equations with Gaussian processes instead of neural networks.
[19] considers a similar noise to ours as filtered white noises, finding an analytic form of corresponding kernels in the same manner as [73].
Unlike our approach focusing on a generic procedure, this work studies a specific class of kernels.
Gaussian processes can be a tool to model posteriors over paths [1] or to match with densities from
Fokker-Planck-Kolmogorov equations [75].
[55] devises a variational inference to approximate posterior inspired by work in sparse GP latent variables.
Background 3.1 White Noise
Brownian motion has a long history dated back from the observation of pollen in water in [5], then was studied formally by Einstein [23].
In this paper, we focus on a generalization of Brownian [Equation] Given inducing points [Equation]
by
[Equation]
approximated by motions under Riemann-Liouville fractional integral defined by:
[Equation] where h ( t ) : R → (0 , 1) is the Hurst exponent , W s is a Brownian motion and Γ( · ) is the Gamma function.
When h ( t ) is a constant H , this stochastic process resembles a fractional Brownian motion (fBm).
However, as noted in [58], it places a great emphasis on the origin ( t = 0 ) and no longer possesses power-law spectral density like in fBm.
As we are interested in the case that h ( t ) is a function of t , we emphasize more on the local variation of Hurst exponent than such properties.
Our formulation in (1) can be understood as a variant of multifractional Brownian motions (mBm) [69].
One of the goals in this paper is to learn h ( t ; θ ) using a function approximator, for example, a neural network with parameter θ .
Interpretation of Hurst exponent The stochastic process in (1) asymptotically shares some properties with fBm and mBm [65] including long-range dependencies and self-similarity .
We refer our further analysis related to (1) as fBm or mBm.
Here, long-range dependencies are defined for a stochastic process if the sum of autocovariance function is unbounded (see Appendix A.4).
Time series analysis has a similar notion for fractional ARIMA models [78].
Deep learning approaches also try to incorporate such long-range dependency to their models [32].
With the Hurst exponent h ( t ) , B t exhibits a long-range dependence when h ( t ) > 1 / 2 .
As h ( t ) = 1 / 2 , the stochastic process falls back to a Brownian motion which no longer has long-range dependency.
When h ( t ) < 1 / 2 , the stochastic process is “rough”, anti-persistent, having irregularity.
This parameter also expresses Hölder continuity of stochastic processes [26].
If h ( t ) changes over time, this means all of such characteristics happen locally.
Brownian motion, B t , or Wiener process has a rich history in modeling physical system [67].
BlackScholes models [40], long-standing models in studying financial markets are based on Brownian motions.
However, recent work shows fBm is more suitable due to the irregularity of volatility [14].
Correlated noise [7] also can be found in fMRI data.
Applications of fractional Brownian motions can be found in [87] [15].
3.2 Background of Neural SDEs
Consider a dynamical system { X t } whose dynamic evolution is governed by the following stochastic differential equation
[Equation]
where µ : [0 , T ] × R n → R n is the drift and σ : [0 , T ] × R n → R n × m is the diffusion.
Both of those functions are assumed to be well-behaved (i.e., smooth and linear growth) and are parameterized by neural networks.
Readers can refer to [80] for a gentle introduction of SDE in machine learning or to [72] for advanced materials.
[83, 84, 54, 46] are excellent starting points to be familiar with neural
SDEs.
Back-propagation using optimize-then-discretize strategy
This strategy is adopted in neural
ODEs, allowing to perform back-propagation in time [11].
Suppose we are interested in optimizing a loss function L at the terminal state X T .
This approach considers a variable ∂ L /∂X t .
The dynamic of this variable can be modeled by a differential equation leading to computing gradient by using chain rules.
In neural SDEs, such a strategy requires a more careful treatment as [54] addresses the randomness in ∂ L /∂X t .
Also, forward and backward SDEs need to share the same sample paths of Wiener process.
Model learning There are two main parameter estimation methods for Neural SDEs.
The first one is latent SDE [54] that follows a variational approach approximating posterior distributions by a neural SDE which shares the same diffusion with priors.
In this framework, the evidence lower bounds are derived by applying the Girsanov theorem.
Recently, [47] proposes SDE-GAN inspired by generative neural networks [30].
In this paper, we mainly use the former in our experiments.
However, it is worth noting that there is no restriction to training with SDE-GAN as long as chosen models satisfy our assumption.
3.3 Sparse Gaussian processes
We also give a brief introduction of sparse Gaussian processes which are known as scalable methods for Gaussian processes.
A Gaussian process (GP) [71] is defined by a set of random variables, any finite number of which have a joint Gaussian distribution.
Formally, a GP is completely specified by a mean function m ( · ) which usually is set to be zero for prior and covariance function k ( · , · ) .
We then can denote a GP by f ( x ) ∼ GP (0 , k ( x, x ′ )) .
Learning or sampling from a Gaussian process usually takes O ( N 3 ) with the number of data points N .
Sparse Gaussian process methods [74, 37] introduce a set of pseudo points (inducing points) u at locations Z which jointly follow a Gaussian distribution with data points.
The density function of f ( x ) , p ( f ( x )) is then approximated by p ( f ( x ) | u ) p ( u ) .
Here the mean and the variance of p ( f ( x ) | u ) can be computed as
[Equation]
where α ( x ) =
k ( Z, Z ) − 1 k
( Z, x ) .
There is a growing interest in this approach because of its scalability compared to traditional GPs [6, 3, 86, 37].
In this paper, we focus on the sampling procedure where we first sample u ∼ N (0 , k
( Z, Z )) , then sample f ( x ) ∼ p ( f ( x ) | u ) .
Learning White Noises in Neural SDEs
This section presents our main model.
We first approximate stochastic integrals using Stieltjes approximations and then use sparse Gaussian processes to leverage dependencies between random variables.
We analyze the convergence to SDEs solutions for this approximation.
Finally, we form an equivalent SDE to our model (see Figure 1).
4.1 Neural SDEs with B t
Consider a stochastic differential equation
[Equation]
with µ : [0 , T ] × R n → R n and σ : [0 , T ] × R n → R n × m .
Compared to (2), the noise coming from B t is an m -dimensional stochastic process, B t = (cid:82) t 0 f
( t, s ) dW s from (1) as we denote f ( t, s ) =
[Equation]
We are interested in learning both the parameters of µ ( t, X t ) , σ ( t, X t ) and that of f ( t, s ) .
Solving
SDEs in (4) is challenging since B t is constructed from an integral.
Direct sampling methods of
B t (i.e., require computing Cholesky decomposition) are costly.
Other approaches [65] use an approximation of stochastic integral as a linear combination of random variables.
However, the number of these random variables prevents optimize-then-discretize methods to scale.
4.2 Integral approximation
We use Stieltjes integration to approximate the integral in (1) by discretizing the interval [0 , T ] with 0 =
s 1 < s [Equation] < · · · < s N = T , i.e.,
[Equation]
where ξ i are independently distributed standard normal random variables coming from W s i +1 − W s i ∼ N (0 , ∆ s i ) with ∆ s i =
s i
+
− s i , and 1 { s ∈ [0 , t ] } is the indicator function.
Note that this is a reasonable approximation because Itô’s integrals, in general, converge in L 2 when N goes to infinity.
Moreover, even if W t has infinite variation, the probability of a sample path leading to the divergence of approximation infinitely often is zero [72].
The discretization is also inevitable whenever it is required to sample B t .
In several numerical approaches, for example, Euler-Maruyama [51], solving stochastic differential equations involves approximating increments, dB t , by
[Equation]
where ∆ g i ( t )
=
g i ( t + ∆ t ) − g i ( t ) .
Comparing to increments ∆ W t when solving (2), simulating ∆ W t is relatively simple as they are independent between each time step.
On the other hand, there exist dependencies between ∆ B t as
[Equation]
Here, ξ i , ξ j ∼ N (0 , 1) , and E [ ξ i ξ j ] = 0 when i ̸
=
j .
Our aim is to represent (6) in a way that is simple to sample from but preserves the dependencies between ∆ B t .
4.3 Sampling increments with sparse Gaussian process
From the observation in (7), we devise a sampling method inspired by sparse Gaussian processes.
In particular, we consider a Gaussian process 1 B ′ t =
∆ B t ∆ t for a fixed ∆ t .
In fact, ∆ t is later used as the time step of SDE solvers.
Note that directly modeling ∆ B t in (6) is numerically unstable when computing (7) with covariance matrices having small values.
Indeed, B ′ t is a Gaussian process because at a given time t , B ′ t is Gaussian with a zero mean.
To compute the covariance function k ( t, s ) , we simply compute k ( t, s ) =
E [ B ′ t B ′ s ] − E
[ B ′ t ] E [ B ′ s ]
= E [ B ′ t B ′ s ] and follow the same step with (7) [Equation]
[Equation]
To this end, we denote B ′ t ∼ GP (0 , k ( t, s )) .
Sampling from a Gaussian process based on a full covariance matrix is not scalable.
To overcome this difficulty, we propose a sampling scheme based on sparse-inducing GP [74].
The main idea of this method is to introduce a bottleneck as a summation of the joint distribution of B ′ t .
Define inducing-point locations as Z = { z 1 , . .
. , z M } which spread across [0 , T ] .
The number of inducing points, M , is much smaller than the number of discretization mesh of numerical solvers.
The inducing points u
=
( B ′ z 1 , . .
B ′ z M ) are distributed as a multivariate Gaussian distribution, i.e., u ∼ N (0 , k
( Z, Z )) .
Following the description of sparse GPs in §3.3, p ( B ′ t ) is approximated by p ( B ′ t | u ) p ( u ) .
The sampling procedure of B ′ t consists of two steps:
[Equation]
Here L Z is the Cholesky decomposition of k ( Z, Z ) , I M is an M × M identity matrix.
m ( · ) and v ( · ) are the conditional mean and variance taken from (3).
The above is a complete sampling procedure for p ( B ′ t | u ) p ( u ) .
Note that in our approximation, B ′ t is independent conditioned on that u are given.
Therefore, we obtain the increment ∆ B t as
[Equation]
Here, we replace √ ∆ tξ t with increments of a (new) Brownian motion W t since ξ t are independent for t ∈ [0 , T ] .
Remark One may consider an alternative approach where we use sparse approximation directly for
B t with given kernel function k ( t, s )
∝ | t | 2 h t,s + | s | h t,s − | t − s | 2 h t,s (see (16)), and can obtain a similar form of increments.
However, this kernel function is not differentiable w.r.t the parameters of Hurst function.
Our approach is to devise a “so-called” differentiation of B t under discretization.
Note that this notion differs from the definition of the derivative of B t in [52] in the sense of Schwartz space.
However, such a representation is not useful for a practical implementation.
Our approach shares a similarity with [49] on controlled differential equations driven by a signal, i.e.,
X t .
The signal X t is approximated by linear interpolations which allows to compute the derivative of X t .
We intentionally model the “derivative” of B t by B ′ t first, and then reconstruct the signal B t from B ′ t .
4.4 Forming equivalent SDEs
To this end, we show that the increments ∆ B t can be represented with increments of a Wiener process, ∆ W t in (11).
Here, the equivalence is under the assumption that the numerical procedure involves only ∆ B t , i.e., Euler-Maruyama methods.
Therefore, the original SDE in (4) is approximated by
[Equation]
This is obtained by conversions, dt ≈ ∆ t and dW t ≈ ∆ W t as we expand the diffusion term
[Equation]
where ν ( t ; u, Z ) = m ( t ; u, Z ) , and ς ( t ; Z )
= v ( t ; Z ) √ ∆ t .
The function ς ( t ; Z ) induced from the approximation of B ′ t is now absorbed into the diffusion function while ν ( t ; u, Z ) together with the diffusion function now is added to the drift function.
Note that ν ( t ; u, Z ) and ς ( t ; Z ) is conditioned on the inducing point u .
Therefore, at the initial time t = 0 , we need to precompute u first.
Although (12) and (4) seem similar in terms of the expressiveness of drift functions and diffusion functions, the drift function of (12) contains a randomness stemming from the sampling u in (9) while that of (4) is deterministic.
This can be understood that the dependency between ∆ B t is translated into the drift functions.
We can elucidate the difference via Algorithm 1 and Algorithmin Appendix.
Note that ν ( t ; u, Z ) and ς ( t ; Z ) are associated with two types of discretizations: that of approximating
Stieltjes integral and that of SDE numerical solver.
One of the key advantages of this new SDE is that it can inherit all theory and practices of wellestablished study of stochastic analysis for Brownian motion or Wiener process.
The preliminary results of Girsanov’s theorem, i.e. [34] cannot used in latent SDE [54].
Itô’s rules for fBm and mBm can be found in [52] but hard to use in practice.
On the other hand, our approach can work for any B t defined in (1).
Another benefit of our formulation is that the sampling procedure is simple, requiring only one ξ t at a time and avoids a multiplication of O ( N ) memory like [65].
This becomes significant as the optimize-then-discretize method requires constructing the same sample paths of B t in forward SDEs and backward SDEs.
Computational analysis The evaluation of diffusion functions in (12) requires computing an aggregate sum over discretization that scales linearly with the number of discretization points.
It may cause some overhead when computing Itô correction term during solving backward SDE [54].
Therefore,
Stratonovich SDEs are more efficient than Itô SDEs.
One limitation of integral approximation is that for a long interval [0 , T ] , one may need more discretized partitions to achieve desired approximation accuracy.
Onthecomputational cost of sparse GPs, we only need to perform the Cholesky decomposition at the initial time of every SDE integration.
This takes O ( M 3 ) with M the number of inducing points.
At a time t during solving SDEs, with the pre-computed L Z , sampling B ′ t has O ( N + M 2 ) time complexity as we perform matrix multiplications in (3).
Limitation Currently, our model only considers fixed step sizes in SDE numerical solvers.
Most theoretical studies in stochastic analysis work for H > 1 / 4 ( H > 1 / 3 in this study) to make sense of stochastic integrals or SDE solutions [70].
The Euler-Maruyama update in our approach may not be sufficiently good when the Hurst exponent is less than 1 / 2 as it is pointed out in [16].
Higher-order solvers can be more accurate, requiring handing double integrals, i.e., (cid:82)
(cid:82) dB t dB s which we wish to address in future work.
Although this paper presents an initial theoretical justification of convergence to solutions, the precise order of convergence is not clear yet.
Therefore, for practical use, we recommend avoiding using Hurst exponents with small values in our method.
4.5 Convergence analysis
The error of our approximation is composed of three main factors.
The first is of SDE solvers.
Due to the assumption of fixed step sizes, only a few solvers, i.e., Euler-Maruyama, are applicable with 1 / 2 strong convergence.
Secondly, the discretization in Stieltjes approximations attributing to the total error leads to the convergence in L 2 .
The convergence of such approximations are known.
The approximation of noise from sparse GPs is unconventional in solving SDEs because the approximation is in distributional sense rather than point-wise or path-wise.
We will show the convergence using rough path theory [56, 26] and sparse GPs bound from [8] for constant Hurst
H ∈ ( 1 / 3 , 1 / 2 ) .
We follow the notation in [39, 56, 26].
Let us denote the canonical lift of B as B = ( B, B ) .
Let (cid:98) B be the path corresponding to the sampling procedure in (9) and (10).
Let X be the solution of d X t = f
( t, X t ) d B t and (cid:98)
X be the solution of d (cid:98) X
= f ( t, (cid:98) X t ) d (cid:98) B where f : [0 , T ] × R n → R n is a bounded function that is four-times continuously differentiable.
We assume that the initial state follows a distribution p 0 which is is continuous and strictly positive in R n .
Let us denote the probability density distribution of the solution X at time t by p t ( x ) ; and that of the approximated solution (cid:98) X by (cid:98) p t ( x ) .
Theorem 1 (Convergence of solutions given sparse GP approximation) .
With the Hurst exponent of B , H ∈ ( 1 / 3 , 1 / 2 ) , assume that E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ for any t ∈ [0 , T ] , and ϱ ∈ ( 1 / 2 H , 3 / 2 ] .
Then, for every α ∈ ( 1 / 3 , 1 / 2 ϱ ) , there exists a constant C such that
[Equation]
Here, θ ∈ (0 , 1 / 2 − ϱα ) and KL( ·∥· ) is the Kullback-Leibler divergence.
This theorem says that if the distribution of (cid:98) B converges to that of B in terms of the KL divergence, their corresponding solutions converge in the same sense.
Note that we make an assumption that
E [ [Equation] B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ , ϱ > 1 / 2 H which means that (cid:98) B is less “rough” than B .
This is a reasonable assumption since (cid:98) B suffers information loss from sparse GP approximations.
Our proof is done by (1) assessing the p -variation of (cid:98) B to bound the distance in rough path space between the canonical lifts of B and (cid:98) B via [26, Corollary 10.6]; and (2) using the local Lipschitz property of Itô-Lyons maps [26, Theorem 8.5] as we follow the technique in [39].
The complete proof is provided in Appendix B along with necessary backgrounds and discussion.
According to [8], we can quantify the convergence of (cid:98) B to B as its KL divergence can be bounded,
KL( (cid:98) B ∥ B ) ≤ C (cid:80) ∞
i =
M +1 λ i .
Here λ i are eigenvalues in eigenvalue-eigenfunction pairs ( λ i , ϕ i ( · ) ) of Mercer’s theorem [62], k ( x, x ′ )
= (cid:80) ∞
i =
λ i ϕ
i ( x ) ϕ i ( x ′ ) .
Recall M is the number of inducing points.
Therefore, the convergence of KL( (cid:98) B ∥ B ) depends on the decaying rate of eigenvalues and the number of inducing points.
We empirically show the fast decaying rate of eigenvalues for finite samples which also can be found in Appendix B.
4.6 Parameterizing white noise
The previous section concerns how to learn white noises in neural SDEs without providing concrete forms of white noise.
This section will show how it affects SDEs in general and discuss the parameterization form of the Hurst exponent.
When the Hurst exponent h ( t ) is constant, say h ( t ) =
H .
There is a qualitative difference between settings of H .
Figure 2 shows how the Hurst parameter affects the model fit trained by Latent SDE.
With a smaller value ( H = 0 . 3 ), sample paths are rough, producing more irregularity.
When H = 0 .
8 , the model has smoother sample paths, and is more conservative as predictions depend on the past.
This parameter has a great influence on the uncertainty quantification of the models.
[Figure] For the case that h ( t ) is function of t , any function approximators can work in general to model h ( t ; θ ) .
In our implementation, we use small neural networks.
With the new representation in (12), we can train the parameters of h ( t ; θ ) together with the parameters of drift and diffusion.
A more general approach is that f ( t, s ) can accept either neural networks or kernel-based functions when we do not consider normalization.
However, their interpretation is not clear as that of mBm.
Experimental Results
This section will first justify how close the samples generated by our method are to true ones.
We then investigate the ability to learn the Hurst function for synthetic data and test our model for real-world data.
Finally, we examine the role of Hurst exponents in score-based generative models.
All experiments are implemented using PyTorch [68] with the library torchsde [54, 47].
Our source code is available at this repository.
How to estimate Hurst from data?
We follow [27] to empirically estimate Hurst exponent from data.
The key technique is based on the correlation of increment of B t and estimates a ratio over sliding windows which is an estimation of Hurst exponent.
5.1 Quality of approximate samples
We would like to investigate if our approximation of B ′ t can produce the sample path of B t that is close enough to those of the true process.
The true Hurst function is set as h ( t ) = 0 .
35 + 0 .
5 t .The samples from [65, 25] are used as a baseline.
This experiment runs 1000 samples.
The estimations of the Hurst exponent for two sampling techniques are in Figure 3.
The Hurst estimation corresponding to our samples is close to the true Hurst function.
It tends to overestimate the long-range dependency ( h ( t ) > 1 / 2 ) and underestimate the irregularity ( h ( t ) < 1 / 2 ) in the samples.
5.2 Recovering Hurst function
[Figure]
Next, we test our model with a synthetic data.
Here the data is generated from a SDE with a known solution:
[Equation]
where B t is a mBm given a fixed h ( t ) = 0 .
3 + 0 .
5 sigmoid (7(1 − t )) .
According to Itô integrals for mBm cases [52], the exact solution is X t = x 0 exp( αt − 12 β 2 t 2 h ( t ) + βB t ) .
We generatedata points for each sample path and obtain up tosample paths.
After training our model, we observe that the obtained Hurst function can capture the dynamic similar to the true Hurst.
As we increase the number of sample paths (from 1 to 5 ), our learned
Hurst tends to approach the true Hurst.
Figure 4 shows an example of our learned Hurst function given 5 sample paths.
We also provide the results for different Hurst functions and an ablation study in Appendix C.3. [Figure]
5.3 Financial data
In this experiment, we compare the Hurst function found by our model with the time-dependent Hurst estimation computed by [27].
Data set We collect two months (from September 26th, 2021 to October 23rd, 2021) of hourly data from four stock indices Amazon (AMZN), Apple (APPL), Google (GOOGL), Microsoft (MSFT).
The data is retrieved from [85].
It is then standardized and split into the first 80% time points to train and the remaining 20% to test.
Obtaining Hurst function Figure 5 compares our found Hurst functions and estimated Hurst computed from [27].
Our obtained Hurst functions have bigger values than that of [27].
This resembles the observation in the previous experiment.
A common observation among these stocks is that Hurst functions made some changes (bump up or drop) around the end of October.
This agrees with many analyses, for example, Amazon missed revenue expectations for the third quarter or Apple revealed new products.
The posterior plots for our model in each stock index are placed in
Appendix C.4.
Quantitative comparison We compare our model to two main baselines: Latent SDE with standard
Brownian motion [54] and Latent ODE [12].
We report three main comparison criteria: training negative log-likelihood (training NLL), test negative log-likelihood (test NLL) and test root mean square error (test RMSE).
Overall, our model can fit training data well as our model outperforms others in terms of training NLLs (see Appendix C.4).
This is because our model can capture correctly noise part while Latent SDE relies on less flexible Brownian noise.
Still, it is not clear to identify which model performs better in predictive tasks.
However, our model produces more consistent results with less variance.
5.4 Effect of Hurst exponent in score-based generative models
This experiment explores a different aspect by studying the effect of Hurst exponent in training score-based generative models.
Recent work [77] on generative models based on score functions and SDEs presents impressive results on par with generative adversarial networks (GAN).
We consider the case where the perturbation in data is under a new SDE, dX t = − 12 β ( t )
X t dt + (cid:112) β
( t ) dB t where
B t = (cid:82) t 0 f ( t, s ) dW s .
We derive the equivalent SDE (discussed more in Appendix C.5):
[Equation]
Varying Hurst exponents We investigate how different types of noise affect learning and sampling in this model.
We consider four scenarios of noise: (1) more anti-persistent as H = 0 . 3 ; (2) longrange dependent with H = 0 . 8 ; (3) linearly increasing h ( t ) ; (4) linearly decreasing h ( t ) .
We only consider the task of generating MNIST images [53].
Figure 6 shows example of generated images.
We observed that H > 1 / 2 is not favorable, and requires SDEs with long terminal time T .
To quantify the quality of generated images, we compute the negative log-likelihood (NLL) of a set of out-of-sample images like in [76].
Discussion Although we do not present a way to find optimal h ( t ) , we show that there is a setting in which the NLL appears to be better (see Figure 7 a).
That is, the NLL of decreasing Hurst setting is 3 .
79 (bits/dim) compared to 4 .
97 (bits/dim) of Brownian motion ( H = 0 . 5 ).
There exists work [50] showing that signal-to-noise ratio (SNR) is the key to establishing an optimal strategy how to inject noises.
There is another work studying the best strategy for reverse variance [2].
In contrast to many diffusion models, our approach has an additional term (cid:112) β ( t ) ν ( t ; u ) which is unrelated to X t but is included in the drift of the backward SDEs when generating data (see Figure 7 b).
Moreover, our approach can suggest a way to control the sample path of B t in the way that we condition on inducing points u since B ′ t is sampled from the conditional distribution p ( B ′ t | u ) .
For example, we can follow the variational inference of [37] to find the posterior q ( u ) for a designed objective, then sample B ′ t given q ( u ) instead of p ( u ) . H = 0 . H = 0 . [Figure]
[Figure]
Conclusion
In conclusion, this work presents a new class of neural SDEs with a generalization of white noise.
We propose an efficient approximate sampling approach for the noises so that neural SDEs can be converted into convenient forms.
The techniques used in this paper are relatively simple, and related to Gaussian process (GP) research.
Our experiments have verified the ability of our models to learn noises from synthetic data to financial time-series as well as showcase an image generation task.
As mentioned earlier, our current work can be improved with higher-order solvers where the double integrals require to model precisely.
Prior work [52, 70] is a good starting point for such an approach.
Another promising direction is to extend neural controlled differential equations with driving signals which are general GPs instead of Brownian motions.
The closure property of GPs under differentiation will be useful to develop this model.
Acknowledgements
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00984, Artificial Intelligence, Explainability, Personalization, Plug and Play, Universal Explanation Platform), (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)), (No. 2022-0-00184,
Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics), and KAIST-NAVER Hypercreative AI Center.
We would like to thank anonymous reviewers for insightful feedback.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the last paragraph of Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Mentioned in the main text and more detailed in Appendix B (b) Did you include complete proofs of all theoretical results?
[Yes] Complete proof can be found in Appendix B 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] MNIST dataset for image generation tasks.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Notation
Meaning
[Table]
evaluated
[Table]
GP
Gaussian process
[Table]
[Table]
solving
SDEs Here, we provide our approach in Algorithm 1 in contrast to the SDEs with Brownian motion in Algorithm 2.
[Table]
Output: Solution X t 1 at t/* Compute drift and diffusion in Eq.
(12) */ 1 def f_and_g( t , x ) : /
Python syntax: global */ global inducing points u /* Mean and variance of GP approximation in Eq.
(3) */ m ( t ; u ) , v ( t ; u ) = GP_approx
( t , u, h ( · ) )
ν ( t ; u ) =
m
( t ; u ) ς ( t ; u ) = v ( t ; u )∆ t drift = µ ( t, x ) + σ ( t, x ) ν ( t ; u ) diffusion = σ ( t, x ) ς
( t ; u ) return drift, diffusion 10 def brownian_increment( t ) : return Brownian noise ∆ W t /*
Use Eq.
(9) and (10) */ 13 inducingpoints u
= sample_inducing( Z , h ( · ) ) /* call SDE solver */ 14 X t 1 = sdeint(f_and_g , brownian_increment , X t 0 , t 0 , t 1 ) 15 return X t
Input: Drift function µ ( t, x ) , diffusion function σ ( t, x ) , initial value X t
Output: Solution X t 1 at t/* Compute drift and diffusion directly */ 1 def f_and_g( t , x ) : return µ ( t, x ) , σ ( t, x ) 4 def brownian_increment( t ) : return Brownian noise ∆ W t /* call SDE solver */ 7 X t 1 = sdeint(f_and_g , brownian_increment , X t 0 , t 0 , t 1 ) 8 return X t
Model training with Latent SDE The training method used in experiments of § 5.2 and § 5. is adopted from [54] via a Bayesian approximation method.
Consider two SDEs sharing the same diffusion function
[Equation]
Applying our approach like in (12), we can obtain new drift and diffusion functions (with conditioned on inducing points) for SDEs with Brownian motion.
Let us write in this form
[Equation]
where ¯ µ θ , ¯ µ ϕ , ¯ σ are derived similar to (12).
The variational bound requires computing the KL divergence between the approximate posterior and prior which is related to the change of measures between prior paths and posterior paths, therefore, is obtained by Girsanov Theorem.
That is, with a well-defined u ( t, x ) satisfying ¯ σ ( t, x ) u ( t, x ) =
¯ µ ϕ ( t, x ) − ¯ µ θ ( t, x ) , the change of measure is written as
[Equation]
Finally, the variational bound consists of the log-likelihood and E [log M t ] .
Details of the learning algorithm can be found in [54].
A Fractional Brownian Motion and Multifractional Brownian Motion
This section contains additional background of fractional Brownian motions (fBm) and multifractional
Brownian motions (mBm) including their covariance function and a detail description of long-range dependency in such processes.
A.1 Fractional Brownian Motion
Fractional Brownian motion can be viewed as a Gaussian process with covariance
[Equation]
Here, H is called the Hurst exponent.
It has some properties: (1) long-range dependence ( H > 1 / 2 ); (2) anti-persistence or irregularity ( H < 1 / 2 ).
Such properties are interesting to applications in modeling internet traffic, highly textured images.
A.2 Multifractional Brownian Motion
Multifractional Brownian motion B ( h ) is a Gaussian process with covariance [52]
[Equation]
16) with h : [0 , T ] → (0 , 1) , h t,s :=
h ( t )+ h ( s ), and c x = (cid:16) [Equation] [Equation] (cid:17) 12 .
This extension is to model cases that behaviors of signals happen locally.
If h ( t ) is close to 0 , we expect irregular local behaviors.
If h ( t ) > 1 / 2 at time t , we expect local long-range dependence.
A.3 Itô Integral for Multifractional Brownian Motion
Traditional Brownian motion case The Itô integral is computed as
[Equation]
Multifractional Brownian motion case The Itô integral has additional derivative of kernel function f
[Equation]
T
[Equation]
The red term ddt R h ( t, t ) is the derivative of variance function of mBm.
This can be interpreted as the second derivative term in Itô’s formula, ∂ 2 f ∂x 2 ( t, B ( h ) t ) , is adjusted according to how fast the variance function changes when time goes.
Note that ddt R h ( t, t ) =
2 t 2 h ( t ) − 1 ( h ′ ( t ) t ln( t )
+ h ( t ))
This result is used in the main text to get the analytic solution of dX t = αX t dt + βdB t
A.4 Properties of Fractional Brownian Motion
As discussed in the main text, fBm exhibits distinct properties including self-similarity and long-range dependency.
We provide a glimpse of these notions to readers.
Self-similarity Simply speaking, self-similarity in general refers to objects which look exact or approximately similar to a part of itself.
This concept is related to fractals.
In statistics, consider a discrete stochastic process { X t } t ∈ N .
We say X is self-similar if a finite sum, i.e., X km + · · · +
X ( k +1) m − 1 and a scaled version a m X k have the same distribution.
In a restricted definition when a m is just a polynomial of m , we say X ( m ) k
= 1 m ( X km + · · · + X ( k +1) m − 1 ) and m 1 − H X ( m ) are distributionally equal or ( X k 1 , . .
, X k d ) distributionally equal to ( m 1 − H X ( m ) k 1 , . . .
, m 1 − H X ( m )
k d ) if X is self-similar.
Now, consider the noise with respect to fBm, defined as
[Equation]
The self-similarity of X is expressed by comparing mX ( m ) and m H X .
Both of them are zero-mean
Gaussian, having the same covariance because
[Equation]
Long-range dependence Long-range dependence in stochastic processes means the sum of the autocovariance function is unbounded.
For fractional Gaussian noise, X k = B Hk +1 − B Hk has the autocovariance function
[Equation]
This can be approximated as γ ( k ) ∼ H (2 H − 1) k 2 H − 2 , leading (cid:80) ∞
k =1 γ ( k ) = ∞
when 1 / 2 <
H < 1 .
Perhaps, fractional ARIMA is a more intuitive model to understand long-range dependency.
While well-known ARIMA model is defined as
[Equation]
where L is the lag operator as LW k = W k − 1 , ϕ
and θ are polynomials.
By introducing
[Equation]
we can construct fractional ARIMA model.
The long-range dependence is expressed by the infinite sum in the right hand side.
Here, the Hurst parameter is H =
d + 1 / 2 .
B Convergence analysis
In this section, we provide some background on rough path theory.
Our proof will investigate the variation of covariance of Gaussian processes and sparse Gaussian processes which help us to judge their Hölder continuity.
Then, we use results of rough path theory [56, 26] to bound the KL divergence between solutions via the KL divergence fractional Brownian motions and their sparse
GP approximations.
Finally, we use [8] to justify the convergence of sparse GPs.
B.1 Rough path theory
Here we give a brief review of rough path theory.
For a complete introduction, readers can refer to [26].
Notation We denote X s,t :=
X t − X s and (cid:82) t s X s,r dX r :
= X s,t .
Also, p -variation seminorm is defined as
[Equation]
where P is a set of disjoint intervals of some partitioning scheme over [0 , T ] .
We will write x ≲ y to express x ≤ Cy for x, y > 0 .
In this section, we will sometimes write the covariance as R ( s, t ) .
A rectangular increment of covariance is denoted by
[Equation]
The p -variation of a rectangle increment is written as ∥ R ∥ p ;[0 ,T 2 .
The space of rough paths The following will provide the specification of the space of rough paths including its norm and its metric.
Definition 2.
Given α ∈ (cid:0) 13 , 12 (cid:3) , the space of α -Hölder rough paths is defined by
• pairs ( X, X
[Equation]
= [Equation] [Equation] • For any X and Y , (inhomogeneous) α -Hölder rough path metrics is
[Equation]
The space of geometric rough paths Many results in rough path theory require rough paths being geometric where the following holds
[Equation]
For Brownian motions, Stratonovich integration leads to geometric Brownian rough paths.
There is a way to convert between Itô and Stratonovich version of Brownian motions (see [39, §3]).
A similar technique can be achieved for fBm by [70].
Itô-Lyons map One of interesting results in rough path theory is the Itô-Lyons maps.
Suppose we have a solution map
[Equation]
Simply speaking, while Ψ is universal as it is just an enhancement of B , the solution map S is a continuous map on both the initial conditions and driving noise.
This property does not hold for Itô maps from sole Brownian motion paths to solution paths.
B.2 Main proof
Before going to the main proof, we provide the Gaussian process form of sample paths (cid:98) B t corresponding to B ′ t described in (9) and (10).
We can understand (cid:98) B t is the result of applying integration operator on B t (with small numerical error).
Since integration operation is linear, (cid:98) B t is also a
Gaussian process:
[Equation]
In fact, one can recover k H ( t, s ) in (15) which is (cid:82) t 0 (cid:82) s 0 k
( u, v ) dudv =
k H ( t, s ) .
For relevant results of rough path theory in the context that driving signals are Gaussian processes, we encourage readers to see [26, Chapter 10] for a complete description.
Proof of Theorem 1.
Since E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ , by [26, Theorem 10.9], the covariance of (cid:98) B t has finite ϱ -variation which is
[Equation]
As B t is a fractional Brownian motion with Hurst exponent H , then the covariance of B t has finite ¯ ϱ -variation with ¯ ϱ = 12 H (see [26, Example 10.11]), or,
[Equation]
We consider R ( B, (cid:98) B ) ( u, v ) :
=
E [ B u (cid:98) B v ] .
Using Cauchy-Schwartz inequality, we have E [ B u (cid:98) B v ] ≤ (cid:112) E
[ B 2 u ] (cid:113) E [ (cid:98) B 2 v ] .
Since ϱ > ¯ ϱ , we use a similar technique in the proof of [26, Corollary 10.6]
[Equation]
Now applying the result in [26, Corollary 10.6], for every α ∈ ( 1 / 3 , 1 / 2 ϱ ) and every θ ∈ (0 , 1 / 2 − ϱα ) and q < ∞
[Equation]
This implies | ϱ α ( B , (cid:98) B ) | L q ≲ sup t ∈ [0 ,T ] W ( B t , (cid:98) B t ) 2 θ .
Using Talagrand inequality [81], we have
[Equation]
[Equation]
We follow closely the proof of [39].
That is, following [26, Theorem 8.15], the solution map S t ( B , · ) is a C 1 -diffeomorphism, i.e., its bijective and its inversion are differentiable.
This means we can obtain the solution X t at time t via canonical lift B and initial state X 0 .
Let us define the inverse map as S − t ( B , · ) which gives us a way to get X 0 given X t .
One can obtain the probability density distribution of solution at time t by a change of variables:
[Equation]
Then continuity of S t or S − t is stated in [26, Theorem 8.5].
That is, given rough paths, X , Y , and initial state x, y , we can obtain a local Lipschitz estimate, for any γ ∈ [ 1 / 3 , α ]
[Equation]
Combining the continuity of p 0 and S t , one can arrive with
[Equation]
It is easy to show that from (23), one can have KL( (cid:98) p t ∥ p t ) ≲ | ϱ α ( B , (cid:98) B ) | L 1 .
Combining with (20),
KL( (cid:98) p t ∥ p t ) ≲ sup t ∈ [0 ,T ] KL( (cid:98) B t ∥ B ) θ .
Discussion In Theorem 1, we make an assumption that ϱ is greater than 1 / 2 H because of the following observations.
Compared to the covariance of B t , that of (cid:98) B t in (18) is subtracted with a positive term.
Therefore, it is reasonable to assume that E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] is smaller than E
[( B t −
B t + τ ) 2 ] = | τ | 2 H .
This also means than we can find ϱ such that E
[( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ < | τ | 2 H .
In other words, ϱ is greater than 1 / 2 H .
As (cid:98) B t tries to approximate B t , we believe ϱ is slightly greater than 1 / 2 H .
Also, we can understand that (cid:98) B t comes from the covariance with truncated spectral information, therefore it is less varying and ϱ tends to bigger.
B.3 Spectral properties of fBm covariance matrix
We empirically compute the eigenvalues of fBm covariance matrices when H = 0 .
The covariance matrices are computed for finite data points in range [0 , 2] .
In Figure 8, we can see that the eigenvalues rapidly decay.
C Experiment results C.1 Set up
We run experiment on a server equipped with Nvidia Tesla P40 GPU.
The number of inducing points is 70 .
In practice, this parameter is less sensitive in training GP when the number of data points (steps) are not too big.
C.2 Sampling mBm
In the experiment in §5.1, the baseline for sampling mBm is based on [65].
To sample mBm given a Hurst function h ( t ) , [65] uses the integral approximation over a discretization of time t 0 < t 1 < · · · < t N ,
[Equation]
Each component in the sum can be expressed analytically, we have the mBm samples over the time mesh as
[Equation]
Here, ξ j are standard Gaussian random variables.
Note that our procedure sampling paths for B t needs only one Gaussian random variable at a time step (see Eq.
(11)) while using [65] requires all
Gaussian random variables in the sum in Eq.
The number of such variables increases as we increase the mesh size.
In the experiment, we used the implementation in [25] to get mBm samples.
Estimating Hurst from data Here, we provide a brief description of how to estimate Hurst from data according to [27].
Motivating from the following property
[Equation]
the estimation is based on the ratio of two statistics
[Equation]
where W is a window size.
The second statistic has a halved resolution.
From [27], the estimation ˆ h ( t ) = 12 log 2 (cid:16) M ′ 2 ( t ) M 2 ( t ) (cid:17) as the coefficient in Eq.
(26) vanishes, remaining Hurst exponent values.
This converges a.s. to h ( t ) .
In our experiments, we set the windows size, W = 10 .
C.3 Synthetic data
Figure 9 shows the log-likelihood of the model when the number of data points N = 100 , 200 , 300 , 400 .
As all the settings have similar average log-likelihoods, the model can fit well data in such settings.
[Figure] overestimates Additional experiments We consider the true Hurst functions including h ( t ) = 0 .
3 + 0 .
5 sigmoid (7(1 − t )) and h ( t ) = 0 .
3 + 0 .
5 sigmoid (7( t − 1)) .
Figures 10 and 11 show the comparisons between our learned Hurst functions and the true one when we increase the number sample paths in the training.
The more sample paths are added, the closer our learned Hurst functions are to the true one.
This observation is similar to the results of [27] of which the plots are placed in the first row of Figure 10 and [-
-]11.
We observe that choosing small step sizes helps training better, especially for the case H < 1 / 2 .
C.4 Financial data
Figure 12 contains the negative log-likelihoods of stock data on the training data.
Figures 13 andshow the negative log-likelihoods and root mean square errors on the test data.
The posterior plot of these stocks are in Figure 16(APPL), 15(AMZN), 17(GOOGL), and 18(MSFT).
C.5 Hurst exponent in score-based generative models
Recent work [77] on generative model based on score functions and SDEs presents impressive results on par with generative adversarial networks (GAN).
The main idea is to learn a score function ∇ log p ( x ) of data.
which is the gradient of log likelihood of data via a process of gradually adding perturbation to data up to the point the result is unrecognizable compared to the original data, similar to random noise.
This process models by a SDE
[Equation]
As perturbing data along the time interval, it is possible to compute analytically the distribution of perturbed data X t at time t given input data X 0 , p ( X t | X 0 ) .
During the sampling phase, such information and the input data X 0 will both not be available.
[77] uses a neural network s ( X t ) to approximate ∇ log p ( X t | X 0 ) . [Figure] [Figure]
Perturbation with noise from B t
We consider the case of perturbation in data under a new SDE, dX t = −
[Equation]
β
[Equation]
β
[Equation]
t ) dB t where B t = (cid:82) t 0 f ( t, s ) dW s .
Our approach leads to a new form
[Equation]
The analytic solution of p ( X t | X 0 ) according to this SDE is rather complicated.
Therefore, we resort a similar form of
[Equation]
We can get a similar p ( X t | X 0 ) but depended on β ( t ) and ς ( t ; Z ) .
Note that ς ( t ; Z ) is the variance produced by sparse GP and ν ( t ; u, Z ) is vanished because its expectation E
[ ν ( t ; u, Z )] = 0 .
Forward vs backward Figures 19 and 20 show how our model adds noise noise during forward phases and how images are reconstructed during backward phases.
These figures are of two settings: decreasing Hurst and increasing Hurst.
The first case having better negative log-likelihood (better than the baseline H = 0 . 5 ) demonstrates that noise perturbation with more dependency at the beginning is preferable.
With a similar setup, we can test the performance on the FashionMNIST data set between“decreasing
Hurst” and baseline “ H = 0 . 5 ”.
We included the NLL plot in Figure 21 where “decreasing Hurst" remains having better NLLs.
Based on the empirical evaluation on MNIST and FashionMNIST, we found that it is better to inject correlated noise (long-range dependent H > 1 / 2 ) at the beginning time of perturbation.
At the end of perturbation, there is no need for such correlated noise but irregular one
( H < 1 / 2 ).
With t close to 0 (close to real images), p ( X t ) is very complex, requiring a careful perturbation in which correlated noise ( H > 1 / 2 ) is a good choice because it retains dependency between two time steps.
On the other hand, with t close to the terminal time T , p ( X t ) becomes close to a Gaussian distribution so that we can accelerate the perturbation by adding even more irregular noise ( H < 1 / 2 ).
APPL
AMZN
GOOGL
MSFT
APPL
AMZN
GOOGL
MSFT
[Figure]
[Figure] [Figure]