Learning Fractional White Noises in Neural Stochastic Differential Equations
Abstract
Differential equations play important roles in modeling complex physical systems.
Recent advances present interesting research directions by combining differential equations with neural networks.
By including noise, stochastic differential equations (SDEs) allows us to model data with uncertainty and measure imprecision.
There are many variants of noises known to exist in many real-world data.
For example, previously white noises are idealized and induced by Brownian motions.
Nevertheless, there is a lack of machine learning models that can handle such noises.
In this paper, we introduce a generalized fractional white noise to existing models and propose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian processes.
Our experimental results demonstrate that the proposed model can capture noise characteristics such as continuity from various time series data, therefore improving model fittings over existing models.
We examine how we can apply our approach to score-based generative models, showing that there exists a case of our generalized noise resulting in a better image generation measure.
Introduction
Differential equations center around modeling relations between functions and their rate of change, playing important roles in many dynamical systems in physics, biology and finance.
Incorporating such a tool in machine learning not only helps us understand more about machine learning models [42, 66, 44] but also expands the range of applications [31, 12].
Recent work on neural ordinary differential equations (neural ODEs) opens a new direction of connecting differential equations and neural networks [11].
Extensions to stochastic differential equations (SDEs) is proposed for handling noise in data or model uncertainty [83, 84, 54, 46]
Noises in data are often encountered in many real-world cases, and can directly affect model performance.
The importance of understanding the characteristics of noise in data has been emphasized in a number of studies including: heteroscedastic noise modeling in time series analysis [82, 57], volatility modeling in financial research and market simulation [40, 14]; the significance of colored noise in neuroscience [20, 7, 4].
Rather than using traditional Brownian noises [5] in neural SDEs, providing a capacity to model more generalized type of noises not only allows this framework to extend to such applications but also retains the expressiveness from the representation powers of neural networks.
By learning noises together with the drift functions and diffusion functions of
SDEs, the models become more flexible as the drift functions and diffusion functions, in turn, will be adjusted according to the learned noises.
Although mathematical research on analyzing general white on Information Systems noise is well-developed [38, 33], it is still challenging to implement practically for machine learning models like [54, 46].
In this paper, we introduce a novel class of white noises defined under a stochastic integral form [38, 33] to existing neural SDE models [83, 84, 54, 46] which only consider noises generated by Wiener process.
The most prominent examples of such stochastic processes include fractional
Brownian motions (fBm) and multifractional Brownian motions (mBm) which are generalizations of Brownian motions or Wiener processes.
The importance of such processes is emphasized in many applications from financial modeling [13, 28], to motion tracking in physical and biophysical systems [29, 35, 41, 17].
Existing techniques on sampling these stochastic processes [18, 65] prohibit models to scale up as they require to keep track of sample paths during SDE solves [54].
To overcome this, we propose an approximation under numerical SDE solvers considering increments at a time.
That is, we first establish a discretized version of stochastic integrals to formulate distribution over noise increments.
The key approach is that we borrow the idea in sparse Gaussian process literature [74, 37] to decouple the dependency between increments.
Therefore, we can reparameterize the increments to obtain an equivalent SDE which is easy to solve.
To show the convergence of our approximations to SDE solutions, we study the approximated covariance functions of underlying stochastic processes to establish error bounds using rough path theory [56, 26] and existing sparse
Gaussian process bounds [8].
The contributions of this paper are threefold.
First, we introduce a new type of neural SDEs with general white noise.
Second, we propose an approximation of that white noise, which is inspired by approximation methods in Gaussian process research to obtain equivalent neural SDEs.
Finally, we demonstrate empirically that our approach can capture noise characteristics and provide insight of how white noise affect learning score-based generative models as well as observe a setting that improves baseline models.
Related work
Differential equations and neural networks [11] presents neural ODEs using numerical method in solving differential equations to learn parameters in neural networks.
There are a number of follow-up works aiming to improvements in different aspects including numerical solver and/or training process [45, 43, 24, 90, 61], implicit termination [10], theoretical understanding [22, 60], and extensions to graph neural networks [9].
Moreover, it has seen some applications in normalizing flows [89, 31, 39] as well as in control [21, 59].
Modeling time series data is one of successful practices of such models [12, 91].
Not close but related, work by [77] demonstrates excellent results on image generation tasks.
Like SDEs are a stochastic extension of differential equations, neural
SDEs are natural extensions of neural ODEs when incorporating uncertainty into models.
However, the formulation is not always straightforward.
[83, 84] established theoretical aspects emphasizing high expressiveness of a class of neural SDEs while [54, 46] present how to train neural SDEs efficiently.
Many advanced tools in stochastic analysis like rough paths [56] are incorporated into differential equations, and therefore empowered machine learning models [49, 48, 64, 63].
SDEs and Gaussian process There are equivalences and connections between Gaussian processes and linear time-invariant SDEs that [80] discusses to a large extent.
[73] presents a faster learning
Gaussian process using filtering and smoothing methods [79].
Models of this research direction often focus on spatial-temporal data and establish state-of-the-art results.
[36, 88] learn differential equations with Gaussian processes instead of neural networks.
[19] considers a similar noise to ours as filtered white noises, finding an analytic form of corresponding kernels in the same manner as [73].
Unlike our approach focusing on a generic procedure, this work studies a specific class of kernels.
Gaussian processes can be a tool to model posteriors over paths [1] or to match with densities from
Fokker-Planck-Kolmogorov equations [75].
[55] devises a variational inference to approximate posterior inspired by work in sparse GP latent variables.
Background 3.1 White Noise
Brownian motion has a long history dated back from the observation of pollen in water in [5], then was studied formally by Einstein [23].
In this paper, we focus on a generalization of Brownian
[Equation]
Given inducing points
[Equation]
by
[Equation]
approximated by motions under Riemann-Liouville fractional integral defined by:
[Equation]
where h ( t ) : R → (0 , 1) is the Hurst exponent , W s is a Brownian motion and Γ( · ) is the Gamma function.
When h ( t ) is a constant H , this stochastic process resembles a fractional Brownian motion (fBm).
However, as noted in [58], it places a great emphasis on the origin ( t = 0 ) and no longer possesses power-law spectral density like in fBm.
As we are interested in the case that h ( t ) is a function of t , we emphasize more on the local variation of Hurst exponent than such properties.
Our formulation in (1) can be understood as a variant of multifractional Brownian motions (mBm) [69].
One of the goals in this paper is to learn h ( t ; θ ) using a function approximator, for example, a neural network with parameter θ .
Interpretation of Hurst exponent The stochastic process in (1) asymptotically shares some properties with fBm and mBm [65] including long-range dependencies and self-similarity .
We refer our further analysis related to (1) as fBm or mBm.
Here, long-range dependencies are defined for a stochastic process if the sum of autocovariance function is unbounded (see Appendix A.4).
Time series analysis has a similar notion for fractional ARIMA models [78].
Deep learning approaches also try to incorporate such long-range dependency to their models [32].
With the Hurst exponent h ( t ) , B t exhibits a long-range dependence when h ( t ) > 1 / 2 .
As h ( t ) = 1 / 2 , the stochastic process falls back to a Brownian motion which no longer has long-range dependency.
When h ( t ) < 1 / 2 , the stochastic process is “rough”, anti-persistent, having irregularity.
This parameter also expresses Hölder continuity of stochastic processes [26].
If h ( t ) changes over time, this means all of such characteristics happen locally.
Brownian motion, B t , or Wiener process has a rich history in modeling physical system [67].
BlackScholes models [40], long-standing models in studying financial markets are based on Brownian motions.
However, recent work shows fBm is more suitable due to the irregularity of volatility [14].
Correlated noise [7] also can be found in fMRI data.
Applications of fractional Brownian motions can be found in [87] [15].
3.2 Background of Neural SDEs
Consider a dynamical system { X t } whose dynamic evolution is governed by the following stochastic differential equation
[Equation]
where µ : [0 , T ] × R n → R n is the drift and σ : [0 , T ] × R n → R n × m is the diffusion.
Both of those functions are assumed to be well-behaved (i.e., smooth and linear growth) and are parameterized by neural networks.
Readers can refer to [80] for a gentle introduction of SDE in machine learning or to [72] for advanced materials.
[83, 84, 54, 46] are excellent starting points to be familiar with neural
SDEs.
Back-propagation using optimize-then-discretize strategy
This strategy is adopted in neural
ODEs, allowing to perform back-propagation in time [11].
Suppose we are interested in optimizing a loss function L at the terminal state X T .
This approach considers a variable ∂ L /∂X t .
The dynamic of this variable can be modeled by a differential equation leading to computing gradient by using chain rules.
In neural SDEs, such a strategy requires a more careful treatment as [54] addresses the randomness in ∂ L /∂X t .
Also, forward and backward SDEs need to share the same sample paths of Wiener process.
Model learning There are two main parameter estimation methods for Neural SDEs.
The first one is latent SDE [54] that follows a variational approach approximating posterior distributions by a neural SDE which shares the same diffusion with priors.
In this framework, the evidence lower bounds are derived by applying the Girsanov theorem.
Recently, [47] proposes SDE-GAN inspired by generative neural networks [30].
In this paper, we mainly use the former in our experiments.
However, it is worth noting that there is no restriction to training with SDE-GAN as long as chosen models satisfy our assumption.
3.3 Sparse Gaussian processes
We also give a brief introduction of sparse Gaussian processes which are known as scalable methods for Gaussian processes.
A Gaussian process (GP) [71] is defined by a set of random variables, any finite number of which have a joint Gaussian distribution.
Formally, a GP is completely specified by a mean function m ( · ) which usually is set to be zero for prior and covariance function k ( · , · ) .
We then can denote a GP by f ( x ) ∼ GP (0 , k ( x, x ′ )) .
Learning or sampling from a Gaussian process usually takes O ( N 3 ) with the number of data points N .
Sparse Gaussian process methods [74, 37] introduce a set of pseudo points (inducing points) u at locations Z which jointly follow a Gaussian distribution with data points.
The density function of f ( x ) , p ( f ( x )) is then approximated by p ( f ( x ) | u ) p ( u ) .
Here the mean and the variance of p ( f ( x ) | u ) can be computed as
[Equation]
where α ( x ) =
k ( Z, Z ) − 1 k
( Z, x ) .
There is a growing interest in this approach because of its scalability compared to traditional GPs [6, 3, 86, 37].
In this paper, we focus on the sampling procedure where we first sample u ∼ N (0 , k
( Z, Z )) , then sample f ( x ) ∼ p ( f ( x ) | u ) .
Learning White Noises in Neural SDEs
This section presents our main model.
We first approximate stochastic integrals using Stieltjes approximations and then use sparse Gaussian processes to leverage dependencies between random variables.
We analyze the convergence to SDEs solutions for this approximation.
Finally, we form an equivalent SDE to our model (see Figure 1).
4.1 Neural SDEs with B t
Consider a stochastic differential equation
[Equation]
with µ : [0 , T ] × R n → R n and σ : [0 , T ] × R n → R n × m .
Compared to (2), the noise coming from B t is an m -dimensional stochastic process, B t = (cid:82) t 0 f
( t, s ) dW s from (1) as we denote f ( t, s ) =
[Equation]
We are interested in learning both the parameters of µ ( t, X t ) , σ ( t, X t ) and that of f ( t, s ) .
Solving
SDEs in (4) is challenging since B t is constructed from an integral.
Direct sampling methods of
B t (i.e., require computing Cholesky decomposition) are costly.
Other approaches [65] use an approximation of stochastic integral as a linear combination of random variables.
However, the number of these random variables prevents optimize-then-discretize methods to scale.
4.2 Integral approximation
We use Stieltjes integration to approximate the integral in (1) by discretizing the interval [0 , T ] with 0 =
s 1 < s
[Equation]
< · · · < s N = T , i.e.,
[Equation]
where ξ i are independently distributed standard normal random variables coming from W s i +1 − W s i ∼ N (0 , ∆ s i ) with ∆ s i
=
s i
+
− s i , and 1 { s ∈ [0 , t ] } is the indicator function.
Note that this is a reasonable approximation because Itô’s integrals, in general, converge in L 2 when N goes to infinity.
Moreover, even if W t has infinite variation, the probability of a sample path leading to the divergence of approximation infinitely often is zero [72].
The discretization is also inevitable whenever it is required to sample B t .
In several numerical approaches, for example, Euler-Maruyama [51], solving stochastic differential equations involves approximating increments, dB t , by
[Equation]
where ∆ g i ( t )
=
g i ( t + ∆ t ) − g i ( t ) .
Comparing to increments ∆ W t when solving (2), simulating ∆ W t is relatively simple as they are independent between each time step.
On the other hand, there exist dependencies between ∆ B t as
[Equation]
Here, ξ i , ξ j ∼ N (0 , 1) , and E [ ξ i ξ j ] = 0 when i ̸
=
j .
Our aim is to represent (6) in a way that is simple to sample from but preserves the dependencies between ∆ B t .
4.3 Sampling increments with sparse Gaussian process
From the observation in (7), we devise a sampling method inspired by sparse Gaussian processes.
In particular, we consider a Gaussian process 1 B ′ t =
∆ B t ∆ t for a fixed ∆ t .
In fact, ∆ t is later used as the time step of SDE solvers.
Note that directly modeling ∆ B t in (6) is numerically unstable when computing (7) with covariance matrices having small values.
Indeed, B ′ t is a Gaussian process because at a given time t , B ′ t is Gaussian with a zero mean.
To compute the covariance function k ( t, s ) , we simply compute k ( t, s ) =
E [ B ′ t B ′ s ] − E
[ B ′ t ] E [ B ′ s ]
=
E [ B ′ t B ′ s ] and follow the same step with (7)
[Equation]
[Equation]
To this end, we denote B ′ t ∼ GP (0 , k ( t, s )) .
Sampling from a Gaussian process based on a full covariance matrix is not scalable.
To overcome this difficulty, we propose a sampling scheme based on sparse-inducing GP [74].
The main idea of this method is to introduce a bottleneck as a summation of the joint distribution of B ′ t .
Define inducing-point locations as Z = { z 1 , . .
. , z M } which spread across [0 , T ] .
The number of inducing points, M , is much smaller than the number of discretization mesh of numerical solvers.
The inducing points u
=
( B ′ z 1 , . .
B ′ z M ) are distributed as a multivariate Gaussian distribution, i.e., u ∼ N (0 , k
( Z, Z )) .
Following the description of sparse GPs in §3.3, p ( B ′ t ) is approximated by p ( B ′ t | u ) p ( u ) .
The sampling procedure of B ′ t consists of two steps:
[Equation]
Here L Z is the Cholesky decomposition of k ( Z, Z ) , I M is an M × M identity matrix.
m ( · ) and v ( · ) are the conditional mean and variance taken from (3).
The above is a complete sampling procedure for p ( B ′ t | u ) p ( u ) .
Note that in our approximation, B ′ t is independent conditioned on that u are given.
Therefore, we obtain the increment ∆ B t as
[Equation]
Here, we replace √ ∆ tξ t with increments of a (new) Brownian motion W t since ξ t are independent for t ∈ [0 , T ] .
Remark One may consider an alternative approach where we use sparse approximation directly for
B t with given kernel function k ( t, s )
∝ | t | 2 h t,s + | s | h t,s − | t − s | 2 h t,s (see (16)), and can obtain a similar form of increments.
However, this kernel function is not differentiable w.r.t the parameters of Hurst function.
Our approach is to devise a “so-called” differentiation of B t under discretization.
Note that this notion differs from the definition of the derivative of B t in [52] in the sense of Schwartz space.
However, such a representation is not useful for a practical implementation.
Our approach shares a similarity with [49] on controlled differential equations driven by a signal, i.e.,
X t .
The signal X t is approximated by linear interpolations which allows to compute the derivative of X t .
We intentionally model the “derivative” of B t by B ′ t first, and then reconstruct the signal B t from B ′ t .
4.4 Forming equivalent SDEs
To this end, we show that the increments ∆ B t can be represented with increments of a Wiener process, ∆ W t in (11).
Here, the equivalence is under the assumption that the numerical procedure involves only ∆ B t , i.e., Euler-Maruyama methods.
Therefore, the original SDE in (4) is approximated by
[Equation]
This is obtained by conversions, dt ≈ ∆ t and dW t ≈ ∆ W t as we expand the diffusion term
[Equation]
where ν ( t ; u, Z ) = m ( t ; u, Z ) , and ς ( t ; Z )
= v ( t ; Z ) √ ∆ t .
The function ς ( t ; Z ) induced from the approximation of B ′ t is now absorbed into the diffusion function while ν ( t ; u, Z ) together with the diffusion function now is added to the drift function.
Note that ν ( t ; u, Z ) and ς ( t ; Z ) is conditioned on the inducing point u .
Therefore, at the initial time t = 0 , we need to precompute u first.
Although (12) and (4) seem similar in terms of the expressiveness of drift functions and diffusion functions, the drift function of (12) contains a randomness stemming from the sampling u in (9) while that of (4) is deterministic.
This can be understood that the dependency between ∆ B t is translated into the drift functions.
We can elucidate the difference via Algorithm 1 and Algorithmin Appendix.
Note that ν ( t ; u, Z ) and ς ( t ; Z ) are associated with two types of discretizations: that of approximating
Stieltjes integral and that of SDE numerical solver.
One of the key advantages of this new SDE is that it can inherit all theory and practices of wellestablished study of stochastic analysis for Brownian motion or Wiener process.
The preliminary results of Girsanov’s theorem, i.e. [34] cannot used in latent SDE [54].
Itô’s rules for fBm and mBm can be found in [52] but hard to use in practice.
On the other hand, our approach can work for any B t defined in (1).
Another benefit of our formulation is that the sampling procedure is simple, requiring only one ξ t at a time and avoids a multiplication of O ( N ) memory like [65].
This becomes significant as the optimize-then-discretize method requires constructing the same sample paths of B t in forward SDEs and backward SDEs.
Computational analysis The evaluation of diffusion functions in (12) requires computing an aggregate sum over discretization that scales linearly with the number of discretization points.
It may cause some overhead when computing Itô correction term during solving backward SDE [54].
Therefore,
Stratonovich SDEs are more efficient than Itô SDEs.
One limitation of integral approximation is that for a long interval [0 , T ] , one may need more discretized partitions to achieve desired approximation accuracy.
Onthecomputational cost of sparse GPs, we only need to perform the Cholesky decomposition at the initial time of every SDE integration.
This takes O ( M 3 ) with M the number of inducing points.
At a time t during solving SDEs, with the pre-computed L Z , sampling B ′ t has O ( N + M 2 ) time complexity as we perform matrix multiplications in (3).
Limitation Currently, our model only considers fixed step sizes in SDE numerical solvers.
Most theoretical studies in stochastic analysis work for H > 1 / 4 ( H > 1 / 3 in this study) to make sense of stochastic integrals or SDE solutions [70].
The Euler-Maruyama update in our approach may not be sufficiently good when the Hurst exponent is less than 1 / 2 as it is pointed out in [16].
Higher-order solvers can be more accurate, requiring handing double integrals, i.e., (cid:82)
(cid:82) dB t dB s which we wish to address in future work.
Although this paper presents an initial theoretical justification of convergence to solutions, the precise order of convergence is not clear yet.
Therefore, for practical use, we recommend avoiding using Hurst exponents with small values in our method.
4.5 Convergence analysis
The error of our approximation is composed of three main factors.
The first is of SDE solvers.
Due to the assumption of fixed step sizes, only a few solvers, i.e., Euler-Maruyama, are applicable with 1 / 2 strong convergence.
Secondly, the discretization in Stieltjes approximations attributing to the total error leads to the convergence in L 2 .
The convergence of such approximations are known.
The approximation of noise from sparse GPs is unconventional in solving SDEs because the approximation is in distributional sense rather than point-wise or path-wise.
We will show the convergence using rough path theory [56, 26] and sparse GPs bound from [8] for constant Hurst
H ∈ ( 1 / 3 , 1 / 2 ) .
We follow the notation in [39, 56, 26].
Let us denote the canonical lift of B as B = ( B, B ) .
Let (cid:98) B be the path corresponding to the sampling procedure in (9) and (10).
Let X be the solution of d X t = f
( t, X t ) d B t and (cid:98)
X be the solution of d (cid:98) X
= f ( t, (cid:98) X t ) d (cid:98) B where f : [0 , T ] × R n → R n is a bounded function that is four-times continuously differentiable.
We assume that the initial state follows a distribution p 0 which is is continuous and strictly positive in R n .
Let us denote the probability density distribution of the solution X at time t by p t ( x ) ; and that of the approximated solution (cid:98) X by (cid:98) p t ( x ) .
Theorem 1 (Convergence of solutions given sparse GP approximation) .
With the Hurst exponent of B , H ∈ ( 1 / 3 , 1 / 2 ) , assume that E [( (cid:98) B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ for any t ∈ [0 , T ] , and ϱ ∈ ( 1 / 2 H , 3 / 2 ] .
Then, for every α ∈ ( 1 / 3 , 1 / 2 ϱ ) , there exists a constant C such that
[Equation]
Here, θ ∈ (0 , 1 / 2 − ϱα ) and KL( ·∥· ) is the Kullback-Leibler divergence.
This theorem says that if the distribution of (cid:98) B converges to that of B in terms of the KL divergence, their corresponding solutions converge in the same sense.
Note that we make an assumption that
E [
[Equation]
B t − (cid:98) B t + τ ) 2 ] ≤ L | τ | 1 / ϱ , ϱ > 1 / 2 H which means that (cid:98) B is less “rough” than B .
This is a reasonable assumption since (cid:98) B suffers information loss from sparse GP approximations.
Our proof is done by (1) assessing the p -variation of (cid:98) B to bound the distance in rough path space between the canonical lifts of B and (cid:98) B via [26, Corollary 10.6]; and (2) using the local Lipschitz property of Itô-Lyons maps [26, Theorem 8.5] as we follow the technique in [39].
The complete proof is provided in Appendix B along with necessary backgrounds and discussion.
According to [8], we can quantify the convergence of (cid:98) B to B as its KL divergence can be bounded,
KL( (cid:98) B ∥ B ) ≤ C (cid:80) ∞
i =
M +1 λ i .
Here λ i are eigenvalues in eigenvalue-eigenfunction pairs ( λ i , ϕ i ( · ) ) of Mercer’s theorem [62], k ( x, x ′ )
= (cid:80) ∞
i =
λ i ϕ
i ( x ) ϕ i ( x ′ ) .
Recall M is the number of inducing points.
Therefore, the convergence of KL( (cid:98) B ∥ B ) depends on the decaying rate of eigenvalues and the number of inducing points.
We empirically show the fast decaying rate of eigenvalues for finite samples which also can be found in Appendix B.
4.6 Parameterizing white noise
The previous section concerns how to learn white noises in neural SDEs without providing concrete forms of white noise.
This section will show how it affects SDEs in general and discuss the parameterization form of the Hurst exponent.
When the Hurst exponent h ( t ) is constant, say h ( t ) =
H .
There is a qualitative difference between settings of H .
Figure 2 shows how the Hurst parameter affects the model fit trained by Latent SDE.
With a smaller value ( H = 0 . 3 ), sample paths are rough, producing more irregularity.
When H = 0 .
8 , the model has smoother sample paths, and is more conservative as predictions depend on the past.
This parameter has a great influence on the uncertainty quantification of the models.
[Figure]
For the case that h ( t ) is function of t , any function approximators can work in general to model h ( t ; θ ) .
In our implementation, we use small neural networks.
With the new representation in (12), we can train the parameters of h ( t ; θ ) together with the parameters of drift and diffusion.
A more general approach is that f ( t, s ) can accept either neural networks or kernel-based functions when we do not consider normalization.
However, their interpretation is not clear as that of mBm.
Experimental Results
This section will first justify how close the samples generated by our method are to true ones.
We then investigate the ability to learn the Hurst function for synthetic data and test our model for real-world data.
Finally, we examine the role of Hurst exponents in score-based generative models.
All experiments are implemented using PyTorch [68] with the library torchsde [54, 47].
Our source code is available at this repository.
How to estimate Hurst from data?
We follow [27] to empirically estimate Hurst exponent from data.
The key technique is based on the correlation of increment of B t and estimates a ratio over sliding windows which is an estimation of Hurst exponent.
5.1 Quality of approximate samples
We would like to investigate if our approximation of B ′ t can produce the sample path of B t that is close enough to those of the true process.
The true Hurst function is set as h ( t ) = 0 .
35 + 0 .
5 t .The
samples from [65, 25] are used as a baseline.
This experiment runs 1000 samples.
The estimations of the Hurst exponent for two sampling techniques are in Figure 3.
The Hurst estimation corresponding to our samples is close to the true Hurst function.
It tends to overestimate the long-range dependency ( h ( t ) > 1 / 2 ) and underestimate the irregularity ( h ( t ) < 1 / 2 ) in the samples.
5.2 Recovering Hurst function
[Figure]
Next, we test our model with a synthetic data.
Here the data is generated from a SDE with a known solution:
[Equation]
where B t is a mBm given a fixed h ( t ) = 0 .
3 + 0 .
5 sigmoid (7(1 − t )) .
According to Itô integrals for mBm cases [52], the exact solution is X t = x 0 exp( αt − 12 β 2 t 2 h ( t ) + βB t ) .
We generatedata points for each sample path and obtain up tosample paths.
After training our model, we observe that the obtained Hurst function can capture the dynamic similar to the true Hurst.
As we increase the number of sample paths (from 1 to 5 ), our learned
Hurst tends to approach the true Hurst.
Figure 4 shows an example of our learned Hurst function given 5 sample paths.
We also provide the results for different Hurst functions and an ablation study in Appendix C.3.
[Figure]
5.3 Financial data
In this experiment, we compare the Hurst function found by our model with the time-dependent Hurst estimation computed by [27].
Data set We collect two months (from September 26th, 2021 to October 23rd, 2021) of hourly data from four stock indices Amazon (AMZN), Apple (APPL), Google (GOOGL), Microsoft (MSFT).
The data is retrieved from [85].
It is then standardized and split into the first 80% time points to train and the remaining 20% to test.
Obtaining Hurst function Figure 5 compares our found Hurst functions and estimated Hurst computed from [27].
Our obtained Hurst functions have bigger values than that of [27].
This resembles the observation in the previous experiment.
A common observation among these stocks is that Hurst functions made some changes (bump up or drop) around the end of October.
This agrees with many analyses, for example, Amazon missed revenue expectations for the third quarter or Apple revealed new products.
The posterior plots for our model in each stock index are placed in
Appendix C.4.
Quantitative comparison We compare our model to two main baselines: Latent SDE with standard
Brownian motion [54] and Latent ODE [12].
We report three main comparison criteria: training negative log-likelihood (training NLL), test negative log-likelihood (test NLL) and test root mean square error (test RMSE).
Overall, our model can fit training data well as our model outperforms others in terms of training NLLs (see Appendix C.4).
This is because our model can capture correctly noise part while Latent SDE relies on less flexible Brownian noise.
Still, it is not clear to identify which model performs better in predictive tasks.
However, our model produces more consistent results with less variance.
5.4 Effect of Hurst exponent in score-based generative models
This experiment explores a different aspect by studying the effect of Hurst exponent in training score-based generative models.
Recent work [77] on generative models based on score functions and
SDEs presents impressive results on par with generative adversarial networks (GAN).
We consider the case where the perturbation in data is under a new SDE, dX t = − 12 β ( t )
X t dt + (cid:112) β
( t ) dB t where
B t = (cid:82) t 0 f ( t, s ) dW s .
We derive the equivalent SDE (discussed more in Appendix C.5):
[Equation]
Varying Hurst exponents We investigate how different types of noise affect learning and sampling in this model.
We consider four scenarios of noise: (1) more anti-persistent as H = 0 . 3 ; (2) longrange dependent with H = 0 . 8 ; (3) linearly increasing h ( t ) ; (4) linearly decreasing h ( t ) .
We only consider the task of generating MNIST images [53].
Figure 6 shows example of generated images.
We observed that H > 1 / 2 is not favorable, and requires SDEs with long terminal time T .
To quantify the quality of generated images, we compute the negative log-likelihood (NLL) of a set of out-of-sample images like in [76].
Discussion Although we do not present a way to find optimal h ( t ) , we show that there is a setting in which the NLL appears to be better (see Figure 7 a).
That is, the NLL of decreasing Hurst setting is 3 .
79 (bits/dim) compared to 4 .
97 (bits/dim) of Brownian motion ( H = 0 . 5 ).
There exists work [50] showing that signal-to-noise ratio (SNR) is the key to establishing an optimal strategy how to inject noises.
There is another work studying the best strategy for reverse variance [2].
In contrast to many diffusion models, our approach has an additional term (cid:112) β ( t ) ν ( t ; u ) which is unrelated to X t but is included in the drift of the backward SDEs when generating data (see Figure 7 b).
Moreover, our approach can suggest a way to control the sample path of B t in the way that we condition on inducing
H = 0 .
H = 0 .increasing decreasing
[Figure]
points u since B ′ t is sampled from the conditional distribution p ( B ′ t | u ) . For example, we can follow the variational inference of [37] to find the posterior q ( u ) for a designed objective, then sample B ′ t given q ( u ) instead of p ( u ) .
Conclusion
In conclusion, this work presents a new class of neural SDEs with a generalization of white noise.
We propose an efficient approximate sampling approach for the noises so that neural SDEs can be converted into convenient forms.
The techniques used in this paper are relatively simple, and related to Gaussian process (GP) research.
Our experiments have verified the ability of our models to learn noises from synthetic data to financial time-series as well as showcase an image generation task.
As mentioned earlier, our current work can be improved with higher-order solvers where the double integrals require to model precisely.
Prior work [52, 70] is a good starting point for such an approach.
Another promising direction is to extend neural controlled differential equations with driving signals which are general GPs instead of Brownian motions.
The closure property of GPs under differentiation will be useful to develop this model.
Acknowledgements
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD) and was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00984, Artificial Intelligence, Explainability, Personalization, Plug and Play,
Universal Explanation Platform), (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)), (No. 2022-0-00184, Development and Study of AI Technologies to Inexpensively
Conform to Evolving Policy on Ethics), and KAIST-NAVER Hypercreative AI Center.
In addition, this research was supported, in part by the DARPA GARD award HR00112020004, NSF CAREER award IIS- and NSF BIGDATA award IIS-1838139 (to TNT thank reviewers feedback
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the last paragraph of Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Mentioned in the main text and more detailed in Appendix B (b) Did you include complete proofs of all theoretical results?
[Yes] Complete proof can be found in Appendix B 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] MNIST dataset for image generation tasks.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]