Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamics
Abstract
Stochastic partial differential equations (SPDEs) are the mathematical tool of choice for modelling spatiotemporal PDE-dynamics under the inﬂuence of randomness.
Based on the notion of mild solution of an SPDE, we introduce a novel neural architecture to learn solution operators of PDEs with (possibly stochastic) forcing from partially observed data.
The proposed Neural SPDE model provides an extension to two popular classes of physics-inspired architectures.
On the one hand, it extends Neural CDEs and variants – continuous-time analogues of RNNs – in that it is capable of processing incoming sequential information arriving at arbitrary spatial resolutions.
On the other hand, it extends Neural Operators – generalizations of neural networks to model mappings between spaces of functions – in that it can parameterize solution operators of SPDEs depending simultaneously on the initial condition and a realization of the driving noise.
By performing operations in the spectral domain, we show how a Neural SPDE can be evaluated in two ways, either by calling an ODE solver (emulating a spectral Galerkin scheme), or by solving a ﬁxed point problem.
Experiments on various semilinear SPDEs, including the stochastic Navier-Stokes equations, demonstrate how the Neural SPDE model is capable of learning complex spatiotemporal dynamics in a resolution-invariant way, with better accuracy and lighter training data requirements compared to alternative models, and up to 3 orders of magnitude faster than traditional solvers.
Introduction
Stochastic partial differential equations (SPDEs) are the mathematical formalism used to model many physical, biological and economic systems subject to the inﬂuence of randomness, be it intrinsic (e.g. quantifying uncertainty) or extrinsic (e.g. modelling environmental random perturbations).
Notable examples of SPDEs include the Kardar–Parisi–Zhang (KPZ) equation for modelling random interface growth such as the propagation of a forest ﬁre from a burnt region to an unburnt region [12], the Ginzburg-Landau equation describing phase transitions of ferromagnets and superconductors near critical temperature [34], or the stochastic Navier-Stokes equations modelling the dynamics of a turbulent ﬂuid ﬂow under the presence of local random ﬂuctuations [31].
For an introduction to the theory of SPDEs see Hairer [11]; a comprehensive textbook is Holden et al.
Classical numerical approaches for solving SPDEs include ﬁnite difference methods and spectral
Galerkin methods [27] among others.
To ensure accuracy and stability of numerical solutions of complex SPDEs, computations must be carried out at high resolution using ﬁne discretization grids, rendering the resulting schemes computationally intractable.
This limitation motivates the study of data-driven methods that can learn solutions to differential equations from partially observed data.
Related work There has been an increased interest in recent years to combine neural networks and differential equations into a hybrid approach [5, 17, 37].
Neural controlled differential equations (Neural CDEs), as popularised by [3, 18, 32], are continuoustime analogues to recurrent neural networks (RNN, GRU, LSTM etc.).
The input to a Neural CDE model is a multivariate time series interpolated into a continuous path X : [0 , T ] → R d ξ ; the model consists of a matrix-valued feedforward neural network f θ : R d h → R d h × d ξ parameterizing the vector ﬁeld of the following dynamical system (and satisfying some minimal Lipschitz regularity to ensure existence and uniqueness of solutions)
[Equation]
where (cid:96) θ : R d u → R d h and π θ : R d h → R d u are feedforward neural networks.
The output response u : [0 , T ] → R d u is then fed to a (possibly pathwise) loss function (mean squared, cross entropy etc.) and trained via stochastic gradient descent in the usual way.
In practice, the term “ dX t ” means that the solution z t of the equation (e.g. attention required from a doctor) can change in response to a change of an external stream of information X t (e.g. heart rate of a patient).
Depending on the level of roughness of the control path X , the integral in eq.
(1) can be interpreted in different ways.
In [18], X is assumed differentiable and is obtained in practice via cubic splines interpolation of the original time series.
In this way, the term “ dX s ” can be interpreted as “ ˙ X s ds ” so that eq.
(1) becomes an ODE of the form ˙
z t = f θ ( z t ) ˙
X t that can be evaluated numerically via a call to an ODE solver of choice (Euler, Runge-Kutta, implicit, adaptive stepsize schemes etc.).
More generally, if X is of bounded variation then the integral above can be seen as a classical
Riemann–Stieltjes or Young integral [38].
Neural SDEs [19, 20, 22, 26] are a special subclass of
Neural CDEs where the control is a sample path from a d ξ -dimensional Brownian motion (which is not of bounded variation), and eq.
(1) is understood via stochastic integration (Itô, Stratonovich etc.).
Neural RDEs [32] allow to relax even further the regularity assumptions on X by treating the integral using rough integration [10, 30].
In practice, Neural RDEs are particularly well suited for long time series.
This is due to the fact that the model can be evaluated via a numerical scheme from stochastic analysis (called the log-ODE method [32]) over intervals much larger than what would be expected given the sampling rate of the time series.
However, the space complexity of the numerical solver increases exponentially in the number of channels d ξ , thus model complexity becomes intractable for high dimensional time series.
Despite offering many advantages for modelling temporal dynamics, these models are not designed to process signals varying both in space and in time such as physical ﬁelds described by SPDEs.
In particular, although these models are time-resolution invariant, they are not space-resolution invariant, and are not well suited to capture nonlinear interactions between the various space-time points typically observed in SPDE-dynamics.
Similar to CDEs, the solution u to an SPDE is characterized by an initial condition u 0 and a driving noise X .
However, in the case of CDEs, ( u 0 , X t , u t ) are vectors, while in the case of SPDEs they are functions.
Neural Operators [21, 23, 24, 28] are generalizations of neural networks capable of modelling mappings between spaces of functions and offer an attractive option for learning with spatiotemporal data [21].
Among all kinds of Neural Operators, Fourier Neural Operators (FNOs)
[25] stand out because of their easier parametrization while demonstrating similar learning performance compared to other Neural Operator models.
However, Neural Operators generally fail to incorporate the effect that an external (possibly random) spatiotemporal signal might have on the system they describe.
In the case of SPDEs, the external signal is indeed random (e.g. sample from a Wiener process) and its presence leads to new phenomena, both at the mathematical and the physical level, often describing more complex and realistic dynamics than the ones arising from deterministic PDEs.
Contributions To overcome the above limitations faced by Neural CDEs and Neural Operators, we introduce the neural stochastic partial differential equation (Neural SPDE) model, capable of learning solution operators of SPDEs from partially observed data by processing, continuously in time and space, incoming sequential information arriving at an arbitrary resolution.
We propose two separate algorithms to evaluate our model: the ﬁrst reduces the Neural SPDE to a system of
ODEs in Fourier space, which can then be solved numerically by means of any ODE solver of choice (emulating a spectral Galerkin scheme); the second rewrites the Neural SPDE as a ﬁxed point problem, which is solved via classical root-ﬁnding schemes.
For both choices of evaluation, the
Neural SPDE model inherits memory-efﬁcient backpropagation capabilities provided by existing adjoint-based and implicit-differentiation-based methods respectively.
Finally, we perform extensive experiments on various semilinear SPDEs, including the stochastic Ginzburg-Landau, Korteweg-De
Vries, Navier-Stokes equations.
The empirical results illustrate several useful aspects of our model: 1) it is space and time resolution-invariant, meaning that even if trained on a lower resolution it can be directly evaluated on a higher resolution; 2) it requires a lower amount of training data to achieve similar or better performance compared to alternative models; 3) its evaluation is up to 3 orders of magnitude faster than traditional numerical solvers.
The outline of the paper is as follows: in Sec.
2 we provide a brief introduction to SPDEs which will help us to deﬁne our Neural SPDE model in Sec. 3, followed by numerical experiments in Sec. 4.
In Appendix A, we provide an overview of the computational aspects of SPDEs used to design the model and solve SPDEs numerically.
Additional experiments can be found in Appendix B.
Background on SPDEs
Let T > 0 and d, d u , d ξ ∈ N .
Let D ⊂ R d be a bounded domain.
Let H u
= { f : D → R d u } and
H ξ = { f : D → R d ξ } be two Hilbert spaces of functions from D to R d u and R d ξ respectively.
We consider a large class of SPDEs of the following type
[Equation]
where W t is either an inﬁnite dimensional Q -Wiener process [27, Def. 10.6] or a cylindrical Wiener process [11, Def. 3.54] with values in H ξ , F : H u
→ H u and G : H u
→ L ( H ξ , H u ) are two continuous operators, L ( H ξ , H u ) is the space of bounded linear operators from H ξ to H u , and L is a linear differential operator generating a semigroup 1 e t L : H u
→ H u .
For further details on Wiener processes see Appendix A.2, and for a primer on semigroup theory see Hairer [11, Section 4].
A function u : [0 , T ] → H u is said to be a mild solution of the SPDE (2) if for any t ∈ [0 , T ] it satisﬁes
[Equation]
where the second integral is a stochastic integral interpreted in the Itô sense [11, Def. 3.57].
Thus, an
SPDE can be informally thought of as an SDE with values in the functional space H u and driven by an inﬁnite dimensional Brownian motion W .
Assuming global Lipschitz regularity on F and G , a mild solution u to (2) exists and is unique [11, Thm. 6.4], at least for short times.
We follow Friz and Hairer [9] and consider a regularization W (cid:15) =
ϕ (cid:15) ∗ W of the driving noise W with a molliﬁer 2 , where ∗ means convolution.
As done in Kidger et al.
[18] for Neural CDEs, we can rewrite the mild solution of the molliﬁed version of eq.
(2) as the following randomly forced PDE
[Equation]
where ξ = ˙ W (cid:15) ,
H u
= L 2 ( D , R d u ) and H ξ =
L 2 ( D , R d ξ ) .
We will refer to ξ as white noise if W is a cylindrical Wiener process and as coloured noise if W is a Q -Wiener process.
In view of machine learning applications, one should think of W as a continuous space-time embedding of an underlying spatiotemporal data stream.
In this paper we are only going to consider W to be a sample path from a Wiener process, but we emphasise that the Neural SPDE model extends, in principle, beyond the scope of SPDEs and could be used for example to process videos in computer vision applications, which we leave as future work.
Next we introduce the Neural SPDE model
( u 0 : 𝒟 → ℝ d u , ξ : [0, T ] → L 2 (𝒟,ℝ d ξ ) )
( z 0 : 𝒟 → ℝ d h , ξ : [0, T ] → L 2 (𝒟,ℝ d ξ ) ) z : [0, T ] → L 2 (𝒟,ℝ d h ) u : [0, T ] → L 2 (𝒟,ℝ d u ) L θ : ℝ d u → ℝ d h Π θ : ℝ d h → ℝ d u Φ θ , ξ ( z ) t =
ℱ −1 d ( ℱ −11 ( B ) t ℱ d ( z 0 ) )
{++ ℱ −1 d +
+}
( B ℱ d +1 ( 1 ≥0 H θ , ξ ( z ⋅ ) ) )
t
Fixed Point Solver :
z = Φ θ , ξ ( z )
ODE Solver : z t = ℱ −1 d ( v t ) v t
{+= v 0 + ∫ t 
+}
Av s + ℱ d
( H θ , ξ (ℱ −1 d ( v s ))), v 0 =
ℱ d ( z 0 )
Time tttt =
T
[Figure]
t
[Figure]
Time t
[Figure]
t 2 t 3 t =
T … t =
Physical Space
Latent Space
Loss ℒ( u , u true )
Solution Operator
[Figure]
Neural SPDEs
For a large class of differential operators L , the action of the semigroup e t L can be written as an integral against a kernel function K t : D × D → R d u × d u such that
[Equation]
for any h ∈ H u , any x ∈ D and t ∈ [0 , T ] , and where µ t is a Borel measure on D .
As in Kovachki et al.
[21], here we take µ t to be the Lebesgue measure on R d but other choices can be made, for example to incorporate prior information.
We assume that K is stationary so that eq.
(3) can be rewritten in terms of the spatial convolution ∗
[Equation]
For a large class of SPDEs of the form (2), both F and G are local operators acting on a function h ∈ H u .
In other words, the evaluations F ( h )( x ) and G ( h )( x ) at any point x ∈ D only depend h ( x ) , and not on the evaluation h ( y ) at some other point y ∈ D in the neighbourhood of x .
3.1 The model
Let H h
=
L 2 ( D , R d h ) for some latent space dimension
d h > d u .
Let
[Equation]
be four feedforward neural networks.
For any differentiable control ξ : [0 , T ] → H ξ , deﬁne the map
H θ,ξ : H u
→ H u so that for any h ∈ H u , x ∈ D , t ∈ [0 , T ]
[Equation]
A Neural SPDE is deﬁned as follows
[Equation]
We note that globally Lipschitz conditions can be imposed by using ReLU or tanh activation functions in the neural networks F θ and G θ .
In secs.
3.2 and 3.3 we propose two distinct algorithms to evaluate the Neural SPDE model (4) which are based on two different parameterization of the kernel K .
3.2 Evaluating the model by solving a system of ODEs
Levereraging the convolution theorem , we can rewrite the integral in eq.
(4) as follows
[Equation]
where F , F − 1 are the d -dimensional Fourier transform (FT) and its inverse (see Def. A.1).
If one further assumes that L is a polynomial differential operator, it can be shown that there exists a map
A : C d → C d h × d h such that F ( K t )( y ) =
e tA ( y ) (see appendix B.1 for a derivation).
It follows that
[Equation]
where v t : C d → C d h is the solution of the following ODE
[Equation]
Hence, z t can be obtained by applying the inverse FT to the output of an ODE solver on [0 , t ] with initial condition F ( z 0 ) , vector ﬁeld Ψ
[Equation]
:=
[Equation]
+
[Equation]
This approach can naturally be seen as a “neural version” of the classical spectral Galerkin method for SPDEs as described in Appendix A.3.
We note that this numerical evaluation of the Neural SPDE model (4) allows to inherit memoryefﬁcient adjoint-based backpropagation capabilities as in the evaluation of a Neural CDE.
For further details on adjoint-based backpropagation we refer the reader to Chen et al.
[5], Kidger [17].
3.3 Evaluating the model by solving a ﬁxed point problem
In our second approach of model evaluation, we make use of three different versions of the FT: the time-only FT F 1 and its inverse F − 1 1 , the space-only FT F d and its inverse F − 1 d , and the space-time
FT F d +1 and its inverse F − 1 d +1 (see Def.
A.1 for details).
Denoting by (cid:63) the space-time convolution, the integral in eq.
(4) can be rewritten as
[Equation]
where 1 ≥ 0 is the indicator function restricting the temporal domain to the positive real line.
Using again the convolution theorem we obtain
[Equation]
where all multiplications are matrix-vector multiplications.
Using the trick introduced in [25], one can parameterize F d +1 ( K )( y ) directly in Fourier space as a complex tensor B , so that the solution of eq.
(4) can be obtained by solving the ﬁxed point problem z = Φ θ,ξ ( z ) with
[Equation]
and where we used the fact that F d ( K t ) =
F − 1 1 ( F d +1 ( K )) t .
This can be solved numerically using classical root-ﬁnding schemes (e.g. by Picard’s iteration)
[Equation]
Analogously to adjoint-based backpropagation for the evaluation approach mentioned in Sec. 3.2, there is a mechanism that leverages the implicit function theorem allowing to backpropagate through the operations of a ﬁxed point solver in a memory-efﬁcient way.
See Bai et al.
[2] for further details.
We note that the FTs are numerically approximated using the discrete Fourier transform (DFT) and selecting a maximum number of frequency modes 3 (see Appendix A.1 for details).
3.4 Space-time resolution-invariance
As depicted in Fig.
1, the input to a Neural SPDE corresponds to a (possibly irregularly sampled) time-indexed sequence of (possibly partially-observed) spatial observations recorded on a space-time grid.
The data is then interpolated into a continuous spatiotemporal signal ξ and initial condition u 0 .
By construction, a Neural SPDE operates in continuous time and space on the tuple of functions ( u 0 , ξ ) and produces a spatiotemporal response u , which is also continuous in space and time; the function u can then be evaluated at an arbitrary space-time resolution, possibly different from the one used during training.
In the next section we will demonstrate empirically that even if trained on a coarser resolution, a Neural SPDE can be evaluated on a ﬁner resolution without sacriﬁcing performance, a property known as zero-shot super-resolution .
3.5 Comparison of the two evaluation methods
Number of parameters For a ﬁxed dimension d h of the latent space, the majority of trainable parameters in the ODE parameterization lies in the complex tensor A , which consists of k 1 max ...
k d max d 2 h parameters, where each k i max is the maximum number of selected frequencies in the Fourier domain.
Regarding the Fixed Point parameterization, the bulk of the parameters is in the complex tensor B , which consists of k 1 max ...
k d +1 max d 2 h , where the additional frequency is due to the fact that we are taking the FFT in space-time rather than just in space as done in the ODE Solver approach. Hence, the latter would in principle have the advantage of using a lower number of parameters than the former; however, to achieve similar performance, we found that the dimensionality of the latent space has to be roughly 20 times higher, which offsets the aforementioned advantage. Time complexities The time complexity of the ODE Solver approach is O ( NN x log( N x )) , where
N is the number of time steps taken by the ODE solver and N x is the number of points on the spatial grid, while the complexity of the Fixed Point approach is O ( IN x N t (log( N x ) + log( N t )))
where
I is the number of Picard iterations and N t is the number of points on the temporal grid.
In our experiments we choose N t ≈ N x and IN t ≈ N , making the two complexities comparable.
Speed of computation We found that the ODE approach is approximately 10 times slower than the Fixed Point approach.
We believe this is largely an implementation issue of the torchdiffeq library, while the FFT is a highly optimised transform in Pytorch.
3.6 Considerations about convergence
We follow Friz and Hairer [9] and consider a regularization W (cid:15) =
ϕ (cid:15) ∗ W of the driving noise W with a compactly supported smooth molliﬁer ϕ (cid:15) .
It is a classical result (Wong-Zakai [35]) from rough path theory [10, 30] that, for the case of SDEs, the sequence of random ODEs driven by the molliﬁcation of Brownian motion converges in probability to a limiting process that does not depend on the choice of molliﬁer and agrees with the Stratonovich solution of the SDE.
Furthermore, the solution map ( u 0 , W ) (cid:55)→ u is continuous in an appropriate rough path topology.
This result nicely extends to the setting of SPDEs driven by a ﬁnite dimensional noise [9, Thm. 1.3]: if u (cid:15) denotes the random
PDE solutions driven by ˙ W (cid:15) dt (instead of ◦ dW t ), then u (cid:15) converges in probability to a limiting process corresponding to the Stratonovich solution of the SPDE.
In our setting though, the driving noise is inﬁnite dimensional and the resulting integral cannot be interpreted in the Stratonovich sense because otherwise the corresponding Itô-Stratonovich correction would be inﬁnite.
Nonetheless,
Hairer and Pardoux [14, Thm. 1.1] show that, in the case of the heat operator and under appropriate renormalization and drift correction, the random PDE solution u (cid:15) converges in probability to the Itô solution of the SPDE, and that the solution map is continuous in an appropriate regularity structures topology.
We note that extending this result to a generic differential operators would require a similarly rigorous proof, which goes beyond the scope of this article and that we leave as future work.
In this section, we run experiments on three semilinear SPDEs: the stochastic Ginzburg-Landau equation in 4.1, the stochastic Korteweg-De Vries equation in 4.2, and the stochastic Navier-Stokes equations in 4.3.
We note that although the assumption of globally Lipschitz vector ﬁelds might be violated for the following SPDEs, well-posedness (i.e. existence of global solutions) can be shown using equation-speciﬁc arguments.
We consider three supervised operator-learning settings:
• u 0 (cid:55)→ u , assuming the noise ξ is not observed; • ξ (cid:55)→ u , assuming the noise ξ is observed, but the initial condition u 0 is ﬁxed across samples; • ( u 0 , ξ ) (cid:55)→ u , assuming the noise ξ is observed and u 0 changes across samples.
We note that learning the operator u 0 (cid:55)→ u of an SPDE without observing the driving noise ξ unavoidably yields poor results for all considered models as only partial information about the system is provided as input.
However, we ﬁnd it informative to include the performances obtained in this setting, as this provides a sanity check that emphasizes the importance of the noise in all the experiments we consider in this paper.
Moreover, the ability to process the initial condition u 0 on its own (in absence of noise) testiﬁes that Neural SPDEs can also be used to learn deterministic PDEs.
We provide an example on the deterministic Navier-Stokes equations in Appendix B.5.
Neural CDE, Neural RDE, FNO and DeepONet [28, 29] will be the main benchmark models.
In addition, we also propose an additional baseline Neural CDE-FNO, which is a hybrid model consisting of a Neural CDE where the drift is modelled by an FNO and the diffusion by a feedforward neural network.
The motivation for using a FNO to represent the drift comes from the universal approximation properties of FNOs studied in Kovachki et al.
[21, Thm. 4].
An interesting line of work to tackle SPDE-learning is provided in [7, 16].
The authors construct a set of features from the pair ( u 0 , ξ ) following the deﬁnition of a model from the theory of regularity structures [13].
They then perform linear [7] and nonlinear [16] regression from these features to the solution of the SPDE at a single time point.
Therefore, these models would have to be retrained for any new prediction.
In addition, both [7, 16] assume knowledge of the differential operator L governing the dynamics, while Neural SPDE learns a representation of L via the parametrization of the associated kernel.
For these reasons these recent models are not included in our benchmark.
For all the experiments, the loss function is the relative pathwise L 2 error.
The hyper-parameters for all the models are selected by grid-search (see Appendix B.2 for further experimental details).
Experiments are run on a Tesla P100 NVIDIA GPU.
The code for the experiments is provided in the supplementary material.
Additional experiments may be found in Appendix B.
4.1 Stochastic Ginzburg-Landau equation
We start with the stochastic Ginzburg-Landau equation, a reaction diffusion equation in 1D given by
[Equation]
This equation is also known as the Allen-Cahn equation in 1 -dimension and is used for modeling various physical phenomena like superconductivity [34].
Here ξ denotes space-time white noise with sample paths generated using classical sampling schemes for Wiener processes detailed in A.2.
[Table]
We consider two data-regimes: a low data regime where the total number of training observations is N = 1 000 , and a large data regime where N = 10 000 .
In both cases, the response paths are generated by solving the SPDE along each sample path of the noise ξ using a ﬁnite difference scheme described in Appendix A.3 using 128 evenly distanced points in space and time and step size ∆ t = 10 − 3 .
Following the same setup as in Chevyrev et al.
[7, eq.
(3.6)], we solve the SPDE until
T = 0 .
05 resulting in 50 time points . We choose as initial condition u 0 ( x ) =
x (1 − x )+ κη ( x ) , {+with η ( x ) = a 
+}
+ (cid:80) k
=10 k = − 10 a k / (1 + | k | 2 ) sin ( kπx ) where a k ∼ N (0 , 1) .
We take κ = 0 and κ = 0 .
1 to generate a dataset where the initial data is either ﬁxed or varies across samples.
We provide extra experiments on this SPDE for larger time horizons T and multiplicative forcing in Appendix B.3.
We report the results in Table 1.
The Neural SPDE model (NSPDE) yields the lowest relative error for all tasks, reaching one order of magnitude improvement on the main task ( u 0 , ξ ) (cid:55)→ u in the large data regime compared to all the applicable benchmark models (NCDE, NRDE, NCDE-FNO).
In all settings, even with a limited amount of training samples ( N = 1 000 ), NSPDE achieves ∼ 1% error rate, and marginally improves to < 1% error when N = 10 000 .
4.2 Stochastic Korteweg–De Vries equation
Next, we consider the stochastic Korteweg–De Vries (KdV) equation, a higher order SPDE given by ∂ t u + γ∂ 3 x u = 6 u∂ x u + ξ, u ( t, 0) =
u ( t, 1) , u (0 , x ) = u 0 ( x ) , ( t, x ) ∈ [0 , T ] × [0 , 1] .
This equation is used to describe the propagation of nonlinear waves at the surface of a ﬂuid subject to random perturbations (another wave equation is studied in Appendix B.4).
We refer the reader to Wazwaz [36] for an overview on the KdV equation and its relations to solitary waves.
The stochastic forcing is given by ξ = ˙ W for W being a partial sum approximation of a Q-Wiener process as per Example 10.8 in Lord et al.
[27] with λ j ∼ j − 5+ ε and φ j ( x ) = sin ( jπx ) (see eq.
(6) in Appendix A.2).
Taking small ε > 0 guarantees that W t is twice differentiable in space for every t ≥ 0 .
To generate the datasets [Table] we solve the SPDE with γ = 0 .
1 until T = 0 .
[Table]
[Table]
The stochastic forcing is simulated using 128 evenly distanced points in space and a time step ∆ t ref = 10 − 3 .
We then approximate realizations of the solution of the KdV equation using a time step ∆ t = 10 − 2 until T = 0 .
Here, the initial condition is given by u 0 ( x ) = sin(2 πx ) + κη ( x ) , where η is deﬁned as in Sec. 4.1.
Similarly to Sec.
4.1 we either take κ = 0 or κ =to generate datasets where the initial condition is either ﬁxed or varies across samples.
Each dataset consists of N = 1 000 training observations.
As reported in Table 2a, Neural SPDEs outperforms the second best model FNO by a full order of magnitude in the task ξ (cid:55)→ u and the second best model
NCDE-FNO by almost two orders of magnitude in the task ( u 0 , ξ ) (cid:55)→ u .
We also perform the same tasks for a larger time horizon T = 1 and report the results of a comparison against FNO in Table 2b.
Partial observations Neural SPDEs are able to process signals that are irregularly sampled both in space and in time by interpolating between observations.
Yet, the ability of a model to process irregular data does not guarantee its robustness when some observations are dropped. Robustness can only be guaranteed if the signal is regular enough so that replacing dropped observations by interpolation results in a new signal that is close, in some suitable norm, to the original signal.
To illustrate this point, we run two additional experiments where we drop uniformly at random 1) 10% of the data in time and 2) 50% of the data in space. As it can be observed in Table 2c, the performance of Neural SPDE remains roughly unchanged when data is dropped in space but decreases when data is dropped in time, which is to be expected since the driving signal is a Q-Wiener process, which is rough in time, but smoother in space.
We also note that to ensure a good approximation of the FT by the FFT, the interpolation must translate the irregular data to a (possibly ﬁner) regular grid.
4.3 Stochastic Navier-Stokes equations in 2D
Finally, we consider the vorticity form of the Navier-Stokes equations for an incompressible ﬂow ∂ t w − ν ∆ w = − u · ∇ w + f + σξ, w (0 , x ) =
w 0 ( x ) , ( t, x ) ∈ [0 , T ] × [0 , 1] 2 , (5) where u is the unique divergence free ( ∇ · u = 0 ) velocity ﬁeld such that w = ∇ × u .
These equations describe the motion of an incompressible ﬂuid with viscosity ν subject to external forces [34].
The deterministic forcing f , deﬁned as in Li et al.
[25], is a function of space only.
The stochastic forcing ξ is given by ξ = ˙ W for W being a Q-Wiener process which is colored in space and rescaled by σ = 0 .
(see Appendix A.2).
The initial condition is generated according to w 0 ∼ N (0 , 3 3 / 2 ( − ∆+49 I ) − 3 ) with periodic boundary conditions.
The viscosity is set to ν = 10 − 4 .
[Figure]
For each realization of the Q-Wiener process (sampled according to the scheme in Appendix A.2) we solve eq.
(5) with a pseudo-spectral solver described in Appendix A.3, where time is advanced with a Crank–Nicolson update.
We solve the SPDE on a 64 × 64 mesh in space and use a time step of size 10 − 3 .
For the tasks u
(cid:55)→ u and ξ (cid:55)→ u , we generate the datasets by solving the SPDE up to time T = 1 and downsample the trajectories by a factor of 10 in time (resulting in 100 time steps) and 4 in space (resulting in a 16 × 16 spatial resolution).
The number of training samples is N = 1 000 .
To generate the training set for the task ( u 0 , ξ ) (cid:55)→ u , we generate 10 long trajectories of 15 000 steps each up to time T = 15 .
We partition each trajectory into consecutive sub-trajectories of 500 time-steps using a rolling window.
This yields a total of 2 000 input-output pairs.
We split the data into shorter sequences of 500 time steps so that one batch ﬁts in memory of the used GPU.
[Table]
DeepONet
[Table] [Table] [Table] [Table]
[Table]
As shown in Table 3, Neural SPDEs marginally outperforms FNO on the task ξ (cid:55)→ u , but with a signiﬁcantly larger gap on the task ( u 0 , ξ ) (cid:55)→ u from the second best model NCDE-FNO.
Fig.indicates that our model is capable of zero-shot super-resolution in space-time, achieving good performance even when evaluated on a larger time horizon and on an upsampled spatial grid.
Finally, we report in Table 4 some run time statistics indicating that NSPDEs can be up to 3 orders of magnitude faster than traditional numerical solvers.  Conclusion
[Table]
We introduced Neural SPDEs, a model capable of learning solution operators of PDEs with (possibly stochastic) forcing from partially observed data.
Our model provides an extension to two classes of physics-inspired models.
It extends Neural CDEs in that it is resolution-invariant both in space and in time, and it extends Neural Operators as it can be used to learn solution operators of SPDEs depending simultaneously on the initial condition and driving noise.
We performed extensive experiments illustrating how the model achieves superior performance while requiring a lower amount of training data compared to other models, and its evaluation is up to 3 orders of magnitude faster than traditional numerical solvers.
Limitations and future work Similarly to other neural operator models, parameterising the kernel in Fourier space is by no means the only available option; other parameterisations could mitigate some of the disadvantages of the FFT (irregular grids, aliasing effect ...), see for example [23].
A question we leave to future work is how to construct a discrepancy between probability measures supported on spatiotemporal signals, generalizing for example the signature kernel MMD in [33].
Neural SPDEs paired with such discrepancy would allow the design of new generative models for spatiotemporal signals.
Another research direction will be to assess whether Neural SPDEs can be used in computer vision to process videos at arbitrary resolution.
Acknowledgments and Disclosure of Funding
This project was supported by G-Research and by DataSig under the grant EP/S026347/1.
Appendix
This appendix is organized as follows.
In Appendix A we provide a summary of the computational aspects of SPDEs used for data simulation and model deﬁnition, emphasizing the important role of the Fourier Transform (A.1) for simulating noise realizations of Wiener processes (A.2) and building numerical solvers for SPDEs (A.3).
In Appendix B we provide additional considerations about our Neural SPDE model and further experimental details (B.2) and additional experiments on the stochastic Ginzburg-Landau (B.3) and wave (B.4) equations, and on the deterministic Navier-Stokes PDE (B.5).
A Computational aspects of SPDEs
We start this section with the deﬁnition of the Fourier Transform (FT).
We then deﬁne the Discrete
Fourier Transform (DFT) as an approximation to the FT of a function observed at ﬁnitely many locations.
Next, we discuss the role played by the FT to sample realizations of Wiener processes, necessary to build spectral solvers for SPDEs.
The interested reader is referred to Briggs and Henson [4] and Lord et al.
[27] for further details.
A.1 The Fourier Transform
Let V be a vector space over the complex numbers (e.g. C d h or C d h × d h ).
Let r ∈ N and let C ⊂ R r be a compact subset of R r .
In the paper we used either r = d and C
= D or r = d
+ 1 and
C = [0 , T ] × D .
Deﬁnition A.1 ( r -dimensional Fourier Transform ) .
The r -dimensional FT F r : L 2 ( R r , V ) →
L 2 ( R r , V ) and its inverse F − 1 r : L 2 ( R r , V ) →
L 2 ( R r , V ) are deﬁned as follows
[Equation]
for any f, g ∈ L 2 ( R r , V ) , where i = √− 1 is the imaginary unit and (cid:104)· , ·(cid:105) denotes the Euclidean inner product on R r .
In practice, we do not observe a function on R r but on a subset C ⊂ R r .
Furthermore, functions are observed at ﬁnitely many locations in C , and another transform—the discrete Fourier transform (DFT)—is used for numerical computations.
In the sequel we denote by Π N the set of periodic sequences indexed on Z r with period vector ( N 1 , . . .
, N r ) .
Deﬁnition A.2 ( r -dimensional Discrete Fourier Transform ) .
The r -dimensional DFT D r : Π N → Π N and its inverse D − 1 r : Π N → Π N are deﬁned as follows,
[Equation]
with N = diag( N 1 , . . .
, N r ) ∈ N r × r , and R N the rectangular domain R N = { x ∈ R r | 0 ≤ x i <
N i , i = 1 , . .
, r } .
The DFT of a sequence can be computed exactly and efﬁciently using the fast Fourier transform (FFT) algorithm [8] which reduces the complexity from O ( M 2 ) to O ( M log M ) where M = N 1 N 2 . . .
N r .
Most importantly, the FFT algorithm is implemented in machine learning libraries such as PyTorch, which provide support for GPU acceleration and automatic differentiation capabilities.
Note that if we have a ﬁnite sequence, we may still deﬁne its DFT by implicitly extending the sequence periodically.
In particular, when a compactly supported function is sampled on its interval of support, and the samples are used as input for a DFT, it is as if the periodic extension of the function had been sampled.
More precisely, consider an input sequence which corresponds to the evaluation of a function f on a regular grid of C = R N .
For simplicity, suppose that N i = N 1 for all i = 1 , . . .
, r and consider the grid points x n = nL/N 1 for n ∈ Z r ∩ R N .
Taking the DFT of the sequence of general term u n = f ( x n ) we obtain for all n ∈ Z r ,
[Equation]
where y n are the reciprocal frequency points given by y n = n/L for n ∈ Z r ∩ R N .
The DFT of a compactly supported (or approximately compactly supported) function f sampled on the regular grid of points x k approximates the FT of f at the frequency points y n (up to a constant multiplicative factor).
The FT is closely related to the notions of Fourier coefﬁcients and Fourier Series deﬁned hereafter.
Deﬁnition A.3 ( r -dimensional Fourier series ) .
Let f be a piecewise smooth function f : R r → V which is periodic in x i with period L i ∈ R + for all i = 1 , . . .
, r .
The r -dimensional Fourier series of f is a representation of the form,
[Equation]
where L = diag( L 1 , . . .
, L r ) ∈ R r × r and c n ( f ) are complex coefﬁcients, called Fourier coefﬁcients , given by
[Equation]
where R L ⊂ R r denotes the rectangular domain of sides
L 1 , . . .
, L r .
We note that in the deﬁnition above, the sign ∼ means that the series is a formal series and no statement is made about the convergence of the series (the forms of convergence are studied in
Alimov et al.
If f is compactly supported on R L , we may still deﬁne its Fourier coefﬁcients, and in this case F r ( f )( y n ) =
| det L | c n ( f ) at the frequency points y n =
L − 1 n .
Numerical consideration
Consider a function f which has compact support (or is periodic) which is observed at M locations in its support (or its unitary cell R L ).
When using the DFT to approximate M points of the spectrum F r ( f )( y k ) (or M coefﬁcients c k ( f ) ), a so-called aliasing error usually occurs: due to the periodicity of the DFT, the k th coefﬁcient of the DFT includes the contributions not only of the k th frequency mode, but also from higher modes of the underlying function f .
In general the accuracy of the highest frequency modes is more impacted by this error, and aliasing occurs speciﬁcally when we compute nonlinear terms in the physical space.
For example, in the main paper we approximate the evaluation on a discretization spatiotemporal grid D ×T of F − 1 d +1 ( F d +1 ( K ) F d +1 ( f )) by D − 1 d
+
( B D d +1 ( f | D ×T )) where f = 1 ≥
H θ,ξ ( z ) and H θ,ξ is nonlinear.
One possibility to mitigate aliasing is to set to zero the DFT terms (arising in nonlinearities) corresponding to the highest frequency modes before we apply the inverse DFT to go back to the physical space.
This is precisely what we do when we parametrize only k 1 max × . . .
× k d +1 max × d h × d h entries of the complex tensor B , and set the others to zero, hence resolving potential aliasing errors.
We note that speciﬁc rules have been proposed (notably in the literature on pseudo-spectral solvers) to deal with speciﬁc nonlinearities.
However, in the context of Neural SPDE we learn the nonlinearities, hence the number of frequency modes that we retain is treated as an hyperparameter.
A.2 Stochastic simulation of Wiener processes
After deﬁning Wiener processes we outline the sampling procedure that we used to simulate the datasets in the main paper.
For more details on computational aspects of SPDEs the reader is referred to Lord et al.
Throughout this section, H will denote a separable Hilbert space (e.g. H =
L 2 ( D ) ) with a complete orthonormal basis { φ k } k ∈ N .
Let (Ω , F , F t , P ) be a ﬁltered probability space.
A.2.1 Q-Wiener process
Consider an operator Q : H → H such that there exists a bounded sequence of nonnegative real numbers { λ k } k ∈ N
such that Qφ k = λ k φ k for all k ∈ N (this is implied by Q being a trace class, non-negative, symmetric operator, for example).
Deﬁnition A.4 ( Q -Wiener process ) .
Let Q be a trace class non negative, symmetric operator on H .
A H -valued stochastic process { W ( t ) : t ≥ 0 } is called a Q -Wiener process if
W (0) = 0 almost surely; 2.
W ( t ; ω ) is a continuous sample trajectory R
+ (cid:55)→ H , for each ω ∈ Ω ; 3.
W ( t ) is F t -adapted and has independent increments W ( t ) − W ( s ) for s < t ; 4.
W ( t ) − W ( s ) ∼ N (0 , ( t − s ) Q ) for all 0 ≤ s ≤ t .
In analogy to the Karhunen Loéve expansion, it can be shown that W ( t ) is a Q -Wiener process if and only if for all t ≥ 0 ,
[Equation]
where β j ( t ) are i.i.d.
Brownian motions, and the series converges in L 2 (Ω , H ) .
Moreover the series is P -a.s.
uniformly convergent on [0 , T ] for arbitrary T > 0 .
(i.e. converges in L 2 (Ω , C ([0 , T ] , H )) ).
In the Navier-Stokes example, we drive the SPDE by samples ξ from a Q -Wiener process in two dimensions.
Here we follow Lord et al.
[27, Example 10.12] and explain how the sampling procedure works in this case.
Let D =
(0 , L 1 ) × (0 , L 2 ) and consider an L 2 ( D ) -valued Q-Wiener process
W ( t ) .
If the eigenfunctions of Q are given by,
[Equation]
numerical approximation of sample paths from W ( t ) are easy to obtain through a DFT.
Denote by λ k the eigenvalues of Q (e.g. λ k
=
e − α | k | 2 for some parameter α > 0 ) and let J be the index set deﬁned by,
[Equation]
The goal is to sample from the truncated expansion of W ( t ) ,
[Equation]
at the collection of sample points,
[Equation]
Consider the random variable Z ( t n , x ) deﬁned by,
[Equation]
meaning that ξ nj = a + ib with a, b i.i.d ∼ N (0 , 1) such that Z ( t n , x k ) is a complex random variable with independent real and imaginary part with the same distribution as two independent copies of the increment W J ( t n + ∆ t, x k ) − W J ( t n , x k ) .
Furthermore, Z ( t n , x k ) can be expressed in the form,
[Equation]
where (cid:101)
Z j 1 ,j 2 = (cid:112) ∆ tλ j 1 ,j 2 J 1 J 2 ξ nj 1 ,j 2 We recognize that the matrix with entries given by eq.
(7) is the 2D inverse DFT of the J 1 × J 2 matrix with entries (cid:101)
Z j 1 ,j 2 .
Therefore, we can sample two independent copies of
[Equation]
J
[Equation]
−by computing a single 2D inverse DFT.
A.2.2 Cylindrical Wiener process
If the operator Q =
I is the identity, then Q is not of trace class on H so that the series in eq.
(6) does not converge in L 2 (Ω , H ) .
This motivates the deﬁnition of cylindrical Wiener processes.
Deﬁnition A.5 ( Cylindrical Wiener process ) .
Let H be a separable Hilbert space.
A cylindrical
Wiener process (a.k.a space-time white noise) is a H -valued stochastic process { W ( t ) : t ≥ 0 } deﬁned by
[Equation]
(8) where { φ j } is any orthonormal basis of H and β j ( t ) are i.i.d.
Brownian motions.
In all examples except Navier-Stokes, we drive the SPDE by samples ξ from a cylindrical Wiener process in one dimension.
Let D =
(0 , L ) and consider an L 2 ( D )
-valued cylindrical Wiener process
W ( t ) .
As explained in Lord et al.
[27, Example 10.31], if we take the basis
[Equation]
numerical approximation of sample paths from W ( t ) are easy to obtain.
The goal is to sample from the truncated expansion,
[Equation]
at the collection of sample points x k = kL/J for k = 1 , . . .
, J .
Observing that a trigonometric identity yields,
[Equation]
the increments W J ( t
[Equation]
+
[Equation]
, x
[Equation]
W
[Equation]
[Equation]
k [Equation] [Equation] tL/J ) for all k = 1 , . . .
, k .
A.3 Numerical solvers
In this section we present an overview of the numerical solvers for SPDEs we used to generate the data for all the experiments.
The stochastic Ginzburg-Landau (Sec. 4.1 and appendix B.3), stochastic wave (Appendix B.4) equations have been solved using the ﬁnite difference method, while the stochastic
Korteweg–De Vries (Sec. 4.2) and Navier Stokes (Sec. 4.3) equations have been solved using the spectral Galerkin method.
We use the same setup as in Sec. 2.
In particular, we focus on stochastic semilinear evolution equations of the form
[Equation]
where W t is either a Q -Wiener process or a cylindrical Wiener process and L is a linear differential operator generating a semigroup e t L .
We consider nonlinearities F, G regular enough (see Lord et al. [27, Assumption 10.23]) to guarantee existence and uniqueness of mild solutions of eq.
Thm. 10.26].
A.3.1 Finite difference method
We illustrate this numerical method for the reaction-diffusion equation
[Equation]
with homogeneous Dirichlet boundary conditions and where (cid:15), σ > 0 are constants.
We assume for simplicity that u 0 , u t , W t are real-valued and D = (0 , a ) .
The generalization to higher dimensions is straightforward.
Consider the grid points x j = jh , where h = aJ and j = 0 , ..., J , for some spatial resolution J ∈ N .
Let u J ( t ) be the ﬁnite difference approximation of [ u ( t, x 1 ) , ..., u ( t, x J − 1 )] (similarly for W J ( t ) ) resulting from the solution of the following SDE
[Equation]
where ˆ f ( u J ) =
[ f ( u 1 ) , ..., f ( u J − 1 )]
T and M is the ( J − 1) ×
( J − 1) matrix approximating Laplacian (with free boundary conditions) which is given by
[Equation]

One could modify M for speciﬁc boundary conditions.
For instance in the case of periodic boundary one should modify M 1 ,J − 1 =
M J − 1 , 1 = − 1 (see Lord et al.
[27, Chapter 3.4] for Dirichlet and
Neuman boundary condition modiﬁcations of M ).
To discretize in time, we may apply numerical methods for SDEs (see for example Lord et al.
[27, Chapter 8]).
Choosing the standard EulerMarayama scheme with time step ∆ t yields an approximation u J,n to u J ( t n ) at t n = n ∆ t deﬁned by
[Equation]
The increments ( W J ( t n +1 ) − W J ( t n )) are generated using techniques discussed in Appendix A.2.
A.3.2 Spectral Galerkin method
Consider again a separable Hilbert space H .
Assume that the differential operator L in eq.
(10) has a complete set of orthonormal eigenfunctions { φ j } j ∈ N and eigenvalues λ j < 0 , ordered so that λ j +1 <
λ j .
Then, we can deﬁne the semigroup e t L as follows
[Equation]
Deﬁne the Galerkin subspace V J
=
Span { φ 1 , ..., φ J } and the orthonormal projections P J : H → V J as follows
[Equation]
Then, the following deﬁnes spectral Galerkin approximation of eq.
[Equation]
where u J := P J u and L J := P J L and W J =
P J W is as in (9).
Using a Euluer-Marayama discretization as above, we obtain the following discretization
[Equation]
This approach is particularly convenient for problems with additive noise where the eigenfunctions of L and Q (the covariance of the Q -Wiener process W ) are equal, which is the case for all the experiments in this paper generated with this method.
The eigenfunctions of the Laplacian with periodic boundary conditions correspond to the Fourier basis exponentials; therefore, one can deﬁne the projection P J in terms of the DFT.
B Further experiments
In this section with discuss additional details about the NSPDE model, its training procedure an of the baseline models, including how the relevant hyperparameters have been selected for each model.
B.1 Derivation of the ODE parameterisation
If one assumes that L is a polynomial differential operator of degree N of the form [Equation] where C n 1 ,...,n d ∈ C d
h × d h are complex matrices, then the FT of the kernel associated to L satisﬁes
[Equation]
for any y ∈ C d , where e is the matrix exponential and P is the following matrix-valued polynomial
[Equation]
Therefore, there exists a map A : C d → C d h × d h such that F ( K t )( y )
=
e tA ( y ) .
It follows that
[Equation]
where v t : C d → C d h is the solution of the following ODE
[Equation]
as shown in section 3.2.
B.2 Additional experimental details
For all experiments the dataset is split into a training, validation and test sets with relative sizes 70% / 15% / 15% .
For all models, a grid search on the hyperparameters is performed using the training and validation sets.
We use the Adam optimizer and a scheduler which reads the validation loss and reduces the learning rate if no improvement is seen for a patience number of epochs.
Additionally, an early stopping method is used to halt the training of the model if no improvement is seen after a patience number of epochs.
The hyperparameters included in the grid search are stated below and examples of hyperparameter selection results are provided in tables 5 to 9.
NSPDE The hyperparameters included in the grid search are the number of frequency modes used to parametrize the kernel in Fourier space B = F d +1 ( K ) and the number of forward iterations used to solve the ﬁxed point problem.
FNO The hyperparameters included in the grid search are the number of frequency modes used to parametrize the kernel and the number of layers M .
Note that the numbers of frequency modes in the grid search differ from the ones used for the NSPDE model by a factor 2 to ensure that the effective number of retained modes is the same.
For both the NSPDE model and FNO, we kept the number of hidden channels ﬁxed to d h = 32 as this systematically yielded better performances than previously included values and enabled to perform the grid search in a reasonable time.
DeepONet The Deep Operator Network (DeepONet) [28] is another popular class of neural network models for learning operators on function spaces.
The DeepONet architecture is based on the universal approximation theorem of Chen and Chen [6].
It consists of two sub-networks referred to as the branch and the trunk networks.
The trunk acts on the coordinates ( t, x ) ∈ [0 , T ] × D , while the branch acts on the evaluation of the initial condition u 0 on a discretized grid D .
Therefore, the
DeepONet is not a space resolution-invariant architecture.
The output of the network is expressed as
[Equation]
where the b k and the τ k are the outputs of the branch and trunk network respectively.
The trunk network is usually a feedforward neural network, and one can chose the architecture of the branch network depending on the structure of the input domain.
We follow Lu et al.
[28] and use feedforward neural networks for both the trunk and the branch networks.
We perform a grid search on the depth and width of the trunk and branch feedforward neural networks.
NRDE/NCDE/NCDE-FNO
The hyperparameters included in the grid search are the number of hidden channels and the type of solver as implemented by torchdiffeq [5].
We note that we used a depth-2 NRDE model (depth-2 already results in d ξ = 8 385 for forcings observed at 128 spatial points and higher depths models could not ﬁt in memory) and recall that NCDE is a depth-1 NRDE. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
B.3 Stochastic Ginzburg-Landau equation
Recall that the stochastic Ginzburg-Landau equations are of the form,
[Equation]
subject to either Periodic or Dirichlet boundary conditions.
Periodic boundary conditions are given by u ( t, 0) =
u ( t, 1) for all t ≥ 0 and Dirichlet boundary conditions are given by u ( t, 0) =
u ( t, 1) =for all t ≥ 0 .
Initial condition we take as in Sec.
4.1 u 0 ( x ) = x (1 − x ) + κη ( x ) with κ = 0 or κ = 0 .
1 depending on a task.
In both Periodic and Dirichlet case we can take η ( x ) as in Sec. 4.
though in Dirichlet case one must take a 0 = 0 to ensure u 0 being zero at the boundary.
We ﬁrst reproduce an experiment from Sec.
4.1 on the additive stochastic Ginzburg-Landau equation but with Dirichlet boundary conditions instead of the periodic.
We compare it to the benchmark of
FNO model which was the most successful among all the benchmarks of Sec. 4.
From Table 10 we see that even though Neural SPDE model depends on the spectral methods the errors did not increase compared to the periodic equation in Sec.
4.1 (see Table 1).
Our algorithm still outperforms FNO whose relative L 2 error increased slightly.
The fact that Neural SPDE can be applied to non-periodic equations could be perhaps explained by interpolation ( L θ ) and projection ( Π θ ) neural networks that could correct for non-periodicity of the data.
[Table]
We now take a look at the speciﬁc hyperparameter: number of forward iterations in the ﬁxed point solver.
We also call this a number of Picard iterations P .
Theoretically as P increases Fixed
Point Solver should converge to the true solution (see [11]).
This suggests that higher P should improve the performance of the Neural SPDE algorithms.
In practise we observed in both additive
Ginsburg Landau equation from Sec.
4.1 and in KdV equation from Sec.
4.2 that P = 1 could already be enough.
This could be explained either by dominance of the linear part of the equation or by overﬁtting in these cases.
Thus we present an experiment on the multiplicative stochastic
Ginzbug-Landau equation over a longer (compared to Sec. 4.1) time interval.
In the Table 11 we compare NSPDE with P ∈ { 1 , 2 , 3 , 4 } and again include FNO benchmark (which performed best in the previous experiments).
We see that NSPDE with even P = 1 outperforms FNO.
Relative
L 2 error for T = 0 .
05 increases for both NSPDE and FNO due to more complicated multiplicative noise.
In Table 11 we present for each P the best result over other hyperparameters obtained by cross validation.
One could clearly see an improvement in error as we increase the number of Picard iterations P (with an exception of the case T = 0 .
05 where P = 3 outperformed P = 4 ).
This improvement becomes more apparent as the time frame T increases.
Heuristically (and qualitatively) this is due to the fact that for the short times solution of the SPDE is relatively close to its linearised version and that nonlinearity of the equation starts to play a bigger role for larger T .
[Table]
B.4 The stochastic wave equation
In this section we consider the following nonlinear wave equation with multiplicative stochastic forcing,
[Equation]
[Equation]
The nonlinear stochastic wave equation arises in relativistic quantum mechanics and is also used in simulations of nonlinear waves that are subject to either noisy observations or random forcing.
We refer a reader to Temam [34] for an overview on the nonlinear wave equation.
The above equation can put in a form of eq.
(2) by rewriting it as a system for ( u, v ) =
( u, ∂ t u ) .
To generate training datasets, we solve the SPDE using a ﬁnite difference method with 128 evenly distanced points in space and a time step size ∆ t = 10 − 3 .
As in Chevyrev et al.
[7, eq.
(3.5)], we solve the SPDE until
T = 0 .
We then downsample the temporal resolution by a factor 5 , resulting in 100 time points.
Here, the initial condition is given by u 0 ( x ) = sin(2 πx ) + κη ( x ) , where η is deﬁned in Sec.
4.1 and for simplicity initial velocity v 0 is taken deterministic v 0 ( x ) =
x (1 − x ) .
Similarly to Sec.
4.1 we either take κ = 0 or κ = 1 to generate datasets where the initial condition is either ﬁxed or varies across samples.
Each dataset consists of N = 1 000 training observations.
[Table]
B.5 Deterministic Navier-Stokes PDE
In this ﬁnal experiment, we demonstrate that our Neural SPDE model can also be used in the setting of PDEs without any stochastic term.
We do so by studying the example from [25] on deterministic Navier-Stokes.
More precisely, we consider the 2D Navier-Stokes equation for a viscous, incompressible ﬂuid in vorticity form:
[Equation]
where u : [0 , T ] × [0 , 1] 2 → R 2 is the velocity ﬁeld, w = ∇× u is the vorticity with w 0 : [0 , 1] 2 → R being the initial vorticity.
Here f is a deterministic forcing term which we take as in [25].
We follow the experimental setup from [25] and use the dataset (available under an MIT license) where ν = 10 − 5 , N = 1000 and T = 20 .
We achieve similar performances as FNO with a L2 error of 0 . 17 .
A comparison between a true and predicted trajectory is depicted in Fig.