Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamics s
AfﬁliationAddress
Abstract
Stochastic partial differential equations (SPDEs) are the mathematical tool ofchoice for modelling spatiotemporal PDE-dynamics under the inﬂuence of randomness.
Based on the notion of mild solution of an SPDE, we introduce a novelneural architecture to learn solution operators of PDEs with (possibly stochastic)forcing from partially observed data.
The proposed Neural SPDE model providesan extension to two popular classes of physics-inspired architectures.
On the onehand, it extends Neural CDEs and variants – continuous-time analogues of RNNs –in that it is capable of processing incoming sequential information arriving irregularly in time and observed at arbitrary spatial resolutions.
On the other hand, itextends Neural Operators – generalizations of neural networks to model mappingsbetween spaces of functions – in that it can parameterize solution operators of SPDEs depending simultaneously on the initial condition and a realization of thedriving noise.
By performing operations in the spectral domain, we show howa Neural SPDE can be evaluated in two ways, either by calling an ODE solver(emulating a spectral Galerkin scheme), or by solving a ﬁxed point problem.
Experiments on various semilinear SPDEs, including the stochastic Navier-Stokesequations, demonstrate how the Neural SPDE model is capable of learning complexspatiotemporal dynamics in a resolution-invariant way, with better accuracy andlighter training data requirements compared to alternative models, and up to 3orders of magnitude faster than traditional solvers.
Introduction
Stochastic partial differential equations (SPDEs) are the mathematical formalism used to model manyphysical, biological and economic systems subject to the inﬂuence of randomness, be it intrinsic(e.g.
quantifying uncertainty) or extrinsic (e g. modelling environmental random perturbations).
Notable examples of SPDEs include the Kardar–Parisi–Zhang (KPZ) equation for modelling randominterface growth such as the propagation of a forest ﬁre from a burnt region to an unburnt region [12],the Ginzburg-Landau equation describing phase transitions of ferromagnets and superconductorsnear critical temperature [34], or the stochastic Navier-Stokes equations modelling the dynamics of aturbulent ﬂuid ﬂow under the presence of local random ﬂuctuations [31].
For an introduction to thetheory of SPDEs see Hairer [11]; a comprehensive textbook is Holden et al.
Classical numerical approaches for solving SPDEs include ﬁnite difference methods and spectral
Galerkin methods [27] among others.
To ensure accuracy and stability of numerical solutions ofcomplex SPDEs, computations must be carried out at high resolution using ﬁne discretization grids,rendering the resulting schemes computationally intractable.
This limitation motivates the study ofdata-driven methods that can learn solutions to differential equations from partially observed data.
In fact, there has been an increased interest in recent years to combine neural networks and differentialequations into a hybrid approach, dubbed neural differential equations [5, 17, 37].
Neural controlled differential equations (Neural CDEs), as popularised by [3, 18, 32], are continuoustime analogues to recurrent neural networks (RNN, GRU, LSTM etc.).
The input to a Neural CDEmodel is a multivariate time series interpolated into a continuous path X : [0 , T ] → R d ξ ; the modelconsists of a matrix-valued feedforward neural network f θ : R d h → R d h × d ξ parameterizing thevector ﬁeld of the following dynamical system (and satisfying some minimal Lipschitz regularity toensure existence and uniqueness of solutions)
[Equation]
where (cid:96) θ : R d u → R d h and π θ : R d h → R d u are feedforward neural networks.
The output responseu : [0 , T ] → R d u is then fed to a (possibly pathwise) loss function (mean squared, cross entropy etc.)and trained via stochastic gradient descent in the usual way.
In practice, the term “ dX t ” means thatthe solution z t of the equation (e.g. attention required from a doctor) can change in response to achange of an external stream of information X t (e.g. heart rate of a patient).
Depending on the level of roughness of the control path X , the integral in eq.
(1) can be interpretedin different ways.
In [18], X is assumed differentiable and is obtained in practice via cubic splinesinterpolation of the original time series.
In this way, the term “ dX s ” can be interpreted as “ ˙ X s ds ”so that eq.
(1) becomes an ODE of the form ˙
z t = f θ ( z t ) ˙
X t that can be evaluated numerically viaa call to an ODE solver of choice (Euler, Runge-Kutta, implicit, adaptive stepsize schemes etc.).
More generally, if X is of bounded variation then the integral above can be seen as a classical
Riemann–Stieltjes or Young integral [38].
Neural SDEs [19, 20, 22, 26] are a special subclass of
Neural CDEs where the control is a sample path from a d ξ -dimensional Brownian motion (which isnot of bounded variation), and eq.
(1) is understood via stochastic integration (Itô, Stratonovich etc.).
Neural RDEs [32] allow to relax even further the regularity assumptions on X by treating the integralusing rough integration [10, 30].
In practice, Neural RDEs are particularly well suited for long timeseries.
This is due to the fact that the model can be evaluated via a numerical scheme from stochasticanalysis (called the log-ODE method [32]) over intervals much larger than what would be expectedgiven the sampling rate of the time series.
However, the space complexity of the numerical solverincreases exponentially in the number of channels d ξ , thus model complexity becomes intractable forhigh dimensional time series.
Despite offering many advantages for modelling temporal dynamics, these models are not designedto process signals varying both in space and in time such as physical ﬁelds described by SPDEs.
Inparticular, although these models are time-resolution invariant, they are not space-resolution invariant,and are not well suited to capture nonlinear interactions between the various space-time pointstypically observed in SPDE-dynamics.
Similar to CDEs, the solution u to an SPDE is characterizedby an initial condition u 0 and a driving noise X .
However, in the case of CDEs, ( u 0 , X t , u t ) arevectors, while in the case of SPDEs they are functions.
Neural Operators [21, 23, 24, 28] are generalizations of neural networks capable of modellingmappings between spaces of functions and offer an attractive option for learning with spatiotemporaldata [21].
Among all kinds of Neural Operators, Fourier Neural Operators (FNOs)
[25] stand outbecause of their easier parametrization while demonstrating similar learning performance comparedto other Neural Operator models.
However, Neural Operators generally fail to incorporate the effectthat an external (possibly random) spatiotemporal signal might have on the system they describe.
Inthe case of SPDEs, the external signal is indeed random (e.g. sample from a Wiener process) and itspresence leads to new phenomena, both at the mathematical and the physical level, often describingmore complex and realistic dynamics than the ones arising from deterministic PDEs.
Contributions To overcome the above limitations faced by Neural CDEs and Neural Operators,we introduce the neural stochastic partial differential equation (Neural SPDE) model, capable oflearning solution operators of SPDEs from partially observed data by processing, continuously intime and space, incoming sequential information arriving at an arbitrary resolution.
We proposetwo separate algorithms to evaluate our model: the ﬁrst reduces the Neural SPDE to a system of
ODEs in Fourier space, which can then be solved numerically by means of any ODE solver ofchoice (emulating a spectral Galerkin scheme); the second rewrites the Neural SPDE as a ﬁxed point problem, which is solved via classical root-ﬁnding schemes.
For both choices of evaluation, the
Neural SPDE model inherits memory-efﬁcient backpropagation capabilities provided by existingadjoint-based and implicit-differentiation-based methods respectively.
Finally, we perform extensiveexperiments on various semilinear SPDEs, including the stochastic Ginzburg-Landau, Korteweg-De
Vries, Navier-Stokes equations.
The empirical results illustrate several useful aspects of our model:1) it is space and time resolution-invariant, meaning that even if trained on a lower resolution it canbe directly evaluated on a higher resolution; 2) it requires a lower amount of training data to achievesimilar or better performance compared to alternative models; 3) its evaluation is up to 3 orders ofmagnitude faster than traditional numerical solvers.
The outline of the paper is as follows: in Sec.
2 we provide a brief introduction to SPDEs which willhelp us to deﬁne our Neural SPDE model in Sec. 3, followed by numerical experiments in Sec. 4.
In Appendix A, we provide an overview of the computational aspects of SPDEs used to design themodel and solve SPDEs numerically.
Additional experiments can be found in Appendix B.
Background on SPDEs
Let T > 0 and d, d u , d ξ ∈ N .
Let D ⊂ R d be a bounded domain.
Let H u
= { f : D → R d u } and
H ξ = { f : D → R d ξ } be two Hilbert spaces of functions from D to R d u and R d ξ respectively.
Weconsider a large class of SPDEs of the following type
[Equation]
where W t is either an inﬁnite dimensional Q -Wiener process [27, Def. 10.6] or a cylindrical Wienerprocess [11, Def. 3.54] with values in H ξ , F : H u
→ H u and G : H u
→ L ( H ξ , H u ) are twocontinuous operators, L ( H ξ , H u ) is the space of bounded linear operators from H ξ to H u , and L is alinear differential operator generating a semigroup e t L : H u
→ H u .
For further details on Wienerprocesses see Appendix A.2, and for a primer on semigroup theory see Hairer [11, Section 4].
Afunction u : [0 , T ] → H u is said to be a mild solution of the SPDE (2) if for any t ∈ [0 , T ] it satisﬁes
[Equation]
where the second integral is a stochastic integral interpreted in the Itô sense [11, Def. 3.57].
Thus, an
SPDE can be informally thought of as an SDE with values in the functional space H u and driven byan inﬁnite dimensional Brownian motion W .
Assuming global Lipschitz regularity on F and G , amild solution u to (2) exists and is unique [11, Thm. 6.4], at least for short times.
We follow Friz and Hairer [9] and consider a regularization W (cid:15) =
δ (cid:15) ∗ W of the driving noise W witha compactly supported smooth molliﬁer δ (cid:15) .
It is a classical result (Wong-Zakai [35]) from rough paththeory [10, 30] that, for the case of SDEs, the sequence of random ODEs driven by the molliﬁcationof Brownian motion converges in probability to a limiting process that does not depend on the choiceof molliﬁer and agrees with the Stratonovich solution of the SDE.
Furthermore, the solution map( u 0 , W ) (cid:55)→ u is continuous in an appropriate rough path topology.
This result nicely extends tothe setting of SPDEs driven by a ﬁnite dimensional noise [9, Thm. 1.3]: if u (cid:15) denotes the random
PDE solutions driven by ˙ W (cid:15) dt (instead of ◦ dW t ), then u (cid:15) converges in probability to a limitingprocess corresponding to the Stratonovich solution of the SPDE.
In our setting though, the drivingnoise is inﬁnite dimensional and the resulting integral cannot be interpreted in the Stratonovich sensebecuase otherwise the corresponding Itô-Stratonovich correction would be inﬁnite.
Nonetheless, Hairer and Pardoux [14, Thm. 1.1] show that, in the case of the heat operator and under appropriaterenormalization and drift correction, the random PDE solution u (cid:15) converges in probability to the Itôsolution of the SPDE, and that the solution map is continuous in an appropriate regularity structurestopology. We note that extending this result to a generic differential operators would require asimilarly rigorous proof, which goes beyond the scope of this article and that we leave as future work. As done in Kidger et al.
[18] for Neural CDEs, we can rewrite the mild solution of the molliﬁedversion of eq.
(2) as the following stochastically forced PDE
[Equation]
where ξ = ˙ W (cid:15) ,
H u
= L 2 ( D , R d u ) and H ξ =
L 2 ( D , R d ξ ) .
We will refer to ξ as white noise if W isa cylindrical Wiener process and as coloured noise if W is a Q -Wiener process.
In view of machine learning applications, one should think of W as a continuous space-time embedding of an underlying spatiotemporal data stream.
In this paper we are only going to consider W tobe a sample path from a Wiener process, but we emphasise that the Neural SPDE model extends, inprinciple, beyond the scope of SPDEs and could be used for example to process videos in computervision applications, which we leave as future work.
Next we introduce the Neural SPDE model.
[-+ ℱ −1 d +
-]
Neural SPDEs
For a large class of differential operators L , the action of the semigroup e t L can be written as anintegral against a kernel function K t : D × D
→ R d u × d u such that
[Equation]
for any h ∈ H u , any x ∈ D and t ∈ [0 , T ] , and where µ t is a Borel measure on D .
As in Kovachkiet al.
[21], here we take µ t to be the Lebesgue measure on R d but other choices can be made, forexample to incorporate prior information.
We assume that K is stationary so that eq.
(4) can berewritten in terms of the spatial convolution ∗
[Equation]
For a large class of SPDEs of the form (2), both F and G are local operators acting on a functionh ∈ H u .
In other words, the evaluations F ( h )( x ) and G ( h )( x ) at any point x ∈ D only depend h ( x ) ,and not on the evaluation h ( y ) at some other point y ∈ D in the neighbourhood of x .
3.1 The model
Let H h
=
L 2 ( D , R d h ) for some latent space dimension
d h > d u .
Let
[Equation] h
[Equation]
Π
[Equation]
: R d
[Equation]
→ R d
[Equation]
be four feedforward neural networks.
For any differentiable control ξ : [0 , T ] → H ξ , deﬁne the map
H θ,ξ : H u
→ H u so that for any h ∈ H u , x ∈ D , t ∈ [0 , T ] [Equation] θ,ξ
[Equation]
A Neural SPDE is deﬁned as follows
[Equation]
We note that globally Lipschitz conditions can be imposed by using ReLU or tanh activation functionsin the neural networks F θ and G θ .
In secs. 3.2 and 3.3 we propose two distinct algorithms to evaluatethe Neural SPDE model (7) which are based on two different parameterization of the kernel K .
3.2 Evaluating the model by solving a system of ODEs
Levereraging the convolution theorem , we can rewrite the integral in eq.
(7) as follows
[Equation]
(8) where F , F − 1 are the d -dimensional Fourier transform (FT) and its inverse (see Def. A.1).
If onefurther assumes that L is a polynomial differential operator of degree N of the form
[Equation] where C n 1 ,...,n d ∈ C d h × d h are complex matrices, then the FT of the kernel associated to L satisﬁes [Equation]
for any y ∈ C d , where e is the matrix exponential and P is the following matrix-valued polynomial [Equation] This means that there exists a map A : C d → C d h × d h such that F ( K t )( y ) = e tA ( y ) .
It follows that
[Equation]
where v t : C d → C d h is the solution of the following ODE
[Equation]
Hence, z t can be obtained by applying the inverse FT to the output of an ODE solver on [0 , t ] withinitial condition F ( z 0 ) , vector ﬁeld Ψ θ, [Equation] [Equation]
[Equation]
This approach can naturally be seen as a “neural version” of the classical spectral Galerkin methodfor SPDEs as described in Appendix A.3.
We note that this numerical evaluation of the Neural SPDE model (7) allows to inherit memoryefﬁcient adjoint-based backpropagation capabilities as in the evaluation of a Neural CDE.
For furtherdetails on adjoint-based backpropagation we refer the reader to Chen et al.
[5], Kidger [17].
3.3 Evaluating the model by solving a ﬁxed point problem
In our second approach of model evaluation, we make use of three different versions of the FT: thetime-only FT F 1 and its inverse F − 1 1 , the space-only FT F d and its inverse F − 1 d , and the space-time
FT F d +1 and its inverse F − 1 d +1 (see Def.
A.1 for details).
Denoting by (cid:63) the space-time convolution,the integral in eq.
(7) can be rewritten as
[Equation]
where 1 ≥ 0 is the indicator function restricting the temporal domain to the positive real line.
Usingagain the convolution theorem we obtain
[Equation]
where all multiplications are matrix-vector multiplications.
Using the trick introduced in [25], one canparameterize F d +1 ( K )( y ) directly in Fourier space as a complex tensor B , so that eq.
(7) becomes
[Equation]
where we used the fact that F d ( K t ) =
F − 1 1 ( F d +1 ( K )) t .
Hence, the solution z can be obtained bysolving the ﬁxed point problem z
= Φ θ,ξ ( z ) , where
[Equation]
This can be solved numerically using classical root-ﬁnding schemes (e.g. by Picard’s iteration)z
≈ FixedPointSolve ( z 0 , Φ θ,ξ ) .
Analogously to adjoint-based backpropagation for the evaluation approach mentioned in Sec.
3.2,there is a mechanism that leverages the implicit function theorem allowing to backpropagate throughthe operations of a ﬁxed point solver in a memory-efﬁcient way.
See Bai et al.
[2] for further details.
We note that the FTs are numerically approximated using the discrete Fourier transform (DFT) andselecting a maximum number of frequency modes 1 (see Appendix A.1 for details). [Figure] [Figure]
[Figure]
As depicted in Fig.
1, the input to a Neural
[Figure] SPDE corresponds to a (possibly irregu- [Figure] larly sampled) time-indexed sequence of(possibly partially-observed) spatial observations recorded on a space-time grid.
Thedata is then interpolated into a continuousspatiotemporal signal ξ and initial condition u 0 .
By construction, a Neural SPDEoperates in continuous time and space onthe tuple of functions ( u 0 , ξ ) and producesa spatiotemporal response u , which is alsocontinuous in space and time; the function u can then be evaluated at an arbitrary space-time resolution, possibly different from the one used during training.
In the next section we will demonstrate empirically that even if trained on a coarserresolution, a Neural SPDE can be evaluatedon a ﬁner resolution without sacriﬁcing performance, a property known as zero-shot super-resolution .
Experiments
In this section, we run experiments on three semilinear SPDEs: the stochastic Ginzburg-Landauequation in 4.1, the stochastic Korteweg-De Vries equation in 4.2, and the stochastic Navier-Stokesequations in 4.3.
We note that although the assumption of globally Lipschitz vector ﬁelds might beviolated for the following SPDEs, well-posedness (i.e. existence of global solutions) can be shownusing equation-speciﬁc arguments.
We consider three supervised operator-learning settings:
• u 0 (cid:55)→ u , assuming the noise ξ is not observed;
• ξ (cid:55)→ u , assuming the noise ξ is observed, but the initial condition u 0 is ﬁxed across samples; • ( u 0 , ξ ) (cid:55)→ u , assuming the noise ξ is observed and u 0 changes across samples.
We note that learning the operator u 0 (cid:55)→ u of an SPDE without observing the driving noise ξunavoidably yields poor results for all considered models as only partial information about thesystem is provided as input.
However, we ﬁnd it informative to include the performances obtainedin this setting, as this provides a sanity check that emphasizes the importance of the noise in all theexperiments we consider in this paper.
Moreover, the ability to process the initial condition u 0 on itsown (in absence of noise) testiﬁes that Neural SPDEs can also be used to learn deterministic PDEs.
We provide an example on the deterministic Navier-Stokes equations in Appendix B.4.
Neural CDE, Neural RDE, FNO and DeepONet [28, 29] will be the main benchmark models.
In addition, we also propose an additional baseline Neural CDE-FNO, which is a hybrid modelconsisting of a Neural CDE where the drift is modelled by an FNO and the diffusion by a feedforwardneural network.
The motivation for using a FNO to represent the drift comes from the universalapproximation properties of FNOs studied in Kovachki et al.
[21, Thm. 4].
An interesting line of work to tackle SPDE-learning is provided in [7, 16].
The authors construct aset of features from the pair ( u 0 , ξ ) following the deﬁnition of a model from the theory of regularitystructures [13].
They then perform linear [7] and nonlinear [16] regression from these features tothe solution of the SPDE at a single time point.
Therefore, these models would have to be retrainedfor any new prediction.
In addition, both [7, 16] assume knowledge of the differential operator Lgoverning the dynamics, while Neural SPDE learns a representation of L via the parametrization ofthe associated kernel.
For these reasons these recent models are not included in our benchmark.
For all the experiments, the loss function is the relative pathwise L 2 error.
The hyper-parametersfor all the models are selected by grid-search (see Appendix B.1 for further experimental details).
Experiments are run on a Tesla P100 NVIDIA GPU.
The code for the experiments is provided in thesupplementary material.
Additional experiments may be found in Appendix B.
4.1 Stochastic Ginzburg-Landau equation
We start with the stochastic Ginzburg-Landau equation, a reaction diffusion equation in 1D given by
[Equation]
This equation is also known as the Allen-Cahn equation in 1 -dimension and is used for modelingvarious physical phenomena like superconductivity [34].
Here ξ denotes space-time white noise withsample paths generated using classical sampling schemes for Wiener processes detailed in A.2.
[Table]
x 0. x NSPDE (Ours) 0. We consider two data-regimes: a low data regime where the total number of training observationsis N = 1 000 , and a large data regime where N = 10 000 .
In both cases, the response paths aregenerated by solving the SPDE along each sample path of the noise ξ using a ﬁnite differencescheme described in Appendix A.3 using 128 evenly distanced points in space and time and step size∆ t = 10 − 3 .
Following the same setup as in Chevyrev et al.
[7, eq.
(3.6)], we solve the SPDE until
T = 0 .
05 resulting in 50 time points and choose as initial condition u 0 ( x )
= x (1 − x ) + κη ( x ) , [Equation]
with a k ∼ N (0 , 1) [Equation] [Equation]
We either take κ = 0 or κ = 0 .
1 to generate a dataset where the initial data is either ﬁxed orvaries across samples.
We provide extra experiments on this SPDE for larger time horizons T andmultiplicative forcing in Appendix B.2.
We report the results in Table 1.
The Neural SPDE model (NSPDE) yields the lowest relative errorfor all tasks, reaching one order of magnitude improvement on the main task ( u 0 , ξ ) (cid:55)→ u in thelarge data regime compared to all the applicable benchmark models (NCDE, NRDE, NCDE-FNO).
In all settings, even with a limited amount of training samples ( N = 1 000 ), the NSPDE model isapplicable and achieves ∼ 1% error rate, and marginally improves to < 1% error when N = 10 000 .
4.2 Stochastic Korteweg–De Vries equation
Next, we consider the stochastic Korteweg–De Vries (KdV) equation, a higher order SPDE given by
[Equation]
This equation is used to describe the propagation of nonlinear waves at the surface of a ﬂuid subjectto random perturbations (another wave equation is studied in Appendix B.3).
We refer the readerto Wazwaz [36] for an overview on the KdV equation and its relations to solitary waves.
The stochastic forcing is given by ξ = ˙ W for W being a partial sum approximation of a Q-Wienerprocess as per Example 10.8 in Lord et al.
[27] with λ j ∼ j − 5+ ε and φ j ( x ) = sin ( jπx ) (seeeq.
(11) in Appendix A.2).
Taking small ε > 0 guarantees that W t is twice differentiable in spacefor every t ≥ 0 .
To generate the datasets, we solve the SPDE with γ = 0 .
1 until T = 0 .
[Table]
The stochastic forcing is simulated using128 evenly distanced points in space and atime step ∆ t ref = 10 − 3 .
We then approximate realizations of the solution of the KdVequation using a time step ∆ t = 10 − 2 until T = 0 .
Here, the initial conditionis given by u 0 ( x ) = sin(2 πx ) + κη ( x ) ,where η is deﬁned in eq.
Similarly to Sec.
4.1 we either take κ = 0 or κ = 1 togenerate datasets where the initial condition is either ﬁxed or varies across samples.
Each dataset consists of N = 1 000 training observations.
Stochastic KdV . We report the relative L2 error on the test set. The symbol x indicates that the model is not applicable. N = 1 000 and T = 1 .
Model u 0 (cid:55)→ u [Equation] (cid:55)→ u [Equation]
u [Equation] (cid:55)→ u FNO 0.
x
NSPDE (Ours) 0. As reported in Table 2, Neural SPDEs outperforms the second best model FNO by afull order of magnitude in the task ξ (cid:55)→ uand the second best model NCDE-FNO byalmost two orders of magnitude in the task( u 0 , ξ ) (cid:55)→ u .
We also perform the sametasks for a larger time horizons T = 1 andreport the results of a comparison against FNO in Sec.
4.3 Stochastic Navier-Stokes equations
Finally, we consider the vorticity form of the Navier-Stokes equations in 2 -dimensions for anincompressible ﬂow
[Equation] where u is the unique divergence free ( ∇ · u = 0 ) velocity ﬁeld such that w = ∇ × u .
Theseequations describe the motion of an incompressible ﬂuid with viscosity ν subject to external forces[34].
The deterministic forcing f , deﬁned as in Li et al.
[25], is a function of space only.
Thestochastic forcing ξ is given by ξ = ˙ W for W being a Q-Wiener process which is colored inspace and rescaled by σ = 0 .
(see Appendix A.2).
The initial condition is generated according tow 0 ∼ N (0 , 3 3 / 2 ( − ∆+49 I ) − 3 ) with periodic boundary conditions.
The viscosity is set to ν = 10 − 4 . [Figure] s
[Figure] l u t i o n p r e d i c t i o n [Figure] For each realization of the Q-Wiener process (sampled according to the scheme in Appendix A.2) wesolve eq.
(10) with a pseudo-spectral solver described in Appendix A.3, where time is advanced witha Crank–Nicolson update.
We solve the SPDE on a 64 × 64 mesh in space and use a time step of size10 − 3 .
For the tasks u
(cid:55)→ u and ξ (cid:55)→ u , we generate the datasets by solving the SPDE up to time
[-G r o und tr u t h t=  50  t=  50  t=  50  t=  50  t=  
 Sup e r - r e s o l u t i o n p r e d i c t i o n 50   50   50   50-]T = 1 and downsample the trajectories by a factor of 10 in time (resulting in 100 time steps) and 4 inspace (resulting in a 16 × 16 spatial resolution).
The number of training samples is N = 1 000 .
Togenerate the training set for the task ( u 0 , ξ ) (cid:55)→ u , we generate 10 long trajectories of 15 000 stepseach up to time T = 15 .
We partition each of these 10 trajectories into consecutive sub-trajectoriesof 500 time-steps using a rolling window.
This procedure yields a total of 2 000 input-output pairs.
We chose to split the data into shorter sequences of 500 time steps so that one batch training could ﬁtin memory on a Tesla P100 NVIDIA GPU.
[Table]
We report the results in Table 3.
Neural
SPDEs marginally outperforms FNO inthe simpler task ξ (cid:55)→ u , but with a signiﬁcantly larger performance gap in theharder task ( u 0 , ξ ) (cid:55)→ u from the secondbest model NCDE-FNO.
Fig.
2 indicatesthat the Neural SPDE model is capableof zero-shot super-resolution in space andin time, achieving good performance evenwhen evaluated on a larger time horizonof 5 000 time steps and on an upsampled64 × 64 spatial grid.
Finally, we report in Table 4 the ratios of inference time of thetrained Neural SPDEs over the runtime ofthe solvers, indicating that NSPDEs can beup to 3 orders of magnitude faster than traditional numerical solvers.
[Table]
Conclusion and future work
In this paper we introduced the Neural SPDE model capable of learning solution operators of PDEswith (possibly stochastic) forcing from partially observed data.
Neural SPDEs provide an extensionto two popular classes of physics-inspired models.
They extend Neural CDEs and variants in thatthey are resolution-invariant both in space and in time, and they extend Neural Operators as theycan be used to learn solution operators of SPDEs depending simultaneously on the initial conditionand driving noise.
We performed extensive experiments illustrating how the model achieves superiorperformance while requiring a lower amount of training data compared to alternative models, and itsevaluation is up to 3 orders of magnitude faster than traditional numerical solvers.
Based on our promising empirical results, we think the following are interesting questions for futureresearch: (i) Does the random PDE model in eq. (4) converge to a limiting Itô SPDE? Is the solutionmap continuous in an a regularity structure topology? Is the limiting process independent on thechoice of molliﬁer?
(ii) Is it possible to construct a discrepancy between probability measuressupported on spatiotemporal signals, generalizing for example the signature kernel MMD in [33]?
Neural SPDEs paired with such discrepancy would allow the design of new generative models forspatiotemporal signals generalizing the results in Kidger et al.
[20]; (iii) Can Neural SPDEs be usedin broader data science contexts, such as computer vision to process videos?
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See the paragraph entitled "Contributions" in Sec.
for a clear statement of the contributions, and Sec. 2 for further details on the paper’sscope.(b)
Did you describe the limitations of your work?
[Yes] See Sec. 2 and Sec. 5.(c)
Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 2.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B.1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
[No] Error bars are not reported because it would be too
computationally expensive for the baseline models NCDE, NRDE and NCDE-FNO.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] The type of GPU is speciﬁed inSec. 4.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Citations can be
found in the descriptions of the experiments in Sec. 4.
(b) Did you mention the license of the assets?
[Yes] See the code provided as supplementary
material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The code, data and models are included in the supplemental material to reproduce the
experiments.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The assets are licensed under the MIT License.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix
This appendix is organized as follows.
In Appendix A we provide a summary of the computationalaspects of SPDEs used for data simulation and model deﬁnition, emphasizing the important role ofthe Fourier Transform (A.1) for simulating noise realizations of Wiener processes (A.2) and buildingnumerical solvers for SPDEs (A.3).
In Appendix B we provide further experimental details (B.1) andadditional experiments on the stochastic Ginzburg-Landau (B.2) and wave (B.3) equations, and onthe deterministic Navier-Stokes PDE (B.4).
A Computational aspects of SPDEs
We start this section with the deﬁnition of the Fourier Transform (FT).
We then deﬁne the Discrete
Fourier Transform (DFT) as an approximation to the FT of a function observed at ﬁnitely manylocations.
Next, we discuss the role played by the FT to sample realizations of Wiener processes,necessary to build spectral solvers for SPDEs.
The interested reader is referred to Briggs and Henson[4] and Lord et al.
[27] for further details.
A.1 The Fourier Transform
Let V be a vector space over the complex numbers (e.g. C d h or C d h × d h ).
Let r ∈ N and let C ⊂ R rbe a compact subset of R r .
In the paper we used either r = d and C
= D or r = d
+ 1 and
C = [0 , T ] × D .
Deﬁnition A.1 ( r -dimensional Fourier Transform ) .
The r -dimensional FT F r : L 2 ( R r , V ) →
L 2 ( R r , V ) and its inverse F − 1 r : L 2 ( R r , V ) →
L 2 ( R r , V ) are deﬁned as follows
[Equation]
for any f, g ∈ L 2 ( R r , V ) , where i = √− 1 is the imaginary unit and (cid:104)· , ·(cid:105) denotes the Euclideaninner product on R r .
In practice, we do not observe a function on R r but on a subset C ⊂ R r .
Furthermore, functionsare observed at ﬁnitely many locations in C , and another transform—the discrete Fourier transform(DFT)—is used for numerical computations.
In the sequel we denote by Π N the set of periodic sequences indexed on Z r with period vector( N 1 , . . .
, N r ) .
Deﬁnition A.2 ( r -dimensional Discrete Fourier Transform ) .
The r -dimensional DFT D r : Π N →Π N and its inverse D − 1 r : Π N → Π N are deﬁned as follows,
[Equation]
with N = diag( N 1 , . . .
, N r ) ∈ N r × r , and R N the rectangular domain R N = { x ∈ R r | 0 ≤ x i <
N i , i = 1 , . .
, r } .
The DFT of a sequence can be computed exactly and efﬁciently using the fast Fourier transform (FFT)algorithm [8] which reduces the complexity from O ( M 2 ) to O ( M log M ) where M = N 1 N 2 . . .
N r .
Most importantly, the FFT algorithm is implemented in machine learning libraries such as PyTorch,which provide support for GPU acceleration and automatic differentiation capabilities.
Note that if we have a ﬁnite sequence, we may still deﬁne its DFT by implicitly extending thesequence periodically.
In particular, when a compactly supported function is sampled on its intervalof support, and the samples are used as input for a DFT, it is as if the periodic extension of thefunction had been sampled.
More precisely, consider an input sequence which corresponds to theevaluation of a function f on a regular grid of C = R N .
For simplicity, suppose that N i = N 1 for alli =
, r and consider the grid points x n = nL/N 1 for n ∈ Z r ∩ R N .
Taking the DFT of thesequence of general term u n = f ( x n ) we obtain for all n ∈ Z r ,
[Equation]
where y n are the reciprocal frequency points given by y n = n/L for n ∈ Z r ∩ R N .
The DFT of acompactly supported (or approximately compactly supported) function f sampled on the regular gridof points x k approximates the FT of f at the frequency points y n (up to a constant multiplicativefactor).
The FT is closely related to the notions of Fourier coefﬁcients and Fourier Series deﬁned hereafter.
Deﬁnition A.3 ( r -dimensional Fourier series ) .
Let f be a piecewise smooth function f : R r → Vwhich is periodic in x i with period L i ∈ R + for all i = 1 , . . .
, r .
The r -dimensional Fourier series off is a representation of the form,
[Equation]
where L = diag( L 1 , . . .
, L r ) ∈ R r × r and c n ( f ) are complex coefﬁcients, called Fourier coefﬁcients ,given by
[Equation]
where R L ⊂ R r denotes the rectangular domain of sides
L 1 , . . .
, L r .
We note that in the deﬁnition above, the sign ∼ means that the series is a formal series and nostatement is made about the convergence of the series (the forms of convergence are studied in
Alimov et al.
If f is compactly supported on R L , we may still deﬁne its Fourier coefﬁcients,and in this case F r ( f )( y n ) =
| det L | c n ( f ) at the frequency points y n =
L − 1 n .
Numerical consideration
Consider a function f which has compact support (or is periodic) which is observed at M locationsin its support (or its unitary cell R L ).
When using the DFT to approximate M points of thespectrum F r ( f )( y k ) (or M coefﬁcients c k ( f ) ), a so-called aliasing error usually occurs: due to theperiodicity of the DFT, the k th coefﬁcient of the DFT includes the contributions not only of the k thfrequency mode, but also from higher modes of the underlying function f .
In general the accuracyof the highest frequency modes is more impacted by this error, and aliasing occurs speciﬁcallywhen we compute nonlinear terms in the physical space.
For example, in the main paper weapproximate the evaluation on a discretization spatiotemporal grid D ×T of F − 1 d +1 ( F d +1 ( K ) F d +1 ( f ))by D − 1 d
+
( B D d +1 ( f | D ×T )) where f = 1 ≥
H θ,ξ ( z ) and H θ,ξ is nonlinear.
One possibility to mitigatealiasing is to set to zero the DFT terms (arising in nonlinearities) corresponding to the highestfrequency modes before we apply the inverse DFT to go back to the physical space.
This is preciselywhat we do when we parametrize only k 1 max × . . .
× k d +1 max × d h × d h entries of the complex tensor B ,and set the others to zero, hence resolving potential aliasing errors.
We note that speciﬁc rules havebeen proposed (notably in the literature on pseudo-spectral solvers) to deal with speciﬁc nonlinearities.
However, in the context of Neural SPDE we learn the nonlinearities, hence the number of frequencymodes that we retain is treated as an hyperparameter.
A.2 Stochastic simulation of Wiener processes
After deﬁning Wiener processes we outline the sampling procedure that we used to simulate thedatasets in the main paper.
For more details on computational aspects of SPDEs the reader is referredto Lord et al.
Throughout this section, H will denote a separable Hilbert space (e.g. H =
L 2 ( D ) ) with a completeorthonormal basis { φ k } k ∈ N .
Let (Ω , F , F t , P ) be a ﬁltered probability space.
A.2.1 Q-Wiener process
Consider an operator Q : H → H such that there exists a bounded sequence of nonnegative realnumbers { λ k } k ∈ N
such that Qφ k = λ k φ k for all k ∈ N (this is implied by Q being a trace class,non-negative, symmetric operator, for example).
Deﬁnition A.4 ( Q -Wiener process ) .
Let Q be a trace class non negative, symmetric operator on H .
A H -valued stochastic process { W ( t ) : t ≥ 0 } is called a Q -Wiener process if
W (0) = 0 almost surely;2.
W ( t ; ω ) is a continuous sample trajectory R
+ (cid:55)→ H , for each ω ∈ Ω ;3.
W ( t ) is F t -adapted and has independent increments W ( t ) − W ( s ) for s < t ;4.
W ( t ) − W ( s ) ∼ N (0 , ( t − s ) Q ) for all 0 ≤ s ≤ t .
In analogy to the Karhunen Loéve expansion, it can be shown that W ( t ) is a Q -Wiener process ifand only if for all t ≥ 0 ,
[Equation]
where β j ( t ) are i.i.d.
Brownian motions, and the series converges in L 2 (Ω , H ) .
Moreover the seriesis P -a.s.
uniformly convergent on [0 , T ] for arbitrary T > 0 .
(i.e. converges in L 2 (Ω , C ([0 , T ] , H )) ).
In the Navier-Stokes example, we drive the SPDE by samples ξ from a Q -Wiener process in two dimensions.
Here we follow Lord et al.
[27, Example 10.12] and explain how the sampling procedure works in this case.
Let D =
(0 , L 1 ) × (0 , L 2 ) and consider an L 2 ( D ) -valued Q-Wiener process
W ( t ) .
If the eigenfunctions of Q are given by,
[Equation]
numerical approximation of sample paths from W ( t ) are easy to obtain through a DFT.
Denote by λ k the eigenvalues of Q (e.g. λ k
=
e − α | k | 2 for some parameter α > 0 ) and let J be the index set deﬁned by,
[Equation]
The goal is to sample from the truncated expansion of W ( t ) ,
[Equation]
at the collection of sample points,
[Equation]
Consider the random variable Z ( t n , x ) deﬁned by,
[Equation]
meaning that ξ nj = a + ib with a, b i.i.d ∼ N (0 , 1) such that Z ( t n , x k ) is a complex random variablewith independent real and imaginary part with the same distribution as two independent copies of theincrement W J ( t n + ∆ t, x k ) − W J ( t n , x k ) .
Furthermore, Z ( t n , x k ) can be expressed in the form,
[Equation]
where (cid:101)
Z j 1 ,j 2 = (cid:112) ∆ tλ j 1 ,j 2 J 1 J 2 ξ nj 1 ,j 2 We recognize that the matrix with entries given by eq.
(12)is the 2D inverse DFT of the J 1 × J 2 matrix with entries (cid:101)
Z j 1 ,j 2 .
Therefore, we can sample twoindependent copies of
[Equation]
∆ t, [Table] k ) − W J ( t n , x
[Table] [Table]
k
[Table]
J [Table]
[Table]
k
[Table]
J
[Table]
A.2.2 Cylindrical Wiener process
If the operator Q =
I is the identity, then Q is not of trace class on H so that the series in eq.
(11)does not converge in L 2 (Ω , H ) .
This motivates the deﬁnition of cylindrical Wiener processes.
Deﬁnition A.5 ( Cylindrical Wiener process ) .
Let H be a separable Hilbert space.
A cylindrical
Wiener process (a.k.a space-time white noise) is a H -valued stochastic process { W ( t ) : t ≥ 0 }deﬁned by
[Equation]
(13) where { φ j } is any orthonormal basis of H and β j ( t ) are i.i.d.
Brownian motions.
In all examples except Navier-Stokes, we drive the SPDE by samples ξ from a cylindrical Wiener process in one dimension.
Let D =
(0 , L ) and consider an L 2 ( D )
-valued cylindrical Wiener process
W ( t ) .
As explained in Lord et al.
[27, Example 10.31], if we take the basis
[Equation]
numerical approximation of sample paths from W ( t ) are easy to obtain.
The goal is to sample fromthe truncated expansion,
[Equation]
(14) at the collection of sample points x k = kL/J for k = 1 , . . .
, J .
Observing that a trigonometricidentity yields,
[Equation]
the increments W J ( t n + ∆ t, x k ) − W J ( t n , x k ) ∼ N (0 , ∆ tL/J ) for all k = 1 , . . .
, k .
A.3 Numerical solvers
In this section we present an overview of the numerical solvers for SPDEs we used to generate the datafor all the experiments.
The stochastic Ginzburg-Landau (Sec. 4.1 and appendix B.2), stochastic wave(Appendix B.3) equations have been solved using the ﬁnite difference method, while the stochastic
Korteweg–De Vries (Sec. 4.2) and Navier Stokes (Sec. 4.3) equations have been solved using thespectral Galerkin method.
We use the same setup as in Sec. 2.
In particular, we focus on stochasticsemilinear evolution equations of the form
[Equation]
[Equation]
where W t is either a Q -Wiener process or a cylindrical Wiener process and L is a linear differentialoperator generating a semigroup e t L .
We consider nonlinearities F, G regular enough (see Lord et al.[27, Assumption 10.23]) to guarantee existence and uniqueness of mild solutions of eq.
Thm. 10.26].
A.3.1 Finite difference method
We illustrate this numerical method for the reaction-diffusion equation
[Equation]
with homogeneous Dirichlet boundary conditions and where (cid:15), σ > 0 are constants.
We assume forsimplicity that u 0 , u t , W t are real-valued and D = (0 , a ) .
The generalization to higher dimensions isstraightforward.
Consider the grid points x j = jh , where h = aJ and j = 0 , ..., J , for some spatial resolution J ∈ N .
Let u J ( t ) be the ﬁnite difference approximation of [ u ( t, x 1 ) , ..., u ( t, x J − 1 )] (similarly for W J ( t ) )resulting from the solution of the following SDE
[Equation]
where ˆ f ( u J ) =
[ f ( u 1 ) , ..., f ( u J − 1 )]
T and M is the ( J − 1) ×
( J − 1) matrix approximating Laplacian(with free boundary conditions) which is given by
[Equation]

[Equation]
−
[Equation]
 
One could modify M for speciﬁc boundary conditions.
For instance in the case of periodic boundaryone should modify M 1 ,J − 1 =
M J − 1 , 1 = − 1 (see Lord et al.
[27, Chapter 3.4] for Dirichlet and
Neuman boundary condition modiﬁcations of M ).
To discretize in time, we may apply numericalmethods for SDEs (see for example Lord et al.
[27, Chapter 8]).
Choosing the standard EulerMarayama scheme with time step ∆ t yields an approximation u J,n to u J ( t n ) at t n = n ∆ t deﬁnedby
[Equation]
The increments ( W J ( t n +1 ) − W J ( t n )) are generated using techniques discussed in Appendix A.2.
A.3.2 Spectral Galerkin method
Consider again a separable Hilbert space H .
Assume that the differential operator L in eq.
(15)has a complete set of orthonormal eigenfunctions { φ j } j ∈ N and eigenvalues λ j < 0 , ordered so thatλ j +1 <
λ j .
Then, we can deﬁne the semigroup e t L as follows
[Equation]
Deﬁne the Galerkin subspace V J
=
Span { φ 1 , ..., φ J } and the orthonormal projections P J : H → V Jas follows
[Equation]
Then, the following deﬁnes spectral Galerkin approximation of eq.
[Equation]
where u J := P J u and L J := P J L and W J =
P J W is as in (14).
Using a Euluer-Marayamadiscretization as above, we obtain the following discretization
[Equation]
This approach is particularly convenient for problems with additive noise where the eigenfunctionsof L and Q (the covariance of the Q -Wiener process W ) are equal, which is the case for all theexperiments in this paper generated with this method.
The eigenfunctions of the Laplacian withperiodic boundary conditions correspond to the Fourier basis exponentials; therefore, one can deﬁnethe projection P J in terms of the DFT.
B Further experiments
We start this section with additional details on the training of NSPDE and the baseline models,including how the relevant hyperparameters have been selected for each model.
B.1 Additional experimental details
For all experiments the dataset is split into a training, validation and test sets with relative sizes70% / 15% / 15% .
For all models, a grid search on the hyperparameters is performed using the trainingand validation sets.
We use the Adam optimizer and a scheduler which reads the validation loss andreduces the learning rate if no improvement is seen for a patience number of epochs.
Additionally,an early stopping method is used to halt the training of the model if no improvement is seen after apatience number of epochs.
The hyperparameters included in the grid search are stated below andexamples of hyperparameter selection results are provided in tables 5 to 9.
NSPDE The hyperparameters included in the grid search are the number of frequency modes usedto parametrize the kernel in Fourier space B = F d +1 ( K ) and the number of forward iterations usedto solve the ﬁxed point problem.
FNO The hyperparameters included in the grid search are the number of frequency modes used toparametrize the kernel and the number of layers M .
Note that the numbers of frequency modes in thegrid search differ from the ones used for the NSPDE model by a factor 2 to ensure that the effectivenumber of retained modes is the same.
For both the NSPDE model and FNO, we kept the number ofhidden channels ﬁxed to d h = 32 as this systematically yielded better performances than previouslyincluded values and enabled to perform the grid search in a reasonable time.
DeepONet
The Deep Operator Network (DeepONet) [28] is another popular class of neural network models for learning operators on function spaces.
The DeepONet architecture is based on the universal approximation theorem of Chen and Chen [6].
It consists of two sub-networks referred to as the branch and the trunk networks.
The trunk acts on the coordinates ( t, x ) ∈ [0 , T ] × D , while the branch acts on the evaluation of the initial condition u 0 on a discretized grid D .
Therefore, the
DeepONet is not a space resolution-invariant architecture.
The output of the network is expressed as
[Equation]
where the b k and the τ k are the outputs of the branch and trunk network respectively.
The trunknetwork is usually a feedforward neural network, and one can chose the architecture of the branchnetwork depending on the structure of the input domain.
We follow Lu et al.
[28] and use feedforwardneural networks for both the trunk and the branch networks.
We perform a grid search on the depthand width of the trunk and branch feedforward neural networks.
NRDE/NCDE/NCDE-FNO
The hyperparameters included in the grid search are the number ofhidden channels and the type of solver as implemented by torchdiffeq [5].
We note that we used adepth-2 NRDE model (depth-2 already results in d ξ = 8 385 for forcings observed at 128 spatialpoints and higher depths models could not ﬁt in memory) and recall that NCDE is a depth-1 NRDE.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
B.2 Stochastic Ginzburg-Landau equation
Recall that the stochastic Ginzburg-Landau equations are of the form,
[Equation]
[Equation]
[Equation]
∈ [Equation] [Equation] [0 , 1] subject to either Periodic or Dirichlet boundary conditions.
Periodic boundary conditions are given byu ( t, 0) =
u ( t, 1) for all t ≥ 0 and Dirichlet boundary conditions are given by u ( t, 0) =
u ( t, 1) =
0for all t ≥ 0 .
Initial condition we take as in Sec.
4.1 u 0 ( x ) = x (1 − x ) + κη ( x ) with κ = 0 orκ = 0 .
1 depending on a task.
In both Periodic and Dirichlet case we can take η ( x ) as in eq.
(9)though in Dirichlet case one must take a 0 = 0 to ensure u 0 being zero at the boundary.
We ﬁrst reproduce an experiment from Sec.
4.1 on the additive stochastic Ginzburg-Landau equationbut with Dirichlet boundary conditions instead of the periodic.
We compare it to the benchmark of
FNO model which was the most successful among all the benchmarks of Sec. 4.
From Table 10 wesee that even though Neural SPDE model depends on the spectral methods the errors did not increasecompared to the periodic equation in Sec.
4.1 (see Table 1).
Our algorithm still outperforms FNOwhose relative L 2 error increased slightly.
The fact that Neural SPDE can be applied to non-periodicequations could be perhaps explained by interpolation ( L θ ) and projection ( Π θ ) neural networks thatcould correct for non-periodicity of the data.
[Table]
We now take a look at the speciﬁc hyperparameter: number of forward iterations in the ﬁxed pointsolver.
We also call this a number of Picard iterations P .
Theoretically as P increases Fixed
Point Solver should converge to the true solution (see [11]).
This suggests that higher P shouldimprove the performance of the Neural SPDE algorithms.
In practise we observed in both additive
Ginsburg Landau equation from Sec.
4.1 and in KdV equation from Sec.
4.2 that P = 1 couldalready be enough.
This could be explained either by dominance of the linear part of the equation or by overﬁtting in these cases.
Thus we present an experiment on the multiplicative stochastic
Ginzbug-Landau equation over a longer (compared to Sec. 4.1) time interval.
In the Table 11 wecompare NSPDE with P ∈ { 1 , 2 , 3 , 4 } and again include FNO benchmark (which performed bestin the previous experiments).
We see that NSPDE with even P = 1 outperforms FNO.
Relative
L 2 error for T = 0 .
05 increases for both NSPDE and FNO due to more complicated multiplicativenoise.
In Table 11 we present for each P the best result over other hyperparameters obtained bycross validation.
One could clearly see an improvement in error as we increase the number of Picarditerations P (with an exception of the case T = 0 .
05 where P = 3 outperformed P = 4 ).
Thisimprovement becomes more apparent as the time frame T increases.
Heuristically (and qualitatively)this is due to the fact that for the short times solution of the SPDE is relatively close to its linearisedversion and that nonlinearity of the equation starts to play a bigger role for larger T .
[Table]
B.3 The stochastic wave equation
In this section we consider the following nonlinear wave equation with multiplicative stochasticforcing,
[Equation]
The nonlinear stochastic wave equation arises in relativistic quantum mechanics and is also used insimulations of nonlinear waves that are subject to either noisy observations or random forcing.
Werefer a reader to Temam [34] for an overview on the nonlinear wave equation.
The above equationcan put in a form of eq.
(2) by rewriting it as a system for ( u, v ) =
( u, ∂ t u ) .
To generate trainingdatasets, we solve the SPDE using a ﬁnite difference method with 128 evenly distanced points inspace and a time step size ∆ t = 10 − 3 .
As in Chevyrev et al.
[7, eq.
(3.5)], we solve the SPDE until
T = 0 .
We then downsample the temporal resolution by a factor 5 , resulting in 100 time points.
Here, the initial condition is given by u 0 ( x ) = sin(2 πx ) + κη ( x ) , where η is deﬁned in eq.
(9) andfor simplicity initial velocity v 0 is taken deterministic v 0 ( x ) =
x (1 − x ) .
Similarly to Sec.
4.1 weeither take κ = 0 or κ = 1 to generate datasets where the initial condition is either ﬁxed or variesacross samples.
Each dataset consists of N = training observations.
Model u 0 (cid:55)→ u ξ (cid:55)→
[Equation]
u
NCDE x 0.
[Table]
B.4 Deterministic Navier-Stokes PDE
In this ﬁnal experiment, we demonstrate that our Neural SPDE model can also be used in thesetting of PDEs without any stochastic term.
We do so by studying the example from [25] ondeterministic Navier-Stokes.
More precisely, we consider the 2D Navier-Stokes equation for aviscous, incompressible ﬂuid in vorticity form:
[Equation]
where u : [0 , T ] × [0 , 1] 2 → R 2 is the velocity ﬁeld, w = ∇× u is the vorticity with w 0 : [0 , 1] 2 → Rbeing the initial vorticity.
Here f is a deterministic forcing term which we take as in [25].
Wefollow the experimental setup from [25] and use the dataset (available under an MIT license) whereν = 10 − 5 , N = 1000 and T = 20 .
We achieve similar performances as FNO with a L2 error of 0 . 17 .
A comparison between a true and predicted trajectory is depicted in Fig.