Generic bounds on the approximation error for physics-informed (and) operator learning
Abstract
We propose a very general framework for deriving rigorous bounds on the approximation error for physics-informed neural networks (PINNs) and operator learningarchitectures such as DeepONets and FNOs as well as for physics-informed operator learning.
These bounds guarantee that PINNs and (physics-informed) DeepONets or FNOs will efﬁciently approximate the underlying solution or solutionoperator of generic partial differential equations (PDEs).
Our framework utilizesexisting neural network approximation results to obtain bounds on more involvedlearning architectures for PDEs.
We illustrate the general framework by derivingthe ﬁrst rigorous bounds on the approximation error of physics-informed operatorlearning and by showing that PINNs (and physics-informed DeepONets and FNOs)mitigate the curse of dimensionality in approximating nonlinear parabolic PDEs.
Introduction
The efﬁcient numerical approximation of partial differential equations (PDEs) is of paramountimportance as PDEs mathematically describe an enormous range of interesting phenomena in thesciences and engineering.
Machine learning techniques, particularly deep learning, are playingan increasingly important role in this context.
For instance, given their universal approximationproperties, deep neural networks serve as ansatz spaces for supervised learning of a variety of(parametric) PDEs [19, 70, 40, 53, 54] and references therein.
In this setting, large amounts of trainingdata might be required.
However, this data is often acquired from expensive computer simulationsor physical measurements [53], necessitating the design of learning frameworks that work withlimited data.
Physics-informed neural networks (PINNs) , proposed by [18, 42, 41] and popularizedby [67, 68], are a prominent example of such a learning framework as the residual of the underlying
PDE is minimized within the class of neural networks and in principle, little (or even no) trainingdata is required.
PINNs and their variants have proven to be a very powerful and computationallyefﬁcient framework for approximating solutions to PDEs, [69,
Often in the context of PDEs, one needs to approximate the underlying solution operator that maps one inﬁnite-dimensional function space into another [27, 39].
As neural networks can only mapbetween ﬁnite dimensional spaces, a new ﬁeld of operator learning is emerging wherein novellearning frameworks need to be designed in order to approximate operators.
These include deepoperator networks (DeepONets) [9, 49] and their variants as well as neural operators [39], whichgeneralize neural networks to this setting.
A variety of neural operators have been proposed, see[45, 46] but arguably, the most efﬁcient form of neural operators is provided by the so-called Fourierneural operators (FNOs) [44].
Both DeepONets and FNOs have been very successfully deployed inscientiﬁc computing [50, 56, 8, 48, 47, 66] and references therein.
Finally, one can combine PINNsand operator learning to design physics-informed DeepONets/FNOs [74, 47, 73, 22].
From a theoretical perspective, one needs to provide a rigorous guarantee that the learning frameworkcan approximate the underlying PDE solution (operator) to desired accuracy.
More precisely, givenan error tolerance " > 0 , we need to rigorously prove that the approximation error of the neuralnetwork (operator) can be made smaller than " .
For efﬁcient approximation , one has to furtherensure that the computational complexity (measured in terms of the size) of the learning architecturegrows at most polynomially in "  1 .
In particular, exponential growth has to be ruled out.
As neuralnetworks, DeepONets and FNOs are all universal approximators [13, 9, 49, 43, 38] of the underlyingfunctions or operators, it is possible to show that the approximation error can be made as small asdesired.
However, these results do not guarantee efﬁcient approximation as the underlying networksize could still grow exponentially with decreasing error, see [77] for neural networks in very highspatial dimensions, [43] for DeepONets and [38] for FNOs.
Hence, the real theoretical challengein this context lies in proving efﬁcient approximation results for the different learning architecturesin scientiﬁc computing.
Such efﬁcient approximation results have mostly been obtained for neuralnetworks in the supervised learning setting e.g. [23, 36, 32, 4, 63] and references therein.
In contrast,there is a relative scarcity of such efﬁcient approximation results for PINNs and operator learningwith notable exceptions being [25, 14, 15] (for PINNs), [43] (for DeepONets) and [38] (for FNOs).
Moreover, the underlying proofs in these works are often on a case-by-case basis and the overallabstract structure is not clearly identiﬁed.
Finally, no similar rigorous approximation results forphysics informed operator learning are available till date.
This paucity of generic efﬁcient approximation results for PINNs and operator learning for PDEs setsthe stage for the current paper where our main contribution is to propose a very general framework(Section 3) for proving bounds on the approximation error for space-time neural networks, PINNs,
DeepONets, FNOs and physics-informed DeepONets and FNOs for very general PDEs.
Consequently,we obtain the ﬁrst rigorous bounds for physics-informed operator learning in literature.
Our frameworkis based on the observation that error estimates for different types of neural network architecturescan all be obtained from one another.
As error estimates for neural network approximations of PDEsolutions at a ﬁxed time are the easiest to obtain, and hence constitute the largest proportion ofcurrently available estimates, we devote particular attention to demonstrating how these availableestimates can be used to obtain novel bounds on the approximation error for space-time networks,
PINNs and (physics-informed) operator learning.
Our results provide a roadmap for derivingmathematical guarantees for deep learning methods in scientiﬁc computing by simplifying the proofs,as the needed work essentially reduces to verifying a small number of assumptions.
We demonstratehow the generic error bounds from Section 3 can be applied in practice in Section 4, among othersby giving short alternative proofs for known results and also proving a number of novel results.
In particular, we show in Section 4.1 that PINNs can overcome the curse of dimensionality fornonlinear parabolic PDEs such as the Allen-Cahn equation i.e., the network size does not growexponentially with increasing spatial dimension.
Moreover, dimension-independent convergencerates are also obtained for (physics-informed) DeepONets and FNOs, provided that the PDE solutionsare sufﬁciently smooth.
These are the ﬁrst results of their kind.
We note that many of the proofs andsome examples are deferred to the supplementary material ( SM ).
Preliminaries2.1 Setting
Given T > 0 and D ⇢ R d compact, consider the function u : [0 , T ] ⇥ D !
R m , for m  1 , thatbelongs to a separable Banach space H and solves the following (time-dependent) PDE,
[Equation]
where u 0 2 Y ⇢
L 2 ( D ) is the initial condition and L a : H ! L 2 ([0 , T ] ⇥ D ) is a differentialoperator that can depend on a parameter (function) a 2 Z ⇢ L 2 ( D ) .
In our notation, we will oftensuppress the dependence of L :=
L a on a for simplicity.
Depending on the context, one mightwant to recover one of the following mathematical objects: for ﬁxed a and u 0 , one might want toapproximate u ( T ) or u ( · , t ) , for all t 2 [0 , T ] with a neural network; a more challenging task wouldbe to learn the bounded solution operator G : X !
L 2 ( ⌦ ) , where X = Y or X = Z and ⌦ = D 86 or ⌦ =
[0 , T ] ⇥ D .
We will use this notation consistently throughout the paper, see SM A.1 for anoverview.
2.2 Approximating PDEs with neural networksNeural networks A (feedforward) neural network
✓ : R d 0 !
R d L is deﬁned as a concatenationof afﬁne maps A l : R d l  1 !
R d l : z 7!
W l z + b l and an activation function  : R !
R that isapplied component-wise resulting in,
[Equation]
The weights and biases of the afﬁne maps ✓ = { W l , b l }
 l  L are the trainable parameters.
We will 93 quantify the size of a neural network by its depth( u ✓ ) :
=
L and its width( u ✓ ) =
max l d l .
In order toobtain a neural network that approximates the solution u of PDE (2.1) at time t =
T , one chooses theparameters of u ✓ such that a discretization (quadrature) of J ( ✓ ) =
k u ( T )  u ✓ k L 2 ( D ) is minimized.
The training data is acquired from either measurements or potentially expensive simulations.
PINNs Physics-informed neural networks are neural networks that are trained with a different,residual-based loss function.
As the PDE solution u satisﬁes L ( u ) = 0 , the goal of physicsinformed learning is to ﬁnd a neural network u ✓ for which the PDE residual is approximately zero,
L ( u ✓ ) ⇡ 0 .
To ensure uniqueness, one also needs to require that the initial condition is satisﬁed i.e.,u ✓ (0 , x ) ⇡ u 0 ( x ) , and similarly for boundary conditions.
In practice one minimizes a quadratureapproximation of J ( ✓ ) =
k L ( u ✓ ) k 2 L 2 ([0 ,T ] ⇥ D ) + k u ✓ (0 , · )  u 0 k
L 2 ( D ) , where additional terms can beadded to (approximately) impose boundary conditions and augmented by data.
A desirable propertyof PINNs is that only very little or even no training data is needed to construct the loss function.
Operator learning In order to approximate operators, one needs to allow the input and output ofthe learning architecture to be inﬁnite-dimensional.
A possible approach is to use deep operatornetworks (DeepONets), as proposed in [49].
Given m , ﬁxed sensor locations { x j } mj =1 ⇢ D andthe corresponding sensor values { v ( x j ) } mj =1 as input, a DeepONet can be formulated in termsof two (deep) neural networks: a branch net  : R m !
R p and a trunk net ⌧ : D !
R p
+1 .
The branch and trunk nets are then combined to approximate the underlying nonlinear operatoras the following DeepONet G ✓ : X !
L 2 ( D ) , with G ✓ ( v )( y ) =
⌧ 0 ( y )
+
P pk =1  k ( v ) ⌧ k
( y ) .
Asecond approach is that of neural operators , which generalize hidden layers by including a non-localintegral operator [45], of which particularly Fourier neural operators (FNOs) [44] are already wellestablished.
The practical implementation (i.e. discretization) of an FNO maps from and to the spaceof trigonometric polynomials of degree at most N 2 N , denoted by L 2 N , and can be identiﬁed witha ﬁnite-dimensional mapping that is a composition of afﬁne maps and nonlinear layers of the form
L l ( z ) j =  ( W l v j + b l,j F  1 N ( P l ( k ) · F N ( z )( k ) j )) , where the P l ( k ) are coefﬁcients that deﬁne anon-local convolution operator via the discrete Fourier transform F N , see [38].
Physics-informed operator learning Both DeepONets and FNOs are trained by choosing a suitableprobability measure µ on X and minimizing a quadrature approximation of J ( ✓ ) =
k G ✓ ( v )
G ( v ) k
L 2 µ ⇥ dx ( X ⇥ ⌦ ) .
Generating training sets might require many calls to an expensive PDE solver,leading to an enormous computational cost.
In order to reduce or even fully eliminate the needfor training data, physics-informed operator learning has been proposed in [74] for DeepONetsand in [47] for FNOs.
Similar to PINNs, the training procedure aims to minimize a quadratureapproximation of J ( ✓ ) =
k L ( G ✓ ) k L 2 µ ⇥ dx
( X ⇥ ⌦ ) .
General results
We propose a framework to obtain bounds on the approximation error for the various neural networkarchitectures introduced in Section 2.2.
Figure 1 visualizes how different types of error estimates canbe obtained from one another.
Every box shows the name of the network architecture, the form of therelevant loss and the theorem which proves the corresponding estimate for the approximation error.
Every arrow in the ﬂowchart represents a proof technique that allows one to transfer an error estimatefrom one type of method to another (see caption of Figure 1 for an overview of those techniques).
We give particular attention to the case where it is known that a neural network can efﬁcientlyapproximate the solution to a time-dependent PDE at a ﬁxed time.
Such neural networks are usuallyobtained by emulating a classical numerical method.
Examples include ﬁnite difference schemes,ﬁnite volume schemes, ﬁnite element methods, iterative methods and Monte Carlo methods, e.g.
Neural network (ﬁxed time)
u
[Figure]
T )
[Figure]
" Assumed to be known
FNO
[Figure]
DeepONet
[Figure]
[Figure]
Neural
[Figure]
(space-
[Figure] [Figure]
[Equation]
[Figure]
PINN
[Equation]
[Figure]
[Equation]
[Figure]
For " > 0 , we denote by U " : X ⇥ [0 , T ] ! H an operator that for any t 2 [0 , T ]maps any initial condition/parameter function v 2 X to a neural network U " ( v, t ) that approximatesthe PDE solution G ( v )( · , t )
= u ( · , t ) 2 L q ( D ) , q 2 { 2 , 1 } , at time t as speciﬁed below.
Moreover, 140 we will assume that we know how its size depends on the accuracy " .
Assumption 3.1.
Let q 2 { 2 , 1 } .
For any B, " > 0 , ` 2 N , t 2 [0 , T ] and any v 2 X withk v k C `  B there exist a neural network U " ( v, t ) : D !
R and a constant C B " , ` > 0 s.t. U "
[Equation] v [Equation] t )  G ( v )
[Equation]
· , t )  L q [Equation] D )  " and max
[Equation]  U
[Equation]  [Equation]
Under this assumption, we prove the existence of space-time neural networks and PINNs thatefﬁciently approximate the PDE solution (Section 3.1), as well as FNOs and DeepONets (Section3.2) and physics-informed FNOs and DeepONets (Section 3.3).
3.1 Estimates for (physics-informed) neural networks
We will construct a space-time neural network u ✓ for which both k u ✓  u k L q ([0 ,T ] ⇥ D ) and the PINNloss k L ( u ✓ ) k L q ([0 ,T ] ⇥ D ) are small.
To accurately approximate the time derivatives of u we emulate 149 Taylor expansions, whereas for the spatial derivatives, we employ ﬁnite difference (FD) operatorsin our proofs.
Depending on whether forward, backward or central differences are used, a FDoperator might not be deﬁned on the whole domain D , e.g. for f 2 C ([0 , 1]) the (forward) operator
+ h
[ f ] := f ( x + h )  f ( x ) is not well-deﬁned for x 2 (1  h, 1] .
This can be solved by resortingto piecewise-deﬁned FD operators, e.g. a forward operator on [0 , 0 . 5] and a backward operator on(0 . 5 , 1] .
In a general domain ⌦ one can ﬁnd a well-deﬁned piecewise FD operator if ⌦ satisﬁes thefollowing assumption, which is satisﬁed by many domains (e.g. rectangular, smooth).
Assumption 3.2.
There exists a ﬁnite partition P of ⌦ such that for all P 2 P there exists " P > 0 and 157 v P 2 B
[Equation]
1 = { x 2 R dim( ⌦ ) : k x k 1  1 } such that for all x 2 P it holds that x + " P ( v P + B 1 1 ) ⇢ ⌦ .
Additionally, we need to assume that the PINN error can be bounded in terms of the errors related toall relevant partial derivatives, denoted by D ( k, ↵ ) :
= D kt D ↵ x := @ kt @
↵ 1 x 1 . . .
↵ d x d , for ( k, ↵ ) 2 N d + .
This assumption is valid for many classical solutions of PDEs, a few worked out examples can befound in SM D.5 (gravity pendulum) and SM D.6 (Darcy ﬂow).
Assumption 3.3.
Let k, ` 2 N , q 2 { 2 , 1 } , C > 0 be independent from d .
For all v 2 X it holds,
[Equation]
In this setting, we prove the following approximation result for space-time networks and PINNs.
Note that for PINNs one can set X := { v } , G ( v ) :
= u and v := u 0 in Assumption 3.3.
Theorem 3.4.
Let r, s 2 N , let u 2 C ( s,r ) ([0 , T ] ⇥ D ) be the solution of the PDE (2.1) and let
Assumption 3.1 be satisﬁed.
There exists a constant C ( s, r ) > 0 such that for every M 2 N and" , h > 0 there exists a tanh neural network u ✓ : [0 , T ] ⇥ D !
R for which it holds that,
[Equation]
and if additionally Assumption 3.2 and Assumption 3.3 hold then,
[Equation]
Moreover, depth( u ✓ )  C · depth( U " ) and width( u ✓ )  CM · width( U " ) .
Proof.
We only provide a sketch of the full proof ( SM B.2).
The main idea is to divide [0 , T ] into Muniform subintervals and construct a neural network that approximates a Taylor approximation in timeof u in each subinterval.
In the obtained formula, we approximate the monomials and multiplicationsby neural networks ( SM A.7) and approximate the derivatives of u by ﬁnite differences and use(A.2) of SM A.2 to ﬁnd an error estimate in C k
([0 , T ] , L q ( D ))
-norm.
We use again ﬁnite differenceoperators to prove that spatial derivatives of u are accurately approximated as well.
The neuralnetwork will also approximately satisfy the initial/boundary conditions as k u ✓  u k L 2 ( @ ([0 ,T ] ⇥ D )) .
177 C poly( d ) k u ✓  u k H 1 ([0 ,T ] ⇥ D ) , which follows from a Sobolev trace inequality.
We note that the bounds (3.3) and (3.4) together imply that there exists a neural network for whichthe total error as well as the PINN loss can be made as small as possible, providing a solid theoreticalfoundation to PINNs for approximating the PDE (2.1).
3.2 Estimates for operator learning
In this section, we use Assumption 3.1 to prove estimates for DeepONets and FNOs.
First, we provea generic error estimate for FNOs.
Using the known connection between FNOs and DeepONets( SM Lemma B.6) this result can then easily be applied to DeepONets (Corollary 3.7).
In order toprove these error estimates, we need to assume that the operator U " from Assumption 3.1 is stablewith respect to its input function, as speciﬁed in Assumption 3.5 below.
Moreover, we will take thed -dimensional torus as domain D = T d = [0 , 2 ⇡ ) d and assume periodic boundary conditions forsimplicity in what follows.
This is not a restriction, as for every Lipschitz subset of T d there exists a(linear and continuous) T d -periodic extension operator of which also the derivatives are T d -periodic[38, Lemma 41].
Assumption 3.5.
Assumption 3.1 is satisﬁed and let p 2 { 2 , 1 } .
For every " > 0 there exists a 192 constant C " stab > 0 such that for all v, v 0 2 X it holds that,
[Equation]
In this setting, we prove a generic approximation result for FNOs.
Theorem 3.6.
Let r 2 N , T > 0 , let G : C r ( T d ) !
C r ( T d ) be an operator that maps a functionu 0 to the solution u ( · , T ) of the PDE (2.1) with initial condition u 0 , let Assumption 3.5 be satisﬁedand let p ⇤ 2 { 2 , 1 } \ { p } .
Then there exists a constant C > 0 such that for every " > 0 , N 2 Nthere is an FNO G ✓ : L 2 N ( T d ) !
L 2 N ( T d ) of depth
O (depth( U " )) and width O ( N d width( U " ))with accuracy,
[Equation]
Proof.
We give a sketch of the proof, details can be found in SM B.3.
Given function values of von a uniform grid with grid size 1 /N , we use trigonometric polynomial interpolation ( SM A.6) toreconstruct v and use this together with Assumption 3.1 to construct a neural network.
The resultingapproximation is then projected onto the space L 2 N , of trigonometric polynomials of degree at most N 2 N , through trigonometric polynomial interpolation.
A recent result, [38, Theorem 36] ( SM Lemma B.6), shows that any error bound for FNOs alsoimplies an error bound for DeepONets, by choosing the trunk nets as neural network approximationsof the Fourier basis.
We apply this result with " ⇠ poly(1 /N ) to Theorem 3.6 to obtain the followinggeneric error bound for DeepONets.
Corollary 3.7.
Assume the setting of Theorem 3.6.
Then for every " > 0 , N 2 N and ev- 209 ery corresponding FNO
G ✓ from Theorem 3.6 there exists a DeepONet G ⇤ ✓ : X !
L 2 ( D ) withwidth(  ) =
O ( N d ) , depth(  ) =
O (depth( G ✓ )) , width( ⌧ ) =
O ( N d +1 ) and depth( ⌧ )  3 thatsatisﬁes (3.6) .
3.3 Estimates for physics-informed operator learning
Using the techniques from previous sections, we now present the very ﬁrst theoretical result forphysics-informed operator learning.
We demonstrate that if an error estimate for a DeepONet/FNOand the growth of its derivatives are known (see SM D.1 on how to obtain these), then one canprove an error estimate for the corresponding physics-informed DeepONet/FNO.
For simplicity,the following result focuses only on operators mapping to C r
( D ) but the generalization to e.g.
C r ([0 , T ] ⇥ D ) is immediate by setting D 0 :=
[0 , T ] ⇥ D .
Theorem 3.8.
Consider an operator G : X !
C r ( D ) , r 2 N , that satisﬁes Assumption 3.2 and 220 Assumption 3.5 with ` 2 N .
Let
⇤ 2 (0 , 1 ] , let  , C (  ) > 0 with    ⇤ and let  : N !
R be a 221 function such that for all p 2 N there is a DeepONet/FNO G ✓ such that
[Equation]
and
G
[Equation]
v [Equation] Then for all  2 R with 0 <   ( r  ` )  ⇤  ` ( r ) r there exists a constant C ⇤ > 0 such that for allv 2 X
and p 2 N it holds that
[Equation]
Proof.
For suitable D ↵ , use SM Lemma B.1 with q = 2 , f 1 = G ( v ) and f 2 = G ✓ ( v ) together with(3.7) to ﬁnd
[Equation]
Let  2 R with 0 <   ( r  ` )  ⇤  ` ( r ) r .
We carefully balance terms by setting h = p   ( r )+  r  ` and = ` r  `  ( r ) + rr  `  to ﬁnd (3.8).
Conclude using Assumption 3.5.
Finally, we use Theorem 3.4 to present an alternative error estimate for a physics-informed DeepONetin the case that Assumption 3.1 is satisﬁed.
As this assumption is different from assuming accessto an error bound for the corresponding DeepONet, it is interesting to use the techniques from theprevious sections rather than directly apply Theorem 3.8.
The proof of the following theorem can befound in SM B.4.
Theorem 3.9.
Let r, s 2 N , T > 0 , let G : C r ( T d ) !
C ( s,r ) ([0 , T ] ⇥ T d ) be an operator that mapsa function u 0 to the solution u of the PDE (2.1) with initial condition u 0 , let Assumption 3.5 besatisﬁed and let p ⇤ 2 { 2 , 1 } \ { p } .
There exists a constant C > 0 such that for every Z, N, M 2 N ," , ⇢ > 0 there is an DeepONet G ✓ : C r ( T d ) !
L 2 ([0 , T ] ⇥ T d ) with Z d sensors with accuracy, [Equation] [Equation] and simultaneously,
[Equation]
for all v .
Moreover, it holds that, depth(  ) = depth
( U " ) , width(  ) =
O ( M ( Z d + N d width( U " ))) ,depth( ⌧ ) = 3 and width( ⌧ )
=
O ( MN d ( N + ln( NM ))) .
Applications
We demonstrate the power and generality of the framework proposed in Section 3 by applying thepresented theory to the following case studies.
First, we demonstrate how these generic bounds canbe used to overcome the curse of dimensionality (CoD) for linear Kolmogorov PDEs and nonlinearparabolic PDEs (Section 4.1).
These are the ﬁrst available results that overcome the CoD for nonlinearparabolic PDEs for PINNs and (physics-informed) operator learning.
Next, we apply the results of
Section 3.3 to both linear and nonlinear operators and provide bounds on the approximation error forphysics-informed operator learning.
4.1 Overcoming the curse of dimensionality
For high-dimensional PDEs, it is not possible to obtain efﬁcient approximation results using standardneural network approximation theory [77, 14] as they will lead to convergence rates that sufferfrom the CoD, meaning that the neural network size scales exponentially in the input dimension.
Inliterature, one has shown for some PDEs that their solution at a ﬁxed time can be approximated toaccuracy " > 0 with a network that has size O (poly( d ) "   ) and therefore overcomes the CoD .
Linear Kolmogorov PDEs We consider linear time-dependent PDEs of the following form.
Setting 4.1.
Let s, r 2 N , u 0 2 C 20 ( R d ) and let u 2 C ( s,r ) ([0 , T ] ⇥ R d ) be the solution of L ( u )( x, t ) = @
t u ( x, t )  1 2Tr (  ( x )  ( x ) T  x [ u ]( x, t ))
µ ( x ) T r x [ u ]( x, t ) = 0 , u (0 , x ) = u 0 ( x ) (4.1) for all ( x, t ) 2 D ⇥ [0 , T ] , where  : R d !
R d ⇥ d and µ : R d !
R d are afﬁne functions and forwhich k u k C ( s, 2) grows at most polynomially in d .
For every " > 0 , there is a neural network b
u 0 ofdepth O
(poly( d ) log
[Equation]
) and width O (poly( d ) [Equation]   ) such that k u 0  [Equation]
Prototypical examples of such linear Kolmogorov PDEs include the heat equation and the BlackScholes equation.
In [23, 7, 36] the authors construct a neural network that approximates u ( T ) andovercomes the CoD by emulating Monte-Carlo methods based on the Feynman-Kac formula.
In[15] one has proven that PINNs overcome the CoD as well, in the sense that the network size growsas O
(poly( d ⇢ d ) "   ) with ⇢ d as deﬁned in SM (C.10)).
For a subclass of Kolmogorov PDEs it isknown that ⇢ d = poly( d ) , such that the CoD is fully overcome.
We provide a shorter proof (in SM C.2) for this result based on Theorem 3.4.
SM Lemma C.6 veriﬁes that Assumption 3.1 is satisﬁed.
Theorem 4.2.
Assume that Setting 4.1 holds.
For every " > 0 and d 2 N , there is a neural networku ✓ of depth O (depth( b u 0 )) and width O (poly( d ⇢ d ) "  (2+  ) r +  r  2 s +1 s  1  1+  s  1 ) such that,
[Equation]
Nonlinear parabolic PDEs Next, we consider nonlinear parabolic PDEs as in Section 4.3, whichtypically arise in the context of nonlinear diffusion-reaction equations that describe the change inspace and time of some quantities, such as in the well-known Allen-Cahn equation [1].
Setting 4.3.
Let s, r 2 N and for u 0 2 X ⇢ C r ( T d ) let u 2 C ( s,r ) ([0 , T ] ⇥ T d ) be the solution of L ( u )( x, t ) = [Equation]
t u [Equation] t,
[Equation]
with period boundary conditions, where F : R !
R is a polynomial and for which k u k C ( s, 2) growsat most polynomially in d .
For every " > 0 , there is a neural network b
u
of depth O (poly( d ) ln  1 / "  )and width O (poly( d ) "   ) such that k u 0  b u 0 k L 1 ( T d ) < " .
Let µ , resp.
µ ⇤ , be the normalized
Lebesgue measure on [0 , T ] ⇥ T d , resp. @
([0 , T ] ⇥ T d ) .
In [32] the authors have proven that ReLU neural networks overcome the CoD in the approximationof u ( T ) .
We have reproven this result in SM Lemma C.14 for tanh neural networks to show that
Assumption 3.1 is satisﬁed.
Using Theorem 3.4 we can now prove that PINNs overcome the CoD fornonlinear parabolic PDEs.
The proof is analogous to that of Theorem 4.2.
Theorem 4.4.
Assume Setting 4.3.
For every  , " > 0 and d 2 N there is a neural network u ✓ ofdepth O (ln  1 / "  ) and width O (poly( d ) "  (2+  ) r +  r  2 s
[Equation]
1 s  1  1+  s  1 ) such that, [Equation]
[Equation]
Similarly, one can use the results from Section 3.2 to obtain estimates for (physics-informed)
DeepONets for nonlinear parabolic PDEs (4.3) such as the Allen-Cahn equation.
In particular, adimension-independent convergence rate can be obtained if the solution is smooth enough, whichimproves upon the result of [43], which incurred the CoD. For simplicity, we present results for C (2 ,r )functions, rather than C ( s,r ) functions, as we found that assuming more regularity did not necessarilyfurther improve the convergence rate.
The proof is given in SM B.4.
Theorem 4.5.
Assume Setting 4.3 and let G : X !
[Equation] [Equation]
T [Equation] and G ⇤ : X !
C (2 ,r ) ([0 , T ] ⇥ T d ) : u 0 7!
u .
For every  , " > 0 , there exists a DeepONets G ✓ and G ⇤ ✓ such that
[Equation]
Moreover, for G ✓ we have O ( "  d +  r ) sensors and,
[Equation]
whereas for G ⇤ ✓ we have O ( "  (3+  ) d r  2 ) sensors and,
[Equation]
4.2 Error bounds for physics-informed
operator learning
We demonstrate how Theorem 3.8 can be used to generalize available error estimates for DeepONetsand FNOs, e.g. [43, 38] and SM D.1, to estimates for their physics-informed counterparts.
Linear operators In the simplest case, the operator G of interest is linear.
In [43, Theorem D.2], ageneral error bound for ReLU DeepONets for linear operators has been established, which still holdsfor tanh DeepONets.
Using Theorem 3.8 it is then straightforward to prove convergence rates forphysics-informed DeepONets for solution operators of linear PDEs (2.1).
Consider an operator G : X !
L 2 ( T d ) : v 7!
u as in Section 2.1, where v is the parameter/initialcondition and u the solution of the PDE (2.1).
Following [43], we ﬁx the measure µ on L 2 ( T d ) asa
Gaussian random ﬁeld, such that v allows the Karhunen-Loève expansion v =
P k 2 Z d ↵ k X k e k ,where | ↵ k |  exp   ` | k |  with ` > 0 , the X k ⇠ N (0 , 1) are iid Gaussian random variablesand {
e k } k
Z d is the standard Fourier basis ( SM A.5).
In this setting, we can prove the following 304 approximation result, the proof of which can be found in SM D.3.
The result can be generalized toother data distributions µ for which a convergence result for DeepONets can be proven, as in [43].
Theorem 4.6.
Assume the setting above and that of Assumption 3.3, and assume that G ( v )
C ` +1 ( T d ) for all v 2 X .
For all  > 0 there exists a constant C > 0 such that for any p 2 N thereexists a DeepONet G ✓ with p sensors and branch and trunk nets such that
[Equation]
Moreover, size( ⌧ )  Cp d +1 d , depth( ⌧ ) = 3 , size(  )  p and depth(  ) = 1 .
Nonlinear operators For nonlinear PDEs a general result like Theorem 4.6 can not be obtainedfrom the currently available tools.
Instead one needs to use Theorem 3.8 for every PDE of interest.
Inthe SM , we demonstrate this for a nonlinear ODE (gravity pendulum with external force, SM D.5)and an elliptic PDE (Darcy ﬂow, SM D.6).
Related work and discussion
This is the ﬁrst paper to rigorously expose the connections between the different deep learningframeworks from Section 2.2 for generic PDEs.
Until now, most available results focus on providinggeneric results for one speciﬁc method.
In [31] and [24] one uses neural networks that approximate solutions to a generic ODE/PDE at a ﬁxed time to construct space-time neural networks.
A generalization to PINNs is not immediate as the proof involves the emulation of the forward Eulermethod.
We have overcome this difﬁculty by constructing space-time neural networks using Taylorexpansions instead (Theorem 3.4).
To bound the approximation error of PINNs one can use thegeneric error bounds in Sobolev norms of e.g. [25, 26] for very general activation functions or themore concrete bounds [14] for tanh neural networks.
In both approaches, the only assumption is thatthe solution of the PDE has sufﬁcient Sobolev regularity.
As a consequence, these results incur thecurse of dimensionality and are not applicable to high-dimensional PDEs.
The authors of [14] analyze
PINNs based on three theoretical questions related to approximation, stability and generalization.
Other theoretical analyses of PINNs include e.g. [71, 72, 30].
For DeepONets, convergence rates foradvection-diffusion equations are presented in [17] and a clear workﬂow for obtaining generic errorestimates as well as worked out examples can be found in [43].
Similar results are obtained for FNOsin [38].
A comprehensive comparison of DeepONets and FNOs is the topic of [50].
To the best ofthe authors’ knowledge, no theoretical results for physics-informed operator learning are currentlyavailable.
A second goal of the paper is to prove that deep learning-based frameworks can overcome the curseof dimensionality (CoD).
PDEs for which the curse of dimensionality has been overcome includelinear Kolmogorov PDEs e.g. [23, 36], nonlinear parabolic PDEs [32] and elliptic PDEs [4, 10, 57].
By assuming that the initial data lies in a Barron class, the authors of [52] proved for elliptic PDEsthat the Deep Ritz Method [20] can overcome the CoD. Since the Barron class is a Banach algebra[10] it is possible that our results, which mostly only involve multiplications and additions of neuralnetworks, can be extended to Barron functions.
For PINNs, it is proven that they can overcomethe CoD for linear Kolmogorov PDEs [15].
We give an alternative proof of this result, improvethe convergence rate (Theorem 4.2) and additionally prove that PINNs can also overcome the CoDfor nonlinear parabolic PDEs (Theorem 4.4).
DeepONets and FNOs can overcome the CoD inmany cases [43, 38] but we note that this does not yet include nonlinear parabolic PDEs such as the Allen-Cahn equation.
In Theorem 4.5 we prove that dimension-independent convergence rates can beobtained if the solution is sufﬁciently regular.
Similar results are expected to hold for e.g. elliptic
PDEs by using the results from [4, 10, 57].
It is evident that the generic bounds presented here can only be obtained under suitable assumptions.
These should always be checked to prevent misleading claims about mathematical guarantees forthe considered deep learning methods.
We brieﬂy discuss how restrictive these are and whetherthey can be relaxed.
Assuming the existence of a neural network that approximates the solutionof PDE at a ﬁxed time (Assumption 3.1) is of course essential, but such a result can usually beobtained by emulating an existing numerical method.
Proving a bound on the Sobolev norm ofthat network is always possible as we only consider smooth networks.
Assumption 3.2 holds formany domains, including rectangular and smooth ones.
Assumption 3.3 and Assumption 3.5 alsohold for a very broad class of PDEs, much like the assumption on the size of the neural networkapproximation in Setting 4.1 and 4.3 holds for most functions of interest.
Therefore, the assumptionthat the PDE solution is C ( s,r ) -regular seems to be the most restrictive.
However, results like Theorem3.4 could be extended to e.g. Sobolev regular functions by using the Bramble-Hilbert lemma insteadof Taylor expansions.
Another restriction is that we exclusively focused on neural networks with thetanh activation function.
This was only for simplicity of exposition.
All results still hold for othersigmoidal activation functions, as well as more general smooth activation functions, which mightgive rise to slightly different convergence rates.
A last restriction is that the obtained rates are notoptimal, but this is not the goal of our framework.
In particular, for PINNs for low-dimensional PDEsit is beneﬁcial to use e.g. [26, 14].
Optimizing the obtained convergence rates and comparing with optimal ones is one direction forfuture research.
Previously mentioned possibilities include extending to more general activationfunctions and less regular functions.
Next, one can consider other sources of error such as theoptimization and generalization error.
We address the latter in SM B.5 by proving an a posteriorierror estimate on the generalization error.
Another direction is to make the connection between ourresults and that of [10] where they prove that Barron spaces are Banach algebras and use this to obtaindimension-independent convergence rates for PDEs with initial data in a Barron class by emulatingnumerical methods.
Finally, we only considered the approximation error in the present analysis.
Itwould be interesting to prove generic results for the total error of the trained learning architectures byalso bounding the underlying generalization error.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See Section 3 and Section 4.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions are either stated in the theorem statement or described in the text right above the theorem
statement.
(b) Did you include complete proofs of all theoretical results?
[Yes]
For each result we
mention where the proof can be found.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]