N EURAL N ETWORK A PPROXIMATIONS OF PDE S B EYOND L INEARITY : R EPRESENTATIONAL P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
A burgeoning line of research has developed deep neural networks capable of approximating the solutions to high dimensional PDEs, opening related lines of theoretical inquiry focused on explaining how it is that these models appear to evade the curse of dimensionality.
However, most theoretical analyses thus far have been limited to simple linear PDEs.
In this work, we take a step towards studying the representational power of neural networks for approximating solutions to nonlinear PDEs.
We focus on a class of PDEs known as nonlinear variational elliptic PDEs, whose solutions minimize an Euler-Lagrange energy functional E ( u ) =
(cid:82) Ω L ( ∇ u ) dx .
We show that if composing a function with Barron norm b with L produces a function of Barron norm at most B L b p , the solution to the PDE can be ϵ -approximated in the L 2 sense by a function with Barron norm O (cid:16) ( dB L ) p log(1 /ϵ ) (cid:17) .
By a classical result due to Barron (1993), this correspondingly bounds the size of a 2-layer neural network needed to approximate the solution.
Treating p, ϵ, B L as constants, this quantity is polynomial in dimension, thus showing that neural networks can evade the curse of dimensionality.
Our proof technique involves “neurally simulating” (preconditioned) gradient in an appropriate Hilbert space, which converges exponentially fast to the solution of the PDE, and such that we can bound the increase of the Barron norm at each iterate.
Our results subsume and substantially generalize analogous prior results for linear elliptic PDEs.
I NTRODUCTION
Scientific applications have become one of the new frontiers for the application of deep learning (Jumper et al., 2021; Tunyasuvunakool et al., 2021;
Sønderby et al., 2020).
PDEs are one of the fundamental modeling techniques in scientific domains, and designing neural network-aided solvers, particularly in high-dimensions, is of widespread usage in many domains (Hsieh et al., 2019; Brandstetter et al., 2022).
One of the most common approaches for applying neural networks to solve PDEs is to parameterize the solution as a neural network and minimize a loss which characterizes the solution (Sirignano & Spiliopoulos, 2018; E & Yu, 2017).
The hope in doing so is to have a method which computationally avoids the “curse of dimensionality”—i.e.
, that scales less than exponentially with the ambient dimension.
To date, neither theoretical analysis nor empirical applications have yielded a precise characterization of the range of PDEs for which neural network-aided methods outperform classical methods.
Active research on the empirical side (Han et al., 2018; E et al., 2017; Li et al., 2020a;b) has explored several families of PDEs, e.g., Hamilton-Bellman-Jacobi and Black-Scholes, where neural networks have been demonstrated to outperform classical grid-based methods.
On the theory side, a recent line of works (Marwah et al., 2021; Chen et al., 2021; 2022) has considered the following fundamental question:
For what families of PDEs can the solution be represented by a small neural network?
The motivation for this question is computational: since the computational complexity of fitting a neural network (by minimizing some objective) will grow with its size.
Specifically, these works focus on understanding when the approximating neural network can be sub-exponential in size, thus avoiding the curse of dimensionality.
Unfortunately, the techniques introduced in this line of work have so far only been applicable to linear PDEs.
In this paper, we take the first step beyond linear PDEs, with a particular focus on nonlinear variational elliptic PDEs .
These equations have the form − div( ∇ L ( ∇ u ))
=
and are instances of nonlinear Euler-Lagrange equations.
Equivalently, u is the minimizer of the energy functional E ( u ) =
(cid:82) Ω L ( ∇ u ) dx .
This paradigm is very generic: its origins are in Lagrangian formulations of classical mechanics, and for different L , a variety of variational problems can be modeled or learned (Schmidt & Lipson, 2009;
Cranmer et al., 2020).
These PDEs have a variety of applications in scientific domains, e.g., (non-Newtonian) fluid dynamics (Koleva & Vulkov, 2018), meteorology(Weller et al., 2016), and nonlinear diffusion equations (Burgers, 2013).
Our main result is to show that when the function L has “low complexity”, so does the solution .
The notion of complexity we work with is the Barron norm of the function, similar to Chen et al.
(2021); Lee et al.
This is a frequently used notion of complexity, as a function with small
Barron norm can be represented by a small, two-layer neural network, due to a classical result (Barron, 1993).
Mathematically, our proof techniques are based on “neurally unfolding” an iterative preconditioned gradient descent in an appropriate function space: namely, we show that each of the iterates can be represented by a neural network with Barron norm not much worse than the Barron norm of the previous iterate—along with showing a bound on the number of required steps.
Importantly, our results go beyond the typical non-parametric bounds on the size of an approximator network that can be easily shown by classical regularity results of the solution to the nonlinear variational PDEs (De Giorgi, 1957; Nash, 1957; 1958) along with universal approximation results (Yarotsky, 2017).
O VERVIEW OF R ESULTS
Let Ω ⊂ R d be a bounded open set with 0 ∈ Ω and ∂ Ω denote the boundary of Ω .
Furthermore, we assume that the domain Ω is such that the Poincare constant C p is greater than 1 (see Theorem 2 for the exact definition of the Poincare constant).
We first define the energy functional whose minimizers are represented by a nonlinear variational elliptic PDE—i.e., the Euler-Lagrange equation of the energy functional.
Definition 1 (Energy functional) .
For all u : Ω → R such that u | ∂ Ω = 0 , we consider an energy functional of the following form:
[Equation]
where L : R d → R is a smooth and uniformly convex function , i.e., there exists constant 0 < λ ≤ Λ such that for all x ∈ R we have λI d ≤ D 2 L ( x ) ≤ Λ I d .
Further, without loss of generality 1 , we assume that λ ≤ 1 /C p .
Note that due to the convexity of the function L , the minimizer u ⋆ exists and is unique.
The proof of existence and uniqueness is standard (e.g., Theorem 3.3 in Fern´andez-Real & Ros-Oton (2020)).
Writing down the condition for stationarity, we can derive a (nonlinear) elliptic PDE for the minimizer of the energy functional in Definition 1 .
Lemma 1.
Let u ⋆ : Ω → R be the unique minimizer for the energy functional in Definition 1.
Then for all φ ∈ H 10 (Ω) the minimizer u ⋆ satisfies the following condition,
[Equation]
where d E [ u ]( φ ) denotes the dirctional derivative of the energy functional calculated at u in the direction of φ .
Thus, the minimizer u ⋆ of the energy functional satisfies the following PDE:
[Equation]
The proof for the Lemma can be found in Appendix Section A.1.
Here, − div( ∇ L ( ∇· ) is a functional operator that acts on a function (in this case u ).
Our goal is to determine if the solution to the PDE in Equation 3 can be expressed by a neural network with a small number of parameters.
In order do so, we utilize the concept of a Barron norm , which measures the complexity of a function in terms of its Fourier representation.
We show that if composing with the function L is such that it increases it has a bounded increase in the Barron norm of u , then the solution to the PDE in Equation 3 will have a bounded Barron norm.
The motivation for using this norm is a seminal paper (Barron, 1993), which established that any function with Barron norm C can be ϵ -approximated in the L 2 sense by a two-layer neural network with size
O ( C 2 /ϵ ) , thus evading the curse of dimensionality if C is substantially smaller than exponential in d .
Informally, we will show the following result:
Theorem 1 (Informal) .
Let L be convex and smooth, such that composing a function with Barron norm b with L produces a function of Barron norm at most B L b p .
Then, for all sufficiently small ϵ > 0 , the minimizer of the energy functional in Definition 1 can be ϵ -approximated in the L 2 sense by a function with Barron norm O (cid:16) ( dB L ) p log(1 /ϵ )
(cid:17) .
As a consequence, when ϵ, p are thought of as constants, we can represent the solution to the EulerLagrange PDE Equation 3 by a polynomially-sized network, as opposed to an exponentially sized network, which is what we would get by standard universal approximation results and using regularity results for the solutions of the PDE.
We establish this by “neurally simulating” a preconditioned gradient descent (for a strongly-convex loss) in an appropriate Hilbert space, and show that the Barron norm of each iterate—which is a function—is finite, and at most polynomially bigger than the Barron norm of the previous iterate.
We get the final bound by (i) bounding the growth of the Barron norm at every iteration; and (ii) bounding the number of iterations required to reach an ϵ -approximation to the solution.
The result in formally stated in Section
R ELATED W ORK
Over the past few years, a growing line of work has focused on parameterizing the solutions to PDEs with neural networks.
Works such as E et al.
; E & Yu (2017); Sirignano & Spiliopoulos (2018); Raissi et al.
(2017) achieved impressive results on a variety of different applications and demonstrated the empirical efficacy of neural networks in solving high-dimensional PDEs, even outperforming previously dominant numerical approaches like finite differences and finite element methods (LeVeque, 2007) that proceed by discretizing the input space, hence limiting their use to problems on low-dimensional input spaces.
Several recent works look to theoretically analyze these neural network based approaches for solving
PDEs.
Mishra & Molinaro (2020) look at the generalization properties of physics informed neural networks.
In Lu et al.
(2021) show the generalization analysis for the Deep Ritz method for elliptic equations like the Poisson equation and Lu & Lu (2021) extends their analysis to the Schr¨odinger eigenvalue problem.
In addition to analyzing the generalization capabilities of the neural networks, theoretical analysis into their representational capabilities has also gained a lot of attention.
Khoo et al.
(2021) show the existence of a network by discretizing the input space into a mesh and then using convolutional NNs, where the size of the layers is exponential in the input dimension.
Sirignano & Spiliopoulos (2018) provide a universal approximation result, showing that for sufficiently regularized PDEs, there exists a multilayer network that approximates its solution.
In Jentzen et al.
Grohs & Herrmann (2020); Hutzenthaler et al.
(2020) provided a better-than-exponential dependence on the input dimension for some special parabolic PDEs, based on a stochastic representation using the Feynman-Kac Lemma, thus limiting the applicability of their approach to PDEs that have such a probabilistic interpretation.
Moreover, their results avoid the curse of dimensionality only over domains with unit volume
A recent line of work has focused on families of PDEs for which neural networks evade the curse of dimensionality—i.e., the solution can be approximated by a neural network with a subexponential size.
Marwah et al.
(2021) show that for elliptic PDE’s whose coefficients are approximable by neural networks with at most N parameters, a neural network exists that ϵ -approximates the solution and has size O ( d log(1 /ϵ ) N ) .
Chen et al.
(2021) extends this analysis to elliptic PDEs with coefficients with small Barron norm, and shows that if the coefficients have Barron norm bounded by B , an ϵ -approximate solution exists with Barron norm at most O ( d log(1 /ϵ ) B ) .
The work by Chen et al.
(2022) derives related results for the Schr¨odinger equation.
As mentioned, while most previous works show key regularity results for neural network approximations of solution to PDEs, most of their analysis is limited to simple linear PDEs.
The focus of this paper is towards extending these results to a family of PDEs referred to as nonlinear variational
PDEs.
This particular family of PDEs consists of many famous PDEs, such as p − Laplacian (on a bounded domain), which is used to model phenomena like non-Newtonian fluid dynamics and nonlinear diffusion processes.
The regularity results for these family of PDEs was posed as Hilbert’s XIX th problem.
We note that there are classical results like De Giorgi (1957) and Nash (1957; 1958) that provide regularity estimates on the solutions of a nonlinear variational elliptic PDE of the form in Equation 3.
One can easily use these regularity estimates, along with standard universal approximation results (Yarotsky, 2017) to show that the solutions can be approximated arbitrarily well.
However, the size of the resulting networks will be exponentially large (i.e. suffer from the curse of dimensionality)—so are of no use for our desired results.
N OTATION AND D EFINITION
In this section, we introduce some key concepts and notation that will be used throughout the paper.
For a vector x ∈ R d , we use ∥ x ∥ 2 to denote its ℓ 2 norm.
Further, C ∞ (Ω) denotes the set of functions f : Ω → R that are infinitely differentiable.
We also define some important function spaces and associated key results below.
Definition 2.
For a vector valued function g : R → R d , we define the L p (Ω) norm for p ∈ [1 , ∞ ) as
[Equation]
For p = ∞ , we have
[Equation]
where ∥ g i ∥
L ∞ (Ω) =
inf { c ≥ 0 : | g ( x ) | ≤ c for almost all x ∈ Ω } .
Definition 3.
For a domain Ω , the space of functions H 10 (Ω) is defined as
[Equation]
The corresponding norm for H 10 (Ω) is defined as ∥ g
∥ H 10 (Ω) = ∥∇ g
∥
L 2 (Ω) .
We will make use of the Poincar´e inequality throughout several of our results.
Theorem 2 (Poincar´e inequality, Poincar´e (1890)) .
For Ω ⊂ R d which is open and bounded, there exists a constant C p > 0 such that for all u ∈ H 10 (Ω)
[Equation]
This constant can be very benignly behaved with dimension for many natural domains—even dimension independent.
Examples include convex domains (Payne & Weinberger, 1960).
4. B ARRON N ORMS
For a function f : R d → R , the Fourier transform and the inverse Fourier transform are defined as
[Equation]
and f ( x ) = (cid:90) R d ˆ f ( ω ) e ix T ω dω.
[Equation]
The Barron norm is an average of the norm of the frequency vector weighted by the Fourier magnitude | ˆ f ( ω ) | .
A slight technical issue is that the the Fourier transform is defined only for f : R d → R .
Since we are interested in defining the Barron norm of functions defined over a bounded domain, we allow for arbitrary extensions of a function outside of their domain.
(This is the standard definition, e.g. in (Barron, 1993).)
Definition 4.
We define F be the set of functions g ∈ L 1 (Ω) such that the Fourier inversion formula g
= (
[Equation] d ˆˆ f ( x ) holds over the domain Ω , i.e.,
[Equation]
Definition 5 (Spectral Barron Norm, (Barron, 1993)) .
Let Γ be a set of functions defined over Ω such that their extension over R d belong to F , that is,
[Equation]
Then we define the spectral Barron norm ∥ · ∥ B (Ω) as
[Equation]
The Barron norm is an L 1 relaxation of requiring sparsity in the Fourier basis—which is intuitively why it confers representational benefits in terms of the size of a neural network required.
We refer to Barron (1993) for a more exhaustive list of the Barron norms of some common function classes.
The main theorem from Barron (1993) formalizes this intuition, by bounding the size of a 2-layer network approximating a function with small Barron norm:
Theorem 3 (Theorem 1, Barron (1993)) .
Let f ∈ Γ such that ∥ f ∥ B (Ω) ≤ C and µ be a probability measure defined over Ω .
There exists a i ∈ R d , b i ∈ R and c
i ∈ R such that (cid:80) ki =
| c i | ≤ 2 C , there exists a function f k ( x )
= (cid:80) ki =1 c i σ (cid:0) a Ti x + b (cid:1) , such that we have,
[Equation]
Here σ denotes a sigmoidal activation function, i.e., lim x →∞ σ ( x ) = 1 and lim x →−∞ σ ( x ) = 0 .
Note that while Theorem 3 is stated for sigmoidal activations like sigmoid and tanh (after appropriate rescaling), the results are also valid for ReLU activation functions, since ReLU ( x ) − ReLU ( x − 1) is in fact sigmoidal.
We will also need to work with functions that do not have Fourier coefficients beyond some size (i.e. are band limited), hence we introduce the following definition:
Definition 6.
Let F W (Ω) be the set of functions whose Fourier coefficients vanish outside a bounded ball, i.e.,
[Equation]
Similarly, we denote
[Equation]
Since we will work with vector valued function, we will also define the Barron norm of a vectorvalued function as the maximum of the Barron norms of its coordinates:
Definition 7.
For a vector valued function g : Ω → R d , we define ∥ g ∥ B (Ω) = max i ∥
g i ∥ B (Ω) .
M AIN R ESULT
Before stating the main result we introduce the key assumption.
Assumption 1.
The function L in Definition 1 can be approximated by a function ˜
L : R d → R such that there exists a constant ϵ L ∈ [0 , λ ) with sup x ∈ R d ∥∇ L ( x ) − ∇ ˜ L ( x ) ∥ 2 ≤ ϵ L ∥ x
∥ 2 .
Furthermore, we assume that ˜ L is such that for any g ∈ H 10 (Ω) , we have ˜ L ◦ g ∈ H 10 (Ω) ,
˜ L ◦ g ∈ F and
[Equation]
[Equation]
for some constants B ˜ L ≥ 0 , and p ≥ 0 .
Furthermore, if g ∈ Γ W then ˜ L ◦ g ∈ Γ kW for a k > 0 .
This assumption is fairly natural: it states that the function L can be approximated (up to ϵ L , in the sense of the gradients of the functions) by a function ˜
L that has the property that when applied to a function g with small Barron norm, the new Barron norm is not much bigger.
The constant p specifies the order of this growth.
The functions for which our results are most interesting are when the dependence of B ˜ L on d is at most polynomial—so that the final size of the approximating network does not exhibit curse of dimensionality.
For instance, we can take L to be a multivariate polynomial of degree up to P : we show in Lemma 7 the constant B ˜ L is O ( d P ) (intuitively, this dependence comes from the total number of monomials of this degree), whereas p and k are both
O ( P ) .
With all the assumptions stated, we now state our main theorem:
Theorem 4 (Main Result) .
Consider the nonlinear variational elliptic PDE in Equation 3 which satisfies Assumption 1 and let u ⋆ ∈ H 10 (Ω) denote the solution to the PDE.
If u 0 ∈ H 10 (Ω) is a function such that u 0 ∈ Γ W 0 , then for all sufficiently small ϵ > 0 , and
[Equation]
, there exists a function u T ∈ H 10 (Ω) such that u T ∈ Γ k T W 0 and Barron norm bounded as
[Equation]
Furthermore, u T satisfies ∥ u T − u ⋆ ∥ H 10 (Ω) ≤ ϵ + ˜ ϵ where
[Equation]
∥ u ⋆ ∥ H 10 (Ω) + 1 λ E ( u 0 ) (cid:17) Λ + ϵ L (cid:32)(cid:18) {+
+}
+ λ( C p + 1)Λ 3 (Λ + ϵ L ) (cid:19) T −(cid:33) .
Remark 1: The function u 0 can be seen as an initial estimate of the solution, that can be refined to an estimate u T , which is progressively better at the expense of a larger Barron norm.
A trivial choice could be u 0 = 0 , which has Barron norm 1, and which by Lemma 2 would satisfy E ( u 0 ) − E ( u ∗ )
≤ Λ ∥ u ∗ ∥ 2 H 10 (Ω) .
Remark 2: The final approximation error has two terms, T goes to ∞ as ϵ tends 0 and is a consequence of the way u T is constructed—by simulating a functional (preconditioned) gradient descent which converges to the solution to the PDE.
The error term ˜ ϵ stems from the approximation that we make between ˜ L and L , which grows as T increases—it is a consequence of the fact that the gradient descent updates with ˜ L and L progressively drift apart as T tends to ∞ .
Remark 3: As in the informal theorem, if we think of p, Λ , λ, C p , k, ∥ u 0 ∥
B (Ω) as constants, the theorem implies u ∗ can be ϵ -approximated in the L 2 sense by a function with Barron norm
O (cid:16) ( dB L ) p log(1 /ϵ ) (cid:17) .
Therefore, combining results from Theorem 4 and Theorem 3 the total parameters required to ϵ -approximate u ∗ by a 2-layer neural network is O (cid:16) 1 ϵ 2 ( dB L ) 2 p log(1 /ϵ ) (cid:17) .
Remark 4: We further note that this result recovers (and generalizes) prior results which bound the Barron norm of linear elliptic PDEs like Chen et al.
In these results, the elliptic PDE takes the form − div( A ∇ u ) and A is assumed to have bounded Barron norm.
Thus, ∥ L ◦ u ∥ B (Ω) ≤ d 2 ∥
A ∥ B (Ω) ∥ u ∥ B (Ω) , hence satisfying Equation 5 in Assumption 1 with p = 1 .
P ROOF OF M AIN R ESULT
The proof will proceed by “neurally unfolding” a preconditioned gradient descent on the objective
E in the Hilbert space H 10 (Ω) .
This is inspired by previous works by Marwah et al.
; Chen et al.
(2021) where the authors show that for a linear elliptic PDE, an objective which is quadratic can be designed.
In our case, we show that E is “strongly convex” in some suitable sense—thus again, bounding the amount of steps needed.
More precisely, the result will proceed in two parts:
First, we will show that the sequence of functions { u t } ∞ t =0 , where u t +1 ← u t − η
( I − ∆) − 1 d E ( u t ) can be interpreted as performing preconditioned gradient descent, with the (constant) preconditioner ( I − ∆) − 1 .
We show that in some appropriate sense (Lemma 2),
E is strongly convex in H 10 (Ω) —thus the updates converge at a rate of O (log(1 /ϵ )) .
We then show that the Barron norm of each iterate u t +1 can be bounded in terms of the
Barron norm of the prior iterate u t .
We show this in Lemma 5, where we show that given
Assumptions1, the ∥ u t +1 ∥ B (Ω) is O ( d ∥ u t ∥ p B (Ω) ) .
By unrolling this recursion we show that the Barron norm of the ϵ -approximation of u ⋆ is of the order O
( d p T ∥ u 0 ∥
p B (Ω) ) , where T are the total steps required for ϵ -approximation and ∥ u 0 ∥
B (Ω) is the Barron norm of the first function in the iterative updates.
We now proceed to delineate the main technical ingredients for both of these parts.
6. C ONVERGENCE R ATE OF S EQUENCE
The proof to show the convergence to the solution u ⋆ is based on adapting the standard proof (in finite dimension) for convergence of gradient descent when minimizing a strongly convex function f .
Recall, the basic idea is to Taylor expand f ( x + δ )
≈ f
( x ) + ∇ f ( x ) T δ + O ( ∥ δ ∥ 2 ) .
Taking δ =
η ∇ f ( x ) , we lower bound the progress term η ∥∇ f ( x ) ∥ 2 using the convexity of f , and upper bound the second-order term η 2 ∥∇ f ( x ) ∥ 2 using the smoothness of f .
We follow analogous steps, and prove that we can lower bound the progress term by using some appropriate sense of convexity of E , and upper bound some appropriate sense of smoothness of E , when considered as a function over H 10 (Ω) .
Precisely, we show:
Lemma 2 (Strong convexity of E in H 10 ) .
If E , L are as in Definition 1, we have
∀ u, v ∈ H 10 (Ω) : ⟨ D E ( u ) , v ⟩ L 2 (Ω) =
(cid:82) Ω − div( ∇ L ( ∇ u )) vdx
= (cid:82) Ω ∇ L ( ∇ u ) · ∇ vdx.
∀ u, v ∈ H 10 (Ω) : λ ∥
u − v ∥ H 10 (Ω) ≤ ⟨ D E ( u ) − D E ( v ) , u − v ⟩ L 2 (Ω) ≤ Λ ∥ u − v ∥ H 10 (Ω) .
∀ u, v ∈ H 10 (Ω) : λ 2 ∥∇ v ∥ 2 L 2 (Ω) + ⟨ D E ( u ) , v ⟩ L 2 (Ω) ≤ E ( u + v ) −E ( u ) ≤ ⟨ D E ( u ) , v ⟩ L 2 (Ω) + Λ2 ∥∇ v ∥ 2 L 2 (Ω) .
∀ u ∈ H 10 (Ω) :
λ 2 ∥ u − u ⋆ ∥ 2 H 10 (Ω) ≤ E ( u ) − E ( u ⋆ ) ≤ Λ2 ∥
u − u ⋆ ∥ 2 H 10 (Ω) .
Part 1 is a helpful way to rewrite an inner product of a “direction” v with D E ( u ) —it is essentially a consequence of integration by parts and the Dirichlet boundary condition.
Part 2 and 3 are common proxies of convexity: they are ways of formalizing the notion that E is strongly convex, when viewed as a function over H 10 (Ω) .
Finally, part 4 is a consequence of strong convexity, capturing the fact that if the value of E ( u ) is suboptimal, u must be (quantitatively) far from u ∗ .
The proof of the
Lemma can be found in the Appendix (Section B.1)
When analyzing gradient descent (in finite dimensions) to minimize a loss function E , the standard condition for progress is that the inner product of the gradient with the direction towards the optimum is lower bounded as ⟨ D E ( u ) , u ∗ − u ⟩ L 2 (Ω) ≥ α ∥
u − u ∗ ∥
2 L 2 (Ω) .
From Parts 2 and 3, one can readily see that the above condition is only satisfied “with the wrong norm”: i.e. we only have ⟨ D E ( u ) , u ∗ − u ⟩ L 2 (Ω) ≥ α ∥
u − u ∗ ∥
2 H 10 (Ω) .
We can fix this mismatch by instead doing preconditioned gradient, using the fixed preconditioner ( I − ∆) − 1 .
Towards that, the main lemma about the preconditioning we require is the following one:
Lemma 3 (Norms with preconditioning) .
For all v ∈ H 10 (Ω) , we have
∥ ( I − ∆) − 1 ∇ · ∇ u ∥
L 2 (Ω) =
∥ ( I − ∆) − 1 ∆ u
∥
L 2 (Ω) ≤ ∥ u
∥ L 2 (Ω) .
⟨ ( I − ∆) − 1 v, v ⟩ L 2 (Ω) ≥ 1 1+ C p ⟨ ∆ − 1 v, v ⟩ L 2 (Ω) .
The first part of the lemma is a relatively simple consequence of the fact that the operators ∆ and ∇ “commute”, and therefore can be re-ordered.
The latter lemma can be understood intuitively as ( I − ∆) − 1 and ∆ − 1 act as similar operators on eigenfunctions of ∆ with large eigenvalues (the extra I does not do much)—and are only different for eigenfunctions for small eigenvalues.
However, since the smallest eigenvalues is lower bounded by 1 /C p , their gap can be bounded.
Next we utilize the results in Lemma 2 and Lemma 3 to show preconditioned gradient descent exponentially converges to the solution to the nonlinear variational elliptic PDE in 3.
Lemma 4 (Preconditioned Gradient Descent Convergence) .
Let u ⋆ denote the unique solution to the PDE in Definition 3.
For all t ∈ N , we define the sequence of functions
[Equation]
If u 0 ∈ H 10 (Ω) after t iterations we have,
[Equation]
The complete proof for convergence can be found in Section B.3 of the Appendix.
Therefore, using the result from Lemma 4, i.e., ∥ u t − u ⋆ ∥ 2 H 10 (Ω) ≤ 2 λ ( E ( u t ) − E ( u ⋆ )) , we have
[Equation]
and ∥ u T − u ⋆ ∥ 2 H 10 (Ω) ≤ ϵ after T steps, where,
[Equation]
6. B OUNDING THE B ARRON N ORM
Having obtained a sequence of functions that converge to the solution u ⋆ , we bound the Barron norms of the iterates.
We draw inspiration from Marwah et al.
; Lu et al.
and show that the Barron norm of each iterate in the sequence has a bounded increase on the Barron norm of the previous iterate.
Note that in general, the Fourier spectrum of a composition of functions can not easily be expressed in terms of the Fourier spectrum of the functions being composed.
However, from Assumption 1, we know that the function L can be approximated by ˜ L such that ˜ L ◦ g has a bounded increase the Barron norm of g .
Thus, instead of tracking the iterates in Equation 6, we track the Barron norm of the functions in the following sequence,
[Equation]
We can derive the following result (the proof is deferred to Section D.1 of the Appendix):
Lemma 5.
Consider the updates in Equation 8, if ˜ u t ∈ Γ W t then for all η ∈ (0 ,
[Equation]
)Λ 3 ] we have ˜ u t +1 ∈ Γ kW t and the Barron norm of ˜ u t +1 can be bounded as
[Equation]
The proof consists of using the result in Equation 5 about the Barron norm of composition of a function with ˜ L , as well as counting the increase in the Barron norm of a function by any basic algebraic operation, as established in Lemma 6.
Precisely we show:
Lemma 6 (Barron norm algebra) .
If h, h 1 , h 2 ∈ Γ , then the following set of results hold,
• Addition: ∥ h 1 + h 2 ∥ B (Ω) ≤ ∥
h 1 ∥ B (Ω) + ∥
h 2 ∥ B (Ω) .
• Multiplication: ∥
h 1 · h 2 ∥ B (Ω) ≤ ∥ h 1 ∥ B (Ω) ∥
h 2 ∥ B (Ω) • Derivative: if
• Preconditioning: if h ∈ Γ , then ∥ ( I − ∆) − 1 h
∥ B (Ω) ≤ ∥ h ∥ B (Ω) .
The proof for the above Lemma can be found in Appendix D.3.
It bears similarity to an analogous result in Chen et al.
(2021), with the difference being that our bounds are defined in the spectral
Barron space which is different from the definition of the Barron norm used in Chen et al.
Expanding on the recurrence in Lemma 6 we therefore have that after T we have, we have W T ≤ k T W 0 and hence u t +1 ∈ Γ k t W 0 , and the Barron norm of u T can be bounded as follows,
[Equation]
Finally, we exhibit a natural class of functions that satisfy the main Barron growth property in
Equations 5.
Precisely, we show (multivariate) polynomials of bounded degree have an effective bound on p and B L :
Lemma 7.
Let f ( x ) = (cid:80) α [Equation] | α |≤ P
[Equation] A α [Equation] di
=1 x α i
i (cid:17) where α is a multi-index and x ∈ R d .
If g :
R d → R d is such that g ∈ Γ W , then we have f ◦ g ∈ Γ PW and the Barron norm can be bounded as,
[Equation]
Hence if ˜ L is a polynomial of degree P the constants in Assumption 1 will take the following values
[Equation]
Finally, since we are using an approximation of the function L we will incur an error at each step of the iteration.
The following Lemma shows that the error between the iterates u t and the approximate iterates ˜
u t increases with t .
The error is calculated by recursively tracking the error between u t and ˜ u t for each t in terms of the error at t − 1 .
Note that this error can be controlled by using smaller values of η .
Lemma 8.
Let ˜ L : R d → R be the function satisfying the properties in Assumption
such that sup x ∈ R d ∥∇ L ( x ) − ∇ ˜ L ( x ) ∥ 2 ≤ ϵ L ∥ x
∥ 2 and we have E ( u ) = (cid:82) Ω L ( ∇ u ) dx and ˜ E ( u ) = (cid:82) Ω ˜ L ( ∇ u ) dx . For η ∈ (0 , λ [Equation]
p +1)Λ 3 ] consider the sequences, [Equation] , and ˜ u t +
[Equation]
) then for all t ∈ N and R ≤ ∥ u ⋆ ∥ H 10 (Ω) + 1 λ E ( u 0 ) we have,
[Equation]
C ONCLUSION AND F UTURE W ORK
In this work, we take a representational complexity perspective on neural networks, as they are used to approximate solutions of nonlinear variational elliptic PDEs of the form − div( ∇ L ( ∇ u ))
= 0 .
We prove that if L is such that composing L with function of bounded Barron norm increases the
Barron norm in a bounded fashion, then we can bound the Barron norm of the solution u ⋆ to the
PDE—potentially evading the curse of dimensionality depending on the rate of this increase.
Our results subsume and vastly generalize prior work on the linear case (Marwah et al., 2021; Chen et al., 2021).
Our proof consists of neurally simulating preconditioned gradient descent on the energy function defining the PDE, which we prove is strongly convex in an appropriate sense.
There are many potential avenues for future work.
Our techniques (and prior techniques) strongly rely on the existence of a variational principle characterizing the solution of the PDE.
In classical
PDE literature, these classes of PDEs are also considered better behaved: e.g., proving regularity bounds is much easier for such PDEs (Fern´andez-Real & Ros-Oton, 2020).
There are many nonlinear PDEs that come without a variational formulation—e.g. the Monge-Ampere equation—for which regularity estimates are derived using non-constructive methods like comparison principles.
It is a wide-open question to construct representational bounds for any interesting family of PDEs of this kind.
It is also a very interesting question to explore other notions of complexity—e.g. number of parameters in a (potentially deep) network like in Marwah et al.
(2021), Rademacher complexity, among others.
A A PPENDIX A. P ROOF FOR L EMMA
The proof follows form Fern´andez-Real & Ros-Oton (2020) Chapter 3.
We are provided it here for completeness.
Proof of Lemma 1.
If the function u ⋆ minimizes the energy functional in Definition 1 then we have for all ϵ ∈ R
[Equation]
where φ ∈ C ∞ c (Ω) .
That is, we have a minima at ϵ = 0 and taking a derivative w.r.t ϵ we get,
[Equation]
where for all x ∈ Ω we have | r ( x ) | ≤ |∇ φ | 2 sup p ∈ R d D 2 L ( p ) .
Since ϵ → 0 the final derivative is of the form,
[Equation]
For functions r, s ∈ H 10 (Ω) note the following Green’s formula,
[Equation]
Using the identity in Equation 11 in Equation 10 we get,
[Equation]
That is the minima for the energy functional is reached at a u which solves the following PDE,
[Equation]
∇ L ( ∇
[Equation]
) = 0 .
B P ROOFS FROM S ECTION 6. B. P ROOF FOR L EMMA
Proof.
In order to prove part 1, we will use the following integration by parts identity, for functions r : Ω → R such that and s : Ω → R , and r, s ∈ H 10 (Ω) ,
[Equation]
[Equation]
where n i is a normal at the boundary and d Γ is an infinitesimal element of the boundary ∂ Ω .
Using the formula in Equation 12 for functions u, v ∈ H 10 (Ω) , we have
[Equation]
where in the last equality we use the fact that the function v ∈ H 10 (Ω) , thus v ( x ) = 0 , ∀ x ∈ ∂ Ω .
To show the second part, first note since L : R d → R is strongly convex and smooth, we have
[Equation]
This implies
[Equation]
Taking square on each side and itegrating over Ω we get
[Equation]
On the other hand, from part 1, we have
[Equation]
Hence, by Cauchy-Schwartz, we get
[Equation]
which proves the right hand side of the inequality in part 2.
For the left size of the inequality, by convexity of L we have ∀ x, y ∈ R d ,
( ∇ L ( x ) − ∇ L ( y )) T ( x − y ) ≥ λ ∥ x − y ∥ 22 is convex,
[Equation]
Integrating over Ω we get,
[Equation]
Using part 1 again, this implies Equation
[Equation]
as we wanted.
To show part 3, we first Taylor expand L to rewrite the energy function as:
[Equation]
) where ˜ x ∈ R d (and is potentially different for every x ∈ Ω ).
Since the function L is strongly convex we have
[Equation]
Plugging in these bounds in Equation
[Equation]
we have:
[Equation]
as well as
[Equation]
Combining Equation 17 and Equation 18 we get,
[Equation]
Finally
[Equation]
part 4 follows by plugging in u = u ⋆ and v = u − u
⋆ in part 3 and using the fact that
D E ( u ⋆ ) = 0 .
B. P ROOF FOR L EMMA
Proof.
Let { λ i , ϕ i } ∞
i
=1 denote the (eigenvalue, eigenfunction) pairs of the operator − ∆ where 0 < λ 1 ≤ λ 2 ≤ · · · , which are real and countable.
( Evans (2010), Theorem 1, Section 6.5)
Using the definition of eigenvalues and eigenfunctions, we have
[Equation]
where in the last equality we use Theorem 2.
Let us write the functions v, w in the eigenbasis as v =
(cid:80) i µ i ϕ i .
Notice that an eigenfunction of − ∆ is also an eigenfunction for ( I − ∆) − 1 , with correspondinding eigenvalue 1
+ λ i .
Thus, to show part 1, we have,
[Equation]
where in the last equality we use the fact that ϕ i are orthogonal.
Now, note that ( I − ∆) − 1 v = (cid:80) ∞
i = 1 (1+ λ i ) µ i ϕ i now, note that since λ 1 ≤ λ 2 ≤ · · · and x 1+ x is monotonically increasing, we have for all i ∈ N
[Equation]
+ λ i
[Equation]
(19) and note that 1 λ i are the eigenvalues for ( − ∆) − 1 for all
i ∈ N .
Now, bounding ⟨ ( I − ∆) − 1 v, v ⟩ L 2 (Ω)
[Equation]
where we the orthogonality of ϕ ′
i s to get Equation 20.
Using the inequality in Equation 19 we can further lower bound ⟨ ( I − ∆) − 1 v, v ⟩ L 2 (Ω) as follows,
[Equation]
where we use the following set of equalities in the last step,
[Equation]
B. P ROOF FOR C ONVERGENCE : P ROOF FOR L EMMA
Proof.
For the analysis we consider η =
[Equation]
Taylor expanding as in Equation
[Equation]
we have
[Equation] [Equation]
where we have in Equation 17 plugged in u t +1 − u t = − η
( I − ∆)
− 1 D E ( u t ) .
First we lower bound Term 1.
Since u ⋆ is the solution to the PDE in Equation 3, we have D E ( u ⋆ ) = 0 .
Therefore we have
[Equation]
Similarly, since u ⋆ is the solution to the PDE in Equation 3 Equation 2 we have for all φ ∈ H 10 (Ω) ⟨∇ L ( ∇ u ⋆ ) , ∇ φ ⟩ L
[Equation]
Ω)
= 0 .
Using this Equation
we get,
[Equation]
Using Equation
[Equation]
we can rewrite Term 1 as
[Equation]
where in step ( i ) we use −∇ · ∇ v = − ∆ v and the fact that ∇ commutes with ( I − ∆) − 1 ( − ∆) .
Plugging part 2 of Lemma 3 in Equation
[Equation]
we get
[Equation]
where ( i ) follows by part 2 of Lemma 2 and we use part 4 of Lemma 2 for the last inequality
[Equation]
We will proceed to upper bounding Term 2 .
Using part 1 of Lemma 3 we have
[Equation]
Here, we use the fact that −∇ · ∇ = − ∆ in step ( i ) and the fact that ∇ commutes with ( I − ∆) − 1 ( − ∆) in step ( ii ) , and finally we use the result from part 2 of Lemma 3 to get the inequality in ( iii ) .
Combining Equation 25 and Equation 26 in Equation 21 we get
[Equation]
Since η = λ 3 / ((1 + C p )Λ 3 ) we have
[Equation]
C E RROR A NALYSIS
First, we will need the following simple technical lemma showing that the H 10 (Ω) norm is self-dual:
Lemma 9.
The dual norm of ∥ · ∥ H 10 (Ω) is ∥ · ∥ H 10 (Ω) .
Proof.
If ∥ u ∥ ∗ denotes the dual norm of ∥ u ∥ H 10 (Ω) , by definition we have,
[Equation]
where the inequality follows by Cauchy- Schwarz.
On the other hand, equality can be achieved by taking v = u ∥∇
[Equation]
∥ 2 .
Thus, ∥ u ∥ ∗ = ∥∇ u
∥
L 2 (Ω) =
∥
u ∥ H 10 (Ω) as we wanted.
With this we can prooceed to the proof of Lemma 8.
C. P ROOF FOR L EMMA
Proof.
We define for all t r t = ˜
u t − u t , and will iteratively bound ∥ r t ∥
L 2 (Ω) .
Starting with u 0 = 0 and ˜ u t = 0 , we define the iterative sequences as,
[Equation]
where η ∈ (0 , λ 3 (1+ C p )Λ 3 ] .
Subtracting the two we get,
[Equation] [Equation]
Taking H 10 (Ω) norm on both sides we get,
[Equation]
Towards bounding (cid:13)(cid:13)(cid:13) ( I − ∆)
[Equation]
D
[Equation]
E ( u t + r t )
[Equation]
D E
[Equation]
u t ) (cid:13)(cid:13)(cid:13)
H 10 (Ω) , from Lemma 9 we know that the dual norm of ∥ w ∥ H 10 (Ω) is ∥ w ∥ H 10 (Ω) , thus,
[Equation]
Using the Lipschitzness of ∇ L
[Equation]
we have
[Equation]
Squaring and integrating over Ω on both sides we get
[Equation]
Pluggin in Equation 31 in Equation 29 we get,
[Equation]
Furthermore, from Lemma 4 we have for all t ∈ N ,
[Equation]
and
[Equation]
Hence we have that for all t ∈ N ,
[Equation]
Putting this all together, we have
[Equation]
Hence using the result from Equation 33 in Equation 28 to get,
[Equation]
[Equation]
where we use the fact that ∥ r t ∥ H 10 (Ω) = 0 .
Notice that ϵ L << Λ .
Further, we have η ∈ (0 , λ 3 ( C p +1)Λ 3 ] that is η ≤ 1 , it implies that η (Λ + ϵ L ) < 1 .
Hence we can further bound Equation 34 as follows,
[Equation]
D P ROOFS FOR B ARRON N ORM A PPROXIMATION : S ECTION 6. D. P ROOF FOR L EMMA 5: B ARRON N ORM R ECURSION
Proof.
Note that the update equation looks like,
[Equation]
From Lemma 6 we have
[Equation] [Equation]
Note that since u t ∈ Γ W t we have ∇ u t ∈ Γ W t and L ( ∇ u t ) ∈ Γ kW t (from Assumption 1).
Therefore, we can bound the Barron norm as,
[Equation]
where we use the fact that for a function h , we have ∥ ( I − ∆) − 1 h
∥ B (Ω) ≤ ∥ h ∥ B (Ω) from Lemmain ( i ) and the fact that L ( ∇ u t ) ∈ Γ kW t in ( ii ) .
Using the result of Addition from Lemma 6 we have
[Equation]
D. P ROOF FOR B ARRON N ORM OF P OLYNOMIAL
Lemma 10 ( Lemma 7 restated) .
Let
[Equation]
where α is a multi-index and x ∈ R d and A α ∈ R is a scalar.
If g : R d → R d is a function such that g ∈ Γ W , then we have f ◦ g ∈ Γ PW and the Barron norm can be bounded as,
[Equation]
Proof.
Recall from Definition 7 we know that for a vector valued function g : R d → R d , we have
[Equation]
Then, using Lemma 6, we have
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
B
[Equation]
Ω)
[Equation]
α, | α | = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
A α d (cid:89) i = g
α i i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
B (Ω) ≤
P (cid:88) α, | α | = | A α | (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
37) where we have repeatedly used Lemma 6 and Cauchy-Schwartz in the last line.
Using the fact that for a multivariate function g : R d → R d we have for all i ∈
[ d ] ∥ g ∥ B (Ω) ≥
∥ g i ∥
B (Ω) .
Therefore, from Equation 37 we get,
[Equation]
Since the maximum power of the polynomial can take is P from Corollary 1 we will have f ◦ g ∈ Γ PW .
D. P ROOF FOR B ARRON N ORM A LGEBRA :L EMMA
The proof of Lemma 6 is fairly similar to the proof of Lemma 3.3 in Chen et al.
(2021)—the change stemming from the difference of the Barron norm being considered
Proof.
We first show the result for Addition and bound ∥ h 1 + h
∥ B (Ω) ,
[Equation]
∥ h 1 + h 2 ∥ B (Ω) ≤ ∥ h 1 ∥ B (Ω) + ∥ h 2 ∥ B (Ω)
[Equation]
For Multiplication , first note that multiplication of functions is equal to convolution of the functions in the frequency domain, i.e., for functions g 1 : R d → d and g 2 : R d → d , we have,
[Equation]
[Equation]
Now, to bound the Barron norm for the multiplication of two functions, ∥
[Equation]
Where we use ∥ ω ∥ 2 ≤ ∥ ω − z ∥ 2 + ∥ z ∥ 2 and the fact that
[Equation]
Collecting the relevant terms together we get,
[Equation]
Hence using Young’s convolution identity from Lemma 11 we have
[Equation]
In order to show the bound for Derivative , since h ∈ Γ W , there exists a function g : R d
→ R such that,
[Equation]
Now taking derivative on both sides we get,
[Equation]
This implies that we can upper bound | (cid:99) ∂ i g ( ω ) as
[Equation]
Hence we can bound the Barron norm of ∂ j h as follows:
[Equation]
In order to show the preconditioning, note that for a function g : R d → R , if f =
( I − ∆)
− 1 g then we have then we have ( I − ∆)
f = g .
Using the result form Lemma 12 we have
[Equation]
Bounding ∥
( I − ∆) − 1 h ∥ B (Ω) ,
[Equation]
Corollary 1.
Let g : R d → R then for any k ∈ N we have ∥ g k ∥ B (Ω) ≤ ∥ g ∥ k B (Ω) .
Furthermore, if the function g ∈ F W then the function g k ∈ Γ kW .
Proof.
The result from ∥ g k ∥ B (Ω) follows from the multiplication result in Lemma 6 and we can show this by induction.
For n = 2 , we have from Lemma 6 we have,
[Equation]
Assuming that we have for all n till k − 1 we have
[Equation]
for n = k we get,
[Equation]
To show that for any k the function g k ∈ Γ kW , we write g k in the Fourier basis.
We have:
[Equation]
In particular, the coefficients with ∥ ω ∥ ∞ > kW vanish, as we needed.
Lemma 11 (Young’s convolution identity) .
For functions g ∈ L p ( R d ) and
h ∈
L q ( R d ) and
[Equation]
where 1 ≤ p, q, r ≤ ∞ we have
[Equation]
Here ∗ denotes the convolution operator.
Lemma 12.
For a differentiable function f : R d → R , such that f ∈ L 1 ( R d ) we have
[Equation]