A neural multilevel method for high-dimensional parametric PDEs
Abstract
In scientiﬁc machine learning, neural networks recently have become a popular tool for learning the solutions of differential equations.
However, practical results often conﬂict the existing theoretical predictions in that observed convergence stagnates early.
A substantial improvement can be achieved by the presented multilevel scheme which decomposes the considered problem into easier to train sub-problems, resulting in a sequence of neural networks.
The efﬁcacy of the approach is demonstrated for high-dimensional parametric elliptic PDEs that are common benchmark problems in uncertainty quantiﬁcation.
Moreover, a theoretical analysis of the expressivity of the developed neural networks is devised.
Introduction
The application of current machine learning methods to problems based on mathematical models has become one of the most promising rising research areas.
In engineering and the natural sciences in particular, complicated mathematical models (usually formulated in terms of differential equations) form the basis for new insights and groundbreaking technologies.
Since the computation of such models can be extremely time consuming, the use of machine learning tools was obvious and has become increasingly common in recent years 2 .
Consequently, signiﬁcant progress has been made on the one hand to understand the expressivity requirements for representing solutions of differential equations [17, 23, 5, 19, 14].
On the other hand, a plethora of architectures has been devised to implement differential equation solvers [4, 21, 24, 25].
Interestingly, while analytical results predict a superior capacity of deep neural networks e.g. for the important cases of partial differential equations (PDE) and stochastic differential equations (SDE), observed results unfortunately do not reﬂect the theory [1].
Instead, the convergence of the error typically stagnates after some training iterations as e.g. illustrated in [13, 16].
In particular, it often does not reach the accuracy of “classical” best-in-class methods (e.g. stochastic Galerkin or collocation, tensor regression, compressed sensing).
In the following, we are concerned with learning the solution map v : R p
→ H 10 ( D ) of linear parametric PDEs of the form
[Equation]
deﬁned with a linear differential operator A , ﬁxed right-hand side f , appropriate boundary conditions and a p -dimensional parameter vector y ∈ R p (determining the model data) on some domain
D ⊂ R d .
Such (possibly very high-dimensional) PDEs are common problems in the ﬁeld of
Uncertainty Quantiﬁcation (UQ).
They can be used as meaningful benchmark problems to assess the quality of novel SciML methods.
Note that in contrast to many classical machine learning problems, a dataset can be generated on the ﬂy with arbitrary precision using the Finite Element (FE) method.
We recollect the notion of multilevel methods that are ubiquitous in numerical methods.
These consist of a hierarchical construction of the approximation, relying on a set of successively reﬁned discretizations and hence exploiting a decomposition of the problem complexity.
Notably, for our method this means that in each approximation step, only the correction to the next ﬁner level has to be represented in the respective sub-network.
The method which comes closest to our work was recently published in [18].
The three main differences are: a) We train the networks on each grid level to also correct the errors of the previous predictions (which turns out to be a signiﬁcant design choice).
b) We learn the solution v ( y ) of the parametric PDE instead of some quantity of interest.
For this to be feasible we make use of a compression of the FE basis.
c)
We examine the required theoretical network complexity and give an evaluation on standard UQ problems.
Our numerical examples illustrate that the multilevel construction leads to a stable convergence of the parametric solutions which is comparable to best-in-class methods such as the ones in [9, 11, 12, 10, 6, 2].
A neural multilevel method
To construct the multilevel scheme for the considered PDE, we denote by V 1 ⊂ . . .
⊂ V L ⊂ H 10 ( D ) a nested sequence of FE spaces resulting from L successive (uniform) reﬁnements of a coarse grid on the spatial domain D and let v (cid:96) :
R p
→
V (cid:96) be the function which maps a (stochastic) parameter
y ∈
R p to the FE solution v (cid:96) ( y ) of (1).
Note that we can express the solution on the ﬁnest mesh v L as a sum of corrections v
[Equation]
(cid:96) = ( v (cid:96) ( y ) − v (cid:96) − 1 ( y ))
=: L (cid:88) (cid:96) = ˜ v (cid:96) ( y ) , where each ˜ v (cid:96) ( y ) ∈ V (cid:96) .
Since dim V (cid:96) grows exponentially with (cid:96) , we make use of a proper orthogonal decomposition (POD) [3, 22, 20] to yield spaces U (cid:96) =
span { u 1 (cid:96) , . .
, u dim U (cid:96) (cid:96) } of lower dimension which allow an efﬁcient approximation of the corrections ˜ v (cid:96) ( y ) .
In our neural multilevel method, we use neural networks NN (cid:96) :
R p
→ R dim U (cid:96) for (cid:96) = 1 , . . .
, L to learn the basis coefﬁcients c i(cid:96) ( y ) of the approximate subspace corrections v L ( y ) = L (cid:88) (cid:96) = ˜ v (cid:96) ( y ) ≈ L (cid:88) (cid:96) =
dim U (cid:96) (cid:88) i = c i(cid:96) ( y ) u i(cid:96) ≈ L (cid:88) (cid:96) =
dim U (cid:96) (cid:88) i = [ NN (cid:96) ( y )] i u i(cid:96) . To avoid an accumulation of the individual errors of NN (cid:96) in the above sum, we train NN (cid:96) to also correct the errors of the output of the preceding levels (see Step 2 below for a precise description).
For this, in addition to the parameter vector y , we also feed the output coefﬁcients of all previous networks as input.
We want to point out that correcting preceding errors provides an essential improvement over the method in [18] (see Section 3).
For simplicity, in the following we just write NN (cid:96) ( y ) for the output vector of the (cid:96) -th network (instead of NN (cid:96) ( y , NN (cid:96) − 1 , . . .
, NN 1 ) ).
To generate the training data, we draw N ∈ N samples y 1 , . . .
, y N and compute solutions v (cid:96) ( y 1 ) , . .
, v (cid:96) ( y N ) for each (cid:96) = 1 , . . .
, L using the FE method.
The iterative training procedure of our neural multilevel procedure reads now as follows:
Algorithm 2.1.
Step 1: (i) Compute the POD basis functions u 11 , . .
, u dim U 1from the snapshot matrix of FE solutions [ v 1 ( y 1 ) , . . .
, v 1 ( y N )] subject to a desired accuracy.
(ii) Train the ﬁrst network NN 1 to minimize N (cid:88)
[Equation]
[Equation] Step
For
(i) Successively update the current approximation
[Equation]
(ii) Compute the POD basis functions u 1 (cid:96) , . .
, u dim U (cid:96) (cid:96) from the snapshot matrix of corrections [ v (cid:96) ( y 1 ) − w (cid:96) − 1 ( y 1 ) , . . .
, v (cid:96) ( y N )
[Equation]
w
[Equation]
1 ( y N )] .
(iii) Train the network NN (cid:96) to approximate the correction by minimizing
[Equation]
Numerical examples
To assess the performance of Algorithm 2.1, we consider a common high-dimensional PDE and compare the solution approximation of our neural multilevel scheme with two related NN models.
More speciﬁcally, we use the stationary parametric diffusion problem − div( T ( a ( y ))
∇ u ( y ))
=with homogeneous Dirichlet boundary conditions on the unit square.
The conductivity is given by a ( y )
= µ + (cid:80) pm =
a m y m where a m ∼ m − 2 are planar Fourier modes, cf.
We consider the “uniform case” with T = id , µ = 1 , p = 50 and y drawn from U ([ − 1 , 1] p ) , as well as the severely more involved “log-normal case” with T = exp , µ = 1 , p = 50 and y drawn from N (0 , I ) .
Algorithm 2.1 is employed with L = 7 feed-forward ReLU neural networks NN 1 , . . .
, NN L consisting of two hidden layers and 512 nodes per layer.
We analyse the approximation quality of our neural multilevel scheme by comparing it with the closely related architecture of [18] (denoted “Lye et al.”)
3 on an independent test set consisting of 1024 randomly drawn samples.
Moreover, we compare the results to a single larger feed-forward ReLU model (denoted “Single Net”) with roughly the same parameter count as all the multilevel networks of one architecture combined.
“Single Net” is trained to directly predict an approximation on the ﬁnest mesh.
For all three approaches, a POD is utilized with a prescribed tolerance of 10 − 7 .
The network training is based on 10 5 samples that are randomly drawn according to the distribution of the respective test case.
On the ﬁnest level 7, the mesh consists of about 2 × 10 5 triangles with ﬁrst order Lagrange elements.
In Figure 1, the H 10 -error with respect to a high ﬁdelity reference solution of our method and “Lye et. al” is displayed for an increasing number of levels (solid line).
It is noteworthy that our “ML Net” essentially keeps its convergence rate throughout while “Lye at al.”
ﬂattens out, starting at level 4 or 5, respectively.
We also reach a relative error that is much better than what is shown in other NN methods such as e.g. [13].
The reason for this may be found in the subtle difference of the two methods.
“Lye et. al” predicts the level corrections independently of the neural network approximations of the previous level (and their errors) leading to a successive error accumulation from level to level.
In contrast, on each level, “ML Net” is aware of the previous errors and trained to correct them. The
H 10 -error with respect to a high ﬁdelity reference solution projected onto the corresponding grid (dotted line) supports this hypothesis.
While the error for “ML Net” stays relatively constant with increasing levels we can observe an error accumulation for “Lye et al”.
The approximation error of the “Single Net” with respect to the high ﬁdelity reference solution is indicated by a red triangle.
Its performance is comparable to “Lye et al”.
The results suggest that it is advantageous to learn the corrections iteratively with respect to the preceding approximation to mitigate error accumulation instead of learning the grid corrections independently.
Throughout the experiments, our method is the only one whose approximation error is not signiﬁcantly larger than the FE grid approximation error.
Theoretical analysis
In this section, we analyse the complexity (in terms of the number of weights) of the neural networks in our multilevel method 4 .
The proof is based on the well-established ability of neural networks to approximate polynomials efﬁciently [15, 8, 26] and will be published in our upcoming paper.
Our result is in line with the literature [7, Section 3.9] and shows that for ﬁxed stochastic dimension p , to reach the accuracy of the FE solution on the ﬁnest grid ( ∼ η − L ) it sufﬁces to have a polynomially growing number of weights assuming a suitable dimensionality reduction.
As expected, this rate deteriorates with increasing stochastic parameter dimension.
A related analysis without using a multilevel scheme is presented in [23, 17].
Theorem 4.1.
For each L ∈ N there exist neural networks NN 1 , . . .
, NN L where the total number of weights is bounded by
[Equation]
such that the multilevel Algorithm 2.1 together with a POD of accuracy ε POD achieves the precision on the order of the FE solution on the ﬁnest grid (up to a multiplication of L )
[Equation]
∗ . Analysing Single Net in a similar way, we can show that its required complexity grows asymptotically faster than the bound in Theorem 4.1.
This crucial observation shows the efﬁciency of the multilevel ansatz in terms of its approximative power.
Conclusion
We describe and numerically demonstrate a neural multilevel scheme which is comparable to best-inclass performance for the solution approximation of high-dimensional parametric PDEs as commonly encountered in UQ.
As such, the proposed scheme outperforms other neural network approaches from the recent literature.
Regarding analytical aspects, we provide complexity bounds which behave favourably with respect to the expected theoretical convergence for this type of problem.
Moreover, asymptotic estimates also show the advantage of the proposed multilevel structure.
For detailed theoretical results, we refer to our upcoming paper.
Nevertheless, despite the favourable observed performance, there are still several open questions.
In particular, a theoretical non-asymptotic multilevel advantage as in other UQ methods has not been uncovered yet.
Moreover, the required POD reduces the scalability of the approach and prevents an application when regularity of the parameter to solution map is low.
In future work, one might consider different architectures to lessen the dependence on dimensionality reduction methods using e.g. CNNs.