Meta-Auto-Decoder for Solving Parametric Partial Differential Equations
Abstract
Many important problems in science and engineering require solving the so-called parametric partial differential equations (PDEs), i e., PDEs with different physical parameters, boundary conditions, shapes of computation domains, etc.
Recently, building learning-based numerical solvers for parametric PDEs has become an emerging new ﬁeld.
One category of methods such as the Deep Galerkin Method (DGM) and Physics-Informed Neural Networks (PINNs) aim to approximate the solution of the PDEs.
They are typically unsupervised and mesh-free, but require going through the time-consuming network training process from scratch for each set of parameters of the PDE.
Another category of methods such as
Fourier Neural Operator (FNO) and Deep Operator Network (DeepONet) try to approximate the solution mapping directly.
Being fast with only one forward inference for each PDE parameter without retraining
, they often require a large corpus of paired input-output observations drawn from numerical simulations, and most of them need a predeﬁned mesh as well.
In this paper, we propose Meta-AutoDecoder (MAD), a mesh-free and unsupervised deep learning method that enables the pre-trained model to be quickly adapted to equation instances by implicitly encoding (possibly heterogenous) PDE parameters as latent vectors.
The proposed method MAD can be interpreted by manifold learning in inﬁnite-dimensional spaces, granting it a geometric insight.
Extensive numerical experiments show that the MAD method exhibits faster convergence speed without losing accuracy than other deep learning-based methods.
The project page with code is available: https://gitee.com/mindspore/mindscience/tree/master/MindElec/.
Introduction
Many important problems in science and engineering, such as inverse problems, control and optimization, risk assessment, and uncertainty quantiﬁcation [1, 2], require solving the so-called parametric
PDEs, i.e., partial differential equations (PDEs) with different physical parameters, boundary conditions, or solution regions.
Mathematically, they require to solve the so-called parametric PDEs that can be formulated as:
[Equation]
where L γ 1 and B γ 2 are partial differential operators parametrized by γ 1 and γ 2 , respectively, and (cid:101) x denotes the independent variable in spatiotemporal-dependent PDEs.
Given U
=
U (Ω; R d u ) and the space of parameters A , η = ( γ 1 , γ 2 , Ω) ∈
A is the variable parameter of the PDEs and u ∈ U is the solution of the PDEs.
Note that the form of η considered here is very general with possible heterogeneity allowed, since the computational domain shape Ω and the functions deﬁned on this domain or its boundary (which may be involved in γ 1 , γ 2 ) is obviously of different type.
Solving parametric PDEs requires to learn an inﬁnite-dimensional operator G : A → U that map any PDE parameter η to its corresponding solution u η (i.e., the solution mapping).
In recent years, learning-based PDE solvers have become very popular, and it is generally believed that learning-based PDE solvers have the potential to improve efﬁciency [3, 4, 5].
The learning-based
PDE solvers can be categorized into two categories in terms of the objects that are approximated by neural networks (NN), i.e., the approximation of the solution u η and the approximation of the solution mapping G .
NN as a new ansatz of solution.
This kind of approaches approximate the solution of the PDEs with a neural network and mainly rely on governing equations and boundary conditions (or their variants) to train the neural networks.
For example, PINNs [3] and DGM [6] constrain the output of deep neural networks to satisfy the given governing equations and boundary conditions.
Deep Ritz
Method (DRM) [7] exploits the variational form of PDEs and can be used to solve PDEs that can be reformulated as equivalent energy minimization problems.
Based on a weak formulation of PDEs,
Weak Adversarial Network (WAN) [8] parameterizes the weak solution and test functions as primal and adversarial neural networks, respectively.
These neural approximation methods can work in an unsupervised manner, without the need to generate labeled data from conventional computational methods.
However, all these methods treat different PDE parameters as independent tasks, and need to retrain the neural network from scratch for each PDE parameter.
When a large number of tasks with different PDE parameters need to be solved, these methods are computationally expensive and impractical.
In order to mitigate retraining cost, E and Yu [7] recommends a transfer learning method that uses a model trained for one task as the initial model to train another task.
However, according to our experiments, transfer learning method is not always effective in improving convergence speed (see Sec.3.1, 3.3).
NN as a new ansatz of solution mapping.
This kind of approaches use neural networks to learn the solution mapping between two inﬁnite-dimensional function spaces [9, 10, 11, 12, 13].
For example,
PDE-Nets [9, 10] are among the earliest neural operators that are speciﬁcally designed convolutional neural networks inspired by ﬁnite difference approximations of PDEs.
They are able to uncover hidden PDE models from observed dynamical data and perform fast and accurate predictions at the same time.
DeepONet [11] uses two subnets to encode the parameters and location variables of the PDEs separately, and merge them together to compute the solution.
FNO [13] utilizes fast
Fourier transform to build the neural operator architecture and learn the solution mapping between two inﬁnite-dimensional function spaces.
A signiﬁcant advantage of these approaches is that once the neural network is trained, the prediction time is almost negligible.
Although they have demonstrated promising results across a wide range of applications, several issues occur.
First, the data acquisition cost is prohibitive in complex physical, biological, or engineering systems, and the generalization ability of these models is poor when there is not enough labeled data [14].
Second, most of these methods [9, 10, 12, 13] require a predeﬁned mesh and utilize the labeled data on the mesh for training and inference.
Third, simply applying one forward inference may lead to unsatisfactory generalization, especially on out-of-distribution (OOD) settings (i.e., PDE parameters for training and inference are from different probability distributions).
Finally, these operators directly takes the
PDE parameter η as network input, which would bring inconvenience in network implementation if η is heterogeneous.
The recently proposed Physics-Informed DeepONet (PI-DeepONet) [15] can learn a mesh-free solution mapping without any labeled data and retraining.
However, it needs to collect a large number of training samples in the parameter space
A to obtain an acceptable accuracy (see
Sec.3.1), and is still inﬂexible dealing with heterogeneous PDE parameters.
Meta-Learning.
Different from conventional machine learning that learns to do a given task, meta-learning learns to improve the learning algorithm itself based on multiple learning episodes over a distribution of related tasks.
As a result, meta-learning can handle new tasks faster and better.
In this ﬁeld, the Model-Agnostic Meta-Learning (MAML) [16] algorithm and its variants [17, 18, 19] have beed widely used.
These algorithms try to ﬁnd an initial model with good generalization ability such that it can be adapted to new tasks with a small number of gradient updates.
For example,
MAML [16] ﬁrstly trains a meta-model with good initialization weight on a variety of learning tasks, which is then ﬁne-tuned on a new task through a few steps of gradient descent to get the target model.
The Reptile [18] algorithm eliminates second-order derivatives in MAML algorithm by repeatedly sampling a task, training on it, and moving the initialization towards the trained weight on that task.
Borrowing the idea of meta-learning may inspire new ways to solve parametric PDEs, where different
PDE parameters correspond to different tasks.
To the best of our knowledge, Meta-MgNet [20] is the ﬁrst work that view solving parametric PDEs as a meta-learning problem, which is based on hypernet and the multigrid algorithm.
Meta-MgNet utilizes the similarity between tasks to generate good smoothing operators adaptively, and thereby accelerates the solution process, but is not directly applicable to PDEs on which the multigrid algorithm is not available.
Recently, the Reptile algorithm is also used to accelerate the PDE solving problems in [21].
However, MAML and Reptile are not always effective in improving convergence speed (see Sec.3.2 and 3.3).
Our contributions.
We propose Meta-Auto-Decoder (MAD), a mesh-free and unsupervised deep learning method that enables the pre-trained model to be quickly adapted to equation instances by implicitly encoding heterogeneous PDE parameters as latent vectors.
Different from Meta-MgNet,
MAD makes use of the similarity between tasks from the perspective of manifold learning, and tries to learn a nonlinear approximation of the solution manifold.
We construct the ansatz of solution as a neural network in the form u θ ( (cid:101) x, z ) .
By taking the spatial (or spatial-temporal) coordinate (cid:101) x directly as the network input, unsupervised training loss is allowed, and a mesh is no longer required.
As the additional input z varies, u θ ( (cid:101) x, z ) moves on a manifold in an inﬁnite-dimensional function space, which may be an approximation of the true solution manifold for certain θ .
The PDE parameter η is implicitly encoded into z by applying the auto-decoder architecture motivated by [22], regardless of the possible heterogeneity.
When a new task comes, MAD achieves fast transfer by projecting the new task to the manifold and ﬁne-tuning the manifold at the same time.
The main contributions of this paper are summarized as follows:
• A mesh-free and unsupervised deep neural network approach is proposed to solve parametric PDEs.
Based on meta-learning concept, once the neural network is pre-trained, solving a new task involves only a small number of iterations.
In addition, the auto-decoder architecture adopted by MAD can realize auto-encoding of heterogeneous PDE parameters.
• The mathematical intuition behind the MAD method is analyzed from the perspective of manifold learning.
In short, a neural network is pre-trained to approximate the solution manifold, and the required solution is searched on the solution manifold or in a neighborhood of the solution manifold.
• Extensive numerical experiments are carried out to demonstrate the effectiveness of our method, which show that MAD can signiﬁcantly improve the convergence speed and has good extrapolation ability for OOD settings.
Methodology 2.1 Meta-Auto-Decoder
We adopt meta-learning concept to realize fast solution of parametric PDEs.
Our basic idea is to ﬁrst learn some universal meta-knowledge from a set of sampled tasks in the pre-training stage, and then solve a new task quickly by combining the task-speciﬁc knowledge with the shared meta-knowledge in the ﬁne-tuning stage.
We also adapt the auto-decoder architecture in [22], and introduce u θ ( (cid:101) x, z ) to approximate the solutions of parametric PDEs.
The architecture of u θ ( (cid:101) x, z ) is shown in Fig.1.
A physics-informed loss is used for training, making the proposed method unsupervised.
Putting all these together, we propose a new method Meta-Auto-Decoder (MAD) to solve parametric PDEs.
For the rest of the subsection, the loss function and the two stages of training will be explained in details.
To enable unsupervised learning, given any PDE parameter η ∈ A , the physics-informed loss
L η : U → [0 , ∞ ) about Eq.(1)
[Equation]
(2) is considered, where λ bc > 0 is a weighting coefﬁcient.
The Monte Carlo estimate of L η [ u ] is
[Equation]
where { (cid:101) x r j } j ∈{ 1 ,...,M r } and { (cid:101) x bc j } j ∈{ 1 ,...,M bc } are two sets of random sampling points in Ω and ∂ Ω , respectively.
This task-speciﬁc loss ˆ L η [ u ] can be computed by automatic differentiation [23], and will be used in the pre-training stage and the ﬁne-tuing stage.
In the pre-training stage, through minimizing the loss function, a pre-trained model parametrized by θ ∗ is learned for all tasks and each task is paired with its own decoded latent vector z ∗ i .
Such a pre-trained model is considered as the meta knowledge as it is learned from the distribution of all tasks and the learned latent vector z ∗ i is the task-speciﬁc knowledge.
When solving a new task in the ﬁne-tuning stage, keep the model weight θ ∗ ﬁxed and minimize the loss by ﬁne-tuning the latent vector z .
Alternatively, we may unfreeze θ and allow it to be ﬁne-tuned along with z .
These two ﬁne-tuning strategies give rise to different versions of MAD, which are called MAD-L and MAD-LM , respectively.
The corresponding problems of pre-training and ﬁne-tuning are formulated as follows:
Pre-training Stage Given N randomly generated PDE parameters η 1 , . .
. , η N ∈ A , both MAD-L and MAD-LM solve the following optimization problem
[Equation]
where θ ∗ is the optimal model weight, { z ∗ i } i ∈{ 1 ,...,N } are the optimal latent vectors for different PDE parameters, and ˆ L η i is deﬁned in Eq.(3)
The regularization 2 (cid:107) z is added for training stability.
Fine-tuning Stage ( MAD-L )
Given a new PDE parameter η new , MAD-L keeps θ ∗ ﬁxed, and minimizes the following loss function to get
[Equation]
then u θ ∗ ( · , z ∗ new ) is the approximate solution of PDEs with parameter η new .
To speed up convergence, we can set the initial value of z to z ∗ i obtained during pre-training where η i is the nearest 2 to η new .
Fine-tuning Stage ( MAD-LM ) MAD-LM ﬁne-tunes the model weight θ with the latent vector z simultaneously, and solves the following optimization problem
[Equation]
with initial model weight θ ∗ .
This would produce an alternative approximate solution u θ ∗ new ( · , z ∗ new ) .
The latent vector is initialized in the same way as MAD-L .
Remark
The MAD method has several key advantages compared with existing methods.
Besides being mesh-free and unsupervised, it can deal with heterogeneous PDE parameters painlessly, since η is not taken as the network input, and is encoded into z in an implicit way.
Introduction of the meta-knowledge θ ∗ would accelerate the ﬁne-tuning process, which can be better understood in the light of the manifold learning perspective.
For MAD-LM, the accuracy on OOD tasks is likely to be at least comparable with training from scratch based on PINNs.
Although the ﬁne-tuning process of
MAD is still slower than one forward inference of a neural network solution mapping, the advantages presented above can make it more suitable for some real applications.
Remark
If we replace the physics-informed loss by certain supervised loss, the MAD-L method would then coincide with the DeepSDF algorithm [22].
Despite of this, the ﬁeld of solving parametric
PDEs is quite different from 3D shape representation in computer graphics.
Moreover, the introduction of model weight ﬁne-tuning in MAD-LM can signiﬁcantly improve solution accuracy, as is explained intuitively in Sec.2.2,2.3 and validated by numerical experiments in Sec.3.
2.2 Manifold Learning Interpretation of MAD-L
We interpret how the MAD-L method works from the manifold learning perspective, which also provides a new interpretation of the DeepSDF algorithm [22].
For the rest of this section, the domain Ω is ﬁxed and excluded from η for simplicity.
Now, we consider the following scenario.
Scenario 1 The set of solutions G ( A ) =
{ G ( η ) | η ∈ A} ⊂ U is contained in a low-dimensional structure.
To be more speciﬁc, there is a ﬁnite-dimensional space Z = R l (with l (cid:28) dim U ) and a Lipschitz continuous mapping ¯ G : Z → U , such that G ( A ) ⊆
¯ G ( Z ) .
In other words, for any η ∈ A , there exists z ∈ Z satisfying ¯ G ( z ) =
G ( η ) .
The mapping ¯ G is Lipschitz continuous if and only if there exists some C > 0 such that (cid:13)(cid:13) ¯ G ( z ) − ¯ G
( z (cid:48) )
(cid:13)(cid:13)
U ≤ C (cid:107) z − z (cid:48) (cid:107) for all z, z (cid:48) ∈ Z .
This Lipschitz continuous constraint excludes highly irregular mappings like space-ﬁlling curves.
When A is a ﬁnite-dimensional space and G is Lipschitz continuous, the parametric PDE would fall into this scenario (just take Z = A , ¯ G = G ).
Since dim Z (cid:28) dim U (the latter is usually inﬁnity) holds, we may view the mapping ¯ G as some sort of “decoder”, and Z is the corresponding latent vector space, despite of the fact that there doesn’t exist an “encoder”.
In many cases, ¯ G ( Z ) ⊂
U forms an embedded submanifold, and therefore our MAD method can be viewed as a manifold-learning approach.
Once the mapping ¯ G is learned as above, then for a given parameter η , searching for the solution u η in the whole space U is no longer needed.
Instead, we may focus on the smaller subset ¯ G ( Z ) , i.e. the class of functions in U that is parametrized by Z , since u η = G ( η ) ∈ ¯ G ( Z ) holds for any η ∈ A .
We then solve the optimization problem
[Equation]
u ηf θ ∗ ( · ,z 1 ) u ηf θ ∗ ( · ,z 2 )
u ηf θ ∗ ( · ,z 3 )
U given η new , we fine-tune z u η new
G ( A ) G θ ∗ ( Z ) (a) Fine-tune z only ( MAD-L )
u ηu ηf θ ∗ ( · ,z 1 ) f θ ∗ ( · ,z 2 )
U u η new fine-tune z fine-tune θ simultaneously
G θ ∗ ( Z ) G ( A ) (b) Fine-tune both z
and θ ( MAD-LM )
and ¯ G ( z η ) is the approximate solution.
Assuming that the dimension of Z is chosen (either empirically or through trial and error), the aim is to ﬁnd the mapping ¯ G .
Since such a mapping is usually complex and hard to design by hand, we consider the θ -parametrized 3 version G θ : Z → U , and ﬁnd the best θ automatically by solving an optimization problem.
G θ can be constructed in the simple form
[Equation]
where u θ is a neural network whose input is the concatenation of (cid:101) x ∈ R d and z ∈ R l .
The next step is to ﬁnd the optimal model weight θ via training, with the target being G ( A ) ⊆
G θ ( Z ) .
Assuming that the PDE parameters are generated from a probability distribution η ∼ p
A , then G ( η ) ∈ G θ ( Z ) holds almost surely if and only if
[Equation]
which suggests taking θ ∗
= arg min θ d ( θ ) .
In case we do not have direct access to the exact solutions u η , the equivalent 4 condition
[Equation]
is considered, and d (cid:48) ( θ ) becomes the alternative loss to be minimized.
In the speciﬁc implementation, the expectation on η ∼ p A is estimated by Monte Carlo samples η 1 , . . .
, η N , and the optimal network weight θ is taken to be
[Equation]
We further estimate the physics-informed loss L η using Monte Carlo method to obtain Eq.(4).
After that, when a new PDE parameter η new ∈ A comes, a direct adaptation of Eq.(7) would then give rise to the ﬁne-tuning process Eq.(5), since u θ ∗ ( · , z ) =
G θ ∗ ( z )
≈ ¯ G ( z ) holds.
An intuitive illustration of how MAD-L works from the manifold learning perspective is given in Fig.2(a).
G ( A ) G θ (30) ( Z )
G θ (60) ( Z )
G θ (200) ( Z ) '
G ( η i ) “ i ∈ ' 1 , , 19 “ '
G θ (30) ( z (30) i ) “ i ∈ ' 1 , , 19 “ '
G θ (60) ( z (60) i ) “ i ∈ ' 1 , , 19 “ '
G θ (200) ( z (200) i ) “ i ∈ ' 1 , , 19 “ (a) Pre-training 5.
G ( A ) G θ ∗ ( Z )
G ( η new )
G θ ∗ ( z (0)new )
G θ ∗ ( z (3)new )
G θ ∗ ( z (6)new )
G θ ∗ ( z (9)new )
G θ ∗ ( z (12)new )
[Figure]
A Visualization Example An ordinary differential equation (ODE) is used to visualize the pretraining and ﬁne-tuning processes of MAD-L .
Consider the following problem with domain Ω = ( − π, π )
⊂ R :
[Equation]
We sample 20 points equidistantly on the interval [0 , 2] as variable ODE parameters, and randomly select one η new for ﬁne-tuning stage and the rest { η i } i ∈{ 1 , ··· , 19 } for pre-training stage.
MAD-L generates a sequence of ( θ ( m ) , { z ( m ) i } i ∈{ 1 , ··· , 19 } ) in pre-training stage, and terminates at m =with the optimal ( θ ∗ , { z ∗ i } i ∈{ 1 , ··· , 19 } ) .
The inﬁnite-dimensional function space U =
C ([ − π, π ]) is projected onto a 2-dimensional plane using Principal Component Analysis (PCA).
Fig.3(a) visualizes how G θ ( Z ) gradually ﬁts G ( A ) in pre-training stage.
The set of exact solutions G ( A ) forms a 1-dimensional manifold (i.e. the red solid curve), and the marked points { G ( η i ) } i ∈{ 1 , ··· , 19 } represent the corresponding ODE parameters used for pre-training.
Each dotted curve represents a solution set G ( m ) θ ( Z ) obtained by the neural network at the m -th iteration with the points G θ ( m ) ( z ( m ) i )
=
u θ ( m ) ( · , z ( m ) i ) also marked on the curve.
As the number of iterations m increases, the network weight θ = θ ( m ) updates, making the dotted curves evolve and ﬁnally ﬁt the red solid curve, i.e., the target manifold G ( A ) .
Fig.3(b) illustrates the ﬁne-tuning process for a given new ODE parameter η new ∈ A .
As in Fig.3(a), the red solid curve represents the set of exact solutions G ( A ) , while the cyan dotted curve represents the solution set G θ ∗ ( Z )
=
G θ (200) ( Z ) obtained by the pre-trained network.
As z = z ( m ) new updates (i.e., through ﬁne-tuning z ), the marked point G θ ∗ ( z ( m ) new ) moves on the cyan dotted curve, and ﬁnally converges to the approximate solution G θ ∗ ( z ∗ new )
=
G θ ∗ ( z (12) new )
≈
G ( η new ) .
2.3 Manifold Learning Interpretation of MAD-LM
The MAD-L method is designed for Scenario 1.
However, many parametric PDEs encountered in real applications do not fall into this scenario, especially when the parameter set A of PDEs is an inﬁnite-dimensional function space.
Simply applying MAD-L method to these PDE solving problems would likely lead to unsatisfactory results.
However, MAD-LM works in a more general scenario, and thus has the potential of getting improved performance for a wide range of parametric PDE problems.
This alternative scenario is given as follows.
Scenario 2 The solution set G ( A ) ⊂
U can be approximated by a set with low-dimensional structure, in the sense that there is a ﬁnite-dimensional space Z = R l (with l (cid:28) dim U ) and a Lipschitz continuous mapping ¯ G : Z → U , such that G ( A ) is contained in the c -neighborhood of ¯ G ( Z ) ⊂
U , where c is a relatively small constant.
In other words, for any η ∈ A , there exists some z ∈ Z satisfying (cid:107) ¯ G
( z ) − G ( η ) (cid:107) U ≤ c .
Appendix A gives an example of G ( A ) that falls into Scenario 2 but not Scenario 1.
In this new scenario, similar derivation leads to the same pre-training stage, which is used to ﬁnd the initial decoder mapping G θ ∗ ≈ ¯ G .
However, in the ﬁne-tuning stage, simply ﬁne-tuning the latent vector z won’t give a satisfactory solution in general due to the existence of the c -gap.
Therefore, we have
[Figure]
to ﬁne-tune the model weight θ with the latent vector z simultaneously, and solve the optimization problem Eq.(6).
It produces a new decoder G θ ∗ new ( z ∗ new ) speciﬁc to the parameter η
new
An intuitive illustration is given in Fig.2(b).
Numerical Experiments
To evaluate the effectiveness of the MAD method, we apply it to solve three parametric PDEs: (1)
Burgers’ equation with variable initial conditions; (2) Maxwell’s equations with variable equation coefﬁcients; and (3) Laplace’s equation with variable solution domains and boundary conditions (heterogeneous PDE parameters).
Accuracy of the model is measured by average relative L 2 error (abbreviated as L 2 error ) between predicted solutions and reference solutions, and we provide the mean value and the 95% conﬁdence interval of L 2 error .
We compare MAD with other methods including learning from scratch (abbreviated as From-Scratch ), Transfer-Learning [7], MAML [16, 17],
Reptile [18] and PI-DeepONet [15].
For each experiment, the PDE parameters are divided into two sets: S 1 and S 2 .
Parameters in S 1 correspond to sample tasks for pre-training, and parameters in
S 2 correspond to new tasks for ﬁne-tuning.
See Appendix B for the default experimental setup, and more detailed experimental setups and results for Burgers’ equation, Maxwell’s equation and
Laplace’s equation are given in Appendix C, D and E respectively.
Unless otherwise speciﬁed, all the experiments are conducted under the MindSpore 5 .
3.1 Burgers’ Equation
We consider the 1-D Burgers’ equation:
[Equation]
Eq.(13) can model one-dimensional ﬂow of a viscous ﬂuid, where u is the velocity, ν is the viscosity coefﬁcient and initial condition u 0 ( x ) is the changing parameter of the PDE, i.e. η = u 0 ( x ) .
The initial condition u 0 ( x ) is generated using Gaussian random ﬁeld (GRF) [24] according to u 0 ( x ) ∼ N (0; 100( − ∆ + 9 I ) − 3 ) with periodic boundary conditions.
Fig.4(a) shows the mean L 2 error of all methods as the number of training iterations increases.
All methods converge to nearly the same accuracy (the mean L 2 error close to 0.013) except for MAD-L , which we guess probably due to the c -gap introduced in Sec.2.3.
In terms of convergence speed,
From-Scratch and Transfer-Learning need about 1200 iterations to converge, whereas MAML , Reptile and MAD-LM need about 200 iterations to converge.
MAD-LM has the fastest convergence speed, requiring only 17% of the training iterations of From-Scratch .
In this experiment, Transfer-Learning does not show any advantage over From-Scratch , which means that Transfer-Learning fails to obtain any useful knowledge in pre-training stage.
PI-DeepONet can directly inference for unseen PDE parameters in S 2 , so it has no ﬁne-tuning process.
Table 1 shows the comparison of the mean L 2 error of PI -DeepONet and MAD under different numbers of training samples in S 1 .
The results show that PI-DeepONet has a strong dependence on the number of training samples and its mean L 2 error is remarkably high when S 1 is small. Moreover, its mean L 2 error is signiﬁcantly higher than that of MAD L or
MAD
-LM in all cases
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
In the above experiments, η s in S 1 and S 2 come from the same GRF, so we can assume that the tasks in the pre-training stage come from the same task distribution as the tasks in the ﬁne-tuing stage.
We investigate the extrapolation capability of MAD, that is, tasks in the ﬁne-tuing stage come from different task distribution than those in the pre-training stage.
Speciﬁcally, S 1 is still the same as above, but S 2 is generated from N (0; 100( − ∆ + 25 I ) − 2 . 5 ) .
Fig.5(a) shows the results of extrapolation experiments.
Since the distribution of tasks has changed, the manifold learned in the pre-training stage ﬁts G ( A ) worse, so MAD-L exhibits worse accuracy than that in Fig.4(a).
However, as with Fig.4(a), the convergence speed of MAD-LM in Fig.5(a) is also better than other methods.
This shows that the extrapolation capability of MAD is also better than other methods in this example.
3.2 Time-Domain Maxwell’s Equations
We consider the time-domain 2-D Maxwell’s equations with a point source in the transverse Electric (TE) mode [25]:
[Equation]
where E x , E y and H z are the electromagnetic ﬁelds, J is the point source term.
The equation coefﬁcients (cid:15) 0 and µ 0 are the permittivity and permeability in vacuum, respectively.
The equation coefﬁcients (cid:15) r and µ r are the relative permittivity and relative permeability of the media, respectively.
[26] uses modiﬁed PINNs method to solve Eq.(14) with ﬁxed (cid:15) r = 1 and µ r = 1 .
However, in this paper, ( (cid:15) r , µ r ) are variable parameters of the PDEs, i.e., η = ( (cid:15) r , µ r ) , which corresponds to the media properties in the simulation region.
Fig.4(b) shows that all methods converge to similar accuracy (mean L 2 error is close to 0.04), and
MAD-LM achieves the lowest mean L 2 error (0.028).
In terms of convergence speed, MAD-L and
MAD-LM are obviously superior to other methods.
It is worth noting that MAML fails to converge in pre-training stage, therefore its data is missing in Fig.4(b).
We guess the reason is that singularity brought by point source and computation of second-order derivatives pose great difﬁculties in solving optimization problem.
Reptile also does not show good generalization ability probably due to the same singularity problem.
We do an extrapolation experiment, where ( (cid:15) r , µ r ) in S 1 comes from [1 , 5] 2 , but in the ﬁne-tuning stage, we only consider the ( (cid:15) r , µ r ) =
(7 , 7) case.
Because the extrapolated task does not lie in the task distribution in the pre-training stage, the point G ( η new ) corresponding to the extrapolated task in the function space U is not on the learned manifold G θ ∗ ( Z ) , which causes MAD-L to converge to poor accuracy.
Fig.5(b) indicates that MAD-LM is signiﬁcantly faster than From-Scratch and Reptile in convergence speed while maintaining high accuracy.
Notably, Transfer-Learning also exhibits faster convergence than From-Scratch and Reptile .
This is because ( (cid:15) r , µ r ) =
(4 , 5) is randomly selected in the pre-training stage and is very close to ( (cid:15) r , µ r ) =
(7 , 7) in Euclidean distance.
3.3 Laplace’s Equation
We consider the 2-D Laplace’s equation as follows:
[Equation]
where the shape of Ω and boundary condition g ( x, y ) are the variable parameters of the PDE, i.e. η = (Ω , g ( x, y )) .
In this experiment, we use triangular domain Ω and vary the shape of Ω by randomly choosing three points on the circumference of a unit circle to form the triangle.
Given that h is the boundary condition on the unit circle, we use GRF to generate h ∼ N (0 , 10 3 / 2 ( − ∆+100 I ) − 3 ) with periodic boundary conditions.
The analytical solution of the Laplace’s equation on the unit circle can be obtained by Fourier method.
Then, we use the analytical solution on the three sides of the triangle as the boundary condition g ( x, y ) .
The variable PDE parameters include the shape of the solution domain (the shape of the triangle) and the boundary conditions on the three sides of the triangle, so the PDE parameters here are heterogeneous.
MAD can implicitly encode such heterogeneous PDE parameters as latent vectors conveniently, whereas PI-DeepONet is unable to handle this case without further adaptations.
Fig.4(c) shows that all methods ﬁnally converge to similar accuracy (mean L 2 error close to 0.001), and MAD-LM achieves the lowest mean L 2 error .
MAD-L , MAD-LM and Reptile show good generalization capability and excellent convergence speed, whereas Transfer-Learning and MAML do not show any advantage over From-Scratch .
Summary of experimental results.
Achieving fast adaptation is the major focus of this paper, and solutions within a reasonable precision need to be found.
Indeed, in many control and inverse problems, a higher precision in solving the forward problem (such as parametric PDEs) does not always lead to better results.
For example, a solution with about 5% relative error is already enough for Maxwell’s equations in certain engineering scenarios.
We are therefore interested in reducing the cost of solving the PDE with a new set of parameters by using only a relatively small number of iterations, in order to obtain an accurate enough solution in practice.
The advantages of MAD (especially MAD-LM ) is directly validated in the numerical experiments, as it achieves very fast convergence in the early stage of the training process.
Some other applications may focus more on the ﬁnal precision, and is not as sensitive to the training cost.
In this alternate criterion, the superiority of the MAD method becomes less obvious in our test cases except Maxwell’s equations, but its performance is still comparable to other methods.
Conclusions
In this paper, a novel mesh-free and unsupervised deep learning method MAD is proposed for solving parametric PDEs based on meta-learning idea.
A good initial model is obtained in pre-training stage to learn useful information from a set of sampled tasks, which is then used to help solve the parametric PDEs quickly in ﬁne-tuning stage.
Moreover, MAD can implicitly encode heterogeneous
PDE parameters as latent vectors.
The effectiveness of MAD method is analyzed from the perspective of manifold learning and veriﬁed by extensive numerical experiments.
Acknowledgments
This work was supported by National Key R&D Program of China under Grant No. 2021ZD0110400.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Yes (b) Did you describe the limitations of your work?
Yes (c) Did you discuss any potential negative societal impacts of your work?
N/A (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Yes 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[-Iterations 10 −10 −
-]
N/A (b) Did you include complete proofs of all theoretical results?
N/A 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
No (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[-y 
-]
No (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
No (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
No 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
N/A
(b) Did you mention the license of the assets?
N/A (c) Did you include any new assets either in the supplemental material or as a URL?
N/A (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Yes (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
N/A 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
N/A (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
N/A (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
N/A