M ETA P: H OW TO T RANSFER Y OUR K NOWLEDGE ON L EARNING H IDDEN P HYSICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Gradient-based meta-learning methods have primarily focused on classical machine learning tasks such as image classification and function regression, where they were found to perform well by recovering the underlying common representation among a set of given tasks.
Recently, PDE-solving deep learning methods, such as neural operators, are starting to make an important impact on learning and predicting the response of a complex physical system directly from observational data.
Since the data acquisition in this context is commonly challenging and costly, the call of utilization and transfer of existing knowledge to new and unseen physical systems is even more acute.
Herein, we propose a novel meta-learnt approach for transfer-learning knowledge between neural operators, which can be seen as transferring the knowledge of solution operators between governing (unknown) PDEs with varying parameter fields.
With the key theoretical observation that the underlying parameter field can be captured in the first layer of the neural operator model, in contrast to typical final-layer transfer in existing meta-learning methods, our approach is a provably universal solution operator for multiple PDE solving tasks.
As applications, we demonstrate the efficacy of our proposed approach on PDE-based datasets and a real-world material modeling problem, demonstrating that our method can handle complex and nonlinear physical response learning tasks while greatly improving the sampling efficiency in new and unseen tasks.
I NTRODUCTION
Few-shot learning is an important problem in machine learning, where new tasks are learned with a very limited number of labelled datapoints (Wang et al., 2020).
In recent years, significant progress has been made on few-shot learning using meta-learning approaches (Koch et al., 2015; Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017; Santoro et al., 2016; Antoniou et al., 2018; Ravi &
Larochelle, 2016; Nichol & Schulman, 2018; Raghu et al., 2019; Tripuraneni et al., 2021; Collins et al., 2022).
Broadly speaking, given a family of tasks, some of which are used for training and others for testing, meta-learning approaches aim to learn a shared multi-task representation that can generalize across the different training tasks, and result in fast adaptation to new and unseen testing tasks.
Although most of meta-learning learning developments focus on conventional machine learning problems such as image classification, function regression, and reinforcement learning, studies on few-shot learning approaches for complex physical system modeling problems have been limited.
The call of developing a few-shot learning approach for complex physical system modeling problems is just as acute, while the typical understanding of how multi-task learning should be applied on this scenario is still nascent.
As a motivating example, we consider the scenario of new material discovery in the lab environment, where the material model is built based on experimental measurements of its responses subject to different loadings.
Since the physical properties (such as the mechanical and structural parameters) in different material specimens vary, the model learnt from experimental measurements on one specimen would have a large generalization error on future specimens.
That means, the data-driven model has to be trained repeatedly with a large number of material specimens, which makes the learning process inefficient.
Further, experimental measurement acquisition of these specimens is often challenging and expensive.
In some problems, a large amount of measurements are not even feasible.
For example, in the design and testing of biosynthetic tissues, performing repeated loading would potentially induces the cross-linking and permanent set phenomenon, which notoriously alter the tissue durability (Zhang & Sacks, 2017).
As a result, it is critical to learn the physical response model of a new specimen with samples sizes as small as possible.
Furthermore, since many characterization methods to obtain underlying material mechanistic and structural properties would require the use of destructive methods (Misfeld & Sievers, 2007;
Rieppo et al., 2008), in practice many physical properties are not measured and can only be treated as hidden and unknown variables.
We likely only have limited access to the measurements on the complex system responses caused by the change of these physical properties.
Supervised operator learning methods are typically used to address this class of problems.
They take a number of observations on the loading field as input, and try to predict the corresponding physical system response field as output, corresponding to one underlying PDE (as one task).
Herein, we consider the meta-learning of multiple complex physical systems (as tasks), such that all these tasks are governed by a common PDE with different (hidden) physical property or parameter fields.
Formally, assume that we have a distribution p ( T ) over tasks, each task T η ∼ p ( T ) corresponds to a hidden physical property field b η ( x ) ∈ B ( R d b ) that contains the task-specific mechanistic and structural information in our material modeling example.
On task T η , we have a number of observations on the loading field g ηi ( x ) ∈
A ( R d g ) and the corresponding physical system response field u ηi ( x ) ∈
U
( R d u ) according to a hidden parameter field
b η ( x ) .
Here, i is the sample index, B ,
A and U are Banach spaces of function taking values in R d b , R d g and R d u , respectively.
For task
T η , our modeling goal is to learn the solution operator G η : A → U , such that the learnt model can predict the corresponding physical response field u ( x ) for any loading field g ( x ) .
Without transfer learning, one needs to learn a surrogate solution operator for each task only based on the data pairs on this task, and repeat the training for every task.
The learning procedure would require a relatively large amount of observation pairs and training time for each task.
Therefore, this physical-based modeling scenario raises a key question: Given knowledge on a number of parametric PDE solving tasks with different unknown parameters, how can one efficiently learn the best surrogate solution operator for a new and unknown parameter, with only a small set of training data pairs 1 ?
To address this question, we introduce MetaP, a novel meta-learnt approach for transfer-learning knowledge between neural operators, which can be seen as transferring the knowledge of solution operators between governing (unknown) PDEs with varying hidden parameter fields.
Our main contributions are:
• MetaP is the first neural-operator-based approach for multi-task learning, which not only preserves the generalizability to different resolutions and input functions from its integral neural operator architecture, but also improves sampling efficiency on new tasks – for comparable accuracy, MetaP saves the required number of measurements by ∼ 90%.
• With rigorous operator approximation analysis, we made the key observation that the hidden parameter field can be captured by adapting the first layer of the neural operator model, in contrast to typical final-layer transfer in existing meta-learning methods.
By construction, MetaP serves as a provably universal solution operator for multiple PDE solving tasks.
• From synthetic, benchmark, to a real-world biological tissue datasets, the proposed method consistently outperforms existing baseline gradient-based meta-learning methods.
B ACKGROUND AND R ELATED W ORK
In this section we introduce the relevant materials on hidden physics learning, neural operators, and gradient-based meta-learning methods, which will later complement the definition of our method.
2. H IDDEN P HYSICS L EARNING AND N EURAL O PERATORS
For many decades, physics-based PDEs have been commonly employed for predicting and monitoring complex system responses, then traditional numerical methods were employed to solve these PDEs and provide predictions for desired system responses.
However, three fundamental challenges present.
First, the choice of governing PDE laws is often determined a priori and free parameters are often tuned to obtain agreement with experimental data.
This fact makes the rigorous calibration and validation process challenging.
Second, traditional numerical methods are solved for specific boundary and initial conditions, as well as loading or source terms.
Therefore, they are not generalizable for other operating conditions and hence not effective for real-time prediction.
Third, complex PDE systems such as turbulence flows and heterogeneous materials modeling problems usually require a very fine discretization, and therefore very time-consuming for traditional solvers.
To provide an efficient surrogate model for physical responses, machine learning methods may hold the key.
Recently, there has been significant progress in the development of deep neural networks (NNs), focusing on learning the hidden physics of a complex system (Ghaboussi et al., 1998; 1991;
Carleo et al., 2019; Karniadakis et al., 2021;
Zhang et al., 2018;
Cai et al., 2022; Pfau et al., 2020;
He et al., 2021; Besnard et al., 2006).
Among these methods, the neural operators show particular promises in resolving the above challenges.
Neural operators aim to learn maps between inputs of a dynamical system and its state, so that the network can serve as a surrogate for a solution operator (Li et al., 2020a;b;c;
You et al., 2022a;
Ong et al., 2022; Gupta et al., 2021;
Lu et al., 2019; 2021b; Goswami et al., 2022a).
Comparing with the classical NNs, the most notable advantages of neural operators are resolution independence and generalizability to different input instances.
Moreover, comparing with the classical PDE modeling approaches, neural operators require only data with no knowledge of the underlying PDE.
All these advantages make neural operators promising tools to PDE learning tasks.
Examples include modeling the unknown physics law of real-world problems (Yin et al., 2022a;
Goswami et al., 2022a; Yin et al., 2022b), and providing efficient solution operator for PDEs (Li et al., 2020a;b;c; Lu et al., 2021c;a).
On the other hand, data in scientific applications are often scarce and incomplete.
Utilization of other relevant data sources could alleviate such a problem, yet no existing work have addressed the transferability of neural operators. Through the meta-learning techniques, our work fulfills the demand of such a transfer setting, with the same type of PDE system but different (hidden) physical properties.
2. G RADIENT -B ASED M ETA -L EARNING M ETHODS
One highly successful meta-learning algorithm has been Model Agnostic Meta-Learning (MAML)
(Finn et al., 2017), which led to the development of a series of related gradient-based meta-learning (GBML) methods (Raghu et al., 2019; Nichol & Schulman, 2018; Antoniou et al., 2018; Hospedales et al., 2020).
Almost-No-Inner-Loop algorithm (ANIL)
(Raghu et al., 2019) modifies MAML by freezing the final layer representation during local adaptation.
Recently, theoretical analysis (Collins et al., 2022) found that the driving force causing MAML and ANIL to recover the general representation is the adaptation of the final layer of their models, which harnesses the underlying task diversity to improve the representation in all directions of interest.
Although MAML and the general meta-learning approaches have achieved impressive performance in some machine-learning applications such as the image classification and reinforcement learning scenarios, a few work has studied the hidden physics learning under meta (Mai et al., 2021; Zhang et al., 2022; Yin et al., 2021; Wang et al., 2021) or even transfer setting (Kailkhura et al., 2019; Goswami et al., 2022b).
Among these meta-learning works, (Mai et al., 2021; Zhang et al., 2022) are designed for specific physical applications, while (Yin et al., 2021; Wang et al., 2021) focus on on dynamics forecasting by learning the temporal evolution information directly (Yin et al., 2021) or learning time-invariant features (Wang et al., 2021).
Hence, none of these works have provided a generic approach nor theoretical understanding on how to transfer the multi-task knowledge between a series of complex physical systems, such that all these tasks are governed by a common parametric PDE with different physical parameters.
M ETA -L EARNT N EURAL O PERATOR 3. I NTEGRAL N EURAL O PERATORS
Here, we first state the base model of this work without the meta aspect.
The integral neural operators, first proposed in (Li et al., 2020a) and further developed in (Li et al.
, 2020b;c; You et al., 2022a;c) comprises of three building blocks.
First, the input function, g ( x ) ∈ A , is lifted to a higher dimensional representation via h ( x , 0) =
P [ g ]( x ) :
= P ( x )[ x , g ( x )] T + p ( x ) .
Here,
P ( x ) ∈ R ( s + d g ) ×
d h and p ( x ) ∈ R d h define an affine pointwise mapping, which are often taken as constant parameters, i.e., P ( x ) ≡ P and p ( x ) ≡ p .
Then, the feature vector function h ( x , 0) goes through an iterative layer block where the layer update is defined via the action of the sum of a local linear operator, a nonlocal integral kernel operator, and a bias function: h ( · , ( l +1)∆ t ) =
J l +1 [ h ( · , l ∆ t )] , for l = 0 , · · · , L − 1 .
Here, h ( · , j ∆ t ) , j = 0 , · · · , L := T/
∆ t , is a sequence of functions representing the values of the network at each hidden layer, taking values in R d h .
J 1 , · · · , J L are the nonlinear operator layers, defined by the particular choice of networks.
In this work, we employ the implicit Fourier neural operator (IFNO) as the base model, because of its theoretical universal approximation property in PDE solving tasks (You et al., 2022c) and robustness in complex physical response modeling tasks (You et al., 2022b) 2 .
In this case, the iterative layers are taken as J 1 = · · · = J L = J , where
[Equation]
Here, F and F − 1 denote the Fourier transform and its inverse, respectively.
In practice, F and F −are computed using the FFT and its inverse to each component of h separately.
Also, c ∈ R d h defines a constant bias, W ∈ R d h ×
d h is the weight matrix, and F [ κ ( · ; v )] :
= R is a circulant matrix that depends on the convolution kernel κ .
σ is an activation function, which is oftenly taken to be the popular rectified linear unit (ReLU) function.
Finally, the output u ( · ) ∈ U is obtained through a projection layer.
In particular, we project the last hidden layer representation h ( · , T ) onto
U as: u ( x ) =
Q [ h ( · , T )]( x ) :
=
Q 2 ( x ) σ ( Q 1 h ( x , T ) + q 1 ( x ))
+ q 2 ( x ) .
Here, Q 1 ( x ) ∈ R d Q × d h ,
Q 2 ( x ) ∈ R d u × d Q , q 1 ( x ) ∈ R d Q and q 2 ( x ) ∈ R
d u are the appropriately sized matrices and vectors that are part of the parameter set that we aim to learn.
Similarly as for the lifting layer, Q 1 ( x ) ,
Q 2 ( x ) , q 1 ( x ) and q 2 ( x ) are also often taken as constant parameters, which will be denoted as Q 1 ,
Q 2 , q 1 and q 2 , respectively.
In the following, we denote the set of trainable parameters in the lifting layer as θ P , the set from the iterative layer block as θ I , and the set in the projection layer as θ Q .
The neural operator can be employed to learn an approximation for the solution operator, G .
Given D := { ( g i , u i ) } Ni =1 , a labelled (context) set of observations, where the input { g i } ⊂
A is a set of independent and identically distributed (i.i.d.)
random fields from a known probability distribution µ on A , and u i ( x ) ∈ U , possibly noisy, is the observed corresponding solution, let Ω ⊂ R s be the domain of interests, we assume that all observations can be modeled with a parametric PDE form [Equation] Here, K b is the operator representing the possibly unknown governing law, e.g., balance laws.
Then, the system response can be learnt by constructing a surrogate solution operator of equation 2: ˜
[Equation]
u
[Equation] x ) , where the parameter set θ =
[ θ P , θ I , θ Q ] is obtained by solving the optimization problem:
[Equation]
Here C denotes a properly defined cost functional which is often taken as the the mean square error.
3. B ASE M ETA M ODEL WITH MAML AND ANIL
To transfer the multi-task knowledge between a series of complex systems governed by different hidden physical parameters, we proposed to leverage the integral neural operator with a meta-learning setting.
Herein, assume that for each training task T η ∼ p ( T ) we have a set of observations of loading field/respond field data pairs D η :=
{ ( g ηi ( x ) , u ηi ( x )) } N η i =1 , and each task can be modeled with a parametric PDE form
[Equation]
where b η ( x ) is the hidden task-specific physical parameter field for the common governing law.
Given a new and unseen test task, T test , and a context set of labelled samples D test := { ( g test i ( x ) , u test i ( x )) } N test i =1 on it, our goal is to obtain the approximated solution operator model on the test task as ˜ G [ g ; θ test ] .
A straightforward approach would be to simply apply MAML and ANIL to a neural operator architecture, which will be treated as the baselines of our studies.
Here we formally state our implementation of ANIL and MAML for the problem described above.
MAML.
The MAML algorithm proposed in (Finn et al., 2017)
aims to find an initialization, ˜ θ , across all tasks, so that new tasks can be learnt with very few examples.
First, we draw a batch {T η } Hη =1 of H tasks from p ( T ) .
For each task T η , we split the available set of loading field/response field data pairs D η to a support set of samples, S η , which will be used for inner loop updates, and a target set of samples, Z η , for outer loop updates.
Then, for the inner loop we let θ η, 0 :=
˜ θ and θ η,i be the task-wise parameter after i gradient updates.
During each inner loop update, we compute
[Equation]
where L S η
( θ η,i − 1 ) is the loss on the support set of the η -th task, and α is the step size.
After m inner loop updates, we update the initial parameter ˜ θ with a fixed step size β :
[Equation]
meta
[Equation]
) , where the meta-loss L meta (˜ θ [Equation]
[Equation]
Then, on the test task, T test , we perform inner loop adaptation based on few labelled samples D test until convergence, and obtain the approximated solution operator model on the test task as ˜ G [ g ; θ test ] .
ANIL.
In (Raghu et al., 2019), ANIL was proposed as a modified version of MAML with inner loop updates only for the final layer.
The inner loop update formulation equation 5 is modified as [Equation] η = [Equation]
H
[Equation]
7) where θ η,iQ is the task-wise parameter on the projection layer after i gradient updates.
Then, we perform the same outer loop updates following equation 6.
3. M ETA P: A N OVEL M ETA -L EARNT N EURAL O PERATOR A RCHITECTURE
We now propose MetaP, which applies task-wise adaptation only to the first layer, i.e., the lifting layer , with the full algorithm outlined in Algorithm 1.
Similar as in other meta-learning approaches (Yoon et al., 2018; Vanschoren, 2018; Yang & Kwok, 2022; Kalais & Chatzis, 2022), the algorithm consists of two phases: (1) a meta-train phase which learns shared iterative layers parameters θ I and projection layer parameters θ P from existing tasks; (2) a meta-test phase which transfers the learned knowledge and rapidly learning surrogate solution operators for unseen tasks with unknown physical parameter field, where only a few test samples are required.
To see the inspiration of the proposed architecture, without loss of generality, we assume that the underlying task parameter field b η ( x ) , modeling the physical property field, is normalized and satisfying (cid:12)(cid:12)(cid:12)(cid:12)
b η ( x ) − b ( x ) (cid:12)(cid:12)(cid:12)(cid:12)
L 2 (Ω) ≤ 1 for all η ∈ { 1 , · · · , H } , where b :=
E T η ∼ p ( T ) [ b η ] .
Denoting F u [ b ] := K b [ u ] as a function from physical parameter fields B to loading fields A , we can take the
Fr´echet derivative of F with respect to b − b and obtain:
[Equation]
[Equation] [Equation]
[Equation]
Substituting the above formulation into equation 4, we obtain
[Equation]
Denoting F 1 [ b η ] :=
[ 1 , b η − b ] and F 2 [ u ηi ] :=
[ F u ηi [ b ] , D F u ηi [ b ]] , we can actually reformulated equation 4 into a more generic form:
[Equation]
We point out that this parametric PDE form is indeed very general and finds applications in many science and engineering applications – besides our motivating example on material modeling, examples also include the monitoring of tissue degeneration problems (Zhang & Sacks, 2017), the detection of subsurface flows (Dejam et al., 2017), the nondestructive inspection in aviation (Fallah et al., 2019), and the prediction of concrete structures deterioration (Wei et al., 2021), etc.
In the following, we show that MetaP are universal solution finding operators for the multi-task PDE solving problem in equation 8, in the sense that they can approximate a fixed point method to a desired accuracy.
For simplicity, we consider a 1 D domain Ω , and scalar functions F 1 [ b η ] , F 2 [ u ηi ] .
These functions are assumed to be sufficiently smooth and measured at uniformly distributed nodes χ := { x 1 , x 2 , . .
, x M } , with F 1 [ b η ]( x j ) ̸ = 0 for all η and j .
Then, equation 8 can be formulated as an implicit system of equations:
[Equation]
where U η, ∗ i :=
[ u ηi ( x 1 ) , u ηi ( x 2 ) , . . .
, u ηi ( x M )] is the solution we seek, ˜ G ηi :=
[ g ηi ( x 1 ) / F 1 [ b η ]( x 1 ) , g ηi ( x 2 ) / F 1 [ b η ]( x 2 ) , . . .
, g ηi ( x M ) / F 1 [ b η ]( x M )] is the reparameterized loading vector, and G ηi :=
[ g ηi ( x 1 ) , g ηi ( x 2 ) , . . .
, g ηi ( x M )] is the original loading vector.
Here, we notice that all task-specific information are encoded in ˜ G ηi and can be captured in the lifting layer parameter.
Therefore, when seeing equation 9 as an implicit problem of U η, ∗ i and ˜ G ηi , it is actually independent of the task parameter field b η , i.e., this problem is task-independent.
In the later contents we refer to equation 9 without the task index, as H ( U ∗ ; ˜ G ) for notation simplicity.
To solve for U ∗ from the nonlinear system H ( U ∗ ; ˜ G ) = 0 , a popular approach would be to use fixed-point iteration methods such as the Newton-Raphson method. In particular, with an initial guess of the solution (denoted as U 0 ), the process is repeated to produce successively better approximations to the roots of equation 9, from the solution of iteration l (denoted as U l ) to the solution of iteration l + 1 (denoted as U l +1 ) as: [Equation] until a sufficiently precise value is reached.
In the following, we show that as long as Assumptionsand 2 hold, i.e., there exists a converging fixed point method, then MetaP can be seen as an resemblance of the fixed point method in equation 10 and hence acts as an universal approximator of the solution operator for equation 8. Assumptions 1 and 2 ensure the hidden PDEs to be numerically solvable with a converging iterative solver, which is a required condition of most numerical PDE solving problems. Then, taking U 0 :=
[ x 1 , · · · , x M ] as the initial guess, we aim to show that for any desired accuracy ε > 0 , one can find a sufficiently large L > 0 and sets of parameters θ
η = { θ ηP , θ I , θ Q } , such that the resultant MetaP model acts as a fixed point method and its prediction satisfies
[Equation]
and samples [Equation]
Assumption 1.
There exists a fixed point equation, U
=
U + R
( U , ˜ G ) for the implicit problem equation 9, such that R : R 2 M
(cid:55)→ R M is a continuous function satisfying R ( U , ˜ G ) = 0 and ||R ( ˆ U , ˜ G ) − R
( ˜ U , ˜ G )
|| l 2 ( R M ) ≤ m ||
ˆ
U − ˜ U
|| l 2 ( R M ) for any two vectors ˆ U , ˜ U ∈ R M .
Here, m > 0 is a constant independent of ˜ G .
Assumption 2.
With the initial guess U 0 :
=
[ x 1 , · · · , x M ] , the fixed-point iteration U l +1 =
U l + R ( U l , ˜ G ) ( l = 0 , 1 , . . . )
converges, i.e., for any given ε > 0 , there exists an integer L such that
[Equation]
for all possible input instances ˜ G ∈ R M and their corresponding solutions U ∗ .
Then, we have our universal approximation theorem as below, with proof provided in Appendix A:
Theorem 1 (Universal approximation) .
Given Assumptions 1-2, let the activation function σ for all iterative kernel integration layers be the ReLU function, and the activation function in the projection layer be the identity function.
Then for any ε > 0 , there exist sufficiently large layer number L >and feature dimension number d h > 0 , such that one can find a parameter set for the multi-task problem, θ η = [ θ ηP , θ I , θ Q ] , such that the corresponding MetaP model satisfies
[Equation]
for all tasks.
E XPERIMENTS
In this section, we demonstrate the empirical effectiveness of the proposed MetaP approach.
Specifically, we conduct experiments on a synthetic dataset from a nonlinear PDE solving problem, a benchmark dataset of heterogeneous materials subject to large deformation, and a real-world dataset from biological tissue mechanical testing, and compare the proposed method against competitive
GBML baselines.
All of the experiments are tested using PyTorch with Adam optimizer, with detailed settings provided in the Appendix B.
In all tests we considered the averaged relative error, || u i,pred − u
i ||
L 2 (Ω) /
|| u
i ||
L 2 (Ω) , as the error metric (lower means better).
We have repeated each experiment for 5 times, and reported the averaged errors and their standard errors.
4. S YNTHETIC D ATA S ETS AND A BLATION S TUDY
We first consider the PDE solution finding problem of the Holzapfel-Gasser-Odgen (HGO) model (Holzapfel et al., 2000), which describes the deformation of hyperelastic, anisotropic, and fiber-reinforced materials.
Different tasks correspond to different material parameter sets, { k 1 , k 2 , E, ν, α } , where k 1 and k 2 are fiber modulus and the exponential coefficient, respectively, E is the Young’s modulus, ν is the Poisson ratio, and α is the fiber angle direction from the reference direction.
In this example the physical response of interests is the displacement field u : [0 , 1] 2 → Rsubject to different traction loadings applied on the top edge of this material.
Therefore, we take the input function g ( x ) as the padded traction loading field, and the output function as the corresponding displacement field.
To investigate the performance of MetaP in few-shot learning, we generate 60 tasks for training, validation, and 1 in-distribution (ID) test task by sampling the physical parameters k 1 , k 2 ∼ U [0 . 1 , 1] ,
E ∼ U [0 . 5 , 1 . 5] , ν ∼ U [0 . 1 , 0 . 49] , and α ∼ U [ π/ 10 , π/ 2] .
Here U stands for uniform distribution.
To further evaluate the generalizability when the physical parameters in test tasks are outside the training regime, we also generate 2 out-of-distribution (OOD) test tasks with physical from different distributions. For the first OOD task we sample E ∼ U [1 . 5 , 2] , while for the second OOD task E ∼ U [0 . 3 , 0 . 5] .
As show in Figure 5 of Appendix B, the first OOD task (denoted as “OOD Task1”) corresponds to a stiffer material sample and smaller deformation for each given loading, while the second OOD task (denoted as “OOD Task2”) generates a softer material sample and larger deformation.
For each training task, we generate 500 data pairs D η :=
{ ( g ηi , u ηi ) } 500 i =1 , by sampling the vertical traction loading from a Gaussian random field.
Then, the corresponding ground-truth displacement field is obtained using the finite element method implemented in FEniCS (Alnæs et al., 2015).
For the test tasks, we train with N test =
{ 2 , 4 , 8 , 12 , 20 , 100 , 300 } numbers of labelled data pairs (the context set), and evaluate the resultant model on an additional dataset with 200 data pairs (the target set).
An 8 -layer IFNO is employed as the base model.
Ablation Study.
We first conduct an ablation study with three settings.
1) Follow the meta-train and meta-test phases as in Algorithm 1, with task-wise adaptation only to the lifting layer in both phases (denotes as “MetaP”).
After MetaP, perform an additional fine-tuning step to all parameters in the meta-test phase (denotes as “MetaP+”).
With this test, we aim to investigate if our algorithm has successfully identified all the common features in the iterative and projection layers.
3) Apply task-wise adaptation only to the projection layer in both meta-train and meta-test phases (denoted as “MetaLast”).
With this test, we study if the successful “adapting last layers” strategy in image classification problems would also apply for our PDE solving problem.
Besides these three settings, we also report the few-shot learning results with four baseline methods: 1) Learn a neural operator model only based on the context data set on the test task (denoted as “Single”), 2) Pretrain a single neural operator model based on all training task data sets, then fine-tune it based on the context test task data set (denoted as “Single+”), 3) MAML, and 4) ANIL.
As shown in the left plot of Figure 2, MetaP and MetaP+ are both able to quickly adapt with few data pairs – to achieve a test error below 5% , “Single” and “Single+” require 100 data pairs, while MetaP and MetaP+ requires only 4 data pairs.
On the other hand, MetaLast, MAML and ANIL have similar performance.
They all require 100 data pairs to achieve a < 5% test error.
This observation verifies our theoretical analysis: on the multi-task parametric PDE solution operator learning problem, one should adapt the first layer, not the last ones.
Moreover, when comparing MetaP and MetaP+ we can see that the additional finetune step barely improves the performance, especially in the few-sample regime.
This fact verifies the efficacy of MetaP, and indicates that our method has successfully capture the underlying task diversity by adapting the first layer, so no further fine-tuning is required.
In-Distribution and Out-Of-Distribution Tests.
On the right plot of Figure 2, we demonstrate the relative test error of MetaP in both ID and OOD tasks.
We can see that these three test errors are both in a similar scale as the error on training tasks.
The error from OOD task1 is slightly smaller than the ID test task error, while the error from OOD task2 is much larger, probably due to the fact that the solutions in OOD task1 generally have smaller magnitude and hence its solution operator lies more in a linear regime, which makes the solution operator learning task easier.
These results validates the good generalization performance of MetaP.
4. B ENCHMARK M ECHNICAL MNIST D ATASETS
To further test the capability of MetaP on benchmark datasets, we test MetaP and four baseline methods on Mechanical MNIST (Lejeune, 2020).
Mechanical MNIST is a dataset of heterogeneous material undergoing large deformation.
It contains 70,000 heterogeneous material specimens, and each specimen is governed by the Neo-Hookean material with a varying modulus converted from the MNIST bitmap images.
On each specimen, 32 loading/response data pairs are provided 3 .
Here in, we randomly select 101 specimens for training and validation, and a new and unseen specimen as the test task.
On the meta-test phase, we reserve 20 data pairs on the test task as the target set for evaluation, then train each model under the few-shot learning setting with N test = { 2 , 4 , 8 , 12 } labelled data pairs as the context set.
All approaches are developed based on an 8-layer IFNO model.
We present the results in the left plot of Figure 3.
The neural operator model learned by MetaP again outperforms the state-of-the-art GBML models.
Our MetaP model achieves 1% when using onlylabelled data pair on the test task, while the “Single” model has around 100% error even usinglabelled data pairs, due to overfitting.
This fact highlights the importance of learning across multitasks in engineering applications – when the total number of measurements on each specimen is limited, it is necessary to transfer the knowledge across specimens.
Moreover, we notice that in this example ANIL is the least effective GBML method, which is even less efficient than the pretrained model (“Single+”), probably due to the inefficacy of the adapting last layers strategy.
4. A PPLICATION ON R EAL -W ORLD D ATA S ETS
We now take a step further to demonstrate the performance of our method on a real-world physical response dataset which is NOT generated by sovling PDEs.
We consider the problem of learning the mechanical response of multiple biological tissue specimens from DIC displacement tracking measurements.
As demonstrated in Figure 1, we measure the biaxial loading of tricuspid valve anterior leaflet (TVAL) specimens from a porcine heart, such that each specimen (as a task) corresponds to a different region of the leaflet.
Due to the material heterogeneity of biological tissues, these specimens are with different mechanical and structural properties.
In this task, we aim to model the tissue response by learning a neural operator mapping the boundary displacement loading to the interior displacement field on each tissue specimen.
On each specimen, we have 500 available data pairs.
Due to the challenges of obtaining the experimental tissue, only 14 specimens are available in total.
This example also stands for a common challenging setting in real-world applications: we not only have the few-shot learning challenge, but also suffer from the difficulty from limited available training tasks.
With a 4-layer IFNO as the base model, we train each model based on N test ∈ [2 , 300] samples, then evaluate the performance on anothersamples.
The results are provided on the right plot of Figure 3.
MetaP performs the best with low data samples among all the methods, and still beat our MAML and ANIL variants when N test = 300 .
Interestingly, MAML and ANIL did not even beat the “Single+” method, possibly due to the low efficacy of the adapting last layers strategy and the small number of training tasks.
C ONCLUSION
In this paper we propose MetaP, the first neural-operator-based meta-learning approach that are designed to achieve good transferability in learning complex physical system responses with significant improvement in sample efficiency.
The first layer adaption used by our method is theoretically motivated and shown to be the universal solution operator for multiple parametric PDE solving tasks.
We demonstrate the effectiveness of our proposed MetaP algorithm on various synthetic and realworld datasets, showing promises over baseline methods.
For future work, we will investigate the applicability of the proposed approach to other neural operators.
[-= 
-]
[-Note that ˜ V is independent of η , and falls into the formulation of V , by letting R 
-]
[-H ( l ∆ t ) + (cid:18) I M ⊗ (cid:20) S + S 
-]