CAPE: C HANNEL -A TTENTION -B ASED PDE P ARAMETER E MBEDDINGS FOR S CI ML
Anonymous authors
Paper under double-blind review
A BSTRACT
Scientiﬁc Machine Learning (SciML) is concerned with the development of machine learning methods for emulating physical systems governed by partial differential equations (PDE).
ML-based surrogate models substitute inefﬁcient and often non-differentiable numerical simulation algorithms and ﬁnd multiple applications such as weather forecasting, molecular dynamics, and medical applications.
While a number of ML-based methods for approximating the solutions of PDEs have been proposed in recent years, they typically do not consider the parameters of the PDEs, making it difﬁcult for the ML surrogate models to generalize to PDE parameters not seen during training.
We propose a new channel-attention-based parameter embedding (C APE ) component for scientiﬁc machine learning models and a simple and effective curriculum learning strategy.
The C APE module can be combined with any neural PDE solvers allowing them to adapt to unseen PDE parameters without harming the original models’ ability to ﬁnd approximate solutions.
The curriculum learning strategy provides a seamless transition between teacher-forcing and fully auto-regressive training.
We compare C APE in conjunction with the curriculum learning strategy using a PDE benchmark and obtain consistent and signiﬁcant improvements over the base models.
The experiments also show several advantages of C APE , such as its increased ability to generalize to unseen PDE parameters without substantially increasing inference time and parameter count.
An implementation of the method and experiments are available at https://anonymous.4open.science/r/CAPE-ML4Sci-145B .
I NTRODUCTION
Many real-world phenomena, ranging from weather forecasts to molecular dynamics and quantum systems, can be modeled with partial differential equations (PDEs).
While for some problems the mathematical description of these equations is available, ﬁnding its solutions is complex and usually needs some numerical treatments.
Numerical simulation methods have been developed for many years and have achieved a high level of accuracy in solving these equations.
However, numerical methods are resource intensive and time-consuming even when run on larger supercomputers to obtain sufﬁciently accurate results.
Especially high-resolution and high-dimensional hydrodynamictype ﬁeld equations are computationally demanding.
The situation becomes even worse if it is necessary to perform simulations with various PDE parameters since a numerical simulation is required for each of the initial conditions and for each PDE parameter’s conﬁgurations.
Recently, there has been a rapidly growing interest in machine learning methods for the problem of solving PDEs due to their various applications in science and engineering Guo et al.
; Lusch et al.
; Sirignano & Spiliopoulos (2018); Raissi (2018);
Kim et al.
Hsieh et al.
Bar-Sinai et al.
; Bhatnagar et al.
; Pfaff et al.
; Wang et al.
Khoo et al.
For example, several prior studies reported that ML models can estimate solutions more efﬁciently than classical numerical simulators (Li et al., 2021a; Stachenfeld et al., 2021).
Moreover, using neural networks as surrogate models allows us to compute derivatives with respect to the input variables.
Differentiable surrogate models allow one to use backpropagation and automatic differentiation to solve the so-called inverse problems which have numerous real-world applications but are difﬁcult to solve using traditional numerical methods (Coros et al., 2013; Allen et al., 2022).
A considerable number of papers have shown the advantage of ML-based surrogate models (Li et al., 2020; 2021a; Stachenfeld et al., 2021).
The majority of these methods, however, are purely data-
[Figure]
driven, which does not allow us to change PDE parameters.
Although a few models are taking into account PDE parameters, they are tailored to speciﬁc neural networks and cannot be used with other state-of-the-art methods.
This makes it difﬁcult for the SciML community to develop models with high generalization capability not only for the initial conditions but both for different types of PDEs and PDE parameters.
To overcome the shortcomings of existing data-driven SciML models, a straightforward approach would include the PDE parameters as additional input.
However, this naive method requires modiﬁcation of the B ASE network which is potentially harmful to its accuracy.
An alternative approach attaches an external parameter embedding module to the network.
However, there are too many possible module structures and methods to provide the embedded parameter information to the base network, and it is in general non-trivial to select the best one.
In this paper, we propose a new and effective parameter embedding module by utilizing the channel-attention method inspired by a numerical solver with the implicit method and style transfer in ML (see Sec. 2.3 and Sec. 2.4).
The crucial idea is that a neural network generates intermediate (approximated) ﬁeld data for future time steps which are then interpolated by a B ASE model such as the FNO (Li et al., 2021a) to predict the ﬁeld data for the next time step.
C APE can be combined with any existing autoregressive neural
PDE solvers.
Fig.
1 illustrates the proposed C APE framework.
We make the following contributions.
First, we propose a C APE module which can be combined with any existing neural PDE solvers and can effectively transfer the PDE parameter information to the base network (B ASE ).
Second, we propose a simple but effective curriculum learning strategy that seamlessly bridges the teacher-forcing and auto-regressive methods.
Third, we perform extensive experiments using various PDEs with a large number of different parameters evaluating the effectiveness and efﬁciency of the proposed method in comparison with state-of-the-art methods.
C APE : A F RAMEWORK FOR N EURAL PDE S OLVERS 2. B ACKGROUND : P ARTIAL D IFFERENTIAL E QUATIONS
Following the notation by Brandstetter et al.
, we consider Partial Differential Equations (PDEs) over time dimension t 2 [0 , T ] and over spatial dimensions x = [ x 1 , . .
, x D ] 2 X ✓ R D which can be written as
[Equation]
where u : [0 , T ] ⇥ X !
R c is the solution of the PDE, where c is the ﬁeld dimension, used to describe various ﬁeld quantities such as velocity, pressure, and density, while u 0 ( x ) is the initial condition at time t = 0 , and B [ u ]( t, x ) = {+
+} are the boundary conditions at x in @
X , which is the boundary of the domain X .
Here, @ x u , @ xx u , . .
. are the partial derivatives of the solution u with respect to the domain, while @ t u is with respect to the time.
The functional F describes the possibly non-linear interactions between the PDE’s terms.
2. P ROBLEM D EFINITION
We consider PDEs (Sec. 2.1) whose solution is described as a temporal sequence of ﬁeld data  u k k
=
0 ,...,N :
= u 0 , u 1 , ..., u N where u k is the ﬁeld data at time step t k , that is, the state of the physical system governed by the PDE under consideration at time t k discretized using  t = T/N .
Each u 2 X ✓ R c ⇥ x 1 ,...,x D represents the ﬁeld tensor data with c , the number of physical variables such as density and velocity, and x i the spatial dimensions of the i -th coordinate.
For example, for a 1-d problem we have X ✓ R c ⇥ x 1 , for a 2-d problem X ✓ R c ⇥ x 1 ⇥ x 2 , and for a 3-d problem
X ✓ R c ⇥ x 1 ⇥ x 2 ⇥ x 3 .
We will often refer to c as the channel dimension.
We aim to emulate numerical simulators of PDEs which iteratively map M : X !
X from u k to u k +1 .
The emulator (or surrogate model) is a learnable function modeled as a neural network NN with weights ✓ .
We refer to the parameters of a neural network as weights to avoid a conﬂict in terminology with the parameters of
PDEs.
In the following, we denote the emulator’s prediction at time index k as ˜ u k .
Auto-regressive neural networks predict the next time step’s ﬁeld data based on a sequence of ﬁeld data tensors of length `
[Equation]
Given the length of the input sequence N 2 N , and an initial input sequence
[Equation]
˜ u
[Equation]
, ..., ˜ u `  of length ` < N , the ML model auto-regressively generates the remaining sequence  ˜ u
` , ..., ˜ u N  .
The training loss is typically the normalised root-mean-square error (RMSE) between the predicted and the true ﬁeld data tensors
[Equation]
3) where || u || 2 is the L 2 -norm of a (vector-valued) variable u .
Since we are training an auto-regressive neural network, the gradients of the above loss can be backpropagated in time in various ways.
We discuss this in the following sections.
Figure 1(left) illustrates this auto-regressive approach to solving PDEs.
In the vast majority of experimental setups, the assumption is made that ` > 1 , and, therefore, an initial input sequence of length ` is available to the model; in practice, this would require a numerical simulation to be run for `  1 -time steps from the initial condition and for each
PDE parameter  .
The main idea of C APE is to learn to generate these sequences based on the current ﬁeld data and parameter values  and use those as input to an off-the-shelf neural surrogate model such as an FNO (Li et al., 2021a) or an U-Net (Ronneberger et al., 2015) to perform a complex interpolation.
2. C OMBINING N EURAL PDE S OLVERS WITH THE C APE M ODULE
The proposed approach is motivated by the need for neural PDE solvers to generalize to PDE parameters unseen during training.
We propose C APE , a novel neural network architecture that takes the prior state of the system ˜ u k and PDE parameters  as input and predicts the ` - intermediate future states [Equation]
[Equation]
[Equation]
✓
[Equation]
The output of C APE is then used by the B ASE network.
The overall structure is provided in Fig.
1(right).
The intuition behind this approach is that the intermediate future states capture information about the PDE parameters’ impact by attending to the results of the convolutional operations.
While we do not change the architecture of the base neural PDE solvers, we propose to use them to predict, given the past temporal states and the intermediate future states, the state for the next time step.
This is contrary to the typical use of neural
PDE solvers.
The base network is trained jointly with the C APE module.
As shown in Sec. 3, this choice improves the prediction capability of the B ASE network.
During training, the output of C APE is regularized by the additional loss term
[Equation]
which forces the C APE module to predict a temporal sequence of future ﬁeld data
[Equation]
Finally, the intermediate sequence  ˆ u k !
k
+ i cape i
=1 ,..., ` is concatenated with u k , the ﬁeld data at time t k , and given to the base network to make the ﬁnal prediction.
In summary, the C APE module transforms the input variables  u k ,  into temporal-sequential intermediate ﬁeld data { u k , ˆ u k !
k +1 cape , . .
, ˆ u k !
k + ` cape } which is then interpolated by the base neural network.
Before we
[Equation]
[Figure]
introduce the inductive bias of the C APE module, we motivate the general approach from a classical numerical simulation perspective.
C APE as style interpolation.
Let us consider the following simple PDE:
[Equation]
The base neural network can be expressed with the equation
[Equation]
where f base is the function expressed by the base network and
[Equation]
C APE ( u k ,  ; ✓ C APE ) are the intermediate future states predicted by the C APE module generated for time index k .
The C APE module can be interpreted as a pre-processor network providing sequential data u k , { u k + i } i =1 ,..., ` , while Eq.
6 can be interpreted as an interpolation network applying a particular style of the given data u k to { ˆ u k
+ i cape } i =1 ,..., ` (see the last ﬁgure of Fig. 4).
C APE as implicit discretization method.
On the other hand, those equations can be understood from a numerical method perspective, where using implicit discretization (Anderson et al., 2016), Eq.
5 reduces to:
[Equation]
Now, Eq. 6 with ` = 1 can be interpreted as an approximation of the implicit method of Eq.
When ` > 1 C APE can therefore be seen as a generalized ML-based variant of the implicit method.
[Equation]
(C APE M ODULE )
C APE computes 3 different d -dimensional channel attention masks a ↵ 2 R d , ↵ = 1 , 2 , 3 from the parameters of the PDE  using a 2 -layer MLP
[Equation]
where d is the channel dimension in the feature space and  is the GeLU activation function (Hendrycks & Gimpel, 2016).
W ↵ =
( W 1 , ↵ , W 2 , ↵ ) are the weights associated with three operators: a 1 ⇥ 1 -convolution ( g 1 ), a depth-wise convolution ( g 2 ), and a spectral convolution (Li et al., 2021a) ( g 3 ), that are used to compute the tensor representations
z k ↵ 2 R d ⇥ n x ... as z k ↵ = g ↵ ( u k , W ↵ ) .
The tensors are then multiplied by the attention
[Equation]
using the Hadamard operator (  1 ) over the ﬁrst dimension (the channel dimension) which is equivalent to the broadcast operation of ML programming languages.
The three convolutions can be interpreted as a ﬁnite difference method since convolution operations accumulate local information of a mesh, which, in principle, can simulate local interactions such as advection and diffusion.
Intuitively, channel attention is equivalent to choosing an appropriate physical process for each PDE parameter.
A similar mechanism has been proposed for visual tasks, called the squeeze-and-excitation networks (Hu et al., 2018) which enhances useful channels of the feature vector of convolutional networks through an attention mechanism.
The feature v k ↵ 2 R d ⇥ n x ... , ↵ = 1 , , 2 , 3 are combined to form an intermediate feature y k 2 R c ⇥ ` ⇥ n x ... as
[Equation]
[Table]
where h 1 ⇥ 1 , ⇤ are 1 ⇥ 1 convolutions that adjust the number of dimensions, in particular h 1 ⇥ 1 ,c !
d : c ⇥ n x · · · !
d ⇥ n x . . .
, while h 1 ⇥ 1 ,d ! c ⇥ ` : d ⇥ n x · · · !
c ⇥ ` ⇥ n x . . . .
Finally, the sequence of predictions is computed
[Equation]
(11) where y ki is the i -th element of the data tensor y k , selected from the second dimension 2 .
For the sake of presentation, we omitted the batch dimension.
Figure 2 illustrates the architecture of the
C APE module.
C URRICULUM L EARNING
For each initial condition u 0 , the B ASE and C APE models (referred to as NN( u k ; ✓ ) ) jointly predict the full temporal sequence { u k } k =1 ,..,N .
We propose the following curriculum learning strat- egy.
For each training epoch, we split the temporal sequence into two parts.
For the ﬁrst part ( u 0 , ..., u k trans ) we use auto-regressive training using ˜ u k , the prediction of the model at time index k , as input to predict the solution for time index k + 1 , that is, ˜ u k +
=
NN(˜ u k ; ✓ ) .
For the second part ( u k trans +1 , ..., u N ) , we train the model using teacher-forcing .
The teacher-forcing strategy (Williams & Zipser, 1989; Bengio et al., 2015) computes the prediction for time index k
+ 1 using a noisy version of the true value at time index k , that is, ˜ u
k +1 =
NN( u k + ✏ ; ✓ ) where ✏ is random noise increasing the stability at inference time (Sanchez-Gonzalez et al., 2018; 2020; Pfaff et al., 2020; Stachenfeld et al., 2021).
The time index k trans determines the time step where we switch from auto-regressive training to teacher-forcing and is computed using the following monotonically increasing function of the epoch number n
[Equation]
where N is the total epoch number, and  is a hyper-parameter controlling the steepness of the transition function.
A plot of the function is provided in Fig. 6) and a detailed algorithm of the training strategy is provided in Appendix (Algorithm 1).
The strategy is based on the following two assumptions: (1) the prediction error decreases as the number of training epochs increases, (2) the accumulated error increases as the number of autoregressive rollout steps increases.
Teacher-forcing training is usually more stable since it avoids the accumulation of prediction errors and should be used exclusively in the ﬁrst phase of training.
The auto-regressive strategy simulates the behavior at test time and exposes the model to inputs that evolved further from the true data, making it more robust to error accumulation.
For the same reasons, however, it tends to be less stable, especially in the early phase of training.
The proposed curriculum-learning strategy is used to combine the advantages of both approaches.
E XPERIMENTS
We used datasets provided by PDEBench (Takamoto et al., 2022) a benchmark for SciML from which we selected the following PDEs:
1D
Advection Equation.
This equation describes the pure-advection of waves
[Equation]
where  is the PDE parameter describing advection velocity.
The exact solution of this equation is: u ( t, x ) = u
( t, x   t ) where u 0 is the initial condition.
Hence, this PDE can be used to check if the ML models understand the property of advection, updating the solution by just advecting the initial proﬁle without changing it.
1D Burgers Equation.
Burgers’ equation is a mathematical model equation simulating the nonlinearity and diffusivity in the hydrodynamic equation by a scalar variable
[Equation]
14) where ⌫ is the diffusion coefﬁcient and the parameter of this equation.
This PDE can be used to check if ML models can understand the non-linear behavior from the second term in the left-hand side of Eq.
14 and the diffusion process whose strength is controlled by the parameter ⌫ .
2D Compressible Navier-Stokes Equations (2D NS).
The compressible Navier-Stokes equations (NS eqs.
, in the following) is one of the basic physics equations describing classical ﬂuid dynamics
[Equation]
(16) where ⇢ is the mass density, v is the velocity, p is the gas pressure, ✏ =
p/ (   1) is the internal energy,  = 5 / 3 ,  0 is the viscous stress tensor, and ⌘ , ⇣ are the shear and bulk viscosity, respectively.
For 1-dimensional PDEs, we used N = 9000 training instances and 1000 test instances for each
PDE parameter with resolution 128.
For 2-dimensional NS equations, we used N = 900 training instances and 100 test instances for each PDE parameter with spatial resolution 64 ⇥ 64 .
Experiment Setup.
We evaluated the neural models U-Net
(Ronneberger et al., 2015) and FNO (Li et al., 2021a)
with some datasets provided by PDEBench (Takamoto et al., 2022) for various parameters for the 1D Advection equation, 1D Burgers equation, and 2D compressible Navier-Stokes equations.
We also evaluated the message passing neural PDE Solvers (MPNN) (Brandstetter et al., 2022) as a baseline allowing conditional treatment of PDE parameters.
We trained each of the neural models (1) Base : without any changes (vanilla model), (2) PINO : with a PINO loss (Li et al., 2021b), (3) Conditional : the parameters are added to the input data as new channel-dimensions, (4) 2-step : with the ﬁeld data for the current and previous time-steps as input ( u k , u k  1 ) , and (5) CAPE : with the C APE module.
Other than case (4), we only provided ﬁeld data for one time step to the models and, therefore, the models cannot obtain PDE parameters’ information from the given data.
The PINO loss function regularizes the ML models to satisfy the residuals of the PDEs and might lead to an improved generalization behavior for unseen PDE parameters.
The C APE module predicts intermediate ﬁeld data for one future time step and this is used as the input to the B ASE model together with the ﬁeld data of the current time step.
The amount of ﬁeld data provided to the B ASE network in cases (4) and (5) is the same: in case (4) the model always obtains ﬁeld data for time steps k and k  1 as input to predict the ﬁeld data for time step k +1 while in case (5) the B ASE model obtains ﬁeld data for time step k and intermediate ﬁeld data for time step k + 1 generated by the C APE model to predict the ﬁeld data for time step k + 1 .
Hence, the model in case (4) obtains two true ﬁeld data for 2 initial steps and, therefore, has the opportunity to adapt to different PDE parameter values.
Hence, while case (4) is more expensive, we consider it a strong baseline for the problem. [Figure] [Table]
Since the solutions of each PDE are not normalized and based on prior results on evaluating PDE solvers Takamoto et al.
(2022), we measure the normalized RMSE (nRMSE).
We used the normalized RMSE loss function L nRMSE with the auxiliary loss function of the C APE module L cape :=
L nMSE + ↵ L cape where ↵ is the weight coefﬁcient.
The optimization was performed with Adam (Kingma & Ba) for 100 epochs.
The learning rate was divided by 2.0 every 20 epochs.
For a fair comparison, we made the model size of the different methods as similar as possible.
A table with parameter sizes is provided in Tab. 6 in the appendix.
A more detailed description of the hyper-parameters is provided in Appendix B.
Varying the parameter values.
Fig.
3 shows bar plots comparing the B ASE models with and without C APE module, the models with PINO loss, and the models with the 2-steps as input.
The
C APE module results in the lowest error in all cases.
In particular, the C APE module leads to an impressible error reduction ranging from 20 % (2D NS equation) to 95 % (1D Advection).
We partly attribute this to the B ASE network’s ability to capture physical dynamics from the PDE parameterdependent data provided by the C APE module.
The vanilla FNO is a state-of-the-art model and is superior to the U-net as a B ASE model 4 .
Interestingly, the PINO loss provides almost no beneﬁt in our setting.
We hypothesize that the PINO loss is heavily affected by and dependent on the timestep size.
A more detailed explanation of this observation is given in Appendix C. Interestingly, the C APE module provides either comparable or a little better results than the case with 2-step information.
This indicates that the C APE module succeeded in providing equivalent and even more useful information to the B ASE network.
Generalization Ability.
Fig.
5 plots the normalized MSE for each parameter value of the PDEs using FNO as the B ASE network.
The parameter of the 1D Advection PDE controls the advection velocity and the parameters of the remaining equations control the strength of the diffusion process.
[Table]
First, the 1D-Advection result shows that the C APE module overﬁts with the trained parameter (  = 0 . 2 , 0 . 4 , 0 .
7 , 2 . 0 , 4 . 0 ), though it showed a very nice generalization performance to the trained PDE parameters.
This could also be indicated by the fact that only the C APE result for the 1D Advection equation showed a much lower error than the approach receiving the 2-step ﬁeld data which in theory provides a similar amount of information as the C APE module.
Hence, to evaluate generalization to the unseen PDE parameters, we expect the difference of these errors to be correlated with the C APE module overﬁtting to the trained parameters.
On the other hand, the other cases (the parameter describes the diffusion process) showed a good generalization to unseen PDE parameters.
Note that the plots also indicate that vanilla models show a preference for the parameter regime; in all the cases, the vanilla models exhibit better results on smaller diffusion coefﬁcients but lose accuracy as the diffusion coefﬁcients increase.
Ablation experiments.
In this section, we performed an ablation study to separate the impact of the curriculum learning strategy. The ablation study on C APE structure is provided in Tab. 9 in Appendix.
Tab.
3 lists the result for the 2D NS equations using FNO and Unet as the B ASE network.
The proposed curriculum learning strategy drastically impacts the accuracy of the model in all cases, indicating the effectiveness of seamlessly bridging teacher-forcing and auto-regressive training.
The full ablation study results are provided in Tab. 8 in Appendix.
Qualitative analysis of the C APE module.
In Fig.
4 we plot some representative outputs of the vanilla FNO, the C APE module, and the overall C APE model, and compare them with the true solutions.
Interestingly, we can see that the B ASE network often interpolates a higher noise approximation of the C APE module into the typical shape (style) of the ﬁnal solution.
Inference time.
We provide the comparison of the inference time between the hybrid approach (including 2 initial steps as input as done in the FNO paper) and C APE (only using the initial step as the input).
Here we only consider a scenario where the initial time steps are obtained from a numerical simulation.
Tab.
4 lists the inference time in the case of the 2D NS equation.
The inference time of the FNO with the C APE module is much shorter than the hybrid method where the inference time is dominated by the simulation time 5 . [Table]
R ELATED W ORK
Scientiﬁc Machine Learning Models Scientiﬁc Machine Learning aims at data-driven modeling of physical systems.
A notable example is Physics Informed Neural Networks (Cai et al., 2022) (PINNs) that, having access to the PDE of a system, learns a neural network over the domain X !
R d by enforcing small residual, i.e. the error when the solution is evaluated by the PDE or the boundary conditions, over a set of sampled points.
While PINNs have the capacity to model various physical systems, they need to be trained for each new condition or parameter.
Neural Operators (Li et al., 2021b), as FNO (Li et al., 2021a) or Graph NO (Li et al., 2020), models the continuous operators over an inﬁnite space and have shown the ability to generalization at multiple scales.
Also, more traditional image-to-image neural networks such as the U-Net (Ronneberger et al., 2015) can be adopted to model NOs.
Physics-Informed Neural Operators (PINO) (Li et al., 2021b) improve the representational power of PINNs by pre-training a NO but having similar limitations to NOs.
Message passing neural PDE solver (Brandstetter et al., 2022) extends the message passing principle to solve PDEs.
In signal processing, for the artiﬁcial bandwidth extension (ABE) task, the TimeFrequency Network(Dong et al., 2020) (TFNet) has been proposed, which shares a similar concept of channel attention (CA).
Training Autoregressive Models As was discussed in Sec.
2.5, there are two representative training strategies for Scientiﬁc ML, that is, teacher-forcing and auto-regressive training.
The teacherforcing strategy was originally developed in natural language processing (Williams & Zipser, 1989;
Bengio et al., 2015) which predicts n+1-th step data (word) using the true n-th step information (word).
This method is known to prevent from the error-accumulation in the predicted sequential data during the model training.
In the case of Scientiﬁc ML, it was found that it is proﬁtable to add a random noise for improving the robustness against the accumulated error at the inference time (Sanchez-Gonzalez et al., 2018; 2020; Pfaff et al., 2020; Stachenfeld et al., 2021).
On the other hand, the auto-regressive strategy uses the previous prediction of the model as n-th timestep information.
Because of the error accumulation problem, not so many works were adopted, e.g. (Li et al., 2021a).
Recently, (Brandstetter et al., 2022) proposed a reconciling method for this problem which is the so-called ”pushforward trick”.
To increase stability, this method uses an adversarial-style loss which predicts the next timestep data using the previous prediction which is calculated using true data.
Note that our method adopted a curriculum training strategy to prevent error accumulation instead of using an additional loss function.
C ONCLUSION
This paper proposed a C APE module which allows any data-driven SciML models to incorporate PDE parameters.
We also proposed a simple but effective curriculum training strategy that allows us to bridge teacher-forcing and auto-regressive learning.
We performed an extensive set of experiments and showed the effectiveness and efﬁciency of our method from various aspects: generalization of seen/unseen PDE parameters during training, parameter efﬁciency, and inference time.
One of our key ﬁndings is the behavior of ML models without parameter embedding.
We found that the ML models without parameter embeddings either (1) show a poor performance uniformly for all the PDE parameters (1D Advection eq.), (2) overﬁtting to a preferred parameter regime (1D Burgers eq.
and 2D NS eqs.).
Hence, the ML model in this case has no generalization power in terms of PDE parameters.
On the other hand, the models with C APE module generalized well for 1D Burgers’ eq.
and 2D NS eqs.
whose parameters govern the physical systems’ diffusion behavior.
C APE cannot be generalized for 1D Advection eq., and we consider it necessary to formulate C APE to be more physics-informed which we consider for future work.