Physics Informed Symbolic Networks
Shirish shirish.karande
Lovekesh Vig TCS Research lovekesh vig Venkataramana Runkana TCS Research
Abstract
We introduce Physics Informed Symbolic Networks (PISN) which utilize physicsinformed loss to obtain a symbolic solution for a system of Partial Differential
Equations (PDE).
Given a context-free grammar to describe the language of symbolic expressions, we propose to use weighted sum as continuous approximation for selection of a production rule.
We use this approximation to define multilayer symbolic networks.
We consider Kovasznay flow (Navier-Stokes) and two-dimensional viscous Burger’s equations to illustrate that PISN are able to provide a performance comparable to PINNs across various start-of-the-art advances: multiple outputs and governing equations, domain-decomposition, hypernetworks.
Furthermore, we propose Physics-informed Neurosymbolic Networks (PINSN) which employ a multilayer perceptron (MLP) operator to model the residue of symbolic networks.
PINSNs are observed to give 2-3 orders of performance gain over standard PINN.
Introduction
Physics Informed Neural Networks [1] (PINNs) have gained popularity for numerically solving
PDEs.
However in the absence of an explicit inductive bias, MLPs can struggle to learn even simple functions in a sample efficient manner [2, 3], Appendix 1.
Meanwhile a symbolic form has the potential to exhibit better systematic generalization, with interpretability as a useful byproduct.
Although symbolic regression has been studied extensively using a variety of algorithms, such as tree search [4–6], sequence generation [7, 8], sparse selection from a bank of primitive expressions [9] and genetic programming [10], none of the existing works have explored the use of physics-informed loss (as done in PINNs) for obtaining a symbolic solution for a system of complex PDEs.
We believe that potentially many of the above mentioned approaches can be extended to accommodate a physicsinformed loss.
However, our preliminary experimentation (A.3) with [5, 7] led to poor performance for complex examples of PDEs.
Hence in this work we consider the problem of physics-informed symbolic regression and make three major contributions towards it.
First, we propose Physics Informed Symbolic Networks (PISNs) that utilize differentiable programming and physics-informed loss for symbolic regression.
We assume that the desired expression belongs to a language defined by a context free grammar with [sin, exp, +, *] as symbolic operations.
Inspired by recent works on differential synthesis of programs [11, 12], we approximate a selection of production rule by a weighted linear sum of the individual rules.
This approximation is used to induce a multilayer symbolic network.
We illustrate that PISNs provide a performance at par with PINNs, with Kovasznay flow (Navier Stokes) and two-dimensional viscous-Burger’s equation as examples.
Second, we demonstrate that physics-informed training of the symbolic networks can be extended to recent advances in PINN, such as HyperNetworks[13] and Domain-decomposition [14, 15] that have enabled applying PINNs to multiple outputs and governing equations, along with faster training.
Lastly, one cannot guarantee universal approximation even with larger number of compositions over an arbitrary set of operators.
Therefore similar to [11], we allow the MLP operator to be a part of the grammar.
Specifically, we capture the residual of the symbolic approximation with a MLP and show that a Physics Informed Neuro-Symbolic Network (PINSN) is able to provide 2-3 orders of magnitude performance gain over PINNs in terms of the maximum error.
Methodology 2.1 Symbolic Networks
We consider symbolic expressions that belong to a language defined by the context-free grammar, G [16], described by (1), where x, y, t, c are the terminal symbols.
We employ a continuous relaxation of G by taking a weighted sum over all the production rules as shown in (2).
[Equation]
Eqn.
(2) is used to induce a Symbolic Network of depth d +2, where d is the total number of hidden layers.
In the following equations, x = [ x, y, t, 1] is the input, W 0 k
=
[ w 0 k 1 , w 0 k 2 , w 0 k 3 , w 0 k 4 ] are the weight vectors for the first hidden layer and W jk =
[ w jk 1 , w jk 2 , w jk 3 , w jk 4 , w jk 5 , w jk 6 , w jk 7 , w jk 8 ] are the weight vectors for all the other hidden layers and W d are the weights used at the output layer.
[Equation]
Eqn.
(2) could be used to induce a multilayer symbolic network similar to the differentiable program architectures proposed in [11].
Such an approach would guarantee that the network is capable of exact representation of all the expressions with a depth proportional to the depth of the network.
Nevertheless in such an architecture the number of parameters would grow exponentially with depth.
On the contrary, the network defined by Eqn.
(3) ensures that the number of parameters increases linearly with depth.
However this does come at the cost of losing out on the power to exactly represent "wide" expressions.
In spite of this, we have observed that for the examples considered in this work, the approximations provided by our symbolic network are comparable with a symbolic network based upon [11], with neither approach providing a consistently better accuracy.
Our approach is able to sometimes provide a advantage in terms of training time.
Detailed investigation of this issue remains.
2.2 NeuroSymbolic Networks
The expressive capacity of G can be improved by adding a MLP operator, F (; θ ) as an additional parameterized terminal symbol to the grammar:
[Equation]
Similar to symbolic networks we can make a continuous approximation of (4) as well.
However instead of adding a neural operator at all depths, for examples considered in this work, we observed substantial gains even by just approximating the residue of the symbolic networks.
Therefore we define the neurosymbolic networks as follows.
[Equation]
[Equation]
[Table]
2.3 Hypernetworks
We propose HyperPISN which consists of a hypernetwork H pisn which takes in the task parameterization λ
h as input and outputs θ pisn (the weights of the PISN network M pisn ).
We consider examples where the initial conditions, boundary conditions and the coefficients of the PDE are all determined by single parameter λ .
We use different values of λ to sample tasks and train the hypernetwork, but show generalization to unseen value of λ .
Since θ pisn can be used to determine the final symbolic expression, HyperPISN can produce symbolic expressions for parameterized PDEs .
[Equation]
We extend PINSNs to HyperPINSNs whose training follows a two step approach.
In first step, H pisn (HyperPISN) is trained for a set of task parameters λ .
In second step, HyperPISN weights are frozen and H res (hypernetwork of MLP operator F ) is trained to predict θ res (weights of F ).
[Equation]
Experiments 3.1 Kovasznay flow (Navier Stokes)
Kovasznay flow is a laminar flow governed by two-dimensional steady state Navier-Stokes equation.
It’s governing equations and true analytical solutions are mentioned in Table 6.
We obtain the boundary conditions from the exact solutions mentioned in Table 6.
The governing equations and boundary conditions are parameterizeed by the Reynolds number Re ∈ [100,1000].
We train the hypernetwork architectures by using 10 equidistant train-tasks.
We elaborate on the training process, hyperparameter details and the mean-L2-Error results in Appendix A.4.1.
3.2 Two-dimensional coupled viscous Burger’s equation
The two-dimensional coupled viscous Burger’s equation is used to model various physical phenomena like shock-wave propogation, turbulence, sedimentation, etc, whose governing equations and true analytical solutions are defined in Table 9.
We consider the range of viscosity coefficient ν ∈
[1e-3,1e2] as parameter λ for governing equations.
In order to capture the dynamics in every sub-domain, we provide sub-domainwise comparative results for one test-task in Appendix A.4.2 and figures 4,5,6,7.
Additional information regarding pointwise mean-L2-error results, training schedule and hyperparameter details are mentioned in Appendix A.4.2. [Table]
Observations
We compare PISN with state-of-the-art Symbolic Regression methods in Appendix A.3.
From table 4, we observe AI-Feynman [5] produces exact symbolic expressions for simpler examples, as the simplifying properties are easier to capture, but struggles for complex examples.
SymbolicGPT [7] on the other hand fails to capture symbolic expressions even for simpler PDE setups without any noisy inputs.
Multivariate Symbolic Function Learner (MSFL) [17] performs better than SymbolicGPT and provides reasonably accurate solutions for simpler PDEs setups, however they fail to scale up to complex examples, as they are constrained by numerical approximations.
We observe, our PISN architecture outperforms all the benchmark methods for complex examples.
We compare PISN to alternative symbolic differentiable program architecture methods in Table 5.
We observe, PISNs outperform MSFL+Autograd on all tasks, and on an average, is 36.5 times better on the ratio of max-error of PISN and MSFL+Autograd.
PISNs also have 1.2 times lower error than differentiable program architecture method proposed in [11], despite having lower number of parameters.
Tables 1 and 2 shows the results for Kovasznay flow and coupled-Burger’s equations respectively.
We provide one illustrative example for Navier Stokes and it’s corresponding error maps in Figures 2 and 3.
For all examples across both the experiments, the difference of errors between a PISN architecture and it’s corresponding PINN architecture is less than that of an order.
This same observation holds for
HyperPISNs, Domain-decomposed-PISNs, and Domain-decomposed-HyperPISNs, which implies recent-advances in PINNs can be combined with PISNs.
When we compare the performance of HyperPISN with PISN we observe ratio of maximum error is on average less than 2 for the 2D Burger’s equations and less than 4.1 for Kovasznay Flow equations.
The increase in ratio of error in case of Kovasznay Flow can be largely attributed to case of Re equal to 125 where the ratio of error averaged across the 3 outputs is 9.6, which is still within an order of magnitude.
We can therefore say that HyperPISNs are hyper symbolic solvers that can generalize across complex parameterized PDEs without the need for training on every value of λ .
Its not straightforward to conceive such a generalization for other symbolic solvers.
We observe PINSN outperform PINN and PISN by 2-3 orders of magnitude across all tasks, and this observation further extends to HyperPINSN, Domain-decomposed-PINSN, and Domain-decomposedHyperPINSN.
This implies adding a MLP operator to the grammar is highly advantageous, as that
MLP operator is capable of capturing the residues of underlying symbolic expression.
We provide an illustrative example for 1 Burger’s equation test-case in Figures 4, 5, 6, 7.
We notice, for every subdomain, PINSNs capture the residues from PISNs and have lower error magnitudes than PINNs from the error heat-maps.
Limitations and Future Work
Our current setup for PISN suffers from training-collapse when we use operators which drastically change the symbolic expression search space e.g. division, logarithm, which restricts the choice of operators.
This leads to our current expressions being long and cumbersome.
For better interpretability of mathematical expressions, crisper analytical solutions are necessary.
Hence, in future, we seek to explore techniques that incorporate additional operators.
Broader Impact
Modeling of a real world phenomenon in terms of differential equations is often an iterative process.
In this context, symbolic solutions not only lend themselves to increased interpretability but also allow for a human in the loop discovery of solutions of PDEs.
We imagine that domain experts and scientist could often guess some approximate behavior of the solution.
This can be used to constrain or initialize the weights of the symbolic network.
A Appendix Contents of appendix
• A.1 Preliminaries A.1.1 Physics-Informed Loss A.1.2 HyperPINN • A.2 Extrapolation failure of MLPs • A.3 PISN
Vs Symbolic Regression Solvers • A.4 Additional Results and Training details
{+A.4.1 Navier Stokes: Kovasznay Flow A.4.2 Two-dimensional coupled Burger’s equation • A.5 Additional Differential Equations A.5.1 Telegraph Equations A.5.
+}
Two-dimensional Burgers: conservation equation • A.6 Sample Mathematical Expressions generated for experiments in section A. • A.7 Impact of L-BFGS Optimizer and increasing the depth of Symbolic Network
A.1 Preliminaries A.1.1 Physics-Informed Loss
We consider Partial Differential Equations of the form
[Equation] Where N is a non-linear differential operator consisting of space and time derivatives, x and t refer to the spatial and temporal variables respectively.
All the three architectures, PINN, PISN and PINSN employ a physics-informed loss to learn their respective parameters which is defined as follows: [Equation] where, L Physics ( θ ) = (cid:88)
[Equation]
C
[-Re=
-]N [Equation] dx [Equation]
(10a)
L
[Equation]
(cid:88)
[Equation]
(10b)
[Equation]
(cid:88)
[Equation]
(10c)
[Equation]
(10d)
C represents the set of collocation points on which the Physics-loss is calculated.
IDC, INC,
BC represents the set of points at which Initial Dirichlet Conditions, Initial Neumann Conditions and Boundary Conditions are evaluated.
λ i , where i ∈ 1,2,3,4 are scalers which are user-defined hyperparameters used to control the loss function.
All the three networks, PINN, PISN and PINSN employ physics-informed loss to train their respective architectures.
In case of PINN, θ represents the weights of the neural network, whereas for PISN and PINSN, θ represents the weights of the continuous approximation of rules defined by context-free grammar.
A.1.2 HyperPINN
Fillipe et al.
in [13] introduced HyperPINNs to solve for parameterized differential equations (Lorenz
System and one-dimensional Burger’s equation).
[Equation]
HyperPINN consists of two components, a hypernetwork H pinn and a main-network M pinn .
H pinn takess in task parameterization λ
h as input and outputs the weights of the main-network θ m .
θ m is directly supplied to the main-network for evaluating the solutions of the differential equation u pinn .
We use different values of λ to sample tasks and train the hypernetwork, but show generalization to unseen value of λ .
A.2 Extrapolation Failure of MLPs
In this section, we illustrate the extrapolation properties of MLPs for various well-known primitive functions.
We consider a MLP with 6 hidden layers, 20 neurons each, considering 100,000 training points within the domain.
Training domain extents for each function are - linear:
exp:[1,3], log:[4,8], sin:[ − 5 π/ 8 , 5 π/ 8 ].
Blue and orange lines in Figure 1 denote the true function and the
- -1 −2.
f(x)=xf(x)=exp(x)
−   f(x)=log(x)
-5π/ 5π/ −   f(x)=sin(x) extrapolation by MLPs respectively.
While MLPs have excellent interpolation properties, they are astonishingly poor extrapolators.
A.3 PISN Vs Symbolic Regression Solvers
In this section, we compare PISN to existing state-of-the-art Symbolic Regression solvers.
AIFeynman [5] comprises simplifying properties of the functions of interest such as symmetry, compositionality and separability among others, to discover the symbolic expressions.
SymbolicGPT [7] [Table] [Table] is a generative-transformer based language model to generate mathematical expressions based on a given set of input-output values.
Panju et al.
in [17] developed a Multivariate Symbolic Function
Learner (MSFL) to generate parse-trees [Table] which can be used to extract mathematical expressions.
While their Taylor-series approximations provide reasonably accurate approximations for lower order derivatives [Table] they deteriorate for higher order and multivariate derivatives.
On the other hand,
Automatic differentiation [18] provides accurate derivatives even for higher order and multivariate derivatives.
Hence, for our PISN architectures [Table] we use automatic differentiation to approximate the physics-informed loss.
Table 3 shows the PDE problem setups for the additional experiments taken from [17] (used to benchmark MSFL) [Table] [Table] [Table]
u x + v y =u ( x, y ) = 1 − e λx cos (2 πy ) [Equation] ) /Re
v (
[Equation]
[Equation]
[Equation]
λe λx sin (
[Equation]
πy ) /
[Equation]
π
[Equation]
/Re
p ( x, y )
= (1 − e 2 λx ) /
Experiment Design: SymbolicGPT and AI-Feynman seek to fit mathematical expressions, given a set of input-output pairs.
Since PDEs don’t have labelled data, we use a PINN to generate inputoutput pairs over the entire domain.
Since the input-output pairs are generated from outputs of a MLP, they are noisy.
We then pass these data points as input to AI-Feynman and pre-trained
SymbolicGPT models to extract the symbolic expression.
As an additional benchmark, we also consider noiseless samples by generating the input-output pairs directly from the ground-truth analytical expression.
Table 4 consists the pointwise-mean-L2-error results, wherein NS refers to 2D Navier-Stokes: Kovasznay flow result for Reynolds number Re=475, and Burger’s refers to twodimensional coupled viscous Burger’s equation for ν = 4 .
3 e − 3 .
The true analytical solutions for the PDEs are referred to from tables 3, 6 and 9.
We display the expressions generated by AI-Feynman and SymbolicGPT, and for 1 Navier-Stokes test-case in Appendix A.6.
We use publicly available
AI-Feynman and SymbolicGPT pretrained-models for conducting the experiments in this section.
Table 5 shows the comparison between differentiable symbolic regression architectures.
We note, performance of MSFL architectures improves by 2-3 orders of magnitude when physics-loss is computed using automatic differentiation as against a Taylor series approximation.
Differentiable programmable architectures refers to [11], which has a higher representation capacity as # Parameters increase exponentially with depth.
We compare the ratio of mean-errors of PISN with MSFL+Autograd, and observe that on an average, PISNs are 36.5 times better than MSFL+Autograd, and PISNs superiority is attributed by it’s significantly better performance in complex examples.
Furthermore, the wave equation has a true solution of sin ( x ) sin ( t ) , which our PISN cannot exactly represent.
Yet,
our
PISN architecture has a very low error of 2.41e-7, indicating the possibility to generate excellent approximations to complex PDEs which have no analytical solutions.
A.4 Additional Results and Training details A.4.1 Kovasznay Flow (Navier Stokes)
Table 6 represents the differential equations and true analytical solutions of Kovasznay flow (Navier
Stokes).
Here, the first equation represents the conservation of mass balance, while the second and third equations represent momentum transfer equations in X and Y dimension respectively.
We consider a 101 x 101 equally spaced grid domain to represent the XY plane, where X, Y ∈ [- 0.5,1.5].
We consider 2601 collocation points and 320 boundary condition points, with 80 points for each face of the grid.
We consider 10 train-tasks equally spread out across the task parameterization space, 5 validation tasks to avoid meta-overfit and 5 test tasks for evaluation.
We train all experiments for 125k epochs.
Remaining hyperparameter details are mentioned in Table 8.
A.4.2 2D Coupled Burger’s Equation
Table 9 represents the governing differential equations and true analytical solutions of 2D-Coupled
Burger’s equation.
The two differential equations represent momentum equations in X and Y directions.
For modeling the temporal component, we consider a timestep interval of 0.1 for t ∈ [0 , 1] .
The solution space is decomposed into 12 components by splitting the grid at X = [0 . 4 , 0 . 8] and
Y =
75] , whose respective MLP architectures and residual points are mentioned in Table 10.
Remaining training details are identical to that used for Kovasznay flow.
Tablerepresents the mean-L2-error results for the tasks considered in Section 3.2. Tables 13 andrepresent domainwise [Table] [Table]
True Analytical solutions u t + u ∗ u x + v ∗ u y =
ν ∗ ( u xx + u yy )
u ( x, y, t ) = 3 / 4 + 1 / 4(1 + exp
[( − 4 x + 4 y − t ) / 32 ν ]) v t + u ∗ v x + v ∗ v y =
ν ∗ ( v xx + v
[Equation]
[Table]
[Figure]
P I N
[Figure]
N
[Figure]
y p
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
N
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
P I N S
N
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
H y p
e r P
I
NN
[Figure]
[Table]
[Figure]
[Table]
[Figure]
H y p
e r P I S
N
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
H y p
e r P
I N
[Figure]
N
[Figure]
1e−
[Table]
[Table]
[Table]
[Table]
[Table]
.62e
[Table] [Table]
[Table] [Table]
.26e2.62e1.69e8.82e2.03e4.12e1.82e4.28e5.58e5.56e1.73e-
[Table]
[Table] [Table]
-  6.62e7.12e7.02e5.05e6.
[Table]
- 6.23e1.05e1.03e2.32e1.31e3.12e1.
[Table]
-  1.26e6.76e7.19e7.74e1.64e6.96e7.26e3.11e5.71e3.29e1.08e2.96e1.98e1.12e7.21e1.11e1.86e1.32e1.33e1.71e5.51e1.25e3.25e8.77e5.03e3.38e3.16e2.72e5.19e2.42e1.37e7.47e7.72e4.11e1.28e2.82e8.08e4.42e3.91e1.19e1.95e2.18eVanilla 2.07e2.03e6.61e1.30e3.82e2.63e5.11e4.71e7.37e2.15e5.54e5.97e4.87e7.02e4.98e1.86e1.36e9.61e3.17e8.72e7.52e5.28e4.65e1.44e2.43e4.19e8.36e3.84e6.31e3.88e5.56e5.44e7.47e5.93e7.86e4.52e6.09e4.42e5.52e3.55e4.91e3.77e4.49e2.73e3.89e1.96e2.06e1.56e3.97e9.16e5.85e1.93e1.66e4.37e5.13e1.16e3.26e3.71e5.57e2.81e6.81e7.63e5.17e4.24e4.86e3.61e2.26e4.82e3.86e1.59e1.71e2.58e-
[Table]
PINN
[Table] [Table]
PINSN − − −
[Table]
[Table]
− − −
[Table] [Table] [Table] [Table]
[Table]
− − −
[Table]
[Table]
[Table]
− − −
[Table] [Table] [Table] [Table]
− − −
[Table]
[Table]
[Table] [Table] [Table] − − −
[Table] [Table]
[Table] [Table]
− − −
[Table]
[Table]
[Table]
[Table]
− − −
[Table] [Table]
− − −
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
PINN
[Table]
[Table]
PINSN − − −
[Table]
[Table]
− − −
[Table]
[Table] [Table] [Table] [Table]
− − −
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
PINN
[Table]
[Table]
PINSN − − −
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
− − −
[Table] [Table] [Table] [Table]
− − −
[Table]
[Table]
− − −
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
PINN
[Table] [Table]
PINSN − − −
[Table] [Table]
− − −
[Table]
[Table] [Table]
[Table] [Table]
− − −
[Table]
[Table] [Table]
− − −
[Table] [Table]
[Table] [Table]
− − −
[Table]
[Table] [Table]
[Table] [Table]
− − −
[Table] [Table]
[Table] [Table]
− − −
[Table] [Table]
[Table] [Table]
− − −
[Table] [Table]
− − −
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
5.89e5.80e3.14e6.36e2.19e5.77e2.85e2.98e2.84e7.27e3.12eVanilla 2.27e1.65e1.91e2.18e3.74e1.99e1.89e1.55e5.16e1.07e3.86e4.49e3.94e2.93e4.41e1.52e2.91e8.84e8.15e2.52e3.51e7.75e1.73e3.91e5.21e3.56e6.82e1.24e2.22e1.19e1.08e7.18e5.32e5.52e1.92e4.74e2.39e1.85e4.59e3.77e6.75e1.97e1.12e7.45e5.12e5.19e1.37e1.39e4.18e2.75e1.87e1.36e3.64e5.12e8.94e6.12e9.73e1.04e4.43e
[Table] [Table]
[Equation]
A.5 Additional Differential Equations A.5.1 Telegraph Equations
Telegraph equations [19] are a sub-derivative of Maxwell equations, used to capture the variations in current and voltage which vary both spatially and temporally.
Telegraph equations are extensively used in modeling transmission lines of varying frequencies, radio frequency conductors, telephone lines, and pulses of direct current.
Table 14 contains the information for parameterized PDEs of
Telegraph equation.
Table 14 represents the PDE information of Telegraph Equations.
Initial Dirichlet and Neumann condition is represented using f(x) and g(x) respectively, with u(x,0)=f(x) and ∂u ( x, 0) ∂t = g(x).
The variable A ∈ R 1 is used to represent the parameterization of the initial conditions, and PDE (in Telegraph-2) and the range column defines the range of values A can take. The final column states the true analytical solution of each of the PDE problems, parameterized by A.
Training Schedule and Hyperparameters: The Telegraph equation is 1-Dimensional x ∈ [0 , 1] and varying in time t ∈ [0 , 1] .
We consider 2601 points as train-collocation points, and sampleinitial and boundary condition points respectively.
Remaining training schedule details are identical to previous experiments.
Tables 15 and 16 represents the mean-L2-error and max-L2-error results for
Telegraph Equations.
We observe, (Hyper)PINSNs boost the accuracy by 2-3 orders of magnitude than (Hyper)PISNs and (Hyper)PINNs.
[Table] [Table]
1.93e-
A.5.2 2D Burger’s: Conservation Equation
The two-dimensional viscous Burger’s equation in it’s conservation form is given by:
[Equation]
Where p ( u ) =
u 2 / 2 and q ( u ) =
u 2 / 2 are the inviscid fluxes, and ν represents the Viscosity coefficient governing the diffusion component of the PDE.
It’s analytical solution is given by u ( x, y, t ) =
(1 + exp ( x + y − t ) / 2 ν ) − 1 .
The training schedule and hyperparameter specifications are identical to 2D-Coupled-Burger’s experiment.
The viscosity coefficient ν ranges between [5 e − 3 , 5 e − 2] .
Table 17 represents the results for domain-decomposed-hypernetwork and domain-decomposed-vanilla architectures.
We observe that (Hyper) domain-decomposed-PINSNs outperform (Hyper) domain-decomposed-PINNs and (Hyper) domain-decomposed-PISNs by 1-2 orders of magnitude.
A.6 Mathematical expressions generated in Appendix A.
Mathematical Expression generated by HyperPISN for Kovasznay flow with Re=475:
[Table]
[Table]
x/ (5 . 56 x + 0 . 6415) − 0 . 0029 log (1 . 9547 t ) 3 + 1 .− 0 . 0115 log (2 . 041 t ) 7 − 3 .− 0 . 0006 log (2 .
{+449 t ) 
+}
+ 1 .− 0 .
0006 log (2 .
449 t ) 5 + 1 .0 .
007 y − 0 . 075 log (2016 . 68 y + 926 . 36) + 1 .− 0 .
{+124 log (125 . 
+}
y + 2 . 26) + 1 .0 .
139 x − 0 .
y − 0 . 011 log (1 . 27 y ) + 1 .6 . 65 x + 4 .0 . 1399 x − 0 .
0112 log (1 . 3568 y ) + 1 .0 .
{+1725 x − 0 . 1256 log (4 . 
 y )+} − 6 .− 0 .
0029 log (1 . 9547 y ) 3 + 1 .− 0 .
0048 log (5 . 174 y )0 .
1399 x + 0 .
0004 log (0 .
773 y )
4 + 1 .0 .
1581 x + 0 .
{+0071 log (2 . 
+}
y ) 4 + 3 .
X-velocity is given by: 
[Table]
[Equation]
[Table]
[Equation]
Y-velocity is given by:  
[Equation] l [Equation] 035 x
[Equation]
l 3 = 0 . 389 x − 0 . 407 y − 0 .l 4 = − 0 .
18 x + 0 .
267 y
+ 0 .l 5 = − 0 .
048 x + 2 . 071 y
+ 0 .l 6 = − 0 .
04 x − 1 .
114 y + 0 .h 11 = 0 .
017 x − 0 .
073 y
+ 0 .
208 sin ( l 1 ) − 0 .
088 exp ( l 2 ) − 0 .
198( l 3 + l 4 ) + 0 .
872 l 5 l 6 − 0 .h
12 = 0 .
192 x − 0 .
215 y + 0 .
158 sin ( l 1 ) − 0 . 332 exp ( l 2 ) − 0 . 21( l 3 + l 4 ) − 0 . 757 l 5 l 6 + 0 .h 13 = − 0 .
036 x − 0 .
001 y + 0 .
295 sin ( l 1 ) + 0 . 141 exp ( l 2 ) + 0 . 098( l 3 + l 4 ) − 0 .[-
-]
l 5 l 6 + 0 .h 14 = − 0 .
122 x − 0 .
11 y − 0 .
043 sin ( l 1 ) − 0 .
19 exp ( l 2 ) + 0 . 109( l 3 + l 4 ) + 0 . 137 l 5 l 6 + 0 .h 15 = − 0 . {+075 x + 0 . 
+}
y + 0 . 128 sin ( l 1 ) + 0 . 197 exp ( l 2 ) + 0 . ( l 3 + l 4 ) + 0 .
769 l 5 l 6 + 0 .h
= 0 .
148 x + 0 .
167 y − 0 .
138 sin ( l 1 ) + 0 .
106 exp ( l 2 ) − 0 .
319( l 3 + l 4 ) + 0 .
764 l
[Equation]
v
[Equation]
Pressure is given by:     l 1 = − 0 .
177 x + 0 .
766 y + 1 .l 2 = − 0 .
037 x + 0 .
y
+ 0 .l 3 = 0 .
03 x + 0 . 296 y + 0 .l 4 = 0 .
458 x − 0 .
142 y − 0 .l
5 = 0 .
115 x − 0 .
249 y + 0 .l 6 = 0 .
726 x − 0 .
y + 0 .h 11 = − 0 .
227 x − 0 .
328 y + 0 .
317 sin ( l 1 ) − 0 .
301 exp ( l 2 ) + 0 .
176( l 3 + l 4 ) + 0 .
341 l 5 l 6 − 0 .h
12 = − 0 .
249 x − 0 .
y − 0 .
403 sin ( l 1 ) + 0 .
041 exp ( l 2 ) − 0 .
308( l 3 + l 4 ) + 0 .
122 l 5 l 6 − 0 .h 13 = − 0 .
3 x + 0 .
115 y
+ 0 .
079 sin ( l 1 )
+ 0 .
45 exp ( l 2 ) + 0 .
229( l 3 + l 4 ) + 0 .
464 l 5 l 6 − 0 .h
14 = − 0 .
267 x + 0 .
127 y
+ 0 .
152 sin ( l 1 ) + 0 .
267 exp ( l 2 ) − 0 .
308( l 3 + l 4 ) + 0 .
384 l 5 l 6 − 0 .h 15 = 0 .
243 x − 0 .
327 y − 0 .
199 sin ( l 1 ) + 0 .
325 exp ( l 2 ) + 0 .
222( l 3 + l 4 ) − 0 .
088 l 5 l 6 + 0 .h 16 = − 0 .
007 x − 0 .
477 y
+ 0 .
257 sin ( l 1 ) + 0 .
347 exp ( l 2 ) − 0 .
( l 3 + l 4 ) − 0 .
213 l 5 l 6 − 0 .p
= 0 .
017 x − 0 .
045 y − 0 .
22 sin ( h 11 ) − 0 . 22 exp ( h 12 ) + 0 . 102( h 13 + h 14 ) + 0 . 163 h 15 h 16 − 0 .(14)
A.7 Impact of L-BFGS Optimizer and increasing the depth of Symbolic Network
We used Adam Optimizer for our experiments in the main section.
L-BFGS is a low memory version of Broyden–Fletcher–Goldfarb–Shanno algorithm, which is popularly used in PINN-based problems due to better convergence guarantees.
In this section
[Table]
we investigate the effects of applying L-BFGS optimizer on top of Adam Optimizer on one example each of Kovasznay flow and two-dimensional coupled viscous Burger’s equation.
[Table]
[Table]
From Tables 19 and
[Table]
we observe
[Table]
applying L-BFGS on top of Adam optimizer improves the accuracy of PINNs by 1-2 orders of magnitude, whereas it improves the accuracy of PISNs and
PINSNs by 1 order of magnitude.
Additionally, for all output variables of interest in both the examples, we observe the accuracy of PISN improves by an order when the depth of the symbolic network is increased from 2 to 3, whereas there is no substantial improvement on further increasing the depth to 4.
This observation further extends to PINSNs, with PINSN of Depth 3 being an order of magnitude better than PINSN of Depth 2.
On an average, the best performing PINSNs are 4.71 and 4.98 times better than best-performing PINNs on Kovasznay flow (Navier Stokes) and two-dimensional Burger’s equation respectively.