Physics-Guided Discovery of Highly Nonlinear Parametric Partial Differential Equations
Abstract
Partial differential equations (PDEs) fitting scientific data can represent physical laws with explainable mechanisms for various mathematically-oriented subjects.
The data-driven discovery of PDEs from scientific data thrives as a new attempt to model complex phenomena in nature, but the effectiveness of current practice is typically limited by the scarcity of data and the complexity of phenomena.
Especially, the discovery of PDEs with highly nonlinear coefficients from low-quality data remains largely under-addressed.
To deal with this challenge, we propose a novel physics-guided learning method, which can not only encode observation knowledge such as initial and boundary conditions but also incorporate the basic physical principles and laws to guide the model optimization.
We empirically demonstrate that the proposed method is more robust against data noise and sparsity, and can reduce the estimation error by a large margin; moreover, for the first time we are able to discover PDEs with highly nonlinear coefficients.
Introduction
Partial differential equations (PDEs) are ubiquitous in many areas, such as physics, engineering and finance.
PDEs are highly concise and understandable expressions of physical mechanisms, which are essential for deepening our understanding of the world and predicting future responses.
The discovery of some typical PDEs is considered as milestones of scientific advances, such as the Navier-Stokes equations and Kuramoto–Sivashinsky equations in fluid dynamics, the Maxwell’s equations and
Helmholtz equations in electrodynamics and the Schrödinger’s equations in quantum mechanics.
Nevertheless, there are still a lot of unknown complex phenomena in modern science such as the micro-scale seepage and turbulence governing equations that await PDEs for description.
Traditionally, PDEs are mainly discovered by: 1) mathematical derivation based on physical laws or principles (e.g., conservation laws and minimum energy principles); and 2) analysis of experimental observations.
With the increasing dimensions and nonlinearity of the physical problems to be solved, the PDE discovery is becoming increasingly challenging, which motivates people to take advantage of machine learning methods.
Pioneering works [1, 2] use symbolic regression to reveal the differential equations that govern nonlinear dynamical systems without using any prior knowledge.
More recently, the representative SINDy[3] and STRidge [4] algorithms are proposed assuming that the dynamical
Coefficients
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
systems are essentially controlled by only few dominant terms.
Through sparse regression, feature selection from candidate terms is performed to estimate the PDE model [5, 6].
Further attempts make use of observation knowledge such as boundary conditions of PDEs [7–9] and low-rank property of scientific data [10], which greatly reduce the amount and quality of data needed for PDE discovery.
Although the aforementioned works show promise in discovering PDEs with constant coefficients (PDEs-CC) as shown in Fig. 1(a) and some simple instances of parametric PDEs (PDEs with variable coefficients, PDEs-VC) as shown in Fig. 1(b)-(c), they do not yet suffice to discover more complex
PDEs (e.g., PDEs with highly nonlinear coefficients) from scarce and noisier data.
An example of highly nonlinear coefficients is the permeability random field [11, 12] shown in Fig. 1(d) for the spatial derivative terms in the PDEs of the seepage problems.
Moreover, the PDEs obtained purely based on data-driven methods can only minimize the estimation error, but these methods may not consider the satisfaction of physical principles, such as the conservation of energy, momentum, etc.
To address these challenges, we rethink how the traditional PDE discovery works.
Based on physical principles, scientists ensure that a newly discovered PDE aligns with the physical world.
For example, the Navier-Stokes (NS) equation originates from the conservation of momentum, thus each term can relate to a certain physical meaning like convective accumulation or viscous momentum.
Inspired by this, we propose a physics-guided learning framework that not only uses observation knowledge such as initial conditions and assumed terms for certain problems but also uses basic physical principles that are universal in nature as learning constraints to guide model optimization.
Under this framework, a spatial kernel sparse regression model is proposed considering the principles of smoothness (as a first principle in PDEs) and conservation to impose smoothing of adjacent spatial coefficients for discovering PDEs with highly nonlinear coefficients.
Experimental results demonstrate that the proposed method can increase the overall accuracy of PDE estimation and the model robustness by a large margin.
In particular, we consider the discovery of
PDEs of different structure complexities with comparisons to baselines.
Our method can discover the PDE structures of all instances that align well with the existing physical principles, while other baselines yield false equation structures for some complex PDEs with excessively high estimation errors.
In summary, our contributions are:
• We propose a novel physics-guided framework for discovering PDEs from sparse and noisy data, which not only encodes observation knowledge but also incorporates physical principles to decrease errors and alleviate data quality issues.
We propose a spatial kernel sparse regression model that considers conservation and differentiation principles.
It presents excellent robustness in spite of noise compared to previous baselines, and can apply to sparse data in continuous spaces without fixed grids.
• We report experiments on representative datasets of nonlinear systems with comparison to strong baselines.
The results show that our method has a lower coefficient estimation error and can discover all the test PDEs with variable coefficients even when the data is extremely noisy while previous baselines cannot.
We also show that the discovered PDEs align well with existing physical principles and can reflect physical meanings.
Preliminary 2.1 Problem description
A physical field dataset u ( x, y, t ) is defined with respect to some input coordinates ( x, y, t ) , where x ∈ [1 , ..., n ] and y ∈ [1 , ..., m ] are spatial coordinates and t ∈ [1 , ..., h ] is a temporal coordinate.
An example of physical field data is shown in the observation data in Fig.
We consider the task of discovering two kinds of PDEs: (1) PDEs with constant coefficients, PDEs-CC; and (2) PDEs with variable coefficients, PDEs-VC.
For simplicity, partial derivative terms are denoted by forms like u x and u xx , which are equivalent to ∂u∂x and ∂ 2 u ∂x 2 .
The time derivatives such as u t (i.e., ∂u∂t ) of a PDE nearly always exist [13], therefore we follow prior works and set u t as the regression label.
Let p denote the number of partial derivative candidate terms considered in the task.
Definition 1 (PDEs with constant coefficients, PDEs-CC) .
PDEs-CC are the simplest PDEs, whose coefficients ξ i are fixed along all coordinates:
[Equation]
Definition 2 (PDEs with variable coefficients, also known as parametric PDEs or PDEs-VC) .
The coefficients of PDEs-VC are changing in some dimensions, e.g., the spatial dimensions:
[Equation]
A simple example of explicit function is ξ i ( x, y )
=
sin x + cos y and other ξ i ( x, y ) may be anistropic random fields [14] that are hard to express by explicit functions.
We can see that a PDE has two parts: the set of Θ( u ) i for ∀ i is the PDE structure, while the set of ξ i ( x, y ) for ∀ i is the PDE coefficients.
Here, each Θ( u ) i represents a monomial basis function of u or the combination of two monomial basis functions of u .
We consider monomial basis functions only up to the third derivative since higher-order derivatives can be inaccurate due to differential precision [4].
In Eqs.(1-2), the coefficient ξ ( x, y ) changes w.r.t.
spatial coordinates x and y .
In this paper, we discuss the case of spatial variations.
If the task is to capture variations in the temporal dimension, we can simply replace ξ ( x, y ) with ξ ( t ) .
Accordingly, the goal of PDE discovery is to determine:
• Terms : which coefficient ξ i is nonzero so that the term Θ( u ) i exists in the PDE structure; • Coefficients : the exact values of all nonzero coefficients at each spatial coordinate.
Naturally, the accuracy of coefficient estimation would affect the correctness of determining which coefficient is nonzero.
This coupling motivates us to choose methods that can perform structure learning and coefficient estimation simultaneously (e.g., sparse regression).
Moreover, since the simplicity of PDE is important, we are looking for the PDE with the fewest terms.
For example, u t = u x is simpler than u t =
u x + u y under similar data fitting.
2.2 Sparse Regression for PDE Discovery
Sparse regression is widely adopted in previous works to estimate both the terms and coefficients of PDEs.
For parametric PDEs with variable coefficients across spatial dimensions, many linear regressions are separately performed for coefficients at different spatial coordinates ( x, y ) :
[Equation]
where Y =
[ Y 1 , Y 2 , ..., Y h ]
⊤ ∈ R h denotes u t of all the h samples along the temporal dimension,
X ji denotes Θ( u ) i of the j -th sample in X ∈ R h × p , W =
[ W 1 , W 2 , ..., W p ] ⊤ ∈ R p denotes all the coefficients ξ i of the p candidate terms, and ϵ denotes the inevitable noise in data.
The above expression describes the scheme where we aim at discovering one PDE from one physical field u , which can also extend to the discovering of multiple PDEs from multiple physical fields.
Here, Eq. and Eq.4 repeat n × m times along the spatial dimensions x and y to get every (cid:99) W [ x,y ] . [Figure] （ a ） （ b ）
2.3 The Challenge of Coefficient Estimation
Though the above methods have been effective for some PDEs with simple variable coefficients [5, 15, 16, 13, 6], they still have difficulty in discovering PDEs with highly nonlinear coefficients due to overfitting.
To illustrate this, we use the mean absolute error (MAE) to measure the error of target ( u t ) fitting across training, development, and test sets.
With the correctness of PDE structure and accurate coefficient estimation, we shall obtain low target fitting MAE on test sets.
As shown in Fig.
2 (a) and extensively mentioned in the literature [4, 14, 10], many physics observations are linearly dependent along the temporal dimension since the coefficient fields that determine the observation are not changing along time.
Linear-dependent observations make the linear equation
Y = XW with rank( X ) ≤ p an underdetermined system that causes overfitting.
Furthermore, data sparsity and noise also impair the data quality and exacerbate the problem.
Fig.
2 (b) shows that the estimated coefficients by baseline sparse regression models such as SGTRidge [5] are irregular and cannot match the ground truth, and the estimation of the target u t cannot generalize to test sets.
The overfitting deviates the model from searching for the correct coefficients and terms, despite its good performance on the training set.
Data details of Fig.
2 are shown in Section 4.3 and Appendix D.
Physics-Guided Spatial Kernel Estimation
While various PDE terms and coefficients could overfit the training data, scientists are only interested in the PDE that is interpretable in terms of physics and can stably describe the natural phenomena.
In this paper, we incorporate physical principles into the PDE learning model.
First, we consider smoothness [11, 14], which is a first principle as PDEs must involve computing derivatives.
A "first principle" refers to a basic assumption that cannot be deduced from any other assumption, which is the foundation of theoretical derivation.
Here, we state the local smooth principle in Definition 3 that ensures the basic accuracy of differentiation.
This aligns well with our observation of many physical data, such as the locally smooth coefficient fields in Fig.
1 and the ground-truth coefficients and data in Fig.
On the contrary, the coefficient estimation and data fitting of SGTRidge are irregular as shown in Fig.
2, because the estimation of coefficients is separate at each spatial point, which does not consider the smoothness of coefficients across spatial dimensions.
Naturally, we expect that a smooth nonlinear function on spatial dimensions can help model the nonlinear coefficients.
Definition 3 (Local smoothness) .
Given coefficient ξ ( x, y ) , the coefficients within a local area (with radius r ) can be considered as a k-Lipschitz continuous function.
Given the spatial distance of any two adjacent coordinates Dist = ∥ S ( x, y ) − S ( x ′ , y ′ )
∥ ≤ r where S ( x, y ) is the spatial coordinate vector, the slope of the coefficient function is bounded by α ≥ 0 as | ξ ( x,y ) − ξ ( x ′ ,y ′ )
|
∥ S ( x,y ) − S ( x ′ ,y ′ )
∥ ≤ α .
Considering the principles of smoothness, we propose a local kernel estimation in the sparse regression that correlates the coefficient estimation at each spatial coordinate to the adjacent coefficient estimation.
A spatially symmetrical kernel (i.e., spatial rotation invariance) for all coordinates (i.e., spatiotemporal translation invariance) would estimate coefficients with respect to conservation laws. [Figure] Discovered Coefficients Sparse
[Figure]
⋯ 𝒖 [Figure] Coefficient [Figure]
Kernel
[Figure]
Physics
[Figure]
Guided Learning
[Figure]
A schematic diagram of the physics-guided learning framework is shown in Fig.
We expect it to enhance the model robustness for learning PDEs with highly nonlinear coefficients.
We prove that the proposed sparse regression with local kernel estimation can reduce the coefficient estimation error and reduce the error caused by noise when the coefficient fields comply with the local smoothness principle, with theorems and proofs in the Appendix B. Furthermore, as long as the spatial coordinates of the coefficients are provided, this local kernel estimation is mesh-free for spatiotemporal data, so nonlinear coefficients can be modeled even with irregularly sparse data.
For each ( x, y ) , the proposed model considers all ( x ′ , y ′ )
that ∥ S ( x, y ) − S ( x ′ , y ′ )
∥ <
r to compute
[Equation]
(6) where [ x, y ] denotes the spatial coordinate of the estimated coefficient while [ x ′ , y ′ ] denotes each spatial coordinate of the adjacent coefficients.
r denotes the radius of the local area.
Here, W ∈ R n × m denotes the model parameters introduced in Eq.3, while Ξ ∈ R n × m is an intermediate tensor replacing
W to represent the estimated coefficients.
γ and q are both hyperparameters.
We denote the spatial coordinate vector as S ( x, y ) and denote the distance between two spatial coordinates ( x, y ) and ( x ′ , y ′ )
as ∥ S ( x, j ) − S ( x ′ , y ′ ) ∥ .
The proposed coefficient estimation only takes the spatial distance as input information and is thus mesh-free to apply to continuous spaces for use in real practices.
We use the local kernel estimated Ξ of spatially adjacent coefficients instead of W as the regression weight to optimize the model (cid:98) Y = X Ξ .
The learning of W at each spatial coordinate is dependent on adjacent X and Y , as W at each spatial coordinate participates in the calculation of all Ξ within the local area.
Therefore, the proposed method calculates adjacent coefficients with nonlinearity when performing sparse regression, which leverages the physical principles to enrich data information and address overfitting. Here we can choose Radius Basis Function (RBF) kernel as K .
Experiments 4.1 Experimental Setting
Setup.
Our experiments aim to discover PDEs terms and coefficients.
The proposed method is compared with PDE-net [15], Sparse Regression (we compare with SGTRidge [5] here; SINDy [3, 17] is also an example of sparse regression) and A-DLGA [6].
We split the first 30% data in the time axis as the training set, the next 30% data as the development set, and the last 40% as the test set.
We perform an additional experiment on the model robustness by adding Gaussian noise in
Appendix E. For each model on each dataset, we tune the hyperparameters, i.e. γ , q and λ , via grid search, so that it has the lowest target u t fitting error on the development (Dev, i.e. validation) set.
The algorithm outline and the implementation details are presented in Appendix C.
Datasets.
To test how well the proposed model performs on the discovery of PDEs-CC, we consider the Burgers’ equation, the Korteweg-de Vries (KdV) equation, and the Chaffe-Infante equation.
For
PDEs-VC, we consider the convection diffusion equation and the governing equation of underground seepage where two cases are spatiotemporal 2D PDEs with simple variable coefficients (see Fig.
1(c)) and five cases are spatiotemporal 3D PDEs with different highly nonlinear coefficient fields (see Fig. 1(d))
that are hard to express explicitly.
The data details are provided in Appendix D.
Evaluation Metrics.
We use three metrics for evaluation:
Recall of the discovered PDE terms compared to ground truth; 2.
The mean absolute error of coefficient ξ estimation; 3.
The mean absolute error of target u t fitting.
The recall rate of PDE terms and the coefficient estimation error indicate whether the discovered PDE is close to the ground-truth PDE that can generalize to future responses with the correct physical mechanism.
Target fitting error tests how well the discovered PDE generalizes to the target u t .
4.2 Results on PDEs with constant coefficients
We present the discovered PDEs with estimated PDE coefficients shown in the brackets and PDE terms discovered correctly.
We use irregular samples to simulate sparsity in the continuous space.
Burgers’ equation.
We consider the discovery of the spatiotemporal 3D Burgers’ equation with two physical fields u and v .
For the sparse regression, we prepare a group of candidate functions that consist of polynomial terms { 1 , u, v, u 2 , uv, v 2 } , derivatives { 1 , u x , u y , v x , v y , ∆ u, ∆ v } and their combinations.
Following the physics-guided learning, we set diffusion terms as known a priori.
The dimensionality of the dataset is 100 × 100 × 200 .
We irregularly sample 10000 data and add 10%
Gaussian noise.
The discovered PDEs are shown below and the coefficients are averaged for each term.
The recall, coefficient error, and target fitting error are shown in Table 1, which shows that our model performs well even with noisy, 3D, irregularly sampled data and multiple physical fields.
[Equation]
= 0 . u xx + 0 . u yy − 1(1 .
uu x − 1(1 .
vu y , v t = 0 . v xx + 0 .
v yy − 1(1 . 0097)
uv x − 1(1 .
vv y .
[Table]
Korteweg-de Vries (KdV) equation.
We consider the discovery of spatiotemporal 2D Korteweg-de
Vries (KdV) equation.
We prepare a group of candidate functions that consist of polynomial terms { 1 , u, u 2 } , derivatives { 1 , u x , u xx , u xxx } and their combinations.
The dimensionality of the dataset is 512 × 201 .
We irregularly sample 5000 data and add 10% Gaussian noise.
The discovered PDEs are shown in below and the coefficients are averaged for each term for the constant coefficient.
The recall, coefficient error and target fitting error are shown in Table 1.
u
[Equation]
Chaffe-Infante equation We consider the discovery of spatiotemporal 2D Chaffe-Infante equation.
We prepare a group of candidate functions that consist of polynomial terms { 1 , u, u 2 , u 3 } , derivatives { 1 , u x , u xx } and their combinations.
The dimensionality of the dataset is 301 × 201 .
We irregularly sample 5000 data and add 10% Gaussian noise.
The discovered PDEs are shown below and the coefficients are averaged for each term for the constant coefficient.
The recall, coefficient error, and target fitting error are shown in Table 1.
The coefficient of u xx is less accurate as u xx is very small.
[Equation]
[Figure]
4.3 Results on PDEs with variable coefficients
Convection diffusion equation.
We consider the discovery of the spatiotemporal 2D convection diffusion equation with two different variable coefficient fields.
We prepare a group of candidate functions that consist of polynomial terms { 1 , u, u 2 } , derivatives { 1 , u x , u xx , u xxx } and their combinations.
The dimensionalities of the two cases are all 100 × 251 .
We randomly sample 5000 data to simulate mesh-free sparsity and add 10% Gaussian noise.
The discovered PDEs are shown below and the relative coefficient errors of the two cases are less than 0.3%.
The recall of terms is 100% and the average u t fitting error for the two cases are 1 .
× 10 − 4 , and the coefficient estimation is visualized in Fig.
A3 in Appendix F.
Our model can discover the terms correctly and estimate coefficients accurately for the parametric convection diffusion equation from noisy and sparse data.
[Equation]
The governing equation of underground seepage.
We consider the discovery of the spatiotemporal 3D governing equation of underground seepage with five different highly nonlinear variable coefficients, namely 1-HNC, 2-HNC, ..., 5-HNC.
We prepare a group of candidate functions that consist of polynomial terms { 1 , u, u 2 } , derivatives { 1 , u x , u xx , u xxx } and their combinations.
The dimensionality of the fives cases are all 50 × 50 × 51 .
We irregularly sample 10000 data and add 5% Gaussian noise.
The discovered PDEs are shown below, and the relative coefficient errors are less than 1% for the five cases.
The recall of terms and target u t fitting errors are shown in Table 2.
Our model can discover the terms correctly from noisy and irregularly sampled sparse data and can generalize to future data for all the five cases with highly nonlinear coefficients.
The coefficient estimation of 1-HNC is visualized in Fig.
4 as an example, with the visualizations of more cases in
Appendix F showing that the relative coefficient estimation error is less than 1%.
[Equation]
The discovered PDEs contain convection terms and diffusion terms along spatial dimensions, which align well with the ground-truth PDEs of underground seepage derived from the conservation of mass and Darcy’s law [18].
On the contrary, all the other baselines render false PDE terms; the test target fitting errors of baselines are much larger than their training errors, reflecting overfitting.
The test fitting errors of our model are much smaller than baselines, showing that our model effectively reduces the estimation error.
To test the robustness of our model, we include results under noise from 5% to 20%.
In most previous works for PDEs-CC [4, 17, 8] or PDEs-VC [5, 15, 10], the model robustness to 5% or 10% noise is tested.
Comparatively, the noise scale we test for our model is fairly large.
We show that our model performs well for most cases under 10% noise.
When the noise level goes up to 20%, in many cases one out of four PDE terms discovered would be wrong as
[Equation]
We find that even under extremely large noise, our model can discover PDEs that can generalize well to future data on test sets, since uu x and u x are very similar when u is not rapidly changing along spatial dimensions, which is visualized in Fig.
A2 in
Appendix E. Moreover, we find that the model performs well in a wide range of hyperparameters, with details in Appendix E. Overall, our model shows excellent robustness against overfitting, especially with sparse and noisy data.
[Table]
Conclusion and Future Work
How to discover Partial Differential Equations (PDEs) with highly nonlinear variable coefficients from sparse and noisy data is an important task.
To address the overfitting of coefficients caused by data quality issues in previous baselines [Table] we propose a physics-guided spatial kernel estimation in sparse regression that aligns well with the local smooth principle in PDEs and conservation laws.
The proposed model incorporates physical principles into a nonlinear smooth kernel to model the highly nonlinear coefficients.
We theoretically prove that it strictly reduces the coefficient estimation error of previous baselines and is also more robust against noise.
With spatial coordinates of coefficients, the model can apply to mesh-free spatiotemporal data without grids.
In experiments, it demonstrates the ability to find various PDEs from sparse and noisy data.
More importantly, it for the first time reports the discovery of PDEs with highly nonlinear coefficients, while previous baselines yield false results.
Our model performs well with a wide range of hyperparameters and noise level up to 20%.
With the state-of-the-art performance, our method brings hope to discover complex PDEs that comply with the continuously differentiable and local smoothness principles to help scientists understand unknown complex phenomena.
In the future, how to avoid the intervention of correlated similar terms and improve the accuracy of differentiation remain important.
Our method works for PDEs that comply with the principles, but may remain intractable for more rarely complex coefficient fields. Also, how to discover equations without the prior knowledge of time-dependent target term is not discussed yet.
Appendix A Related Work
Dynamical system modeling Machine learning is widely leveraged to predict the future response of desired physical fields from data [19, 20].
As an alternative, scientists can also obtain the future response by solving a partial differential equation (PDE) that describes the dynamical system.
Early pioneering works [21, 22] using neural networks to simulate dynamical systems can date back to three decades ago.
More recent machine learning algorithms [23–26] can be mainly divided into two branches: the mesh-based discrete learning and the meshfree continuous learning of simulation.
Within the meshfree learning branches, pure data-driven approaches [27, 15] are mainly based on high-quality data and physics-informed approaches [7, 28–30] use physics knowledge to enhance models to adapt to noisier and sparser data.
Recent studies on neural operators [31, 32] also use neural networks to learn the meshfree and infinite-dimensional mapping for dynamical systems.
Within the mesh-based learning branches, convolutional networks are widely adopted [33, 34] to simulate PDEs for spatiotemporal systems [35–38].
The geometry-adaptive learning of nonlinear
PDEs with arbitrary domains [39–41] and the particle-based dynamical system modeling [42, 43] by graph neural networks rises as a promising direction.
Moreover, deep learning also renders giving symbolic representation of solutions to PDEs[44] possible and demonstrate higher accuracy [45–47].
Data-driven discovery Early trials for equation discovery in the last century [48] uses inductive logic programming to find the natural laws.
Two research streams have been proposed to search the governing equations.
The first stream aims at identifying a symbolic model [49] that describes the dynamical systems from data, which uses symbolic regression [1, 2] and symbolic neural networks [50, 51] to discover functions by comparing differentiation of the experimental data with analytic derivatives of candidate function.
The second stream is mainly to incorporate prior knowledge [52, 9, 8] and perform sparse regressions [4, 53, 3, 54, 35] to discover PDEs by selecting term candidates.
Evolutionary algorithm
[13] is also proposed to start with an incomplete library and evolve through generations.
While these algorithms only discover the equation structure for PDEs with constant coefficients, later works also start to work on the discovery of PDEs with variable coefficients.
For PDEs with variable coefficients, we need to determine their PDE structures (the partial derivative terms that form the PDE) and coefficients (the variable coefficients that multiply partial derivative terms in the
PDE)
at the same time.
Sequential Group Threshold Regression [5] combines coefficient regression and term selection to find PDEs with variable coefficients.
PDE-Net [15, 16], Graph-PDE [55] and
Differential Spectral Normalization [56] are proposed to use neural blocks such as convolution to discover the PDEs models.
In addition, DLrSR [10] solves the noise problem by separating the clean low-rank data and outliers.
A-DLGA [6] proposes to alleviate data linear dependency at the sacrifice of estimation error.
Up until now, the current state-of-the-art approaches have proven to discover some PDEs with variable coefficients, but the discovery of PDEs with highly nonlinear coefficients remains a challenge [5, 6, 16] due to the overfitting of the sparse regressions and data quality issues.
B Theoretical Analysis
In this section, we introduce the theoretical analysis to demonstrate the advantages of our model.
We provide several theorems in the following with proofs.
The proposed spatial kernel estimation (See Eqs.5 and 6) uses the spatial distance to estimate the probability density function of coefficients with nonlinearity.
To help understand its advantage, we first introduce a spatial averaging estimation with linearity here.
We intend to show that the spatial averaging estimation can have an estimation error upper bounded by the upper limit of coefficient difference between adjacent coordinates.
The coefficient estimation error without such spatial averaging estimation, on the contrary, has no upper bound (i.e., can be fairly large).
Furthermore, we will show that our proposed spatial kernel estimation has a lower estimation error than the spatial averaging estimation.
We also prove that the estimation error caused by noise can be greatly alleviated by the spatial kernel.
The analysis not only proves that our proposed spatial estimation strategy can reduce estimation errors but also demonstrates that the kernel estimation with nonlinearity is very suitable for the tasks.
The local averaging estimation is defined as follows.
For each ( x, y ) , the spatial averaging estimation considers all ( x ′ , y ′ ) that ∥ S ( x, y ) − S ( x ′ , y ′ )
∥ <
r to compute
[Equation]
where W has the same definition as used in the spatial kernel estimation, i.e. model parameters, as described by Eqs.
3 and 4.
We use (cid:98) Ξ ( avg ) to denote the estimated coefficients and ξ to denote the ground-truth coefficients.
To express its upper bound error, we introduce the Lipschitz continuity to express the local smoothness with a Lipschitz constant α ≥ 0 , as introduced in Definition 3.
Here, we consider the upper limit of coefficient difference between adjacent coefficients within the local area for all x, y, x’, and y’ as
[Equation]
According to the local smoothness principle stated in Definition 3, the coefficient variation along the spatial dimensions is an increasing function.
In the worse case, we have all the coefficients on only one side with differences approximating αr .
Therefore, the upper bound of estimation error is
[Equation]
While the spatial averaging coefficient estimation has a upper bound of coefficient estimation error, the spatially independent estimation in Eqs.
3 and 4 practiced by many baselines cannot guarantee to match the ground-truth coefficients even if Eq. 4 is optimized due to the existence of many linearly dependent observations.
We assume that the spatial averaging estimation can avoid this issue by using extra data from adjacent coordinates within the local area in the sacrifice of introducing the estimation error as described in Eq.
We can also easily demonstrate that the local averaging estimation has a lower estimation error than the strategy practiced in A-DLGA [6] that makes coefficients grids coarser by merging grids within each spatial area into one grid, which also uses extra adjacent data to alleviate the issue caused by linearly independent observations.
We formalize this in Theorem 1.
Theorem 1 (Reduction on coefficient error by local averaging estimation) .
With respect to the local smoothness principle, the coefficients estimated by the spatial averaging estimation has strictly lower upper-bound coefficient estimation error than A-DLGA.
Proof.
Assume that the Local smooth Principle in Definition 3 applies, we consider the estimation of coefficient ξ ( x, y ) .
We denote the estimated coefficients of A-DLGA as (cid:99) W † .
For the (cid:99) W † , the upper bound of estimation error should be the case where ( x, y ) locates at the edge of a local area, instead of at the center as in the spatial averaging estimation, so that the upper limit of coefficient difference between coordinates within the area would be 2 αr .
For each ( x, y ) , A-DLGA considers all ( x ′ , y ′ )
that ∥ S ( x ′′ , y ′′ ) − S ( x ′ , y ′ )
∥ < r , where ∥ S ( x ′′ , y ′′ ) − S ( x, y ) ∥ < r .
Therefore,
[Equation]
We further demonstrate in Theorem 2 that our proposed spatial kernel estimation with nonlinearity reduces the coefficient estimation error of the local averaging estimation with linearity.
Our model is more accurate than local averaging estimation, so it is more accurate than A-DLGA.
Theorem 2 (Reduction on coefficient error by local kernel) .
With respect to the local smooth principle, the coefficients estimated by the spatial kernel estimation has strictly lower coefficient estimation error than the spatial averaging estimation.
Proof.
Assume that the Local smooth Principle in Definition 3 applies, we consider the estimation of coefficient ξ ( x, y ) .
Because the coefficient function is a k-Lipschitz continuous function, the coefficient difference increases with spatial distance: the closer the coordinates, the smaller the coefficient difference.
The spatial kernel estimation is stated in Eqs.
5 and 6.
We denote the estimated coefficients as Ξ .
Note that the kernel value defined as K [ x ′ ,y ′ ]
= exp( −∥ S ( x,y ) − S ( x ′ ,y ′ )
∥
[Equation]
) in Eq.decreases in the spatial distance, so closer coefficients give more contributions.
Therefore, we have
[Equation]
The above equations and inequalities prove that the coefficient estimation error of spatial kernel estimation is strictly lower than the coefficient estimation error of spatial averaging estimation.
Moreover, the spatial kernel estimation reduces the coefficient estimation error caused by noise in the sparse regression, which is proved in Theorem 3.
This makes the PDE discovery more robust.
For coefficient at a spatial coordinate ( x, y ) , its estimation is affected by both the noise in its Y and the noises in Y at adjacent coordinates within the local area.
We only discuss the estimation error caused by noise here, so we assume (cid:98) Ξ
= (cid:99) W = ξ if η = 0 , ϵ = 0 , which means that the estimation of coefficients must be with a balance distribution of adjacent coefficients.
This allows us to discuss the noise effect alone without the error caused by kernel discussed in Theorem 2.
We prove that the weighted addition of independent Gaussian noises by kernel estimation has a lower error than the original sparse regression and the error is lower when more adjacent coefficients are considered.
Theorem 3 (Reduction on coefficient error caused by noise) .
Assume that the coefficient estimation error is only caused by noise
so that (cid:98) Ξ = (cid:99) W = ξ if η = 0 , ϵ = 0 , then we must have | (cid:98) Ξ − ξ | < | (cid:99) W − ξ | if η ̸ = 0 , ϵ ∼ η N (0 , σ 2 ) ∈ R h .
Proof.
Consider ϵ ∼ η N h (0 , σ 2 ) in estimation that (cid:99) W =
( X T X + λI ) X T ( Y + ϵ ) .
| (cid:99) W − ξ | =
( X T X + λI ) X T ϵ .
We assume (cid:98) Ξ
= (cid:99) W = ξ if η = 0 , ϵ = 0 , which means for each (cid:98) Ξ [ x ′ ,y ′ ]
there must always be another (cid:98) Ξ [ x ′′ ,y ′′ ] such that 1) they have the same distance to (cid:98) Ξ [ x,y ] so they have the same kernel value, i.e. ∥ S ( x ′ , y ′ )
− S ( x, y ) ∥ =
∥ S ( x ′′ , y ′′ ) − S ( x, y ) ∥ , and 2) they are symmetrical to the value of ξ [ x,y ] so that their biases can be offset, i.e. ξ [ x,y ] − ξ [ x ′ ,y ′ ]
i = ξ [ x ′′ ,y ′′ ] − ξ [ x,y ] .
Based on this, for all ( x ′ , y ′ ) that ∥ S ( x, y ) − S ( x ′ , y ′ )
∥ < r , the estimated coefficients should be
[Equation]
For each ϵ ( x ′ ,y ′ ) ∼ ηN (0 , σ 2 ) that is i.i.d., as (cid:80) ϵ ∼ ηN (0 , (cid:80) σ 2 i ) , we have | (cid:98) Ξ
[ x,y ] − ξ [ x,y ] | ∼ ηN (0 , (cid:80) ( x ′ ,y ′ )
K [ x ′ ,y ′ ]
σ 2 K [ x ′ ,y ′ ] ) .
As each σ is the same for all adjacent coefficients within the area, we have | (cid:98) Ξ
[ x,y ] − ξ [ x,y ] | ∼ η (cid:80) ( x ′ ,y ′ )
1 N (0 , σ 2 ) .
However, | (cid:99) W [ x,y ] − ξ [ x,y ] | ∼ ηN (0 , σ 2 ) .
Therefore, | (cid:98) Ξ − ξ | < | (cid:99) W − ξ | and | (cid:99) W − ξ | | (cid:98) Ξ − ξ | = (cid:80) ( x ′ ,y ′ )
C Algorithm Details C.1 Iterative One-Out Sparse Regression
We use an iterative one-out regression that filters out one X ; ,i which gives the least Akaike information criterion (AIC) at each iteration of coefficient estimation.
If we use M to denote the set of indexes of reserved coefficients, the formula of AIC is approximately used as follows
[Equation]
The iteration ends when there are only L coefficients left in the regression.
This aims to filter out the most irrelevant ξ i that maximize the least square errors to avoid its intervention in estimating coefficients.
The iterative one-out regression repeats
[Equation]
Iterative one-out regression is an approximation of sparse group regression that improves the accuracy in determining the nonzero ξ i without the interference of irrelevant terms, as a similar method has pointed out [5].
The overall algorithm is expressed in Algorithm 1.
Algorithm
The physics-guided spatial kernel sparse regression approach.
D Data statistics
We introduce the governing equation of underground seepage in the following.
The subsurface flows in the field of fluid mechanics with different coefficients are taken as (1-5)-HNCs to perform the experiments.
The governing equation for the data is:
[Equation]
where S s denotes the specific storage; K ( x, y ) denotes the hydraulic conductivity field; and u denotes the hydraulic head.
In our work, u is the physical field the desired PDE describes and K is the coefficient field.
This equation is also used by PDE-net [15] but its coefficient field is much simpler.
The two variable coefficient fields used in PDE-net are 0 .
5(cos( y ) + x (2 π − x ) sin( x ))
+ 0 .
6 and b ( x, y ) = 2(cos( y ) + sin( x ))
+ 0 . 8 .
The hydraulic conductivity field K ( x, y ) in the governing equation is set to be heterogeneous to better simulate real situations in practice.
The heterogeneous fields are often regarded as random fields with higher complexity following a specific distribution with corresponding covariance [11, 12, 14, 57].
In detail, in our paper a two-dimensional transient saturated flow in porous medium is considered.
The domain is a square, which is evenly divided into 51 × 51 grid blocks and the length in both directions is 1020 [L], where [L] denotes any consistent length unit.
The left and right boundaries are set as constant pressure boundaries and the hydraulic head takes values of H x =
=
[L] and Hx=1020=200 [L], respectively.
Furthermore, the two lateral boundaries are assigned as noflow boundaries.
The specific storage is assumed as a constant, taking a value of SS=0.0001 [L-1].
The total simulation time is 10 [T], where [T] denotes any consistent time unit, with each time step being 0.2 [T], resulting in 50 time steps.
The initial conditions are H t =0 ,x =
=
[L] and
H t =0 ,x ̸ =0 =
[L].
The mean and variance of the log hydraulic conductivity are given as 0 and 1, respectively.
In addition, the correlation length of the field is 408[ L ] .
The hydraulic conductivity field is parameterized through KLE and 20 terms are retained in the expansion.
Therefore, this field is represented by 20 random variables ξ = ξ ′ 1 ( τ ) , ξ ′ 2 ( τ ) , ..., ξ ′ 20 ( τ ) in the considered cases.
An example of conductivity field obtained through KLE is shown in Fig.
A1(a), which exhibits strong anisotropy.
MODFLOW software is adopted to perform the simulations to obtain the dataset, and the data distributions at two time steps are presented in Fig.
A1(b) and (c) as an example.
For the two convection diffusion equation cases, we simulate the two-dimensional PDEs u t = − 1 x u x + 0 .
1 u xx and u t = − 1 ( x +sin πx ) u x + 0 .
1 u xx with variable coefficients − 1 x and − 1 ( x +sin πx ) that can be expressed explicitly and do not have physical meanings or references in the nature, respectively.
These two equations are also generated by MODFLOW.
All the data used are available in the Supplementary Materials .
We use a single GPU machine of GTX1080 and the error variation to random seeds is statistically insignificant.
E Hyperparameter and Robustness Analysis
In addition to the main experiment and the robustness experiment, we also conduct a hyperparameter analysis to discuss the suitable range of hyperparameters to ensure model performance.
We set the radius r within [2, 5, 10] and set the γ value of the Gaussian kernel within [0.03, 0.1, 0.3, 1] to find out whether our model is stable over a wide hyperparameter range.
The γ value determines the variance of the kernel.
When γ → 0 , the kernel estimation is equivalent to the averaging estimation introduced in Appendix B. When γ → ∞ , the kernel estimation degrades to separate regression at each spatial coordinate.
We find that as the value of γ decreases, the coefficient error increases.
This reflects the gradual approximation to the local averaging estimation when the values of the kernel function at each coordinate are almost the same.
The increasing error aligns well with the Theorems 2 and 3 in Appendix B that local averaging estimation has larger estimation error.
Although a wide range of hyperparameters can all give the correct PDE structure, we can tune the hyperparameters for each specific case to obtain the best performance.
Table A5 shows that r = 5 , γ = 1 is the best hyperparameter setting for 1-HNC in the experiment.
The radius r here is the spatial distance that is normalized for each dataset.
These kernel functions that have tiny coefficient estimation error mainly rely on the most adjacent coefficients for estimation.
The optimal values of both r and γ are determined by the graininess of the local smooth principle in real practice.
If the radius is too large, the kernel will no longer be "local" to match the principle.
In all, our model performs well within a wide range of hyperparameters.
The hyperparameters we use in the main experiments are r = 10 , γ = 1 .
We show the full results of hyperparameter analysis of (1-5)-HNCs in the following.
[Table]
[Equation]
[Table]
[Equation]
[Table]
Hyperparameters
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
r = 2 , γ = 0 .
[Equation]
[Table] [Table]
[Equation] [Equation] [Equation]
[Table] [Table]
[Equation]
= 2 , γ =
[Equation] [Equation]
[Table] [Table]
r = 10 ,
[Equation]
[Table]
r = 10 , γ
[Equation] [Equation]
[Table]
[Equation]
=
[Equation]
γ = 0 .
[Equation]
[Table]
[Equation] [Equation]
[Table]
[Equation]
=
[Equation] [Equation]
[Table]
r = 5 , γ = 0 .
[Equation] [Table]
r
[Equation]
γ
[Equation] [Equation]
[Table]
r
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Equation]
[Table]
Hyperparameters
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Equation]
[Equation]
[Table]
[Equation]
[Equation]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation] [Equation] [Equation] [Table]
[Equation] [Equation]
[Equation]
r = 2 , γ = 0 .
[Equation]
like The main experiments are mostly conducted with up to 20% noise.
Here, we conduct a robustness analysis to discuss whether the proposed model can handle even larger noise of 25% and 30%.
We find that for some PDE cases with constant coefficients, our model can perform well even under 30% noise.
We report the discovered PDE structure and coefficient error of the 1-HNC dataset in Table
A6.
In the table, C-I equation denotes the Chaffe-Infante equation and C-D equation denotes the first case of convection diffusion equation.
From the table, we find that our model performs well under 30% noise for Burgers’ equation and KdV equation, which outperforms the robustness reported in previous baselines.
Compared to results in Table.
1, we find that the coefficient errors and fitting error of Burgers’ equation and KdV equation are close to the results under 20%.
However, for C-I equation and convection diffusion equation, the coefficient errors and fitting errors increases drastically since the discovered PDE terms are partially wrong.
The wrongly discovered PDE terms for C-I equation is { u, u 2 , u xx } instead of { u, u 3 , u xx } .
The wrongly discovered PDE terms for the first case of C-D equation is { u, u x } instead of { u x , u xx } .
We also show in Fig.
A2 that some terms such as u x and uu x are highly correlated and have similar distributions.
Correlated terms and coefficients like these increase are challenging for data-driven machine learning models to identify correct PDE terms.
[Table]
F More Experimental Results
In this section, we visualize the estimated coefficients and compare them with the ground-truth coefficients.
The visualized residual errors of the estimated coefficients show that the proposed our model is very accurate in coefficient estimation.
The results of 1-HNC is shown in Fig.
4 in the main manuscript, so we only show the results of the other four cases of the governing equation of underground seepage here and the two cases of the convection diffusion equation.
The estimated coefficients of all datasets are all obtained with the optimal hyperparameters tuned for each dataset.
[Figure]
[Figure] [Figure]
[Figure] [Figure]