Bayesian Spline Learning for Equation Discovery of Nonlinear Dynamics with Quantified Uncertainty
Anonymous Author(s)
AffiliationAddress email
Abstract
Nonlinear dynamics are ubiquitous in science and engineering applications, but thephysics of most complex systems is far from being fully understood.
Discoveringinterpretable governing equations from measurement data can help us understandand predict the behavior of complex dynamic systems.
Although extensive workhas recently been done in this field, robustly distilling explicit model forms fromvery sparse data with considerable noise remains intractable.
Moreover, quantifying and propagating the uncertainty of the identified system from noisy data ischallenging, and relevant literature is still limited.
To bridge this gap, we developa novel Bayesian spline learning framework to identify parsimonious governingequations of nonlinear (spatio)temporal dynamics from sparse, noisy data withquantified uncertainty.
The proposed method utilizes spline basis to handle thedata scarcity and measurement noise, upon which a group of derivatives can beaccurately computed to form a library of candidate model terms.
The equationresiduals are used to inform the spline learning in a Bayesian manner, whereapproximate Bayesian uncertainty calibration techniques are employed to approximate posterior distributions of the trainable parameters.
To promote the sparsity,an iterative sequential-threshold Bayesian learning approach is developed, usingthe alternative direction optimization strategy to systematically approximate L0sparsity constraints.
The proposed algorithm is evaluated on multiple nonlinear dynamical systems governed by canonical ordinary and partial differential equations,and the merit/superiority of the proposed method is demonstrated by comparisonwith state-of-the-art methods.
Introduction
In the realm of science and engineering, dynamical systems are ubiquitous.
However, in actualcircumstances, the governing equations behind complicated dynamics may not be completely understood, preventing researchers from developing first-principled models.
On the other hand, theever-increasing data availability opens up new avenues for scientists to identify predictive modelsfrom enormous observation data, a process known as system identification (SI).
Recent advances indeep learning have prompted the rapid development of powerful SI models for high-dimensionalproblems using deep neural networks (DNNs).
Many DNN-based SI models have been proposed tolearn differential operators for complex (spatio)temporal physics from data and shown good potentialin terms of data reproduction and state prediction [1–3].
However, deep learning models usuallylack interpretability and are difficult to comprehend.
Furthermore, it is questionable in terms ofgeneralizability when compared to first-principle models, as such black-box DNN models provideless insight into the underlying processes.
Instead of identifying a black-box model, we focus on extracting analytical equation forms from data,which has higher interpretability and has the potential to advance our knowledge of unknown physics.
The sparse identification of non-linear dynamic (SINDy) algorithm [4] is an excellent developmentalong this path.
The central idea is to use sparse linear regression to uncover parsimonious governingequations from a dictionary of basis functions constructed by data, where the sparsity is promotedby pruning out redundant terms based on certain specified thresholds [5].
SINDy, although showinggreat promise, faces several challenges: (1) it heavily relies on high-quality data to extract derivativeinformation, which is typically based on finite difference (FD) methods, making it impossible tohandle incomplete, scarce, or noisy data; (2) it is formulated in a deterministic fashion and cannotaccount for the uncertainties from multiple sources, which is critical for real-world applications wheredata is frequently corrupted and can be very sparse.
In the past few years, the SINDy framework hasbeen further improved in various aspects to address these challenges, e.g., enhancing the library [6]or using deep learning for denoising and derivative computation by fitting the noisy data in adecoupled [7–9] or coupled manner [10–14].
For uncertainty quantification, the dictionary-basedequation discovery algorithms have been recently extended to Bayesian settings [15–17], based onthe idea of sparse Bayesian learning pioneered by Tipping and co-workers [18–22].
Despite recent progress and extensive work in this field, reliably distilling explicit equation formsfrom very sparse data with significant noise remains an unsolved challenge.
There are still significantgaps in handling data scarcity and noise, quantifying and reducing multi-source uncertainties, andpromoting sparsity, which most existing equation discovery techniques struggle to simultaneouslyaddress.
To this end, we propose a novel Bayesian Spline Learning (BSL) approach to identify parsimonious ordinal/partial differential equations (ODEs/PDEs) from sparse and noisy measurements;meanwhile, the associated uncertainties are quantified.
To deal with data scarcity and measurementnoise, the proposed BSL uses a spline basis, on which a collection of derivatives can be reliablycomputed for the library construction.
The posterior distributions of spline-based model parametersare approximated by a stochastic gradient descent (SGD) trajectory-based training scheme, wherethe first moment of SGD iterations is computed by stochastic weight averaging approach [23].
Theproposed BSL approach is effective in two aspects.
On the one hand, the spline-based representationcan help to interpolate locally the spatiotemporal field and perform differentiation analytically.
As aresult, it considerably enhances learning efficiency in cases with sparse and noisy data.
The posteriordistributions of spline, library, and equation coefficients are estimated simultaneously, without addingtoo much overhead to the training process.
Besides the measurement uncertainty, the model-formuncertainty is also obtained in the proposed method, which can be used for downstream Bayesian dataassimilation, where online data is assimilated to improve the predictability of the identified systemfor chaotic scenarios
Related Work
The data in real-world circumstances is frequently sparse in spatial/temporal domains and maycontain considerable noise, posing significant challenges to SINDy or its variants [4–6, 16, 24–27].
Deep learning (DL) has been leveraged as a superb interpolator for concurrently generating metadataand smoothing high-frequency noise [28], effectively improving the performance in identifyingequations from imperfect measurements [7, 10–14].
For example, automatic differentiation (AD)is effective for computing derivatives analytically from sparse, noisy data using a point-wise multilayer perceptron (MLP) [13, 29, 30].
However, it is difficult to impose locality constraints in thepoint-wise formulation.
Wandel et al.
[31] demonstrated the merit of employing a spline basis foranalytically calculating derivatives while enforcing the locality of spatiotemporal fields.
Owing to thesuperiority of spline-based differentiation compared to numerical discretization, physics-informedspline networks remarkably outperform point-wise physics-informed neural networks (PINN) in thecontext of solving PDEs [31] and discovering ODEs [14].
However, these models fail to simultaneously deal with ODE and PDE systems when the data is sparseand substantially corrupted.
For the PDE datasets, for example, the PINN-based sparse regression(PINN-SR) [13] works admirably, but it fails to converge on ODE datasets.
On the other hand,algorithms such as physics-informed spline learning (PiSL) [14] that performs very well for ODEdiscovery cannot handle PDE problems due to the limitation of the B-spline basis adopted.
Moreimportantly, all these SOTA sparse learning algorithms are formulated in deterministic settings anduncertainties introduced from data/library imperfection cannot be quantified.
For inverse problems like equation discovery, it is natural to use Bayesian framework to quantify andanalyze the prediction uncertainty.
To enable the posterior computation for high-dimensional trainableparameters that are intractable by traditional Bayesian inference, people have resorted to variousapproximation strategies, e.g., variational inference [32–34], Monte Carlo dropout [35], Bayes bybackprop [36], Laplace approximation [37], and deep ensemble approaches [38].
Although thesetechniques have had great success, training may still be challenging, and costs will rise dramatically asthe problems become more sophisticated.
Alternatively, we chose to employ an SGD trajectory-basedalgorithm, Stochastic Weight Averaging-Gaussian (SWAG) [23], where the information in the SGDtrajectory is exploited to approximate the posterior.
As demonstrated empirically, SWAC is wellscalable to high-dimensional problems and can accurately estimate uncertainty across many different
Bayesian learning tasks [23, 39].
Data assimilation (DA) has been widely used in numerical weather prediction (NWP) by fusing onlinesensing data into a predictive model for nonlinear dynamics forecast.
People have recently integratedeep learning into DA to improve online prediction performance [40–42].
However, the predictivemodel in DA is assumed to be known a priori , and uncertainties from the model and observations,which are required for DA, are usually hard to obtain.
In our framework, the predictive model can beunknown a priori and will be identified to assimilate additional data for online forecasting.
Moreover,instead of arbitrarily guessing the observation and model-form errors, this information can be learnedin the proposed BSL framework, as the data and model-form uncertainties are quantified.
The UQcapability of BSL naturally integrates the equation discovery with Bayesian DA techniques.
The main contributions of this work are three-fold: (1) we extended spline learning for sparseequation discovery of spatiotemporal physics governed by ODEs or PDEs; (2) we developed sparsitypromoting Bayesian learning for UQ tasks; and (3) the proposed BSL framework can seamlesslyintegrate with Bayesian data assimilation techniques to improve online dynamics forecasting.
Methodology
Let us consider a dynamical system, which is governed by a parametric ODE/PDE system in thefollowing general form
[Equation]
where u denotes the state vector, for ODE systems,
u = [ u 1 ( t ) , u 2 ( t ) , · · · , u d ( t )] T ∈ R
d dependsonly on time t , for PDE systems, u ( x , t ) =
[ u 1 ( x , t ) , u 2 ( x , t ) , · · · , u d ( x , t )] d ∈ R d depends onboth time t and space x , and F : R d → R d represents unknown nonlinear functions.
The states areobserved at discrete times { t i } ni =1 and at spatial locations { x j } sj =1 for PDE systems.
Let ˜ u denotethe noisy observation vector, the observation set is ˜ U
= { ˜ u ( t 1 ) ,
˜ u
( t 2 ) , · · · , ˜ u ( t n ) } T ∈ R n × d and˜ U = { ˜ u ( x 1 , t 1 ) , ˜ u ( x 2 , t 1 ) , · · · , ˜ u ( x s , t n ) } T ∈ R ( n × s ) × d for ODE/PDE systems, respectively.
Ourgoal here is to explicitly discover the parsimonious form of F ( · ) from a library of candidate basisfunctions and quantify the associated uncertainty given noisy observation data.
3.1 Overview
Given the dataset ˜ U , the SI problem can be solved by sparse regression techniques with a predefinedlibrary Φ ( u ) of m basis functions, [Equation]
[Equation]
) where ϕ i : R d → R , 1 ≤ i ≤ m denotes a basis function, which, for instance, can be the polynomialor trigonometric function.
Hence, the matrix of library terms evaluated at observed states is definedas,
[Equation]
Given the library, the overview of the BSL framework is shown in Fig.
Firstly, a spline-basedmodel is constructed to represent state variables and their derivatives by denoising the observation
[Figure]
W ⋆ , B , P EnKF
[Figure]
L ∝ priors + ( ˜ U − N m θ ) T B −1 ( ˜ U − N m θ ) +( · N c θ − Φ ( N c θ ) W ⋆ ) T P −1 ( · N c θ − Φ ( N c θ ) W ⋆ )
L ′ ∝ priors +( · N c θ − Φ ( N c θ ) W ) T P −1 ( · N c θ − Φ ( N c θ ) W ) , Φ ( θ ) · U ( θ ), P
W ⋆ n × d = W m × d m ≤ n = d u 1 dt d u 2 dt · U … 1 u 1 u 2 uΦ w 1 w W Stage1 ADO Itr = 1 {+Stage
 ++} 1 Stage
[Figure]
1-D basis 2-D basis data, which is illustrated in the blue box and discussed in Sec.3.2.
Then the spline-based model andsparse regression are trained simultaneously by alternating direction optimization (ADO), as shownin the red box.
Specifically, a single ADO iteration contains two sub-processes, where sub-process
I trains the spline-based model using log posterior loss and passes the trainable parameters andthe noise estimations to sub-process II, which uses a Bayesian SINDy-like method to prune outredundant terms in the library as defined in Eq. 3, and then updates the number of relevant terms in thetraining loss for sub-process I. After several ADO iterations, the parsimonious form of the governingequations and the posterior distribution for the coefficients will be estimated.
More details about the
ADO iterations in the Bayesian framework are shown in Sec.
3.3 and supplemental materials.
Finally,with the estimated posterior, the predictive uncertainty can be quantified by evaluating the identifiedsystem with an ensemble of parameters.
To further improve the prediction capability, especially forchaotic systems, we propose to leverage data assimilation techniques, which is shown in the greenbox and discussed in Sec.3.4 and supplemental materials.
3.2 Spline-based learning
Several previous works have already shown the potential of spline-based learning and demonstratedthe advantages compared with the classical DL structures, e.g.,(MLP, CNN) [14, 31, 43].
Therefore,we use this structure to smooth the solution fields based on the noisy measurement and then identifythe underlying PDE/ODE and calculate the corresponding derivatives from the spline-reconstructedfields.
The B-spline curves, based on the De Boor’s algorithm, are defined in a recursive way as:
[Equation]
where τ s is the location of knots, k is the degree of polynomial.
When k = 3 , it is the well-used
Cubic-B Spline curve.
With the defined basis, the spline interpolation can be write as y ( t ) =
Σ r + k − 1 s =
N s,k ( t ) θ s .
And the number of control points (trainable weights) θ ∈ R r + k is chosen empirically.
It can be proved that the derivative of p order B-spline basis is a function of p − 1 order B-spline, written as:
[Equation]
The proof of Eq.
5 is attached in supplemental materials.
The higher-order derivative can be calculatedby recursively using Eq. 5.
The analytical derivatives of spline basis are very beneficial for PDEdiscovery tasks since it always involves constructing library terms containing high order derivatives.
With a proper order p , the first p derivatives are accurate, and there is no error introduced duringthe derivation, as opposed to using the numerical methods to approximate derivatives.
The splinefunction and its derivatives are defined in a one-dimensional scenario.
It is straightforward to extendto n-dimension by direct using tensor-product.
For example, for a two-dimensional problem withspatial-temporal fields, the basis can be defined as
[Equation]
Here the two-dimensional basis is denoted by a different style to write it compactly.
Similarly, thepartial derivative for two-dimensional basis is defined as:
[Equation]
With the definition of sparse system identification and spline reconstruction, the whole spline learningcan be stated as follows: given noisy measurement data ˜ U , find the best sets of weights θ and
W so that data fitting loss and the weakly physics-informed loss can be minimized under sparsityconstraints , as shown in Eq. 8:
[Equation]
Here, N m and N c denote the spline basis matrices evaluated at measurement and collocation locations.
Moreover, N m and N c are numbers of measurement data and collocation points.
Furthermore, thealternating direction optimization (ADO) shown in previous works [13, 14] can be adopted tominimize the loss function efficiently, and the details are attached in supplemental materials.
3.3 Sparse system identification in Bayesian formulation
Bayesian methods provide a natural probabilistic representation of uncertainty, which is crucial formodel predictions.
System identification from noisy and sparse measurements generally contains twotypes of errors (similar to the hidden Markov model): (1) Observation error, where the data is noisy,and the smoothed data is reconstructed through spline-based learning, as shown in Eq. 9˜
[Equation]
where ϵ 1 represents the observation error.
(2) Evolution error or model form error, since the discoveredsystem cannot be exact due to library imperfection and needs to be reformulated as,
[Equation]
where ϵ 2 represents model form error.
And Eq.
(10) is evaluated on collocation points.
In this work, these error terms are modeled as zero-mean multivariate Gaussian random variables:ϵ 1 ∼ N ( 0 , B ) and ϵ 2 ∼ N ( 0 , P ) .
And we further assume that these error covariance matrices B , Pare diagonal matrix with diagonal terms { b k } and { p k } with 1 ≤ k ≤ d and they are learn-ableparameters during training.
According to Bayes’ rule, the posterior can be written as:
[Equation] [Equation]
decomposed
[Equation]
Currently, we specify the prior for the linear coefficient matrix as a zero mean Gaussian distributionp ( W | α ) =
N ( W | 0 , α − 1 I ) with the hyper prior as a Gamma distribution p ( α ) =
Gamma ( α | a 0 , b 0 ) .
Similarly, we also define the prior for the spline trainable parameters as zero mean Gaussiandistribution with the hyper prior as another Gamma distribution p ( θ | β ) =
N ( θ | 0 , β − 1 I ) andp ( β ) =
Gamma ( β | a 1 , b 1 ) .
To account for the data uncertainty and process uncertainty, the diagonal covariance matrices B and P are also set as learn-able during the training.
And henceimproper uniform priors are used for B and P .
The likelihood consists of two parts, as:
[Equation] [Equation]
where
[Equation] [Equation]
[Equation]
Traditional Bayesian sampling approaches (i.e., Markov chain Monte Carlo methods) can be usually intractable and expensive, especially when the parameter dimensionality is high.
Therefore,researchers tend to use alternative approximation approaches.
Current work uses a stochastic gradientdescent (SGD) trajectory-based approach, Stochastic Weight Averaging Gaussian (SWAG) [23]algorithm to approximately sample from the posterior distribution.
This method approximates theposterior by collecting the parameters near the loss plateau after a sufficient number of the trainingsteps.
To further reduce the inference cost, we construct a subspace by finding the PCA componentsof the SWAG trajectories [44] and then draw samples in the subspace instead.
In terms of the lossfunction in the probabilistic model, we chose to maximize the log form of the posterior density Eq. 11,also by leveraging the ADO algorithm.
The sub-process I minimizes the log density function definedby Eq.
11-13 and the sub-process II adopts the Bayesian variants of SINDy algorithm modified from[17, 19].
The likelihood function in sub-process II has the same form as in Eq.
but with a differentsparsity promoting prior p ( W | A )
= m (cid:89)
j =
N ( W j | 0 , α ′− 1 j ) , where A =
[ α ′ 1 , α ′ 2 , ..., α ′ m ] T .
In a single
ADO iteration, the sub-process
I provides the updated θ and P to sub-process II for constructing thelibrary terms.
While the sub-process II shrinks the library terms and passes updated library/weightstructure W back to sub-process I.
The whole training requires multiple ADO iterations before itreaches the final balance, where no more terms will be pruned out in sub-process II.
The detailed
ADO algorithm is listed in supplemental materials.
After obtaining the approximated posteriordistribution, predicting uncertainty can be estimated by marginalizing out the model parameters.
3.4 Data assimilation for enhanced predictability
For a chaotic system, one notorious problem is that a slight perturb in any model parameters cansignificantly influence the prediction.
For example, in numerical weather prediction (NWP) taskswhere the researcher always needs to predict the behavior of chaotic weather systems, various dataassimilation (DA) techniques have been developed to assimilate the available data and the knownequations.
Kalman filter and its variants are successful mathematical tools for data assimilation.
Our Bayesian formulation, providing the observation error covariance matrix B and evolution errorcovariance matrix P , can be naturally incorporated with the Kalman filter frameworks to improve thepredictability of chaotic systems.
Current works choose the ensemble Kalman filter for the task, andmore background can be found in supplemental materials.
Experiment and Result
In this section, we first show the equation discovery and uncertainty quantification results for nonlinear
ODE systems.
We also show that incorporating DA techniques can improve the predicting ability forchaotic systems.
Finally, we present the PDE discovery results for several canonical PDE systems.
The first pedagogical ODE example is the Van der Pol oscillator, which is defined as dxdt = yand dydt = µ (1 − x 2 ) y − x with µ = 0 .
The data is corrupted with 5% noise, and the library t y x t xxx
[Figure]
[Equation]
coefficients (red lines − ) fall within the confidence interval withhigh probability.
The right sub-figures show the propagated ensemble results (blue lines − ) based onthe discovered equations, measurement data (green circles ◦ ), and the true state trajectories of x andy (red dashed lines −− ).
The result clearly shows that the ensemble predictions can cover the data,and the uncertainty range fluctuates around the truth state values.
The second ODE example is a chaotic system, Lorenz 96.
It is a simplified mathematical model foratmospheric convection, defined as:
[Equation]
t =t = 1. x =x x t t t x x t t =t = 9. x =[-X n and X n +
-]
[Figure] with periodic boundary conditions X − 1 = X n − 1 , X 0 = X n and X n +1 =
X 1 .
We chose n = 6 incurrent case and F = 8 for the forcing terms.
The measurement states variables are corrupted with5% Gaussian noise, and the library consists of polynomials of state variables up to 3 rd order.
Thediscovery result for these systems is shown in the lower blue box in Fig.
2 following the same layoutas the Van der Pol systems.
The upper left figure indicates that the sparsity structure can be identified,and the horizontal axis marks 24 parsimonious terms (4 terms for each state, 6 states in total) outof 84 library terms.
The detailed PDF plots show that posterior distributions still cover truth valueswith high probability.
Although the discovery result is quite accurate, the forward simulations ofidentified Lorenz 96 system would induce a large phase difference compared with truth trajectoriesdue to the chaotic nature of the underlying system (shown in Supplemental Materials).
To alleviatethe chaotic behavior and improve the predictability, we seamlessly coupled the ensemble Kalmanfilter (EnKF), a classical DA technique, to assimilate noisy measurements with the identified system.
The observation error matrix B and evolution error matrix P required for the EnKF scheme canbe directly passed from the Bayesian spline learning framework.
The right subplots show the DAresults for the 3 out of 6 state variables to save space.
The horizontal axis marks the evolving time,and the vertical axis represents the state variables.
In each subplot, the green shaded region indicatesthe time interval with available noisy measurement data (marked by green circles ( ◦ )).
And the redsolid-line ( − ) is the true L96 states.
Finally, the blue curves ( − ) are the ensemble predictions.
It canbe seen that the ensemble predictions inside the region with measurement data almost overlap with thetrue trajectory.
Furthermore, the ensemble uncertainty grows more significantly in the extrapolationregion, but the ensemble ranges still fluctuate around the true trajectory.
Note that the prediction forthe chaotic system with the DA process is much better than directly forward simulating the identifiedsystem, where the predictable interval is only about 1 second.
More discovery results are attached in
Supplemental Materials.
Then, we use the proposed BSL framework to identify classical PDE systems and show two ofthem in the main text.
They include advection equation ∂u∂t
+ {+∂u∂x = 
+}
and Burgers’ equation∂u∂t + u ∂u∂x − 0 .
5 ∂ 2 u [Equation] 2 = 0 .
The results are shown in Fig.
3, inside the red and blue boxes, respectively.
Inside each box, the leftmost contour plot shows the true state value in the spatiotemporal field, andthe right three sub-figures show the UQ results for different cross-sections from the left contourplot.
Specifically, the first 2 out of 3 subplots are sampled from different time instances, and the lastsubplot is sampled from a fixed coordinate point, which gives a clearer view of the UQ trend alongthe spatiotemporal field.
The red lines ( − ) represent the truth value, blue lines ( − ) are obtained byensemble predictions, and black stars ( ∗ ) represent sparse and noisy measurements.
The predictionfrom the identified PDE system is accurate in the spatiotemporal field, and the ensemble fluctuatesaround the truth state variables.
Finally we benchmark our proposed method with several state-of-theart discovery algorithms (PINN-SR [13], SINDy [5] and RVM [45]).
The error metric is defined as:
[Equation] [Equation]
where C Discovery are the non-zero mean prediction from the posterior distribution and C True are thetrue coefficients of the governing equations.
If the method fails to converge or cannot identify thecorrect parsimonious form, we will report the final result as Fail .
The errors are scaled by × 10 − [-
-]3 tohave a clear comparison.
Two additional metrics are defined as precision M P and recall M R andthey are defined in Eq.
19,where
the ⊙ symbols element-wise product of vectors and the l 0 norm isthe non-zero terms in a vector. It can be seen from Table 1 that our BSL method always performsbest, when the noise is significant ( > 5% ), demonstrating its robustness to noise.
The PINN-SR candiscover the PDE equation with corrupted data set, but it fails to predict accurate time trajectoriesfor ODE systems.
This is also reported in a relevant paper [46], that the plain MLP structure isnot satisfactory for predicting time series.
The SINDy method can behave much better when largehigh-quality data exist.
However, the SINDy method can easily fail when noise is significant.
Our
BSL requires fewer parameters and is easier to converge due to the enforcement of locality constraintsby the spline basis model.
Furthermore, the SINDy method requires much more data ( 100% ) totrain but is still vulnerable to data noise.
These benchmark cases show the potential of our BSL forequation discovery tasks. The full table with extended cases can be found in Tab. 3 in appendix. [Table] discovery
[Table]
Discussion and Limitation
In the current work, we developed a novel Bayesian spline learning (BSL) framework for equationdiscoveries from sparse and noisy datasets with quantified uncertainty. The proposed frameworkimproves the result in the following aspects.
Firstly,the spline basis can accurately interpolatesolution surfaces and analytically compute derivatives to form the candidate library, outperformingthe other benchmark methods based on the finite difference (FD) or auto differentiation (AD).
Theimprovement in performance can be accounted for two reasons; Compared to numerical methods FD,it can represent the derivatives with closed-form expression.
Compared to AD, it leverages the localsupport of spline basis, which gives a more physical inductive bias to the learning process.
In thecase of the Bayesian learning part, the weakly physics-informed likelihood function is constrainedto inform the model learning in a Bayesian manner,without introducing significant computationaloverhead.
During the training, we adopt a Bayesian sparsity-promoting ADO iteration strategy torecover the parsimonious governing equation as well as the posterior distribution of its coefficients.
Last but not least, the Bayesian DA is also integrated into the BSL framework to improve the onlinepredictability of the chaotic systems, which can potentially benefit real-world tasks like numericalweather forecasting (NWP).
The proposed framework is evaluated on discovering multiple canonical
ODE and PDE systems, and great superiority has been demonstrated in comparison with state-ofthe-art methods.
Similarly to all machine learning models, the current work should be used withcaution in order not to generate potential negative societal impacts, e.g., modeling natural systems.
Admittedly, this work still relies on a pre-defined library of candidate terms, and thus the identifiedsystem is largely limited to the functional space determined by the user-specified library.
In general,for library-based methods, how to design an inclusive but not unnecessarily large library a priori isimportant yet very challenging, which may require prior knowledge of the system to be identifiedand thus limit their applications for systems involving complex governing physics. Moreover, interms of the spline representation, the current work only uses a uniform displacement of knot anduses the tensor product for high dimensional problems.
It is not an optimal setting in general andlimits the method’s ability when the spatial dimension is higher. In future work, we aim to applymore advanced techniques, e.g., to find the best knot size and to reduce the computational cost aslisted in [47, 48]. We will also adopt a better spline definition as shown in [43], namely, definingspline kernel and only expanding the basis function in the subdomain. It would allow a significantreduction of trainable parameters and B-spline basis functions.
A more detailed implementation ofthe current work can be found in Appendix A.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See supplementalmaterials
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] It is included inthe supplemental materials
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See supplemental materials
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4 and supplemental materials(d)
Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental materials
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] It is general open source code available
in public for research purpose
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 Proof: analytical form of spline derivatives
Theorem 1. .
The derivative of B-spline curve can be analytically evaluated as:
[Equation]
where
[Equation]
Proof.
We will prove by the induction.
For the base case k =
[Equation]
we have
[Equation]
Let’s suppose that the formula holds for k up to n .
We will prove that this formula also holds fork = n + 1 .
By the definition in Eq.
4 and the chain rule
[Equation]
we can get that:
[Equation]
A.2 Spline representation
In this section, we give error bounds for spline representation.
For simplicity, we consider 1D scenarioand assume the target function u : [0 , 1] → R is periodic and defined on the unit interval Ω = [0 , 1] .
Consider a set of uniform knots Γ : 0 = τ 0 ≤ τ 1 ≤ · · · ≤ τ r + k = 1 with .
The space of k th degree{ N s,k } r + k − 1 s = splines is
S k
(Ω , Γ)
= { p | p ( t ) is a polynomial of degree k in each ( τ i , τ i +1 ) }
∩ C k − 1 (Ω) .
Spline interpolation seeks ˆ u ∈ S k (Ω , Γ) that satisfies u ( τ i )
=
ˆ u ( τ i ) ∀ 0 ≤ r + k .
Theorem 2 (Spline interpolation error bounds [49, 50]) .
Assume that u is periodic, for odd degreespline interpolation k , we have
[Equation]
Here C k +1 and D k +1 − l are constant parameters, which are independent of ∆ τ and u , andω ( u ( k +1) , ∆ τ ) =
sup | x − y |≤
[Equation]
| .
In the present work, we focus on using spline for smoothing noisy data.
It is essentially a nonparameteric estimation of function u from noisy data { t i , ˜ u i } ni =1 ,˜ u
i = u ( t i ) + η i with η i ∼ N (0 , δ 2 η ) .
We further assume that { t i } are uniformly sampled from Ω .
The unregularized smoothing process estimates spline basis coefficients
[Equation]
through minimizing ∥
˜ U − N m · θ ∥ .
Here ˜ U
= (cid:2) ˜
u 1 , ˜ u 2 , · · · , ˜ u n (cid:3) T and N m is the spline basis matrixevaluated at these measurement locations.
The optimal θ opt satisfies ˆ u ( t ) =
N ( t ) · θ opt , where ˆ u isthe spline interpolation of u , and N ( t ) =
[ N 0 ,k ( t ) , N 1 ,k
( t ) , · · · , N r + k − 1 ,k
( t )]
T denotes spline basisfunction vector.
Following [51], we have spline fitting error bounds, as following.
Theorem 3 (Spline fitting error bounds) .
Assume that u is periodic and the number of data n issufficient large, for odd degree spline interpolation k , we have
[Equation]
where C 1 and C 2 are constant and independent of n .
Proof.
Let us denote
[Equation]
We have
[Equation]
We will first prove that
[Equation]
For any θ , 1 n θ T N Tm N m θ is the Monte Carlo approximation of (cid:82) (cid:0) N ( t ) T · θ (cid:1) 2 dt , and hence
[Equation]
Bringing the following property of B-splines [52]
[Equation]
[Equation]
into Eq.
(27) leads to Eq.
Then we prove that
[Equation]
Since (cid:80) s N s,k ( t ) = 1 and ∥ e ∥ ∞ =
O (∆ τ k +1 ) , we have
[Equation]
Finally, combining Eq.
(25), Eq.
(26) and Eq.
(28) leads to
[Equation]
In our sparse Bayesian regression, in stead of solving the aforementioned minimization problem, wehave additional regularization terms.
A.3 Algorithms
In this section, we present detailed algorithms used in the present work, which include Bayesian Alternative Direction Optimization (ADO) Learning 1,
Sequential Threshold Sparse Bayesian Learning 2,and Ensemble Kalman Filter 3.
Input : Library Φ , spline basis N , time derivative of spline basis matrix ˙ N c , negative log form of equation
Eq.
11 L Output: Mean estimation: θ SWA , W SWA , B SWA , P SWA
Samples from posterior distributions: ˚ θ , ˚ W , ˚ B , ˚ P
Pretrain : for i = 1 : T Pretrain do
SDG optimization with the fixed library { θ i +1 ,
W
i
+1 ,
B i +1 ,
P ′ i +1 } =
arg min
L
Update library
{+W i +1 , P i +
+}
=
STSparseBayesian ( Φ , ˙
U
i +1 = ˙
N c
θ
i +1 , W ′
i
+1 , P ′
i +1 )
if L ( θ i +1 , W i +1 ,
B i +1 , P i +1 )
<
L ⋆ then L ⋆ =
L ( θ i +1 ,
W i +1 ,
B i +1 , P i +1 )
θ ⋆ , W ⋆ , B ⋆ , P ⋆
= θ i +1 , W i +1 ,
B i +1 , P i +1 else break endStochastic Weight Averaging-Gaussian (SWAG) for posterior approximation : θ SWA , W SWA , B SWA , P SWA
= θ ⋆ , W ⋆ , B ⋆ , P ⋆
With a constant learning rate for i = 1 : T SWAG do SGD update θ i , W i , B i , P i θ SWA , W SWA , B SWA , P SWA
= i
θ SWA
+ θ i
i + ,
i W SWA
+ W
i
i + , i B SWA
+ B i
i + ,
i P SWA + P
i
i + endCompute low-rank square root of empirical covariance matrices Λ θ , Λ W , Λ B , Λ P from { θ i − θ SWA } , { W i − W SWA } , { B i − B SWA } , { P i − P SWA } Sampling : ˚ θ = θ SWA + Λ θ ˚ z θ ˚
[Equation]
=
[Equation]
SWA
[Equation]
W ˚ z W
[Equation]
=
[Equation]
SWA
[Equation]
Λ B ˚
z B ˚ P = P
SWA + Λ P ˚ z P where ˚ z ∗ are the random samples from N (0 , I ) .
Input : Spline trainable parameter θ , library Φ ( θ ) , approximated derivative ˙
U , library weight W , and process error matrix P Output: Best solution library Φ ⋆ , W ⋆ , P ⋆
Initialize:
Threshold ϵ library terms p old
Flag
True while Flag do do endfor j = p do
W j
W end
Find the nonzero entries
W
, record the end
Input : ensemble number J , sampled weights { W j } Jj =1 , discovered dynamical model M ( • ; W ) , process noise covariance P , observation model h , observation ˜ U , observation noise covariance B
Output: Analysis ensemble trajectories
U ja ( t )
Forecast: U jf
( t i +1 )
=
M ( U ja ( t i ); W j ) + ϵ j 2 , ϵ j 2 ∼ ( 0 , P )
U f
( t i +1 )
=
1 J (cid:80)
Jj =
1 U jf
( t i +1 )
Analysis : U jh ( t i +1 )
=
h
( U jf ( t i +1 ))
U h
( t i +1 )
=
1 J (cid:80)
Jj =
1 U jh ( t i +1 )
C fh ( t i +1 )
=
{+1 J − 1 Σ Jj =
+}
( U if ( t i +1 )
− U f ( t i +1 ))
( U ih ( t i +1 ) − U h
( t i +1 ))
T
C hh ( t i +1 )
=
{+1 J − 1 Σ Jj =
+}
( U ih ( t i +1 )
− U h ( t i +1 ))
( U ih ( t i +1 ) − U h
( t i +1 ))
T + B
K ( t i +1 )
=
C fh (cid:0) C hh (cid:1) −
U ja ( t i +1 )
=
U jf ( t i +1 )
+ K
( t i +1 ) (cid:18) ˜
U ( t i +1 ) − U jh ( t i +1 ) − ϵ j 1 (cid:19) , ϵ j 1 ∼ ( 0 , B )
A.4 Training Details
Additional training hyper parameters used in Sec. 4 is shown in the Tab. 2.
[Table]
A.5 Additional Result
We list additional discovery and UQ results in this section.
Fig.
4 shows 4 distributions of thecoefficients for Van der Pol system in red box and Lorenz 96 system in blue box.
Fig.
5 showsadditional UQ result for all the 6 state variables for Lorenz 96 system, incorporating EnKF algorithms.
Lastly, Fig.
7 shows additional UQ results for advection results in red box and Burgers’ results inblue box.
A.6 Additional Discussion: PDE with source term
In this section, we chose to investigate the discovery performance when a source term exists in thegoverning equation.
For example, we chose to discover Burgers’ equation with a source term, withthe form:
[Equation]
To account for the source term, we add source terms to the library, as shown in Eq.
[Equation]
The discovery result is with 20% Gaussian noise is shown in Tab.
The error is calculated by thesame metric in Eq.
17 It can be seen that our model is more accurate than the PINN-SR model, withapproximately the same number of training parameters and the training time.
The SINDy algorithmfailed at this large and cannot give the parsimonious form of the governing equation.
Finally thecontour plot and the cross section result are shown in Fig.
A.7 Additional Discovery result
In this section, we list the full table that includes all the experiments made for current work, asattached in Tab. 3.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
A.9 Implementation detail for the spline
In the current work, we apply direct tensor product to extend spline for solving spatial-temporal fieldand the relevant statistics are attached in Tab. 7, where the control points (trainable weights) θ in 1-dis marked by red and the total number of parameters are listed in the last column.
We only store thenon-zero elements for two-dimensional basis to leverage the sparsity (local support) of the spline.
However, we must claim that it is not an optimal way to
[Table]
A.10 Relevant Terminologies
[Table]
[Table]
to intrinsic randomness by nature,
[Table]
is irreducible
[Table]
lack of knowledge
[Table]
can be
[Table]
by
[Table]
more information
[Table]