D-CIPHER: Discovery of Closed-form Partial Differential Equations
Abstract
Closed-form differential equations, including partial differential equations and higher-order ordinary differential equations, are one of the most important tools used by scientists to model and better understand natural phenomena.
Discovering these equations directly from data is challenging because it requires modeling relationships between various derivatives that are not observed in the data ( equationdata mismatch ) and it involves searching across a huge space of possible equations.
Current approaches make strong assumptions about the form of the equation and thus fail to discover many well-known phenomena.
Moreover, many of them resolve the equation-data mismatch by estimating the derivatives, which makes them inadequate for noisy and infrequent observations.
To this end, we propose
D-CIPHER, which is robust to measurement artifacts and can uncover a new and very general class of differential equations.
We further design a novel optimization procedure, CoLLie, to help D-CIPHER search through this class efﬁciently.
Finally, we demonstrate empirically that it can discover many well-known equations that are beyond the capabilities of current methods.
Introduction
Scientists have been using mathematical equations to describe the world for centuries.
In particular, closed-form differential equations turned out to be one of the best tools to model physical phenomena.
A differential equation describes a relationship between a quantity and its derivatives (rates of change); it is called closed-form if this relationship is described by a mathematical expression consisting of a ﬁnite number of variables, constants, arithmetic operations and some well-known functions (e g., exponent, logarithm, trigonometric functions) 1 .
Closed-form differential equations provide a general description of reality in a concise representation that is amenable to closer inspection by scientists.
This renders them transparent and interpretable to human experts.
Discoveries of these equations required a thorough knowledge of the theory, strong mathematical skills, substantial creativity and good intuition.
The goal of this work is to discover closed-form differential equations directly from data thus accelerating the process of scientiﬁc discovery.
Challenges in discovering differential equations from data
• Partial and higher-order derivatives.
Many algorithms [10, 37] can only identify Ordinary Differential Equations (ODEs) which evolve only with respect to one variable (usually time).
In contrast, many natural phenomena are described by equations involving many variables (e.g., spatial coordinates) called Partial Differential Equations (PDEs) and higher-order derivatives. • Derivatives not observed.
Discovering differential equations from data is challenging because the derivatives are usually not observed in the dataset ( equation-data mismatch [37]).
This makes
verifying a candidate equation a non-trivial task.
Most of the methods proposed in the literature try to resolve this issue by estimating the derivatives [10, 42].
However, estimating the derivative is difﬁcult, especially when the data is sampled infrequently or with high noise [37, 30]. • Strong assumptions and constrained search space.
The majority of algorithms for identifying differential equations make many assumptions about the form of the equation.
In particular, they make the evolution assumption (deﬁned and explained later) and assume that the equation can be represented as a linear combination of prespeciﬁed functions and differential operators [10, 30]. However, many well-known equations, such as a forced harmonic oscillator or an inhomogeneous wave equation, cannot be represented in that way.
Currently a few algorithms tackle only some of these challenges.
In particular, Weak SINDy [30] is able to discover PDEs without estimating the derivative by utilizing a variational approach.
However, the form of the equation is constrained to be in a form amenable for a sparse regression algorithm.
DCODE [37], on the other hand, uses a variational approach in conjunction with a symbolic regression algorithm to discover closed-form ODEs.
However, it cannot handle higher-order derivatives or multiple independent variables, so it cannot be used to discover closed-form PDEs.
The algorithms that do not require the evolution assumption appeared in [28] and [19] but they require derivative estimation and only consider equations represented as linear combinations of prespeciﬁed functions.
Contributions.
In this work we develop the D iscovery of C losed-form P artial and H igher-order
Differential E quations in a R obust Way framework (D-CIPHER) that does not estimate the derivatives, requires fewer assumptions, has a broader search space than previous techniques and works for both higher order ODEs and PDEs.
Our contributions are as follows:
• We examine the landscape of different types of PDEs from the discovery perspective.
In particular, we introduce new notions such as evolution form , evolution assumption , derivative-bound part, and derivative-free part.
We use them to describe what kinds of PDEs can be discovered with current methods and to motivate our new class of differential equations.
(Section 2) •
We propose a new general class of PDEs ( Variational-Ready PDEs) that admit the variational formulation (and thus allows to circumvent the derivative estimation).
We also prove a theorem that motivates a novel objective function.
(Section 4) • We use the novel objective function to develop D-CIPHER, a new algorithm that searches over the Variational-Ready PDEs.
(Section 5) • We develop a new optimization procedure (CoLLie) to efﬁciently solve a constrained least-squares problem and thus help D-CIPHER search through this space efﬁciently.
(Section 6)
Partial Differential Equations
In this section, we provide background information about Partial Differential Equations and introduce new notions necessary for the following discussion.
Notation and deﬁnitions.
We denote the set { 1 , 2 , . . . , n } as [ n ] and the set of non-negative integers as N 0 .
Throughout this paper we let M, N, K ∈ N be some natural numbers and let Ω ⊂ R M be an open set inside R M .
A comprehensive table with all symbols used in this work can be found in
Appendix A together with some deﬁnitions restated more formally.
Going beyond ODEs.
The simplest differential equations are ODEs that describe quantities that evolve with respect to only one independent variable, usually time.
Most methods assume that the
ODE is explicit and as such can be represented as a system of ﬁrst-order ODEs:
[Equation]
where ˙
u j represents the derivative of u j .
Then the discovery problem is reduced to deciding the order of the derivative (usually ﬁrst or second) and the discovery of f j .
For PDEs, it is not enough to talk about the derivative, as we can take derivatives with respect to different variables.
We denote the mixed derivative as ∂ α , where α ∈ N M 0 is called a multi-index, and deﬁne it as ∂ α = ∂ α 1 1 ∂ α 2 2 . . .
∂
α M M .
Each ∂
α
i i =
∂
α
i /∂x α
i i is a α th i -order partial derivative with respect to x i (the i th independent variable) 2 .
We deﬁne the order of α as | α | = (cid:80) Mi
=1 α i .
We call ∂ α non-trivial if | α | > 0 .
A PDE of order K is any equation of the form
[Equation]
where ∂ [ K ] u are all non-trivial mixed derivatives of all u j ( j ∈ [ N ] ) up to the K th order.
We call a
PDE closed-form if f is closed-form.
As PDEs might include many different combinations of derivatives, there is no generally accepted counterpart of explicit ODEs in the space of PDEs.
Evolution Assumption.
Although there is no generally accepted notion of an explicit PDE, we deﬁne an evolution form of a PDE to be an equation of the form
[Equation]
where ∂ [ K ] / α is ∂ [ K ] with ∂ α omitted, α is a known multi-index and j ∈ [ N ] .
Note that if M =and | α | = K then Equation 3 becomes exactly the deﬁnition of an explicit ODE.
Many PDE discovery algorithms assume a particular evolution form [30].
We call it an evolution assumption (EA).
This assumption requires the knowledge of α and j which might not be trivial.
Usually, ∂ α is assumed to be the ﬁrst derivative with respect to time ( ∂ t ) [42] which is not true for many well-known PDEs such as the wave equation or Gauss’s law.
D-CIPHER does not need the evolution assumption.
Moreover, it can discover some PDEs that cannot be put into the evolution form.
Linear combinations.
Current PDE discovery algorithms [42, 30, 11] consider PDEs that can be represented as a linear combination of functions.
That means the PDE has the form
[Equation]
where θ p ∈ R for p ∈
[ P ] are the only constants that are optimized.
As there are lot of expressions that cannot be put in that form, these algorithms fail to discover more complex equations.
In particular, for an unknown θ ∈ R functions such as sin( θx i ) , e θx i or 1 x i + θ cannot be learned by these algorithms.
Unlike previous methods, D-CIPHER is not limited to PDEs that can be represented as a linear combination of functions.
To describe the family of PDEs that D-CIPHER can discover we need to introduce the notions of derivative-bound and derivative-free parts.
Derivative-bound part and derivative-free part.
Any PDE can be expressed in the form
[Equation]
where we collect all the terms with the derivatives into f ( x , u ( x ) ,
∂ [ K ] u ( x )) and all terms without the derivatives into g ( x , u ( x )) .
We call f the derivative-bound part and g the derivative-free part.
We also denote them as ∂ -bound
and ∂
-free .
The signiﬁcance of the ∂ -free part is that it can be evaluated directly given u , whereas the ∂ -bound part requires access to the derivatives which are not observed.
It is important to note that for ﬁrst-order ODEs, f is trivial and is just equal to ˙ u j .
The challenge of unobserved derivatives might put constraints on the ∂ -bound part of the PDE and thus in practice, it might not be possible to search over all closed-form PDEs.
However, we observe that no such constraints need to be put on the ∂ -free part (as it does not include the derivatives).
We take full advantage of this observation and search separately for the two parts.
We search over all closed-form functions g and for each candidate, we try to ﬁnd the best counterpart f among the allowed expressions.
This is very different from the previous approaches, which either do not need to ﬁnd f as they work only for ﬁrst-order ODEs [37] or they constrain equally both the ∂ -bound part and ∂ -free part to be a linear combination [30].
Related Works
A comparison of related works is shown in Table 1.
Full review of related works in Appendix G.
Data-driven discovery of physical laws is an established area of machine learning [6, 44].
The pioneering work in that area was SINDy [10] that constrained the space of equations to linear
Can it discover PDEs?
Does it avoid derivative estimation?
Is the evolution assumption unnecessary (Equation 3)?
Can it discover any closed-form ∂ -free part (Equation 5)?
[Table]
combinations of functions (Equation 4) from a predeﬁned library and used sparse regression to discover explicit ODEs.
It was later extended to include implicit ODEs [28] and PDEs [42, 43].
As these methods assume the PDE to be a linear combination, their search space is signiﬁcantly limited.
Recently, the variational approach has been used as a viable alternative to derivative estimation.
However, they have only been used for differential equations in a linear combination form [30, 31, 41] or closed-form ﬁrst-order ODEs [37].
Extending the variational approach to closed-form PDEs is not trivial as PDEs are much more complex than ODEs and not all closed-form PDEs admit the variational formulation.
To address these challenges we use the new notions deﬁned in Section 2 to deﬁne a new and general class of PDEs in Section 4 that admit the variational formulation.
Variational-Ready PDEs
In this section, we propose a new and very general class of PDEs, the Variational-Ready PDEs (VRPDEs), which can be characterized without referring to the derivative.
The VR-PDEs allow arbitrary ∂ -free part but make some minor restrictions on the ∂ -bound part.
These restrictions allow one to use the variational formulation of PDEs to circumvent derivative estimation entirely.
Despite the minor restriction, VR-PDEs contain many well-known PDEs, including all linear PDEs, Maxwell’s equations, and Navier-Stokes equations (additional examples provided in Appendix B).
To deﬁne the new class of PDEs, we need the following deﬁnition.
Deﬁnition 1 (Extended derivative and differential operator) .
Let α ∈ N M 0 , | α | ≤ K , be a multiindex.
Let h : R M + N → R and a : R M → R be smooth functions.
An extended derivative E , denoted ( α , a, h ) , maps a vector ﬁeld u : R M → R N to a function E [ u ] : R M → R deﬁned as:
[Equation]
E is called closed-form if a and h are closed-form.
We call E non-degenerate if | α | > 0 .
Now, let ( E p ) p ∈
[ P ] be a ﬁnite sequence of non-degenerate extended derivatives.
The extended differential operator, denoted as E [ P ] is an operator deﬁned as:
[Equation]
Remark.
Any linear operator L = (cid:80) α ∈A a α ∂ α acting on u j is an extended differential operator.
Deﬁnition 2 (Variational-Ready PDE) .
Let E [ P ] be an extended differential operator, and let g :
R M + N → R be a continuous function.
We denote a Variational-Ready PDE (VR-PDE) by a pair (cid:0)
E [ P ] , g (cid:1) and deﬁne it as:
[Equation]
We extend the standard variational formulation of PDEs (Proposition 1 in Appendix B) from linear
PDEs to all VR-PDEs.
The following deﬁnition is useful in further discussion.
Deﬁnition 3.
Consider a ﬁeld u : Ω → R N , and an extended derivative E
= ( α , a, h ) .
Let φ : Ω → R be a testing function ( C K function 3 with compact support).
We deﬁne the functional
[Equation]
We can now use this functional to formulate variational characterization of VR-PDEs.
Theorem 1.
u : Ω → R N , where u j ∈ C K , is a solution to a VR-PDE in Equation 8 if and only if
[Equation]
for all testing functions φ : Ω → R .
Proof.
Appendix B.
This theorem motivates the variational loss function as we expect the left-hand side of Equation 9 to be closer to 0 the closer the canditate PDE is to the true one.
To calculate how well a set of vector ﬁelds D = { u ( d ) } Dd =1 matches a VR-PDE (cid:0) E
[ P ] , g (cid:1)
we propose the following loss function.
[Equation]
where { φ s } S s =1 is a set of predeﬁned testing functions.
This novel loss function makes it possible to evaluate how well any VR-PDE matches the observed data.
It can be used as an optimization objective in any algorithm that searches over some subspace of closed-form VR-PDEs.
We propose D-CIPHER in Section 5 as an example of such an algorithm.
D-CIPHER
In this section, we introduce a novel algorithm, D-CIPHER, for PDE discovery.
Problem formulation.
We are given a dataset of observed ﬁelds D = { v ( d ) } Dd =1 with a ﬁnite sampling grid G ⊂ Ω .
Each v ( d ) ( x ) is a noisy measurement, i.e., v ( d ) :
G → R N is deﬁned as v ( d
[Equation]
where (cid:15) ( d ) j ( x ) is a realization of a zero-mean random variable (noise), each u ( d ) j : Ω → R is a C K function, and every true ﬁeld u ( d ) is governed by the same closed-form PDE f .
The task is to infer the closed-form PDE, f , from the dataset D = { v ( d ) } Dd =1 and the sampling grid G .
We assume that f is inside the class of closed-form VR-PDEs (Section 4) and its ∂ -bound part is inside a subspace of extended differential operators spanned by a user-speciﬁed dictionary (see Step 1 below).
We propose an algorithm that consists of three steps.
For more details, check Appendix C.
Step 1: Choose the form and incorporate prior knowledge.
A human expert should encode their prior knowledge of the problem into a dictionary of non-degenerate extended derivatives Q
= { ˆ E p } p ∈
[ P ] .
We use this dictionary to search over a ﬁnite-dimensional subspace of closed-form operators spanned by this set.
In other words, we assume that the VR-PDE is of the form:
[Equation]
where β ∈ R P , g is any closed-form function of M + N variables, and ˆ E p =
( α p , a p , h p ) .
For instance, a dictionary might include only the partial derivatives up to a certain order.
For a 1+ second-order equation that means Q = { ∂ t , ∂ x , ∂ tx , ∂ 2 t , ∂ 2 x } .
That is already enough to discover heat and wave equations with any closed-form source.
If, for instance, the user suspects the presence of the advection term uu x (as in the Burgers’ equation), the term ∂ x ( u 2 ) can be included in the library.
It’s important to note that we do not assume any particular form of g apart from being closed-form.
Step 2: Estimate the ﬁelds.
As the dataset D consists of noisy and infrequently sampled ﬁelds, we ﬁrst need to estimate the "true" ﬁelds ˆ u ( d ) from v ( d ) .
Any choice of reconstruction algorithm can be used and the user should choose it according to the problem setting and their domain knowledge.
Step 3: Optimize.
We minimize the loss function in Equation 10 for the estimated ﬁelds { ˆ u ( d ) } Dd = among all PDEs of the form in Equation 12.
We solve the following optimization problem:
[Equation]
As we want to discover both g and β , we need to enforce ||
β || 1 = 1 .
Otherwise the loss is minimized by g = 0 and β = 0 .
We choose the L 1 norm to encourage sparsity in the coordinates of β .
The inner minimization in Equation 13 can be rewritten as a constrained least-squares problem.
[Equation]
where ˆ E p = ( α p , a p , h p ) and z ( d,s ) ∈ R P , w ( d,s ) ∈ R are deﬁned as
[Equation]
We show the full derivation in Appendix C. z ( d,s ) can be precomputed at the beginning of the algorithm without estimating the derivatives of the reconstructed ﬁelds.
They can be easily calculated if the derivatives of the testing functions φ s and the derivatives of a p can be analytically computed.
As the optimization problem in Equation 14 has to be solved many times for different closed-form expressions g , it poses some unique challenges.
As standard approaches are not sufﬁciently fast, we design a new heuristic algorithm to solve this problem.
We describe it in the next section.
CoLLie
The problem in Equation 14 from the previous section can be formulated as follows.
Given matrix
A ∈ R m × n and vector b ∈ R m , ﬁnd a vector z ∈ R n that minimizes || Az − b
||
such that
||
z || 1 = 1 .
The task is challenging as the unit L 1 sphere is not convex.
A method that guarantees an optimal solution is based on an observation that the n -dimensional L1 sphere consists of 2 ( n +1) n -simplices (which are convex).
Minimizing ||
Az − b
|| 22 on a simplex is a quadratic program [8] with many available solvers [2, 46, 3].
However, then the computation time scales exponentially with the number of dimensions.
This is prohibitively long for the inner optimization of D-CIPHER.
Therefore we design a heuristic algorithm CoLLie ( Co nstrained L 1 Norm Lea st Squares) that ﬁnds an approximate solution but is signiﬁcantly faster (Figure 1).
For a detailed description of CoLLie, see Appendix D.
[Figure]
Experiments
We perform a series of synthetic experiments to show how well D-CIPHER is able to discover some well-known differential equations (Table 2).
First, we demonstrate that D-CIPHER performs better than current methods when discovering PDEs in a linear combination form (Section 7.1).
Then we demonstrate it can discover PDEs with a closed-form ∂ -free part that cannot be expressed as a linear combination and thus are beyond the capabilities of current methods (Section 7.2).
We contrast
[Table] [Table]
[Table]
D-CIPHER with its ablated version where the derivatives are estimated and the standard MSE loss is used instead of the variational loss (details in Appendix E.1).
For additional information about the experiments (e.g., implementation details, data generation, experimental settings) see Appendix E.
Evaluation metrics.
To establish how well a discovered PDE matches the ground truth, we evaluate its ∂ -free and ∂ -bound parts separately.
For the ∂ -free part, we assign a binary variable indicating whether the correct functional form of the equation was recovered (please check Appendix E.8 for details).
For the ∂ -bound part, we measure the RMSE between the found coefﬁcients of β and the target ones.
We report the averages and standard deviations for both parts.
We call the averages respectively Success Probability and Average RMSE .
7.1 Discovering Linear Combinations: comparison with other methods
We compare D-CIPHER against two variants of PDE-FIND [42] and WSINDy [41] with optimization performed by Stepwise Sparse Regression [7] or Forward Regression Orthogonal Least-Squares [5] [Figure]
We note that D-CIPHER is speciﬁcally designed to discover PDEs that are beyond the capabilities of current methods, i [Figure] e., where the derivative-free part can be any closed-form expression.
Current methods are usually tested on equations where the derivative-free part is trivial (identically equal to 0).
Even though these algorithms are specialized to discover these simpler kinds of equations,
D-CIPHER performs better than (or equally well as) PDE-FIND and WSINDy, regardless of the optimization algorithm, when tested on Burgers’ equation the homogeneous heat equation [Figure] and
Kuramoto–Sivashinsky equation (Figure 2).
This demonstrates gain from both the variational loss and the new optimization routine.
[Figure]
7.2 Discovering equations beyond current methods
Inhomogeneous heat equation.
D-CIPHER is able to discover the correct equation even in settings with very high noise.
It performs better than the ablated version, thus showing the importance of the variational objective.
The result are presented in Table
[Table]
[Table]
[Figure]
Forced and damped harmonic oscillator.
As the oscillator is described by a second-order ODE, it cannot be discovered by D-CODE [37].
D-CIPHER discovers the correct functional form of the ∂ -free part and achieves a low RMSE for the coefﬁcients of β in most of the experimental settings.
The performance is higher than or comparable to the ablated version of D-CIPHER, thus demonstrating gain from using the variational approach.
We present the results in Figure [Figure]
Inhomogeneous wave equation.
This equation does not have the standard evolution form, as it does not involve the ∂ t term.
Thus, even without the source term, most of the current methods cannot be applied directly to discover this equation.
In Figure 4 we show the absolute difference between the true ﬁeld and the ﬁelds computed from the sources discovered by D-CIPHER and its ablated version across different measurement settings.
D-CIPHER ﬁnds the correct functional form with coefﬁcients not far from the ground truth.
The ablated version fails to discover the correct functional form and the found ∂ -free part does not reproduce the correct behavior of the equation.
Discussion
As D-CIPHER can potentially discover any closed-form ∂ -free part, it is especially useful when this part of the PDE captures an essential component of the phenomenon.
We demonstrate it by ﬁnding the heat and vibration sources as well as the driving force of an oscillator.
Beyond the spatio-temporal physical equations, D-CIPHER might prove useful in discovering population models structured by age, size, and spatial position [53, 52], age-dependent epidemiological models [17], and predator-prey models with age-structure [36].
All these equations are VR-PDEs where the ∂ -free parts are crucial elements of the equations signifying the rates of mortality, infection, recovery, or growth.
We discuss the limitations and open challenges in Appendix F.5.
Acknowledgments and Disclosure of Funding
This work is supported by the US Ofﬁce of Naval Research (ONR), the National Science Foundation (NSF Grant number:1722516) and Roche.
We want to thank Samuel Holt and Jonathan Crabbé for their useful comments and feedback on earlier versions of this work.
We are grateful to Professor
Yuanzhang Xiao for insightful discussions about the optimization algorithms.
A Notation and deﬁnitions
[Table] [Table]
extended derivatives β a vector describing the ∂ -bound part of the VR-PDE σ R a noise ratio ∆ t a sampling interval
A.2 Deﬁnitions
In this section, we collect the deﬁnitions of some of the important terms used in the paper for easy reference.
Deﬁnition 4 (Closed-form expressions and functions) .
A closed-form expression is a mathematical expression that consists of a ﬁnite number of variables, constants, arithmetic operations, and certain well-known functions (e.g., logarithm, trigonometric functions).
A function f is called closed-form if it can be represented by a closed-form expression.
E.g., f ( x, y ) =
x 2 log( y ) + sin(3 z ) .
Remark.
In practice, we do not want to consider any ﬁnite expression.
Any symbolic regression algorithm penalizes expressions that are too long putting a soft constraint on the number of elements used.
That is why deep neural networks are not considered closed-form even if they satisfy the conditions in Deﬁnition 4.
Deﬁnition 5 (Multi-index) .
An n -dimensional multi-index
α is an n -tuple
[Equation]
where ∀ i ∈ [ n ] α i ∈ N 0 .
Thus α ∈ N n 0 .
We deﬁne the order of α as | α | = (cid:80) ni
=
α i .
Deﬁnition 6.
For any n -dimensional multi-index α we deﬁne a mixed derivative
[Equation]
where ∂ α
i i =
∂
α
i /∂x α
i i is a α th i -order partial derivative with respect to x
i
(the i th independent variable).
We call ∂ α non-trivial if | α | > 0 .
We denote the list of all non-trivial partial derivatives of u up to order K as ∂ [ K ] u .
Deﬁnition 7 (Closed-form Partial Differential Equation) .
Let f be a closed-form real smooth function.
We say that a vector ﬁeld u : Ω → R N is governed by a K th -order closed-form PDE described by f if
[Equation]
where ∂ [ K ] u are all non-trivial mixed derivatives of all u j ( j ∈ [ N ] ) up to the K th order.
Deﬁnition 8 (Testing function) .
Support of a function φ : Ω → R is deﬁned as [Equation] where B is the topological closure of B in Ω . φ is called a testing function if it is a C K function with compact support.
Deﬁnition 9 (True Field) .
We deﬁne a true ﬁeld on Ω as a vector valued function u : Ω → R N where each u j : Ω → R is a C K function.
Deﬁnition 10 (Observed ﬁeld and sampling grid) .
We deﬁne a sampling grid G to be a ﬁnite subset of Ω .
Let u : Ω → R N be a true ﬁeld on Ω .
An observed ﬁeld sampled from u on a grid G is a function v : G → R N of the form
[Equation]
where (cid:15) j ( x ) corresponds to noise , a realisation of a zero-mean random variable.
Deﬁnition 11 ( L 1 sphere) .
Let n ∈ N .
We deﬁne n -dimensional L 1 sphere to be a subset of R n + deﬁned as:
[Equation]
Deﬁnition 12 (Standard simplex) .
Let n ∈ N .
We deﬁne standard n -simplex to be a subset of R n + deﬁned as:
[Equation]
B Variational-Ready PDEs B.1 Variational Formulation of PDEs
In this section, we provide the standard variational formulation of PDEs for linear PDEs [16].
Deﬁnition 13 (Linear differential operator) .
Let A be a ﬁnite set of multi-indices.
A linear differential operator L is deﬁned as
[Equation]
where a α ∈ C K is a non-zero sufﬁciently smooth function of dependent variables.
If max α ∈A | α | = n then we call L an n th -order linear differential operator.
If all a α are constants we say that L has constant coefﬁcients .
The adjoint of L , denoted L † , is a linear differential operator deﬁned as
[Equation]
Proposition 1 (Variational Formulation of PDEs for linear PDEs) .
Let K ∈ N .
Consider a scalar ﬁeld u : Ω → R , such that u ∈ C K , a K th -order linear differential operator L , and a continuous function g : Ω → R .
Let φ : Ω → R be a testing function.
Then u satisﬁes a linear PDE
L [ u
[Equation] [Equation]
if and only if
[Equation]
for all testing functions φ : Ω → R .
Note that the integrals are always well-deﬁned as φ has a compact support.
B.2 Theorem
Before we prove the Theorem 1, we need the following lemma, which is a particular formulation of the Fundamental lemma of calculus of variations [15].
We also need a generalized version of the divergence theorem [26].
Lemma (Fundamental lemma of calculus of variations) .
Let K ∈ N , Ω be an open set in R M , and u : Ω → R be a continuous function.
Then u is equal to 0 on the whole Ω if and only if (cid:82) Ω u ( x ) φ ( x ) d x = 0 for all C K functions φ : Ω → R with compact support.
Proof.
If u is identically 0 on Ω then all the integrals are trivially equal to 0 .
We now prove the converse.
Let as assume for contradiction that there exists a point x 0 ∈ Ω such that u ( x 0 ) (cid:54) = 0 .
Without loss of generality we assume u ( x 0 ) =
(cid:15) > 0 .
As u is continuous there exists an open ball around x 0 of radius δ , denoted B δ x 0 = { x ∈ Ω | || x − x 0 || 2 < δ } , such that ∀ x ∈ B δ x 0 u ( x ) >
(cid:15)/ 2 > 0 .
Now let φ be a C K function that is positive on B δ x 0 and 0 elsewhere.
Such a function can always be created by appropriately shifting and scaling φ ( x ) =
e − 1 / (1 −|| x || 22 ) · 1 {|| x || 2 < 1 } .
Its support is a closed ball ¯ B δ x 0 = { x ∈ Ω | || x − x 0 || 2 ≤ δ } which is compact.
Then
[Equation] [Equation]
as both u ( x ) and φ ( x ) are positive on B δ x 0 .
Thus we found a continuous function φ with compact support such that:
[Equation]
Therefore u is identically 0 on Ω .
To make this section self-contained, we provide the statement of the generalized divergence theorem [26].
Theorem 2 (Divergence theorem) .
Let Ω be an open set in R M and let f, g be continuous on ¯Ω
= Ω ∪ ∂ Ω and continuously differentiable on Ω .
Then
[Equation]
where ν is a normal unit vector to the boundary ∂ Ω .
In a 1-dimensional setting, the statement of the theorem reduces to the integration by parts.
We can now prove Theorem 1.
Proof.
Let us denote E p
= ( α p , a p , h p ) .
Then the PDE in Equation 8 can be written as:
[Equation]
The LHS is continuous as all a p and h p are smooth, g is continuous, u ∈ C K , and | α p | ≤ K ∀
p ∈
[ P ] .
Thus we can use the fundamental lemma of calculus of variations to say that the Equation 25 is true if and only if
[Equation]
for all testing functions φ .
We transform the LHS of Equation 26 using linearity to:
[Equation] [Equation]
Let us now focus on
[Equation]
[Equation]
and let us denote α p
= ( α p 1 , . . .
, α pM ) .
Then ∂ α p =
∂ α p 1. . .
∂
[Equation]
and the expression can be written as
[Equation]
Let us denote the support of φ as B .
As φ is equal to zero outside of its support, we can write the expression as
[Equation]
[Equation]
Without loss of generality, let us assume that α p 1 > 0 .
By the divergence theorem, this can be rewritten as
[Equation]
31) because the integral over the boundary is equal to[-
-]
[Equation]
as φ has a compact support (and thus vanishes on the boundary).
We can perform this operation α ptimes to shift the whole derivative ∂ α p 1to the second part of the equation and obtain
[Equation] [Equation]
Then we repeat this for other derivatives and we end up with the following expression:
[Equation]
[Equation]
As the integrand is zero outside of B , this can be rewritten as:
[Equation]
[Equation]
or more compactly, using the functional deﬁned in Deﬁnition 3, as:
[Equation]
Therefore Equation 27 can be written as:
[Equation]
Thus, we proved that Equation 25 is true if and only if
[Equation]
38) for all testing functions φ .
[Table] [Table]
B.3 Examples
The examples of VR-PDEs can be found in Table 5.
C D-CIPHER
C.1 Rewrite the inner optimization as a constrained least squares
Let us rewrite the objective in Equation 13.
[Equation]
First, let us observe that
F
[Equation]
if we let z ( d,s ) p ∈ R be deﬁned as
[Equation]
Moreover, if we deﬁne w ( d,s ) ∈ R as
[Equation]
we can rewrite expression 39 as
[Equation]
Now
[Equation]
the sum over p can be written as a dot product between z ( d,s ) ∈ R P and β ∈ R P .
We can also combine the sums over d and s .
We obtain
[Equation]
which is exactly the same as the objective in Equation 14.
C.2 Pseudocode
The pseudocode of D-CIPHER is presented in Algorithm 1.
Input: Observed ﬁelds D = { v ( d ) } Dd =1 , grid G
Input: Symbolic regression optimization algorithm
O
Input: Smoothing algorithm S
Input: Testing functions { φ s } Ss =
Input: Dictionary Q = { ˆ E p } Pp =1 , ˆ E p =
( α p , a p , h p ) (cid:46)
Step
Output: Target PDE ˆ u ( d ) =
S ( v ( d ) ) ∀ d ∈
[ D ] (cid:46) Stepinitialize matrix Z ∈ R D × S × R P
Z ( d,s ) p ← (cid:82) Ω h p ( x , ˆ u ( d ) ( x ))( − 1) | α p | ∂ α p ( a p ( x ) φ s ( x ))
d x procedure L OSS ( g ) initialize vector w ∈ R D × S w ( d,s ) ← (cid:82) Ω g ( x , ˆ u ( d ) ( x ))
φ s ( x ) d x β ← C O LL IE ( Z , w ) (cid:46) Section
L = ||
Zβ − w ||return L end procedure g
= O
( L OSS )
(cid:46)
Stepinitialize vector w ∈ R D × S w ( d,s ) ← (cid:82) Ω g ( x , ˆ u ( d ) ( x ))
φ s ( x ) d x β ← C O LL IE ( Z , w ) (cid:46) Sectionreturn (cid:80)
Pp =1 β p ˆ E p
[ u ]( x ) − g ( x , u ( x )) =
C.3 Testing functions
Testing functions, by deﬁnition, need to be sufﬁciently smooth functions (in C K class) with compact support.
Moreover, the result proved in [37] suggests that these functions should be a subset of a
Hilbert basis in L2 space.
In particular, that means they should be orthonormal.
We use B-Splines [13] as the testing functions in our experiments because we can control their smoothness and the derivatives are easy to compute.
We scale and shift them appropriately so that they are orthonormal.
D CoLLie D.1 Lagrangian
The problem that CoLLie is supposed to solve is a constrained least-squares optimization deﬁned as: minimize || Az
[Equation]
subject to
[Equation]
z
[Equation]
(45) where A ∈ R m × n has a full column rank, b ∈ R m , and z ∈ R n for some m, n ∈ N .
We consider the Lagrangian L : R n × R → R associated with this problem [8] deﬁned as
[Equation]
Now let us deﬁne ˆ z : R → R n as
[Equation]
The goal of our algorithm is to ﬁnd λ ∗ ∈ R such that
|| ˆ z ( λ ∗ )
|| 1 = 1 .
Let us deﬁne a function q : R → R as
[Equation]
The goal can be phrased as ﬁnding λ ∗ ∈
R such that q ( λ ∗ )
= 1 .
Let us note that ˆ z (0) is just a solution to the ordinary least squares (OLS) problem with no constraints and its norm is q (0) .
D.2 Extending LARS
Case 1.
q (0) ≥ 1 .
If we assume that ˆ z is continuous then q is also continuous.
From the continuity and the fact that lim λ → + ∞ q ( λ ) = 0 and q (0) ≥ 1 we infer that there exists a λ ≥
such that q ( λ ) = 1 .
Moreover, for λ ≥
the problem in Equation 47 is the same as in LASSO [48].
Therefore we just need to perform LASSO for different λ and choose the one that gives the solution with L1 norm equal to 1.
To do it in practice we use Least Angle Regression (LARS)
[14], a popular algorithm used to minimize the LASSO objective.
It generates complete solution paths, i.e., a function c : R + → R n deﬁned as
[Equation]
(49) which is equivalent to ˆ z for λ ≥ 0 .
An illustration of LARS solution paths can be seen in Figure 5.
Each line corresponds to a function c i which describes the coefﬁcient for the i th covariate.
The paths are deﬁned from some λ 0 where all c i ( λ 0 )
=
to λ = 0 where c (0)
=
ˆ z (0) .
In other words, the solution paths cover the whole range of constraints from the strictest, effectively imposing the L norm of z to be 0 , up to no constraints, solving the OLS problem.
The solution paths from the LARS algorithm are piecewise linear and the outputs are the values of the coefﬁcients for points ( λ 0 > . . .
>
λ n = 0) where the slopes change.
We calculate the norm at each of these points, || c ( λ i ) || 1 , and ﬁnd j ∈ [ n ] such that || c
( λ j − 1 ) || 1 < 1 ≤ || c ( λ j )
|| 1 .
As each c i is a linear function on [ λ j , λ j − 1 ] and we know both c ( λ j − 1 ) and c ( λ j ) , we can effectively search for λ ∈ [ λ j , λ j − 1 ] such that || c ( λ ) || 1 = 1 .
The search can be performed by any root-ﬁnding algorithm.
We use Brent’s method [9].
Case 2.
< q (0) < 1 .
This is much more difﬁcult as it corresponds to solving the problem in Equation 47 for λ < 0 .
The solutions given by the LARS algorithm are too small.
In fact, the solution with the biggest norm is c (0) =
ˆ z (0) , the OLS solution, with norm exactly q (0) < 1 .
To address this challenge, we propose the following heuristic.
We extend the solution paths generated by LARS beyond λ = 0 for λ < 0 .
We assume that the paths will continue to be piecewise linear and that they will keep the slope they have in the last interval [ λ n = 0 , λ n − 1 ] .
Let us denote this slope as
[Equation]
This is graphically represented in Figure 5.
Formally, these extended paths, ¯ c : R → R are deﬁned as:
[Equation]
Now, we want to ﬁnd λ <
such that || ¯ c ( λ ) ||
1 = 1 .
To achieve this in practice, we ﬁrst make the following observations.
For any λ < 0 we say that
¯ c i ( λ ) is on the right side if ¯ c i ( λ )∆ c i ≤
and we say that
¯ c i ( λ ) is on the wrong side if ¯ c i ( λ )∆ c i > 0 .
In other words, being on the wrong side just means that the path is yet to cross the x-axis if we keep decreasing λ .
We can easily ﬁnd λ (cid:48)
such that for all λ < λ (cid:48)
all ¯ c i ( λ ) are on the right side (none of the paths will ever cross the x-axis).
[Equation]
If || ¯ c ( λ (cid:48) )
|| 1 ≥ 1 we just need to search the interval [ λ (cid:48) , 0] for λ such that || ¯ c ( λ (cid:48) )
||
1 = 1 .
If || ¯ c ( λ (cid:48) )
||
1 <
then we need to search λ < λ (cid:48) .
However, by deﬁnition, for all λ < λ (cid:48) , all c i ( λ ) are on the right side.
That means || ¯ c ( λ ) || 1 as a function of λ is just a linear function on the interval ( −∞ , λ (cid:48) ) .
To see that, let us observe that
[Equation]
Additionally, for λ < λ (cid:48)
all c i ( λ ) are on the right side, so we have sign (¯ c i ( λ ))
= − sign (∆ c i ) .
We can rewrite || ¯ c ( λ ) || 1 as:
[Equation]
Therefore the solution can be found using the following equation
[Equation]
Case 3.
q (0) = 0 .
In that case, we just return a precomputed solution to the problem minimize || Az | [Equation]
subject to
[Equation]
[Equation]
which we compute by subdividing the problem into 2 n quadratic programs and solving each of them separately using CVXOPT algorithm
[2] as described in Section 6.
D.3 Comparison
We perform a comparison between CoLLie and an algorithm based on CVXOPT [2] as described in
Section 6 .
CoLLie uses different procedures depending on the problem, thus for a fair comparison we generate an equal number of tests falling under Case 1 and Case 2 (two main cases) as described in D.2.
To achieve that, we generate a random m × n matrix A where each entry is sampled from the standard normal distribution.
We use m = 1000 and n ranging from 2 to 7 .
We then generate a vector ˆ z ∈ R n such that each entry is sampled from a uniform distribution on [ − 0 . 5 , 0 . 5] .
Then ˆ z is normalized to have L1 norm equal to 1 .
We sample a number l from a uniform distribution on [ − 1 , 3] and multiply ˆ z by l to obtain z (cid:48)
=
l ˆ z .
By this procedure, we are guaranteed that cases || z (cid:48)
||
1 <
and || z (cid:48)
|| 1 ≥ 1 are equally likely.
We then let b =
Az (cid:48) .
The task is then to ﬁnd z with L1 norm equal to 1 that minimizes ||
Az − b
|| 22 .
As q (0) =
||
z (cid:48)
|| 1 = l , Case 1 and Case 2 are equally likely.
We perform 1000 experiments for each n .
As losses for optimal solutions can be on widely different scales, we report the relative error between the loss obtained by CoLLie and the loss obtained by the algorithm based on CVXOPT, which seems to always ﬁnd
the optimal solution.
The time is measured on a single computer with an Intel Core i5-6500 CPU (4 cores) and 16GB of
RAM.
E Experiments E.1 Ablated D-CIPHER
The ablated version uses the standard MSE loss with estimated derivatives and thus solves the following optimization problem:
[Equation]
where v ( d ) is the observed ﬁeld, G is the sampling grid, and ˆ E p [ v ( d ) ]( x ) requires derivative estimation.
The ablated version uses the same symbolic regression algorithm to search over closed-form g and
CoLLie for the inner optimization.
E.2 Implementation
Testing functions.
We use B-Splines [13] as the testing functions in our experiments because we can control their smoothness and the derivatives are easy to compute.
We scale and shift them appropriately so that they are orthonormal.
Step 1.
For the homogeneous heat equation and Burgers’ equation we use dictionary Q = { u, ∂ t u, ∂ x u, ∂ x ( u 2 ) , ∂ 2 x u, ∂ 2 x ( u 2 ) } .
For Kuramoto-Sivashinsky equation we use dictionary Q = { u, ∂ t u, ∂ x u, ∂ x ( u 2 ) , ∂ 2 x u, ∂ 2 x ( u 2 ) , ∂ 3 x u, ∂ 3 x ( u 2 ) , ∂ 4 x u, ∂ 4 x ( u 2 ) } .
For the damped and forced harmonic oscillator we use the dictionary { ∂ t , ∂ 2 t } , and for the wave and heat equations, we use { ∂ t , ∂ x , ∂ 2 t , ∂ t ∂ x , ∂ 2 x } .
Step 2.
Field estimation is performed using the Gaussian Process Regression from the Python library scikit-learn [34].
The kernel is chosen to be the RBF kernel [54] with an added White kernel to account for noise.
The observed ﬁeld is initially standardized by subtracting the mean and dividing by the standard deviation.
Then the GaussianProcessRegressor is ﬁtted to the data.
The estimated ﬁelds are generated by predicting the values of a trained Gaussian Process on a full integration grid and then scaling back to their original range (by multiplying by the standard deviation and adding the mean).
Step 3.
The search over the closed-form expression is performed using the symbolic regression library gplearn [47].
We use a custom ﬁtness function that solves the inner optimization problem in Equation 13.
This inner optimization is performed by CoLLie (Section 6).
The integration is performed using Riemann sums.
Ablated version of D-CIPHER.
The derivative estimation is performed by ﬁrst ﬁtting a Gaussian process (in the same way as in Step 2) and then using the ﬁnite difference to estimate the derivative in one of the coordinates for all points in the sampling grid.
To obtain higher-order derivatives, a
Gaussian process is ﬁtted again and the derivative is once again calculated using the ﬁnite difference (possibly in a different direction than the ﬁrst time).
E.3 Hyperparameters
Gaussian process regression.
The kernel parameters of the Gaussian Process are automatically adjusted during training.
The default bounds of the length scale of the RBF kernel and the noise level of the White kernel are used, i.e., (1 e − 5 , 1 e 5) .
GPlearn.
We do not perform parameter tuning for the gplearn library and use the same parameters as in D-CODE [37] except for the parsimony coefﬁcient and the number of generations.
The number of generations is chosen to be 30 for the damped and forced harmonic oscillator andfor the inhomogeneous heat and wave equations.
Please check [47] for the detailed description of these parameters.
[Table]
We modify the implementation of the parsimony coefﬁcient.
The standard implementation adds to the loss the length of the equation multiplied by the parsimony coefﬁcient.
In our implementation, we increase the loss by the parsimony coefﬁcient.
This modiﬁcation is performed because for different experiments we record the loss on widely different scales.
To prevent tuning this parameter for every experimental setting we introduce a penalty that can work on different scales.
The parsimony coefﬁcient is chosen manually by performing experiments for a few values.
The value used in the experiments is 0 . 05 .
The set of allowed mathematical operations is: { + , − , × , ÷ , sin , exp , log }
We want to emphasize that we use the same conﬁguration of gplearn in D-CIPHER and its ablated version.
Integration and number of testing functions.
For the damped and forced harmonic oscillator we use 10 testing functions and the integration step 0 .
For the inhomogeneous heat and wave equations we use 100 testing functions and integrate on a grid with steps δt = 0 .
01 and δx = 0 .
Derivative estimation in the ablated version of D-CIPHER.
The Gaussian process is conﬁgured the same way as described above.
The interval used in the ﬁnite difference method to estimate the derivative was chosen to be: 10 − 3 .
E.4 Choice of equations
Equations used in Section 7.1 are canonical equations from physics that often appear in other works about PDE discovery [42].
The homogeneous heat equation is a second-order PDE that models how heat diffuses through a region.
It contains the dissipative term ∂ 2 x u .
Burgers’ equation is second-order
PDE used, for instance, in ﬂuid mechanics or nonlinear acoustics [12].
It contains the advection term u∂ x u and the diffusion term that prevents shock formation.
Kuramoto-Sivashinsky equation is a fourth-order PDE used in modelling reaction-diffusion systems [22] and is known for its chaotic behavior [18].
In Section 7.2, we chose equations of physical signiﬁcance that have an interesting ∂ -free part that is not a linear combination (as discussed in Section 2).
That makes them impossible to discover by the current methods.
A forced and damped harmonic oscillator is a second-order ODE.
Although DCODE [37] can discover any closed-form ﬁrst-order ODE, it cannot be used to discover second-order
ODEs.
Thus there is currently no algorithm capable of discovering this equation.
Inhomogeneous heat and wave equations are second-order PDEs where the ∂ -free part is a source (of heat and wave respectively).
Moreover, the wave equation does not have the standard evolution form, as it does not involve the ∂ t term.
Thus even without the source term, most of the current methods cannot be applied directly to discover this equation.
E.5 Data generation
Homogeneous heat equation.
The ﬁelds were generated by solving the equation ∂ t u − θ 1 ∂ 2 x u =( θ 1 = 0 .
with Neumann boundary conditions ∂ x u ( t, 0) =
∂ x u ( t, X ) = 0 and an initial condition u (0 , x ) = u 0 ( x ) , where u 0 is randomly sampled from a Gaussian process.
The equation is solved using the implicit BTCS scheme [20] with steps δt = 0 . 001 and δx = 0 .
The observed ﬁeld is generated by sampling ( t, x ) ∈ [0 , T ] × [0 , X ] , evaluating the true ﬁeld u ( t, x ) and adding Gaussian noise.
T = 2 and X = 2 are used in the experiments.
Burger’s equation.
The ﬁelds are computed by solving ∂ t u + u∂ x u − θ 1 ∂ 2 x U =
( θ 1 = 0 . 2 ) with an initial condition u (0 , x ) = u 0 ( x ) , where u 0 is randomly sampled from a Gaussian process, and with Dirichlet boundary conditions u ( t, 0)
=
u 0 (0) , u ( t, X ) = u 0 ( X ) .
The equation is solved the using Crank-Nicolson scheme [51] with steps δt = 0 .
002 and δx = 0 .
The observed ﬁeld is generated by sampling ( t, x ) ∈ [0 , T ] × [0 , X ] , evaluating the ﬁeld u ( t, x ) and adding Gaussian noise.
T = 2 and X = 2 are used in the experiments.
Kuramoto-Sivashinsky equation.
The solution is the same as the one used in [42].
The observed ﬁeld is generated by sampling ( t, x ) ∈ [0 , T ] × [0 , X ] , evaluating the ﬁeld u ( t, x ) and adding Gaussian noise.
T = 100 and X = 100 are used in the experiments.
Damped and forced harmonic oscillator.
The true ﬁelds are created by analytically solving the equation ∂ 2 t u ( t ) + 2 θ 1 θ 2 ∂ t u ( t ) + θ 22 u ( t )
= θ 3 sin( θ 4 t ) , where θ 1 = 0 .
5 , θ 2 = 4 . 0 , θ 3 = 5 . 0 , θ 4 = 3 . 0 , with random initial conditions for u (0) and ∂ t u (0) .
The observed ﬁelds are then created by sampling t ∈ [0 , T ] , evaluating u ( t ) , and adding Gaussian noise.
T = 2 was used in the experiments.
Inhomogeneous heat equation.
The true ﬁelds are computed by solving ∂ t u ( t, x ) − θ 1 ∂ 2 x u ( t, x )
= θ 2 e θ 3 t , where θ 1 = 0 .
25 , θ 2 = 1 .
25 , θ 3 = 1 . 8 , with Neumann boundary conditions ∂ x u ( t, 0) =
∂ x u ( t, X ) = 0 and an initial condition u (0 , x ) = u 0 ( x ) , where u 0 is randomly sampled from a
Gaussian process.
The equation is solved using the implicit BTCS scheme [20] with steps δt = 0 .and
δx = 0 .
The observed ﬁeld is generated by sampling ( t, x ) ∈ [0 , T ] × [0 , X ] , evaluating the true ﬁeld u ( t, x ) and adding Gaussian noise.
T = 2 and X = 2 are used in the experiments.
Inhomogeneous wave equation.
The true ﬁelds are computed by solving ∂ 2 t u ( t, x ) − θ 1 ∂ 2 x u ( t, x ) = θ 2 e t sin( θ 3 t ) , where θ 1 = 1 . 0 , θ 2 = 2 . 0 , θ 3 = 3 . 0 , with Dirichlet boundary conditions u ( t, 0) =
u 0 (0) , u ( t, X ) = u 0 ( X ) , where u 0 is randomly sampled from a Gaussian process and speciﬁes the initial condition u (0 , x ) = u 0 ( x ) .
The equation is solved using the Implicit Difference Method [33] with steps δt = 0 . 001 and δx = 0 .
The observed ﬁeld was generated by sampling ( t, x ) ∈ [0 , T ] × [0 , X ] , evaluating the true ﬁeld u ( t, x ) and adding Gaussian noise.
T = 2 and
X = 2 were used in the experiments.
E.6 Experimental settings Homogeneous heat equation
Noise ratio ( σ R ): 0 .
Number of samples ( D ):[-
-]
Domain ( Ω ): [0 , 2] × [0 . 2]
Grid ( G ): { 0 , 0 . 07 , . . .
, 2 } × { 0 , 0 .
Burgers’ equation
Noise ratio ( σ R ): 0 .
Number of samples ( D ):[-
-]
Domain ( Ω ): [0 , 2] × [0 . 2]
Grid ( G ): { 0 , 0 . 1 , . . .
, 2 } × { 0 , 0 . 1 , . .
Kuramoto [Table] equation
Noise ratio ( σ R ): 0 .
Number of samples ( D ):[-
-]
Domain ( Ω ): [0 , 100] × [0 . 100]
Grid ( G ): { 0 , 1 . 67 , . . .
, 100 } × { 0 , 1 . 67 , . .
Damped and forced harmonic oscillator
Default values are in bold.
Noise ratio ( σ R ): 0 .
Number of samples ( D ): 1 , 2 , 5 , 10 ,[-
-]
Domain ( Ω ): [0 , 2]
Grid ( G ): { 0 , 0 .
Inhomogeneous heat equation
Noise ratio ( σ R ): 0 . 05 , 0 . 1 , 0 .[-
-]
Number of samples ( D ):[-
-]
Domain ( Ω ): [0 , 2] × [0 . 2]
Grid ( G ): { 0 , 0 . 07 , . . .
, 2 } × { 0 , 0 .
Inhomogeneous wave equation
Noise ratio ( σ R ): 0 .
Number of samples ( D ):[-
-]
Domain ( Ω ): [0 , 2] × [0 . 2]
Grid ( G ): { 0 , 0 . 07 , . . .
, 2 } × { 0 , 0 .
E.7 Benchmarks
An important hyperparameter in PDE-FIND and WSINDy is the library Θ used.
We impose that Q and Θ ∪ { ∂ t u } have the same number of elements.
Moreover, solutions given by PDE-FIND and
WSINDy are scaled to have the L1 norm equal to 1.
Both of these measures are undertaken to ensure that RMSE error is comparable between the algorithms.
For the homogeneous heat equation and Burgers’ equation we use a library Θ = { u, ∂ x , ∂ 2 x u, u∂ x u, u∂ 2 x u } .
For the Kuramoto-Sivashinsky equation, we use a library Θ = { u, ∂ x , ∂ 2 x u, ∂ 3 x u, ∂ 4 x u, u∂ x u, u∂ 2 x u, u∂ 3 x , u∂ 4 x } .
In the experiments, we have not optimized for the derivative-free part as it is identically equal to 0 in all equations.
E.8 Correct functional form
To measure success probability we need to establish whether two closed-form functions match.
The previous approach [37] considered their functional forms , i.e., expressions where all numeric constants are replaced by placeholders.
By this measure, functions sin(3 x ) and sin(3 .
5 x ) match as they have the same functional form sin( Cx ) , where C is a placeholder.
However, this deﬁnition is quite restrictive because functions sin(3 x ) , sin(3 x )+0 .
001 sin(3 x ) , and sin(3 x + 0 . 001) all have different functional forms.
We consider it an open challenge to design a good metric that would meaningfully reﬂect whether the correct equation is discovered.
We propose the following.
For a target function f , we consider its augmented form ˜ f , deﬁned as ˜ f
( x ) =
C 1 f ( C 3 x + C 4 )
+ C 2 , where all C i are placeholders.
Then all numeric constants are turned into placeholders as well.
In the end, we combine the constants.
For instance, C 1 + C 2 becomes just C 3 .
As an example, let us consider a function f ( x ) = 1 .
3 e 2 x .
The augmented functional form is created in the following way:
Augment: C 1 × 1 .
3 e 2 × ( C 3 × x + C 4 ) + C2.
Replace: C 1 × C 5 e C 6 ×
( C 3 × x + C 4 ) + C3.
Combine: C 1 e C 3 x + C 4 + C[-
-]
We perform this procedure for the target function.
We can now take the standard functional form of the candidate function and check whether it matches the augmented functional form of the target function, taking into account that some of the constants might not be present in the candidate expression.
To aid in this procedure, we use a Python library for symbolic mathematics, SymPy [32].
E.9 Computation time
The average computation time for a single experiment with the damped and forced harmonic oscillator is 281 seconds with a standard error of 4.5 seconds.
The average computation time for a single experiment with an inhomogeneous heat equation is 68 minutes with a standard error of 38 seconds.
This time is measured on a single computer with an Intel Core i5-6500 CPU (4 cores) and 16GB of
RAM.
The experiments are run simultaneously on 5 computers like the one described above.
The total time for all experiments (all seeds, all equations, all experimental settings, and both versions of
D-CIPHER) is 65 hours.
E.10 Licenses
The licenses of the software used in this work are presented in Table
[Table]
F Additional experiments and discussion F.1 Sharpe-Lotka-McKendrick model
We test D-CIPHER on a population model called the Sharpe-Lotka-McKendrick model [53].
The model is described by the following equation
[Equation]
(58) where m ( a ) is age-speciﬁc mortality rate.
We choose m ( a ) = 2 e θa .
We set θ = 1 . 5 in the experiments.
We note that the derivative-free part of the target PDE cannot be expressed as a linear combination of functions from a ﬁnite dictionary if the parameters are not known a priori.
D-CIPHER is uniquely positioned among other discovery algorithms as the only technique that can recover any mortality rate that can be represented as a closed-form expression.
We show a comparison between D-CIPHER and the Ablated D-CIPHER in Table[-
-]
[Table]
F.2 Discovering systems
Discovering systems of PDEs is a much harder problem than discovering a single PDE.
One of the issues is the fact that we would call indeterminism .
It follows from the following fact.
If a vector ﬁeld u is a solution to two differential equations f 1 and f 2 , i [Equation] e.,
[Equation]
then it is also a solution to any linear combination of these equations, i
[Equation]
e.,
[Equation]
for any λ 1 , λ 2 ∈ R .
Moreover, equations can sometimes be differentiated to yield more equations.
Let us take as an example the Cauchy-Riemann equations deﬁned as:
[Equation]
Let us assume that we have a true vector ﬁeld ( u 1 , u 2 ) that satisﬁes both equations.
Then the following equations are also satisﬁed
[Equation]
We can also differentiate the Cauchy-Riemann equations to arrive at the Laplacian equations for uand u 2 .
[Equation]
We could also combine the ﬁrst-order equations with second-order equations or consider even higherorder derivatives.
Although all these equations are compatible with our vector ﬁeld ( u 1 , u 2 ) , not all of them are equally desirable to discover.
That is why we believe that in any algorithm for discovering systems of differential equations substantial expert knowledge or inductive biases have to be encoded to guide the algorithm into the right equations.
Current methods do not consider systems of equations or consider a system of equations of a very particular form.
In the latter case, each equation models a derivative with respect to time of a different scalar ﬁeld.
The system is assumed to look like this:
[Equation]
In addition, the LHS of these equations is often assumed to only contain spatial derivatives.
D-CIPHER can be used to discover some systems of differential equations if enough prior knowledge is provided in the choice of the dictionary Q .
Moreover, the discovered equations are not required to have a particular evolution form as is the case in current approaches.
Firstly, we note that D-CODE [37] has been shown to discover a system of equations that looks like
Equations 64 when all equations are ﬁrst-order ODEs.
As D-CIPHER reduces to D-CODE when applied to ﬁrst-order ODEs, it is also capable of discovering such a system.
We demonstrate that D-CIPHER is able to discover both Cauchy-Riemann equations if we use two different dictionaries.
Each of the dictionaries yields a different equation.
Additionally, it is a well-known fact that if a vector ﬁeld satisﬁes Cauchy-Riemann equations then the constituent scalar ﬁelds are harmonic, i.e., they satisfy Laplace’s equation.
Based on the same dataset we are also able to discover both Laplace’s equations given another set of two different dictionaries.
We note that Laplace’s equation does not contain ∂ x term, so most of the current methods cannot be directly applied to discover this equation.
The results are presented in Figure 6.
The dictionaries used to discover each of the equations are the following.
For ∂ x u 1 − ∂ y u 2 =
we use Q 1 = { ∂ x u 1 , ∂ y u 2 , ∂ 2 x u 1 , ∂ y u 2 }
For ∂ x u 2 + ∂ y u 1 =
we use Q 2 = { ∂ x u 2 , ∂ y u 1 , ∂ 2 x u 2 , ∂ 2 y u 1 }
For ∂ 2 x u 1 + ∂ 2 y u 1 =
we use Q 3 = { ∂ x u 1 , ∂ y u 1 , ∂ 2 x u 1 , ∂ 2 y u 1 }
For ∂ 2 x u 2 + ∂ 2 y u 2 =
we use Q 4 = { ∂ x u 2 , ∂ y u 2 , ∂ 2 x u 2 , ∂ 2 y u 2 }
In the experiments, we have not optimized for the derivative-free part as it is identically equal to 0 in all equations.
[Figure]
F.3 Increasing the size of the dictionary Q
We test D-CIPHER on the Sharpe-Lotka-McKendrick model (Equation 58) with different dictionaries.
We start with a small dictionary Q 1 = { ∂ t u, ∂ a u } , and we create every new dictionary from the previous one by adding one more extended derivative.
The ﬁnal dictionary contains 10 elements,
Q 10 = { ∂ t u, ∂ a u, ∂ 2 a u, ∂ 2 t u, ∂ t ∂ a u, ∂ a ( u 2 ) , ∂ 2 a ( u 2 ) , ∂ t ( u 2 ) , ∂ 2 t ( u 2 ) , ∂ t ( u 3 ) , ∂ a ( u 3 ) } .
The Average
RMSE of the ∂ -bound part is shown in Figure 7.
We do not observe any increase in average error.
Note that in these experiments we just focus on the ∂ -bound part and do not optimize the ∂ -free part.
F.4 Computational complexity
We want to emphasize that PDE discovery is not a time-critical application (usually this process is performed manually by scientists) and we believe D-CIPHER’s computation time is acceptable for such a task.
In this section, we describe which parts of the algorithms are most computationally intensive.
Computation in D-CIPHER is performed in Step 2 and Step 3.
Step 2.
Computational complexity of Step 2 depends on the choice of the smoothing algorithm.
We want to emphasize that the user can use any smoothing algorithm based on their domain knowledge and experience, including spline regression, LOWESS, and Kalman ﬁlters.
Gaussian Process has time complexity O ( n 3 ) where n is the number of data points in a grid G .
D-CIPHER is speciﬁcally designed to work for sparse and noisy data, so we have not encountered major computational issues
[Figure] [Figure]
while performing Gaussian process regression.
We also note that signiﬁcant progress has been made in adapting Gaussian processes for datasets with many data points [23].
Step 3.
D-CIPHER consists of two optimization loops.
The outer optimization is performed by a symbolic regression algorithm (in our case genetic programming).
The inner optimization is performed by CoLLie (Section 6).
Searching through a space of closed-form expression requires testing many candidate equations.
D-CIPHER is designed to work with many different algorithms for symbolic regression and it is advised to choose an algorithm that can search through this space most efﬁciently.
CoLLie was speciﬁcally designed to solve the optimization problem as quickly as possible with a minor accuracy trade-off (see Figure 1).
It is based on LARS which has time complexity O ( mn 2 ) [14] where m is the number of samples and n is the number of features.
In our case, n = P is usually small as it corresponds to the size of the dictionary and m = SD .
In our experiments, S was set up [-to
-]to 100 and D up to 10.
Overall LARS is performed very quickly.
The additional steps in CoLLie require only a few arithmetical operations (linear in n ) and a possible root searching of a single variable function that is efﬁciently implemented using Brentq algorithm [9].
Other important parts of the algorithm are the numerical integrations.
One such integration is performed at the beginning of the algorithm to compute the matrix Z
(see Algorithm 1).
It does not contribute much to the computation time as it is performed only once.
The other integration is performed for each candidate equation to compute vector w .
Also, substantial time is spent on computing the values of the candidate function g used in the integration.
Fortunately, both of these operations can be implemented as vectorized operations which are designed to run very efﬁciently on modern hardware.
We also discourage overly long equations for g (check the discussion in Appendix
E.3) to limit the number of operations performed.
F.5 Limitations and open challenges
D-CIPHER may fail in some scenarios, either due to challenging experimental settings or a challenging underlying PDE .
Challenging experimental settings might include unobserved variables, high measurement noise, infrequent sampling, and inadequate domain (e.g., small time horizon).
Challenging PDE forms might include a PDE outside of the VR-PDE class or a ∂ -free part with a complex expression that is difﬁcult to ﬁnd.
We note that we address some of these challenges by utilizing a variational approach, deﬁning VR-PDEs to be a very general class of equations, and designing CoLLie enabling a thorough search across closed-form expressions.
G Related works
Symbolic Regression.
The goal of symbolic regression is to ﬁnd a closed-form expression that best models the given dataset both in terms of accuracy and simplicity.
In contrast with the conventional regression analysis which optimizes the parameters of a pre-speciﬁed model, symbolic regression aims at discovering both the general structure and the parameters of the model.
Most of the existing work focuses on developing optimization algorithms.
Genetic Programming [21] has been widely used for that task [44].
A different strategy has been employed in AI Feynman [49, 50] that uses neural networks to reduce the search space by identifying simplifying properties like symmetry or separability.
Optimization methods based on pre-trained neural networks [4], reinforcement learning [35], and Meijer-G functions [1] have also been proposed.
Data-driven discovery of closed-form differential equations.
Data-driven discovery of physical laws is an established area of machine learning [6, 44].
The pioneering work in that area was SINDy [10] that constrained the space of equations to linear combinations of functions from a predeﬁned library and used sparse regression to discover explicit ODEs.
It was later extended to include implicit ODEs [28, 19] and PDEs [42, 43].
Various other extensions were proposed by improving the derivative estimation and the training procedure [40, 55], adding additional selection criteria [27] and learning the library using genetic programming [29, 11, 56].
A different approach is taken by [24] (an extension of [25]) which uses convolutional and symbolic neural networks.
It is important to note that all of these methods still assume the PDE to be a linear combination as discussed in
Section 2 (Equation 4) which signiﬁcantly limits their search space.
Some other developments are based on Gaussian processes [39, 38] but they require the exact form of the PDE and only optimize the parameters.
Variational approach.
Recently, the variational approach has been used as a viable alternative to derivative estimation.
However, they have only been used for differential equations in a linear combination form [30, 31, 41] or closed-form ﬁrst-order ODEs [37].
Extending the variational approach to closed-form PDEs is not trivial as PDEs are much more complex than ODEs and not all closed-form PDEs admit the variational formulation.
In fact, the approaches that learn the library mentioned in the previous paragraph can produce exactly such terms which prohibits the use of variational formulation.
To address these challenges we use the new notions deﬁned in Section 2 to deﬁne a new and general class of PDEs in Section 4 that admit the variational formulation.