Constraining Gaussian Processes to Systems of Linear Ordinary Differential Equations
Abstract
Data in many applications follows systems of Ordinary Differential Equations (ODEs).
This paper presents a novel algorithmic and symbolic construction for covariance functions of Gaussian Processes (GPs) with realizations strictly following a system of linear homogeneous ODEs with constant coefﬁcients, which we call LODE-GPs.
Introducing this strong inductive bias into a GP improves modelling of such data.
Using smith normal form algorithms, a symbolic technique, we overcome two current restrictions in the state of the art: (1) the need for certain uniqueness conditions in the set of solutions, typically assumed in classical ODE solvers and their probabilistic counterparts, and (2) the restriction to controllable systems, typically assumed when encoding differential equations in covariance functions.
We show the effectiveness of LODE-GPs in a number of experiments, for example learning physically interpretable parameters by maximizing the likelihood.
Introduction
Many real world tasks have underlying dynamic behavior, for example chemical reactions [22], systems in bioprocess engineering [26] or population dynamics [68].
One currently widely debated application are compartmental models in epidemiology [20, 34, 50].
Many such systems are linear or can be decently linearized, such as in control theory [71], biology [16], process engineering [1, §9] or engines [7].
Including prior knowledge in the form of differential equations beneﬁts the model ﬁt and enhances interpretability for a model.
Hence, modelling differential equations in Machine
Learning (ML) has therefore been the focus of much research in both Gaussian Processes (GPs) (e g. [37, 29, 49, 24, 61]) and Deep Learning (DL) (e.g. [47, 36, 62, 17]).
The class of probabilistic Ordinary Differential Equation (ODE) solvers allow to estimate the solution of ODE initial value problems through approximations, often based on e.g. Runge-Kutta methods or Kalman ﬁlters [50, 52, 51, 10], and thereby do not strictly guarantee to yield solutions of the
ODE. This class of algorithms is commonly used to solve non-linear ODEs, but typically require that systems are well-posed, in particular the solution needs to be unique once a ﬁnite number of initial conditions is known.
While this second limitation is typically irrelevant in physical or biological systems, it is strongly relevant in systems in engineering, such as in control systems with their freely choosable inputs.
With early works like [25], the introduction of derivatives as linear operators in GPs became prominent for partial differential equations.
This inspired several works that encode differential equations in the covariance structure of GPs [29, 49, 24, 61, 46] to introduce a strong inductive bias into GPs.
This motivated [37] to make these approaches algorithmic, build a mathematical foundation, and show the hidden assumption in previous works of being limited to controllable systems.
This paper overcomes the necessity of approximations, the restriction to systems where initial conditions lead to unique solutions, and the restriction to controllable systems.
For that purpose, we algorithmically construct LODE-GPs (Theorem 1), a novel class of GPs whose realizations strictly follow a given system of homogenuous linear ODEs with constant coefﬁcients.
Our construction of LODE-GPs starts by describing our system of ODEs via a so-called operator matrix A .
Calculating its Smith Normal Form (SNF) gives us a decoupled representation of the system via a diagonal operator matrix D , at the cost of going through two invertible base change matrices U, V with the relationship U (cid:1)
A (cid:1) V = D .
We can easily construct a multi-output GP g = GP (0 , k ) (cf.
Lemma 1) with as many outputs as the system channels and whose covariance function encodes the decoupled system.
Then, applying the base change matrix V to g yields the pushforward GP V ∗ g = GP (0 , V kV ′ )
whose covariance function encodes the original system, i.e. the LODE-GP.
We refer to Theorem 1 and its algorithmic proof for details.
Our paper makes the following contributions:
It develops and proves an algorithmic and symbolic construction of LODE-GPs, GPs with the strong inductive bias that their realizations strictly satisfy any given system of homogeneous linear ODEs with constant coefﬁcients (see Theorem 1).
It demonstrates that the constructed GPs are numerically stable and robust (see Section 5).
It automatically includes ODE system parameters as GP parameters, learns them during the training process of the GP, and allows an interpretation of the data (see Subsection 5.2).
It provides a free and public implementation based on the GP library GPyTorch 1 [21]. We successfully test our approach on controllable and non-controllable systems of differential equations, with and without system parameters.
The LODE-GP strictly satisﬁes the ODEs and outperforms GPs by several magnitudes in its precision, additionally it correctly reconstructs the system parameters used to generate the data with a small relative error, even despite noise.
We also discuss how this strict behavior inﬂuences results for metrics like the RMSE with e.g. Figure 4.
Related Work
Using physical information in ML systems became a central research topic over recent years in both
DL and GPs.
With the GP research in two important categories: (1) Works that introduce derivative information as inductive bias into the GP [29, 37, 38, 25, 49, 24] and (2) works that train GPs without inductive bias on physical data, with specially selected standard kernels, as an approximate model for additional processing steps [67, 61, 55, 32, 9, 12, 13].
Of all the works on GPs, [37, 38] are the closest to us, as they also provide general algorithms to introduce inductive bias into GPs, based on Gröbner bases.
These approaches are only applicable to controllable systems.
By basing our algorithm on the SNF instead of Gröbner bases, we overcome this limitation to controllable systems, but are restricted to ODEs .
Probabilistic ODE solvers like [50, 33, 60, 10, 41, 51, 52, 8, 12] are able to include inductive bias of ODEs but sometimes require a number of comparably complex calculations and some approximations.
We avoid any approximation through calculation of the SNF and the application of the pushforward with a linear differential operator.
Hence, our work is limited to linear ODEs, whereas probabilistic ODE solvers are able to work with non-linear ODEs.
For GP priors for decoupled ODE systems with constant coefﬁcients and right hand side functions see [2], which consideres the right hand side functions as latent, hence these models are called Latent
Force Model (LFM).
A GP prior on these latent forces and is pushed forward through differential operators and Green’s operator. We empirically compare LODE-GPs to LFMs in the appendix. Of the other works that introduce inductive bias, [25] and [49] are the earliest works and already discuss a general formulation of differential equations by viewing derivatives as linear operators that can be applied to GPs.
Subsequent works target speciﬁc ODEs and create covariance functions with inductive bias [24, 61, 15].
They can be considered special cases of our approach with a speciﬁc manually constructed pushforward.
Finally, others constructed covariance functions for partial differential equations from standard covariance functions with e.g. curl-free behaviour [67] to model physical behaviour [67, 61, 55, 32] or use the GP as a surrogate model to learn physical problems like the inverse pole problem [13] or a variety of applications [9].
Most DL models explore physically informed models through additional loss terms that punish solutions that deviate from the system [62, 36, 47], or in the case of [17] show the equivalence of the loss to their assumed physical behavior.
This is often combined with different modiﬁcations to the networks like additional features [17] or speciﬁc network structures [36].
Background 3.1 Gaussian Processes
A GP g = GP ( µ, k ) deﬁnes a probability distribution over the space of functions R d !
R ℓ , such that the outputs g ( x i ) at any set of x i 2 R d are jointly Gaussian [70].
Such a (multi-output) GP is deﬁned by its mean function (often set to zero)
[Equation]
and its (multi-output) positive semi-deﬁnite covariance function (also called kernel)
[Equation]
A popular kernel is the Squared Exponential (SE) kernel
[Equation]
, with its signal variance σ and lengthscale ℓ . It models smooth data very well and is thus usable for many datasets [70, p. 83], as its realizations are dense in the space of smooth, i.e. inﬁnitely differentiable, functions C ∞ ( R , R ) [39, Prop. 1].
Manipulating existing GPs allows to introduce indutive biases in various ways [18, 19, 54, 11, 59].
One important example is the application of matrices of linear operators to a GP.
Formally, this is the pushforward operation of an operator matrix B on the GP g as B ∗ g = GP
( Bµ ( x ) ,
Bk ( x, x ′ )
( B ′ ) T ) [4], where B ′ denotes the operation of B on the second argument of k ( x, x ′ )
[38, Lemma 2.2].
These operators can, for example, be differential operators.
The matrix B induces the strong bias such that all realizations lie in the image of B .
This pushforward is typical for applications in differential equations [29, 37, 38, 25, 49] or geometry [27].
Example 1. GPs can be constrained to realizations satisfying a system of linear equations given by a matrix A , e.g. A =
[2 (cid:0) 3] sol F ( A ) :
= [Equation]
with F = C ∞ ( R , R ) the space of smooth input functions.
The matrix B = (cid:20) (cid:21) is maximal in that it solves {+A (cid:1) B = 
+}
and hence sol F ( A ) =
B (cid:1) F = f B (cid:1) f
( x ) j f ( x ) 2 Fg .
Taking a GP prior g = GP (0 , k ) for f ( x ) 2 F and applying the pushforward then yields a new, constrained GP prior B ∗ g = GP (0 , Bk ( B ′
) T )
=
GP (cid:18)(cid:20)  (cid:21) , (cid:20) 9 (cid:1) k SE 6 (cid:1) k SE 6 (cid:1)
k SE 4 (cid:1) k SE (cid:21)(cid:19) for B (cid:1) f
( x ) 2 sol F ( A ) , whose realizations are guaranteed to lie in sol F ( A ) .
This example is similar to our use case of systems of differential equations for LODE-GPs, where we just replace the matrix B of numbers with a suitable matrix of differential operators. x ′′ ( t ) ℓℓg f 1 ( t ) f 2 ( t )
3.2 Smith normal form
The SNF [53, 43] is a normal form for a matrix
A 2 R [ x ] m × n over the polynomial ring R [ x ] , such that U (cid:1)
A (cid:1) V = D .
Here, D 2 R [ x ] m × n is a (not necessarily square) diagonal matrix of same size as A and the base change matrices U 2 R [ x ] m × m and V 2 R [ x ] n × n are invertible square matrices, i.e. det( U ) , det( V ) 2 R nf 0 g [23, 14].
Algorithms to construct the SNF are implemented in many computer algebra systems such as Matlab [42], Maple [40], or SageMath [58] (based on
PARI [57]) which is a free and open source python library for computer algebra.
Intuitively, the
SNF can be compared to the eigendecomposition of a R n × n matrix, which—if it exists—produces a square matrix of eigenvectors W and the diagonal matrix of eigenvalues Λ .
This analogy is lacking since e.g. the SNF produces two independent base change operator matrices V resp.
U instead of a single eigenvector matrix W resp.
its inverse W − 1 .
Still, the matrix D decouples and thereby simpliﬁes a system given by the input matrix A , as does the diagonal eigenvalue matrix Λ [48, 14].
Since the SNF exists for matrices over polynomial rings over any ﬁeld, we can also compute it over a polynomial ring over the function ﬁeld R ( a 1 , . . .
, a k ) .
Hence, we can model differential equations containing parameters a 1 , . .
. , a k .
The heating example in Subsection 5.2 demonstrates how such parameters are being used in the SNF and subsequently learned from data.
Example 2.
A bipendulum acts as our running example.
It consists of a rod with a pendulum attached to each end of the rod, see Figure 2.
The linearized equations of an idealized bipendulum:
[Equation]
with the operator matrix A , ℓ i > 0 the length of the i -th pendulum, g = 9 . 81 the gravitational constant, u ( t ) =
(cid:0) x ′′ ( t ) the rod’s acceleration, and f i ( t ) , i = 1 , 2 the angle between the i -th pendulum and the vertical axis.
The SNF of A reveals its properties, speciﬁcally when ℓ 1 = ℓ 2 and ℓ 1 6 = ℓ 2 .
An algorithm calculates the SNF D of A and the base change matrices U, V as follows: case ℓ 1 = 1 6 = ℓ 2 = 2 :
[Equation] [Equation]
| {z }
V = (cid:20) 1 00 1(cid:21)
| {z }
D case l 1 = l 2 = 1 : (cid:20)
(cid:0) 1(cid:21) | {z }
U (cid:20) ∂ 2 t + g (cid:0) ∂ 2 t + g (cid:0)(cid:21) | {z }
A 4    1 (cid:0) 1 0 ∂ 2 t + g  | {z }
V = (cid:20)  0 ∂ 2 t + g (cid:21) | {z }
D
In case of unequal rope length, the controllability is intuitive when imagining that, by accelerating the rod at the right time, the two ropes can reach every combination of valid positions.
Formally, this follows that D (cid:1) [ h 1 ( t ) h 2 ( t ) h 3 ( t )]
T =
if and only if h 1 ( t ) =
h 2 ( t ) =
and h 3 ( t ) is an arbitrarily choosable output.
In case of equal rope length, the system is not controllable, indicated by the diagonal entry ∂ 2 t + g of D [44, p. 154].
Constructing a GP for differential equations
We introduce a polynomial time algorithm to construct LODE-GPs, a class of GPs with realizations dense in the space of solutions of linear ODEs with constant coefﬁcients.
This ensures that the
LODE-GP is able to produce all possible solutions to the ODEs and nothing but solutions for the
ODEs.
All this is guaranteed to be strictly accurate, since we make no approximations.
Consider a system of linear homogenous ordinary differential equations with constant coefﬁcients
[Equation]
with operator matrix A 2 R [ ∂ t ] m × n determining the relationship between the smooth functions f i ( t ) 2 C ∞ ( R , R ) of f ( t ) =
( f 1 ( t ) . . .
f n ( t )) T .
For such systems our main result holds.
Theorem 1.
(LODE-GPs)
For every system as in Equation (2) there exists a GP g , such that the set of realizations of g is dense in the set of solutions of A (cid:1) f
( t ) = 0 .
The following lemma play a crucial role in the proof of this theorem by constructing base covariance functions for a system as in Equation (2) that is decoupled (via the SNF) into scalar equations. The proof in Appendix D makes use of the solution space being decomposed, in particular is deal wih ﬁnite dimensional vector space like Bayesian linear regression.
Lemma 1.
Covariance functions for solutions of the scalar linear differential equations d (cid:1) f =with constant coefﬁcients, i.e. d 2 R [ ∂ t ] , are given by Table 1 for d is primary, i.e. a power of an irreducible real polynomial.
In the case of a non-primary d , each primary factor d i of d = Q ℓ − 1 i =
d i is ﬁrst translated to its respective covariance function k
i separately before they are added up to give the full covariance function k ( t 1 , t 2 ) =
P ℓ − 1 i =
k i ( t 1 , t 2 ) .
[Table]
[-j (cid:16)P j −i =
-]
For the case of single (i.e. j = 1 ) zeroes, the sum in the covariance function simpliﬁes to a factor 1 .
Example 3.
The ODE d = ∂ t (cid:0) 1 has solutions of the form a (cid:1) exp( t ) for a 2 R . This onedimensional solution space is described by the covariance function k
( t 1 , t 2 ) = exp( t 1 ) (cid:1) exp( t 2 ) = exp( t 1 + t 2 ) .
Proof of Theorem 1.
We begin with a neutral multiplication of A (cid:1) f
= 0 with V (cid:1) V − 1 and a left multiplication by U to decouple the system using the SNF D =
U (cid:1) A (cid:1) V (cf.
Equation (
[Equation]
We assume the GP-prior h (cid:24) GP ( 0 , k ) for this vector p via a GP, where k is a multi-output diagonal covariance function such that each diagonal entry of k is given by Lemma 1.
By this Lemma, the set of realizations of this prior h is dense in the set of solutions of D (cid:1) p = 0 .
The pushforward of h with V yields a GP g that can learn the original system of ODEs.
[Equation]
with V ′ =
V T the operation applied on the second entry of the kernel (i.e. t 2 ) using [38, Lemma 2.2].
As V is invertible, the set of realizations of g is dense in the set of solutions of A (cid:1) f = 0 .
This proof shows that diagonalizing the system to D (cid:1) p = 0 decouples the components of p to expose the intrinsic behavior of the system.
We can also easily interpret the controllability of a system in this decoupled form.
A zero column (or 0 (cid:1) p i = 0 ) stands for a freely choosable function in p , i.e. an output of the system.
A one in a column (or 1 (cid:1) p i = 0 ) stands for something we have no choice in, e.g. an input or a state that is ﬁxed once the desired output was chosen.
Any other entry leads to sinusoidal-exponential behavior, which is uncontrollable in the sense that it cannot be inﬂuenced.
This behaviour is classic in systems of differential equations, as most of them can be split into a controllable and uncontrollable solution space, which can be clearly seen in the entries of D .
Summing up, a system is controllable iff all diagonal entries of D are either zero or one.
This paper improves upon [37] for two reasons: ﬁrst, it decouples the uncontrollable part of a system via the SNF as a direct summand and, second, it constructs a covariance function for this uncontrollable summand.
This is generally impossible for partial differential equations due to [3, 45].
The runtime of the construction of LODE-GPs in Theorem 1 is dominated by computing the SNF.
This has a deterministic polynomial complexity in the size of the (usually small) operator matrices
A [63, 65, 35, 69] with quicker probabilistic Las Vegas algorithms [56], and it can be parallelized [64, 66].
For our examples, the SNF terminates instantaniously.
The application of the base change matrix V is cubic in the (usually small) size of matrices, assuming constant time to apply operators to the covariance functions from Lemma 1.
Once the covariance function is constructed, the complexity of O ( n 3 ) for GPs applies, where n is the (potentially big) number of data points.
Example 4.
We continue Example 2 and consider the SNF of A for l 1 6 = l 2 .
Since the three diagonal entries of the matrix D are 1 , 1 ,
we conclude that latent kernel k has diagonal entries 0 , 0 , k SE with Lemma 1.
Now the pushforward creates the LODE-GP, which requires the following application of two operator matrices; we simplify by removing irrelevant zeroes of the latent covariance.
[Equation]
i
Applying these two operators to the kernel k SE results in the covariance function for the bipendulum.
Experimental evaluation
We demonstrate the effectiveness of LODE-GPs in three examples, where we constrain a LODE-GP using the system description and train it with datapoints, similar to solving an initial value problem.
The Gröbner basis approach of [37] yields precisely the same results for the two controllable examples and is not applicable to the uncontrollable example.
The only other method that could deal with our class of differential equations is [2] 2 , for which a comparison with the three tank system is discussed in the appendix, in the following we compare our model mainly to classic GPs.
Our comparison includes the error in satisfying the ODEs, speciﬁed by the median error the GPs posterior mean function has in satisfying the ODEs at evenly spread points, where we calculate derivatives through ﬁnite differences.
The LODE-GPs hyperparameters (lengthscale and signal variance) are randomly initialized from a uniform distribution on [ (cid:0) 3 , 3] and were trained using Adam [31].
The 25 training datapoints are created uniformly in the intervals [1 , 6] and [ (cid:0) 5 , 5] for the bipendulum and three tank system, resp.
the heating system.
Training and evaluation is repeated [Table] (cid:0) 0 . 0 .0 .f Noisydata Posteriormean
Noisefreedata 2 σ conﬁdence   8 (cid:0) 0 .(cid:0) 0 . 0 .0 .f
Noisydata Posteriormean Noisefreedata 2 σ conﬁdence   8 (cid:0) 0 .(cid:0) 0 . 0 .0 .0 .u Noisydata Posteriormean Noisefreedata 2 σ conﬁdence ten times in each experiment using a GPyTorch [21] implementation of our LODE-GP construction with SageMath [58] to symbolically calculate the SNF.
The training loss is the negative marginal log likelihood (c.f.
[70, Eq.
(2.30)]) which, in GPyTorch, is additionally divided by the number of training data points.
Details of the training, data generation, veriﬁcation process, additional experiments, e.g. without noise, and further system details can be found in the appendix.
5.1 Bipendulum - Controllable
We continue with the controllable variant of the bipendulum from Example 2 with the rope lengths ℓ 1 = 1 and ℓ 2 = 2 .
We create 25 points of training data from a solution of the ODEs as shown in
Figure 7.
We add white noise to the data with standard deviation of 2% of the maximal signal.
Figure 3 compares the error in satisfying the ODE of a LODE-GP and a GP.
The LODE-GP is producing samples that satisfy the ODE with a median error 3 of 2e (cid:0) 6 , for both differential equations.
In comparison, the symbolically computed solution used to generate the training data, has an error of 1 . 75e (cid:0) 7 , only one order of magnitude better than the LODE-GP.
Hence, we conclude that, up to numerical precision, the LODE-GP produces samples that strictly satisfy the ODE, whereas regression models like GPs do not satisfy the differential equations.
The LODE-GP produces roughly 30000 times more precise samples ( 0 .
06 to 2e (cid:0) 6 ) and deviates only slightly from this error.
Further we investigate the RMSE of the models by comparing their posterior mean to noiseless data in the training and evaluation intervals.
The LODE-GP RMSE mostly performs similar compared to the GP RMSE (see Table 2). For the training interval, the LODE-GP achieves basically the
[Figure]
[Figure] [Figure] [Figure] u ( t ) same performance as the GP.
For the evaluation interval the performance worsens due to a SE
GP extrapolating to its (constant zero) prior mean.
The GP can easily reach this mean zero, by smoothly modifying all channels without regard for the system equations.
Since the ODE solution also approaches zero, this leads to a small error in extrapolation.
The LODE-GP has to satisfy the system equations to move the mean function back to mean zero.
Hence, the system produces a large spike 4 in the input channel u ( t ) , before the system can reach mean zero, as seen in Figure 4.
We also analyse the training runtime and see that the LODE-GP is as fast as the GP during the O ( n 3 ) model inference and slower by a factor smaller than two in the O ( n 2 ) covariance matrix calculation.
5.2 Heating system - Controllable with parameters
In this experiment we use a parameterized heating system to show that a LODE-GP can learn physically interpretable system parameters during GP hyperparameter training.
The controllable heating system is depicted in Figure 5.
It uses an input u ( t ) to control a heating element f 1 ( t ) which exchanges heat with an object f 2 ( t ) .
Two physical parameters a and b determine how quickly the heat moves between the objects according to the ODEs:
[Equation]
We generate training data using a solution of the differential equation (see Figure 5) with parameter values a = 3 , b = 1 .
The LODE-GP can be constructed with parameters in the ODEs.
Learning the physical parameters a and b reconstructs the original values successfully with a maximal relative error of less than 2.8% in ten training runs on data without noise.
After adding white noise to the data with standard deviation of 1% of the maximal signal, the parameters were successfully reconstructed with a maximal relative error of 5.3% in ten training runs.
Again, the LODE-GP satisﬁes the original ODEs (with parameters a = 3 and b = 1 ) with an median error of 1e (cid:0) 2 , trained on the noisy data, we refer the reader to Appendix B.3 for a visualization similar to Figure 7. This is bigger than in the previous example, as the LODE-GP only uses approximate parameter values for u ( t ) f 1 ( t ) f 2 ( t )        (cid:0)
f 1 ( t ) f 2 ( t ) u ( t ) f 1 ( t ) f 2 ( t ) f 3 ( t ) u 1 ( t ) u 2 ( t )      0 .f 1 ( t ) f 2 ( t ) f 3 ( t ) u 1 ( t ) u 2 ( t ) a and b , but still smaller than the error of a GP of 0 .
Our interpretation of these results is that the LODE-GP can learn the physically interpretable parameters.
5.3 Three tank system - Non-controllable
We conclude the experiments with a non-controllable ﬂuid system where the water level in three tanks is changed by two pipes.
The system is non-controllable due to pipes’ overlap over the center tank, whose changes directly affect the other tanks.
This system requires multiple non-zero covariance functions in the latent GP to describe both the non-controllable subsystem and the two degrees of freedom, which corresponds to the columns of the SNFs matrix D via Lemma 1:
D = " 1 00 00 0 (cid:0) ∂ t 0#
We use a solution to the system of ODEs (see Figure 6) to generate 25 datapoints, to which we add white noise with standard deviation of 10% of the maximal signal.
The system ODEs are again strictly satisﬁed by the LODE-GP with a median error of 1e (cid:0) 5 in each
ODE.
We refer the reader to Appendix B.4 for a visualization similar to Figure 7.
Hence, LODE-GP can handle larger, non-controllable systems of ODEs, despite high noise.
Conclusion
In this paper we have introduced an algorithmical approach to automatically and symbolically create
LODE-GP, a class of covariance functions for GPs such that their realizations strictly follow a given system of linear homogeneous ODEs.
We have proven that this approach is mathematically sound and veriﬁed its effectiveness in experiments.
We have demonstrated that the learned posterior mean strictly satisﬁes the given system of ODEs up to numerical precision and discussed the consequences of this strict behavior e.g. in their extrapolation, leading to drastic behavior when both following the
ODEs and going back to the prior mean.
Additionally, we automatically trained the physically interpretable parameters of the system and were able to reconstruct their original values with low relative error.
The runtime of the LODE-GP is still asymptotically dominated by the O ( n 3 ) Cholesky decomposition of the covariance matrix, despite bigger constants in the O ( n 2 ) part of constructing the covariance matrix and a (usually very small) constant time precomputation of the covariance function.
Of course, approximate GPs are applicable to our covariance functions.
Beyond applications where the dynamic behavior is known through corresponding ODEs, the combination with kernel search methods [30, 19, 28, 6, 5] has the potential to detect unknown dynamical behavior in data by systematic exploration and use these ﬁndings to generate new interpretable knowledge about the data through the corresponding system. Acknowledgments and Disclosure of Funding This research was supported by the research training group “Dataninja” (Trustworthy AI for Seamless Problem Solving:
Next Generation Intelligence Joins Robust Data Analysis) funded by the
German federal state of North Rhine-Westphalia.
We want to thank the anonymous reviewers for their valuable feedback. Appendices A Constructing and training a LODE-GP
The construction of a LODE-GP, as discussed in the proof of Theorem 1, involves the following steps: Calculate the SNF symbolically using SageMath [58] 2. Construct the covariance function for the GP-prior h (cid:24) GP ( 0 , k ) , according to Lemmaand the last line of Equation3. Calculate the ﬁnal LODE-GP through the pushforward with the matrix V , including the matrix multiplication V (cid:1) k (cid:1) V ′ and applying the derivatives to k
This construction gives us our LODE-GP, which we train using the standard GP training procedure.
All SE kernels that are part of the LODE-GPs have randomly initialized signal variance σ and lengthscale ℓ values chosen uniformly from [ (cid:0) 3 , 3] , which are passed through an exp , during calculation, to ensure numerical stability (e.g. preventing negative lengthscales) and smooth training.
We use an
Adam optimizer with a learning rate of 0 .
1 for 300 training iterations in all experiments, further we use the GPyTorch multitask likelihood, since the LODE-GP inherits the class of a GPyTorch multitask GP.
We reduced the noise constraint for the likelihood from 1e (cid:0) 4 to 1e (cid:0) 10 .
GPyTorch uses the softplus operation to ensure the constraint.
We performed calculations with 64 bit precision.
Training and evaluation was done on our server equipped with a Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz and 64GB RAM running at 3200MHz.
All calculations were done on the CPU.
Training was done on 25 datapoints and was repeated ten times for noisy and noise-free, for both
LODE-GP and GP. At no point in training or evaluation is any data or GP output scaled, i.e. we use all the data as is. B Additional details on experiments and results We compare the values for the GP and LODE-GP parameters across the three experiments in Table 3.
The values for the parameters are their actual values as used in the calculation, i.e. after applying the softplus and exp functions.
The behaviour discussed in Section 5 is clearly visible in the maximum
LODE-GP lengthscale for the bipendulum, where it sometimes learns a basically constant function.
For the bipendulum and the three tank experiments, the LODE-GP generally learns a higher lengthscale, which translates to a smoother behaviour in the outputs.
Analyzing the eigenvalues for the LODE-GP and GP shows that the LODE-GP produces roughly half as many non-zero eigenvalues (i.e. eigenvalue λ > 1e (cid:0) 6 ) compared to the GP, as shown in
Table 4.
This behaviour is expected since the LODE-GP is underlying additional strong constraints due to the inductive bias, which is reﬂected in its eigenvalues.
B.1 Calculating the ODE error
To calculate the error in satisfying the ODEs via ﬁnite differences, we generate uniformly distributed datapoints in the evaluation interval and add duplicate points, shifted by a step of ∆ = 1 e (cid:0) 3 .
We generate either 500 or 333 datapoints, dependent on whether only the ﬁrst or also the second derivative is necessary, ensuring that always a total of 1000 datapoints are created.
We pass the data to the GP resp.
LODE-GP
and extract the resulting mean function.
The ﬁrst and second derivative are then approximated using forward difference.
To ensure numerical stability in the calculation of the second derivative, we pass the ﬁrst order forward difference result through a low-pass ﬁlter, removing high frequency noise from the result.
To ﬁnally calculate the ODE error, we insert the ﬁnite difference values for the corresponding derivatives in the original ODEs and average over the error for each ODE.
[Table]
LODE-GP noisy noise 0.
3.64772e0.
signal variance 10.
lengthscale 5.
LODE-GP no noise noise 1.41396e1.97306e4.83248e5.48139esignal variance 6.
lengthscale 5. GP noisy noise 0. 2.93415e0. signal variance 1.
lengthscale 4. GP no noise noise 1.61976e5.0974e1.57147e1.72362esignal variance 6. lengthscale 5. LODE-GP noisy noise 0.
signal variance 0. lengthscale 5. LODE-GP no noise noise 1.18695e6.09864e [Table]
[Table] B.2 Bipendulum We illustrate how a LODE-GP covariance function might look like by showing the (1,1) entry for the bipendulum LODE-GP covariance function: [Equation]
System details Since the bipendulum has been thoroughly discussed in the paper [Equation] we just brieﬂy mention the system equations, operator matrix and the SNF: [Equation]
[Table] [Equation] To generate the data we use the following solution to the ODE (cf. Figure 7):
[Equation] [Figure] [Equation] [Equation]
Training details To train a LODE-GP we generate uniformly spaced training data from Equationin the interval [1 , 6] to which we add noise of the form 0 . 012 (cid:1) σ n for σ n (cid:24) N (0 , 1) , i.e. noise with standard deviation 0 . 012 ( 2 % of the maximal signal value). The data is shown in Figure 7.
To validate our results, e.g. using RMSE, we generate uniformly spaced evaluation data from Equationin the interval [1 , 11] , without adding noise. B.3 Heating System details Original system equations:
[Equation]
u
[Equation] [Figure] [Figure]
(cid:0) 6 (cid:0) 4 (cid:0)    (cid:0)   f NoisydataPosteriormeanNoisefreedata 2 σ conﬁdence (cid:0) 6 (cid:0) 4 (cid:0)    (cid:0)   f NoisydataPosteriormeanNoisefreedata 2 σ conﬁdence (cid:0) 6 (cid:0) 4 (cid:0)    (cid:0)(cid:0)  u NoisydataPosteriormean
Noisefreedata 2 σ conﬁdence (cid:0) 6 (cid:0) 4 (cid:0)    (cid:0)
DataPosteriormean Originaldata 2 σ conﬁdence (cid:0) 6 (cid:0) 4 (cid:0)    (cid:0) DataPosteriormean Originaldata 2 σ conﬁdence (cid:0) 6 (cid:0) 4 (cid:0)    (cid:0)(cid:0)
DataPosteriormean Originaldata 2 σ conﬁdence (cid:0)  (cid:0)(cid:0)  f 1 ( t ) f 2 ( t ) u ( t ) GP is shown. The SNF of the ODEs with all relevant matrices:
[Equation]
To generate data, we use the following solution to the heating ODEs (cf. Figure 8). [Equation] [Equation]
[Equation]
Training details To train a LODE-GP we generate uniformly spaced training data from Equationin the interval [ (cid:0) 5 , 5] to which we add noise of the form 0 . 02 (cid:1) σ n for σ n (cid:24) N (0 , 1) , i.e. noise with
[Table]
standard deviation 0 . 02 ( 1 % of the maximal signal value). The data is shown in Figure 8. To validate our results, e.g. using RMSE, we generate uniformly spaced evaluation data from Equation 6 in the interval [ (cid:0) 9 , 9] , without adding noise.
Additional results We present the exact parameter values trained by the LODE-GP in Table 5. It can be seen that the LODE-GP consistently learn the system parameters a and b with a small relative error, even despite noise. B.4 Three tank Original system equations:
[Equation]
[Equation]
[Equation]
" (cid:0) ∂
[Equation]
1 (cid:0) ∂ t 1  (cid:0) ∂ t 0# | {z } A (cid:1)  f 1 ( t ) f 2 ( t ) f 3 ( t ) u
[Equation]
t ) u
[Equation]
t
[Equation]
The SNF of the ODEs with all relevant matrices:
[Equation]
[Equation] # | [Equation] [Equation]
{+1 (cid:0) [Equation] 1  (cid:0) ∂ t 0# | {z } A  0 0(cid:0) 0 
+}
(cid:0)0 0 (cid:0)1 0 0 (cid:0) ∂ t 0 1∂ t (cid:0) ∂ t  | {z } V = " 1 00 00 0 (cid:0) ∂ t 0# | {z } D We use the following solution to the ODEs (cf.
Figure 9).
[Equation] [Equation] [Equation] exp
[Equation] [Equation] Training details To train a LODE-GP we generate uniformly spaced training data from Equationin the interval [1 , 6] to which we add noise of the form 0 . 08 (cid:1) σ n for σ n (cid:24) N (0 , 1) , i.e. noise with standard deviation 0 .
( 10 % of the maximal signal value). The data is shown in Figure 9. To validate our results, e.g. using RMSE, we generate uniformly spaced evaluation data from Equationin the interval [1 , 11] , without adding noise.
For the three tank system we hard coded the case (cid:0) x = x , which is a legal operation since we can take out the (cid:0) 1 as part of a left matrix multiplication and integrate it in the U matrix, which does not inﬂuence the LODE-GP. [Table] a t a [Table]
o r m ea n [Table] e fr [Table]
a t a [Table] c
[Table] ﬁ d e n ce [Table] r m ea
[Table]
F i gu r e 9 :
T h
e po s t e r i o r
L OD E - G P m od e l s f o r t h e t
h r ee t a nk s y s t
e m , t r a i n e d on no
i s y d a t a ( t op ) a nd
no i s e -fr ee
d a t a ( bo tt o m ) .
T h
e b l ac k s t a r
s
i nd
i ca t e t h e
d a t a po i n t s ( w it h
o r w it hou
t
no i s e ) , t h e r e d li n e i s t
h e s o l u t
i on t o t h e
OD
E s , t
h e
b l u e d a s h e d li n e i s t h e
L
OD
E - G
P s po s t e r
i o r m ea n , t h e t r a n s p a r e n t b l u e a r
ea i s t
h e 2 σ c on ﬁ d e n ce i n t e r v a l .
I n t h e bo tt o m p l o t , a s a m p l e fr o m t h e
G P
i s s ho w n .
Comparison to Latent Force Model
We compare our LODE-GP with the LFM introduced by [2].
To do so we set the variables from Equation 3 in [2] as follows:
D q = 0 , B q = 0 ,
S rq =
[Equation]
[Equation]
Further we calculate the covariance function as the solution of the integral k =
R t 2 0 R t 1 0 exp − ( x 1 − x 2 ) 2 dx 1 dx 2 , effectively setting ℓ = 1 and σ = 1 .
Following the steps of [2], we get a GP that can estimate the solution of the three tank systems differential equations.
The resulting GP marginalizes the (in their model considered) latent function u 1 and u 2 and only considered the three data channels f 1 , f 2 , and f 3 .
To calculate the ODE error, we inserted the original data for the channels u 1 and u 2 into the calculation.
For a fair comparison with our LODE-GP, we also marginalize u 1 and u 2 there.
Similarly as for the
LFM, we have set ℓ = 1 and σ = 1 .
The resulting ODE errors of the two models are as shown in Table 6.
The performance of the marginalized LODE-GP is better but comparable to the LFM.
The performance of the full LODE
GP, having learned all 5 channels and also set all lengthscales and signal variances to 1 , shows a signifcant increase in performance.
[Table]
C Discussion of the code and runtime
We use GPyTorch as the core of our code and build our LODE-GP on top of their work, which makes it an essential part of our implementation.
Nevertheless, it could be replaced with other GP libraries if the kernel would be rewritten for those.
We use SageMath to calculate the symbolic SNF, given the system of ODEs.
In our current implementation, SageMath is a necessary library, but could be replaced by any other computer algebra system, with a Python interface, that symbolically solves the SNF for matrices A 2 R [ x ] m × n over the polynomial ring R [ x ] , and additionally the function ﬁeld R ( a 1 , . .
. , a k )
[ x ] if there are parameters in the ODEs.
Training a LODE-GP on 25 multidimensional 5 datapoints with the stated training speciﬁcations needed, in the median, between 3 and 5 seconds per training and, 7 seconds for the three tank system, due to its greater size, as shown in Table 7.
This result is in accordance to our ﬁndings in Section 5, where we discuss that the O ( n 2 ) calculation takes longer, whereas the O ( n 3 ) stays roughly the same.
Due to the small number of datapoints, the O ( n 2 ) calculations have a strong inﬂuence on the runtime.
The GP needed roughly 15 seconds per training run.
Additionally, generating the model, i.e. the SNF calculation, preparing the differentiated covariance function and initializing the model object, took averagely 0.3 seconds.
Of this time, the SNF calculation takes, on average, 0.15 seconds.
These calculations are performed only once for the model and are neglegtable, compared to the training runtime.
During development, we encountered an interesting problem we want to document for other developers.
Namely that PyTorch only trains parameters that are part of the model object itself (i.e. which can be called using the self keyword).
For example, we can’t just use a list, that is part of the model object, in which we store the covariance functions, this would cause PyTorch to not recognize the parameters and not train them.
In our case, the various kernels and ODE parameters, that are added dynamically, required us to give each parameter and kernel a name and add it to the kernel individually.
We also store a position matrix with references to the respective kernel at that position, e.g. for
[Table]
the bipendulum covariance function we have a 3 (cid:2) 3 matrix with different kernel objects for each position.
Additionally, we want to highlight the On-Line Encyclopedia of Integer Sequences, which we used to ﬁnd the construction formula of the number sequence for the general SE kernel derivative.
D Proof for base covariance function construction
We prove the following Lemma
Lemma 1.
Covariance functions for solutions of the scalar linear differential equations d (cid:1) f
=with constant coefﬁcients, i.e. d 2 R [ ∂ t ] , are given by Table 8 for d is primary, i.e. a power of an irreducible real polynomial.
{+In the case of a non-primary d , each primary factor d i of d = Q ℓ − 1 i =
 d i is ﬁrst translated to its respective covariance function k i separately before they are added up to give the full covariance function k ( t 1 , t 2 ) = P ℓ − 1 i =
+}
k i ( t 1 , t 2 ) .
[Table] [Table]
a
[Table]
)( ∂
[Table]
(cid:0)
[Table]
+
[Table]
{+j (cid:16)P j − [Table] i =
 t+}
[Table]
1 t
[Table]
(cid:17)
[Table]
[Table]
[Equation] Proof. For a function f we have 1 (cid:1) f = 0 if and only if f = 0 .
Such functions are described by the zero covariance function (and zero mean function).
{+The real differential equation ( ∂ t (cid:0) a ) j (cid:1) f = 0 only has the analytic solutions given symbolically by P j −i =
+}
a i (cid:1) t i (cid:1) exp( a (cid:1) t ) for arbitrary a i 2 R .
This is a ﬁnite dimensional space and hence (as for linear regressions problems) a covariance function is given by P ji =
[Equation]
t [Equation] + [Equation]
The real differential equation (( ∂ t (cid:0) a ) 2 + b 2 ) j
{+= (( ∂ t (cid:0) a (cid:0) ib )( ∂ t (cid:0) a + ib )) j only has the analytic solutions given symbolically by exp( a (cid:1) t ) (cid:1) P j − 1 i =
+}
t i (cid:1) ( a i (cid:1) cos( b (cid:1) t )
+ b i (cid:1) sin( b (cid:1) t )) for arbitrary
a i 2 R .
This is again ﬁnite dimensional space and hence a covariance function is given by
[Equation]
The factor cos( b (cid:1) ( t 1 (cid:0) t 2 )) is sometimes called the cosine covariance function.
For a smooth function f 2 C ∞ ( R , R ) , the equation 0 (cid:1) f = 0 poses no restriction.
Hence, by [39, Prop. 1], the squared exponential covariance function yields a GP with realizations densely contained in the space C ∞ ( R , R ) .
E Licenses and versions
We use Python 3.9.7, which uses the PSF license agreement.
We use GPyTorch version 1.5.0 for our experiments, GPyTorch is distributed under the MIT license.
We use SageMath version 9.5, released 2022-01-30, for our experiments, SageMath is distributed under the CC BY-SA 4.0 license.