Symplectic Spectrum Gaussian Processes: Learning Hamiltonians from Noisy and Sparse Data
Abstract
Hamiltonian mechanics is a well-established theory for modeling the time evolution of systems with conserved quantities (called Hamiltonian ), such as the total energy of the system.
Recent works have parameterized the Hamiltonian by machine learning models (e.g., neural networks), allowing Hamiltonian dynamics to be obtained from state trajectories without explicit mathematical modeling.
However, the performance of existing models is limited as we can observe only noisy and sparse trajectories in practice.
This paper proposes a probabilistic model that can learn the dynamics of conservative or dissipative systems from noisy and sparse data.
We introduce a Gaussian process that incorporates the symplectic geometric structure of Hamiltonian systems, which is used as a prior distribution for estimating Hamiltonian systems with additive dissipation.
We then present its spectral representation, Symplectic Spectrum Gaussian Processes (SSGPs) , for which we newly derive random Fourier features with symplectic structures.
This allows us to construct an efﬁcient variational inference algorithm for training the models while simulating the dynamics via ordinary differential equation solvers.
Experiments on several physical systems show that SSGP offers excellent performance in predicting dynamics that follow the energy conservation or dissipation law from noisy and sparse data.
Introduction
There is great interest in the data-driven approach for learning the dynamics of physical systems.
Modeling, simulating, and forecasting dynamics from data are fundamental in engineering and physical sciences; the data-driven approach also has the potential to discover new laws in physics [17, 38].
Many real-world systems can be described by ordinary differential equations (ODEs).
Classical data-driven approaches make strong assumptions about the form of the equations; thus, they are not applicable if the system’s equation is unknown [8, 32, 33, 37, 42].
The pioneering work of neural ordinary differential equation (NODE) [4] has been presented as a general black-box model for learning vector ﬁelds , represented by functions, to output time derivatives of the system’s state; the state of the system is continuously transformed along the vector ﬁelds (see the right part of
Figure 1).
The Hamiltonian neural network (HNN) [14] and variants (e.g., [6, 12, 49]) allow one to learn vector ﬁelds such that the total energy of the system (called Hamiltonian ) is conserved.
This formulation is advantageous for learning dynamics that follow the basic laws of physics (i.e., the energy conservation law).
Several extended models cover Hamiltonian systems with additive dissipation [26, 48].
However, neural network based models implicitly assume that a large amount of training data with high temporal resolution is available.
This paper addresses the problem of learning Hamiltonian dynamics from noisy and sparse trajectories and predicting the dynamics from an arbitrary initial condition.
We illustrate our problem in Figure 1.
The state trajectories are assumed to be sparse in the following senses: 1) Limited observation trials and 2) low temporal resolution (see Figure 1).
One promising approach to alleviate overﬁtting in sparse settings is a Gaussian process (GP) [35], which allows for model learning while considering that the data contains uncertainties.
GP models have been proposed for inferring unknown dynamics from trajectories [16, 28, 29], in which GPs are used for modeling the vector ﬁelds.
They utilize a GP approximation with random Fourier features (RFFs) [34] to learn the dynamics via ODE solvers.
The approximation is needed for the following reasons.
Since we can obtain only trajectory data, not derivative observations (i.e., direct observations of vector ﬁelds), we cannot use the standard GP posterior conditioned on data points; thus, we require posterior approximation.
The RFF-based approximation avoids the prohibitive computational complexity of generating sample paths (i.e., realizations of the vector ﬁeld) from the GP posterior (details are discussed in [45, 46]).
This advantage in computation is essential for ODE-based learning.
Since they do not, however, consider Hamiltonian mechanics, it is difﬁcult to accurately capture dynamics that follow physical laws such as energy conservation and dissipation.
GP regression models for derivative observations have been proposed [39], in which the covariance function is given by matrix-valued kernels for estimating vector ﬁelds.
Recently, symplectic Gaussian process regression (SympGPR) [36] has been proposed; the covariance function for learning conservative vector ﬁelds is derived by incorporating the theory of Hamiltonian mechanics into GP modeling.
However, SympGPR assumes that one can obtain derivative observations, not trajectories, for training.
Although it can be naively applicable to our problem by using ﬁnite differences, it is difﬁcult to learn the dynamics accurately from noisy state observations with low temporal resolution.
Also, it is not applicable to Hamiltonian systems with dissipation.
This paper proposes a GP framework that can infer Hamiltonian systems with additive dissipative terms from noisy and sparse trajectories.
We ﬁrst extend the GP prior for Hamiltonian dynamics proposed in [36] to handle additive dissipation.
The conserved quantity of the systems (i.e., Hamiltonian) is assumed to be a single-output GP.
By employing Hamilton’s equations, the vector ﬁelds are derived as a multi-output GP whose covariance function incorporates the symplectic structure for the energy conservation law. Moreover, one can handle Hamiltonian dynamics with friction by introducing the dissipation matrix.
The most signiﬁcant contribution of this work is the derivation of the RFFs that encode the symplectic structure to obtain the GP approximation for dissipative
Hamiltonian systems, which we call Symplectic Spectrum Gaussian Processes (SSGPs) .
This can efﬁciently generate samples of the Hamiltonian vector ﬁelds from the GP posterior and construct the scalable learning algorithm based on ODE solvers.
Inference in SSGP is based on variational Bayes, which allows for learning the Hamiltonian vector ﬁelds while considering the uncertainties posed by noisy and sparse data.
Another beneﬁt of SSGP is that it can be used for decomposing the dynamics into conservative and dissipative terms allowing the dynamics for unseen friction coefﬁcients to be predicted.
Such a task cannot be handled with models (e.g., NODE) that do not use prior knowledge available in physics.
The main contributions of this work are as follows:
• We introduce a GP prior for modeling Hamiltonian systems with additive dissipation.
• We propose its spectral representation (called SSGP) by deriving RFFs that incorporate the symplectic structure of Hamiltonian systems.
• We develop a variational inference procedure for SSGP that offers numerical integration by ODE solvers as a subroutine.
• Experiments on several physical systems show that SSGP can accurately predict the dynamics that follow the conservation or dissipation laws from noisy and sparse trajectories 1 . [Table]
Related Work
This work is positioned in the series of studies for learning black-box models that can estimate
Hamiltonian dynamics from data.
Here, black-box means that explicit mathematical modeling of differential equations is not required.
We describe below prior works in this research line.
Tableshows the comparison of the proposed model with the existing representative models.
Neural network models.
Time-evolution of system states is generally described by differential equations.
The neural ordinary differential equation (NODE) [4] and its variants [3, 20] have been proposed for learning the continuous-time evolution of the states from data.
In these studies, the vector ﬁelds that determine the state evolution are parameterized by deep neural networks (DNNs) and estimated by back-propagating errors of observed trajectories via the ODE solver.
The Hamiltonian neural network (HNN) [15] introduced prior knowledge of Hamiltonian mechanics as an inductive bias for training DNNs; the core concept is to parameterize the Hamiltonian (i.e., energy function) using DNNs.
This formulation allows one to obtain dynamics that follow the energy conservation laws.
Although the HNN assumes derivative observations for training, a learning procedure with ODE solvers from state trajectories was proposed in [6, 49].
Several models have been extended to handle Hamiltonian systems with additive dissipation [40, 48].
Most recent studies have further expanded the scope of application, including Hamiltonian systems with controllable inputs [9], stiff Hamiltonian dynamics [24], energy-conserving partial differential equation systems [26], odddimensional chaotic systems [7], and Poisson systems [5, 18]. Another approach, SympNets [19, 47], models the symplectic map using neural networks, which are shown to be universal approximators.
However, existing DNN-based models implicitly assume a situation wherein a large amount of training data with high temporal resolution is available; they might fail to capture dynamics in the noisy and sparse settings this work focuses on.
Gaussian process models.
Gaussian process (GP) modeling has the advantage that it makes it possible to learn models while considering uncertainties present in data [35].
Solak et al.
[39] proposed GP regression models for derivative observations, in which the covariance function is given by matrix-valued kernels for inferring vector ﬁelds.
Curl-free and the divergence-free kernels have been introduced for learning conservative vector ﬁelds [1, 25]; however, they lack the ﬂexibility to be extended to cover various physical dynamics as described in the previous paragraph.
Rath et al.
extended the model in [39] to learn Hamiltonian vector ﬁelds, which is called symplectic Gaussian process regression (SympGPR)
In SympGPR, the conserved quantity (i.e., Hamiltonian) is deﬁned by a single-output GP; the covariance function for Hamiltonian vector ﬁelds can then be naturally derived on the basis of Hamilton’s equations.
However, these GP regression models assume derivative observations, not state trajectories.
Although the use of ﬁnite differences allows for naively applying them to the problem of learning from trajectories, they might fail to capture dynamics, especially when the temporal resolution is lower, because each ﬁnite difference is computed from state pairs at neighboring discrete time steps.
Most recently, a few models combined GPs with symplectic integrators [11, 30]; however, all the GP models [11, 30, 36] are not applicable to Hamiltonian systems with additive dissipation.
Learning with ODE solvers.
Numerical integration methods play an essential role in learning dynamics from trajectory data.
Chen et al.
[4] provide a convenient tool for solving ODEs and learning parameters via back-propagation or memory-efﬁcient adjoint methods, which are widely used for learning Hamiltonian dynamics (e.g., [9, 12, 49]).
One can also use advanced learning methods based on symplectic integrators [10, 27, 50] or discrete gradients [26], which can evaluate the loss function and its gradient while preserving energy accurately.
These studies are related to ours but different in the research focus.
They focus on how to estimate the parameters given the models (e.g., DNNs); whereas our proposal is a model whose learning method can be chosen as needed.
It might be possible to employ the above techniques for learning our model.
Different from the above studies, this work can also consider uncertainties to learn the model from noisy and sparse data.
Hamiltonian Mechanics
In this section, we brieﬂy review Hamiltonian mechanics [13].
Let us consider a system with N degrees of freedom.
In the Hamiltonian formalism, the continuous-time evolution of the system is described in phase space, that is, the product space of generalized coordinates x q = ( x q1 , . . .
, x q N ) and generalized momenta x p = ( x p1 , . . .
, x p N ) .
Let x = ( x q , x p ) ∈ R D be a state of the system, where D = 2 N .
The system’s evolution is determined by the Hamiltonian H ( x ) : R D → R , which denotes the system’s total energy.
Traditionally, the Hamiltonian is manually designed to suit the system.
The dynamics of a Hamiltonian system with additive dissipative terms is given by
[Equation]
where S = (cid:18) O I
[Equation] I O (cid:19) . Here, ∇ H ( x ) : R D → R D is the gradient of the Hamiltonian with respect to state x , S ∈ R D
× D is the skew-symmetric matrix, R ∈ R D × D is the positive semi-deﬁnite dissipation matrix, I is the identity matrix, and O is the zero matrix.
In (1), we deﬁne the time derivatives of the state by the function f ( x ) : R D → R D , which is a special kind of vector ﬁeld that has a symplectic geometric structure (called Hamiltonian vector ﬁeld or symplectic gradient).
The dynamics on this vector ﬁeld conserve the total energy when R =
O .
One example of the dissipation matrix is
R = diag(0 , . .
. , 0 , r 1 , . .
. , r N ) , representing a dissipative system with friction coefﬁcient r n ≥ 0 .
Although we assume this kind of dissipation matrix in the following, this work is extendable to the general dissipation matrix, such as the state-dependent damping term [9].
Given vector ﬁeld f ( x ) and initial condition x 1 at time t 1 , one can predict state x t at time t by integrating f ( x ) from t 1 to t , as follows: x t = x [Equation] +
[Equation]
t t 1 f
[Equation]
dt .  Model
We propose SSGP (Symplectic Spectrum Gaussian Process), a probabilistic model for learning
Hamiltonian systems with additive dissipation from noisy and sparse trajectories.
Figure 2 shows a schematic diagram of the proposed generative processes of noisy trajectories.
In the following, we ﬁrst introduce a Gaussian process (GP) prior for modeling conservative and dissipative vector ﬁelds by incorporating the theory of Hamiltonian mechanics.
We then derive its spectral representation, for which we propose random Fourier features that encode symplectic structures.
This spectral representation is used for constructing a scalable inference algorithm described in Section 5.
Finally, we describe the generative processes of noisy state observations.
GP priors for Hamiltonian systems with additive dissipation.
In the proposed model, the unknown
Hamiltonian H ( x ) is assumed to be a single-output GP with zero mean.
Let L :=
( S − R ) ∇ denote a differential operator.
According to (1), the vector ﬁeld can be represented using L , as follows:
[Equation]
where H ( x [Equation] ∼ GP (0 , γ ( x , x (cid:48)
[Equation] [Equation]
Here, γ ( x , x (cid:48) ) : R D × R D
→ R is a covariance function.
Since differentiation is a linear operator, the derivative of a GP is again a GP [39]; thus, f ( x ) is given by a multi-output GP,
[Equation]
where 0 is a column vector of 0’s, and K ( x , x (cid:48) )
: R D × R D → R D × D is the matrix-valued covariance function represented by
[Equation]
Here, ∇ 2 is the Hessian operator.
As described above, we can obtain the GP prior (3) with the covariance function (4) that encodes the geometric structure of Hamiltonian systems with additive dissipation.
This formulation can be regarded as a generalization of the GP for non-dissipative
Hamiltonian systems [36].
Though one can opt for any positive deﬁnite kernel as γ ( x , x (cid:48) ) , we assume that γ ( x , x (cid:48) ) is shift-invariant in the approximation described in the next paragraph.
One of the most widely used shift-invariant kernels is the ARD (Automatic Relevance Determination)
Gaussian kernel,
[Equation]
where Λ = diag (cid:0) λ 21 , . .
, λ 2 D (cid:1) .
Here, σ 20 ∈ R > 0 and λ 2 d ∈ R > 0 are the signal variance and the length scale, respectively.
Spectral representations.
We present the approximation of the GP prior (3), for which we newly derive random Fourier features (RFF) that encode symplectic structures of Hamiltonian systems.
The RFF-based approximation is advantageous for 1) estimating the GP posterior for vector ﬁelds without derivative observations and 2) efﬁciently sampling the vector ﬁeld from the GP posterior, which has been suggested in recent studies on learning ODEs [16, 28, 29, 45].
These advantages make it feasible to train our GP model for Hamiltonian systems with dissipation by utilizing ODE solvers, even when only trajectory data, not derivative observations, are available (see also the On computational complexity paragraph in Section 5 and Appendix E).
We begin by modeling the Hamiltonian H ( x ) as a GP approximation,
[Equation]
where w
[Equation]
Here, we adopt the M pairs of basis functions 2 , φ m ( x ) = (cid:2) cos(2 π s (cid:62) m x ) , sin(2 π s (cid:62) m x ) (cid:3) (cid:62) , parameterized by a D -dimensional column vector s m of spectral points.
The point s m is sampled from the kernel’s spectral density, which is given by
[Equation]
in the case in which the ARD Gaussian kernel (5) is used [23].
In (6), w m ∈ R 2 is a row vector of weights for the m th pair of basis.
We apply the differential operator L to the approximation (6); which yields the spectral representation (i.e., SSGP) of (3), represented by
[Equation]
where w = ( w 1 , . . . , w M ) , and the resulting feature maps are deﬁned by Ψ ( x ) = ( Ψ 1 ( x ) , . . .
, Ψ M ( x )) ; the m th feature map Ψ m ( x ) : R D
→ R D × 2 is given by
[Equation] [Equation]
which we call symplectic random Fourier features (S-RFF) .
The derivation of (8) is described in
Appendix A. S-RFF (9) is the ﬁrst to incorporate symplectic structures into the random features for modeling vector ﬁelds by leveraging the knowledge of Hamiltonian mechanics.
This result is not trivial in that we bridge two clearly distinct research topics, random features for kernel machines (i.e.,
GPs) and Hamiltonian mechanics.
Let us explore the connection between the exact GP (3) and its spectral approximation (8).
Denoting Dirac’s delta function by p ( f | w ) , the distribution of f is given by integrating out w from (8), as follows 3 :
[Equation]
where
[Equation]
Here, ˜ K ( x , x (cid:48) ) :
R D × R D → R D × D is the covariance function for GP approximation.
We observed that approximation quality improves with the number of spectral points (i.e., the number of basis functions) by comparing the gram matrix of K ( x , x (cid:48) ) (4) with that of ˜ K ( x , x (cid:48) ) in (10), where the spectral points s m are sampled from (7).
We show the gram matrices in Appendix B.
Generative processes of noisy observations.
Suppose that we have a collection of I trajectories { ( t ij , y ij ) | i = 1 , . .
, I ; j = 1 , . . .
, J i } , where J i is the number of samples in the i th trajectory.
Each sample is speciﬁed by a pair ( t ij , y ij ) , which represents the observation of noisy state y ij at time t ij .
We treat the noiseless state x ij , the counterpart of y ij , as a latent variable.
We assume that the observation model of y ij is a Gaussian distribution with a variance of σ 2 .
Letting Y =
{ y ij } , the marginal likelihood (i.e., evidence) is given by
[Equation]
where p ( f ) is the GP prior (10) of the vector ﬁeld, and p ( x i 1 ) is the prior distribution 4 of the initial condition x i 1 .
Given f and x i 1 , the state x ij is deterministically given by solving the ODE; thus, we can write the conditional distribution p ( x ij | f , x i 1 ) in (11) using Dirac’s delta function, as follows:
[Equation]
Note that, although we omit the observation time points { t ij } in (11), it is actually conditioned on { t ij } .
For simplicity, we adopt this notation hereinafter.
Inference
Although we would like to estimate the model parameters w , Λ , σ 20 , σ 2 and R by maximizing the logarithm of (11), we cannot calculate the exact marginal likelihood (11) analytically as it includes the process of solving the ODEs.
We then present a variational inference procedure for learning Hamiltonian systems with additive dissipation from noisy and sparse trajectories.
The spectral representation of SSGP is used for efﬁcient sampling of the conservative or dissipative vector ﬁelds from the GP posterior, allowing the learning to proceed with ODE solvers while handling uncertainties in the vector ﬁelds.
Parameter learning.
We consider the following evidence lower bound (ELBO),
[Equation]
where q ( x ij ) and q ( w ) are the variational distributions of the noiseless state x ij and the weights w , respectively.
We assume that the variational distribution of w is given by a Gaussian distribution, q ( w )
=
N ( b , C ) , where b ∈ R 2 M and C ∈ R 2 M [-⇥ 
-]× 2 M are the mean and the covariance matrix, respectively.
For j = 1 , we assume that the variational distribution of initial condition x i 1 is given by a Gaussian distribution with the mean being the observed state y i 1 , q ( x i 1 ) =
N ( y i 1 , A ) , where
A ∈ R D × D is the covariance matrix.
For j ≥ 2 , the variational distribution of x ij is given by
[Equation]
where the factor in square brackets is the variational distribution of f .
The derivation of (13) is described in Appendix C. We approximate the expectation in (13) using Monte Carlo integration,
[Equation]
where the Monte Carlo samples are given by
[Equation]
where (cid:15) ( k ) i
[Equation] [Equation]
where we use the reparameterization trick [21] in (16), and we perform the numerical integration by the ODE solvers (e.g., the Runge-Kutta method) in (17).
Here, K is the number of Monte Carlo samples to obtain (15), and √ · denotes a matrix square root. f ( k ) ( x ) in (17) is the sample of the vector ﬁeld generated from the variational posterior of f ,
[Equation]
where w
[Equation] [Equation]
Here, L is the number of Monte Carlo samples to obtain (18), and (cid:15) ( k,l ) ∼ N ( 0 , I ) .
In the spectral representation, the randomness of function f is totally controlled by the distribution of weights w .
Accordingly, we can optimize both model parameters and variational parameters while considering uncertainties in the vector ﬁeld by sampling w at each training iteration.
The inference procedure is shown in Appendix D. We can easily extend our proposed approach to learn Hamiltonians from high-dimensional data (such as images) by combining an autoencoder with an SSGP, as in [14, 43].
The states x and x (cid:48) in the covariance function K ( x , x (cid:48) ) (4) are modeled as the latent vectors of the autoencoder.
This formulation can be regarded as an extension of the SSGP based on deep kernel learning [44].
Prediction.
The variational posterior, q ( f ) =
(cid:82) p ( f | w ) q ( w ) d w , has a closed-form solution, a
Gaussian distribution whose mean function and covariance function are given by ˜ m ∗ ( x ) = Ψ ( x ) b (cid:62) and ˜ K ∗ ( x , x (cid:48) ) =
Ψ ( x ) CΨ (cid:62) ( x (cid:48) ) , respectively.
One can predict the dynamics from an arbitrary initial condition by numerically integrating the mean function ˜ m ∗ ( x ) .
Also, one can evaluate the predictive uncertainty for the vector ﬁeld by using the covariance function ˜ K ∗ ( x , x (cid:48) ) .
On computational complexity.
The computational bottleneck is the sampling of w ; the cost of computing √ C in (18) is O ( M 3 ) .
The cost increases with the number of basis functions; however, the most important thing is that the bottleneck is outside of the ODE solver.
The sampling of w makes it possible to efﬁciently evaluate function values at arbitrary point x , as in (18), and to use it in the ODE solver in (17).
The advantage in computational costs is discussed in Appendix E. In addition, previous studies have shown experimentally that the GP models approximated by RFF have, in many cases, high predictive performance even when setting M lower than 10 3 (e.g., [23]).
Actually, our experiments show that SSGP yields an accurate prediction using a small set of RFFs.
The average training time when setting M = 250 was 2943.0 seconds for the dataset of the pendulum with friction; the experiments were conducted on the AMD EPYC 7313 CPU (3.0GHz).
Experiments
Data.
We evaluated the proposed model, SSGP, using two physical systems: pendulum , and Dufﬁng oscillator . The Hamiltonian of the pendulum system is [Equation]
where we denote the gravitational constant by g , the mass constant by m , and the length of the pendulum by l .
In the experiments, we set g = 3 and m = l = 1 .
The Hamiltonian of the Dufﬁng oscillator system is [Equation] where we set the parameters, α = β = 1 . For each system, the dissipation matrix was set to R =
O (energy conservation) or R = diag(0 , 0 . 05)
(energy dissipation).
We generated trajectory data by employing a numerical integrator, i.e., the Dormand–Prince method with adaptive time-stepping, implemented in torchdiffeq 5 [3, 4].
We sampled initial conditions with total energies uniformly distributed across a predeﬁned range.
The energy ranges for pendulum and Dufﬁng oscillator were [1 . 3 , 2 . 3] and [0 . 5 , 1 . 5] , respectively.
To evaluate the robustness of our model against the degree of sparsity, we prepared { 10 , 15 , 20 , 30 , 50 } trajectories sampled at a frequency of { 3 , 5 , 10 } Hz forseconds, and added Gaussian noise with variance σ 2 = 0 .
1 to each sample.
We randomly split the trajectory data and used 70% for training and 30% for validation.
Here, if the number of trajectories for validation was less than ﬁve, we used ﬁve trajectories for validation and the rest for training.
We generated a test set of 25 trajectories independently from training and validation sets; each test trajectory was sampled at a frequency of 100 Hz for 15 seconds.
The experiments were conducted ﬁve times by resampling the training and validation sets.
Task 1: Normal prediction.
We evaluated performance by comparing the predicted state trajectories { ˆ x ij } with the ground truth { x true ij } (i.e., the test set described above).[-The evaluation metric is themean squared error (MSE), 
 I-]
The evaluation metric is the mean squared error (MSE), 1 I (cid:80) Ii =1 ( 1 J i (cid:80) J i j =1 (cid:107) ˆ x ij − x true ij (cid:107) 2 ) , where (cid:107) · (cid:107) 2 is the Euclidean norm.
In the evaluation, we used the datasets from four settings: pendulum and Dufﬁng oscillator with or without energy dissipation.
Task 2: Predicting the dynamics for unseen friction coefﬁcients.
One beneﬁt of SSGP is that it can decompose the dynamics into its conservative and dissipative terms, thus predicting the dynamics for arbitrary friction coefﬁcients.
To show this, we evaluated SSGP by the following procedure: 1) we trained the model using the dataset from pendulum or Dufﬁng oscillator with the friction coefﬁcients,
R = diag(0 , 0 . 05) ; 2) we predicted its conservative system by using the learned Hamiltonian, where we set the friction coefﬁcients, R = O .
The evaluation metric is the same as that in Task 1 .
Setup of the SSGP.
We trained the model using the Adam optimizer [22] with learning rate of 10 −for 10 4 epochs, implemented in PyTorch [31].
We performed numerical integration by the adaptive
Dormand–Prince method [3, 4] with the relative and absolute tolerances of 10 − 8 .
We set the numbers of Monte Carlo samples to K = 1 and L = 100 .
The number M of spectral points was chosen from { 100 , 250 , 500 } based on the validation error.
We used a block-diagonal approximation of C , namely, q ( w ) is factorized as two factors separating the weights for the sines and cosines, where the computational complexity is the same as that of the original.
We ﬁxed the friction coefﬁcients R =
O when training the conservative systems in Task 1 .
Baselines.
We compared the SSGP with the existing models (see Table 1): Hamiltonian neural network (HNN) [15], dissipative HNN (D-HNN)
[40], neural ordinary differential equation (NODE) [4], symplectic ODE-Net (SymODEN) [49], dissipative SymODEN (D-SymODEN) [48] and symplectic Gaussian process regression (SympGPR)
Since HNN, D-HNN and SympGPR require derivative observations for training, we used the ﬁnite difference instead.
Another baseline is the GP model using standard random Fourier features (RFF), which corresponds to the case that the vector ﬁeld f ( x ) in SSGP is modeled as a multi-output GP approximated by the standard RFF (not considering Hamiltonian mechanics).
In the experiments, we call it RFF .
The details of each baseline are described in Appendix F.
Results.
We present the experimental results that are picked up from the case where the sampling frequency was 5 Hz.
Appendix G shows all the results, including those for the frequencies of 3 and 10 Hz.
Figure 3 shows MSE and standard errors for SSGP and the baselines in Task 1 and Task 2 .
As expected, in many cases, MSE decreased as the number of trajectories increased.
The performance of HNN, D-HNN and SympGPR was limited because their training was based on ﬁnite differences.
NODE and RFF had large errors except for the case of Figure 3 (b) because they cannot use the prior knowledge of Hamiltonian mechanics as the inductive bias.
SymODEN and D-SymODEN were more competitive with SSGP; nevertheless, SSGP matched or bettered their predictive performance.
[Table] Moreover, SSGP improved the performance rather than all baselines, especially when the number of trajectories was small.
These results show that SSGP is advantageous in sparse settings.
In Figure(b) , NODE and RFF yielded relatively low errors; however, since they cannot distinguish between the conservative and dissipative terms, their performance was worse in Task 2 , as shown in Figure(c) .
In contrast, SSGP can estimate the conservative and dissipative terms separately by incorporating knowledge of Hamiltonian mechanics, which yields better performance in both Task 1 and Taskthan all baselines.
Table 2 shows MSE of the friction coefﬁcients estimated by SSGP, D-SymODEN and D-HNN when inferring the dissipative systems.
Here, MSE was averaged for all experimental settings.
This result shows that SSGP can accurately estimate the friction coefﬁcients.
In the following, we deeply explore the results when the number of observed trajectories was 20.
Figure 4 shows the visualization of the trajectories predicted by SSGP.
One observes that SSGP appropriately captured the dynamics of the dissipative Hamiltonian systems (the third column of Figure 4) and better discerned the conservative and dissipative terms (the fourth and ﬁfth columns of Figure 4).
Figure (5) shows the cumulative errors for the trajectories and energies for the pendulum data.
Let I test denote the number of time steps in the test set, and j t denote a time step at time t .
We calculated the cumulative error of trajectories at time t : (cid:80) j t j =1 ( 1 I test
[Table]
The cumulative error of energies was obtained similarly by calculating the squared error of the energy, H true (ˆ x ij ) , evaluated using the predicted trajectories and the true energy, H true ( x true ij ) .
Here, H true ( · ) is the true Hamiltonian of each system.
As shown in Figure 5, SSGP achieved lower errors than the baselines for both trajectories and energies at all time points.
The performance improvement of SSGP was more signiﬁcant in the period [10 , 15] , which is out of the simulation period (10 seconds) when generating the training data.
These results show that SSGP can accurately predict the dynamics with energy conservation and dissipation in both short-term and long-term simulations.
Conclusion
We have proposed the Symplectic Spectrum Gaussian Process (SSGP), which allows one to predict systems whose dynamics follow energy conservation and dissipation laws from noisy and sparse data.
Our result, the symplectic random Fourier feature, is a general tool and has the potential to use the design of kernel machines with prior knowledge in physics.
As described in Section 5, the SSGP can easily extend to learn Hamiltonians from high-dimensional data (e g., images).
Our future work is to evaluate the effectiveness of its extension
Acknowledgments and Disclosure of Funding
This work was supported by JST, ACT-X Grant Number JPMJAX210D, Japan.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] The main contributions of this work have been summarized in the last paragraph of Section 1.
(b) Did you describe the limitations of your work?
[Yes] The limitations have been described in Appendix H. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We have discussed the societal impacts in Appendix H. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the guidelines.
No ethical impact is supposed.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All the assumptions have been described in Sections 4 and 5.
(b) Did you include complete proofs of all theoretical results?
[Yes] Derivation procedures of our models have been included in Appendices A and C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code is available at https://github.com/yusuk-e/SSGP .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We have described how to generate datasets in the Data paragraph of Section 6.
The setups of the proposed model and the baselines were speciﬁed in the Setup of the SSGP paragraph of Section 6 and Appendix F, respectively.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We have provided the standard errors of the performance in Figure 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We used AMD EPYC [-
-][Table]
CPU (3.0GHz) in the experiments, as mentioned in the On computational complexity paragraph of Section 5. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We used the python library, torchdiffeq [3, 4], which we have cited in Section 6.
(b) Did you mention the license of the assets?
[Yes] We have mentioned the license information in the footnote on page 7.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code is available at https://github.com/yusuk-e/SSGP .
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
The spectral representation of the GP prior (3) for Hamiltonian systems with additive dissipation can be derived by applying the differential operator L = ( R − S ) ∇ to the 2 M -dimensional GP approximation (6) of the Hamiltonian H ( x ) , as follows:
[Equation]
As shown in (10), the spectral representation of f ( x ) corresponds to the GP approximation equipped with covariance function ˜ K ( x , x (cid:48) )
arising from symplectic random Fourier features (S-RFFs) Ψ ( x ) .
The covariance function ˜ K ( x , x (cid:48) ) is given by
[Equation]
One observes that ˜ K ( x , x (cid:48) ) is an extention of the covariance function arising from standard RFFs [23, p. 1870] to encode the geometric structure of Hamiltonian systems with additive dissipation.
B Comparisons of gram matrices
To investigate the connection between the exact GP (3) for Hamiltonian systems with additive dissipation and its spectral approximation (8), we show the comparisons of the gram matrix of
K ( x , x (cid:48) )
(4) and that of ˜ K
( x , x (cid:48) ) (10) in Figure 6.
We observe that approximation quality improves as the number M of spectral points increases.
C Derivation of the ELBO
In this appendix
[Equation]
we derive the evidence lower bound (ELBO) (13).
Let X =
{ x ij } denote the set of the noiseless states.
We assume that the variational distribution is given by
[Equation]
As described in Section
[Equation]
p ( f | w ) is Dirac’s delta function; q ( x i 1 ) and q ( w ) are the variational distributions of the initial condition x i 1 and the weights w , respectively.
p ( x ij | f , x i 1 ) is Dirac’s delta function represented by (12).
The ELBO is derived from Jensen’s inequality as follows:
[Equation]
where p ( f | w ) and p ( x ij | f , x i 1 ) are cancelled in (32), and we deﬁne q ( f )
=
(cid:82) p ( f | w ) q ( w ) d w in (33).
For j ≥ 2 , the variational distribution q ( x ij ) in (34) is given by
[Equation]
[Equation]
It should be noted that, given f and x i 1 , x ij ( j ≥ 2) is deterministically generated by the ODE.
D Inference procedure of SSGP
The inference procedure of SSGP is shown in Algorithm 1.
Here, unif { 1 , J i − S } in Algorithm 1 is a discrete uniform distribution with support { 1 , . . .
, J i − S } .
Although we assume that, for simplicity, the sampling frequency G s is constant, our algorithm can be easily extended to irregularly sampled trajectories.
In addition, this algorithm requires a hyperparameter, integration time window τ , which is often used for effectively learning ODEs when the trajectory length
J i is large.
In the implementation, we randomly re-select a state taken to be an initial condition from the observed samples in each iteration and calculate the ELBO by generating trajectories up to τ seconds later, as in previous works (e.g., [7, 49]).
In the experiments, we set the integration time window τ = 1 .
E On computational complexity
As described in Section 5, our inference algorithm is computationally efﬁcient because the proposed model is based on symplectic random Fourier features (S-RFFs) and the bottleneck (i.e., sampling of weights) is outside of the ODE solver.
To clarify it, let us consider the case where we do not use the
Input: Trajectory data { ( t ij , y ij ) | i = 1 . .
, I ; j = 1 . . .
, J i } , sampling frequency G s , number of spectral points M numbers
Monte Carlo samples integration τ epochs E
Output: Parameters Λ , σ 20 , σ 2 , R , A , b , C 1: Initialize the parameters 2: e ←3: repeat 4: /* Sample mini-batches D */ 5:
S ← τ/G s 6:
D ← ∅ 7: for i = 1 , . . .
I do 8: s i 1 ∼ unif { 1 , J i − S } 9:
D ← D ∪ { ( t i,s i 1 , y i,s i 1 ) , . .
, ( t i,s i 1 + S , y i,s i 1 + S ) } 10: end for 11: 12: /* Generate trajectories */ 13: for k
= 1 , . .
, K do 14: for l = 1 , . . .
, L do 15: w ( k,l ) ← b + √ C (cid:15) ( k,l ) where (cid:15) ( k,l ) ∼ N ( 0 , I ) 16: end for 17:
Construct the vector ﬁeld f ( k ) ( x ) by (18).
18: for i = 1 , . .
, I do 19: x ( k ) i,s i
[Equation]
← y i,s i 1 +
[Equation]
(cid:15)
[Equation]
i , where (cid:15) ( k ) i ∼ N ( 0 , I ) 20: x ( k ) i,s i 1 +1 , . . .
, x ( k ) i,s i 1 + S ← ODESolve (cid:16) x ( k ) i,s i 1 , f ( k ) ( x ) , t i,s i 1 +1 , . . .
, t i,s i 1 + S (cid:17) 21: end for 22: end for 23: 24:
Update the parameters by maximizing the ELBO (13) evaluated using D .
25: e ← e +26: until e > E
RFF-based approximation.
Since the derivative observations are not available, we should approximate the GP posterior for vector ﬁelds; one possible way is to introduce latent variables (similar to inducing variables) for the derivative observations, as in [A1].
Unfortunately, this technique does not reduce the cost of sampling paths from the GP posterior which scales cubically with the number of test points [45, 46].
Note that sampling paths is different from sampling function values independently at each test point.
Denote the number of test points by N t , which is proportional to the inverse of step size ∆ in the ODE solver, as follows: N t ∝ 1 / ∆ , where ∆ should be set to a sufﬁciently small value (often lower than 10 − 3 ).
Moreover, the GP posterior should be updated at each numerical integration step by conditioning all data points (i.e., test points) generated by the ODE solver.
The cost of this approach results in O ( N 4t ) .
On the other hand, the cost of our inference algorithm is O ( M 3 ) , where
M is the number of spectral points (often lower than 10 3 ).
Accordingly, our proposed model based on RFF is also beneﬁcial in terms of computational complexity.
F Baselines
In this appendix, we describe baseline models for the experiments in Section 6.
In all experiments, we trained the model using the Adam optimizer with a learning rate of 10 − 3 and a weight decay of 10 − 4 for 10 4 epochs.
We also used the ODE solver, the Dormand–Prince method with adaptive time-stepping, with the relative and absolute tolerances of 10 − 8 .
Neural network models.
HNN can utilize the prior knowledge of energy conservation as an inductive bias for training neural networks; D-HNN is its extension to handle not only energy conservation but also energy dissipation.
These models require derivative observations, not trajectories, for training; thus, we used ﬁnite differences instead in the experiments.
NODE can infer unknown dynamics from trajectory data by utilizing ODE solvers, which does not consider prior knowledge in physics.
SymODEN is the extension of HNN to allow the model learning from trajectory data by using ODE solvers, as in NODE.
D-SymODEN can also apply to the dissipative systems.
For all the models, we used a fully connected neural network with 3 layers, 200 hidden units, and tanh activations.
Gaussian process models.
SympGPR can estimate conservative vector ﬁelds from derivative observations by considering Hamiltonian mechanics; we used ﬁnite differences for training.
RFF is a baseline that uses standard random Fourier features for modeling vector ﬁelds (not considering
Hamiltonian mechanics); its training is based on ODE solvers.
This baseline is equivalent to [16] except for using the decoupled sampling.
SSGP is our proposal, which can handle conservative and dissipative systems by considering Hamiltonian mechanics and train the model with uncertainties from trajectories using ODE solvers.
For all the GP models, we used the ARD Gaussian kernel.
G Experimental results
This appendix contains additional results for our experiments in Section 6.
Figures 7(a) and 7(b) show MSEs and standard errors for SSGP and the baselines when the sampling frequency was 3 or 10 Hz.
We obtained similar results to those for 5 Hz shown in Figure 3.
SSGP had a more signiﬁcant performance improvement when the number of trajectories was smaller and the sampling frequency was lower (see Figure 7(a)).
On the other hand, the performance of SSGP was closer to those of the baselines when the trajectory data with relatively high time resolution was available (see Figure 7(b)).
These results are convincing to clarify that SSGP is more advantageous in the sparse setting.
We provide the visualization comparisons of the predicted trajectories when the sampling frequency was 5 Hz.
Figures 8 and 9 show the visualization results for predicting the conservative and dissipative systems, respectively, in Task 1 .
As shown in Figure 8, NODE and RFF do not capture the energy conservation law.
As one can see from the results of Dufﬁng oscillator (see Figures 8(b) and 9(b)),
HNN, D-HNN, SymODEN, and D-SymODEN failed to estimate the vector ﬁelds precisely because they suffered from overﬁtting to noisy observations; thus, they output false trajectories.
Compared with the baselines, SSGP estimated the vector ﬁelds without overﬁtting and accurately predicted the trajectories with energy conservation and dissipation.
Figure 10 shows the visualizations of the predicted conservative and dissipative terms in Task 2 .
One observes that SSGP accurately decomposed the dynamics into conservative and dissipative terms rather than the baseline models.
H Discussion
Limitations.
Finite-dimensional approximation of Gaussian processes (GPs) via a set of random
Fourier features (RFFs) is helpful for efﬁcient sampling from GP posterior.
In the proposed approach, we introduce the RFFs that encode symplectic geometric structures of Hamiltonian systems to construct the efﬁcient variational inference algorithm for learning dynamics with energy conservation or dissipation.
However, the RFF-based approximation is well-known to yield undesirable extrapolatory predictions as the number of samples increases, which is called variance starvation [A2, A3].
This might prevent the proposed model from learning Hamiltonian dynamics effectively.
It would be promising to employ the decoupled sampling technique [45], as in [11, 16], which might be useful to infer unknown dynamics while avoiding the variance starvation.
Although our experimental results are encouraging, they are the ﬁrst step to investigating the empirical properties of the proposed model.
We will apply it to various types of systems.
In addition, the proposed model is easily extendable to learning Hamiltonians from high-dimensional data (such as images), as described in Section 5.
We will empirically evaluate the effectiveness of its extension.
Societal impact.
This paper presents a framework for data-driven modeling that incorporates prior knowledge of Hamiltonian mechanics to improve predictive performance in noisy and sparse settings.
Generally, a data-driven approach can have a signiﬁcant impact on a wide range of applications, including climate and robotics.
However, since we have not evaluated our proposed framework for such real-world applications, we need to carefully assess the prediction and uncertainty estimation performance when applying it to them.
[Figure]
[Figure]
[Figure]
[Figure]