Constants of motion network
Anonymous Author(s)
AffiliationAddress email
Abstract
The beauty of physics is that there is usually a conserved quantity in an alwayschanging system, known as the constant of motion.
Finding the constant of motionis important in understanding the dynamics of the system but typically requiresmathematical proficiency and manual analytical work.
In this paper, we presenta neural network that can simultaneously learn the dynamics of the system andthe constants of motion from data.
By exploiting the discovered constants ofmotion, it can produce better predictions on dynamics and can work on a widerrange of systems than Hamiltonian-based neural networks.
In addition, the trainingprogresses of our method can be used as an indication of the number of constantsof motion in a system which could be useful in studying a novel physical system
Introduction
Noether’s theorem [1] states that if a system has a symmetry, then there’s a constant of motioncorresponding to it.
As it turns out, constants of motion play a significant role in understanding theworld around us and are ubiquitous in almost every aspect of physics.
Among of the most prominentexamples are the conservation of energy, the conservation of momentum, as well as the conservationof angular momentum.
Historically, constants of motion were discovered by doing analytical works from observational dataor from mathematical descriptions of the systems’ equations of motion.
For example, the law ofconservation of energy was proposed by Châtelet based on Newton’s work on classical mechanics [2].
With the recent emergence of employing neural networks for scientific discovery and learning systems’behaviour from observational data [3, 4, 5, 6], naturally it raises a question, " can we find constants ofmotion of dynamical systems from their data and exploit them to make a better prediction? "
A large body of literature has been moving into this direction by learning the Hamiltonian [6, 7] orits variations [8, 9] of a system.
However, as most of the previous works focus on Hamiltonian andits variations, they work well in conserving the Hamiltonian or energy.
However, when the systemshave other constants of motion, the Hamiltonian-based works fail to discover and exploit thosequantities.
Here we present the "Constant Of Motion nETwork" (COMET) that can discover constantsof motion of a system and exploit them to make a better prediction.
In contrast to Hamiltonian-basednetworks [6, 7, 8, 9], COMET is not constrained to Hamiltonian systems and its coordinate choice,making it generally applicable to a wider range of systems as shown in Table 1.
In addition, we alsofound that the training progress of COMET can be used as an indication of how many independentconstants of motion there are in the system (see section 6) which could be a valuable hint in studyinga novel physical system.
Our implementation and experiments can be found in the public domain 1 .
NODE [10]
HNN [6]
NSF [7]
LNN [8]
COMET
Conserves energy ✓ ✓ ✓ ✓
Works on general coordinates ✓ ✓ ✓ ✓
Works on dissipative systems ✓ ✓
Conserves other quantities
Related works
The simplest way to learn the dynamics of systems with neural networks is by using neural ordinarydifferential equation (NODE)
NODE takes the full states of the system and produces thedynamics of the systems, i.e. the time derivative of the states.
The simulation can then be run bysolving the ODE from the output of NODE.
As there is no inductive bias in the NODE, they typicallystruggle to conserve quantities that are important in some systems’ dynamics, such as energy.
Hamiltonian neural network (HNN) [6] is an attempt to solve this conservation problem by learningthe Hamiltonian and calculate the state dynamics from the Hamiltonian.
It has been shown that with
HNN, one can conserve the energy and produce better motion prediction in a long time horizon.
Dueto its simplicity and the elegance of the idea, HNN has been applied on a wide range of tasks andneural network architectures [11, 12, 13], and even on dissipative systems by adding a dissipationterm [14, 15].
HNN can also be combined with symplectic integrator [16, 17] to produce a betterresult from the trajectory observations.
Despite its ability to conserve the energy, HNN is limited by the requirement of using canonicalcoordinates instead of arbitrary coordinates.
Works on Lagrangian neural networks [18, 8] solvethis limitation by learning the Lagrangian.
Other attempts use neural symplectic form to learn thecoordinate-free representation of Hamiltonian [7] or Poisson neural network to learn the Poissonsystem [9].
However, those works are still limited to Hamiltonian or Poisson systems.
COMET: constants of motion network
We start by denoting a set of states in a system as s ∈ R n s where n s is the number of states.
Statesare the internal parameters of a system that completely determines its dynamics without externalinfluence.
For example, in a classical particle motion, the particle’s position and velocity constitutethe states of the system.
Without external influence, the change of the states typically depends on thestates itself, i.e. d s /dt
= ˙s
( s ) .
A constant of motion is a quantity that is conserved over the time in the system, like energy.
In somesystems, such as integrable systems [19], there are other quantities other than energy that is conserved,for example, momentum or angular momentum.
These constants of motion can typically be describedas a function of the states of the system, so we denote it as c ( s ) ∈ R n c with n c is the number ofconstants of motion.
As their quantity is constant throughout the motion, their time derivative mustbe 0, or d c /dt = 0 .
By taking the dependency of c on s , the condition on c can be written as
[Equation]
where ∂ c /∂
s is an n c × n s Jacobian matrix where each row of the matrix is the gradient of eachconstant of motion with respect to the states s .
The equation above means that the state dynamics ˙smust be perpendicular to the gradient of each constant of motion.
To design a deep learning architecture that can simultaneously learn the constants of motion andlearn the dynamics that conserve the constant of motion, we define two functions that depends on thestates that can be constructed with neural networks, ˙s 0 ( s ) and c ( s ) .
The function ˙s 0 : R n s → R n s is the initial guess of the rate of change of the states.
The function c : R n s → R n c is the function thatcomputes the constants of motion of the system.
To ensure the constants of motion are conserved asin equation 1, we compute the state dynamics by orthogonalizing the initial guess ˙s 0 with respect tothe gradient of every constant of motion,
[Equation]
where c i is the i -th element of the constants of motion c and ortho( a , V ) is an operation to orthogonalize the vector a to every vector in the set V .
3.1 Orthogonalization process
One way to produce an orthogonal vector against a set of vectors is by using QR decomposition, i.e.
[Equation]
where Q ( · ,n c ) is the last column of the matrix Q , and R ( n c ,n c ) is the element at the last row andlast column of the matrix R .
The first row of the equations above shows a construction of a tallmatrix A ∈ R n s × ( n c +1) where the first n c columns are the gradient of the constants of motion andthe last column is the initial guess of the states rate of change, ˙s 0 .
QR decomposition is usuallyimplemented using Householder transformation [20] which produces much smaller numerical errorthan the alternative Gram-Schmidt process [21] in practice.
The QR procedure above imposes a constraint that the number of constants of motion must beless than the number of states, i.e. n c < n s .
This is in agreement with the maximum numberof independent constants of motion in an integrable system is n c = n s − 1 [19].
By using QRdecomposition, COMET will try to find n c independent constants of motion.
Reasoning behind thisis given in Appendix A.
3.2 Training loss function
We need to train the two trainable functions in COMET, ˙s 0 ( s ) and c ( s ) , so that the state dynamics˙s from equation 3 match the dynamics from the observation or training data, ˆ˙s .
In order to train
COMET, the loss function in this case is constructed as
[Equation]
where w · are the tunable regularization weights.
The first term of the loss function is the standard L 2error where the prediction must match the training data.
The second term of the equation above isincluded to accelerate the training process by making the initial guess ˙s 0 to be as close as possible tothe actual value of the states’ rate of change.
The third term is an additional regularization to help thediscovery of the constants of motion.
Learning constants of motion from data
To demonstrate the capability of COMET to simultaneously learn both the dynamics and the constantsof motion, we tested it in a variety of cases.
For all the cases in this section, the training data weregenerated by simulating the dynamics of the system from t = 0 to t = 10 .
From the simulations, wecollected the states s as well as the states rate of change, ˆ ˙s , which were calculated analytically andwere added a Gaussian noise with standard deviation σ = 0 .
There are 6 simple experiments performed to demonstrate the capability of COMET: (1) mass-spring,(2) 2D pendulum, (3) damped pendulum, (4) two body, (5) nonlinear spring, and (6) Lotka-Volterra.
The cases were selected to represent a wide variety of cases.
It includes cases with Hamiltonian in
[Table]
[Table]
canonical coordinates (case 1, 4, 5), Hamiltonian with non-canonical coordinates (case 2, 6), a casewith redundant states (case 2), dissipative system (case 3), and a case with a moderate number ofstates (case 4).
The details of each case, including the number of constants of motion that we set for
COMET training, as well as the training setups are described in appendix B.
For each case, we compared the performance of COMET with other methods: (1) simple neural ODE(NODE) [10], (2) Hamiltonian neural network (HNN) [6] with the coordinates given in each casebelow, (3) neural symplectic form (NSF) [7], and (4) Lagrangian neural network (LNN) [8].
Theneural network architecture for each method is detailed in appendix C.
4.1 Results
For each case, we tested each method by running another 100 simulations from t = 0 to t = 100 (10 timeslonger than the training) using 1000 sampled pointswith the initial condition randomly initialized as aboveusing different seed.
The root mean squared errors ofthe state predictions are shown in table 2.
From table 2, we can see that COMET performs wellacross the test cases.
In the mass-spring case, all methods perform well.
However, when it goes to the pendulum cases and Lotka-Volterra, HNN fails to predictthe dynamics due to the chosen coordinates not beingthe canonical coordinates.
Although NSF can performreasonably well in 2D pendulum, it fails on the dampedpendulum case because it is not a Hamiltonian system.
COMET takes the advantage of having constantsof motion in the cases above that can be exploited toguide the true trajectory, therefore, it can achieve betterpredictions of the dynamics regardless of the chosencoordinates and whether it conserves energy or not.
Figure 1 shows the discovered constant of motion in mass-spring and Lotka-Volterra cases.
As it canbe seen from the figure, COMET can successfully discovered the constants of motion from the data.
Figure 2 shows the evolution of the known constants of motion for every method in the mass-spring,2D pendulum, and the two body cases.
The periodic variation from the true constants of motion aredue to the added noise in the training data.
In mass-spring case, figure 2(a) shows that the HNN, NSF,and COMET conserves the energy while the NODE gets the energy decreasing over time. [Figure]
A different story can be found in figures 2(b) where they show 3 constants of motion of pendulumin 2D coordinate.
In this case, NODE and HNN fail to conserve the constants of motion, while
COMET can conserve those constants of motion during long period of time.
The failure of HNN canbe attributed to the state coordinates not being the canonical coordinates.
This shows that COMETcan discover the constants of motion with much less constraints in the state coordinates than HNN.
For the two body case in Figure 2(c), we can see that NODE and NSF diverges quite quickly.
Thefailure of NSF in this case might be due to the added noise in the training data.
Among the testedmethods, only HNN and COMET conserves the energy.
However, as HNN is only designed for
Hamiltonian or energy conservation, it fails to conserve other quantities, such as the momentum andangular momentum.
COMET, on the other hand, can successfully conserve those quantities.
Figure 3 shows the trajectory of the two body system simulated using various methods tested herefrom t = 0 to t = 20 .
From the figure, we can see that only our method (COMET) that can produceclosed trajectory.
HNN produces almost closed trajectory, but it slightly deviates from the closedtrajectory because it conserves only the energy, but does not necessarily conserve the other quantities.
By exploiting as many constants of motion as possible, COMET can reproduce the motion with thesmall error compared to the other methods.
Systems with external influences
One advantage of COMET is that it can easily work with systems with external influences, suchas external forces.
If the system has conserved quantities when the external influences are keptconstant, then COMET with a simple modification can be used to learn the constants of motion andexploit them to get more accurate dynamics.
The modification is just to make the initial guess of thedynamics and the constants of motion to depend on the external influences as well as the states, {+i.e. ˙s 
+} ( s , x ) and c ( s , x ) , where x ∈ R n x is the external influence.
The dynamics can still be calculatedfollowing the equation 2.
We conducted an experiment using the 2D pendulum from the section 4, but with additional externalforce in the x -direction, F x .
The training data was generated by having the external force withprofile F x ( t ) =
a 0 cos( a 1 t + a 2 ) with uniformly-distributed random values of a 0 ∼ U ( − 0 .
5 , 0 . 5) ,a 1 ∼ U (0 , 5) , and a 2 ∼ U (0 , 2 π ) .
The experiment was done similarly like in section 4, by addingthe force as the input to the neural network for NODE, HNN, NSF, and LNN as well.
Figure 4 shows the constants of motion on the test system with constant external force.
As seen onthe figure, the values of the true constants of motion produced by COMET is oscillating slightlyaround a constant offset, due to the added noise in the training.
In contrast, NODE produces the shiftof the energy values, NSF produces large oscillation even for the energy, and LNN quickly diverges.
Although HNN can produce similar energy deviation with COMET, it has larger deviation on otherconserved quantities.
This shows that even with external forces, COMET can still find and exploitthe constants of motion for its dynamic predictions.
Finding the number of constants of motion
For most systems, the number of independent constants of motion is usually not known beforehandand not so obvious.
Knowing the number of constants of motion can be useful in understanding themanifold dimension of the motion, however, this problem is not an easy problem to solve.
Duringthe research of this work, we observe that COMET’s training progresses might provide a valuableindication to the number of constants of motion.
The parameter to look at is the first term in the loss function in equation 4, i.e
[Equation]
If we set the number of constants of motion greater than the true number, then that term could notget lower than a certain value.
It is because ˙s is constrained to be perpendicular to the constants ofmotion and if there are excessive constants of motion, then it may not be able to match the value fromexperiments to a certain accuracy.
We ran a simple experiment to find the number of constants of motion for known systems.
Specifically,
COMET was trained in the damped pendulum, two body, and 2D nonlinear spring cases from section4 without added noise and ran for 3000 epochs.
Those cases are known to have 2, 7, and 2 constantsof motion out of 4, 8, and 4 number of states, respectively.
The numbers of constants of motion werescanned from 0 to the maximum value, n s − 1 .
Figure 5 shows the value of L 1 for the various caseswith varying number of constants of motion.
From the figure 5 (top), we can see that once the number of constants of motion is set above acertain number, the value of L 1 suddenly increases.
This gives an indication of the actual number ofconstants of motion.
If the system has the maximum number of constants of motion, then the valuesof L 1 will always be similar to the values with n c = 0 .
Besides the final value of L 1 , the evolution value of L 1 for various number of constants of motion can be an indication on the true number of the constants motion as we can see in figure bottom
From figure see that the of motion for damped pendulum
6.1 Failure mode
[Figure]
This technique in determining the number of constants of motiondepends on the ability of the neural network to find the constants ofmotion.
Therefore, if the neural network is not expressive enough,it could fail to find the constants of motion and indicate a lowernumber of constants of motion than it should be.
Figure 6 illustrates this case where we scanned the number ofconstants of motion from 0 to 3 in the 2D nonlinear spring casewhere the neural network only has 50 hidden elements per layerinstead of 250.
It gives an indication that the number of constantsof motion to be 1 instead of the true number of 2.
More complex cases
Simulating a system with infinite number of states — The previous examples only involve systemswith finite or countable number of states.
To demonstrate the general applicability of COMET, we ranan experiment on simulation of systems with infinite (but discretized) number of states.
Specifically,we trained the COMET to learn the dynamics of shallow wave following Korteweg-De Vries (KdV)equation [22, 23] of u ( x, t ) , ∂u∂t = − u ∂u∂x − δ 2 ∂ 3 u ∂x 3 .
The states in this case are the values of u alongthe x -axis which constitutes infinite number of states.
In our experiment, we simulate the behaviour of u from x = 0 to x =
L = 5 with periodicboundary condition, sampled in 100 points with uniform spacing.
We also set δ = 0 .
00022 fornumerical stability.
The training dataset was generated by running 100 simulations with randominitial condition from t = 0 to t = 10 with 100 steps.
The initial condition in the training dataset is u ( x, 0)
= a 0 + a 1 cos (2 πx/L + a 2 ) where a 0 , a 1 , and a 2 are randomly chosen within the range of(1 . 5 , 2 . 5) , (0 , 1) , and (0 , 2 π ) , respectively.
The neural network was constructed with 1D convolutional layers with kernel size 5 and circularpadding, followed by logsigmoid activation function.
The pattern above was repeated 4 times butwithout the activation function for the last one, using 250 channels in the hidden layers.
The trainingwas done as described in section 4 which takes about 5-7 hours on an NVIDIA T4 GPU.
The numberof channels in the input is 1 (only for u ), and for the output it is 1 + n c where n c is the number ofconstants of motion that we set.
The first channel of the output is to represent the initial guess of thedynamics, ˙ u 0 ( x ) .
The last n c channels are for what we call as the constants of motion density, p i ( x ) ,for i = 1 , ..., n c .
From p i ( x ) , the constants of motion can be calculated as c i =
(cid:82) L 0 p
i ( x ) dx .
Usingthe outputs from the network, the dynamics can be calculated following the equation 2.
We compared the performance of NODE and COMET in solving the KdV equation for t =
to20 .
It is not obvious how to apply HNN, NSF, and LNN as the KdV equation has only u as itsstates and do not include velocity nor momentum.
Figure 7 shows the states u ( x, t ) at t = 20 of thesimulations using the true dynamics, NODE, and COMET.
From the figure, we can see that at t = 20 ,the simulation done by NODE has diverged while COMET simulations are still intact.
This showsthat COMET can take advantage of the constants of motion to make its prediction more accurate.
Learning from pixels — Another experiment that we run to show COMET’s capability is to learnthe dynamics from pixels and to see if it can exploit the constants of motion in the latent space of anauto-encoder. In this case, we simulate the dynamics of two-body with gravitational interactions (likein section 4) and present the data as 30 × 30 pixels images that show the positions of the 2 objects. Our model consists of an encoder, a COMET, and a decoder. The inputs to our model are 2 consecutiveframes to capture the information about positions and velocities which we denote as x .
The encoderconverts the pixel inputs to latent variables, s =
f
e ( x ) .
The dynamics of the latent variables are thenlearned by COMET which will then be converted back to pixel images,
˜x = f d ( s ) , by the decoder.
The loss function in this case is simply a sum of the COMET loss from equation 8 and the autoencoder loss, i.e. L ae =
∥ x − ˜x ∥ 2 .
The observed state derivatives ˆ˙ s to compute the COMET loss inequation 8 is approximated by finite difference of the encoded pixel data from 2 consecutive timesteps, i.e. ˆ˙ s ≈ [ f e ( x ( t + ∆ t ))
− f e ( x ( t ))] / ∆ t .
In contrast to HNN and LNN, there is no requirementin COMET to make half of the latent states to be the time derivative of the other half.
In this experiment, we compared the performance of COMET, NODE, HNN, and NSF to learn thedynamics of the latent variables of the auto-encoder.
The auto-encoder architectures are kept to bethe same for all methods.
The number of latent states are picked arbitrarily to be 10.
It is more thanthe true number of states which is 8.
For COMET, the number of constants of motion for COMETwas found by following the procedure in section 6 which is 9.
Figure 8 shows a sample of the decoded images of the dynamics predicted by COMET, NODE, HNN,and NSF, compared to the ground truth.
From the figure, we can see that the dynamics predictedby NODE diverges as soon as t = 20 and at some point the decoder produces non-sensible images.
Although NSF and HNN can produce good images until the end of simulation at t = 180 , the dynamics predicted by NSF and HNN diverges from the ground truth.
In contrast, COMET can stillmatch the dynamics of the ground truth simulation until the end of simulation.
This shows COMET’scapability in accurately learn the dynamics of latent states of an auto-encoder.
Limitations – COMET works better if there is at least one constant of motion in the system.
If thereis no constant of motion, then COMET works similarly like the neural ODE [10].
Although wepresented a way to find out the number of constants of motion in section 6, it still requires multipletraining processes and manual insight.
In the case of successful training, COMET sometimes produces dynamics that are stiffer than thetrue dynamics, although LNN and NSF more often produce stiffer dynamics.
In a rare case, thedynamics from COMET are so stiff that the integration by scipy’s solve_ivp cannot be completedin a reasonable time.
This only happens in the KdV case and did not happen in the other cases wetested for this paper.
We believe that the limitations above should be addressed to move forward.
impact – The impact of deep learning on physical sciences is expected to be similar to theimpact of scientific computational method.
Although it enables new fields of study, it adds more pointof failure.
For example, if a new or unexpected result is discovered using deep learning methods, itcould be a true discovery or false discovery due to the failure/imperfection in training, unsuitableneural network architecture, bug in the code, among other things.
Therefore, deep learning methodssuch as COMET should be accompanied with other different and independent methods to confirmobtained results in scientific works.
Conclusions
We have shown that COMET can simultaneously learn the constants of motion and the dynamics of asystem from observational data.
Because the assumption made by COMET (i.e. have constants ofmotion) is less strict than Hamiltonian-based neural networks, it can be applied to a wider range ofsystems than the Hamiltonian-based neural networks, including dissipative systems and systems withexternal influences.
The training progresses of COMET can also give an indication on the number ofconstants of motion in a system.
With all the advantages we presented, we believe that COMET canbe a valuable tool for scientific machine learning in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](
b) Did you describe the limitations of your work
Yes ] See sections 6.1 and 8.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See section8.(d)
Have you read the ethics review guidelines and ensured that your paper conforms
to
them?
[Yes]
If you are including theoretical results a) Did you
state the full set of assumptions of all theoretical results?
[N/A]
b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See the footnote
at the end of section 1.
b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See section 4.
c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See table 2 and figure 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A More explanation on the training of COMET
The training loss of COMET is
[Equation]
With the terms above, it seems that the deep learning could cheat the training either by: (1)
makingall c i to be constants or (2) getting ∇ c
i terms to be linearly dependent to each other, thus discoveringfewer constants of motion than intended.
However, in practice, we did not find both effects in thetraining.
In fact, COMET training tends to find the constants of motion such that ∇ c
i are all linearlyindependent although there is no explicit term to encourage linear-dependency of ∇ c i in the lossfunction.
This is due to the involvement of QR decomposition in computing the term ˙s .
For a QR decompositionof a matrix A , i.e. Q , R = QR( A ) , the gradient of ¯A =
∂
L /∂
A is
[Equation]
where M = R¯R T − ¯Q T Q and copyltu( · ) is the operation to copy the lower triangular elements tothe upper triangular of the matrix.
If the matrix A contains almost linearly dependent columns, thematrix R will be ill-conditioned.
As the gradient above involves the inverse term of R , the almostlinearly dependent columns of A will produce very high gradient in ¯A .
This will make it harder toget the columns of matrix A (and therefore ∇ c i ) to be completely linearly dependent as it has to gothrough the region where the gradient is very high.
As the training of COMET tends to find independent constants of motion, setting the number ofconstants of motion higher than it should be would make the training fail.
This is what we exploit insection 6 to give us the indication of the true number of constants of motion.
The last term in the loss function of COMET does not need the information from the dataset.
Moreover, we would like the constraints to be fulfilled for the states outside the ones listed in thetraining dataset.
Therefore, when training our COMET, the last term of the loss function is calculatedusing the states from the training plus some noise, i.e.
[Equation]
where ˜s ∼ N ( 0 ; σ 2 I ) is the Gaussian noise with standard deviation σ = 0 .
B Experiment details
This section contains the experiment details for cases tested in section 4.
In section 4, there are 6simple experiments performed to demonstrate the capability of COMET: (1) mass-spring, (2) 2Dpendulum, (3) damped pendulum, (4) two body, (5) nonlinear spring, and (6) Lotka-Volterra.
Thecases were selected to represent a wide variety of cases.
For each case, 100 simulations with random initial conditions were generated with 100 sampled timepoints each from t = 0 to t = 10 .
The dataset is split to 70% for training, 10% for validation, and20% for test. The training was performed with batch size of 32 using a neural network with 3 hiddenlayers of 250 elements each using logsigmoid as the activation function to get infinitely differentiablefunction. There are n s inputs to the neural network with n s + n c outputs. The first n s elements areassigned to the initial guess, ˙s 0 , while the next n c elements are assigned to the constants of motion, c .
The training procedures were performed using Adam optimizer [24] with the learning rate 3 × 10 − 4until 1,000 epochs, which takes about an hour with an NVIDIA T4 GPU.
The regularization weightsare w 1 = w 2 = 1 . 0 .
The description for each simulation case can be found below.
Case 1: Frictionless mass and spring.
This is the simplest case to test COMET’s capability wherean object of mass m = 1 is connected to a stationary point by a spring with constant k = 1 .
Thestates of this system is s = ( x, ˙ x ) T where x is the displacement of the object from its equilibriumposition and ˙ x is the velocity of the object.
The training data was generated by randomly initializingthe position and velocity with a uniform random distribution between ( − 0 .
In this case, thereis only one independent constant of motion which is energy, E = ( x 2 + ˙ x 2 ) / 2 .
Case 2: 2D Pendulum.
The second case is a 2D pendulum of length l = 1 and mass m = 1 with aninfluence of gravity g = 1 .
The observed states in this case are the pendulum’s x
and y coordinatefrom the pivot as well as its velocity in x and y coordinate, i.e. s =
( x, y, ˙ x, ˙ y ) T , making it redundant.
The training data were generated by randomly initializing the angle and angular velocity with uniformdistribution in the range ( − 1 . 0 , 1 . 0) .
There are three independent constants of motion in this case,(1) energy: E = ( ˙ x 2 + ˙ y 2 ) / 2 + y , (2) length: x 2 + y 2 = 1 , and (3) angle: x ˙ x + y ˙ y
= 0 .
Case 3: 2D damped pendulum.
This case is similar to the previous case, except that we introducedthe damping force proportional to the velocity with damping coefficient α = 1 , making it an underdamped system.
The training data were generated in a similar way as the previous case.
As the energyis not conserved, only the second and third constants of motion from the previous case are valid.
Case 4: Two body interactions.
We considered a case where two bodies of the same masses m = 1are interacting with gravitational force with constant G = 1 and rotating around their centre of mass.
The training data were generated by initializing it with a distance randomly chosen between (1 . 0 , 3 .
0)with perpendicular velocity between 0 .
7 v 0 to 1 .
0 v 0 where v 0 is the velocity to make the orbits circular.
As the motion is planar, we only considered their motion on a 2D plane.
Therefore, there are 8 statevariables, s = ( x 1 , y 1 , x 2 , y 2 , ˙ x 1 , ˙ y 1 , ˙ x 2 , ˙ y 2 ) T .
As the two-body motion is well-known to be fullyintegrable, the number of constants of motion is n s − 1 , which equals to 7.
Among them are: totalenergy, total angular momentum, and total x and y momentum.
Case 5: 2D nonlinear spring.
We consider a case of a motion of an object of mass m = 1 in 2Dwhere it is connected to the origin with a nonlinear spring with force F
= −| r | 2 r where r is theposition of the object in 2D coordinate.
The states in this case is s = ( x, y, ˙ x, ˙ y ) T .
The datasetwas generated by starting the simulation with randomly selected states between ( − 1 . 0 , 1 . 0) for allpositions and velocities.
The constants of motion of this systems are the energy and the angularmomentum, which makes n c = 2 .
Case 6: Lotka-Volterra equation is an ordinary differential equation modelling the population ofpredator and prey.
It is known to have a symplectic structure [25], therefore it has a constant ofmotion.
We consider the equations ˙ x = x − xy and ˙ y
=
− y
+ xy where x and y represent the preyand predator populations respectively.
There are only 2 states here, s = ( x, y ) T with n c = 1 .
Theinitial values of x and y are sampled randomly from uniform distribution within (0 . 5 , 2 . 0) .
As thereis no time derivative variable in the states, it does not make sense to apply LNN for this case.
C Neural network architectures
In this section, we will explain in more detail about the architecture of the tested methods in section 4as
well as in section 5 in incorporating external forces.
We use the same notations as in section 5,where the state is s ∈ R n s , its time derivative is ˙s ∈ R n s , the external force is x ∈ R n x , n s is thenumber of states, and n x is the number of external forces.
The neural networks for all methods tested here (including COMET) consists of 3 hidden layers with250 elements each with logsigmoid activation function.
The activation function is only applied tothe hidden layers, but not applied to the output layer.
However, the number of inputs and outputsmight be different, depending on the needs of each method.
The neural network architecture above ischosen to produce good training results for all methods.
C.1 Neural ODE
With neural ODE [10], the states’ time derivative is directly represented by a neural network thattakes the states, s , as its input, i.e. [Equation]
The neural network takes n s inputs, produces n s outputs, with hidden layers and the activationfunction follow the description above.
If external forces, x , exist, the neural network is modified to take x as the input as well, i.e.
[Equation]
= g NODE ( s , x ) . In this case, the neural network architecture stays the same, but with concatenated s and x as its input,therefore taking n s + n x inputs.
C.2 Hamiltonian neural network (HNN)
HNN [6] was implemented by having a neural network takes n s input for s and 1 output for thepredicted Hamiltonian, H ,
[Equation]
For cases with canonical coordinate, the first half of the states represents the position while the lasthalf of the states represents the canonical momentum.
The time derivative of states is calculated by
[Equation]
The hidden layers and the activation function of the neural network follow the implementation inother cases.
If there are external forces, then the neural network is modified to take the concatenated s and x as itsinput, i.e.
H = g HNN ( s , x ) .
The time derivative of states is still calculated according to equation 12.
C.3 Neural Symplectic Form (NSF)
NSF [6] was implemented by having 2 neural networks where both take the states s as the input.
Oneneural network produces one output, H , that represents the Hamiltonian, and the other one producesn s outputs, Y , for its skew-symmetric matrix.
Mathematically, it can be written as
[Equation]
[Equation]
Both neural networks have 3 hidden layers with 250 elements each with logsigmoid as their activationfunction.
This follows the neural network implementation in other cases.
Combining both functionsabove into a single neural network produces worse training results than having two neural networks.
The time derivative of the states is obtained by calculating
[Equation]
Similar to the other methods with external force, the neural networks are modified to take theconcatenated s and x as their inputs, i.e. [Equation]
[Equation]
The time derivative of the states is still calculated according to equation 16 above. [Table]
C.4 Lagrangian neural network (LNN)
The neural network in LNN [8] takes n s inputs (for s ) and produces 1 output for the predicted
Lagrangian, L ,
L = f LNN ( s ) .
For cases with position and velocity as the states, the states are arranged so that the first half of thestates is the position and the last half is the velocity, i.e. s = (cid:0) q T , ˙q T (cid:1) T .
The time derivative of thestates is given by ˙s
= (cid:0) ˙q T , ¨q T (cid:1) T where the acceleration ¨q is calculated by [Table]
[Equation]
With external force, the neural network takes the concatenated s and x as its input,
[Equation]
(21) where the acceleration still following the equation 20 above.
D Additional experimental results
D.1 Partial number of constants of motion
Although the number of constants of motion can be found with the procedure in section 6, it canproduce a number lower than the true number of constants of motion.
Therefore, it is interestingto see the effect of setting lower number of constants of motion to COMET’s predictions accuracy.
Table 3 shows the error range of the COMET’s predictions with varying number of constants ofmotion.
From the table, it can be seen that setting the number of constants of motion at least 1 could givebetter worst case predictions than at least one of the other tested methods (NODE, HNN, LNN, and
NSF) in most cases.
Increasing the number of constants of motion also decreases the 95th percentilebound of the prediction error, which means that it can keep the stability of the trajectory better as thenumber of constants of motion increases.