D ECONSTRUCTING THE I NDUCTIVE B IASES OF H AMILTONIAN N EURAL N ETWORKS
A BSTRACT
Physics-inspired neural networks (NNs), such as Hamiltonian or Lagrangian NNs, dramatically outperform other learned dynamics models by leveraging strong inductive biases.
These models, however, are challenging to apply to many real world systems, such as those that don’t conserve energy or contain contacts, a common setting for robotics and reinforcement learning.
In this paper, we examine the inductive biases that make physics-inspired models successful in practice.
We show that, contrary to conventional wisdom, the improved generalization of
HNNs is the result of modeling acceleration directly and avoiding artiﬁcial complexity from the coordinate system, rather than symplectic structure or energy conservation.
We show that by relaxing the inductive biases of these models, we can match or exceed performance on energy conserving systems while dramatically improving performance on practical, non conservative systems.
We extend this approach to constructing transition models for common Mujoco environments, showing that our model can appropriately balance inductive biases with the ﬂexibility required for model-based control.
[Figure]
Reality
I NTRODUCTION
The inductive biases of convolutional neural networks, such as translation equivariance, locality, and parameter sharing, play a key role in their generalization properties.
Other biases have similarly guided the development of deep models for other domains like graphs and point clouds.
Yet since models often encode multiple biases simultaneously, it can be challenging to identify how each contributes to generalization in isolation.
By understanding which inductive biases are essential, and which are merely ancillary, we can simplify our models, and improve performance.
For example,
Wu et al.
(2019) demonstrate that Graph Convolutional Networks can be radically simpliﬁed to mere logistic regression by removing nonlinearities, leading to dramatic gains in computational efﬁciency, while retaining comparable accuracy.
Hamiltonian Neural Networks (HNNs) (Greydanus et al., 2019) have emerged as a leading approach for modeling dynamical systems.
These dynamics models encode physical priors and outperform alternative Neural ODE approaches (Chen et al., 2018).
In the spirit of Wu et al.
(2019), we seek to identify the critical components of HNNs.
Since Lagrangian models (LNNs) (Lutter et al., 2019a;
Cranmer et al., 2020) share the same structure and inductive biases as HNNs, we focus on HNNs where energy conservation and symplecticity are more explicit.
HNNs encode a number of inductive biases that help model physical systems:
ODE bias : HNNs model derivatives of the state rather than the states directly.
Second-order (SO) bias : HNNs model changes in position through changes in velocity.
Energy conservation bias : HNNs conserve their learned energy function.
Symplectic bias : HNNs dynamics are symplectic : phase space areas are conserved, and the vector ﬁeld has a Hamiltonian structure.
In this paper we theoretically and empirically examine the role of these biases.
In contrast to conventional wisdom, we ﬁnd that the generalization beneﬁt of HNNs is not explained by their symplectic or energy-conserving properties, but rather by their implicit second-order bias.
We highlight these ﬁndings in Figure 1.
Abstracting and extracting out this second-order bias, we show how to improve the performance of Neural ODEs, empowering applications when HNNs assumptions are violated as is often the case. Code for our experiments can be found at: https://github.com/ngruver/decon-hnn.
R ELATED W ORK
Physics-inspired models and energy conservation Since Greydanus et al.
(2019), many researchers have sought to extend the HNN approach to make it more general or applicable to systems that break energy conservation.
Jin et al.
(2020), Li et al.
(2020), Tong et al. (2021), and Xiong et al. (2020) propose methods for creating neural networks that preserve symplectic structure directly.
Zhong et al.
(2020), and later Desai et al.
(2021), Li et al. (2021), and Lee et al.
(2021) propose models with additional capacity for changes to the system energy.
In our analysis, we compare against approaches that add additional capacity for changes in energy, although our approach is fundamentally different — we attempt to remove unnecessary biases rather than add complexity.
Physics-inspired models for control A large and rapidly expanding body of work explores how to use physics-based biases in dynamics models with controls or contacts (Lutter et al., 2019b;
Zhong et al., 2019; Gupta et al., 2019; 2020; Chen et al., 2019; Hochlehnert et al., 2021; Chen et al., 2020; Zhong et al., 2021b) intended for application in model-based planning.
Especially relevant to our evaluations, Alvarez et al.
(2020) models MuJoCo
(Todorov et al., 2012) trajectories with a numerically integrated neural network but does not explore other physics-inspired inductive biases.
Analyzing physics-based inductive biases Karniadakis et al.
(2021) and Liu et al.
(2021) propose conceptual frameworks for deep learning with physics-based inductive biases, but present minimal empirical analysis of common design decisions.
Zhong et al.
(2021a) compare many approaches to physics-inspired deep learning, with results that parallel some ﬁndings here.
The contribution of our work, however, is not simply benchmarking but also an actionable theory of HNNs’ success. In spirit, our work is similar to Botev et al. (2021), which also critically examines the role of physicsbased priors in dynamics models, though their focus is on learning latent space dynamics, while we focus on temporal models in observation space.
B ACKGROUND
We consider dynamical systems described by ordinary differential equations (ODEs) which determine how the system evolves over time.
Even with high order derivatives, these systems can be arranged into the form dzdt =
F ( z, t ) for z ∈ R n .
If the dynamics F are time-independent, then the
ODE can be understood as a vector ﬁeld , specifying at each point where the next state will be.
Neural ODEs (NODEs) (Chen et al., 2018) parametrize a vector ﬁeld with a neural network and learn the dynamics directly from observed trajectories.
A NODE dynamics model ˆ F θ is rolled out from the initial condition z 0 with ODE integration, ˆ z t = ODESolve( z 0 , ˆ F θ , t ) , and ﬁt to trajectory data by minimizing an L2 loss L ( θ ) = (cid:80) Tt =1 (cid:107) ˆ z t − z t (cid:107) 2 between the predicted and observed trajectories, ˆ z t and z t .
Like NODEs, HNNs also model dynamical systems as a parameterized vector ﬁeld,
[Equation]
where ˆ H θ is a neural network with scalar output (Greydanus et al., 2019).
This differential equation expresses Hamiltonian dynamics where p and q are the canonical positions and momenta,
[Equation]
d
[Equation]
It is common practice to also explicitly deﬁne H
= T + V , where T and V are the kinetic and potential energy (Zhong et al., 2019; Gupta et al., 2019; Finzi et al., 2020).
The assumption that the
Hamiltonian is separable holds for mechanical systems and allows for further simpliﬁcation,
[Equation]
where positive deﬁnite mass matrix M and scalar V are outputs of the neural network.
B REAKING D OWN HNN PERFORMANCE
In the following section we investigate to what extent commonly held beliefs about HNN properties actually explain the ability of HNNs to generalize.
To separate out the different properties of these models, we select synthetic environments from Finzi et al.
(2020) and Finzi et al.
(2021) that are derived from a time independent Hamiltonian, where energy is preserved exactly.
We use kChainPendulum , a k link pendulum in angular coordinates, and kSpringPendulum , a pendulum connected with k spring links in Cartesian coordinates.
We compute relative error between predicted states, ˆ z , and ground truth, z , as (cid:107) ˆ z − z (cid:107) 2 / (cid:107)
ˆ z (cid:107) 2 (cid:107) z (cid:107)and between the predicted and ground truth Hamiltonian as | ˆ H − H | / | ˆ H
|| H
| .
Following Finzi et al.
(2020), we evaluate the performance of the model by computing the geometric mean of the relative error of the state over rollouts of length 20 times the size used at training, which more faithfully predicts downstream performance compared to other metrics like MSE (Finzi et al., 2020).
The geometric mean of a function f over time T is given by exp( 1 T (cid:82) T t =0 log f ( t ) dt ) .
4. E NERGY C ONSERVATION
HNNs are commonly believed to be superior for energy conservation than comparable models (Greydanus et al., 2019; Cranmer et al., 2020).
While there is some empirical evidence to support this claim, a precise mathematical explanation for why this should be the case has not been established.
Surprisingly, we ﬁnd that conditioned on the trajectory reconstruction error, HNNs are no better at conserving the true energy of the system than unconstrained NeuralODE models.
Up to the numerical accuracy of an ODE solver, HNNs do conserve their own learned energy function ˆ H (different from the true energy of the system H ), due to basic properties of Hamiltonian mechanics,
[Equation]
where the last equality follows from the antisymmetry of J .
If the HNN has ﬁt the data well, we might expect ˆ H to be close to the true Hamiltonian H , and so if ˆ H is conserved then it seems that H should be too.
As we show in Appendix B.2, the problem with this argument is that we have no guarantees that ˆ H and H are close even if we have ﬁt the data well with low error in the rollouts or the dynamics.
Since the dynamics error and the training rollout error depend only on gradients ∇ ˆ H , while the gradients may be close, the differences between the two scalar functions can grow arbitrarily large.
In Figure 2 (left) we show that the degree of energy conservation is highly correlated with the rollout error of the model, regardless of the choice of architecture.
While HNNs have better rollout [Figure] n e [Figure]
gy V i
o l a ti [Figure] C h a i n
Sp r i n g
E n e r g y E rr
[Figure]
performance than NeuralODEs, they are on the same regression line with Rollout Error ∝ Energy
Violation.
The differences in energy violation are best predicted by the rollout error and not the architecture.
In Appendix B.1, we derive a bound that helps explain this behaviour.
Given that the trajectories are in a bounded region of the phase space and that there is a ﬁxed amount of error in the dynamics model, the energy violation grows at most linearly in time and the dynamics error.
In Figure 2 (right) we demonstrate empirically that energy is not conserved as time progresses for HNNs.
In fact, the energy error of both NODE and HNN models grow linearly as our bound suggests.
Although energy conservation may be helpful for generalization, the evidence does not indicate that HNNs are inherently better at conserving energy than NODEs, suggesting that the superior generalization of HNNs cannot be attributed to superior energy conservation.
4. S YMPLECTIC VECTOR FIELDS
A deﬁning property of Hamiltonian mechanics is the fact that the dynamics are symplectic .
If energy conservation does not explain the effectiveness of HNNs, the symplectic property of HNN dynamics may be the cause.
Informally, one of the consequences of symplectic dynamics is that every part of the state space is equally attractive and repulsive.
There are no sources where all nearby trajectories ﬂow out from, or sinks where all nearby trajectories ﬂow into, only saddle points and centers where the inﬂow and outﬂow is balanced.
Symplectic integrators (Leimkuhler and Skeel, 1994) make use of this property for more stable integration of Hamiltonian systems over very long timespans, and it is intuitive that enforcing this property in learned dynamics would have beneﬁts also, at the very least for reducing the size of the hypothesis space.
More formally, symplecticity is the property that the J matrix (the symplectic form) is preserved by the dynamics (Equation 1).
This condition can be expressed as a constraint on the Jacobian DF of the vector ﬁeld F ( z ) = dzdt (here the derivative D maps from a function to its Jacobian).
In terms of the Jacobian, symplecticity is the condition DF (cid:62) J
+ JDF = 0 or equivalently ( JDF ) T
= JDF since J (cid:62) =
− J .
The signiﬁcance of this condition is that areas occupied by states in phase space (which have units of energy) are preserved by symplectic transformations.
One consequence is that a volume of solutions in phase space will continue to occupy the same volume over time and will not be compressed or expanded.
In other words, the vector ﬁeld has 0 divergence:
Tr( DF ) = 0 , which one can derive from the above expression. [Figure]
On R n , it can be shown that all symplectic vector ﬁelds can be expressed as the dynamics of some
Hamiltonian system, and vice versa:
[Equation]
To show the forward direction, one can simply substitute in Hamiltonian dynamics.
It is clear that the symplecticity property is satisﬁed: using J 2 =
−
I , JDF =
JD ( J ∇ H )
=
J 2 ∇ 2 H = −∇ 2 H , and −∇ 2 H is a symmetric matrix.
The reverse direction is less obvious; however, by Poincaré’s lemma if a vector ﬁeld F satisﬁes ( JDF ) T = JDF on R n , then there exists a Hamiltonian function H such that F =
J ∇ H , which is shown in subsection B.3.
The equivalence of Hamiltonian dynamics and symplecticity allows us separate the unique properties of HNNs from other inductive biases that result indirectly from modeling F through H . Following Ghosh et al.
(2020), we can create a regularizer SymplecticError( F ) =
(cid:107) ( JDF )
T − JDF (cid:107)that directly measures the degree to which the symplectic property is violated.
By parametrizing a NeuralODE and regularizing the symplectic error, we can enforce Hamiltonian structure while still directly modeling dzdt = F rather than H .
Alongside an unregularized NeuralODE, we can isolate and evaluate the beneﬁt of this Hamiltonian structure bias with a direct comparison.
Surprisingly, we ﬁnd that the Hamiltonian structure bias, as enforced by the symplectic regularizer, provides no real beneﬁt to the model’s ability to generalize over the long test rollouts (Figure 3 left).
The achieved symplectic error Figure 3 (right) is not positively correlated with the ﬁnal test rollout error of the model, and in some cases is even negatively correlated. Even when the symplectic error is very low and the symplecticity condition is enforced, there is no consistent improvement on the rollout generalization.
4. S ECOND - ORDER STRUCTURE
If the superior performance of HNNs over NeuralODEs does not come from their better energy conservation properties, nor from the symplectic structure of the predicted vector ﬁeld, what is the true cause?
In previous work, authors have used slightly different implementations of HNNs.
One subtle improvement over the original work (Greydanus et al., 2019) comes from explicitly splitting the Hamiltonian as ˆ H =
T + V = p (cid:62) M ( q ) − 1 p/ 2 + V ( q ) and modeling the mass matrix M ( q ) and the potential V ( q ) with separate neural networks rather than using a single neural network for ˆ H (Zhong et al., 2019).
This splitting enforces a strong assumption about the functional form of the Hamiltonian that applies to mechanical systems that makes it easier to learn and extrapolate.
Through
Hamiltons equations ˆ F =
J ∇ ˆ H , this splitting is in fact specifying the relationship between position and momentum dq dt
= ∂H ∂p =
M − 1 ( q ) p , and that forces can only affect dp dt . [Figure] [Figure]
In fact [Figure] we can see that this assumption essentially leads to the second-order differential equation (cid:20) [Equation]
[Equation]
This second-order (SO) structure,
[Equation]
) (cid:21) , is a direct by-product of the separable Hamiltonian inductive bias, but is more general, applying to both conservative and non-conservative physical systems.
We can isolate the effect of this bias by directly observing its effect on both HNNs and NODEs.
For HNNs the bias is made explicit in separable Hamiltonians, but not in the general case, when H ( q, p ) is the direct output of the network instead of V ( q ) .
We can design a NODE with secondorder structure (NODE + SO) by setting z = [ q, p ] with p =
Mv and dqdt = v, dpdt = ˜
A θ ( q, p ) , or equivalently just the second order equation d 2 q dt 2 =
A θ ( q, dqdt ) .
Figure 4 shows the effect of second order structure on the test predictions of NODEs and HNNs.
It is clear that this bias explains the superior performance of HNNs much more than other biases that are frequently given more credit.
In fact, when we add this bias to NODE models, we see that their performance more closely matches HNNs than vanilla NODEs without creating a conservative or symplectic vector.
4. F UNCTIONAL COMPLEXITY
Adding second order structure to NODEs is always helpful, and matches the HNN performance for many of the systems.
However, we see that there is still a gap for some systems, and curiously in each of these cases the system is described in a non-Cartesian coordinate system, such as with joint angles and Euler angles.
Recall that with the symplecticity bias, we only found no beneﬁt for enforcing that there exists a function ˆ H such that
ˆ F
=
J ∇ ˆ H bias while parametrizing ˆ F ; however, for HNNs this function not only exists, it is directly parametrized by the neural net.
If the function [Figure]
Spring Spring [Figure] R o ll ou t E rr o r
NODE
NODE
[Figure]
SO
HNN
SymODEN
ˆ H happens to be a simpler function to express and learn than ∇ H , then representing the solution in this way can be
For systems expressed in Cartesian coordinates like the spring pendulum, the mass matrix M ( q ) =
M is a constant, and so the gradients of H = p (cid:62)
Mp + V ( q ) are simple to express and learn.
However, for systems with constraints such as the Chain pendulum, where states are typically expressed in angular coordinates, the mass matrix M ( q ) will have a complicated form and that complexity will be magniﬁed when taking the derivative.
As an example, consider the Hamiltonian that
an
HNN must learn for the 2 link chain pendulum versus the vector ﬁeld that a NODE + SO model must learn for this system (derived in Finzi et al.
(2020), Appendix F.2).
For the spring pendulum the functional complexity of the Hamiltonian and vector ﬁelds is comparable, while for the chain pendulum, the vector ﬁeld contains many more terms.
Parameterizing such a system via its Hamiltonian simpliﬁes the learning problem, and enables a neural network to converge more rapidly towards a plausible solution.
This observation aligns with the insight in Finzi et al.
(2020), which shows that changing the coordinate system to Cartesian dramatically simpliﬁes the learning problem, at the expense of needing to enforce the constraints to the conﬁguration space more directly.
Figure 5 shows the relative performance of the NODE+SO across the chain pendulum and spring pendulum environments.
As we would now expect, the gap between the NODE+SO and HNN vanishes (and even favors NODE+SO) when complexity of the Hamiltonian and vector ﬁeld are comparable.
4.4. N ON - CONSERVATIVE SYSTEMS
Perfectly energy-conserving systems are useful for analyzing the limiting behaviour of physicsinformed networks, but in the vast majority of real world applications, we do not model closed systems.
Energy is changed through contact with the environment (as in friction or drag) or an actor applying controls.
In these cases, HNNs can be generalized by adding a forcing term
[Equation] as in SymODEN (Zhong et al., 2019; 2020) and Desai et al.
(2021), where u is the control input, which can be ﬁxed as constant in systems with drag or friction.
Though SymODEN can accommodate controls and damping, we show that simply using the bias of second-order dynamics is sufﬁcient to achieve nearly the same performance with much less complexity.
We demonstrate the matching performance on our n-body pendulum systems, augmented with drag, dp dt = − ∂H
∂q − λv (Figure 6). [Figure]
M ODELING M UJOCO T RANSITION D YNAMICS
HNNs are typically evaluated on relatively simple systems, like those considered in the previous sections.
In principle, we would expect these results to extend to more complex systems that are governed by similar physical laws.
In practice, however, there is little evidence to suggest that applying HNN methods to complex systems is easy or effective.
The MuJoCo physics simulator (Todorov et al., 2012) is one such complex system that we would expect to beneﬁt from HNN inductive biases.
Gym Mujoco systems are heavily used in model-based reinforcement learning (MBRL) literature, but the dynamics models commonly used are surprisingly simple (often just MLPs trained to predict the next change in state)
(Chua et al., 2018; Wang and
Ba, 2019).
Given how much beneﬁt other applications of deep learning have derived from specialized architectures, such as CNNs for computer vision (LeCun et al., 1989), RNNs for NLP (Mikolov et al., 2010), or WaveNets for audio (Oord et al., 2016), we would expect analogous improvements to be possible in MBRL.
Algorithms for MBRL are infamously sensitive to choice of prediction horizon, and one possible explanation is poor generalization caused by weak inductive biases (Janner et al., 2019; Pan et al., 2020; Amos et al., 2021).
Improving model design for mechanical systems has the potential to improve both the sample efﬁciency of MBRL algorithms and their robustness.
We train NODEs and HNNs on trajectories from several OpenAI Gym Mujoco environments (Brockman et al., 2016).
Crucially, we compare NODEs endowed with second-order structure (NODE + SO) against pre-existing NODE and HNN models, as we did in Section 4.4.1.
Note that with ﬁxed step size integrators, NODEs are equivalent to discrete transition models that predict the next state or delta directly with an MLP, and therefore our NODE baseline is representative of models commonly used in MBRL.
See Appendix C for implementation details.
In Figure 7 we show that NODE + SO signiﬁcantly outperforms baseline methods.
Surprisingly HNNs underperform NODEs on all the systems we consider.
Although the HNN could in principle learn the dynamics, in practice the bias towards Hamiltonian dynamics makes ﬁtting the training data very difﬁcult and provides no tangible beneﬁt to generalization.
This outcome is notably different from what we observe in toy tasks, where HNNs can ﬁt non-conservative systems (e.g., pendulums with drag) with little difﬁculty.
In spirit, our results parallel the ﬁndings of Wu et al.
in the different setting of graph CNNs. We are able to distill the inductive biases of HNNs into a NODE + SO without losing performance on systems that HNNs perform well on, and, even more importantly, these reduced systems are much more capable of scaling to complex systems and larger datasets.
C ONCLUSION
In this paper, we deconstructed the inductive biases of highly performing HNN models into their component parts, a NeuralODE, symplecticity, conservation of the learned energy function, and second order structure.
Contrary to conventional wisdom, the success of HNNs is not from their energy conservation or symplecticity, but rather from the assumption that the system can be expressed as a single second order differential equation.
Stripping away the other components of an HNN, we are left with a model that is simpler, more computationally efﬁcient, and less restrictive in that it can be directly applied to non-Hamiltonian systems.
As a consequence, we are able to apply the resulting model to constructing transition models for the challenging Mujoco locomotion environments, with promising performance. A CKNOWLEDGEMENTS
The authors would like to thank Greg Benton and Pavel Izmailov for helpful comments on our original submission.
This research is supported by an Amazon Research Award, Facebook Research,
Google Research, NSF I-DISRE 193471, NIH R01DA048764-01A1, NSF IIS-1910266, and NSF 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science.
A A PPENDIX O UTLINE
This appendix is organized as follows.
In Section B, we present proofs for the energy conservation properties of HNNs and Neural ODEs, as well as a more detailed description of symplecticity.
In
Section C, we described the details of our experiments on Mujuco, including our data preprocessing, training, and architectural decisions.
Lastly, in Section D, we provide additional experimental results requested by reviewers of our original submission.
These include a comparison of alternative loss functions and a comparison on additional rigid body systems.
B M ATHEMATICAL D ETAILS B. E NERGY C ONSERVATION FOR N EURAL ODE S
Let F
= J ∇ H be the ground truth dynamics of a time independent Hamiltonian, and ˆ F be the dynamics learned by a neural network through an learned Hamiltonian ˆ H for HNNs or otherwise.
Given some initial condition z 0 , let ˆ z T denote the solution to ˙
z
= ˆ F ( z ) at time T starting from zand z T be the solution to the ground truth dynamics from z 0 .
Suppose the error in the dynamics model e ( z )
=
ˆ F ( z ) − F ( z ) is bounded ∀
z : (cid:107) e
( z ) (cid:107) <
δ and that we are only considering a bounded region of the state space (such as the states of a pendulum with bounded energy).
Since energy is conserved H ( z t ) =
H ( z 0 )
=
H (ˆ z 0 ) , we can write the the energy error
[Equation]
Since the value is independent of the path, we may consider the path given by the approximated dynamics ˆ z t .
Noting that d ˆ z/dt = ˆ F
(ˆ z t ) =
J ∇ H (ˆ z t )
+ e (ˆ z t ) , we have
[Equation]
Bounding the maximum value of the integrand along the path, we have that
[Equation]
which grows only linearly with time. This linear bound on the energy error is in stark contrast with the state error which could grow {+exponentially according to the Lyapunov exponents, even if the dynamics error is arbitrarily small. Advancing the ground truth and learned dynamics forward to some small time t = (cid:15) , ˆ z (cid:15) = z 
+}
+ (cid:15) ˆ F ( z 0 ) yields error (cid:107) ˆ z (cid:15)
− z (cid:15) (cid:107) = (cid:107) (cid:15) ˆ F ( z 0 ) − (cid:15)F ( z 0 ) (cid:107) = (cid:15) (cid:107) e ( z ) (cid:107) <
(cid:15)δ .
And yet, this error even if propagated by the ground truth dynamics will grow exponentially (cid:107) ˆ z T − z T (cid:107)
≈ e λT (cid:107) ˆ z (cid:15)
− z (cid:15) (cid:107) =
e λT (cid:15) (cid:107) e ( z ) (cid:107) B. HNN E NERGY C ONSERVATION A simple but erroneous argument for why HNNs approximately conserve the true energy goes as follows:
We would like to know if HNNs achieve better energy conservation given the same levels of error in the predicted dynamics.
For HNNs, ˆ F = J ∇ ˆ H , and we can see that the dynamics error e ( z ) can also be written as e ( z ) =
J ∇ ( ˆ H − H ) .
If we could convert a bound on the derivatives (cid:107) e (cid:107) =
(cid:107)∇ ( ˆ H − H ) (cid:107) <
δ (since J is an orthogonal matrix) into a bound on the learned Hamiltonian itself E ( z ) :
= ˆ H ( z ) − H ( z ) − c and | E | < ∆ holding globally for some constant c , then we would have a constraint on the energy error that doesn’t grow with time.
Expanding the difference in initial and ﬁnal energy, the constant c cancels out and we have
[Equation]
using the fact that the learned energy function ˆ H is conserved over the model rollout ˆ z t .
If there was a constraint | E | < ∆ then
[Equation]
Unfortunately, even if the gradients are close and δ is small, that does not imply that ∆ is small.
Small differences in gradient can add up to very large differences in the values of the two functions.
While the dynamics may well approximate the data, and achieve low rollout error, there is no reason to believe that at a given point in phase space the learned Hamiltonian should have a value that is close to the true Hamiltonian.
B. S YMPLECTICITY
Symplecticity is the requirement that the dynamics satisfy ( JDF ) (cid:62) =
JDF .
Deﬁning G =
JF , the requirement is simply that the antisymmetric part of the jacobian is 0 , DG (cid:62) − DG = 0 .
Unpacking Poincare’s lemma requires some familiarity with differential geometry concepts such as differential forms and exterior derivatives, and so we will assume them but for this section only.
Poincare’s lemma states that on a contractible domain (such as R n ) if a differential k -form ω is closed dω =
(the exterior derivative of ω is 0 ) then it is also exact ω
=
dν
(it is the exterior derivative of another differential ( k − 1 )-form ν ).
While F is a vector ﬁeld, G = JF is a differential 1 -form (dual to a vector ﬁeld).
If DG is symmetric, then it is also closed: dG =
(cid:80) i ∂ i
G j dx i ∧ dx j = 2 (cid:80) i ( ∂ i
G j − ∂ j G i )
dx i ∧ dx j = 0 since ( ∂ i G j − ∂ j G i )
= 0 is just another way of expressing DG (cid:62) − DG = 0 .
Therefore by Poincare’s lemma, G = dφ for some 0 -form (scalar function) φ .
Therefore F = J − 1 dφ =
Jd ( − φ ) since J − 1 = − J .
As the exterior derivative of scalar function is just the gradient, we can deﬁne H = − φ and see that there exists a scalar function H such that
F = J ∇ H .
C M UJOCO E XPERIMENT D ETAILS
Data collection: for each control task we trained a standard soft actor-critic RL agent to convergence (Haarnoja et al., 2018).
Note that we had to use modiﬁed versions of the Gym environments since the standard environments preprocess observations in ad-hoc ways.
For example, Hopper clips the velocity observations to [ − 10 , 10] d and truncates part of the position.
2 Our versions of the environments simply return [ q, v ] as the observation.
Then we randomly split the episodes in the replay buffer into train and test.
The training data was 40K 3-step trajectories (i.e. two transitions) randomly sampled from the training episodes.
The test data was 200 200-step trajectories randomly sampled from the test episodes.
This data-collection strategy is important to the experiment because random controls typically do not cause the agent to cover the entire state-action space.
Similarly many control policies are highly cyclical, so it is important to separate train and test splits at the episode level.
Training: we trained each model for 256 epochs using Adam with a batch size of 200 and weight decay ( λ = 1e-4).
We used a cosine annealing learning rate schedule, with η max = 2e-4, η min = 1e-6.
Model Architecture Each network was parameterized as a 2-layer MLP with 128 hidden units.
Each model used the Euler integration rule with 8 integration steps per transition step.
The step size was determined by the integration step size of the underlying environment, h = ∆ t/ 8 .
NODE + SO: given the state z and controls u , a standard NODE takes dz/dt = f
( z, u, θ ) .
However, if z = [ q, v ] (that is, if both position and velocity are observed), then we already have a good estimate of dq/dt in the observation itself, namely v .
Hence we propose only using the network to model acceleration dv/dt = f
( z, u, θ ) , and to model dq/dt implicitly.
It is important to note that we cannot take dq/dt = v because v is observed before the control u is applied.
Instead we take dq/dt = v/ 2 + ( v + h × dv/dt ) / 2 , averaging the velocity at time t (before the control is applied) and the predicted velocity at time t + 1 (after the control is applied), given an Euler integration step of size h on v .
This integration rule can be viewed as an approximate RK2 step on q t , where v t +1 is approximated via a learned Euler step on v t .
This approach has two beneﬁts.
In the ﬁrst place it constrains the predicted velocity and acceleration to be consistent across time.
Second the model is able to take an approximate RK2 step on q at the cost of a single forward pass (instead of 2).
The latter is important because integration error can accumulate over long rollouts, even if the model ﬁts the dynamics very well.
D A DDITIONAL E XPERIMENTAL R ESULTS D. C OMPARISON OF LOSS FUNCTIONS
In the experimental results presented in the body of the paper were obtained training on l2 loss between integrated and ground truth trajectories.
As noted in feedback the an early version of the paper, this practice goes contrary to prior work using l1 loss for stability (Finzi et al., 2020).
In
Figure 8 we show the result of changing the loss.
[Figure]
rr
[Figure]
NODE
[Equation]
HNN
D. A DDITIONAL S YSTEMS
To extend the comparison of NODE and HNN models, we trained models on three additional systems presented by Finzi et al.
Figure 9 shows the rollout error of NODE, NODE + SO, and
HNN models.
One the gyroscope system, we observe a similar result to the one above.
In the magnet pendulum and rotor systems, the results are slightly more counterintuitive, with the NODE model outperforming the more sophisticated alternatives.
We suspect the small difference in performance in the models is due to the challenge of stably training HNNs in complex systems (with magnet pendulum having complex dynamics and rotor having a complex coordinate system).
[Figure]