Original: H AMILTONIAN G ENERATIVE N ETWORKS
Revised: H AMILTONIAN G ENERATIVE N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The Hamiltonian formalism plays a central role in classical and quantum physics.
Revised: The Hamiltonian formalism plays a central role in classical and quantum physics.

Original: Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserved quantities, and they come equipped with many useful properties, like time reversibility and smooth interpolation in time.
Revised: Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserved quantities, and they come equipped with many useful properties, like time reversibility and smooth interpolation in time.

Original: These properties are important for many machine learning problems —from sequence prediction to reinforcement learning and density modelling —but are not typically provided out of the box by standard tools such as recurrent neural networks.
Revised: These properties are important for many machine learning problems —from sequence prediction to reinforcement learning and density modelling —but are not typically provided out of the box by standard tools such as recurrent neural networks.

Original: In this paper, we introduce the Hamiltonian Generative Network (HGN), the ﬁrst approach capable of consistently learning Hamiltonian dynamics from high-dimensional observations (such as images) without restrictive domain assumptions.
Revised: In this paper, we introduce the Hamiltonian Generative Network (HGN), the ﬁrst approach capable of consistently learning Hamiltonian dynamics from high-dimensional observations (such as images) without restrictive domain assumptions.

Original: Once trained, we can use HGN to sample new trajectories, perform rollouts both forward and backward in time, and even speed up or slow down the learned dynamics.
Revised: Once trained, we can use HGN to sample new trajectories, perform rollouts both forward and backward in time and even speed up or slow down the learned dynamics.

Original: 1 We demonstrate how a simple modiﬁcationofthenetworkarchitectureturnsHGNintoapowerfulnormalisingﬂowmodel, called Neural Hamiltonian Flow (NHF), that uses Hamiltonian dynamics to model expressive densities.
Revised: 1 We demonstrate how a simple modiﬁcation of the network architecture turns HGN into a powerful normalising ﬂow model, called Neural Hamiltonian Flow (NHF), that uses Hamiltonian dynamics to model expressive densities.

Original: Hence, we hope that our work serves as a ﬁrst practical demonstration of the value that the Hamiltonian formalism can bring to machine learning.
Revised: We hope that our work serves as a ﬁrst practical demonstration of the value that the Hamiltonian formalism can bring to deep learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Any system capable of a wide range of intelligent behaviours within a dynamic environment requires a good predictive model of the environment’s dynamics.
Revised: Any system capable of a wide range of intelligent behaviours within a dynamic environment requires a good predictive model of the environment’s dynamics.

Original: This is true for intelligence in both biological (Friston, 2009; 2010; Clark, 2013) and artiﬁcial (Hafner et al., 2019; Battaglia et al., 2013; Watter et al., 2015; Watters et al., 2019) systems.
Revised: This is true for intelligence in both biological (Friston, 2009; 2010; Clark, 2013) and artiﬁcial (Hafner et al., 2019; Battaglia et al., 2013; Watter et al., 2015; Watters et al., 2019) systems.

Original: Predicting environmental dynamics is also of fundamental importance in physics, where Hamiltonian dynamics and the structurepreserving transformations it provides have been used to unify, categorise and discover new physical entities (Noether, 1915; Livio, 2012).
Revised: Predicting environmental dynamics is also of fundamental importance in physics, where Hamiltonian dynamics and the structurepreserving transformations it provides have been used to unify, categorise and discover new physical entities (Noether, 1915; Livio, 2012).

Original: Hamilton’s fundamental result was a system of two ﬁrst-order differential equations that, in a stroke, uniﬁed the predictions made by prior
Revised: Hamilton’s fundamental result was a system of two ﬁrst-order differential equations that, in a stroke, uniﬁed the predictions made by prior

Original: Newtonian and Lagrangian mechanics (Hamilton, 1834).
Revised: Newtonian and Lagrangian mechanics (Hamilton, 1834).

Original: After well over a century of development, it has proven to be essential for parsimonious descriptions of nearly all of physics.
Revised: After well over a century of development, it has proven to be essential for parsimonious descriptions of nearly all of physics.

Original: Hamilton’s equations provide a way to predict a system’s future behavior from its current state in phase space (that is, its position and momentum for classical Newtonian systems, and its generalized position and momentum more broadly).
Revised: Hamilton’s equations provide a way to predict a system’s future behavior from its current state in phase space (that is, its position and momentum for classical Newtonian systems, and its generalized position and momentum more broadly).

Original: Hamiltonian mechanics induce dynamics with several nice properties: they are smooth, they include paths along which certain physical quantities are conserved (symmetries), and their time evolution is fully reversible.
Revised: Hamiltonian mechanics induce dynamics with several nice properties: they are smooth, they include paths along which certain physical quantities are conserved (symmetries) and their time evolution is fully reversible.

Original: These properties are also useful for machine learning systems.
Revised: These properties are also useful for machine learning systems.

Original: For example, capturing the time-reversible dynamics of the world state might be useful for agents attempting to account for how their actions led to effects in the world; recovering an dt dt dt dt
Revised: For example, capturing the time-reversible dynamics of the world state might be useful for agents attempting to account for how their actions led to effects in the world; recovering an dt dt dt dt

Original: E
Revised: E

Original: H γ z ~ q ɸ (z|x 0 ...x T ) sxx T t q ɸ f ψ q p
Revised: H γ z ~ q ɸ (z|x 0 ...x T ) sxx T t q ɸ f ψ q p

Original: E
Revised: E

Original: H γ + - s T q p d θ x T - +
Revised: H γ + - s T q p d θ x T - +

Original: I n f e r e n c e
Revised: I n f e r e n c e

Original: D e c o d e r
Revised: D e c o d e r

Original: H a m i l t o n i a n
Revised: H a m i l t o n i a n

Original: Forward rollout +dt
Revised: Forward rollout +dt

Original: Reverse rollout -dt
Revised: Reverse rollout -dt

Original: dt dt sq p d θ x abstract low-dimensional manifold with paths that conserve various properties is tightly connected to outstanding problems in representation learning (see e.g. Higgins et al.
Revised: dt dt sq p d θ x abstract low-dimensional manifold with paths that conserve various properties is tightly connected to outstanding problems in representation learning (see e.g. Higgins et al.

Original: (2018) for more discussion); and the ability to conserve energy is related to expressive density modelling in generative approaches (Rezende & Mohamed, 2015).
Revised: (2018) for more discussion); and the ability to conserve energy is related to expressive density modelling in generative approaches (Rezende & Mohamed, 2015).

Original: Hence, we propose a reformulation of the well-known image manifold hypothesis by extending it with a Hamiltonian assumption (illustrated in Fig.
Revised: Hence, we propose a reformulation of the well-known image manifold hypothesis by extending it with a Hamiltonian assumption (illustrated in Fig.

Original: 1): natural images lie on a low-dimensional manifold embedded within a high-dimensional pixel space and natural sequences of images trace out paths on this manifold that follow the equations of an abstract Hamiltonian .
Revised: 1): natural images lie on a low-dimensional manifold embedded within a high-dimensional pixel space and natural sequences of images trace out paths on this manifold that follow the equations of an abstract Hamiltonian .

Original: Given the rich set of established tools provided by Hamiltonian descriptions of system dynamics, can we adapt these to solve outstanding machine learning problems?
Revised: Given the rich set of established tools provided by Hamiltonian descriptions of system dynamics, can we adapt these to solve outstanding machine learning problems?

Original: When it comes to adapting the
Revised: When it comes to adapting the

Original: Hamiltonian formalism to contemporary machine learning, two questions need to be addressed: 1) how should a system’s
Revised: Hamiltonian formalism to contemporary machine learning, two questions need to be addressed: 1) how should a system’s

Original: Hamiltonian be learned from data; and 2) how should a system’s abstract phase space be inferred from the high-dimensional observations typically available to machine learning systems?
Revised: Hamiltonian be learned from data; and 2) how should a system’s abstract phase space be inferred from the high-dimensional observations typically available to machine learning systems?

Original: Note that the inferred state may need to include information about properties that play no physical role in classical mechanics but which can still affect their behavior or function, like the colour or shape of an object.
Revised: Note that the inferred state may need to include information about properties that play no physical role in classical mechanics but which can still affect their behavior or function, like the colour or shape of an object.

Original: The ﬁrst question was recently addressed by the Hamiltonian Neural Network (HNN)
Revised: The ﬁrst question was recently addressed by the Hamiltonian Neural Network (HNN)

Original: (Greydanus et al., 2019) approach, which was able to learn the Hamiltonian of three simple physical systems from noisy phase space observations.
Revised: (Greydanus et al., 2019) approach, which was able to learn the Hamiltonian of three simple physical systems from noisy phase space observations.

Original: However, to address the second question, HNN makes assumptions that restrict it to Newtonian systems and appear to limit its ability to scale to more challenging video datasets.
Revised: However, to address the second question, HNN makes assumptions that restrict it to Newtonian systems and appear to limit its ability to scale to more challenging video datasets.

Original: In this paper we introduce the ﬁrst model that answers both of these questions without relying on restrictive domain assumptions.
Revised: In this paper we introduce the ﬁrst model that answers both of these questions without relying on restrictive domain assumptions.

Original: Our model, the Hamiltonian Generative Network (HGN), is a generative model that infers the abstract state from pixels and then unrolls the learned Hamiltonian following the Hamiltonian equations (Goldstein, 1980).
Revised: Our model, the Hamiltonian Generative Network (HGN), is a generative model that infers the abstract state from pixels and then unrolls the learned Hamiltonian following the Hamiltonian equations (Goldstein, 1980).

Original: We demonstrate that HGN is able to reliably learn the Hamiltonian dynamics from noisy pixel observations on four simulated physical systems: a pendulum, a mass-spring and two- and three- body systems.
Revised: We demonstrate that HGN is able to reliably learn the Hamiltonian dynamics from noisy pixel observations on four simulated physical systems: a pendulum, a mass-spring and two- and three- body systems.

Original: Our approach outperforms HNN by a signiﬁcant margin.
Revised: Our approach outperforms HNN by a signiﬁcant margin.

Original: After training, we demonstrate that HGN produces meaningful samples with reversible dynamics and that the speed of rollouts can be controlled by changing the time derivative of the integrator at test time.
Revised: After training, we demonstrate that HGN produces meaningful samples with reversible dynamics and that the speed of rollouts can be controlled by changing the time derivative of the integrator at test time.

Original: Finally, we show that a small modiﬁcation of our architecture yields a ﬂexible, normalising ﬂow-based generative model that respects Hamiltonian dynamics.
Revised: Finally, we show that a small modiﬁcation of our architecture yields a ﬂexible, normalising ﬂow-based generative model that respects Hamiltonian dynamics.

Original: We show that this model, which we call Neural Hamiltonian Flow (NHF), inherits the beneﬁcial properties of HGN (including volume preservation) and is capable of expressive density modelling, while offering computational beneﬁts over standard ﬂow-based models.
Revised: We show that this model, which we call Neural Hamiltonian Flow (NHF), inherits the beneﬁcial properties of the Hamiltonian formalism (including volume preservation) and is capable of expressive density modelling, while offering computational beneﬁts over standard ﬂow-based models.

Original: Related work Most machine learning approaches to modeling dynamics use discrete time steps, which often results in an accumulation of the approximation errors when producing rollouts and, therefore, to a fast drop in accuracy.
Revised: Most machine learning approaches to modeling dynamics use discrete time steps, which often results in an accumulation of the approximation errors when producing rollouts and, therefore, to a fast drop in accuracy.

Original: Our approach, on the other hand, does not discretise continuous dynamics and models them directly using the Hamiltonian differential equations, which leads to slower divergence for longer rollouts.
Revised: Our approach, on the other hand, does not discretise continuous dynamics and models them directly using the Hamiltonian differential equations, which leads to slower divergence for longer rollouts.

Original: The density model version of HGN (NHF) uses the Hamiltonian dynamics as normalising ﬂows along with a numerical integrator, making our approach somewhat related to the recently published neural ODE work (Chen et al., 2018; Grathwohl et al., 2018).
Revised: ThedensitymodelversionofHGN(NHF)usestheHamiltoniandynamicsasnormalisingﬂows along with a numerical integrator, making our approach somewhat related to the recently published neural ODE work (Chen et al., 2018; Grathwohl et al., 2018).

Original: What makes our approach different is that Hamiltonian dynamics are both invertible and volume-preserving (as discussed in Sec. 2.3), which makes our approach computationally cheaper than the alternatives and more suitable as a model of physical systems and other processes that have these properties.
Revised: What makes our approach different is that Hamiltonian dynamics are both invertible and volume-preserving (as discussed in Sec. 3.3), which makes our approach computationally cheaper than the alternatives and more suitable as a model of physical systems and other processes that have these properties.

Revised: Also related is recent work attempting to learn a model of physical system dynamics end-to-end from image sequences using an autoencoder (de Avila Belbute-Peres et al., 2018).

Revised: Unlike our work, this model does not exploit Hamiltonian dynamics and is trained in a supervised or semi-supervised regime.

Original: M ETHODS 2.T HE H AMILTONIAN FORMALISM
Revised: 2. H AMILTONIAN N EURAL N ETWORK

Revised: One of the most comparable approaches to ours is the Hamiltonian Neural Network (HNN)

Revised: (Greydanus et al., 2019).

Revised: This work, done concurrently to ours, proposes a way to learn Hamiltonian dynamics from data by training the gradients of a neural network (obtained by backpropagation) to match the time derivative of a target system in a supervised fashion.

Revised: In particular, HNN learns a differentiable function

Revised: H ( q,p ) that maps a system’s state (its position, q , and momentum, p ) to a scalar quantity interpreted as the system’s Hamiltonian.

Revised: This model is trained so that H ( p,q ) satisﬁes the Hamiltonian equation by minimizing

Revised: [Equation]

Revised: where the derivatives ∂ H ∂q and ∂ H ∂p are computed by backpropagation.

Revised: Hence, this learning procedure is most directly applicable when the true state space (in canonical coordinates) and its time derivatives are known.

Revised: Accordingly, in the majority of the experiments presented by the authors, the Hamiltonian was learned from the ground truth state space directly, rather than from pixel observations.

Revised: The single experiment with pixel observations required a modiﬁcation of the model.

Revised: First, the input to the model became a concatenated, ﬂattened pair of images o t =[ x t , x t +1 ] , which was then mapped to a low-dimensional embedding space z t =

Revised: [ q t ,p t ] using an encoder neural network.

Revised: Note that the dimensionality of this embedding ( z ∈ R 2 in the case of the pendulum system presented in the paper) was chosen to perfectly match the ground truth dimensionality of the phase space, which was assumed to be known a priori.

Revised: This, however, is not always possible.

Revised: The latent embedding was then treated as an estimate of the position and the momentum of the system depicted in the images, where the momentum was assumed to be equal to the velocity of the system – an assumption enforced by the additional constraint found necessary to encourage learning, which encouraged the time derivative of the position latent to equal the momentum latent using ﬁnite differences on the split latents:

Revised: [Equation]

Revised: This assumption is appropriate in the case of the simple pendulum system presented in the paper, however it does not hold more generally.

Revised: Note that our approach does not make any assumptions on the dimensionality of the learned phase space, or the form of the momenta coordinates, which makes our approach more general and allows it to perform well on a wider range of image domains as presented in

Revised: Sec. 4.

Revised: H

Revised: i f i u x u ~ 𝛑 (u) x ~ p(x) ss i s 0 ~ 𝛑 (s 0 ) s i ~ p(s i )

Revised: E + -

Original:  The Hamiltonian formalism describes the continuous time evolution of a system in an abstract phase space s = ( q , p ) ∈
Revised: A B  M ETHODS 3. T HE H AMILTONIAN FORMALISM The Hamiltonian formalism describes the continuous time evolution of a system in an abstract phase space s = ( q , p ) ∈

Original: R 2 n , where q ∈ R n is a vector of position coordinates, and p ∈ R n is the corresponding vector of momenta.
Revised: R 2 n , where q ∈ R n is a vector of position coordinates, and p ∈ R n is the corresponding vector of momenta.

Original: The time evolution of the system inthe phase space is given by the
Revised: The time evolution of the system in phase space is given by the

Original: Hamiltonian equations:
Revised: Hamiltonian equations:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the Hamiltonian H : R 2 n → R maps the state s = ( q , p ) to a scalar representing the energy of the system.
Revised: where the Hamiltonian H : R 2 n → R maps the state s = ( q , p ) to a scalar representing the energy of the system.

Original: The Hamiltonian speciﬁes a vector ﬁeld over the phase space that describes all possible dynamics of the system.
Revised: The Hamiltonian speciﬁes a vector ﬁeld over the phase space that describes all possible dynamics of the system.

Original: For example, a Hamiltonian for an undamped mass-spring system is H ( q,p )=
Revised: For example, the Hamiltonian for an undamped mass-spring system is H ( q,p )=

Original: 12 kq 2 + p 2 2 m , where m is the mass, q ∈ R 1 is its position, p ∈ R 1 is its momentum and k is the spring stiffness coefﬁcient.
Revised: 12 kq 2 + p 2 2 m , where m is the mass, q ∈ R 1 is its position, p ∈ R 1 is its momentum and k is the spring stiffness coefﬁcient.

Original: The Hamiltonian can often be expressed as the sum of the kinetic T and potential V energies H
Revised: The Hamiltonian can often be expressed as the sum of the kinetic T and potential V energies H

Original: = T ( p ) + V ( q ) , as is the case for the mass-spring example.
Revised: = T ( p )+ V ( q ) , as is the case for the mass-spring example.

Original: Identifying a system’s
Revised: Identifying a system’s

Original: Hamiltonian is in general a very difﬁcult problem, requiring carefully instrumented experiments and researcher insight produced by years of training.
Revised: Hamiltonian is in general a very difﬁcult problem, requiring carefully instrumented experiments and researcher insight produced by years of training.

Original: In what follows, we describe a method for modeling a system’s
Revised: In what follows, we describe a method for modeling a system’s

Original: Hamiltonian from raw observations (such as pixels) by inferring a system’s state with a generative model and rolling it out with the Hamiltonian equations.
Revised: Hamiltonian from raw observations (such as pixels) by inferring a system’s state with a generative model and rolling it out with the Hamiltonian equations.

Original: 2. L EARNING H AMILTONIANS WITH THE H AMILTONIAN G ENERATIVE N ETWORK
Revised: 3. L EARNING H AMILTONIANS WITH THE H AMILTONIAN G ENERATIVE N ETWORK

Original: Our goal is to build a model that can learn a Hamiltonian from observations.
Revised: Our goal is to build a model that can learn a Hamiltonian from observations.

Original: We assume that the data
Revised: We assume that the data

Original: X = { ( x 10 ,..., x 1 T ) ,..., ( x K 0 ,..., x KT ) } comes in the form of high-dimensional noisy observations, where each x i =
Revised: X = { ( x 10 ,..., x 1 T ) ,..., ( x K 0 ,..., x KT ) } comes in the form of high-dimensional noisy observations, where each x i =

Original: G ( s i )=
Revised: G ( s i )=

Original: G ( q i ) is a non-deterministic function of the generalised position in the phase space, and the full state is a non-deterministic function of a sequence of images
Revised: G ( q i ) is a non-deterministic function of the generalised position in the phase space, and the full state is a non-deterministic function of a sequence of images

Original: s i =(
Revised: s i =(

Original: q i , p i )
Revised: q i , p i )

Original: =
Revised: =

Original: D ( x i 0 ,..., x it ) , since the momentum (and hence the full state) cannot in general be recovered from a single observation.
Revised: D ( x i 0 ,..., x it ) , since the momentum (and hence the full state) cannot in general be recovered from a single observation.

Original: Our goal is to infer the abstract state and learn the Hamiltonian dynamics in phase space by observing
Revised: Our goal is to infer the abstract state and learn the Hamiltonian dynamics in phase space by observing

Original: K motion sequences, discretised into T  time steps each.
Revised: K motion sequences, discretised into T +1 time steps each.

Original: In the process, we also want to learn an approximation to the generative process G ( s ) in order to be able to move in both directions between the high dimensional observations and the low-dimensional abstract phase space.
Revised: In the process, we also want to learn an approximation to the generative process G ( s ) in order to be able to move in both directions between the high dimensional observations and the low-dimensional abstract phase space.

Original: Although the Hamiltonian formalism is general and does not depend on the form of the observations, we present our model in terms of visual observations, since many known physical Hamiltonian systems, like a mass-spring system, can be easily observed visually.
Revised: Although the Hamiltonian formalism is general and does not depend on the form of the observations, we present our model in terms of visual observations, since many known physical Hamiltonian systems, like a mass-spring system, can be easily observed visually.

Original: In this section we introduce  Hamiltonian
Revised: In this section we introduce the Hamiltonian

Original: Generative Network (HGN), a generative model that is trained to behave according to the Hamiltonian dynamics in an abstract phase space learned from raw observations of image sequences.
Revised: Generative Network (HGN), a generative model that is trained to behave according to the Hamiltonian dynamics in an abstract phase space learned from raw observations of image sequences.

Original: HGN consists of three parts (see Fig. 2): an inference network, a Hamiltonian network and a decoder network, which are discussed next.
Revised: HGN consists of three parts (see Fig. 2): an inference network, a Hamiltonian network and a decoder network, which are discussed next.

Original: The inference network takes in a sequence of images ( x i 0 ,... x iT ) , concatenated along the channel dimension, and outputs a posterior over the initial state z ∼ q φ ( ·| x 0 ,... x T ) , corresponding to the system’s coordinates in phase space at the ﬁrst frame of the sequence.
Revised: The inference network takes in a sequence of images ( x i 0 ,... x iT ) , concatenated along the channel dimension, and outputs a posterior over the initial state z ∼ q φ ( ·| x 0 ,... x T ) , corresponding to the system’s coordinates in phase space at the ﬁrst frame of the sequence.

Original: We parametrise q φ ( z ) as a diagonal Gaussian with a unit Gaussian prior p ( z ) =
Revised: We parametrise q φ ( z ) as a diagonal Gaussian with a unit Gaussian prior p ( z ) =

Original: N (0 , I ) and optimise it using the usual reparametrisation trick (Kingma & Welling, 2014).
Revised: N (0 , I ) and optimise it using the usual reparametrisation trick (Kingma & Welling, 2014).

Original: To increase the expressivity of the abstract phase space s 0 , we map samples from the posterior with another function s [-
-]
Revised: To increase the expressivity of the abstract phase E H Tdt dt dt dt s [-
-]T f ψ

Original: = f
Revised: E H T - + s Tq Tp T- - s 0 q 0 ~ 𝝿 (q 0 ) p 0 ~ 𝝿 (p 0 ) +

Original: ψ ( z ) to obtain the system’s
Revised: q T p T ~ q(p T |q T ) Inference

Original: H
Revised: Sampling -dt

Original: i f i u x u ~ 𝛑 (u) x ~ p(x) ss i s 0 ~ 𝛑 (s 0 ) s i ~ p(s i ) E + -{+
 =+}
Revised: +dt dt dt space s 0 , we map samples from the posterior with another function s {+
 =+}

Original: A
Revised: f

Original: B initial state.
Revised: ψ ( z ) to obtain the system’s initial state.

Original: As mentioned in Sec. 2.1, the Hamiltonian function expects the state to be of the form s = ( q , p ) , hence we initialise s 0 ∈ R 2 n, and arbitrarily assign the ﬁrst half of the units to represent abstract position q and the other half to represent abstract momentum p .
Revised: As mentioned in Sec. 3.1, the Hamiltonian function expects the state to be of the form s = ( q , p ) , hence we initialise s 0 ∈ R 2 n and arbitrarily assign the ﬁrst half of the units to represent abstract position q and the other half to represent abstract momentum p .

Original: The Hamiltonian network is parametrised as a neural network with parameters γ that takes in the inferred abstract state and maps it to a scalar H γ ( s t ) ∈ R .
Revised: The Hamiltonian network is parametrised as a neural network with parameters γ that takes in the inferred abstract state and maps it to a scalar H γ ( s t ) ∈ R .

Original: We can use this function to do rollouts in the abstract state space using the Hamiltonian equations (Eq. 1), for example by Euler integration: s t+1 =
Revised: We can use this function to do rollouts in the abstract state space using the Hamiltonian equations (Eq. 3), for example by Euler integration: s t

Original: ( q t +1 , p t +1 ) =
Revised: +1 =( q t +1 , p t +1 )=(

Original: ( q t + ∂ H ∂ p t dt, p t − ∂ H ∂ q t dt ) .
Revised: q t + ∂ H ∂ p t dt, p t − ∂ H ∂ q t dt ) .

Original: In  practice we use a more sophisticated leapfrog integratortorolloutthesystem, sinceithasbettertheoreticalpropertiesandresultsinbetterperformance in practice (see Sec.
Revised: In this work we assume a separable Hamiltonian, so in practice we use a more sophisticated leapfrog integrator to roll out the system, since it has better theoretical properties and results in better performance in practice (see Sec.

Original: A.5 in Supplementary Materials for more details).
Revised: A.6 in Supplementary Materials for more details).

Original: The decoder network is a standard deconvolutional network (we use the architecture from Karras et al.
Revised: The decoder network is a standard deconvolutional network (we use the architecture from Karras et al.

Original: that takes in a low-dimensional representation vector and produces a high-dimensional pixel reconstruction.
Revised: that takes in a low-dimensional representation vector and produces a high-dimensional pixel reconstruction.

Original: Given that each instantaneous image does not depend on the momentum information, we restrict the decoder to take only the position coordinates of the abstract state as input: p θ ( x t )=
Revised: Given that each instantaneous image does not depend on the momentum information, we restrict the decoder to take only the position coordinates of the abstract state as input: p θ ( x t )=

Original: d θ ( q t ) .
Revised: d θ ( q t ) .

Original: The objective function.
Revised: The objective function.

Original: Given a sequence of T  images, HGN is trained to optimise the following objective:
Revised: Given a sequence of T +1 images, HGN is trained to optimise the following objective:

Original: [Equation]
Revised: [Equation]

Original: which can be seen as a temporally extended variational autoencoder (VAE) (Kingma & Welling, 2014;
Revised: which can be seen as a temporally extended variational autoencoder (VAE) (Kingma & Welling, 2014;

Original: Rezende et al., 2014) objective, consisting of a reconstruction term for each frame, and an additional term that encourages the inferred posterior to match a prior.
Revised: Rezende et al., 2014) objective, consisting of a reconstruction term for each frame, and an additional term that encourages the inferred posterior to match a prior.

Original: The key difference with a standard VAE lies in how we generate rollouts – these are produced using the Hamiltonian equations of motion in learned Hamiltonian phase space.
Revised: The key difference with a standard VAE lies in how we generate rollouts – these are produced using the Hamiltonian equations of motion in learned Hamiltonian phase space.

Original: 2. L EARNING H AMILTONIAN F LOWS
Revised: 3. L EARNING H AMILTONIAN F LOWS

Original: In this section, we describe how the architecture described above can be modiﬁed to produce a model for ﬂexible density estimation.
Revised: In this section, we describe how the architecture described above can be modiﬁed to produce a model for ﬂexible density estimation.

Original: Learning computationally feasible and accurate estimates of complex densities is an open problem in generative modelling.
Revised: Learning computationally feasible and accurate estimates of complex densities is an open problem in generative modelling.

Original: A common idea to address this problem is to start with a simple prior distribution π ( u ) and then transform it into a more expressive form p ( x ) through a series of composable invertible transformations
Revised: A common idea to address this problem is to start with a simple prior distribution π ( u ) and then transform it into a more expressive form p ( x ) through a series of composable invertible transformations

Original: f i ( u ) called normalising ﬂows (Rezende &
Revised: f i ( u ) called normalising ﬂows (Rezende &

Original: Mohamed, 2015) (see Fig.
Revised: Mohamed, 2015) (see Fig.

Original: 3A).
Revised: 3A).

Original: Sampling can then be done according to x = f T ◦ ... ◦
Revised: Sampling can then be done according to x = f T ◦ ... ◦

Original: f 1 ( u ) , where u ∼ π ( · ) .
Revised: f 1 ( u ) , where u ∼ π ( · ) .

Original: Density evaluation, however, requires more expensive computations of both inverting the ﬂows and calculating the determinants of their Jacobians.
Revised: Density evaluation, however, requires more expensive computations of both inverting the ﬂows and calculating the determinants of their Jacobians.

Original: For a single ﬂow step, this equates to the  following:p ( x )=
Revised: For a single ﬂow step, this equates to the [Figure] following:

Original: π ( f − 1 ( x )) (cid:12) (cid:12) (cid:12) det (cid:16) ∂f − 1 ∂ x (cid:17)(cid:12) (cid:12) (cid:12) .
Revised: [Equation]

Original: While a lot of work has been done recently into proposing better alternatives for ﬂow-based generative models in machine learning (Rezende & Mohamed, 2015;
Revised: While a lot of work has been done recently into proposing better alternatives for ﬂow-based generative models in machine learning (Rezende & Mohamed, 2015;

Original: Kingma et al., 2016;
Revised: Kingma et al., 2016;

Original: Papamakarios et al., 2017; Dinh et al., 2017; Huang et al., 2018; Kingma &
Revised: Papamakarios et al., 2017; Dinh et al., 2017; Huang et al., 2018; Kingma &

Original: Dhariwal, 2018; Hoogeboom et al., 2019; Chen et al., 2019; Grathwohl et al., 2018), none of the approaches manage to produce both sampling and density evaluation steps that are computationally scalable.E H Tdt dt dt dt s T f ψ E
Revised: Dhariwal, 2018; Hoogeboom et al., 2019; Chen et al., 2019; Grathwohl et al., 2018), none of the approaches manage to produce both sampling and density evaluation steps that are computationally scalable.

Original: H T - + s Tq

Original: Tp T- - s 0 q 0 ~ 𝝿 (q 0 ) p 0 ~ 𝝿 (p 0 )

Original: +

Original: q T p T ~ q(p T |q T )

Original: Inference Sampling -dt +dt dt dt Note that the two requirements for normalising ﬂows are that they are invertible and volume preserving, which are exactly the two properties that Hamiltonian dynamics posses.
Revised: The two requirements for normalising ﬂows are that they are invertible and volume preserving, which are exactly the two properties that Hamiltonian dynamics possess.

Original: The latter can be seen by computing the determinant of the Jacobian of the inﬁnitesimal transformation induced by the Hamiltonian H :
Revised: This can be seen by computing the determinant of the Jacobian of the inﬁnitesimal transformation induced by the Hamiltonian H . By Jacobi’s formula, the derivative of the determinant at the identity is the trace and: det

Original: [Equation]
Revised: [Equation]

Original: where i (cid:54) =
Revised: where i (cid:54) =

Original: j are the off-diagonal entries of the determinant of the Jacobian.
Revised: j are the off-diagonal entries of the determinant of the Jacobian.

Original: Hence, in this section we describe a simple modiﬁcation of HGN that allows it to act as a normalising ﬂow.
Revised: Hence, in this section we describe a simple modiﬁcation of HGN that allows it to act as a normalising ﬂow.

Original: We will refer to this modiﬁcation as the Neural Hamiltonian Flow (NHF) model.
Revised: We will refer to this modiﬁcation as the Neural Hamiltonian Flow (NHF) model.

Original: First, we assume that the initial state s 0 is a sample from a simple prior s 0 ∼ π 0 ( · ) .
Revised: First, we assume that the initial state s 0 is a sample from a simple prior s 0 ∼ π 0 ( · ) .

Original: We then chain several Hamiltonians H i to transform the sample to a new state s T
Revised: We then chain several Hamiltonians H i to transform the sample to a new state s T

Original: = H T ◦ ... ◦
Revised: = H T ◦ ... ◦

Original: H 1 ( s 0 ) which corresponds to a sample from the more expressive ﬁnal density s T ∼ p ( x ) (see Fig.
Revised: H 1 ( s 0 ) which corresponds to a sample from the more expressive ﬁnal density s T ∼ p ( x ) (see Fig.

Original: 3B for an illustration of a single Hamiltonian ﬂow).
Revised: 3B for an illustration of a single Hamiltonian ﬂow).

Original: Note that unlike HGN, where the Hamiltonian dynamics are shared across time steps (a single Hamiltonian is learned and its parameters are shared across time steps of a rollout), in NHF each step of the ﬂow (corresponding to a single time step of a rollout) can be parametrised by a different Hamiltonian.
Revised: Note that unlike HGN, where the Hamiltonian dynamics are shared across time steps (a single Hamiltonian is learned and its parameters are shared across time steps of a rollout), in NHF each step of the ﬂow (corresponding to a single time step of a rollout) can be parametrised by a different Hamiltonian.

Original: The inverse of such a Hamiltonian ﬂow can be easily obtained by replacing dt by − dt in the Hamiltonian equations and reversing the order of the transformations, s 

Revised: The inverse of such a Hamiltonian ﬂow can be easily obtained by replacing dt by − dt in the Hamiltonian equations and reversing the order of the transformations, s

Original: =
Revised: =

Original: H − dt◦ ... ◦
Revised: H − dt◦ ... ◦

Original: H − dt T ( s T ) (we will use the appropriate dt or − dt superscript from now on to make the direction of integration of the Hamiltonian dynamics more explicit).
Revised: H − dt T ( s T ) (we will use the appropriate dt or − dt superscript from now on to make the direction of integration of the Hamiltonian dynamics more explicit).

Original: The resulting density p ( s T ) is given by the following equation:
Revised: The resulting density p ( s T ) is given by the following equation:

Original: [Equation]
Revised: [Equation]

Original: Our proposed NHF  modelthen constitutes a more structured form of a Neural ODE ﬂow (Chen et al., 2018). The few notable differences are the following: (i) The Hamiltonian ODE is volume-preserving, which makes the computation of log-likelihood cheaper than for a general ODE ﬂow.
Revised: Our proposed NHF is more computationally efﬁcient that many other ﬂow-based approaches, because it does not require the expensive step of calculating the trace of the Jacobian. Hence, the NHF model constitutes a more structured form of a Neural ODE ﬂow (Chen et al., 2018), but with a few notable differences: (i) The Hamiltonian ODE is volume-preserving, which makes the computation of loglikelihood cheaper than for a general ODE ﬂow.

Original: (ii) General ODE ﬂows are only invertible in the limit dt → 0 , whereas for some Hamiltonians we can use more complex integrators (like the symplectic leapfrog integrator described in Sec.
Revised: (ii) General ODE ﬂows are only invertible in the limit dt → 0 , whereas for some Hamiltonians we can use more complex integrators (like the symplectic leapfrog integrator described in Sec.

Original: A.5) that are both invertible and volume-preserving for any dt> 0 .
Revised: A.6) that are both invertible and volume-preserving for any dt> 0 .

Original: The structure s =( q , p ) on the state-space imposed by the Hamiltonian dynamics can be constraining from the point of view of density estimation.
Revised: The structure s =( q , p ) on the state-space imposed by the Hamiltonian dynamics can be constraining from the point of view of density estimation.

Original: We choose to use the trick proposed in the Hamiltonian
Revised: We choose to use the trick proposed in the Hamiltonian

Original: Monte Carlo (HMC) literature (Neal et al., 2011; Salimans et al.; Levy et al., 2017), which treats the momentum p as a latent variable (see Fig. 4).
Revised: Monte Carlo (HMC) literature (Neal et al., 2011; Salimans et al., 2015; Levy et al., 2017), which treats the momentum p as a latent variable (see Fig. 4).

Original: This is an elegant solution which avoids having to artiﬁcially split the density into two disjoint sets.
Revised: This is an elegant solution which avoids having to artiﬁcially split the density into two disjoint sets.

Original: As a result, the data density that our Hamiltonian ﬂows are modelling becomes exclusively parametrised by p ( q T ) , which takes the following form:
Revised: As a result, the data density that our Hamiltonian ﬂows are modelling becomes exclusively parametrised by p ( q T ) , which takes the following form:

Original: [Equation]
Revised: [Equation]

Original: ◦ .
Revised: p T .

Original: [Equation]
Revised: This integral is intractable, but the Pendulum Mass-spring

Original: [Equation] T .
Revised: T r a i n M S E

Original: This integral is intractable, but the model can still be trained via variational methods where we introduce a variational density f ψ ( p T | q T ) with parameters ψ and instead optimise the following ELBO:
Revised: T e s t M S E T r a i n M S E

Revised: T e s t M

Revised: S E

Revised: Two-Body

Revised: Three-Body

Original: [Equation]
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: model can still be trained via variational methods where we introduce a variational density f ψ ( p T | q T ) with parameters ψ and instead optimise the following ELBO:

Original:  Note that, in contrast to VAEs (Kingma & Welling, 2014; Rezende et al., 2014), the ELBO in Eq.
Revised: [Equation] Note that, in contrast to VAEs (Kingma & Welling, 2014; Rezende et al., 2014), the ELBO in Eq.

Original: 4 is not explicitly in the form of a reconstruction error term plus a KL term.
Revised: 6 is not explicitly in the form of a reconstruction error term plus a KL term.

Original:  R ESULTS
Revised:  R ESULTS

Original: In order to directly compare the performance of HGN to that of its closest baseline, HNN, we generated four datasets analogous to the data used in Greydanus et al.
Revised: In order to directly compare the performance of HGN to that of its closest baseline, HNN, we generated four datasets analogous to the data used in Greydanus et al.

Original: The datasets contained observations of the time evolution of four physical systems: mass-spring, pendulum, two- and three-body (see
Revised: The datasets contained observations of the time evolution of four physical systems: mass-spring, pendulum, two- and three-body (see

Original: Fig. 5).
Revised: Fig. 5).

Original: In order to generate each trajectory, we ﬁrst randomly sampled an initial state, then produced a 30 step rollout following the ground truth Hamiltonian dynamics, before adding Gaussian noise with standard deviation σ 2 =0 .
Revised: In order to generate each trajectory, we ﬁrst randomly sampled an initial state, then produced a 30 step rollout following the ground truth Hamiltonian dynamics, before adding Gaussian noise with standard deviation σ 2 =0 .

Original: 1 to each phase-space coordinate, and rendering a corresponding 64x64 pixel observation.
Revised: 1 to each phase-space coordinate, and rendering a corresponding 64x64 pixel observation.

Original: We generated 50 000 train and 10 000 test trajectories for each dataset.
Revised: We generated 50 000 train and 10 000 test trajectories for each dataset.

Original: When sampling initial states, we start by ﬁrst sampling the total energy of the system denoted as a radius r in the phase space, before sampling the initial state ( q,p ) uniformly on the circle of radius r .
Revised: When sampling initial states, we start by ﬁrst sampling the total energy of the system denoted as a radius r in the phase space, before sampling the initial state ( q,p ) uniformly on the circle of radius r .

Original: Note that our pendulum dataset is more challenging than the one described in Greydanus et al.
Revised: Note that our pendulum dataset is more challenging than the one described in Greydanus et al.

Original: (2019), where the pendulum had a ﬁxed radius and was initialized at a maximum angle of 30 ◦ from the central axis.
Revised: (2019), where the pendulum had a ﬁxed radius and was initialized at a maximum angle of 30 ◦ from the central axis.

Original: Mass-spring.
Revised: Mass-spring.

Original: The dynamics of a frictionless mass-spring system are modeled by the Hamiltonian
Revised: The dynamics of a frictionless mass-spring system are modeled by the Hamiltonian

Original: H = 12 kq 2 + p 2 2 m , where k is the spring constant and m is the mass.
Revised: H = 12 kq 2 + p 2 2 m , where k is the spring constant and m is the mass.

Original: We ﬁx k = 2 and m = 0 .
Revised: We ﬁx k = 2 and m = 0 .

Original: 5 , then sample a radius from a uniform distribution r ∼ U (0 . 1 , 1 . 0) .
Revised: 5 , then sample a radius from a uniform distribution r ∼ U (0 . 1 , 1 . 0) .

Original: Pendulum.
Revised: Pendulum.

Original: The dynamics of a frictionless pendulum are modeled by the Hamiltonian H =2 mgl (1 − cos ( q ))
Revised: The dynamics of a frictionless pendulum are modeled by the Hamiltonian H =2 mgl (1 − cos ( q ))

Original: + p 2 2 ml 2 , where g is the gravitational constant and l is the length of the pendulum.
Revised: + p 2 2 ml 2 , where g is the gravitational constant and l is the length of the pendulum.

Original: We ﬁx g =3 , m =0 .
Revised: We ﬁx g =3 , m =0 .

Original: 5 , l =1 , then sample a radius from a uniform distribution r ∼ U (1 . 3 , 2 . 3) .
Revised: 5 , l =1 , then sample a radius from a uniform distribution r ∼ U (1 . 3 , 2 . 3) .

Original: Two- and three- body problems.
Revised: Two- and three- body problems.

Original: In an n-body problem, particles interact with each other through an attractive force, like gravity.
Revised: In an n-body problem, particles interact with each other through an attractive force, like gravity.

Original: The dynamics are represented by the following Hamiltonian H 
Revised: The dynamics are represented by the following Hamiltonian H =

Original: = (cid:80) n i || p i ||
Revised: [Table]

Original: m i − (cid:80) 1 ≤ i<j ≤ n gm i m j || q j − q

Original: i || .
Revised: || .

Original: We set m =1 and g =1 for both systems.
Revised: We set m =1 and g =1 for both systems.

Original: For the two-body problem, we set r ∼ U (0 . 5 , 1 . 5) , and we also change the observation noise to σ 2 = 0 . 05 .
Revised: For the two-body problem, we set r ∼ U (0 . 5 , 1 . 5) , and we also change the observation noise to σ 2 = 0 . 05 .

Original: For the three-body problem, we set r ∼ U (0 . 9 , 1 . 2) , and set the observation noise to σ 2 =0 .
Revised: For the three-body problem, we set r ∼ U (0 . 9 , 1 . 2) , and set the observation noise to σ 2 =0 .

Original: Learning the Hamiltonian We tested whether HGN and the HNN baseline could learn the dynamics of the four systems described above.
Revised: Learning the Hamiltonian We tested whether HGN and the HNN baseline could learn the dynamics of the four systems described above.

Original: To ensure that our re-implementation of HNN was correct, we replicated all the results presented in the original paper (Greydanus et al., 2019) by verifying that it could learn the dynamics of the mass-spring, pendulum and two-body systems well from the ground truth state, and the dynamics of a restricted pendulum from pixels.
Revised: To ensure that our re-implementation of HNN was correct, we replicated all the results presented in the original paper (Greydanus et al., 2019) by verifying that it could learn the dynamics of the mass-spring, pendulum and two-body systems well from the ground truth state, and the dynamics of a restricted pendulum from pixels.

Original: We also compared different modiﬁcations of HGN: a version trained and tested with an Euler rather than a leapfrog integrator (HGN Euler), a version trained with no additional function between the posterior and the prior (HGN no f ψ ), and a deterministic version (HGN determ).
Revised: We also compared different modiﬁcations of HGN: a version trained and tested with an Euler rather than a leapfrog integrator (HGN Euler), a version trained with no additional function between the posterior and the prior (HGN no f ψ ) and a deterministic version (HGN determ), which did not include the sampling step from the posterior q φ ( z | x 0 ... x T ) . [Table]

Original: Tbl. 1 and Fig.
Revised: Tbl. 1 and Fig.

Original: 6 demonstrate that HGN and its modiﬁcations learned well on all four datasets.
Revised: 6 demonstrate that HGN and its modiﬁcations learned well on all four datasets.

Original: However, when we attempted to train HNN on the four datasets described above, its Hamiltonian often collapsed to 0 and the model failed to reproduce any dynamics, defaulting to a static single image.
Revised: However, when we attempted to train HNN on the four datasets described above, its Hamiltonian often collapsed to 0 and the model failed to reproduce any dynamics, defaulting to a static single image.

Original: We were unable to improve on this performance despite our best efforts, including a modiﬁcation of the architecture to closely match ours (referred to as HNN conv) (see Section A
Revised: We were unable to improve on this performance despite our best efforts, including a modiﬁcation of the architecture to closely match ours (referred to as HNN Conv) (see Sec.

Original: 3 of  details).
Revised: A.3 of the appendix for details).

Original: Tbl.
Revised: Tbl.

Original: T r a i n M S

Original: E T e s t M S E [Figure] shows that the average mean squared error (MSE) of the pixel reconstructions on both the train and test data is an order of magnitude better for HGN compared to both versions of HNN.
Revised: 1 shows that the average mean squared error (MSE) of the pixel reconstructions on both the train and test data is an order of magnitude better for HGN compared to both versions of HNN.

Original: The same holds when visualising the average per-frame MSE of a single train and test rollout for each dataset shown in Fig.
Revised: The same holds when visualising the average per-frame MSE of a single train and test rollout for each dataset shown in Fig.

Original: Note that the different versions of HGN have different trade-offs.
Revised: Note that the different versions of HGN have different trade-offs.

Original: The deterministic version produces more accurate reconstructions, however it does not allow sampling.
Revised: The deterministic version produces more accurate reconstructions but it does not allow sampling.

Original: This effect is equivalent to a similar distinction between autoencoders (Hinton & Salakhutdinov, 2006) and VAEs.
Revised: This effect is equivalent to a similar distinction between autoencoders (Hinton & Salakhutdinov, 2006) and VAEs.

Original: Using the simpler Euler integrator rather than the more involved leapfrog one might be conceptually more appealing, however it does not provide the same energy conservation and reversibility properties as the leapfrog integrator, as evidenced by the increase by an order of magnitude of the variance of the learned Hamiltonian throughout a sequence rollout as shown in Fig. 6).
Revised: Using the simpler Euler integrator rather than the more involved leapfrog one might be conceptually more appealing, however it does not provide the same energy conservation and reversibility properties as the leapfrog integrator, as evidenced by the increase by an order of magnitude of the variance of the learned Hamiltonian throughout a sequence rollout as shown in Tbl.

Original: The full version of HGN, on the other hand, is capable of reproducing the dynamics well, is capable of producing diverse yet plausible rollout samples (Fig.
Revised: The full version of HGN, on the other hand, is capable of reproducing the dynamics well, is capable of producing diverse yet plausible rollout samples (Fig.

Original: 8), and its rollouts can be reversed it time, sped up or slowed down by either changing the value or the sign of dt used in the integrator (Figs. 7).[Figure] [Table]
Revised: 8) and its rollouts can be reversed in time, sped up or slowed down by either changing the value or the sign of dt used in the integrator (Fig. 7).

Original: an Euler rather than a leapfrog integrator, a deterministic rather than a generative version, and a version of HGN with no extra network between the posterior and the initial state.

Original: Expressive density modelling using learned Hamiltonian ﬂows We evaluate whether NHF is capable of expressive density modelling by stacking learned Hamiltonians into a series of normalising ﬂows.
Revised: Expressive density modelling using learned Hamiltonian ﬂows We evaluate whether NHF is capable of expressive density modelling by stacking learned Hamiltonians into a series of normalising ﬂows.

Original: Fig.
Revised: Fig.

Original: 9 demonstrates that NHF can transform a simple soft-uniform prior distribution
Revised: 9 demonstrates that NHF can transform a simple soft-uniform prior distribution

Original: π ( s 0 ; σ,β ) into signiﬁcantly more complex densities with arbitrary numbers of modes.
Revised: π ( s 0 ; σ,β ) into signiﬁcantly more complex densities with arbitrary numbers of modes.

Original: The soft-uniform density, π ( s 0 ; σ,β ) ∝ f ( β ( s + σ 1 2 ))
Revised: The soft-uniform density, π ( s 0 ; σ,β ) ∝ f ( β ( s + σ 12 ))

Original: f ( − β ( s − σ 1 2 )) , where f is the sigmoid function, was chosen to make it easier to visualise the learned attractors.
Revised: f ( − β ( s − σ 12 )) , where f is the sigmoid function and β is a constant, was chosen to make it easier to visualise the learned attractors.

Original: The model also performed well with other priors, including a Normal distribution.
Revised: The model also performed well with other priors, including a Normal distribution.

Original: It is interesting to note that the trained model is well interpretable.
Revised: It is interesting to note that the trained model is very interpretable.

Original: When decomposed into the equivalents of the kinetic and potential energies, it can beseen that the learned potential energy V ( q ) learned to have several local minima, one for each mode of the data. As a consequence, the trajectory of the initial samples through the ﬂow has attractors at the modes of the data. [Figure]
Revised: When decomposed into the equivalents of the kinetic and potential energies, it can be

Original: [Figure]
Revised: [Table]

Original: S a

Original: [Figure]
Revised: [Figure]

Original: S
Revised: HNN. HNN did not learn the dynamics of the system and instead learned to reconstruct an average image.

Original: [Figure]
Revised: [Figure]

Original: p l
Revised: t t t t

Original: [Figure]

Original: S a m [Figure] e
Revised: S a m p l e S a m p l e

Original: [Figure] S [Figure]
Revised: S a m p l e S a m p l e

Original:  e
Revised: S a m p l e

Original: S a m p l e S a m p l e
Revised: S a m p l e S a m p l e

Original: [Figure]
Revised: S a m p l e seen that the learned potential energy V ( q ) learned to have several local minima, one for each mode of the data. As a consequence, the trajectory of the initial samples through the ﬂow has attractors at the modes of the data.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have presented HGN, the ﬁrst deep learning approach capable of reliably learning Hamiltonian dynamics from pixel observations.
Revised: We have presented HGN, the ﬁrst deep learning approach capable of reliably learning Hamiltonian dynamics from pixel observations.

Original: We have evaluated our approach on four classical physical systems  and demonstrated that it outperformed the only relevant baseline by a large margin.
Revised: We have evaluated our approach on four classical physical systems [Figure] and demonstrated that it outperformed the only relevant baseline by a large margin.

Original: Hamiltonian dynamics have a number of useful properties that can be exploited more widely by the machine learning community.
Revised: Hamiltonian dynamics have a number of useful properties that can be exploited more widely by the machine learning community.

Original: For example, the Hamiltonian induces a smooth manifold and a vector ﬁeld in the abstract phase space along which certain physical quantities are conserved.
Revised: For example, the Hamiltonian induces a smooth manifold and a vector ﬁeld in the abstract phase space along which certain physical quantities are conserved.

Original: The time evolution along these paths is also completely reversible.
Revised: The time evolution along these paths is also completely reversible.

Original: These properties can have wide implications in such areas of machine learning as reinforcement learning, representation learning and generative modelling.
Revised: These properties can have wide implications in such areas of machine learning as reinforcement learning, representation learning and generative modelling.

Original: We have demonstrated the ﬁrst step towards applying the learnt Hamiltonian dynamics as normalising ﬂows for more expressive yet computationally efﬁcient density modelling.
Revised: We have demonstrated the ﬁrst step towards applying the learnt Hamiltonian dynamics as normalising ﬂows for more expressive yet computationally efﬁcient density modelling.

Original: We hope that this work serves as the ﬁrst step towards a wider adoption of the rich body of physics literature around the Hamiltonian principles in the machine learning community.
Revised: We hope that this work serves as the ﬁrst step towards a wider adoption of the rich body of physics literature around the Hamiltonian principles in the machine learning community.

Original: A S UPPLEMENTARY M ATERIALS A. H AMILTONIAN G ENERATIVE N ETWORK
Revised: A S UPPLEMENTARY M ATERIALS A. H AMILTONIAN G ENERATIVE N ETWORK

Original: The Hamiltonian Generative Network (HGN) consists of three major parts, an encoder, the Hamiltonian transition network and a decoder.
Revised: The Hamiltonian Generative Network (HGN) consists of three major parts, an encoder, the Hamiltonian transition network and a decoder.

Original: During training the encoder starts with a sequence
Revised: During training the encoder starts with a sequence

Original: [Equation]
Revised: [Equation]

Original: of raw training images and encodes it into a probabilistic prior representation transformed with an additional network on top
Revised: of raw training images and encodes it into a probabilistic prior representation transformed with an additional network on top

Original: [Equation]
Revised: [Equation]

Original: into a start state
Revised: into a start state

Original: [Equation]
Revised: [Equation]

Original: consisting of a downsized spatial representation in latent space ( 4 × 4 ), where each abstract pixel is the the concatenation of abstract position ( q ) and momentum ( p ) (each of dimension 16).
Revised: consisting of a downsized spatial representation in latent space ( 4 × 4 ), where each abstract pixel is the the concatenation of abstract position ( q ) and momentum ( p ) (each of dimension 16).

Original: The encoder network is a convolutional neural network with 8 layers, with 32 ﬁlters on the ﬁrst layer, then 64 ﬁlters on each subsequent layer, while in the last layer we have 48 ﬁlters.
Revised: The encoder network is a convolutional neural network with 8 layers, with 32 ﬁlters on the ﬁrst layer, then 64 ﬁlters on each subsequent layer, while in the last layer we have 48 ﬁlters.

Original: The ﬁnal encoder transformer network is a convolutional neural network with 3 layers and 64 ﬁlters on each layer.
Revised: The ﬁnal encoder transformer network is a convolutional neural network with 3 layers and 64 ﬁlters on each layer.

Original: Starting from this initial embedded state, the Hamiltonian transition network generates subsequent states using a symplectic integrator approximating the Hamiltonian equations.
Revised: Starting from this initial embedded state, the Hamiltonian transition network generates subsequent states using a symplectic integrator approximating the Hamiltonian equations.

Original: The Hamiltonian transition network represents the Hamiltonian function
Revised: The Hamiltonian transition network represents the Hamiltonian function

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: as a function from the abstract position and momentum space to the real numbers at any time step t .
Revised: as a function from the abstract position and momentum space to the real numbers at any time step t .

Original: The Hamiltonian transition network is a convolutional neural network of 6 layers, each consisting ofﬁlters.
Revised: The Hamiltonian transition network is a convolutional neural network of 6 layers, each consisting ofﬁlters.

Original: The discrete timestep we use for the symplectic integrator update step is dt =0 .
Revised: The discrete timestep we use for the symplectic integrator update step is dt =0 .

Original: At each time step t the decoder network d θ takes only the abstract position part q t of the state s t and decodes it back to an output image (cid:101) x
Revised: At each time step t the decoder network d θ takes only the abstract position part q t of the state s t and decodes it back to an output image (cid:101) x

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: of the same shape as the input images.
Revised: of the same shape as the input images.

Original: [Equation]
Revised: [Equation]

Original: The decoder network is a progressive network consisting of 3 residual blocks, where each residual block resizes the current input image by a factor of 2 using the nearest neighbor method (at the end we have to upscale our latent spatial dimension of 4 to the desired output image dimension of 32 in these steps), followed by 2 blocks of a one layer convolutional neural network with 64 ﬁlters and a leaky
Revised: The decoder network is a progressive network consisting of 3 residual blocks, where each residual block resizes the current input image by a factor of 2 using the nearest neighbor method (at the end we have to upscale our latent spatial dimension of 4 to the desired output image dimension of 32 in these steps), followed by 2 blocks of a one layer convolutional neural network with 64 ﬁlters and a leaky

Original: ReLU activation function, closing by a sigmoid activation in each block.
Revised: ReLU activation function, closing by a sigmoid activation in each block.

Original: After the 3 blocks a ﬁnal one layer convolutional neural network outputs the output image with the right number of channels.
Revised: After the 3 blocks a ﬁnal one layer convolutional neural network outputs the output image with the right number of channels.

Original: We use Adam optimisier (Kingma & Ba, 2014) with learning rate 1.5e-4.
Revised: We use Adam optimisier (Kingma & Ba, 2014) with learning rate 1.5e-4.

Original: When optimising the loss, in practice we do not learn the variance of the decoder p θ ( x | s ) and ﬁx it to 1, which makes the reconstruction objective equivalent to a scaled L2 loss.
Revised: When optimising the loss, in practice we do not learn the variance of the decoder p θ ( x | s ) and ﬁx it to 1, which makes the reconstruction objective equivalent to a scaled L2 loss.

Original: Furthermore, we introduce a Lagrange multiplier in front of the KL term and optimise it using the same method as in Rezende & Viola.
Revised: Furthermore, we introduce a Lagrange multiplier in front of the KL term and optimise it using the same method as in Rezende & Viola (2018).

Original: A. N EURAL H AMILTONIAN F LOW
Revised: A. N EURAL H AMILTONIAN F LOW

Original: For all NHF experiments the Hamiltonian was of the form H ( q,p ) =
Revised: For all NHF experiments the Hamiltonian was of the form H ( q,p ) =

Original: K ( p ) + V ( q ) .
Revised: K ( p ) + V ( q ) .

Original: The kinetic energy term K and the potential energy term V are soft-plus MLPs with layer-sizes [ d, 128 , 128 , 1] where d is the dimension of the data.
Revised: The kinetic energy term K and the potential energy term V are soft-plus MLPs with layer-sizes [ d, 128 , 128 , 1] where d is the dimension of the data.

Original: Soft-plus non-linearities were chosen because the optimisation of Hamiltonian ﬂows involves second-order derivatives of the MLPs used for parametrising the
Revised: Soft-plus non-linearities were chosen because the optimisation of Hamiltonian ﬂows involves second-order derivatives of the MLPs used for parametrising the

Original: Hamiltonians.
Revised: Hamiltonians.

Original: This makes ReLU non-linearities unsuitable.
Revised: This makes ReLU non-linearities unsuitable.

Original: The encoder network was parametrized as f ψ ( p | q )=
Revised: The encoder network was parametrized as f ψ ( p | q )=

Original: N ( p ; µ ( q ) ,σ ( q )) , where µ and σ are ReLU MLPs with size [ d, 128 , 128 ,d ] .
Revised: N ( p ; µ ( q ) ,σ ( q )) , where µ and σ are ReLU MLPs with size [ d, 128 , 128 ,d ] .

Original: The Hamiltonian ﬂow H dt , was approximated using a leapfrog integrator (Neal et al., 2011) since it preserves volume and is invertible for any dt (see also section A.5).
Revised: The Hamiltonian ﬂow H dt , was approximated using a leapfrog integrator (Neal et al., 2011) since it preserves volume and is invertible for any dt (see also section A.6).

Original: We found that only two leapfrog steps where sufﬁcient for all our examples.
Revised: We found that only two leapfrog steps where sufﬁcient for all our examples.

Original: Parameters were optimised using Adam (Kingma & Ba, 2014) (learning rate -4) and Lagrange multipliers were optimised using the same method as in Rezende & Viola  kernel density estimate 
Revised: Parameters were optimised using Adam (Kingma & Ba, 2014) (learning rate 3e-4) and Lagrange multipliers were optimised using the same method as in Rezende & Viola (2018). All shown kernel density estimate (KDE) plots used 1000 samples and isotropic Gaussian kernel bandwidth of 0 .

Original: A. H AMILTONIAN N EURAL N ETWORK
Revised: A. H AMILTONIAN N EURAL N ETWORK

Original: The Hamiltonian Neural Network (HNN) (Greydanus et al., 2019) learns a differentiable function
Revised: The Hamiltonian Neural Network (HNN) (Greydanus et al., 2019) learns a differentiable function

Original: H ( q , p ) that maps a system’s state in phase space (its position q and momentum p ) to a scalar quantity interpreted as the system’s Hamiltonian.
Revised: H ( q , p ) that maps a system’s state in phase space (its position q and momentum p ) to a scalar quantity interpreted as the system’s Hamiltonian.

Original: This model is trained so that H ( q , p ) satisﬁes the Hamiltonian equation by minimizing
Revised: This model is trained so that H ( q , p ) satisﬁes the Hamiltonian equation by minimizing

Original: [Equation]
Revised: [Equation]

Original:  where the derivatives ∂ H ∂ q and ∂ H ∂ p are computed by backpropagation.
Revised: (12) where the derivatives ∂ H ∂ q and ∂ H ∂ p are computed by backpropagation.

Original: In the original paper, these targets are either assumed to be known or are estimated by ﬁnite differences using the state at times t and t +1 .
Revised: In the original paper, these targets are either assumed to be known or are estimated by ﬁnite differences using the state at times t and t +1 .

Original: Accordingly, in the majority of the experiments presented by the authors, the Hamiltonian was learned from the ground truth state space directly, rather than from pixel observations.
Revised: Accordingly, in the majority of the experiments presented by the authors, the Hamiltonian was learned from the ground truth state space directly, rather than from pixel observations.

Original: The original HNN model is trained in a supervised fashion on the ground truth state of a physical system and its time derivatives.
Revised: The original HNN model is trained in a supervised fashion on the ground truth state of a physical system and its time derivatives.

Original: As such, it is not directly comparable to our method, which learns a
Revised: As such, it is not directly comparable to our method, which learns a

Original: Hamiltonian directly from pixels.
Revised: Hamiltonian directly from pixels.

Original: Instead, we compare to the PixelHNN variant of the HNN, which is introduced in the same paper, and which is able to learn a Hamiltonian from images and in the absence of true state or time derivative in some settings.
Revised: Instead, we compare to the PixelHNN variant of the HNN, which is introduced in the same paper, and which is able to learn a Hamiltonian from images and in the absence of true state or time derivative in some settings.

Original: This required a modiﬁcation of the model.
Revised: This required a modiﬁcation of the model.

Original: First, the input to the model became a concatenated, ﬂattened pair of images X t = ( x t , x t +1 ) , which was then mapped to a low-dimensional embedding space z t =( q t , p t ) using an encoder neural network.
Revised: First, the input to the model became a concatenated, ﬂattened pair of images X t = ( x t , x t +1 ) , which was then mapped to a low-dimensional embedding space z t =( q t , p t ) using an encoder neural network.

Original: Note that the dimensionality of this embedding ( z ∈ Rin the case of the pendulum system presented in the paper) is chosen to perfectly match the ground truth dimensionality of the phase space, which was assumed to be known a priori.
Revised: Note that the dimensionality of this embedding ( z ∈ Rin the case of the pendulum system presented in the paper) is chosen to perfectly match the ground truth dimensionality of the phase space, which was assumed to be known a priori.

Original: This, however, is not always possible, as when a system has not yet been identiﬁed.
Revised: This, however, is not always possible, as when a system has not yet been identiﬁed.

Original: The latent embedding was then treated as an estimate of the position and the momentum of the system depicted in the images, where the momentum was assumed to be equal to the velocity of the system – an assumption enforced by the additional constraint found necessary to encourage learning, which encouraged the time derivative of the position latent to equal the momentum latent using ﬁnite differences on the split latents:
Revised: The latent embedding was then treated as an estimate of the position and the momentum of the system depicted in the images, where the momentum was assumed to be equal to the velocity of the system – an assumption enforced by the additional constraint found necessary to encourage learning, which encouraged the time derivative of the position latent to equal the momentum latent using ﬁnite differences on the split latents:

Original: [Equation]
Revised: [Equation]

Original: This loss is motivated by the observation that in simple Newtonian systems with unit mass, the system’s state is fully described by the position and its time derivative (the system’s velocity).
Revised: This loss is motivated by the observation that in simple Newtonian systems with unit mass, the system’s state is fully described by the position and its time derivative (the system’s velocity).

Original: An image embedding that corresponds to the position and velocity of the system will minimize this loss.
Revised: An image embedding that corresponds to the position and velocity of the system will minimize this loss.

Original: This assumption is appropriate in the case of the simple pendulum system presented in the paper, however it does not hold more generally.
Revised: This assumption is appropriate in the case of the simple pendulum system presented in the paper, however it does not hold more generally.

Original: As mentioned earlier, PixelHNN takes as input a concatenated, ﬂattened pair of images and maps them to an embedding space z t =( q t , p t ) , which is treated as an estimate of the position and momentum of the system depicted in the images.
Revised: As mentioned earlier, PixelHNN takes as input a concatenated, ﬂattened pair of images and maps them to an embedding space z t =( q t , p t ) , which is treated as an estimate of the position and momentum of the system depicted in the images.

Original: Note that X t always consists of two images in order to make the momentum observable.
Revised: Note that X t always consists of two images in order to make the momentum observable.

Original: This embedding space is used as the input to an HNN, which is trained to learn the Hamiltonian of the system as before, but using the embedding instead of the true system state.
Revised: This embedding space is used as the input to an HNN, which is trained to learn the Hamiltonian of the system as before, but using the embedding instead of the true system state.

Original: To enable stable learning in this conﬁguration, the PixelHNN uses a standard mean-squared error autoencoding loss:
Revised: To enable stable learning in this conﬁguration, the PixelHNN uses a standard mean-squared error autoencoding loss:

Original: [Equation]
Revised: [Equation]

Original: where ˆ X t is the autoencoder output and X it is the value of pixel i of N total pixels in X t .
Revised: where ˆ X t is the autoencoder output and X it is the value of pixel i of N total pixels in X t .

Original: This loss encourages the network embedding to reﬂect the content of the input images and to avoid the trivial solution to (13).
Revised: This loss encourages the network embedding to reﬂect the content of the input images and to avoid the trivial solution to (12).

Original: The full PixelHNN loss is:
Revised: The full PixelHNN loss is:

Original: [Equation]
Revised: [Equation]

Original: 13) where L HNN is computed using the ﬁnite difference estimate of the time derivative of the embedding.
Revised: where L HNN is computed using the ﬁnite difference estimate of the time derivative of the embedding.

Original: λ
Revised: λ

Original: HNN is a Lagrange multiplier, which is set to 0.1, as in the original paper.
Revised: HNN is a Lagrange multiplier, which is set to 0.1, as in the original paper.

Original: L WD is a standard L2 weight decay and its Lagrange multiplier λ
Revised: L WD is a standard L2 weight decay and its Lagrange multiplier λ

Original: WD is set to 1e-5, as in the original paper.
Revised: WD is set to 1e-5, as in the original paper.

Original: In the experiments presented here, we reimplemented the PixelHNN architecture as described in
Revised: In the experiments presented here, we reimplemented the PixelHNN architecture as described in

Original: Greydanus et al.
Revised: Greydanus et al.

Original: and trained it using the full loss (13).
Revised: and trained it using the full loss (15).

Original: As in the original paper, we used a PixelHNN with HNN, encoder, and decoder subnetworks each parameterized by a multi-layer perceptron (MLP).
Revised: As in the original paper, we used a PixelHNN with HNN, encoder, and decoder subnetworks each parameterized by a multi-layer perceptron (MLP).

Original: The encoder and decoder MLPs use ReLU nonlinearities.
Revised: The encoder and decoder MLPs use ReLU nonlinearities.

Original: Each consists oflayers, with 200 units in each hidden layer and an embedding of the same size as the true position and momentum of the system depicted (2 for mass-spring and pendulum, 8 for two-body, and 12 for 3-body).
Revised: Each consists oflayers, with 200 units in each hidden layer and an embedding of the same size as the true position and momentum of the system depicted (2 for mass-spring and pendulum, 8 for two-body, and 12 for 3-body).

Original: The HNN MLP uses tanh nonlinearities and consists of two hidden layers with 200 units and a one-dimensional output.
Revised: The HNN MLP uses tanh nonlinearities and consists of two hidden layers with 200 units and a one-dimensional output.

Original: To ensure the difference in performance between the PixelHNN and HGN are not due primarily to archiectural choices, we also compare to a variant of the PixelHNN architecture using the same convolutional encoder and decoder as used in HGN.
Revised: To ensure the difference in performance between the PixelHNN and HGN are not due primarily to archiectural choices, we also compare to a variant of the PixelHNN architecture using the same convolutional encoder and decoder as used in HGN.

Original: We used identical hyperparameters to those described in section A.1.
Revised: We used identical hyperparameters to those described in section A.1.

Original: We map between the convolutional latent space used by the encoder and decoder and the vector-valued latent required by the HNN using one additional linear layer for the encoder and decoder.
Revised: We map between the convolutional latent space used by the encoder and decoder and the vector-valued latent required by the HNN using one additional linear layer for the encoder and decoder.

Original: In the original paper, the PixelHNN model is trained using full-batch gradient descent.
Revised: In the original paper, the PixelHNN model is trained using full-batch gradient descent.

Original: To make it more comparable to our approach, we train it here using stochastic gradient descent using minibatches of size 64 and around 15000 training steps.
Revised: To make it more comparable to our approach, we train it here using stochastic gradient descent using minibatches of size 64 and around 15000 training steps.

Original: As in the original paper, we train the model using the Adam optimizer (Kingma & Ba, 2014) and a learning rate of 1e-3.
Revised: As in the original paper, we train the model using the Adam optimizer (Kingma & Ba, 2014) and a learning rate of 1e-3.

Original: As in the original paper, we produce rollouts of the model using a Runge-Kutta integrator (RK4).
Revised: As in the original paper, we produce rollouts of the model using a Runge-Kutta integrator (RK4).

Original: See Section A.5 for a description of RK4.
Revised: See Section A.6 for a description of RK4.

Original: Note that, as in the original paper, we use the more sophisticated algorithm implemented in scipy ( scipy.integrate.solve_ivp ) (Jones et al., 2001).
Revised: Note that, as in the original paper, we use the more sophisticated algorithm implemented in scipy ( scipy.integrate.solve_ivp ) (Jones et al., 2001).

Original: A. D ATASETS
Revised: A. D ATASETS

Original: The datasets for the experiments described in 3 were generated in a similar manner to Greydanus et al.
Revised: The datasets for the experiments described in 4 were generated in a similar manner to Greydanus et al.

Original: (2019) for comparative purposes.
Revised: (2019) for comparative purposes.

Original: All of the datasets simulate the exact Hamiltonian dynamics of the underlying differential equation using the default scipy initial value problem solver Jones et al.
Revised: All of the datasets simulate the exact Hamiltonian dynamics of the underlying differential equation using the default scipy initial value problem solver Jones et al.

Original: After creating a dataset of trajectories for each system, we render those into a sequence of images.
Revised: After creating a dataset of trajectories for each system, we render those into a sequence of images.

Original: The system depicted in each dataset can be visualized by rendering circular objects:
Revised: The system depicted in each dataset can be visualized by rendering circular objects:

Original: • For the mass-spring the mass object is rendered as a circle and the spring and pivot are invisible.
Revised: • For the mass-spring the mass object is rendered as a circle and the spring and pivot are invisible.

Original: • For the pendulum only the weight (the bob) is rendered as a circle, while the massless rod and pivot are invisible.
Revised: • For the pendulum only the weight (the bob) is rendered as a circle, while the massless rod and pivot are invisible.

Original: • For the two and three body problem we render each point mass as a circle in a different color.
Revised: • For the two and three body problem we render each point mass as a circle in a different color.

Original: Additionally, we smooth out the circles such that they do not have hard edges, as can be seen in Fig.
Revised: Additionally, we smooth out the circles such that they do not have hard edges, as can be seen in Fig.

Revised: A. C ONVERGENCE OF HGN AND HNN

Revised: In order to obtain the results presented in Tbl.

Revised: 1 we trained both HGN and HNN for 15000 iterations, with batch size of 16 for HGN, and 64 for the HNN.

Revised: Given the dataset sizes, this means that HGN was trained for around 5 epochs and HNN was trained for around 19 epochs, which took around 16 hours.

Original:  A. I NTEGRATORS
Revised: Figs. 10-11 plot the convergence rates for HGN (leapfrog) and HNN (conv) on the four datasets. [Figure] A. I NTEGRATORS

Original: Throughout this paper, we estimate the future state of systems from inferred values of the system position and momentum by numerically integrated the Hamiltonian.
Revised: Throughout this paper, we estimate the future state of systems from inferred values of the system position and momentum by numerically integrated the Hamiltonian.

Original: We explore three methods of numerical integration: (i) Euler integration, (ii) Runge-Kutta integration and (iii) leapfrog integration.
Revised: We explore three methods of numerical integration: (i) Euler integration, (ii) Runge-Kutta integration and (iii) leapfrog integration.

Original: Euler integration estimates the value of a function at time t + dt by incrementing the function’s value with the value accumulated by the function’s derivative, assuming it stays constant in the interval [ t,t + dt ] .
Revised: Euler integration estimates the value of a function at time t + dt by incrementing the function’s value with the value accumulated by the function’s derivative, assuming it stays constant in the interval [ t,t + dt ] .

Original: In the Hamiltonian framework, Euler integration takes the form:
Revised: In the Hamiltonian framework, Euler integration takes the form:

Original: [Figure] [Equation]
Revised: [Equation] [Equation]

Original: Because Euler integration estimates a function’s future value by extrapolating along its ﬁrst derivative, the method ignores the contribution of higher-order derivatives to the function’s change in time.
Revised: Because Euler integration estimates a function’s future value by extrapolating along its ﬁrst derivative, the method ignores the contribution of higher-order derivatives to the function’s change in time.

Original: Accordingly, while Euler integration can reasonably estimate a function’s value over short periods, its errors accumulate rapidly as it is integrated over longer periods or when it is applied multiple times.
Revised: Accordingly, while Euler integration can reasonably estimate a function’s value over short periods, its errors accumulate rapidly as it is integrated over longer periods or when it is applied multiple times.

Original: This limitation motivates the use of methods that are stable over more steps and longer integration times.
Revised: This limitation motivates the use of methods that are stable over more steps and longer integration times.

Original: One such method is four-step Runge-Kutta integration (RK4), the most widely used member of the
Revised: One such method is four-step Runge-Kutta integration (RK4), the most widely used member of the

Original: Runge-Kutta family of integrators.
Revised: Runge-Kutta family of integrators.

Original: Whereas Euler integration estimates the value of a function at time t + dt using only the function’s derivative at time t , RK4 accumulates multiple estimates of the function’s value in the interval [ t,t + dt ] .
Revised: Whereas Euler integration estimates the value of a function at time t + dt using only the function’s derivative at time t , RK4 accumulates multiple estimates of the function’s value in the interval [ t,t + dt ] .

Original: This integral more correctly reﬂects the behavior of the function in the interval, resulting in a more stable estimate of the function’s value.
Revised: This integral more correctly reﬂects the behavior of the function in the interval, resulting in a more stable estimate of the function’s value.

Original: RK4 estimates the state at time t + dt as: 
Revised: RK4 estimates the state at time t + dt as: [Figure]

Original: [Equation]
Revised: [Equation]

Original: (16) where, for compactness, we write the full system state as x t =[ q t , p t ] and the Hamiltonian as H ( x ) .
Revised: where, for compactness, we write the full system state as x t =[ q t , p t ] and the Hamiltonian as H ( x ) .

Original: In the Hamiltonian framework, k 1 , k 2 , k 3 , k 4 are obtained by evaluating the derivative at four points in the interval [ t,t + dt ] :[Equation] [Equation] [Equation]
Revised: In the Hamiltonian framework, k 1 , k 2 , k 3 , k 4 are obtained by evaluating the derivative at four points in the interval [ t,t + dt ] :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: While RK4 may produce reasonably stable estimates over short periods of time, it is not guaranteed to behave stably indeﬁnitely.
Revised: While RK4 may produce reasonably stable estimates over short periods of time, it is not guaranteed to behave stably indeﬁnitely.

Original: Neither RK4 nor Euler integration is guaranteed to preserve the energy of the system being integrated, and in practice both will produce estimates that drift away from the true system dynamics over timescales that are relevant for simulating real systems.
Revised: Neither RK4 nor Euler integration is guaranteed to preserve the energy of the system being integrated, and in practice both will produce estimates that drift away from the true system dynamics over timescales that are relevant for simulating real systems.

Original: Fortunately, there are well-known methods for numerical integration that preserve energy and can be applied to Hamiltonian systems, like the one we propose here.
Revised: Fortunately, there are well-known methods for numerical integration that preserve energy and can be applied to Hamiltonian systems, like the one we propose here.

Original: One such method is leapfrog integration, which is a special method for integrating differential equations of the form: 
Revised: One such method is leapfrog integration, which is a special method for integrating differential equations of the form: [Figure]

Original: [Equation]
Revised: [Equation]

Original: If we assume that the Hamiltonian equations take this form, we can integrate them using the leapfrog integrator, which in essence updates the position and momentum variables at interleaved time points in a way that resembles the updates “leapfrogging” over each other (see Fig.
Revised: If we assume that the Hamiltonian equations take this form, we can integrate them using the leapfrog integrator, which in essence updates the position and momentum variables at interleaved time points in a way that resembles the updates “leapfrogging” over each other (see Fig.

Original: 10B for an illustration).
Revised: 12B for an illustration).

Original: In particular, the following updates can be applied to a Hamiltonian of the form H = V ( q )+ T ( p ) , where V is the potential energy and T is the kinetic energy of the system:
Revised: In particular, the following updates can be applied to a Hamiltonian of the form H = V ( q )+ T ( p ) , where V is the potential energy and T is the kinetic energy of the system:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As discussed above, leapfrog integration is more stable and accurate over long rollouts than integrators like Euler or RK4.
Revised: As discussed above, leapfrog integration is more stable and accurate over long rollouts than integrators like Euler or RK4.

Original: This is because the leapfrog integrator is an example of a symplectic integrator, which means it is guaranteed to preserve the special form of the Hamiltonian even after repeated application.
Revised: This is because the leapfrog integrator is an example of a symplectic integrator, which means it is guaranteed to preserve the special form of the Hamiltonian even after repeated application.

Original: An example visual comparison between a symplectic (leapfrog) and non-symplectic (Euler) integrator applied over the Hamiltonian for a harmonic oscilator is shown in Fig.
Revised: An example visual comparison between a symplectic (leapfrog) and non-symplectic (Euler) integrator applied over the Hamiltonian for a harmonic oscilator is shown in Fig.

Original: 10A. For a more thorough discussion of the properties of leapfrog integration, see (Neal et al., 2011).
Revised: 12A. For a more thorough discussion of the properties of leapfrog integration, see (Neal et al., 2011).

Revised: A. P ROOF OF THE H AMILTONIAN F LOW ELBO

Revised: The Hamiltonian Flow consists of two components.

Revised: Firstly, it deﬁnes a density model over the joint space s T =

Revised: ( q T ,p T ) using the Hamiltonian Flow as described in section 3.3.

Revised: However, we assume that the observable variable represents only q T and treat p T as a latent variable which we have to marginalize over.

Revised: Since the integral is intractable using the introduced variational distribution we can derive the lower bound on the marginal likelihood:

Revised: [Equation]