Original: FINDE: N EURAL D IFFERENTIAL E QUATIONS FOR F INDING AND P RESERVING I NVARIANT Q UANTITIES
Revised: FINDE: N EURAL D IFFERENTIAL E QUATIONS FOR F INDING AND P RESERVING I NVARIANT Q UANTITIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many real-world dynamical systems are associated with first integrals (a.k.a.
Revised: Many real-world dynamical systems are associated with first integrals (a.k.a.

Original: invariant quantities), which are quantities that remain unchanged over time.
Revised: invariant quantities), which are quantities that remain unchanged over time.

Original: The discovery and understanding of first integrals are fundamental and important topics both in the natural sciences and in industrial applications.
Revised: The discovery and understanding of first integrals are fundamental and important topics both in the natural sciences and in industrial applications.

Original: First integrals arise from the conservation laws of system energy, momentum, and mass, and from constraints on states; these are typically related to specific geometric structures of the governing equations.
Revised: First integrals arise from the conservation laws of system energy, momentum, and mass, and from constraints on states; these are typically related to specific geometric structures of the governing equations.

Original: Existing neural networks designed to ensure such first integrals have shown excellent accuracy in modeling from data.
Revised: Existing neural networks designed to ensure such first integrals have shown excellent accuracy in modeling from data.

Original: However, these models incorporate the underlying structures, and in most situations where neural networks learn unknown systems, these structures are also unknown.
Revised: However, these models incorporate the underlying structures, and in most situations where neural networks learn unknown systems, these structures are also unknown.

Original: This limitation needs to be overcome for scientific discovery and modeling of unknown systems.
Revised: This limitation needs to be overcome for scientific discovery and modeling of unknown systems.

Original: To this end, we propose first integral-preserving neural differential equation (FINDE) .
Revised: To this end, we propose first integral-preserving neural differential equation (FINDE) .

Original: By leveraging the projection method and the discrete gradient method,
Revised: By leveraging the projection method and the discrete gradient method,

Original: FINDE finds and preserves first integrals from data, even in the absence of prior knowledge about underlying structures.
Revised: FINDE finds and preserves first integrals from data, even in the absence of prior knowledge about underlying structures.

Original: Experimental results demonstrate that
Revised: Experimental results demonstrate that

Original: FINDE can predict future states of target systems much longer and find various quantities consistent with well-known first integrals in a unified manner.
Revised: FINDE can predict future states of target systems much longer and find various quantities consistent with well-known first integrals in a unified manner.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modeling and predicting real-world systems are fundamental aspects of understanding the world in natural science and improving computer simulations in industry.
Revised: Modeling and predicting real-world systems are fundamental aspects of understanding the world in natural science and improving computer simulations in industry.

Original: Target systems include chemical dynamics for discovering new drugs (Raff et al., 2012), climate dynamics for climate change prediction and weather forecasting (Rasp et al., 2020; Trigo & Palutikof, 1999), and physical dynamics of vehicles and robots for optimal control (Nelles, 2001).
Revised: Target systems include chemical dynamics for discovering new drugs (Raff et al., 2012), climate dynamics for climate change prediction and weather forecasting (Rasp et al., 2020; Trigo & Palutikof, 1999), and physical dynamics of vehicles and robots for optimal control (Nelles, 2001).

Original: Although neural networks have achieved remarkable results in image  and natural language processing (Devlin et al., 2018; He et al., 2016), they havealso been actively investigated for modeling dynamical systems (Nelles, 2001).
Revised: In addition to image processing and natural language processing (Devlin et al., 2018; He et al., 2016), neural networks have been actively studied for modeling dynamical systems (Nelles, 2001).

Original: Their history dates back to at least the 1990s, and many approaches have been proposed so far (see Chen et al.
Revised: Their history dates back to at least the 1990s (see Chen et al.

Original: (1990); Clouse et al.
Revised: ; Clouse et al.

Original: ; Levin & Narendra (1995); Narendra & Parthasarathy (1990)
Revised: ; Levin & Narendra (1995); Narendra & Parthasarathy (1990)

Original: Sj ¨ oberg et al.
Revised: Sj ¨ oberg et al.

Original: ; Wang & Lin (1998) for examples).
Revised: ; Wang & Lin (1998) for examples).

Original: Recently, the neural ordinary differential equation (NODE) has redefined neural networks for continuous-time dynamics (Chen et al., 2018).
Revised: Recently, two notable but distinct families have been proposed. Physics-informed neural networks (PINNs) directly solve partial differential equations (PDEs) given as symbolic equations (Raissi et al., 2019).

Original: NODE defines an ordinary differential equation (ODE) using a neural network and employs a numerical integrator to obtain a solution.
Revised: Neural ordinary differential equations (NODEs) learn ordinary differential equations (ODEs) from observed data and solve them using numerical integrators (Chen et al., 2018). Our focus this time is on NODEs.

Original: Most real-world systems are associated with first integrals (a.k.a.
Revised: Most real-world systems are associated with first integrals (a.k.a.

Original: invariant quantities), which are quantities that remain unchanged over time (Hairer et al., 2006).
Revised: invariant quantities), which are quantities that remain unchanged over time (Hairer et al., 2006).

Original: First integrals arise from intrinsic geometric structures of systems and are sometimes more important than superficial dynamics in understanding systems (see Appendix A for details).
Revised: First integrals arise from intrinsic geometric structures of systems and are sometimes more important than superficial dynamics in understanding systems (see Appendix A for details).

Original: Many previous studies have incorporated prior knowledge about first integrals and attempted to accurately learn a target system.
Revised: Many previous studies have extended NODEs by incorporating prior knowledge about first integrals and attempted to accurately learn a target system.

Original: Greydanus et al.
Revised: Greydanus et al.

Original: (2019) proposed the Hamiltonian neural network (HNN), which employs a neural network to approximate Hamilton’s equation, thereby conserving the system energy called the Hamiltonian.
Revised: (2019) proposed the Hamiltonian neural network (HNN), which employs a neural network to approximate Hamilton’s equation, thereby conserving the system energy called the Hamiltonian.

Original: Finzi et al.
Revised: Finzi et al.

Original: (2020a) proposed neural network architectures that conserve linear and angular momenta by utilizing the graph structure.
Revised: (2020a) proposed neural network architectures that conserve linear and angular momenta by utilizing the graph structure.

Original: Finzi et al.
Revised: Finzi et al.

Original: (2020b) also extended an HNN to a system with holonomic constraints, which led to first integrals such as a pendulum length.
Revised: (2020b) also extended an HNN to a system with holonomic constraints, which led to first integrals such as a pendulum length.

Original: Matsubara et al.
Revised: Matsubara et al.

Original: (2020) proposed a model that preserves the total mass of a discretized partial differential equation (PDE).
Revised: (2020) proposed a model that preserves the total mass of a discretized PDE.

Original: These studies have demonstrated that the more prior knowledge a neural network has about first integrals, the more accurate their dynamics prediction.
Revised: These studies have demonstrated that the more prior knowledge a neural network has about first integrals, the more accurate their dynamics prediction.

Original: See Table 1 for comparisons.
Revised: See Table 1 for comparisons.

Original: [Table]
Revised: [Table]

Original: Previous studies have mainly attempted to preserve known first integrals for better computer simulations.
Revised: Previous studies have mainly attempted to preserve known first integrals for better computer simulations.

Original: However, in situations where a neural network learns an unknown target system [Table] it is naturally expected that first integrals associated with the target system are also unknown [Table]
Revised: However, in situations where a neural network learns an unknown target system, it is naturally expected that first integrals associated with the target system are also unknown, and it is not clear which of the above methods are available.

Original: and it is not clear which of the above methods are available.

Original: Therefore, this study proposes first integral-preserving neural differential equation (FINDE) to find and preserve unknown first integrals from data in a unified manner.
Revised: Therefore, this study proposes first integral-preserving neural differential equation (FINDE) to find and preserve unknown first integrals from data in a unified manner.

Original: FINDE has two versions for continuous and discrete time; these have the following advantages.
Revised: FINDE has two versions for continuous and discrete time; these have the following advantages.

Original: Finding First Integrals Many studies have designed architectures or operations of neural networks to model continuous-time dynamics with known types of first integrals.
Revised: Finding First Integrals Many studies have designed architectures or operations of neural networks to model continuous-time dynamics with known types of first integrals.

Original: However, the underlying geometric structures of a target system are generally unknown in practice.
Revised: However, the underlying geometric structures of a target system are generally unknown in practice.

Original: In contrast, FINDE finds various types of first integrals from data in a unified manner and preserves them in predictions.
Revised: In contrast, FINDE finds various types of first integrals from data in a unified manner and preserves them in predictions.

Original: For example, from an energy-dissipating system, FINDE can find first integrals other than energy.
Revised: For example, from an energy-dissipating system, FINDE can find first integrals other than energy.

Original: FINDE can find not only known first integrals, but also unknown ones.
Revised: FINDE can find not only known first integrals, but also unknown ones.

Original: Hence, FINDE can lead to scientific discoveries.
Revised: Hence, FINDE can lead to scientific discoveries.

Original: Combination with Known First Integrals FINDE can be combined with previously proposed neural networks designed to preserve known first integrals, such as HNNs.
Revised: Combination with Known First Integrals FINDE can be combined with previously proposed neural networks designed to preserve known first integrals, such as HNNs.

Original: In addition, when some first integrals are known in advance, they can also be incorporated into FINDE to avoid rediscovery.
Revised: In addition, when some first integrals are known in advance, they can also be incorporated into FINDE to avoid rediscovery.

Original: Therefore, FINDE is available in various situations.
Revised: Therefore, FINDE is available in various situations.

Original: Exact Preservation of First Integrals
Revised: Exact Preservation of First Integrals

Original: The first integral associated with a continuous-time system is destroyed after the dynamics is temporally discretized for computer simulations.
Revised: The first integral associated with a continuous-time system is destroyed after the dynamics is temporally discretized for computer simulations.

Original: By leveraging the discrete gradient, the discrete-time version of FINDE preserves first integrals exactly (up to rounding errors) in discrete time and further improves the prediction performance.
Revised: By leveraging the discrete gradient, the discrete-time version of FINDE preserves first integrals exactly (up to rounding errors) in discrete time and further improves the prediction performance.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: First Integrals Let us consider a time-invariant differential system dd t u
Revised: First Integrals Let us consider a time-invariant differential system dd t u

Original: = f ( u ) on an Ndimensional manifold M , where u denotes the system state and f : M → T u M represents a vector field on M .
Revised: = f ( u ) on an Ndimensional manifold M , where u denotes the system state and f : M → T u M represents a vector field on M .

Original: For simplicity, we suppose the manifold M to be a Euclidean space R N .
Revised: For simplicity, we suppose the manifold M to be a Euclidean space R N .

Original: Definition 1 (first integral) .
Revised: Definition 1 (first integral) .

Original: A quantity V : M → R is referred to as a first integral of a system dd t u = f ( u ) if it remains constant along with any solution u ( t ) , i.e., dd t V ( u ) = 0 .
Revised: A quantity V : M → R is referred to as a first integral of a system dd t u = f ( u ) if it remains constant along with any solution u ( t ) , i.e., dd t V ( u ) = 0 .

Original: If a differential system dd t u = f ( u ) has K functionally independent first integrals V 1 , . .
Revised: If a differential system dd t u = f ( u ) has K functionally independent first integrals V 1 , . .

Original: , V K , the solution u ( t ) given an initial value u 0 stays at the ( N − K ) -dimensional submanifold
Revised: , V K , the solution u ( t ) given an initial value u 0 stays at the ( N − K ) -dimensional submanifold

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The tangent space T u M ′ ⊂ T u M of the submanifold M ′ ⊂ M at a point u is the orthogonal complement to the space spanned by the gradients ∇ V k ( u ) of the first integrals V k for k = 1 , . .
Revised: The tangent space T u M ′ ⊂ T u M of the submanifold M ′ ⊂ M at a point u is the orthogonal complement to the space spanned by the gradients ∇ V k ( u ) of the first integrals V k for k = 1 , . .

Original: , K ;
Revised: , K ;

Original: [Equation]
Revised: [Equation]

Revised: u

Original: [Equation]
Revised: [Equation]

Original: k
Revised: u

Original:  = 1 , . .
Revised: [Equation] ⊤ w [Equation] 0 for k = 1 , . .

Original: . , K } .
Revised: . , K } .

Original: Conversely, if the time-derivative f at point u is on the tangent space T u M ′ for certain functions
Revised: Conversely, if the time-derivative f at point u is on the tangent space T u M ′ for certain functions

Original: V k ’s, the quantities V k ’s are first integrals of the system dd t u = f ( u ) ; it holds that dd t V k ( u ) = ∇ V k ( u ) ⊤ d d t u = ∇ V k ( u ) ⊤ f ( u ) = 0 .
Revised: V k ’s, the quantities V k ’s are first integrals of the system dd t u = f ( u ) ; it holds that dd t V k ( u ) = ∇ V k ( u ) ⊤ d d t u = ∇ V k ( u ) ⊤ f ( u ) = 0 .

Original: One of the most well-known first integrals is the Hamiltonian H , which represents the system energy of a Hamiltonian system.
Revised: One of the most well-known first integrals is the Hamiltonian H , which represents the system energy of a Hamiltonian system.

Original: Noether’s theorem states that a continuous symmetry of a system leads to a conservation law (and hence a first integral)
Revised: Noether’s theorem states that a continuous symmetry of a system leads to a conservation law (and hence a first integral)

Original: (Hairer et al., 2006).
Revised: (Hairer et al., 2006).

Original: A Hamiltonian system is symmetric to translation in time, and the corresponding first integral is the Hamiltonian.
Revised: A Hamiltonian system is symmetric to translation in time, and the corresponding first integral is the Hamiltonian.

Original: Symmetries to translation and rotation in space lead to the conservation of linear and angular momenta.
Revised: Symmetries to translation and rotation in space lead to the conservation of linear and angular momenta.

Original: However, not all first integrals are related to symmetries.
Revised: However, not all first integrals are related to symmetries.

Original: A pendulum can be expressed in Cartesian coordinates, and then the rod length constrains the mass position.
Revised: A pendulum can be expressed in Cartesian coordinates, and then the rod length constrains the mass position.

Original: This type of constraint is called a holonomic constraint and leads to first integrals.
Revised: This type of constraint is called a holonomic constraint and leads to first integrals.

Original: Models of disease spreads and chemical reactions have the total mass (population) as the first integral.
Revised: Models of disease spreads and chemical reactions have the total mass (population) as the first integral.

Original: Also for a system described by a PDE, the total mass is sometimes a first integral (Furihata & Matsuo, 2010).
Revised: Also for a system described by a PDE, the total mass is sometimes a first integral (Furihata & Matsuo, 2010).

Original: See Appendix A for the theoretical classification of dynamics and corresponding geometric structures.
Revised: See Appendix A for the classes of dynamics, their geometric structures, and related studies to find or preserve first integrals.

Original: First Integrals in Numerical Analysis For computer simulations, differential systems are discretized in time and solved by numerical integration.
Revised: First Integrals in Numerical Analysis For computer simulations, differential systems are discretized in time and solved by numerical integration, causing numerical errors (which is composed of temporal discretization errors and rounding errors).

Original: Then, the geometric structures of the system are often destroyed, and the corresponding first integrals are no longer preserved.
Revised: Moreover, the geometric structures of the system are often destroyed, and the corresponding first integrals are no longer preserved.

Original: A common remedy is a symplectic integrator, which preserves the symplectic structure and accurately integrates Hamiltonian systems (Hairer et al., 2006).
Revised: A common remedy is a symplectic integrator, which preserves the symplectic structure and accurately integrates Hamiltonian systems (Hairer et al., 2006).

Original: However, the Ge–Marsden theorem states that a symplectic integrator only approximately conserves the Hamiltonian (Zhong & Marsden, 1988).
Revised: However, the Ge–Marsden theorem states that a symplectic integrator only approximately conserves the Hamiltonian (Zhong & Marsden, 1988).

Original: Hence, many numerical schemes have also been investigated to preserve first integrals exactly, while these schemes cannot preserve the symplectic structure.
Revised: Hence, many numerical schemes have also been investigated to preserve first integrals exactly, while these schemes cannot preserve the symplectic structure.

Original: Some examples are shown below.
Revised: Some examples are shown below.

Original: Let the superscript s ∈ { 0 , 1 , . .
Revised: Let the superscript s ∈ { 0 , 1 , . .

Original: . , S } denote the state u s or time t s at the s -th time step, and ∆ t s = t s +1 − t s denote a time-step size.
Revised: . , S } denote the state u s or time t s at the s -th time step, and ∆ t s = t s +1 − t s denote a time-step size.

Original: A projection method uses a numerical integrator to predict the next state ˜ u
Revised: A projection method uses a numerical integrator to predict the next state ˜ u

Original: s +1 from the current state u s and then projects the state ˜ u
Revised: s +1 from the current state u s and then projects the state ˜ u

Original: s +1 onto the submanifold
Revised: s +1 onto the submanifold

Original: M ′ (Gear, 1986; Hairer et al., 2006, Section IV.4).
Revised: M ′ (Gear, 1986; Hairer et al., 2006, Section IV.4).

Original: The projected state u s +1 preserves the first integrals V k .
Revised: The projected state u s +1 preserves the first integrals V k .

Original: In particular, the projected state u s +1 is obtained by solving the optimization problem arg min u s +1 ∥ u s +1 − ˜ u
Revised: In particular, the projected state u s +1 is obtained by solving the optimization problem arg min u s +1 ∥ u s +1 − ˜ u

Original: s +1 ∥ subject to V k ( u s +1 )
Revised: s +1 ∥ subject to V k ( u s +1 )

Original: − V k ( u s ) =
Revised: − V k ( u s ) =

Original: for k = 1 , . . .
Revised: for k = 1 , . . .

Original: , K.
Revised: , K.

Original: The local coordinate method defines a coordinate system on the neighborhood of the current state u s and integrates a differential equation on it (Potra & Yen, 1991; Hairer et al., 2006, Section IV.5).
Revised: The local coordinate method defines a coordinate system on the neighborhood of the current state u s and integrates a differential equation on it (Potra & Yen, 1991; Hairer et al., 2006, Section IV.5).

Original: The discrete gradient method defines a discrete analogue to a differential system and integrates it in discrete time, thereby preserving the Hamiltonian exactly (up to rounding errors) in discrete time (Furihata & Matsuo, 2010; Gonzalez, 1996; Hong et al., 2011).
Revised: The discrete gradient method defines a discrete analogue to a differential system and integrates it in discrete time, thereby preserving the Hamiltonian exactly (up to rounding errors) in discrete time (Furihata & Matsuo, 2010; Gonzalez, 1996; Hong et al., 2011).

Original: Neural Networks to Preserve First Integrals NODE defines the right-hand side f of a differential system dd t u = f ( u ) using a neural network in the most general way with no associated first integrals (Chen et al., 2018).
Revised: Neural Networks to Preserve First Integrals NODE defines the right-hand side f of a differential system dd t u = f ( u ) using a neural network in the most general way with no associated first integrals (Chen et al., 2018).

Original: NODE is a universal approximator to ODEs and can approximate any
Revised: NODE is a universal approximator to ODEs and can approximate any

Original: ODE with arbitrary accuracy if there is an infinite amount of training data (Teshima et al., 2020).
Revised: ODE with arbitrary accuracy if there is an infinite amount of training data (Teshima et al., 2020).

Original: In practice, the amount of training data is limited, and prior knowledge about the target system is helpful for learning (see Sannai et al.
Revised: In practice, the amount of training data is limited, and prior knowledge about the target system is helpful for learning (see Sannai et al.

Original: (2021) for the case with convolutional neural networks (CNNs)).
Revised: (2021) for the case with convolutional neural networks (CNNs)).

Original: HNN (Greydanus et al., 2019) assumes the target system to be a Hamiltonian system in the canonical form, thereby guaranteeing various properties of Hamiltonian systems by definition, including the conservation of energy and preservation of the symplectic structure in continuous time (Hairer et al., 2006).
Revised: HNN (Greydanus et al., 2019) assumes the target system to be a Hamiltonian system in the canonical form, thereby guaranteeing various properties of Hamiltonian systems by definition, including the conservation of energy and preservation of the symplectic structure in continuous time (Hairer et al., 2006).

Original: Some studies have employed a symplectic integrator for HNN to preserve the energy and symplectic structure with smaller numerical errors (Chen et al., 2020).
Revised: Some studies have employed a symplectic integrator for HNN to preserve the energy and symplectic structure with smaller numerical errors (Chen et al., 2020).

Original: LieConv and EMLPHNN employ neural network architectures with translational and rotational symmetries to preserve momenta (Finzi et al., 2020a; 2021).
Revised: LieConv and EMLPHNN employ neural network architectures with translational and rotational symmetries to preserve momenta (Finzi et al., 2020a; 2021).

Original: CHNN incorporates a known holonomic constraint in the dynamics (Finzi et al., 2020b).
Revised: CHNN incorporates a known holonomic constraint in the dynamics (Finzi et al., 2020b).

Original: Deep conservation extracts latent dynamics of a PDE system and preserves a quantity of interest by forcing its flux to be zero (Lee & Carlberg, 2021).
Revised: Deep conservation extracts latent dynamics of a PDE system and preserves a quantity of interest by forcing its flux to be zero (Lee & Carlberg, 2021).

Original: HNN++ also guarantees the conservation of mass in PDE systems by using a coefficient matrix derived from differential operators (Matsubara et al., 2020).
Revised: HNN++ also guarantees the conservation of mass in PDE systems by using a coefficient matrix derived from differential operators (Matsubara et al., 2020).

Original: These methods preserve known types of first integrals and suffer from temporal discretization errors.
Revised: These methods preserve known types of first integrals and suffer from temporal discretization errors.

Original: In contrast, FINDE learns any types of first integrals from data and preserves them even after temporal discretization.
Revised: In contrast, FINDE learns any types of first integrals from data and preserves them even after temporal discretization.

Original: The neural projection method (NPM) learns fixed holonomic constraints using the projection (and inequality constraints)
Revised: The neural projection method (NPM) learns fixed holonomic constraints using the projection (and inequality constraints)

Original: (Yang et al., 2020).
Revised: (Yang et al., 2020).

Original: DGNet employed discrete gradient methods to guarantee the energy conservation in Hamiltonian systems (and the energy dissipation in friction systems)
Revised: DGNet employed discrete gradient methods to guarantee the energy conservation in Hamiltonian systems (and the energy dissipation in friction systems)

Original: (Matsubara et al., 2020).
Revised: (Matsubara et al., 2020).

Original: While these methods preserve the aforementioned first integrals exactly in discrete time, their formulations are not available for other first integrals.
Revised: While these methods preserve the aforementioned first integrals exactly in discrete time, their formulations are not available for other first integrals.

Original: Several studies have proposed neural networks to learn Lyapunov functions, which are expected to be non-increasing over time, in contrast to first integrals (Manek & Kolter, 2019; Takeishi &
Revised: Several studies have proposed neural networks to learn Lyapunov functions, which are expected to be non-increasing over time, in contrast to first integrals (Manek & Kolter, 2019; Takeishi &

Original: Kawahara, 2020).
Revised: Kawahara, 2020).

Original: If the state moves in the direction of increasing the function, it is projected onto or moved inside the contour line of the Lyapunov function.
Revised: If the state moves in the direction of increasing the function, it is projected onto or moved inside the contour line of the Lyapunov function.

Original: This concept is similar to that of the continuous-time version of FINDE but focuses on a single non-increasing quantity in continuous time; FINDE preserves multiple quantities in both continuous and discrete time.
Revised: This concept is similar to that of the continuous-time version of FINDE but focuses on a single non-increasing quantity in continuous time; FINDE preserves multiple quantities in both continuous and discrete time.

Original:  F IRST I NTEGRAL -P RESERVING N EURAL D IFFERENTIAL E QUATION
Revised:  F IRST I NTEGRAL -P RESERVING N EURAL D IFFERENTIAL E QUATION

Original: We suppose that a target system has at least K unknown functionally independent first integrals.
Revised: We suppose that a target system has at least K unknown functionally independent first integrals.

Original: When a neural network learns the dynamics of the target system, it is not guaranteed to learn these first integrals.
Revised: When a neural network learns the dynamics of the target system, it is not guaranteed to learn these first integrals.

Original: We suppose that a certain neural network ˆ f for modeling the target dynamics is given, and in addition to this model
Revised: We suppose that a certain neural network ˆ f for modeling the target dynamics is given, and in addition to this model

Original: ˆ f , we introduce a neural network that outputs a K -dimensional vector
Revised: ˆ f , we introduce a neural network that outputs a K -dimensional vector

Original: V ( u ) =
Revised: V ( u ) =

Original: ( V 1 ( u ) V 2 ( u ) . . .
Revised: ( V 1 ( u ) V 2 ( u ) . . .

Original: V K ( u )) ⊤ .
Revised: V K ( u )) ⊤ .

Original: Each element is expected to learn one of the first integrals as
Revised: Each element is expected to learn one of the first integrals as

Original: V k : R N → R for k = 1 , . . .
Revised: V k : R N → R for k = 1 , . . .

Original: , K .
Revised: , K .

Original: Then, the submanifold M ′ is defined as in Eq.
Revised: Then, the submanifold M ′ is defined as in Eq.

Original: C ONTINUOUS FINDE: T IME -D ERIVATIVE P ROJECTION M ETHOD
Revised: C ONTINUOUS FINDE: T IME -D ERIVATIVE P ROJECTION M ETHOD

Original: We propose a time-derivative projection method called continuous FINDE (cFINDE) .
Revised: We propose a time-derivative projection method called continuous FINDE (cFINDE) .

Original: The cFINDE projects the time-derivative onto the tangent space T u M ′ .
Revised: The cFINDE projects the time-derivative onto the tangent space T u M ′ .

Original: Roughly speaking, the cFINDE projects the dynamics on the space of the directions in which the first integrals do not change.
Revised: Roughly speaking, the cFINDE projects the dynamics on the space of the directions in which the first integrals do not change.

Original: In this way, the method can learn dynamics while preserving first integrals V , thereby finding unknown first integrals from data.
Revised: In this way, the method can learn dynamics while preserving first integrals V , thereby finding unknown first integrals from data.

Original: We refer to the neural network that defines the time-derivative ˆ f : R N → R N as the base model.
Revised: We refer to the neural network that defines the time-derivative ˆ f : R N → R N as the base model.

Original: Applying the method of Lagrange multipliers to the projection method in Eq.
Revised: Applying the method of Lagrange multipliers to the projection method in Eq.

Original: (3), and taking the limit as the time-step size approaches zero, we have
Revised: (3), and taking the limit as the time-step size approaches zero, we have

Original: [Equation]
Revised: [Equation]

Original: where M = ∂ V ∂ u and λ ∈ R N is the Lagrange multiplier (see Appendix B.1 for detailed derivation).
Revised: where M = ∂ V ∂ u and λ ∈ R N is the Lagrange multiplier (see Appendix B.1 for detailed derivation).

Original: We transform the second equation to obtain
Revised: We transform the second equation to obtain

Original: [Equation]
Revised: [Equation]

Original: from which we obtain the Lagrange multiplier λ ( u )
Revised: from which we obtain the Lagrange multiplier λ ( u )

Original: =
Revised: =

Original: ( M ( u ) M ( u ) ⊤ ) − 1 M ( u ) ˆ f ( u ) .
Revised: ( M ( u ) M ( u ) ⊤ ) − 1 M ( u ) ˆ f ( u ) .

Original: By eliminating λ ( u ) , we define the cFINDE as
Revised: By eliminating λ ( u ) , we define the cFINDE as

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 (continuous-time first integral preservation) .
Revised: Theorem 1 (continuous-time first integral preservation) .

Original: The cFINDE dd t u = f ( u ) preserves all first integrals V k for k = 1 , . .
Revised: The cFINDE dd t u = f ( u ) preserves all first integrals V k for k = 1 , . .

Original: , K in continuous time, that is, dd t V k = 0 .
Revised: , K in continuous time, that is, dd t V k = 0 .

Original: See Appendix B.1 for proof.
Revised: See Appendix B.1 for proof.

Original: The base model ˆ f can be a NODE, an HNN, or any other model depending on the available prior knowledge.
Revised: The base model ˆ f can be a NODE, an HNN, or any other model depending on the available prior knowledge.

Original: Additionally, if a first integral is already known, it can be directly used as one of the first integrals V k instead of being found by the neural network.
Revised: Additionally, if a first integral is already known, it can be directly used as one of the first integrals V k instead of being found by the neural network.

Original: Note that even though the base model ˆ f is an HNN, due to the projection, the cFINDE f is no longer a
Revised: Note that even though the base model ˆ f is an HNN, due to the projection, the cFINDE f is no longer a

Original: Hamiltonian system in the strict sense.
Revised: Hamiltonian system in the strict sense.

Original: Compared to the base model ˆ f , the cFINDE requires the additional computation of the neural network V , several matrix multiplications, and an inverse operation.
Revised: Compared to the base model ˆ f , the cFINDE requires the additional computation of the neural network V , several matrix multiplications, and an inverse operation.

Original: The inverse operation has a computational cost of O
Revised: The inverse operation has a computational cost of O

Original: ( K 3 ) , which is not costly if the number K of first integrals is small.
Revised: ( K 3 ) , which is not costly if the number K of first integrals is small.

Original: Many previous models also need the inverse operation to satisfy the constraints and geometric structures, such as Lagrangian neural network (LNN) (Cranmer et al., 2020), neural symplectic form (Chen et al., 2021), and CHNN (Finzi et al., 2020b).
Revised: Many previous models also need the inverse operation to satisfy the constraints and geometric structures, such as Lagrangian neural network (LNN) (Cranmer et al., 2020), neural symplectic form (Chen et al., 2021), and CHNN (Finzi et al., 2020b).

Original: 3. D ISCRETE FINDE: D ISCRETE -T IME D ERIVATIVE P ROJECTION M ETHOD
Revised: 3. D ISCRETE FINDE: D ISCRETE -T IME D ERIVATIVE P ROJECTION M ETHOD

Original: The cFINDE is still an ODE and hence needs to be solved using a numerical integrator, which causes the temporal discretization errors in the first integrals.
Revised: The cFINDE is still an ODE and hence needs to be solved using a numerical integrator, which causes the temporal discretization errors in the first integrals.

Original: In order to eliminate these errors, it is necessary to constrain the destination (i.e., finite difference) rather than the direction (i.e., time-derivative).
Revised: In order to eliminate these errors, it is necessary to constrain the destination (i.e., finite difference) rather than the direction (i.e., time-derivative).

Original: For this purpose, we propose discrete FINDE (dFINDE) by employing discrete gradients to define discrete tangent spaces , which are needed to constraint the state variables on the submanifold M ′ .
Revised: For this purpose, we propose discrete FINDE (dFINDE) by employing discrete gradients to define discrete tangent spaces , which are needed to constraint the state variables on the submanifold M ′ .

Original: A discrete gradient ∇ V is a discrete analogue to a gradient ∇ V (Furihata & Matsuo, 2010; Gonzalez, 1996; Hong et al., 2011).
Revised: A discrete gradient ∇ V is a discrete analogue to a gradient ∇ V (Furihata & Matsuo, 2010; Gonzalez, 1996; Hong et al., 2011).

Original: Recall that a gradient ∇ V of a function V : R N → R can be regarded as a function R N → R N that satisfies the chain rule dd t V ( u ) = ∇ V ( u )
Revised: Recall that a gradient ∇ V of a function V : R N → R can be regarded as a function R N → R N that satisfies the chain rule dd t V ( u ) = ∇ V ( u )

Original: ⊤ dd t u .
Revised: ⊤ dd t u .

Original: Analogously, a discrete gradient ∇ is defined as follows:
Revised: Analogously, a discrete gradient ∇ is defined as follows:

Original: Definition 2 (discrete gradient) .
Revised: Definition 2 (discrete gradient) .

Original: A discrete gradient ∇ V of a function V : R N → R is a function R N × R N → R N that satisfies V ( v ) − V ( u ) =
Revised: A discrete gradient ∇ V of a function V : R N → R is a function R N × R N → R N that satisfies V ( v ) − V ( u ) =

Original: ∇ V ( v , u ) ⊤ ( v − u ) and ∇ V ( u , u ) = ∇ V ( u ) .
Revised: ∇ V ( v , u ) ⊤ ( v − u ) and ∇ V ( u , u ) = ∇ V ( u ) .

Original: The first condition is a discrete analogue to the chain rule when replacing the time-derivatives dd t V and dd t u with finite differences ( V ( v ) − V ( u )) and ( v − u ) , respectively, and the second condition ensures consistency with the ordinary gradient ∇ V .
Revised: The first condition is a discrete analogue to the chain rule when replacing the time-derivatives dd t V and dd t u with finite differences ( V ( v ) − V ( u )) and ( v − u ) , respectively, and the second condition ensures consistency with the ordinary gradient ∇ V .

Original: A discrete gradient ∇ V is not uniquely determined and has been obtained manually.
Revised: A discrete gradient ∇ V is not uniquely determined and has been obtained manually.

Original: Recently, the automatic discrete differentiation algorithm (ADDA) has been proposed by Matsubara et al.
Revised: Recently, the automatic discrete differentiation algorithm (ADDA) has been proposed by Matsubara et al.

Original: (2020), which obtains a discrete gradient of a neural network in a manner similar to the automatic differentiation algorithm (Abadi et al., 2016;
Revised: (2020), which obtains a discrete gradient of a neural network in a manner similar to the automatic differentiation algorithm (Abadi et al., 2016;

Original: Paszke et al., 2017).
Revised: Paszke et al., 2017).

Original: The discrete gradient is defined in discrete time; hence, the prediction using the discrete gradient is free from temporal discretization errors.
Revised: The discrete gradient is defined in discrete time; hence, the prediction using the discrete gradient is free from temporal discretization errors.

Original: See Appendix B.2 and the references Furihata & Matsuo (2010); Matsubara et al.
Revised: See Appendix B.2 and the references Furihata & Matsuo (2010); Matsubara et al.

Original: (2020) for more details.
Revised: (2020) for more details.

Original: Following Christiansen et al.
Revised: Following Christiansen et al.

Original: (2011); Dahlby et al.
Revised: (2011); Dahlby et al.

Original: (2011), we introduce a discrete analogue to the tangent space T u M ′ called the discrete tangent space T ( v , u ) M ′ .
Revised: (2011), we introduce a discrete analogue to the tangent space T u M ′ called the discrete tangent space T ( v , u ) M ′ .

Original: In particular, for a pair of points ( v , u ) ∈ M ′ , the discrete tangent space is defined as
Revised: In particular, for a pair of points ( v , u ) ∈ M ′ , the discrete tangent space is defined as

Original: [Equation]
Revised: [Equation]

Original: If the finite difference ( u s +1 − u s ) between the predicted and current states is on the discrete tangent space T ( u s +1 , u s ) M ′ , the first integrals V k are preserved because V k ( u s +1 ) − V k
Revised: If the finite difference ( u s +1 − u s ) between the predicted and current states is on the discrete tangent space T ( u s +1 , u s ) M ′ , the first integrals V k are preserved because V k ( u s +1 ) − V k

Original: ( u s )
Revised: ( u s )

Original: = ∇ V k
Revised: = ∇ V k

Original: ( u s +1 , u s ) ⊤ ( u s +1 − u s )
Revised: ( u s +1 , u s ) ⊤ ( u s +1 − u s )

Original: = 0 .
Revised: = 0 .

Original: Note that similar concepts defined in different ways are also referred to as discrete tangent spaces (Cuell & Patrick, 2009; Dehmamy et al., 2021).
Revised: Note that similar concepts defined in different ways are also referred to as discrete tangent spaces (Cuell & Patrick, 2009; Dehmamy et al., 2021).

Original: We suppose that a neural network (e.g., NODE) ˆ f defines an ODE and a numerical integrator predicts the next state ˜ u
Revised: We suppose that a neural network (e.g., NODE) ˆ f defines an ODE and a numerical integrator predicts the next state ˜ u

Original: s +1 from a given state u s .
Revised: s +1 from a given state u s .

Original: We call this process a discrete-time base model ˆ ψ , which satisfies ˜ u
Revised: We call this process a discrete-time base model ˆ ψ , which satisfies ˜ u

Original: s +1 − u s ∆ t s = ˆ ψ ( u s ; ∆ t s ) .
Revised: s +1 − u s ∆ t s = ˆ ψ ( u s ; ∆ t s ) .

Original: Subsequently, we consider the model
Revised: Subsequently, we consider the model

Original: [Equation]
Revised: [Equation]

Original: where M ( u s +1 , u s ) = ( ∇ V 1 ( u s +1 , u s ) .
Revised: where M ( u s +1 , u s ) = ( ∇ V 1 ( u s +1 , u s ) .

Original: ∇ V K ( u s +1 , u s ))
Revised: ∇ V K ( u s +1 , u s ))

Original: ⊤ .
Revised: ⊤ .

Original: As shown in Appendix B.1, this formulation is also derived from the projection method in Eq.
Revised: As shown in Appendix B.1, this formulation is also derived from the projection method in Eq.

Original: Using the chain rule of the discrete gradient,
Revised: Using the chain rule of the discrete gradient,

Original: [Equation]
Revised: [Equation]

Original: t s ) , (9)
Revised: t s ) , (9)

Original: Substituting this into Eq.
Revised: Substituting this into Eq.

Original: (8) and eliminating the Lagrange multiplier λ , we define the dFINDE as u 
Revised: (8) and eliminating the Lagrange multiplier λ , we define the dFINDE as u s +1 − u s ∆ t s = ψ ( u s +1 , u s ; ∆ t s ) = ( I − Y ( u

Original: [Equation]
Revised: [Equation]

Original:  s = [Equation]
Revised: +1 , u s ))

Original: [Equation] ) for Y =M ⊤ [Equation]
Revised: ˆ ψ ( u s ; ∆ t s ) for Y =

Original: M  M [Equation] 10) where we have abbreviated M ( u s +1 , u s ) and Y ( u s +1 , u s ) to M and Y , respectively.
Revised: M ⊤ ( M M ⊤ ) − 1 M, (10) where we have abbreviated M ( u s +1 , u s ) and Y ( u s +1 , u s ) to M and Y , respectively.

Original: Theorem 2 (discrete-time first integral preservation) .
Revised: Theorem 2 (discrete-time first integral preservation) .

Original: The dFINDE u s +1 − u s ∆ t s = ψ ( u [Equation] 1 , u s ; ∆ t s ) preserves all first integrals V k for k = 1 , . .
Revised: The dFINDE u s +1 − u s ∆ t s = ψ ( u s +1 , u s ; ∆ t s ) preserves all first integrals V k for k = 1 , . .

Original: , K in discrete time, that is, V k ( u s +1 ) − V k
Revised: , K in discrete time, that is, V k ( u s +1 ) − V k

Original: ( u s )
Revised: ( u s )

Original: = 0 .
Revised: = 0 .

Original: See Appendix B.1 for proof.
Revised: See Appendix B.1 for proof.

Original: Intuitively, dFINDE projects the finite difference (discrete-time derivative)
Revised: Intuitively, dFINDE projects the finite difference (discrete-time derivative)

Original: ˆ ψ onto the discrete tangent space T ( u s +1 , u s ) M ′ after the numerical integration for each step, whereas cFINDE projects the time-derivative
Revised: ˆ ψ onto the discrete tangent space T ( u s +1 , u s ) M ′ after the numerical integration for each step, whereas cFINDE projects the time-derivative

Original: ˆ f onto the tangent space T u M ′ at every substep inside a numerical integrator.
Revised: ˆ f onto the tangent space T u M ′ at every substep inside a numerical integrator.

Original: In the discrete-time base model ˆ ψ , the ODE ˆ f can be defined by any model, such as NODE or HNN, and the numerical integrator can be implemented by any method, such as the Runge–Kutta method or the leapfrog integrator.
Revised: In the discrete-time base model ˆ ψ , the ODE ˆ f can be defined by any model, such as NODE or HNN, and the numerical integrator can be implemented by any method, such as the Runge–Kutta method or the leapfrog integrator.

Original: The projection method in Eq.
Revised: The projection method in Eq.

Original: (3), the method in Eq.
Revised: (3), the method in Eq.

Original: (8), and the dFINDE in Eq (10) are implicit methods.
Revised: (8), and the dFINDE in Eq (10) are implicit methods and hence relatively computationally expensive.

Original: However, only the dFINDE can be trained non-iteratively by standard backpropagation algorithms.
Revised: However, only the dFINDE can be trained non-iteratively by standard backpropagation algorithms.

Original: As explained in Appendix B.3, this is because the next state u s +1 is given during training and the ADDA can explicitly obtain the discrete gradient and its computational graph.
Revised: As explained in Appendix B.3, this is because the next state u s +1 is given during training and the ADDA can explicitly obtain the discrete gradient and its computational graph.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTINGS

Original: Target Systems We evaluated FINDE and base models using datasets associated with first integrals; these are summarized in Table 2.
Revised: Target Systems We evaluated FINDE and base models using datasets associated with first integrals; these are summarized in Table 2.

Original: A gravitational two-body problem (2-body) on a 2dimensional configuration space is a typical Hamiltonian system in the canonical form.
Revised: A gravitational two-body problem (2-body) on a 2dimensional configuration space is a typical Hamiltonian system in the canonical form.

Original: In addition to the total energy, the system has first integrals related to symmetries in space, namely, the linear and angular momenta.
Revised: In addition to the total energy, the system has first integrals related to symmetries in space, namely, the linear and angular momenta.

Original: The Korteweg–De Vries (KdV) equation is a PDE model of shallow water waves.
Revised: The Korteweg–De Vries (KdV) equation is a PDE model of shallow water waves.

Original: This equation is a Hamiltonian system in a non-canonical form and has the Hamiltonian, total mass, and many other quantities as first integrals.
Revised: This equation is a Hamiltonian system in a non-canonical form and has the Hamiltonian, total mass, and many other quantities as first integrals.

Original: We discretized the KdV equation in space, obtaining a fifty-dimensional state u .
Revised: We discretized the KdV equation in space, obtaining a fifty-dimensional state u .

Original: A double pendulum (2-pend) is a Hamiltonian system in polar coordinates.
Revised: A double pendulum (2-pend) is a Hamiltonian system in polar coordinates.

Original: However, we transformed it to Cartesian coordinates; hence, it became a Poisson system.
Revised: However, we transformed it to Cartesian coordinates; hence, it became a Poisson system.

Original: The lengths of the two rods work as holonomic constraints and lead to four first integrals in addition to the Hamiltonian.
Revised: The lengths of the two rods work as holonomic constraints and lead to four first integrals in addition to the Hamiltonian.

Original: The FitzHugh–Nagumo model is a biological neuron model as an electric circuit, which exhibits a rapid and transient change of voltage called a spike.
Revised: The FitzHugh–Nagumo model is a biological neuron model as an electric circuit, which exhibits a rapid and transient change of voltage called a spike.

Original: As an electric circuit, the currents through and voltages applied to the inductor and capacitor can be regarded as system states, which are constrained by the circuit topology and Kirchhoff’s current and voltage laws.
Revised: As an electric circuit, the currents through and voltages applied to the inductor and capacitor can be regarded as system states, which are constrained by the circuit topology and Kirchhoff’s current and voltage laws.

Original: Then, this system has a state of four elements and two first integrals.
Revised: Then, this system has a state of four elements and two first integrals.

Original: Because the resistor dissipates the energy, the system is not a Poisson system, but a Dirac structure can be found (van der Schaft & Jeltsema, 2014).
Revised: Because the resistor dissipates the energy, the system is not a Poisson system, but a Dirac structure can be found (van der Schaft & Jeltsema, 2014).

Original: We generated a time-series set of each dataset with different initial conditions (hence, different values of first integrals).
Revised: We generated a time-series set of each dataset with different initial conditions (hence, different values of first integrals).

Original: See Appendix C for more details.
Revised: See Appendix C for more details.

Original: Implementation We implemented the proposed FINDE and evaluated it under the following settings.
Revised: Implementation We implemented the proposed FINDE and evaluated it under the following settings.

Original: We implemented all codes by modifying the officially released codes of HNN (Greydanus et al., 2019) 1 and DGNet (Matsubara et al., 2020) 2 .
Revised: We implemented all codes by modifying the officially released codes of HNN (Greydanus et al., 2019) 1 and DGNet (Matsubara et al., 2020) 2 .

Original: We used Python v. 3.8.12 with packages scipy v. 1.7.3, pytorch v. 1.10.2, torchdiffeq v. 0.1.1, functorch v. 1.10 preview, and gplearn v. 0.4.2.
Revised: We used Python v. 3.8.12 with packages scipy v. 1.7.3, pytorch v. 1.10.2, torchdiffeq v. 0.1.1, functorch v. 1.10 preview, and gplearn v. 0.4.2.

Original: We used the Dormand–Prince method (dopri5) (Dormand & Prince, 1986) as the numerical integrator, except in Section 4.2.
Revised: We used the Dormand–Prince method (dopri5) (Dormand & Prince, 1986) as the numerical integrator, except in Section 4.2.

Original: All experiments were performed on a single NVIDIA A100.
Revised: All experiments were performed on a single NVIDIA A100.

Original: Following HNN (Greydanus et al., 2019) and DGNet (Matsubara et al., 2020), we used fullyconnected neural networks with two hidden layers.
Revised: Following HNN (Greydanus et al., 2019) and DGNet (Matsubara et al., 2020), we used fullyconnected neural networks with two hidden layers.

Original: The input was the state u , and the output represented the first integrals V for FINDE, time-derivative ˆ f for NODE, or the Hamiltonian H for HNN.
Revised: The input was the state u , and the output represented the first integrals V for FINDE, time-derivative ˆ f for NODE, or the Hamiltonian H for HNN.

Original: Each hidden layer had 200 units and preceded a hyperbolic tangent activation function.
Revised: Each hidden layer had 200 units and preceded a hyperbolic tangent activation function.

Original: Each weight matrix was initialized as an orthogonal matrix.
Revised: Each weight matrix was initialized as an orthogonal matrix.

Original: For the KdV dataset, we used a 1-dimensional CNN, wherein the kernel size of each layer was 3.
Revised: For the KdV dataset, we used a 1-dimensional CNN, wherein the kernel size of each layer was 3.

Original: The double pendulum is a second–order system, implying that the time-derivative dd t q of the position q is known to be the velocity v .
Revised: The double pendulum is a second–order system, implying that the time-derivative dd t q of the position q is known to be the velocity v .

Original: Hence, we treated only the acceleration dd t v as the output to learn in the 2-pend dataset.
Revised: Hence, we treated only the acceleration dd t v as the output to learn in the 2-pend dataset.

Original: This assumption slightly improved the absolute performances but did not change the relative trends.
Revised: This assumption slightly improved the absolute performances but did not change the relative trends.

Original: As the loss function for the cFINDE, we used the mean squared error (MSE) between the ground truth future state u s +1 GT and the future state u s +1 pred.
Revised: As the loss function for the cFINDE, we used the mean squared error (MSE) between the ground truth future state u s +1 GT and the future state u s +1 pred.

Original: predicted from the current step u s GT normalized by the time-step size ∆ t s ; we named this the 1-step error .
Revised: predicted from the current step u s GT normalized by the time-step size ∆ t s ; we named this the 1-step error .

Original: For the dFINDE, we used the MSE between the left- and right-hand sides of Eq.
Revised: For the dFINDE, we used the MSE between the left- and right-hand sides of Eq.

Original: (10) because the ground truth states u s GT and u s +1 GT are available during the training phase.
Revised: (10) because the ground truth states u s GT and u s +1 GT are available during the training phase.

Original: The base model and FINDE were jointly trained using the Adam optimizer (Kingma & Ba, 2015) with the parameters ( β 1 , β 2 ) =
Revised: The base model and FINDE were jointly trained using the Adam optimizer (Kingma & Ba, 2015) with the parameters ( β 1 , β 2 ) =

Original: (0 . 9 , 0 . 999) and a batch size of 200.
Revised: (0 . 9 , 0 . 999) and a batch size of 200.

Original: The learning rate was initialized to 10 − 3 and decayed to zero with cosine annealing (Loshchilov &
Revised: The learning rate was initialized to 10 − 3 and decayed to zero with cosine annealing (Loshchilov &

Original: Hutter, 2017).
Revised: Hutter, 2017).

Original: See Appendix B.3 and the enclosed source code for details about implementations.
Revised: See Appendix B.3 and the enclosed source code for details about implementations.

Original: Evaluation Metric We used the 1-step error as an evaluation metric, which is identical to the loss function for the cFINDE, and displayed it in the scale × 10 − 9 .
Revised: Evaluation Metric We used the 1-step error as an evaluation metric, which is identical to the loss function for the cFINDE, and displayed it in the scale × 10 − 9 .

Original: The lower this indicator, the better, as indicated by ↓ .
Revised: The lower this indicator, the better, as indicated by ↓ .

Original: The MSEs of the state or system energy over a long period are misleading indicators, as suggested in prior studies (Botev et al., 2021;
Revised: The MSEs of the state or system energy over a long period are misleading indicators, as suggested in prior studies (Botev et al., 2021;

Original: Jin et al., 2020b; Vlachas et al., 2020).
Revised: Jin et al., 2020b; Vlachas et al., 2020).

Original: For example, a periodic orbit that is correctly learned except for a slight difference in angular velocity would have the same MSE as an orbit that never moves from its initial position.
Revised: For example, a periodic orbit that is correctly learned except for a slight difference in angular velocity would have the same MSE as an orbit that never moves from its initial position.

Original: Instead, we used the valid prediction time ( VPT ) (Botev et al., 2021;
Revised: Instead, we used the valid prediction time ( VPT ) (Botev et al., 2021;

Original: Jin et al., 2020b; Vlachas et al., 2020).
Revised: Jin et al., 2020b; Vlachas et al., 2020).

Original: VPT denotes the time point s divided by the length S of the time-series at which the MSE of the predicted state u s pred.
Revised: VPT denotes the time point s divided by the length S of the time-series at which the MSE of the predicted state u s pred.

Original: first exceeds a given threshold θ in an initial value problem, that is,
Revised: first exceeds a given threshold θ in an initial value problem, that is,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The higher this indicator, the better, as indicated by ↑ .
Revised: The higher this indicator, the better, as indicated by ↑ .

Original: To obtain VPTs, we normalized each element of state to have the zero mean and unit variance in the training data and set θ to 0.01.
Revised: To obtain VPTs, we normalized each element of state to have the zero mean and unit variance in the training data and set θ to 0.01.

Original: For systems with “spiking” behaviors, a small error in phase may be regarded as a significant error in the state; for the FitzHugh–Nagumo model, we obtained the VPTs by allowing for a delay and advance of up to 5 steps.
Revised: For systems with “spiking” behaviors, a small error in phase may be regarded as a significant error in the state; for the FitzHugh–Nagumo model, we obtained the VPTs by allowing for a delay and advance of up to 5 steps.

Original: 4. D EMONSTRATION OF F IRST I NTEGRAL P RESERVATION
Revised: 4. D EMONSTRATION OF F IRST I NTEGRAL P RESERVATION

Original: [Figure]
Revised: [Figure]

Original: e r g y analytical cFINDE leapfrog dFINDE
Revised: e r g y analytical cFINDE leapfrog dFINDE

Original: Before learning first integrals from data, we demonstrate that dFINDE can preserve first integrals without temporal discretization errors.
Revised: Before learning first integrals from data, we demonstrate that dFINDE can preserve first integrals without temporal discretization errors.

Original: We used a mass-spring system, which had the state u
Revised: We used a mass-spring system, which had the state u

Original: = ( q v ) ⊤ , dynamics dd t q = v and dd t v = − q , and system energy
Revised: = ( q v ) ⊤ , dynamics dd t q = v and dd t v = − q , and system energy

Original: E ( q, v ) = 12 ( q 2 + v 2 ) .
Revised: E ( q, v ) = 12 ( q 2 + v 2 ) .

Original: Using an initial value of (1 . 0 0 . 0) ⊤ and a time-step size of ∆ t = 0 .
Revised: Using an initial value of (1 . 0 0 . 0) ⊤ and a time-step size of ∆ t = 0 .

Original: 2 , we solved the initial value problem of the true ODE using the leapfrog integrator with or without FINDE, with the true system energy E as the first integral V .
Revised: 2 , we solved the initial value problem of the true ODE using the leapfrog integrator with or without FINDE, with the true system energy E as the first integral V .

Original: Notably, no neural networks nor training were involved.
Revised: Notably, no neural networks nor training were involved.

Original: The results, along with the analytical solution, are illustrated in Fig.
Revised: Figure 1 shows the results, along with the analytical solution.

Original: The upper panel shows that the time-series predicted by comparison methods overlap and are apparently identical.
Revised: The states predicted by comparison methods overlap and are apparently identical.

Original: However, thelower panel shows that the energy obtainedfrom the states predicted by the leapfrog integrator fluctuates and the same is true for cFINDE.
Revised: However, the energy obtained by the leapfrog integrator fluctuates and the same is true for cFINDE.

Original: This is because the leapfrog integrator and cFINDE suffer from temporal discretization errors in first integrals.
Revised: This is because the leapfrog integrator and cFINDE suffer from temporal discretization errors in first integrals.

Original: In contrast, dFINDE preserves the energy accurately, the same as the analytical solution.
Revised: In contrast, dFINDE preserves the energy accurately, the same as the analytical solution.

Original: This is because dFINDE projects the state ( q v ) ⊤ onto the discrete tangent space T ( v , u ) M ′ at every step.
Revised: This is because dFINDE projects the state ( q v ) ⊤ onto the discrete tangent space T ( v , u ) M ′ at every step.

Original: Although a smaller time-step size reduces numerical errors, this result demonstrates the advantage of dFINDE.
Revised: Although a smaller time-step size reduces temporal discretization errors, this result demonstrates the advantage of dFINDE.

Original: See Appendix D.1 for the case with the Dormand-Prince integrator.
Revised: See Appendix D.1 for the case with the Dormand-Prince integrator.

Original: 4. F INDING N ON -H AMILTONIAN F IRST I NTEGRALS OF H AMILTONIAN S YSTEMS
Revised: 4. F INDING N ON -H AMILTONIAN F IRST I NTEGRALS OF H AMILTONIAN S YSTEMS

Original: We evaluated FINDE on learning from the 2-body dataset.
Revised: We evaluated cFINDE and dFINDE on learning from the 2-body dataset.

Original: We used HNN as the base model ˆ f .
Revised: We used HNN as the base model ˆ f .

Original: We found that FINDE obtained better performances if it did not treat the Hamiltonian H of the HNN as one of the first integrals V k .
Revised: We found that cFINDE and dFINDE obtained better performances if it did not treat the Hamiltonian H of the HNN as one of the first integrals V k .

Original: The medians and standard deviations of five trials are summarized in the leftmost column of Table 3.
Revised: The medians and standard deviations of five trials are summarized in the leftmost column of Table 3.

Original: The cFINDE achieved better VPTs than the original HNN with K =to 2 , and its performance was suddenly degraded with K = 3 .
Revised: The cFINDE achieved better VPTs than the original HNN with K = 1 to 2 , and its performance was suddenly degraded with K = 3 .

Original: The dFINDE showed a similar trend with slightly better performances.
Revised: The dFINDE showed a similar trend with slightly better performances; there is a trade-off between performance and computational cost.

Original: The HNN with FINDE found two first integrals in addition to the Hamiltonian H of the HNN.
Revised: The HNN with either cFINDE or dFINDE found two first integrals in addition to the Hamiltonian H of the HNN.

Original: Even though a two-body problem is a Hamiltonian system that an HNN can learn, the prior knowledge that there exist first integrals other than the Hamiltonian H can be a clue that enables better learning.
Revised: Even though a two-body problem is a Hamiltonian system that an HNN can learn, the prior knowledge that there exist first integrals other than the Hamiltonian H can be a clue that enables better learning.

Original: The HNN with FINDE yielded 1-step errors worse than the HNN results, suggesting that the HNN without FINDE overfitted short-term changes and had difficulty predicting long-term dynamics.
Revised: Despite their better long-term prediction performance, the HNN with either cFINDE or dFINDE yielded 1-step errors worse than the HNN, indicating that the 1-step error is misleading as an evaluation criterion.

Original: These example results are depicted in Fig.
Revised: These example results are depicted in Fig.

Original: In the absence of FINDE, the mass positions ( x 1 , y 1 ) and ( x 2 , y 2 ) became inaccurate in a short time and the center-of-gravity position ( x c , y c ) 
Revised: In the absence of FINDE, the mass positions ( x 1 , y 1 ) and ( x 2 , y 2 ) became inaccurate in a short time and the center-of-gravity position ( x c , y c ) =

Original: =
Revised: [Table]

Original: ( x 1 + x 2, y 1 + y 2) deviated rapidly.
Revised: [Figure] [Figure] ) deviated rapidly.

Original: The HNN with cFINDE accurately predicted the state for a longer [Table] [Figure] [Figure]
Revised: The HNN with cFINDE accurately predicted the state for a longer period.

Original: period.

Original: Even after errors in the mass positions became non-negligible, errors in the center-of-gravity position were still small.
Revised: Even after errors in the mass positions became non-negligible, errors in the center-of-gravity position were still small.

Original: We show the absolute errors averaged over all trials in Fig.
Revised: Figure 3 shows the absolute errors averaged over all trials, which demonstrate how the trend changes with cFINDE.

Original: In both the xand y -directions, the HNN without FINDE produced errors in the center-of-gravity position x c (or y c ), and those in the mass positions x 1 , x 2 (or y 1 , y 2 ) at a similar level.
Revised: In both the x - and y -directions, the HNN without FINDE produced errors in the center-of-gravity position x c (or y c ), and those in the mass positions x 1 , x(or y 1 , y 2 ) at a similar level.

Original: In contrast, with the cFINDE, errors in the center-of-gravity position were much smaller than those in the mass positions, implying that errors in one mass position canceled out errors in the other.
Revised: In contrast, with the cFINDE, errors in the center-of-gravity position were much smaller than those in the mass positions, implying that errors in one mass position canceled out errors in the other.

Original: We performed a symbolic regression of first integrals V found by the neural network.
Revised: We performed a symbolic regression of first integrals V found by the neural network.

Original: For K = 2 , the found first integrals V were identical to the linear momenta in the x - and y -directions up to affine transformation in most cases.
Revised: For K = 2 , the found first integrals V were identical to the linear momenta in the x - and y -directions up to affine transformation in most cases.

Original: See Appendix D.2 for detailed results.
Revised: See Appendix D.2 for detailed results.

Original: Therefore, we conclude that FINDE not only had better prediction accuracy but also found and preserved linear momenta (which are related to symmetries in space) more accurately despite not having prior knowledge about symmetries.
Revised: Therefore, we conclude that FINDE not only had better prediction accuracy but also found and preserved linear momenta (which are related to symmetries in space) more accurately despite not having prior knowledge about symmetries.

Original: 4. F INDING F IRST I NTEGRALS OF U NKNOWN S YSTEMS
Revised: 4. F INDING F IRST I NTEGRALS OF U NKNOWN S YSTEMS

Original: It is often unclear whether a target system is a Hamiltonian system or not, but one can expect that it has several first integrals.
Revised: It is often unclear whether a target system is a Hamiltonian system or not, but one can expect that it has several first integrals.

Original: We evaluated FINDE using NODE as the base model and display the results in Table 3.
Revised: We evaluated cFINDE and dFINDE using NODE as the base model and display the results in Table 3.

Original: For the KdV dataset, the NODE with FINDE obtained improved1-step errors and VPTs for a wide range of K .
Revised: For the KdV dataset, the NODE with either cFINDE or dFINDE obtained improved VPTs for a wide range of K .

Original: Figure 4 shows an example result.
Revised: Figure 4 shows an example result.

Original: Thetop panels show that the prediction results were apparently similar.
Revised: The prediction states were apparently similar.

Original: The bottom panels summarize the mean absolute errors in states u , total mass (cid:80) k u k , and energy.

Original: In the absence of FINDE, the NODE increased all of its errors in proportion to time.
Revised: In the absence of FINDE, the NODE increased all of its errors in proportion to time.

Original: Withthe cFINDE, the error in total mass increased at the point where the two solitons collided, but then returned to the original level.
Revised: With cFINDE, the error in total mass increased at the point where the two solitons collided, but then returned to the original level.

Original: Although the calculation is slightly inaccurate, the cFINDE learned to preserve the total mass.
Revised: Although the calculation is slightly inaccurate, the cFINDE learned to preserve the total mass.

Original: Therightmost panel shows that the error in energy continued to 
Revised: The error in energy continued to increase for K = 2 , but remained within a small range for K = 3 . These results suggest that the first or second quantity learned by the cFINDE was total

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: increase for K = 2 , but remained within a small range for K = 3 . These results suggest that the first or second quantity learned by the cFINDE was total mass, the third quantity was system energy, and the remaining quantity may correspond to one of the many first integrals of the KdV equation.
Revised: mass, the third quantity was system energy, and the remaining quantity may correspond to one of the many first integrals of the KdV equation.

Original: For the 2-pend dataset, the NODE with FINDE obtained improved1-step errors and VPTs with
Revised: For the 2-pend dataset, the NODE with either cFINDE or dFINDE obtained improved VPTs with

Original: K = 1 to 5; the sole exception is the 1-step error of the dFINDE with K = 5 .
Revised: K = 1 to 5 .

Original: In addition to the system energy, the double pendulum has two holonomic constraints on the position, which lead to two additional constraints involving the velocity (see Appendix C for details).
Revised: In addition to the system energy, the double pendulum has two holonomic constraints on the position, which lead to two additional constraints involving the velocity (see Appendix C for details).

Original: Thus, it is reasonable that the NODE with FINDE obtained the best VPTs for K = 5 first integrals and completely failed when assuming K > 5 first integrals.
Revised: Thus, it is reasonable that the NODE with either cFINDE or dFINDE obtained the best VPT for K = 5 first integrals and completely failed for K > 5 first integrals.

Original: As exemplified in Fig.
Revised: As exemplified in Fig.

Original: 5, the NODE without FINDE did not preserve the lengths of rods, making the states deviate gradually.
Revised: 5, the NODE without FINDE did not preserve the lengths of rods, making the states deviate gradually.

Original: See Appendix D.3 for the case when actual constraints are known.
Revised: See Appendix D.3 for the case when actual constraints are known.

Original: For the FitzHugh–Nagumo dataset, the NODE with FINDE obtained improved1-step errors and VPTs for K = 2 .
Revised: For the FitzHugh–Nagumo dataset, the NODE with either cFINDE or dFINDE obtained improved VPTs for K = 2 .

Original: As exemplified in Fig.
Revised: As exemplified in Fig.

Original: 6, the ground truth state converged to a periodic orbit, and only the NODE with cFINDE for K = 2 reproduced similar dynamics.
Revised: 6, the ground truth state converged to a periodic orbit, and only the NODE with cFINDE for K = 2 reproduced similar dynamics.

Original: Without FINDE, the state did not remain in a limited region.
Revised: Without FINDE, the state did not remain in a limited region.

Original: For K = 1 , the state converged to a wrong equilibrium; the sole quantity V 1 may have attempted  to learn both first integrals and remained under-trained.
Revised: For K = 1 , the state converged to a wrong equilibrium; the sole quantity V 1 may have attempted and failed to learn both first integrals.

Original: We conclude that FINDE found all first integrals of the 2-pend and FitzHugh–Nagumo datasets;the numbers were K = 5 and K = 2 , respectively.
Revised: We conclude that both cFINDE and dFINDE found all first integrals of the 2-pend and FitzHugh–Nagumo datasets; K = 5 and K = 2 , respectively.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This study proposed first integral-preserving neural differential equation (FINDE), which can find and preserve any type of first integrals from data in a unified manner.
Revised: This study proposed first integral-preserving neural differential equation (FINDE), which can find and preserve any type of first integrals from data in a unified manner.

Original: FINDE projects the time evolution onto the submanifold defined using the (discrete) gradients of first integrals represented by a neural network.
Revised: FINDE projects the time evolution onto the submanifold defined using the (discrete) gradients of first integrals represented by a neural network.

Original: We experimentally demonstrated that FINDE found and preserved first integrals that come from the energy and mass conservation laws, symmetries in space, and constraints, thereby predicting the dynamics for far longer.
Revised: We experimentally demonstrated that FINDE found and preserved first integrals that come from the energy and mass conservation laws, symmetries in space, and constraints, thereby predicting the dynamics for far longer.

Original: FINDE is available even for an energy-dissipating system.
Revised: FINDE is available even for an energy-dissipating system.

Original: When FINDE obtains the best prediction accuracy with K = K ′ , it suggests that the target system has at least K ′ first integrals.
Revised: When FINDE obtains the best prediction accuracy with K = K ′ , it suggests that the target system has at least K ′ first integrals.

Original: Hence, FINDE has the potential to make scientific discoveries by revealing geometric structures of dynamical systems.
Revised: Hence, FINDE has the potential to make scientific discoveries by revealing geometric structures of dynamical systems.

Revised: See Appendix D.4 for more discussions on K .

Original: The numerical error tolerance 10 − 9 was negligible compared to the 1-step errors (which were 10 −to 10 − 4 in absolute error).
Revised: The numerical error tolerance 10 − 9 was negligible compared to the 1-step errors (which were 10 −to 10 − 4 in absolute error).

Original: However, the dFINDE tended to obtain much better VPTs than the cFINDE.
Revised: However, the dFINDE tended to obtain much better VPTs than the cFINDE.

Original: This result suggests that a method leading to smaller numerical errors produces a model with smaller modeling errors, as observed in previous works (Chen et al., 2020; Matsubara et al., 2020).
Revised: This result suggests that a method leading to smaller numerical errors produces a model with smaller modeling errors, as observed in previous works (Chen et al., 2020; Matsubara et al., 2020).

Original: These results may form a new frontier for integrating numerical and modeling errors.
Revised: These results may form a new frontier for integrating numerical and modeling errors.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: See Section 4.1 for experimental settings.
Revised: See Section 4.1 for experimental settings.

Original: More detailed descriptions can be found in Appendix B. for training procedure and Appendix C for datasets.
Revised: More detailed descriptions can be found in Appendix B. for training procedure and Appendix C for datasets.

Original: The authors have enclosed the source code for generating the datasets and running the experiments as supplementary material
Revised: The authors have enclosed the source code for generating the datasets and running the experiments as supplementary material

Original: A H AMILTONIAN S YSTEM , ITS G ENERALIZATION , AND F IRST I NTEGRALS
Revised: A H AMILTONIAN S YSTEM , ITS G ENERALIZATION , AND F IRST I NTEGRALS

Original: Preliminary In this section, we briefly introduce potential target systems and related works.
Revised: Preliminary In this section, we briefly introduce potential target systems and related works.

Original: Methods proposed by related works use specific prior knowledge about target systems, such as constraints.
Revised: Methods proposed by related works use specific prior knowledge about target systems, such as constraints.

Original: In contrast, our proposed FINDE assumes a situation where neural networks learn systems with unknown properties.
Revised: In contrast, our proposed FINDE assumes a situation where neural networks learn systems with unknown properties.

Original: See, for example, Hairer et al.
Revised: See, for example, Hairer et al.

Original: (2006); van der Schaft & Jeltsema (2014) for more details about geometric mechanics.
Revised: (2006); van der Schaft & Jeltsema (2014) for more details about geometric mechanics.

Original: On an N -dimensional manifold M , an ODE is defined using a vector field f : M → T u M , which maps a point u on the manifold M to a tangent vector f ( u ) on the tangent space T u M .
Revised: On an N -dimensional manifold M , an ODE is defined using a vector field f : M → T u M , which maps a point u on the manifold M to a tangent vector f ( u ) on the tangent space T u M .

Original: The NODE defines an ODE in this way (Chen et al., 2018).
Revised: The NODE defines an ODE in this way (Chen et al., 2018).

Original: Given a scalar-valued function H : M
Revised: Given a scalar-valued function H : M

Original: → R on the manifold M , its differential d H : M → T ∗ u M is a cotangent vector field (a.k.a.
Revised: → R on the manifold M , its differential d H : M → T ∗ u M is a cotangent vector field (a.k.a.

Original: a differential 1-form), which maps a point u on the manifold M to a cotangent vector d H ( u ) on the cotangent space T ∗ u M .
Revised: a differential 1-form), which maps a point u on the manifold M to a cotangent vector d H ( u ) on the cotangent space T ∗ u M .

Original: Hamiltoanian System
Revised: Hamiltoanian System

Original: A Hamiltonian system is defined using a non-degenerate closed differential 2-form ω called symplectic form, which is a skew-symmetric bilinear map ω
Revised: A Hamiltonian system is defined using a non-degenerate closed differential 2-form ω called symplectic form, which is a skew-symmetric bilinear map ω

Original: u : T u M × T u M → R at point u .
Revised: u : T u M × T u M → R at point u .

Original: A symplectic form assigned to a manifold is called the symplectic structure.
Revised: A symplectic form assigned to a manifold is called the symplectic structure.

Original: The coordinate-free form of Hamilton’s equation is dd t u =
Revised: The coordinate-free form of Hamilton’s equation is dd t u =

Original: X H ( u ) ,
Revised: X H ( u ) ,

Original: ω u ( X H ( u ) , w ) = ⟨ d H ( u ) , w ⟩ for any w ∈ T u M , where X H is the Hamiltonian vector field.
Revised: ω u ( X H ( u ) , w ) = ⟨ d H ( u ) , w ⟩ for any w ∈ T u M , where X H is the Hamiltonian vector field.

Original: The symplectic form ω gives rise to a bundle map ω ♭ u : T u M → T ∗ u M , with which Hamilton’s equation is rewritten as dd t u =
Revised: The symplectic form ω gives rise to a bundle map ω ♭ u : T u M → T ∗ u M , with which Hamilton’s equation is rewritten as dd t u =

Original: X H ( u ) = ( ω ♭ u ) − 1 (d H ( u )) .
Revised: X H ( u ) = ( ω ♭ u ) − 1 (d H ( u )) .

Original: The right-hand side is locally equivalent to the product of a coefficient matrix S and the gradient ∇ H of the Hamiltonian H .
Revised: The right-hand side is locally equivalent to the product of a coefficient matrix S and the gradient ∇ H of the Hamiltonian H .

Original: Then, Hamilton’s equation is obtained as dd t u = S ∇ H ( u ) .
Revised: Then, Hamilton’s equation is obtained as dd t u = S ∇ H ( u ) .

Original: Hamiltonian systems are often expressed in the canonical form, in other words, they are defined on Darboux coordinates, on which the state u is the paired generalized position q and generalized momentum p .
Revised: Hamiltonian systems are often expressed in the canonical form, in other words, they are defined on Darboux coordinates, on which the state u is the paired generalized position q and generalized momentum p .

Original: The corresponding coefficient matrix is S = (cid:0) 

Revised: The corresponding coefficient matrix is S = (cid:0)

Original: I n − I n 0 (cid:1) for 2 n = N and the n -dimensional identity matrix I n .
Revised: I n − I n 0 (cid:1) for 2 n = N and the n -dimensional identity matrix I n .

Original: The HNN was developed to model Hamiltonian systems in the canonical forms (Greydanus et al., 2019).
Revised: The HNN was developed to model Hamiltonian systems in the canonical forms (Greydanus et al., 2019).

Original: An Euler–Lagrange equation with a hyperregular Lagrangian and a Lotka–Volterra equation are also Hamiltonian systems; however, their coordinate systems are not Darboux coordinates.
Revised: An Euler–Lagrange equation with a hyperregular Lagrangian and a Lotka–Volterra equation are also Hamiltonian systems; however, their coordinate systems are not Darboux coordinates.

Original: A neural symplectic form (NSF) handles this class of equations (Chen et al., 2021).
Revised: A neural symplectic form (NSF) handles this class of equations (Chen et al., 2021).

Original: The KdV equation is also a Hamiltonian system not on Darboux coordinates.
Revised: The KdV equation is also a Hamiltonian system not on Darboux coordinates.

Original: For Hamiltonian PDE systems, HNN++ was proposed (Matsubara et al., 2020).
Revised: For Hamiltonian PDE systems, HNN++ was proposed (Matsubara et al., 2020).

Original: According to Darboux’s theorem, any Hamiltonian system on an even–dimensional manifold can be transformed into the canonical form.
Revised: According to Darboux’s theorem, any Hamiltonian system on an even–dimensional manifold can be transformed into the canonical form.

Original: Noether’s theorem states that a continuous symmetry of a system leads to a conservation law.
Revised: Noether’s theorem states that a continuous symmetry of a system leads to a conservation law.

Original: A
Revised: A

Original: Hamiltonian system is symmetric (invariant) to translation in time and conserves the Hamiltonian
Revised: Hamiltonian system is symmetric (invariant) to translation in time and conserves the Hamiltonian

Original: H .
Revised: H .

Original: A two-body problem is symmetric to translation and rotation in space and conserves linear and angular momenta.
Revised: A two-body problem is symmetric to translation and rotation in space and conserves linear and angular momenta.

Original: These quantities are first integrals.
Revised: These quantities are first integrals.

Original: LieConv and EMLP-HNN had such symmetries implemented in their architectures (Finzi et al., 2020a; 2021).
Revised: LieConv and EMLP-HNN had such symmetries implemented in their architectures (Finzi et al., 2020a; 2021).

Original: A pendulum is not symmetric to translation and rotation in space and does not conserve linear and angular momenta, but does exchange them with the base to which it is fixed.
Revised: A pendulum is not symmetric to translation and rotation in space and does not conserve linear and angular momenta, but does exchange them with the base to which it is fixed.

Original: Poisson System A Poisson system is named after a Poisson bracket {· , ·} , but it is convenient to refer to it as a degenerate Hamiltonian system.
Revised: Poisson System A Poisson system is named after a Poisson bracket {· , ·} , but it is convenient to refer to it as a degenerate Hamiltonian system.

Original: A Poisson bracket is defined using a Poisson 2-vector
Revised: A Poisson bracket is defined using a Poisson 2-vector

Original: B , which is a skew-symmetric bilinear map B u : T ∗ u M × T ∗ u M → R at point u .
Revised: B , which is a skew-symmetric bilinear map B u : T ∗ u M × T ∗ u M → R at point u .

Original: The Poisson 2-vector B gives rise to a bundle map B ♯ u : T ∗ u M → T u M and defines Hamilton’s equation as dd t u =
Revised: The Poisson 2-vector B gives rise to a bundle map B ♯ u : T ∗ u M → T u M and defines Hamilton’s equation as dd t u =

Original: B ♯ (d H ( u )) .
Revised: B ♯ (d H ( u )) .

Original: The Darboux–Lie theorem states that any Poisson system can be transformed into the canonical form
Revised: The Darboux–Lie theorem states that any Poisson system can be transformed into the canonical form

Original: dd t u = S ∇ H ( u ) by using a matrix S = (cid:16) 

Revised: dd t u = S ∇ H ( u ) by using a matrix S = (cid:16)

Original: I k 0 − I k 0 0 0 0 0 (cid:17) for 2 k < N .
Revised: I k 0 − I k 0 0 0 0 0 (cid:17) for 2 k < N .

Original: The last
Revised: The last

Original: N − 2 k elements remain unchanged and correspond to the first integrals.
Revised: N − 2 k elements remain unchanged and correspond to the first integrals.

Original: In this sense, a Poisson system is a degenerate Hamiltonian system.
Revised: In this sense, a Poisson system is a degenerate Hamiltonian system.

Original: A Poisson 2-vector assigned to a manifold is called a
Revised: A Poisson 2-vector assigned to a manifold is called a

Original: Poisson structure.
Revised: Poisson structure.

Original: Several models of the dynamics of disease spreading and chemical reactions are
Revised: Several models of the dynamics of disease spreading and chemical reactions are

Original: Poisson systems, and total population and molecular mass are typical first integrals.
Revised: Poisson systems, and total population and molecular mass are typical first integrals.

Original: A Poisson neural network (PNN) learns to transform a given Poisson system into a canonical form (Jin et al., 2020a).
Revised: A Poisson neural network (PNN) learns to transform a given Poisson system into a canonical form (Jin et al., 2020a).

Original: Constrained Hamiltonian System A constraint C ( q ) = 0 on the position q is called a holonomic constraint.
Revised: Constrained Hamiltonian System A constraint C ( q ) = 0 on the position q is called a holonomic constraint.

Original: Holonomic constraint appear, for example, when the arm’s length restricts the position of a robot’s hand.
Revised: Holonomic constraint appear, for example, when the arm’s length restricts the position of a robot’s hand.

Original: Differentiating a holonomic constraint C ( q ) =
Revised: Differentiating a holonomic constraint C ( q ) =

Original: yields a constraint involving the velocity G ( q , v ) =
Revised: yields a constraint involving the velocity G ( q , v ) =

Original: ∂C∂ q v = 0 , which is simply called a velocity constraint.
Revised: ∂C∂ q v = 0 , which is simply called a velocity constraint.

Original: Hence, each holonomic constraint leads to two first integrals C and G .
Revised: Hence, each holonomic constraint leads to two first integrals C and G .

Original: A Hamiltonian system with holonomic constraints is also a Poisson system; in particular, it is a constrained Hamiltonian system.
Revised: A Hamiltonian system with holonomic constraints is also a Poisson system; in particular, it is a constrained Hamiltonian system.

Original: A CHNN incorporates the known holonomic constraints C ( q ) and corresponding velocity constraints G ( q , v ) of a Hamiltonian system in the canonical form (Finzi et al., 2020b).
Revised: A CHNN incorporates the known holonomic constraints C ( q ) and corresponding velocity constraints G ( q , v ) of a Hamiltonian system in the canonical form (Finzi et al., 2020b).

Original: The original study suggested that CHNN may learn holonomic constraints from data, but this has not been tested.
Revised: The original study suggested that CHNN may learn holonomic constraints from data, but this has not been tested.

Original: For modeling a constrained Hamiltonian system, it is sufficient to incorporate only velocity constraints G ( q , v ) because a holonomic constraint C ( q ) is implicitly satisfied if the corresponding velocity constraint G ( q , v ) is satisfied.
Revised: For modeling a constrained Hamiltonian system, it is sufficient to incorporate only velocity constraints G ( q , v ) because a holonomic constraint C ( q ) is implicitly satisfied if the corresponding velocity constraint G ( q , v ) is satisfied.

Original: Celledoni et al.
Revised: Celledoni et al.

Original: (2022) used such formulation, and extended
Revised: (2022) used such formulation, and extended

Original: HNN and CHNN to systems on non-Euclidean spaces.
Revised: HNN and CHNN to systems on non-Euclidean spaces.

Original: A neural projection method learns fixed holonomic constraints, as well as inequality constraints, which are outside the scope of this study (Yang et al., 2020).
Revised: A neural projection method learns fixed holonomic constraints, as well as inequality constraints, which are outside the scope of this study (Yang et al., 2020).

Original: This method updates the state by solving an optimization problem similar to Eq.
Revised: This method updates the state by solving an optimization problem similar to Eq.

Original: (3) iteratively using the gradient descent method at every training step.
Revised: (3) iteratively using the gradient descent method at every training step.

Original: Subsequently, it applies the backpropagation algorithm to all the optimization iterations.
Revised: Subsequently, it applies the backpropagation algorithm to all the optimization iterations.

Original: Thus, it has high computational and memory costs.
Revised: Thus, it has high computational and memory costs.

Original: These studies mainly focused on physically-induced holonomic constraints and may not work for other first integrals, as shown in Appendices D.3 and D.4.
Revised: These studies mainly focused on physically-induced holonomic constraints and may not work for other first integrals, as shown in Appendices D.3 and D.5.

Original: However, the purpose of FINDE is to find and preserve general first integrals, including energy and mass not limited by constraints.
Revised: However, the purpose of FINDE is to find and preserve general first integrals, including energy and mass not limited by constraints.

Original: A concurrent study, “Constants-of-motion network,” introduced the penalty loss function so that neural ODEs learn to preserve first integrals (Kasim & Lim, 2022); however, unlike other related methods, this method does not guarantee preservation.

Original: A Noether network was proposed to model videos that do not always capture physical phenomena (Alet et al., 2021).

Original: A subset of the latent variable is assumed to represent image features that do not change during a video, such as the appearance of objects.

Original: For prediction, these features are forced not to change.

Original: The Noether network is potentially useful for learning physical phenomena from videos, but is more similar to semantic manipulation of latent variables (Shen et al., 2020).

Original: Dirac Structure A Dirac structure is named after a Dirac bracket, a generalization of the Poisson bracket (van der Schaft & Jeltsema, 2014), and can be found in various systems.
Revised: Dirac Structure A Dirac structure is named after a Dirac bracket, a generalization of the Poisson bracket (van der Schaft & Jeltsema, 2014), and can be found in various systems.

Original: For a rolling disk, the direction in which the disk can move forward without slipping is limited by the disk’s orientation.
Revised: For a rolling disk, the direction in which the disk can move forward without slipping is limited by the disk’s orientation.

Original: This constraint is called a non-holonomic constraint.
Revised: This constraint is called a non-holonomic constraint.

Original: In an electric circuit, when elements are connected in series, the current flow through each element is always the same.
Revised: In an electric circuit, when elements are connected in series, the current flow through each element is always the same.

Original: This constraint is called Kirchhoff’s current law.
Revised: This constraint is called Kirchhoff’s current law.

Original: One can find Dirac structures in these systems.
Revised: One can find Dirac structures in these systems.

Original: The dissipative
Revised: The dissipative

Original: SymODEN was proposed to model a port-Hamiltonian system in the canonical form (Zhong et al., 2020b), which is a special case of the Dirac structure.
Revised: SymODEN was proposed to model a port-Hamiltonian system in the canonical form (Zhong et al., 2020b), which is a special case of the Dirac structure.

Original: To the best of our knowledge, a neural network model for a general Dirac structure has not yet been proposed.
Revised: To the best of our knowledge, a neural network model for a general Dirac structure has not yet been proposed.

Original: FINDE is the first neural network method to learn Dirac structures better than NODE can, even though it is not specialized for Dirac structures.
Revised: FINDE is the first neural network method to learn Dirac structures better than NODE can, even though it is not specialized for Dirac structures.

Original: PDE with Mass Conservation The total mass of a PDE system is sometimes preserved (Furihata & Matsuo, 2010).
Revised: with Mass Conservation The total mass of a PDE system is sometimes preserved (Furihata & Matsuo, 2010).

Original: The KdV equation is a Hamiltonian system that describes shallow water waves, in which the energy and total mass are preserved.
Revised: The KdV equation is a Hamiltonian system that describes shallow water waves, in which the energy and total mass are preserved.

Original: The Cahn–Hilliard equation is a model of phase separation of copolymer melts, in which the total mass is preserved, but the energy is dissipated.
Revised: The Cahn–Hilliard equation is a model of phase separation of copolymer melts, in which the total mass is preserved, but the energy is dissipated.

Original: In general, a quantity in an area is preserved if its flux entering minus its flux leaving is zero.
Revised: In general, a quantity in an area is preserved if its flux entering minus its flux leaving is zero.

Original: Deep conservation extracts latent dynamics of a PDE system and preserves a quantity of interest by forcing its flux to be zero (Lee & Carlberg, 2021).
Revised: Deep conservation extracts latent dynamics of a PDE system and preserves a quantity of interest by forcing its flux to be zero (Lee & Carlberg, 2021).

Original: HNN++ also ensures mass conservation by designing a coefficient matrix that determines local interaction (Matsubara et al., 2020).
Revised: HNN++ also ensures mass conservation by designing a coefficient matrix that determines local interaction (Matsubara et al., 2020).

Revised: General First Intergals A concurrent study, “Constants-of-motion network,” introduced the penalty loss function so that NODEs learn to preserve first integrals (Kasim & Lim, 2022); however, unlike other related methods, this method does not guarantee preservation.

Revised: A Noether network was proposed to model videos that do not always capture physical phenomena (Alet et al., 2021).

Revised: A subset of the latent variable is assumed to represent image features that do not change during a video, such as the appearance of objects.

Revised: For prediction, these features are forced not to change.

Revised: The Noether network is potentially useful for learning physical phenomena from videos, but is more similar to semantic manipulation of latent variables (Shen et al., 2020).

Original: B D ETAILS OF M ETHODS B. D ERIVATION OF FINDE 
Revised: Some studies have investigated methods that do not predict dynamics but specialize in finding first integrals (Fukunaga & Olsen, 1971; Liu & Tegmark, 2021). These methods can be used to help FINDE determine the hyperparameter K . They commonly estimate the number ( N − K ) of dimensions of the tangent space T u M ′ of the submanifold M ′ at point u using its neighbors.

Revised: For example,

Revised: AI Poincar´e proposed by Liu & Tegmark (2021) assumes that all data points share the submanifold

Revised: M ′ and uses an autoencoder to reconstruct the tangent space T u M ′ .

Revised: Hence, it can only process a single long time series with fixed first integrals.

Revised: In contrast, our proposed FINDE can leverage a dataset of multiple time series with different values of the first integrals.

Revised: B D ETAILS OF M ETHODS B. D ERIVATION OF FINDE

Original: Continuous FINDE (cFINDE) Let u s denote a current state and ˆ f denote a vector field.
Revised: Continuous FINDE (cFINDE) Let u s denote a current state and ˆ f denote a vector field.

Original: After a time interval ∆ t , the state transitions to ˆ u s +1 .
Revised: After a time interval ∆ t , the state transitions to ˆ u s +1 .

Original: A typical projection method projects the state ˜ u s +1 onto a submanifold M ′ and obtains a state u s +1 , which preserves the first integrals V
Revised: A typical projection method projects the state ˜ u s +1 onto a submanifold M ′ and obtains a state u s +1 , which preserves the first integrals V

Original: =
Revised: =

Original: ( V 1 . . .
Revised: ( V 1 . . .

Original: V K ) ⊤ .
Revised: V K ) ⊤ .

Original: This procedure is defined as an optimization problem in Eq.
Revised: This procedure is defined as an optimization problem in Eq.

Original: [Equation]
Revised: [Equation]

Original: One can solve the problem using the method of Lagrange multipliers.
Revised: One can solve the problem using the method of Lagrange multipliers.

Original: A Lagrangian function is
Revised: A Lagrangian function is

Original: [Equation]
Revised: [Equation]

Original: where λ ′ is the Lagrange multiplier.
Revised: where λ ′ is the Lagrange multiplier.

Original: The stationary point satisfies
Revised: The stationary point satisfies

Original: [Equation]
Revised: [Equation]

Original: Subsequently [Equation] a projection method can be redefined as
Revised: Subsequently, a projection method can be redefined as

Original: [Equation]
Revised: [Equation]

Original: We transform Eq.
Revised: We transform Eq.

Original: (A4) into
Revised: (A4) into

Original: [Equation]
Revised: [Equation]

Original: where λ = λ ′ / ∆ t .
Revised: where λ = λ ′ / ∆ t .

Original: Taking the limit as ∆ t → +0 , we obtain Eq.
Revised: Taking the limit as ∆ t → +0 , we obtain Eq.

Original: [Equation]
Revised: [Equation]

Original: The second equation ensures that a state transition following the new vector field f preserves the first integrals V .
Revised: The second equation ensures that a state transition following the new vector field f preserves the first integrals V .

Original: By eliminating the Lagrange multiplier λ ( u ) , we define the cFINDE as in Eq.
Revised: By eliminating the Lagrange multiplier λ ( u ) , we define the cFINDE as in Eq.

Original: (6), that is,
Revised: (6), that is,

Original: [Equation]
Revised: [Equation]

Original: where M = ∂ V ∂ u .
Revised: where M = ∂ V ∂ u .

Original: Because of the above derivation, the cFINDE can be considered a continuous-time version of a projection method.
Revised: Because of the above derivation, the cFINDE can be considered a continuous-time version of a projection method.

Original: The preservation of first integrals can be proved as follows.
Revised: The preservation of first integrals can be proved as follows.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: [Equation]
Revised: [Equation]

Original: Hence, it holds that dd t V k ( u ) = 0 for k = 1 , . . .
Revised: Hence, it holds that dd t V k ( u ) = 0 for k = 1 , . . .

Original: , K , indicating that the cFINDE dd t u = f ( u ) preserves all first integrals V k in continuous time.
Revised: , K , indicating that the cFINDE dd t u = f ( u ) preserves all first integrals V k in continuous time.

Original: Discrete FINDE (dFINDE)
Revised: Discrete FINDE (dFINDE)

Original: For dFINDE, we take the discrete gradient of the Lagrangian equation in Eq.
Revised: For dFINDE, we take the discrete gradient of the Lagrangian equation in Eq.

Original: (A2)
Revised: (A2)

Original: and obtain the discrete version of the necessary conditions for the stationary point;
Revised: and obtain the discrete version of the necessary conditions for the stationary point;

Original: [Equation]
Revised: [Equation]

Original: M ( u s +1 , u s ) corresponds to the Jacobian ∂ V ∂ u .
Revised: M ( u s +1 , u s ) corresponds to the Jacobian ∂ V ∂ u .

Original: By substituting the base model ˜ u
Revised: By substituting the base model ˜ u

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ˆ ψ ( u s ; ∆ t s ) and the dFINDE u
Revised: ˆ ψ ( u s ; ∆ t s ) and the dFINDE u

Original: s +1 − u s ∆ t s = ψ
Revised: s +1 − u s ∆ t s = ψ

Original: [Equation]
Revised: [Equation]

Original: u s
Revised: u s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: u s ; ∆ t s ) into the above equation and dividing the first equation by ∆ t , we obtain Eq.
Revised: u s ; ∆ t s ) into the above equation and dividing the first equation by ∆ t , we obtain Eq.

Original: [Equation]
Revised: [Equation]

Original: where λ = λ ′ /
Revised: where λ = λ ′ /

Original: ∆ t s .
Revised: ∆ t s .

Original: By eliminating the Lagrange multiplier λ , we define the dFINDE as in Eq.
Revised: By eliminating the Lagrange multiplier λ , we define the dFINDE as in Eq.

Original: (10), that is,
Revised: (10), that is,

Original: [Equation]
Revised: [Equation]

Original: The preservation of first integrals can be proved as follows.
Revised: The preservation of first integrals can be proved as follows.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: [Equation]
Revised: [Equation]

Original: Hence, it holds that V k ( u s +1 ) =
Revised: Hence, it holds that V k ( u s +1 ) =

Original: V k
Revised: V k

Original: ( u s ) for k = 1 , . . .
Revised: ( u s ) for k = 1 , . . .

Original: , K , indicating that the dFINDE u s +1 − u s ∆ t s = ψ ( u s +1 , u s ; ∆ t s ) preserves all first integrals V k in discrete time.
Revised: , K , indicating that the dFINDE u s +1 − u s ∆ t s = ψ ( u s +1 , u s ; ∆ t s ) preserves all first integrals V k in discrete time.

Original: B. D ISCRETE G RADIENT
Revised: B. D ISCRETE G RADIENT

Original: A discrete gradient is a discrete analogue to a gradient (Furihata & Matsuo, 2010; Gonzalez, 1996;
Revised: A discrete gradient is a discrete analogue to a gradient (Furihata & Matsuo, 2010; Gonzalez, 1996;

Original: Hong et al., 2011).
Revised: Hong et al., 2011).

Original: Discrete gradients that satisfy Definition 2 are not unique, and many variations have been proposed.
Revised: Discrete gradients that satisfy Definition 2 are not unique, and many variations have been proposed.

Original: For a neural network, Matsubara et al.
Revised: For a neural network, Matsubara et al.

Original: (2020) proposed the automatic discrete differentiation algorithm (ADDA).
Revised: (2020) proposed the automatic discrete differentiation algorithm (ADDA).

Original: We briefly introduce the algorithm in the case of finitedimensional Euclidean spaces.
Revised: We briefly introduce the algorithm in the case of finitedimensional Euclidean spaces.

Original: The differential d g of a function g : R N → R M is a linear operator d g u : R N → R M at point u and satisfies
Revised: The differential d g of a function g : R N → R M is a linear operator d g u : R N → R M at point u and satisfies

Original: [Equation]
Revised: [Equation]

Original: The differential d g acting on a vector w is equivalent to the product of a vector w with the Jacobian
Revised: The differential d g acting on a vector w is equivalent to the product of a vector w with the Jacobian

Original: J g ( u ) of the function g at point u : d g u ( w ) =
Revised: J g ( u ) of the function g at point u : d g u ( w ) =

Original: J g ( u ) w .
Revised: J g ( u ) w .

Original: Similarly, according to the chain rule, the differential d ( h ◦ g ) of a composition h ◦ g of functions g, h is equivalent to the multiplication with a series J h ( g ( u ))
Revised: Similarly, according to the chain rule, the differential d ( h ◦ g ) of a composition h ◦ g of functions g, h is equivalent to the multiplication with a series J h ( g ( u ))

Original: J g ( u ) of Jacobians.
Revised: J g ( u ) of Jacobians.

Original: Therefore, the automatic differentiation algorithm obtains the differential of a neural network.
Revised: Therefore, the automatic differentiation algorithm obtains the differential of a neural network.

Original: The differential d g of a function g : R N → R is a horizontal vector, and the gradient ∇ g of the function g is a vertical vector dual to the differential.
Revised: The differential d g of a function g : R N → R is a horizontal vector, and the gradient ∇ g of the function g is a vertical vector dual to the differential.

Original: Therefore, the gradient ∇ g is obtained by transposing the differential d g .
Revised: Therefore, the gradient ∇ g is obtained by transposing the differential d g .

Original: The ADDA replaces each Jacobian with its discrete analogue.
Revised: The ADDA replaces each Jacobian with its discrete analogue.

Original: For linear layers, such as fully-connected and convolution layers, the discrete Jacobian is identical to the ordinary Jacobian.
Revised: For linear layers, such as fully-connected and convolution layers, the discrete Jacobian is identical to the ordinary Jacobian.

Original: For element-wise nonlinear layers, such as activation functions, a diagonal matrix composed of the slopes between two inputs can act as the discrete Jacobian.
Revised: For element-wise nonlinear layers, such as activation functions, a diagonal matrix composed of the slopes between two inputs can act as the discrete Jacobian.

Original: A discrete gradient obtained by the above steps satisfies Definition 2.
Revised: A discrete gradient obtained by the above steps satisfies Definition 2.

Original: B. P REDICTION AND T RAINING P ROCEDURES
Revised: B. P REDICTION AND T RAINING P ROCEDURES

Original: For ODEs modeled by neural networks, various training and prediction strategies have been proposed to date (Chen et al., 2018; 2020;
Revised: For ODEs modeled by neural networks, various training and prediction strategies have been proposed to date (Chen et al., 2018; 2020;

Original: Course et al., 2020; Matsubara et al., 2020; Zhong et al., 2020a); FINDE can adopt any of these.
Revised: Course et al., 2020; Matsubara et al., 2020; Zhong et al., 2020a); FINDE can adopt any of these.

Original: In our experiments, we used the following simple strategies.
Revised: In our experiments, we used the following simple strategies.

Original: In the case of the cFINDE and base models, taking a state u s GT from the dataset, a numerical integrator solves the ODE dd t u = f ( u ) and predicts the next state u s +1 pred. .
Revised: In the case of the cFINDE and base models, taking a state u s GT from the dataset, a numerical integrator solves the ODE dd t u = f ( u ) and predicts the next state u s +1 pred. .

Original: This process can be informally expressed as
Revised: This process can be informally expressed as

Original: [Equation]
Revised: [Equation]

Original: We solved this integration using torchdiffeq.odeint .
Revised: We solved this integration using torchdiffeq.odeint .

Original: The prediction accuracy can be evaluated using the difference between the predicted state u
Revised: The prediction accuracy can be evaluated using the difference between the predicted state u

Original: s +1 pred.
Revised: s +1 pred.

Original: and ground truth u s +1 GT taken from the dataset.
Revised: and ground truth u s +1 GT taken from the dataset.

Original: We normalized the difference by the time-step size ∆ t s and defined the 1-step error L 1-step as
Revised: We normalized the difference by the time-step size ∆ t s and defined the 1-step error L 1-step as

Original: [Equation]
Revised: [Equation]

Original: The cFINDE and base models were trained to minimize the 1-step error L 1-step .
Revised: The cFINDE and base models were trained to minimize the 1-step error L 1-step .

Original: In the case of the dFINDE, the next state u s +1 pred. is predicted by solving Eq.
Revised: In the case of the dFINDE, the next state u s +1 pred. is predicted by solving Eq.

Original: (10) as an implicit scheme; in particular,
Revised: (10) as an implicit scheme; in particular,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Therefore, prediction by the dFINDE is implicit.
Revised: Therefore, prediction by the dFINDE is implicit.

Original: For evaluation, we solved this scheme using scipy.
Revised: For evaluation, we solved this scheme using scipy.

Original: optimize.fsolve and obtained the 1-step error in Eq.
Revised: optimize.fsolve and obtained the 1-step error in Eq.

Original: (A13).
Revised: (A13).

Original: However, during the training phase, the ground truth u s +1 GT of the next state is known.
Revised: However, during the training phase, the ground truth u s +1 GT of the next state is known.

Original: Hence, we substituted this into Eq.
Revised: Hence, we substituted this into Eq.

Original: (10), and then used the difference between the left- and right-hand sides of the dFINDE as the loss function:
Revised: (10), and then used the difference between the left- and right-hand sides of the dFINDE as the loss function:

Original: [Equation]
Revised: [Equation]

Original: The discrete Jacobian M (and hence Y ) can be obtained explicitly, and an explicit numerical integrator can be used for the base model ˆ ψ .
Revised: The discrete Jacobian M (and hence Y ) can be obtained explicitly, and an explicit numerical integrator can be used for the base model ˆ ψ .

Original: Hence, the process to obtain the value of the loss function is explicit, and the dFINDE can be trained in an explicit way, whereas the prediction is still implicit.
Revised: Hence, the process to obtain the value of the loss function is explicit, and the dFINDE can be trained in an explicit way, whereas the prediction is still implicit.

Original: Some previous studies have proposed alternative strategies.
Revised: Some previous studies have proposed alternative strategies.

Original: For example, a loss function can be defined as the sum of the errors at multiple time points during a long-term prediction.
Revised: For example, a loss function can be defined as the sum of the errors at multiple time points during a long-term prediction.

Original: The cFINDE can naturally adopt such a training strategy, and the dFINDE can adopt it after a minor modification.
Revised: The cFINDE can naturally adopt such a training strategy, and the dFINDE can adopt it after a minor modification.

Original: While it is helpful to pursue absolute performance, it requires additional hyperparameters, such as the length of prediction time, and additional effort to adjust them.
Revised: While it is helpful to pursue absolute performance, it requires additional hyperparameters, such as the length of prediction time, and additional effort to adjust them.

Original: We used the 1-step error in the present study for simplicity and fair comparisons.
Revised: We used the 1-step error in the present study for simplicity and fair comparisons.

Original: The function V ( u ) learning a first integral may become a constant function during training; subsequently, its Jacobian matrix vanishes ( ∂V ( u ) ∂ u ≡ 0 ).
Revised: The function V ( u ) learning a first integral may become a constant function during training; subsequently, its Jacobian matrix vanishes ( ∂V ( u ) ∂ u ≡ 0 ).

Original: In this case, our algorithm returns a division-byzero error because it requires the inverse of the matrix ∂V ( u ) ∂ u ∂V ( u ) ∂ u ⊤ for the projection.
Revised: In this case, our algorithm returns a division-byzero error because it requires the inverse of the matrix ∂V ( u ) ∂ u ∂V ( u ) ∂ u ⊤ for the projection.

Original: We have not taken any special measures to prevent such errors, but no errors occurred in any experiments with proper settings.
Revised: We have not taken any special measures to prevent such errors, but no errors occurred in any experiments with proper settings.

Original: The division-by-zero errors have occurred only when FINDE assumes an unreasonable number of first integrals (e.g., K = 6 for the double pendulum, which has five first integrals).
Revised: The division-by-zero errors have occurred only when FINDE assumes an unreasonable number of first integrals (e.g., K = 6 for the double pendulum, which has five first integrals).

Original: FINDE works correctly even when the functions f ( u ) and V ( u ) learn the same first integrals; we verified such a case in Section 4.2, where both functions are known.
Revised: FINDE works correctly even when the functions f ( u ) and V ( u ) learn the same first integrals; we verified such a case in Section 4.2, where both functions are known.

Original: FINDE learns first integrals point-by-point, and the found first integral is not always consistent over the domain.
Revised: FINDE learns first integrals point-by-point, and the found first integral is not always consistent over the domain.

Original: The same can be said about the energy function of HNN, and this type of problem is an open problem for neural network models of dynamical systems.
Revised: The same can be said about the energy function of HNN, and this type of problem is an open problem for neural network models of dynamical systems.

Original: C D ETAILS OF D ATASETS
Revised: C D ETAILS OF D ATASETS

Original: To generate each dataset, we used scipy package and the Dormand–Prince method (dopri5) with the default relative tolerance of 10 − 9 , unless otherwise stated.
Revised: To generate each dataset, we used scipy package and the Dormand–Prince method (dopri5) with the default relative tolerance of 10 − 9 , unless otherwise stated.

Original: Experiments on the KdV dataset were performed with double precision, and all other experiments were performed with single precision.
Revised: Experiments on the KdV dataset were performed with double precision, and all other experiments were performed with single precision.

Original: Hamiltonian System in Canonical Form: Two-Body Problem
Revised: Hamiltonian System in Canonical Form: Two-Body Problem

Original: A gravitational two-body problem on a 2-dimensional configuration space has a state u composed of the 4-dimensional position q = ( x 1 y 1 x 2 y 2 ) ⊤ and 4-dimensional velocity v = ( v x 1 v y 1 v x 2 v y 2 ) ⊤ .
Revised: A gravitational two-body problem on a 2-dimensional configuration space has a state u composed of the 4-dimensional position q = ( x 1 y 1 x 2 y 2 ) ⊤ and 4-dimensional velocity v = ( v x 1 v y 1 v x 2 v y 2 ) ⊤ .

Original: This is a secondorder ODE, indicating that dd t q = v .
Revised: This is a secondorder ODE, indicating that dd t q = v .

Original: The momentum p x 1 of x 1 equals m 1 v x 1 .
Revised: The momentum p x 1 of x 1 equals m 1 v x 1 .

Original: The timederivative dd t v of the velocity v is called the acceleration.
Revised: The timederivative dd t v of the velocity v is called the acceleration.

Original: The acceleration of x 1 is given by
Revised: The acceleration of x 1 is given by

Original: [Equation]
Revised: [Equation]

Original:  2 , where G , m 1 , and m 2 denote the constant of gravity and masses of two bodies, respectively.
Revised: [Equation] 2 , where G , m 1 , and m 2 denote the constant of gravity and masses of two bodies, respectively.

Original: The same process applies for the remaining positions.
Revised: The same process applies for the remaining positions.

Original: The total energy of the two-body problem is given by
Revised: The total energy of the two-body problem is given by

Original: [Equation]
Revised: [Equation]

Original: The first and second terms denote the kinetic and potential energies, respectively.
Revised: The first and second terms denote the kinetic and potential energies, respectively.

Original: The two-body problem is a Hamiltonian system, and the dynamics mentioned above can be rewritten as Hamilton’s equation.
Revised: The two-body problem is a Hamiltonian system, and the dynamics mentioned above can be rewritten as Hamilton’s equation.

Original: The Hamiltonian H is one of the first integrals; the two-body problem has other first integrals, such as the linear momenta in the x - and y -directions
Revised: The Hamiltonian H is one of the first integrals; the two-body problem has other first integrals, such as the linear momenta in the x - and y -directions

Original: [Equation]
Revised: [Equation]

Original: and angular momentum (Hairer et al., 2006).
Revised: and angular momentum (Hairer et al., 2006).

Original: We set G , m 1 , and m 2 to 1.0.
Revised: We set G , m 1 , and m 2 to 1.0.

Original: The initial distance r 1 = (cid:112) x 21 + y 21 of a mass m 1 from the origin was set to r 1 ∼ U (0 . 5 , 1 . 0) , and the initial angle θ 1 = tan − 1 ( y 1 x 1 ) was set to θ 1 ∼ U (0 , 2 π ) .
Revised: The initial distance r 1 = (cid:112) x 21 + y 21 of a mass m 1 from the origin was set to r 1 ∼ U (0 . 5 , 1 . 0) , and the initial angle θ 1 = tan − 1 ( y 1 x 1 ) was set to θ 1 ∼ U (0 , 2 π ) .

Original: The initial speed | v 1 | = (cid:113) v 2 x 1 + v 2 y 1 was set to 12 r 2 ϵ v , where ϵ v ∼ N (1 , 0 . 05) .
Revised: The initial speed | v 1 | = (cid:113) v 2 x 1 + v 2 y 1 was set to 12 r 2 ϵ v , where ϵ v ∼ N (1 , 0 . 05) .

Original: The initial angle of the velocity was set to θ ± 0 .
Revised: The initial angle of the velocity was set to θ ± 0 .

Original: 5 π
Revised: 5 π

Original: + ϵ θ π , where ϵ θ ∼ N (0 , 0 . 05) .
Revised: + ϵ θ π , where ϵ θ ∼ N (0 , 0 . 05) .

Original: The initial condition of the other mass m 2 was set to the opposite of the mass m 1 .
Revised: The initial condition of the other mass m 2 was set to the opposite of the mass m 1 .

Original: Subsequently, the two masses trace elliptical orbits, and when ϵ v =
Revised: Subsequently, the two masses trace elliptical orbits, and when ϵ v =

Original: ϵ θ = 0 , they trace exactly circular orbits.
Revised: ϵ θ = 0 , they trace exactly circular orbits.

Original: In addition, we added a perturbation following
Revised: In addition, we added a perturbation following

Original: N (0 , 0 . 01) to the velocities of both masses, which corresponds to the center-of-gravity velocity.
Revised: N (0 , 0 . 01) to the velocities of both masses, which corresponds to the center-of-gravity velocity.

Original: We set the time-step size ∆ t to 0.01 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 10 , 000 steps for evaluation.
Revised: We set the time-step size ∆ t to 0.01 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 10 , 000 steps for evaluation.

Original: We trained each model for 100,000 iterations.
Revised: We trained each model for 100,000 iterations.

Original: Hamiltonian System in Non-Canonical Form: KdV equation The KdV equation is a model of shallow water waves and is known to have soliton solutions (Furihata, 2001).
Revised: Hamiltonian System in Non-Canonical Form: KdV equation The KdV equation is a model of shallow water waves and is known to have soliton solutions (Furihata, 2001).

Original: The dynamics is given by
Revised: The dynamics is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) where x denotes the spatial position and the subscripts denote partial derivatives; for example, u t = ∂u∂t .
Revised: (A18) where x denotes the spatial position and the subscripts denote partial derivatives; for example, u t = ∂u∂t .

Original: The Hamiltonian is given by
Revised: The Hamiltonian is given by

Original: [Equation]
Revised: [Equation]

Revised: (A19)

Original: As Hamilton’s equation dd t u = S ∇ H , the partial differential operator ∂∂x acts as the coefficient matrix S .
Revised: As Hamilton’s equation dd t u = S ∇ H , the partial differential operator ∂∂x acts as the coefficient matrix S .

Original: This system is Liouville integrable and has infinitely many first integrals, including the
Revised: This system is Liouville integrable and has infinitely many first integrals, including the

Original: Hamiltonian H , total mass I 1 = (cid:82) u d x , and T 2 = (cid:82) u
Revised: Hamiltonian H , total mass I 1 = (cid:82) u d x , and T 2 = (cid:82) u

Original: 2 d x (Miura et al., 1968).
Revised: 2 d x (Miura et al., 1968).

Original: Other first integrals are defined using higher-order partial derivatives.
Revised: Other first integrals are defined using higher-order partial derivatives.

Revised: For PDEs, PINNs are known to provide solutions when symbolic equations and boundary conditions are given (Raissi et al., 2019).

Revised: We, in contrast, consider learning spatially discretized PDEs as ODEs from observed data and solving them using numerical integrators, in the same context as NODEs and HNNs; this topic has also been studied extensively (Long et al., 2018; Matsubara et al., 2020;

Revised: Sun et al., 2020; Holl et al., 2020).

Original: Following the experiments in a previous study (Matsubara et al., 2020), we discretized the KdV equation; it no longer has infinitely many first integrals.
Revised: Following the experiments in a previous study (Matsubara et al., 2020), we discretized the KdV equation in space; it no longer has infinitely many first integrals.

Original: We set α = − 6 , β = 1 , spatial size to 10 space units, and space mesh size to 0.2; the system state u had 50 elements.
Revised: We set α = − 6 , β = 1 , spatial size to 10 space units, and space mesh size to 0.2; the system state u had 50 elements.

Original: We generated two solitons as the initial condition; each was expressed as − 12 α κ 2 sech 2 ( κ ( x − d )) , where the size κ followed U (0 . 5 , 2) and the initial position d of one soliton was set to be at least 2.0 from that of the other.
Revised: We generated two solitons as the initial condition; each was expressed as − 12 α κ 2 sech 2 ( κ ( x − d )) , where the size κ followed U (0 . 5 , 2) and the initial position d of one soliton was set to be at least 2.0 from that of the other.

Original: We set the time-step size ∆ t to 0.001 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 10 , 000 steps for evaluation, using the discrete gradient method to ensure energy conservation (Furihata, 2001).
Revised: We set the time-step size ∆ t to 0.001 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 10 , 000 steps for evaluation, using the discrete gradient method to ensure energy conservation (Furihata, 2001).

Original: We trained each model for 30,000 iterations.
Revised: We trained each model for 30,000 iterations.

Revised: Due to the spatial discretization, the KdV dataset contains spatial truncation errors.

Revised: When the neural network learns this dataset, no spatial truncation errors are additionally introduced.

Revised: An evaluation using the analytical solution as a dataset or datasets created with different spatial resolutions is included in future work.

Original: Poisson System: Double Pendulum A double pendulum (2-pend) is depicted in Fig.
Revised: Poisson System: Double Pendulum A double pendulum (2-pend) is depicted in Fig.

Original: A1.
Revised: A1.

Original: In polar coordinates, this is a Hamiltonian system.
Revised: In polar coordinates, this is a Hamiltonian system.

Original: The state is composed of the angles ( θ 1 , θ 2 ) of the two rods and their angular velocities ( ω 1 , ω 2 ) .
Revised: The state is composed of the angles ( θ 1 , θ 2 ) of the two rods and their angular velocities ( ω 1 , ω 2 ) .

Original: This is also a second-order ODE, indicating that dd t θ 1 = ω 1 and dd t θ 2 = ω 2 .
Revised: This is also a second-order ODE, indicating that dd t θ 1 = ω 1 and dd t θ 2 = ω 2 .

Original: Let l 1 , l 2 denote the lengths of the two rods, m 1 , m 2 denote the masses of the two weights, and g denote the gravitational acceleration.
Revised: Let l 1 , l 2 denote the lengths of the two rods, m 1 , m 2 denote the masses of the two weights, and g denote the gravitational acceleration.

Original: The acceleration is given by
Revised: The acceleration is given by

Original: [Equation]
Revised: [Equation]

Original: where ∆ = θ 1 − θ 2 .
Revised: where ∆ = θ 1 − θ 2 .

Original: In 2-dimensional Cartesian coordinates, the state is composed of the positions ( x 1 , y 1 , x 2 , y 2 ) of the two masses and the corresponding velocities ( v x 1 , v y 1 , v x 2 , v y 2 ) .
Revised: In 2-dimensional Cartesian coordinates, the state is composed of the positions ( x 1 , y 1 , x 2 , y 2 ) of the two masses and the corresponding velocities ( v x 1 , v y 1 , v x 2 , v y 2 ) .

Original: The position is transformed by x 1 = l 1 sin θ 1 , y 1 =
Revised: The position is transformed by x 1 = l 1 sin θ 1 , y 1 =

Original: l 1 cos θ 1 , x 2 = x 1 + l 2 sin θ 2 , and y 

Revised: l 1 cos θ 1 , x 2 = x 1 + l 2 sin θ 2 , and y

Original: = y 1 + l 2 cos θ 2 , and the velocity is transformed accordingly.
Revised: = y 1 + l 2 cos θ 2 , and the velocity is transformed accordingly.

Original: The total energy H is given by
Revised: The total energy H is given by

Original: [Equation]
Revised: [Equation]

Original: The first and second terms denote the kinetic and potential energies, respectively.
Revised: The first and second terms denote the kinetic and potential energies, respectively.

Original: The double pendulum is no longer a Hamiltonian system in Cartesian coordinates.
Revised: The double pendulum is no longer a Hamiltonian system in Cartesian coordinates.

Original: Because the lengths of the two rods are constant, the double pendulum has two constraints on the position: l 21 = x 21 + y 21 and l 22 = ( x 2 − x 1 ) 

Revised: Because the lengths of the two rods are constant, the double pendulum has two constraints on the position: l 21 = x 21 + y 21 and l 22 = ( x 2 − x 1 )

Original: + ( y 2 − y 1 ) 2 .
Revised: + ( y 2 − y 1 ) 2 .

Original: These constraints are holonomic constraints, and they lead to constraints involving the velocity, namely 

Revised: These constraints are holonomic constraints, and they lead to constraints involving the velocity, namely

Original: = x 1 v x 1 + y 1 v yand 

Revised: = x 1 v x 1 + y 1 v yand

Original: = ( x 2 − x 1 )( v x 2 − v x 1 ) + ( y 2 − y 1 )( v y 2 − v y 1 ) .
Revised: = ( x 2 − x 1 )( v x 2 − v x 1 ) + ( y 2 − y 1 )( v y 2 − v y 1 ) .

Original: When the constraints involving the velocity are satisfied, the holonomic constraints are implicitly satisfied.
Revised: When the constraints involving the velocity are satisfied, the holonomic constraints are implicitly satisfied.

Original: Therefore, the number of first integrals is five; however, three first integrals are sufficient to determine the dynamics.
Revised: Therefore, the number of first integrals is five; however, three first integrals are sufficient to determine the dynamics.

Original: The dynamics is degenerate and classified as a constrained Hamiltonian system, or a Poisson system in a more general case.
Revised: The dynamics is degenerate and classified as a constrained Hamiltonian system, or a Poisson system in a more general case.

Original: We set the masses of the two weights to m 1 = m 2 = 1 . 0 and the gravitational acceleration g to 9 . 8 .
Revised: We set the masses of the two weights to m 1 = m 2 = 1 . 0 and the gravitational acceleration g to 9 . 8 .

Original: We set the lengths l 1 , l 2 of the two rods to follow U (0 . 9 , 1 . 1) , the initial angles θ 1 , θ 2 to follow
Revised: We set the lengths l 1 , l 2 of the two rods to follow U (0 . 9 , 1 . 1) , the initial angles θ 1 , θ 2 to follow

Original: U ( − 0 .
Revised: U ( − 0 .

Original: 5 , 0 . 5) , and the initial angular velocities ˙ θ 1 , ˙ θ 2 to follow U ( − 0 .
Revised: 5 , 0 . 5) , and the initial angular velocities ˙ θ 1 , ˙ θ 2 to follow U ( − 0 .

Original: We set the time-step size ∆ t to 0.1 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 5 , 000 steps for evaluation.
Revised: We set the time-step size ∆ t to 0.1 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 5 , 000 steps for evaluation.

Original: We trained each model for 100,000 iterations.
Revised: We trained each model for 100,000 iterations.

Original: 𝐶 𝑅 𝐿 𝐷 𝐼 𝑉 𝐸 model (Izhikevich &

Original: FitzHugh, 2006). Dirac Structure: FitzHugh–Nagumo Model R. FitzHugh proposed a model of the electrical dynamics of a biological neuron, and J. Nagumo created an equivalent electric circuit.
Revised: Dirac Structure: FitzHugh–Nagumo Model R. FitzHugh proposed a model of the electrical dynamics of a biological neuron, and J. Nagumo created an equivalent electric circuit.

Original: This model is called the FitzHugh–Nagumo model (Izhikevich & FitzHugh, 2006) and is a modified version of the van der Pol oscillator; the state oscillates when the magnitude of the external current source I is within an appropriate range.
Revised: This model is called the FitzHugh–Nagumo model (Izhikevich & FitzHugh, 2006) and is a modified version of the van der Pol oscillator; the state oscillates when the magnitude of the external current source I is within an appropriate range.

Original: The circuit comprises a resistor R , inductor L , capacitor C , tunnel diode D , and voltage source E connected as shown in Fig.
Revised: The circuit comprises a resistor R , inductor L , capacitor C , tunnel diode D , and voltage source E connected as shown in Fig.

Original: A2.
Revised: A2.

Original: The whole circuit is connected to an external current source I .
Revised: The whole circuit is connected to an external current source I .

Original: Let I R denote the current through the resistor R , and V R denote the applied voltage.
Revised: Let I R denote the current through the resistor R , and V R denote the applied voltage.

Original: Ohm’s law and other properties of the elements give V R
Revised: Ohm’s law and other properties of the elements give V R

Original: =
Revised: =

Original: I R R , C dd t V C
Revised: I R R , C dd t V C

Original: = I C , L dd t I L
Revised: = I C , L dd t I L

Original: =
Revised: =

Original: V L , and I D
Revised: V L , and I D

Original: = D ( V D ) , where we treat D as a nonlinear function.
Revised: = D ( V D ) , where we treat D as a nonlinear function.

Original: Kirchhoff’s current law (KCL) obtains I C
Revised: Kirchhoff’s current law (KCL) obtains I C

Original: + I D
Revised: + I D

Original: + I R
Revised: + I R

Original: =
Revised: =

Original: I and I R = I L , and Kirchhoff’s voltage law (KVL) obtains V C 
Revised: I and I R = I L , and Kirchhoff’s voltage law (KVL) obtains V C =

Original: = V D
Revised: V D

Original: =
Revised: =

Original: V R + V L + E .
Revised: V R + V L + E .

Original: We denote W
Revised: We denote W

Original: = I R and V = V C , and set L = 1 / 0 .
Revised: = I R and V = V C , and set L = 1 / 0 .

Original: 08 , R = 0 .
Revised: 08 , R = 0 .

Original: 8 , C = 1 . 0 , V E = − 0 .
Revised: C = 1 . 0 , V E = − 0 .

Original: 7 , and D ( V ) =
Revised: 7 , and D ( V ) =

Original: V 3 / 3 − V .
Revised: V 3 / 3 − V .

Original: Subsequently, we obtain the FitzHugh–Nagumo model of the original parameters as
Revised: Subsequently, we obtain the FitzHugh–Nagumo model of the original parameters as

Original: [Equation]
Revised: [Equation]

Original: Due to the resistor R , the FitzHugh–Nagumo model is not an energy-conserving system.
Revised: Due to the resistor R , the FitzHugh–Nagumo model is not an energy-conserving system.

Revised: 𝐶 𝑅 𝐿 𝐷 𝐼 𝑉 𝐸

Revised: (Izhikevich &

Revised: FitzHugh, 2006)

Original: Consider a situation where the current through and the voltage applied to stateful elements (capacitors and inductors) are measurable, but the connections between the elements are unknown.
Revised: Consider a situation where the current through and the voltage applied to stateful elements (capacitors and inductors) are measurable, but the connections between the elements are unknown.

Original: We treated I C , I L , V C , V L as the system state u .
Revised: We treated I C , I L , V C , V L as the system state u .

Original: Because the state is in 4-dimensional space and the dynamics is intrinsically 2-dimensional, there exist two first integrals; for example, but not limited to, 
Revised: Because the state is in 4dimensional space and the dynamics is intrinsically 2-dimensional, there exist two first integrals; for example, but not limited to, I =

Original: I= I C
Revised: I C

Original: + D ( V C )
Revised: + D ( V C )

Original: + I L and E
Revised: + I L and E

Original: = V C − I L R − V L .
Revised: = V C − I L R − V L .

Original: This type of electric circuit is an example of a Dirac structure because the state variables are constrained by the circuit topology and Kirchhoff’s current and voltage laws (van der Schaft & Jeltsema, 2014).
Revised: This type of electric circuit is an example of a Dirac structure because the state variables are constrained by the circuit topology and Kirchhoff’s current and voltage laws (van der Schaft & Jeltsema, 2014).

Original: From the viewpoint of generalized Hamiltonian systems, ( I L , V C ) corresponds to the position, and ( V L , I C ) corresponds to the momentum.
Revised: From the viewpoint of generalized Hamiltonian systems, ( I L , V C ) corresponds to the position, and ( V L , I C ) corresponds to the momentum.

Original: The electric circuit can be described as a port-Hamiltonian system in a non-canonical form.
Revised: The electric circuit can be described as a port-Hamiltonian system in a non-canonical form.

Original: Because of the non-canonical form, the FitzHugh–Nagumo model is outside the scope of CHNN and dissipative SymODEN (Finzi et al., 2020b; Zhong et al., 2020b).
Revised: Because of the non-canonical form, the FitzHugh–Nagumo model is outside the scope of CHNN and dissipative SymODEN (Finzi et al., 2020b; Zhong et al., 2020b).

Original: We set the external current source I to follow U (0 . 7 , 1 . 1) , set the initial values of V and W to follow
Revised: We set the external current source I to follow U (0 . 7 , 1 . 1) , set the initial values of V and W to follow

Original: U ( − 1 . 5 , 1 . 5) and U (0 . 0 , 2 . 0) , and transformed them to the state.
Revised: U ( − 1 . 5 , 1 . 5) and U (0 . 0 , 2 . 0) , and transformed them to the state.

Original: We set the time-step size ∆ t to 0.1 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 2 , 000 steps for evaluation.
Revised: We set the time-step size ∆ t to 0.1 and generated 1,000 time-series of S = 500 steps for training and 10 time-series of S = 2 , 000 steps for evaluation.

Original: We trained each model for 30,000 iterations.
Revised: We trained each model for 30,000 iterations.

Original: D A DDITIONAL R ESULTS AND D ISCUSSION D. D EMONSTRATION OF F IRST I NTEGRAL P RESERVATION
Revised: D A DDITIONAL R ESULTS AND D ISCUSSION D. D EMONSTRATION OF F IRST I NTEGRAL P RESERVATION

Original: [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 1, we examined a mass-spring system and FINDE using the leapfrog integrator.
Revised: 1, we examined a mass-spring system and FINDE using the leapfrog integrator.

Original: We also examined the case with the Dormand–
Revised: We also examined the case with the Dormand–

Original: Prince integrator (dopri5), as shown in Fig.
Revised: Prince integrator (dopri5), as shown in Fig.

Original: A3.
Revised: A3.

Original: We increased the number of steps to 10 5 , and displayed the MSEs of the state instead of the state itself.
Revised: We increased the number of steps to 10 5 , and displayed the MSEs of the state instead of the state itself.

Original: First, we focus on the energy in the bottom panel.
Revised: First, we focus on the energy.

Original: Even using the Dormand–Prince integrator, a fourth-order method, the energy is slightly decreased.
Revised: Even using the Dormand–Prince integrator, a fourth-order method, the energy is slightly decreased.

Original: The cFINDE with the Dormand– Prince integrator shows the same tendency.
Revised: The cFINDE with the Dormand–Prince integrator shows the same tendency.

Original: This phenomenon is due to numerical errors and is called energy drift.
Revised: This phenomenon is due to temporal discretization errors and is called energy drift.

Original: The dFINDE with the Dormand–Prince integrator significantly suppresses the energy error.
Revised: The dFINDE with the Dormand–Prince integrator significantly suppresses the error in energy.

Original: The remaining error is caused by rounding errors.
Revised: The remaining error is caused by rounding errors.

Original: When the focus is on the MSEs of the state in the upper panel, the trend is different: the dFINDE with the Dormand–Prince integrator suffers from the most significant state errors.
Revised: When the focus is on the MSEs of the state, the trend is different: the dFINDE with the Dormand–Prince integrator suffers from the most significant errors in state.

Original: Although the dFINDE is designed to eliminate energy errors, it does not necessarily minimize state errors.
Revised: Although the dFINDE is designed to eliminate temporal discretization errors in energy, it does not necessarily reduce those in state.

Original: However, the Dormand–Prince integrator is designed to suppress state errors.
Revised: In contrast, the Dormand–Prince integrator is designed to suppress temporal discretization errors in state.

Original: Therefore, there is no guarantee that the dFINDE improves the prediction performance when defined using state errors.
Revised: Therefore, there is no guarantee that the dFINDE improves the prediction performance when defined using errors in state.

Original: Conversely, the experimental results in Table 3 demonstrate that the dFINDE is superior to the base model and cFINDE in VPT.
Revised: Conversely, the experimental results in Table 3 demonstrate that the dFINDE is superior to the base model and cFINDE in VPT.

Revised: This is because dFINDE reduces the modeling errors rather.

Original: For the mass-spring system, the governing equation is already known as an ODE and is discretized by the dFINDE, leading to temporal discretization errors.
Revised: For the mass-spring system, the governing equation is already known as an ODE and is discretized by the dFINDE, leading to temporal discretization errors.

Original: However, when dFINDE learns dynamics from data, the training data points are already sampled in discrete time, and the dFINDE predicts future states in discrete time.
Revised: However, when dFINDE learns dynamics from data, the training data points are already sampled in discrete time, and the dFINDE predicts future states in discrete time.

Original: Therefore, no  discretization error occurs, and we obtain only the advantages of exactly preserving the first integral.
Revised: Therefore, no temporal discretization error occurs, and we obtain only the advantages of exactly preserving the first integral.

Original: This type of paradox has been repeatedly discovered in previous studies.
Revised: This type of paradox has been repeatedly discovered in previous studies.

Original: For example, the leapfrog integrator and discrete gradient method are second-order methods.
Revised: For example, the leapfrog integrator and discrete gradient method are second-order methods.

Original: However, they are superior to the Dormand–Prince integrator when combined with neural networks and learning dynamics from data (Matsubara et al., 2020).
Revised: However, they are superior to the Dormand–Prince integrator when combined with neural networks and learning dynamics from data (Matsubara et al., 2020).

Original: For learning, the preservation of specific properties of target systems is more important than the order of accuracy.[Table] [Table]
Revised: For better learning (i.e., smaller modeling errors), the preservation of specific properties of target systems is more important than the order of accuracy.

Original: D. S YMBOLIC R EGRESSION OF F OUND F IRST I NTEGRALS
Revised: D. S YMBOLIC R EGRESSION OF F OUND F IRST I NTEGRALS

Original: Using gplearn (based on genetic programming) [Table] we performed a symbolic regression of the first integrals V found by the neural network.
Revised: Using gplearn (based on genetic programming), we performed a symbolic regression of the first integrals V found by the neural network.

Original: We prepared addition, subtraction, multiplication [Table] and division as candidate operations, used Pearson’s correlation coefficient as the evaluation criterion, set the early stopping threshold to 0. [Table]
Revised: We prepared addition, subtraction, multiplication, and division as candidate operations, used Pearson’s correlation coefficient as the evaluation criterion, set the early stopping threshold to 0.9, and set the population size to 10,000.

Original: and set the population size to 10,000.

Original: We set the other hyperparameters to their default values, e [Table] g., the maximum number of generations was 20.
Revised: We set the other hyperparameters to their default values, e.g., the maximum number of generations was 20.

Original: We summarize the regression results of the HNN with cFINDE for K = 2 trained using the twobody dataset in Table A1.
Revised: We summarize the regression results of the HNN with cFINDE for K = 2 trained using the twobody dataset in Table A1.

Original: Note that Pearson’s correlation coefficient is invariant to biases and scale factors.
Revised: Note that Pearson’s correlation coefficient is invariant to biases and scale factors.

Original: FINDE is also invariant because it only uses the directions of the gradients of first integrals.
Revised: FINDE is also invariant because it only uses the directions of the gradients of first integrals.

Original: Hence, we removed biases and scale factors from the regression results.
Revised: Hence, we removed biases and scale factors from the regression results.

Original: When the focus is on the symbolic regression of the training data, V 1 , V 1 , V 2 , and V 2 for trials 0, 1, 2, and 3 are identical to the linear momentum in the x -direction up to scale factors; recall that we set m 

Revised: When the focus is on the symbolic regression of the training data, V 1 , V 1 , V 2 , and V 2 for trials 0, 1, 2, and 3 are identical to the linear momentum in the x -direction up to scale factors; recall that we set m

Original: = m 2 = 1 . 

Revised: = m 2 = 1 .

Original: and see Eq.
Revised: and see Eq.

Original: (A17).
Revised: (A17).

Original: V 2 , V 2 , V 1 , and V 1 for trials 0, 1, 2, and 3 are also identical to the linear momentum in the y -direction.
Revised: V 2 , V 2 , V 1 , and V 1 for trials 0, 1, 2, and 3 are also identical to the linear momentum in the y -direction.

Original: V 1 and V 2 for trial 4 are weighted sums of the linear momenta in the x - and y -directions; in particular, they can be regarded as the linear momenta in the (1 , − 1) - and (1 , 1) - directions, respectively.
Revised: V 1 and V 2 for trial 4 are weighted sums of the linear momenta in the x - and y -directions; in particular, they can be regarded as the linear momenta in the (1 , − 1) - and (1 , 1) - directions, respectively.

Original: When the quantities V 1 ( u ) and V 2 ( u ) are first integrals, any function of only V 1 ( u ) , V 2 ( u ) , and arbitrary constants is a first integral functionally dependent on V 1 ( u ) and V 2 ( u ) .
Revised: When the quantities V 1 ( u ) and V 2 ( u ) are first integrals, any function of only V 1 ( u ) , V 2 ( u ) , and arbitrary constants is a first integral functionally dependent on V 1 ( u ) and V 2 ( u ) .

Original: Thus, there isno guarantee that FINDE will find the first integrals in their well-known forms.
Revised: Thus, it is in principle impossible to re-discover a first integral as a well-known symbolic expression, and a failure in symbolic regression is not a problem in any way.

Revised: Previous studies introduced certain constraints (such as “gauge fixing”) for symbolic regression (Liu & Tegmark, 2021); a combination of such method may improve the results.

Original: However, recent studies  have revealed that typical initialization and training proceduresof neural networks tend to learn simple functions (Barrett & Dherin, 2021; Cao et al., 2021).
Revised: However, recent studies on neural networks have revealed that typical initialization and training procedures tend to learn simple functions (Barrett & Dherin, 2021; Cao et al., 2021).

Original: Additionally, the symbolic regression limited the depth of the computation graph, biasing the results toward simple functions; hence, the found first integrals were identical to the well-known forms and were separated in the x - and y -directions in most cases.
Revised: Additionally, the symbolic regression limited the depth of the computation graph, biasing the results toward simple functions; hence, the found first integrals were identical to the well-known forms and were separated in the x - and y -directions in most cases.

Original: The same is true for the symbolic regression of the test data, except for V 1 for trial 0, which had a small perturbation α .
Revised: The same is true for the symbolic regression of the test data, except for V 1 for trial 0, which had a small perturbation α .

Original: Because of the limited extrapolation ability, neural networks cannot always accurately represent functions outside the training data range.
Revised: Because of the limited extrapolation ability, neural networks cannot always accurately represent functions outside the training data range.

Original: Once first integrals are found by
Revised: Once first integrals are found by

Original: FINDE and identified as equations by symbolic regression, one can use the equations instead of neural networks, ensuring the preservation of first integrals in the entire domain.
Revised: FINDE and identified as equations by symbolic regression, one can use the equations instead of neural networks, ensuring the preservation of first integrals in the entire domain.

Original: From these results, we can conclude that cFINDE identified the linear momenta.
Revised: From these results, we can conclude that cFINDE identified the linear momenta.

Original: The state of the KdV dataset has 50 elements, which is too large to apply a symbolic regression.
Revised: The state of the KdV dataset has 50 elements, which is too large to apply a symbolic regression.

Original: For the 2-pend and FitzHugh–Nagumo datasets, we did not find consistent equations of first integrals.
Revised: For the 2-pend and FitzHugh–Nagumo datasets, we did not find consistent equations of first integrals.

Original: For example, the symbolic regression identified a quantity x 21 − y 1 as a first integral in the 2-pend dataset, which is not directly related to well-known first integrals.
Revised: For example, the symbolic regression identified a quantity x 21 − y 1 as a first integral in the 2-pend dataset, which is not directly related to well-known first integrals.

Original: When the angle θ 1 of the upper rod is small, y 1 takes a value close to − 1 , and the quantity x 21 − y 1 is close to x 21 + y 21 , which is a well-known first integral, namely the square l 21 of the upper rod length l 1 .
Revised: When the angle θ 1 of the upper rod is small, y 1 takes a value close to − 1 , and the quantity x 21 − y 1 is close to x 21 + y 21 , which is a well-known first integral, namely the square l 21 of the upper rod length l 1 .

Original: It is difficult to determine whether this inaccuracy is because of the training of FINDE or symbolic regression.
Revised: It is difficult to determine whether this inaccuracy is because of the training of FINDE or symbolic regression.

Original: There may still be room for improvement in the training of FINDE or symbolic regression. 
Revised: There may still be room for improvement in the training of FINDE or symbolic regression. [Table] [Table]

Original: D. C OMPARISON WITH M ODEL OF K NOWN H OLONOMIC C ONSTRAINTS
Revised: D. C OMPARISON WITH M ODEL OF K NOWN H OLONOMIC C ONSTRAINTS

Original: The double pendulum (2-pend) is classified as a constrained Hamiltonian system.
Revised: The double pendulum (2-pend) is classified as a constrained Hamiltonian system.

Original: CHNN was proposed for cases when holonomic constraints are known (Finzi et al., 2020b).
Revised: CHNN was proposed for cases when holonomic constraints are known (Finzi et al., 2020b).

Original: We evaluated compari- [Table] son methods under the assumption that the holonomic constraints were known.
Revised: We evaluated comparison methods under the assumption that the holonomic constraints were known.

Original: We summarized the results in Table A2.
Revised: We summarized the results in Table A2.

Original: The HNN, without constraints, completely failed to learn the dynamics.
Revised: The HNN, without constraints, completely failed to learn the dynamics.

Original: This is unsurprising because the dynamics of the double pendulum is outside the scope of the HNN.
Revised: This is unsurprising because the dynamics of the double pendulum is outside the scope of the HNN.

Original: The two known holonomic constraints lead to two constraints involving the velocity; the CHNN took into account all four known constraints and worked remarkably.
Revised: The two known holonomic constraints lead to two constraints involving the velocity; the CHNN took into account all four known constraints and worked remarkably.

Original: The HNN with cFINDE was given all four known constraints as the first integrals, but did not work properly.
Revised: The HNN with cFINDE was given all four known constraints as the first integrals, but did not work properly.

Original: The original purpose of projection methods is to eliminate numerical errors of first integrals but not to change the class to which the dynamics belong.
Revised: The original purpose of projection methods is to eliminate temporal discretization errors of first integrals but not to change the class to which the dynamics belong.

Original: Therefore, when a target system is not a subject of the base model, the base model with FINDE does not work.
Revised: Therefore, when a target system is not a subject of the base model, the base model with FINDE does not work.

Original: The NODE learns an ODE in a general way, and thus constrained Hamiltonian systems are included in its subjects.
Revised: The NODE learns an ODE in a general way, and thus constrained Hamiltonian systems are included in its subjects.

Original: Given all four known constraints, the NODE with cFINDE worked better but never surpassed the CHNN.
Revised: Given all four known constraints, the NODE with cFINDE worked better but never surpassed the CHNN.

Original: However, the CHNN works only for Hamiltonian systems in the canonical form with holonomic constraints.
Revised: However, the CHNN works only for Hamiltonian systems in the canonical form with holonomic constraints.

Original: We also evaluated comparison methods using the 2-body dataset under the assumption that the linear momenta were known as first integrals.
Revised: We also evaluated comparison methods using the 2-body dataset under the assumption that the linear momenta were known as first integrals.

Original: The CHNN attempted to obtain the inverse of a singular matrix and could not learn the dynamics.
Revised: The CHNN attempted to obtain the inverse of a singular matrix and could not learn the dynamics.

Original: In contrast, the cFINDE improved the performances of both NODE and HNN.
Revised: In contrast, the cFINDE improved the performances of both NODE and HNN.

Revised: Existing methods (e.g., HNN and CHNN) assume geometric structures (e.g., Hamiltonian structure) described in Appendix A in order to guarantee conservation laws.

Revised: When multiple structures are assumed at the same time, they must be integrated using appropriate prior knowledge.

Revised: If it is possible, it would achieve extremely high performance.

Revised: Otherwise, the geometric structures would conflict with each other and would not produce an appropriate model.

Revised: This is the reason why CHNN failed to learn the 2-body dataset and HNN+FINDE failed to learn the 2-pend dataset.

Original:  When the detailed properties of target systems are known, one can choose the best models.
Revised: In contrast, NODE+FINDE does not assume any geometric structure and assumes first integrals in the most general way, being available to any situation. Hence, FINDE can assume one or more first integrals without changing anything. When the detailed properties of target systems are known, one can choose the best models.

Original: If the chosen model is inappropriate, the training procedure totally fails.
Revised: If the chosen model is inappropriate, the training procedure totally fails.

Original: FINDE provides a better alternative when prior knowledge is limited.
Revised: FINDE provides a better alternative when prior knowledge is limited.

Original: Moreover, a constrained Hamiltonian system can have first integrals other than holonomic constraints and the Hamiltonian.
Revised: Moreover, a constrained Hamiltonian system can have first integrals other than holonomic constraints and the Hamiltonian.

Original: In this case, the CHNN with
Revised: In this case, the CHNN with

Original: FINDE is potentially the best choice.
Revised: FINDE is potentially the best choice.

Original:  D. C OMPARISON WITH M ODIFIED N EURAL P ROJECTION M ETHOD
Revised: [Table] D. H OW TO D ETERMINE N UMBER OF F IRST I NTEGRALS

Revised: The theoretical explanation for the high performance of neural networks (e.g., HNN) that assume first integrals for physical phenomena is an open question.

Revised: Sannai et al.

Revised: (2021) has theoretically shown that neural networks (e

Revised: [Table]

Revised: g., CNNs and GNNs) with symmetry have faster learning convergence, and we consider this approach can be applied to the above question.

Revised: At least for cFINDE and dFinde, we have an intuitive but not rigorous explanation; assuming one more first integral (i

Revised: [Table]

Revised: e., increasing K by 1) reduces the number of degrees of freedom in the dynamics by 1, narrows the hypothesis space, accelerates learning convergence, and suppresses generalization errors.

Revised: As shown in Table

Revised: [Table]

Revised: the performance of cFINDE and dFINDE is sensitive to the assumed number

Revised: K of first integrals.

Revised: Because K is a hyperparameter, it is basically a subject to be adjusted through evaluations on a validation set.

Revised: With inappropriately large K , both cFINDE and dFINDE dropped their performance significantly.

Revised: See the results of the 2-pend and FitzHugh–Nagumo datasets for

Revised: K = 6 and K = 2 , respectively.

Revised: However, the performance drop can be found even with the training set.

Revised: Table A3 summarizes the prediction performance on the training set of the 2-pend dataset.

Revised: As was the case with the test set, the performance significantly dropped at K = 6 .

Revised: This is because NODE with cFINDE for K =assumes the submanifold M ′ to be 2-dimensional.

Revised: The submanifold M ′ is in fact 3-dimensional, so NODE with cFINDE for K = 6 is incapable of learning the dynamics and performs poorly even on the training set.

Revised: Hence, the training set is enough to avoid a fatally inappropriate K .

Revised: Alternatively, K can be determined by using other methods (e.g., Fukunaga & Olsen (1971);

Revised: Liu &

Revised: Tegmark (2021)).

Revised: Although these methods have some drawbacks introduced in Appendix A, they may be complementary to FINDE.

Revised: D. C OMPARISON WITH M ODIFIED N EURAL P ROJECTION M ETHOD

Original: The neural projection method (NPM) also employs a projection method (Yang et al., 2020).
Revised: The neural projection method (NPM) also employs a projection method (Yang et al., 2020).

Original: Using a manner similar to Newton’s method, it enforces the constraint C ( u ) = 0 by the projection of the state u under the assumption that the quantity C ( u ) is always zero.
Revised: Using a manner similar to Newton’s method, it enforces the constraint C ( u ) = 0 by the projection of the state u under the assumption that the quantity C ( u ) is always zero.

Original: This assumption holds for some cases (e.g., holonomic constraints in a fixed environment), but not for most first integrals, whose values depend on initial conditions.
Revised: This assumption holds for some cases (e.g., holonomic constraints in a fixed environment), but not for most first integrals, whose values depend on initial conditions.

Original: For example, the linear momentum in the x -direction of the two-body problem is the first integral expressed as V ( u ) = m 1 v x 1 ( t ) + m 2 v x 2 ( t ) .
Revised: For example, the linear momentum in the x -direction of the two-body problem is the first integral expressed as V ( u ) = m 1 v x 1 ( t ) + m 2 v x 2 ( t ) .

Original: This quantity V is constant within a trial (i.e.,
Revised: This quantity V is constant within a trial (i.e.,

Original: V ( u ( t ))
Revised: V ( u ( t ))

Original: =
Revised: =

Original: V ( u (0)) ) and varies between trials depending on the initial speed v x 1 (0) and v x 2 (0) .
Revised: V ( u (0)) ) and varies between trials depending on the initial speed v x 1 (0) and v x 2 (0) .

Original: The total energy, the total mass, and many other first integrals depend on the initial condition in the same manner; hence, they are outside the scope of the NPM.
Revised: The total energy, the total mass, and many other first integrals depend on the initial condition in the same manner; hence, they are outside the scope of the NPM.

Original: In contrast, by imposing the constraint on the gradient ∇ V = 0 or discrete gradient ∇ V = 0 , our proposed FINDE keeps the quantity V constant and can handle any first integrals.
Revised: In contrast, by imposing the constraint on the gradient ∇ V = 0 or discrete gradient ∇ V = 0 , our proposed FINDE keeps the quantity V constant and can handle any first integrals.

Original: For comparison, we replaced the constraint C ( u ) = 0 with C ( u s +1 , u s )
Revised: For comparison, we replaced the constraint C ( u ) = 0 with C ( u s +1 , u s )

Original: =
Revised: =

Original: V ( u s +1 )
Revised: V ( u s +1 )

Original: − V ( u s ) = 

Revised: − V ( u s ) =

Original: and adopted the NPM to first integrals varying from trial to trial.
Revised: and adopted the NPM to first integrals varying from trial to trial.

Original: We evaluated the modified
Revised: We evaluated the modified

Original: NPM using the 2-pend dataset.
Revised: NPM using the 2-pend dataset.

Original: Because the modified NPM is a discrete-time projection method, we compared it with the discrete-time version of the proposed FINDE (dFINDE).
Revised: Because the modified NPM is a discrete-time projection method, we compared it with the discrete-time version of the proposed FINDE (dFINDE).

Original: The results are summarized in Table A3.
Revised: The results are summarized in Table A4. [Table]

Original: The dFINDE successfully learned the dynamics in all trials, but the modified NPM failed to learn the dynamics in half the trials (see the rightmost column for the numbers of successful trials out of 5).
Revised: The dFINDE successfully learned the dynamics in all trials, but the modified NPM failed to learn the dynamics in half the trials (see the rightmost column for the numbers of successful trials out of 5).

Original: The modified NPM often encountered of the underflow of the time-step size or a division by the zero[Table] gradient of the first integral.
Revised: The modified NPM often encountered of the underflow of the time-step size or a division by the zero gradient of the first integral.

Original: Even when the learning was successful, the performance of the NPM was inferior to that of the dFINDE.
Revised: Even when the learning was successful, the performance of the NPM was inferior to that of the dFINDE.

Original: The modified NPM solved the optimization problem in Eq.
Revised: The modified NPM solved the optimization problem in Eq.

Original: (3) at every step, but it sometimes diverged or failed to converge, especially in the early phase of learning.
Revised: (3) at every step, but it sometimes diverged or failed to converge, especially in the early phase of learning.

Original: The NPM was successful for fixed environments but might be unsuited for general first integrals varying from trial to trial.
Revised: The NPM was successful for fixed environments but might be unsuited for general first integrals varying from trial to trial.

Original: However, the dFINDE does not require solving an optimization problem during training, making the learning process robust against randomness such as initialization.
Revised: However, the dFINDE does not require solving an optimization problem during training, making the learning process robust against randomness such as initialization.