N ONSEPARABLE S YMPLECTIC N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Predicting the behaviors of Hamiltonian systems has been drawing increasingattention in scientiﬁc machine learning.
However, the vast majority of the literaturewas focused on predicting separable Hamiltonian systems with their kinematicand potential energy terms being explicitly decoupled, while building data-drivenparadigms to predict nonseparable Hamiltonian systems that are ubiquitous in ﬂuiddynamics and quantum mechanics were rarely explored.
The main computationalchallenge lies in the effective embedding of symplectic priors to describe the inherently coupled evolution of position and momentum, which typically exhibitsintricate dynamics.
To solve the problem, we propose a novel neural networkarchitecture, Nonseparable Symplectic Neural Networks (NSSNNs), to uncoverand embed the symplectic structure of a nonseparable Hamiltonian system fromlimited observation data.
The enabling mechanics of our approach is an augmentedsymplectic time integrator to decouple the position and momentum energy termsand facilitate their evolution.
We demonstrated the efﬁcacy and versatility of ourmethod by predicting a wide range of Hamiltonian systems, both separable andnonseparable, including chaotic vortical ﬂows.
We showed the unique computational merits of our approach to yield long-term, accurate, and robust predictionsfor large-scale Hamiltonian systems by rigorously enforcing symplectomorphism.
I NTRODUCTION
A Hamiltonian dynamic system refers to a formalism for modeling a physical system exhibitingsome speciﬁc form of energy conservation during its temporal evolution.
A typical example is apendulum whose total energy (referred as the system’s Hamiltonian) is conserved as a temporallyinvariant sum of its kinematic energy and potential energy.
Mathematically, such energy conservationindicates a speciﬁc geometric structure underpinning its time integration, named as a symplecticstructure, which further spawns a wide range of numerical time integrators to model Hamiltoniansystems.
These symplectic time integrators have proven their effectiveness on simulating a varietyof energy-conserving dynamics when Hamiltonian expressions are known as a prior.
Examplesencompass applications in plasma physics (Morrison, 2005), electromagnetics (Li et al., 2019), ﬂuidmechanics (Salmon, 1988), and celestial mechanics (Saari & Xia, 1996), to name a few.
On another front, the emergence of the various machine learning paradigms with their particularfocus on uncovering the hidden invariant quantities and their evolutionary structures enable a faithfulprediction of Hamiltonian dynamics without knowing its analytical energy expression beforehand.
The key mechanics underpinning these learning models lie in a proper embedding of the strongmathematical inductive priors to ensure Hamiltonian conservation in a neural network data ﬂow.
Typically, such priors are realized in a variational way or a structured way.
For example, in Greydanuset al.
(2019), the Hamiltonian conservation is encoded in the loss function.
This category of methodsdoes not assume any combinatorial pattern of the energy term and therefore relies on the inherentexpressiveness of neural networks to distill the Hamiltonian structure from abundant training datasets(Choudhary et al., 2019).
Another category of Hamiltonian networks, which we refer to as structuredapproaches, implements the conservation law indirectly by embedding a symplectic time integrator(DiPietro et al., 2020; Tong et al., 2020; Chen et al., 2020) or a composition of linear, activation andgradient modules (Jin et al., 2020) into the network architecture .
One of the main limitations of the current structured methods lies in the separable assumption ofthe Hamiltonian expression.
Examples of separable Hamiltonian systems include the pendulum, the
Lotka–Volterra (Zhu et al., 2016), the Kepler (Antohe & Gladwell, 2004), and the Hénon–Heiles systems (Zotos, 2015).
However, beyond this scope, there exist various nonseparable systemswhose Hamiltonian has no explicit expression to decouple the position and momentum energies.
Examples include incompressible ﬂows (Suzuki et al., 2007), quantum systems (Bonnabel et al. ,2009), rigid body dynamics (Chadaj et al., 2017), charged particle dynamics (Zhang et al.
2016),and nonlinear Schrödinger equation (Brugnano et al., 2018).
This nonseparability typically causeschaos and instability, which further complicates the systems’ dynamics. Although SympNet in Jinet al.
(2020) can be used to learn and predict nonseparable Hamiltonian systems, multiple matricesof the same order with system dimension are needed in the training process of SymNet, resultingdifﬁculties in generalizing into high-dimensional large-scale N-body problems which are commonin a series of nonseparable Hamiltonian systems, such as quantum multibody problems and vortexparticle dynamics problems.
Such chaotic and large-scale nature jointly add shear difﬁculties for aconventional machine learning model to deliver faithful predictions.
In this paper, we propose an effective machine learning paradigm to predict nonseparable Hamiltoniansystems.
We build a novel neural network architecture, named as nonseparable symplectic neuralnetworks (NSSNNs), to enable accurate and robust predictions of long-term Hamiltonian dynamicsbased on short-term observation data.
Our proposed method belongs to the category of structurednetwork architectures: it intrinsically embeds the symplectomorphism into the network designto strictly preserve the symplectic evolution and further conserves the unknown, nonseparable Hamiltonian energy.
The enabling techniques we adopted in our learning framework consist in anaugmented symplectic time integrator to asymptotically “decouple” the position and momentumquantities that were nonseparable in their original form.
We also introduce Lagrangian multiplierin the augmented phase space to improve the system’s numerical stability.
Our network design ismotivated by ideas originated from physics (Tao, 2016) and optimization (Boyd et al., 2004).
Thecombination of these mathematical observations and numerical paradigms enables a novel neuralnetwork architecture that can drastically enhances both the scale and scope of the current predictions.
We show a motivational example in Figure 1 by comparing our approach with a traditional HNNmethod (Greydanus et al., 2019) regarding their structural designs and predicting abilities.
We referthe readers to Section 6 for a detailed discussion.
As shown in Figure 1, the vortices evolved using NSSNN are separated nicely as the ground truth, while the vortices merge together using HNN dueto the failure of conserving the symplectic structure of a nonseparable system.
The conservativecapability of NSSNN springs from our design of the auxiliary variables (red x and y ) which convertsthe original nonseparable system into a higher dimensional quasi-separable system where we canadopt a symplectic integrator.
R ELATED WORKSData-driven physical prediction.
Data-driven approaches have been widely applied in physicalsystems including ﬂuid mechanics (Brunton et al., 2020), wave physics (Hughes et al., 2019),quantum physics (Sellier et al., 2019), thermodynamics (Hernandez et al., 2020), and material science(Teicherta et al., 2019).
Among these different physical systems, data-driven ﬂuid receives increasingattention.
We refer the readers to Brunton et al.
for a thorough survey of the fundamentalmachine learning methodologies as well as their uses for understanding, modeling, optimizing, andcontrolling ﬂuid ﬂows in experiments and simulations based on training data.
One of the motivationsof our work is to design a versatile learning approach that can predict complex ﬂuid motions.
Onanother front, many pieces of research focus on incorporating physical priors into the learningframework, e.g., by enforcing incompressibility (Mohan et al., 2020), the Galilean invariance (Linget al., 2016), quasistatic equilibrium (Geng et al., 2020), the Lagrangian invariance (Cranmer et al. ,2020), and Hamiltonian conservation (Hernandez et al., 2020; Greydanus et al., 2019; Jin et al., 2020;
Zhong et al., 2020).
Here, inspired by the idea of embedding physics priors into neural networks, weaim to accelerate the learning process and improve the accuracy of our model.
Neural networks for Hamiltonian systems.
Greydanus et al.
introduced Hamiltonianneural networks (HNNs) to conserve the Hamiltonian energy of the system by reformulating the lossfunction.
Inspired by HNN, a series of methods intrinsically embedding a symplectic integrator intothe recurrent neural network were proposed, such as SRNN (Chen et al., 2020), TaylorNet (Tonget al., 2020) and SSINN (DiPietro et al., 2020), to solve separable Hamiltonian systems. Combinedwith graph networks (Sanchez-Gonzalez et al., 2019; Battaglia et al., 2016), these methods werefurther generalized to large-scale N-body problems induced by interaction force between the particlepairs. Jin et al.
(2020) proposed SympNet by directly constructing the symplectic mapping of systemvariables within neighbouring time steps to handle both separable and nonseparable Hamiltoniansystems.
However, the scale of parameters in SympNet for trainning N dimensional Hamiltoniansystem is O ( N 2 ) , which makes it hard to be generalized to the high dimensional N-body problems. Our NSSNN overcomes these limitations by devising a new Hamiltonian network architecture that isspeciﬁcally suited for nonseparable systems (see details in Section 5). In addition, the Hamiltonianbased neural networks can be extended to further applications. Toth et al.
(2020) developed the Hamiltonian Generative Network (HGN) to learn Hamiltonian dynamics from high-dimensionalobservations (such as images).
Moreover, Zhong et al.
(2020) introduced Symplectic ODE-Net(SymODEN), which adds an external control term to the standard Hamiltonian dynamics.
F RAMEWORK
3. A UGMENTED H AMILTONIAN EQUATION
We start by considering a Hamiltonian system with N pairs of canonical coordinates (i.e. Ngeneralized positions and N generalized momentum).
The time evolution of canonical coordinates isgoverned by the symplectic gradient of the Hamiltonian (Hand & Finch, 2008).
Speciﬁcally, the timeevolution of the system is governed by Hamilton’s equations asd q d t = ∂ H ∂ p , d p d t = − ∂ H ∂ q , (1) with the initial condition ( q , p ) | t
= t 0 = ( q 0 , p 0 ) .
In a general setting, q = ( q 1 , q 2 , · · · , q N ) representsthe positions and p = ( p 1 , p 2 , ...
p N ) denotes their momentum.
Function H = H ( q , p ) is the
Hamiltonian, which corresponds to the total energy of the system.
An important feature of Hamilton’sequations is its symplectomorphism (see Appendix B for a detailed overview).
The symplectic structure underpinning our proposed network architecture draws inspirations from theoriginal research of Tao (2016) in computational physics.
In Tao (2016), a generic, high-order, explicitand symplectic time integrator was proposed to solve (1) of an arbitrary separable and nonseparable Hamiltonian H .
This is implemented by considering an augmented Hamiltonian
[Equation]
with
[Equation]
in an extended phase space with symplectic two form d q ∧ d p + d x ∧ d y , where ω is a constant thatcontrols the binding of the original system and the artiﬁcial restraint.
Notice that the Hamilton’s equations for H [Equation]
[Equation]
with the initial condition ( q , p , x , y ) | t
= t 0 = ( q 0 , p 0 , q 0 , p 0 ) have the same exact solution as (1)in the sense that ( q , p , x , y )
=
( q , p , q , p ) .
Hence, we can get the solution of (1) by solving (4).
Furthermore, it is possible to construct high-order symplectic integrators for H in (4) with explicitupdates.
Our model aims to learn the dynamical evolution of ( q , p ) in (1) by embedding (4) into theframework of NeuralODE (Chen et al., 2018).
The coefﬁcient ω acts as a regularizer, which stabilizesthe numerical results (see Section 4).
3. N ONSEPARABLE H AMILTONIAN N EURAL N ETWORK
We learn the nonseparable Hamiltonian dynamics (1) by constructing an augmented system (4), from which we can obtain the energy function H ( q , p ) by training the neural network H θ ( q , p ) with parameter θ and calculate the gradient ∇ H θ ( q , p ) by taking the in-graph gradient.
For the constructed network H θ ( q , p ) , we integrate (4) by using the second-order symplectic integrator (Tao, 2016).
Speciﬁcally, we will have an input layer ( q , p , x , y ) =
( q 0 , p 0 , q 0 , p 0 ) at t = t 0 and an output layer ( q , p , x , y ) =
( q n , p n , x n , y n ) at t = t 0 + n d t . Input: q 0 , p 0 , t 0 , t, d t ; φ δ 1 , φ δ 2 , and φ δ 3 in (5); Output: (ˆ q , ˆ p , ˆ x , ˆ y ) =
( q n , p n , x n , y n ) 1 ( q 0 , p 0 , x 0 , y 0 )
= ( q 0 , p 0 , q 0 , p 0 ) 2 n = ﬂoor [( t − t 0 {+) / d t ] 3 for i = 
 →+} n do ( q i , p i , x i , y i ) = φ d t/ 2◦ φ d t/ 2◦ φ d t 3 ◦ φ d t/ 2◦ φ d t/ 2◦ ( q i − 1 , p i − 1 , x i − 1 , y i − 1 ) ; 5 end
The recursive relations of ( q i , p i , x i , y i ) , i = 1 , 2 , · · · , n , can be expressed by the algorithm 1 (alsosee Figure 8 in Appendix A).
The input functions φ δ 1 ( q , p , x , y ) , φ δ 2 ( q , p , x , y ) , and φ δ 3 ( q , p , x , y )in algorithm 1 are  q
[Equation]

[Equation]
p ) /∂ p ] px y − δ [ ∂ H θ ( x , p ) /∂ q ]   , and 1  (cid:18)
[Equation]
x
[Equation]
(cid:19)
[Equation]
(cid:18)
[Equation]
p
[Equation]
y (cid:19) (cid:18)
[Equation]
(cid:19) − R δ (cid:18)
q
[Equation]
x p − y (cid:19) 
[Equation]
, (5) respectively.
Here
[Equation]
where I is a identity matrix
[Equation]
[-= ( q 0 , p 0 , q 0 , p 0 ) 2 n = ﬂoor [( t − t 0 ) / d t ] 3 for i = 
 → n do ( q i , p i , x i , y i ) = φ d t/ 2◦ φ d t/ 2◦ φ d t 3 ◦ φ d t/ 2◦ φ d t/ 2◦ ( q i − 1 , p i − 1 ,-]We remark that x and y are just auxiliary variables, which are theoretically equal to q and p .
Therefore, we can use the data set of ( q , p ) to construct the data set containing variables ( q , p , x , y ) .
In addition, by constructing the network H θ , we show that theorem B.1 in Appendix B holds, so thenetworks φ δ 1 , φ δ 2 , and φ δ 3 in (5) preserve the symplectic structure of the system.
Suppose that Φ 1and Φ 2 are two symplectomorphisms.
Then, it is easy to show that their composite map Φ 2 ◦ Φ 1 isalso symplectomorphism due to the chain rule.
Thus, the symplectomorphism of algorithm 1 can beguaranteed by the theorems B.1.
T RAINING SETTINGS AND ABLATION TESTS
We use 6 linear layers with hidden size 64 to model H θ , all of which are followed by a Sigmoidactivation function except the last one.
The derivatives ∂ H θ /∂ p , ∂ H θ /∂ q , ∂ H θ /∂ x , ∂ H θ /∂ y are all obtained by automatic differentiation in Pytorch (Paszke et al., 2019).
The weights of the linear layers are initialized by Xavier initializaiton (Glorot & Bengio, 2010).
We generate the dataset for training and validation using high-precision numerical solver (Tao, 2016),where the ratio of training and validation datasets is 9 : 1 .
We set the dataset ( q j 0 , p j 0 ) as the start inputand ( q j , p j ) as the target with j = 1 , 2 , · · · , N s , and the time span between ( q j 0 , p j 0 ) and ( q j , p j ) is
T train .
Feeding ( q 0 , p 0 ) = ( q j 0 , p j 0 ) , t 0 = 0 , t = T train , and time step d t in Algorithm 1 to get the predicted variables (ˆ q j , ˆ p j , ˆ x j , ˆ y j ) . Accordingly, the loss function is deﬁned as
[Equation]
where N b = 512 is the batch size of the training samples.
We use the Adam optimizer (Kingma & Ba, 2015) with learning rate 0.05.
The learning rate is multiplied by 0.8 for every 10 epoches.
Taking system H ( q, p ) = 0 . 5( q 2 + 1)( p 2 + 1) as an example, we carry out a series of ablation testsbased on our constructed networks.
Normally, we set the time span, time step and dateset size as T = 0 . 01 , d t = 0 . 01 and N s = 1280 .
The choice of ω in (4) is largely ﬂexible, since NSSNN is not sensitive to the parameter ω when it islarger than a certain threshold.
Figure 2 shows the training and validation losses with different ω inthe network trained by clean and noise datasets. Though the convergence rates are slightly differentin a small scope, the examples with various ω are able to converge to the same size of training andvalidation losses.
Here, we set ω = 2000 , but ω can be smaller than 2000 . The only requirement forpicking ω is that it has to be larger than O (10) , which is detailed in Appendix C. We pick L 1 loss function to train our network due to its better performance. Figure 3 comparesthe validation losses with different training loss functions in the network trained by clean and noisedatasets.
Figure 3(a) shows that either the network trained by L 1 or MSE with clean dataset canconverge to a small validation loss, but the network trained by L 1 loss converges relatively faster.
Figures 3(b) and 3(c) both show that the network trained by L 1 with noise dataset can converge to asmaller validation loss.
In addition, we already introduced a regularization term in the symplecticintegrator embedded in the network; thus, there is no need to add the regularization term in the lossfunction.
The integral time step in the sympletic integrator is a vital parameter, and the choice of d t largelydepends on the time span T train .
Figure 4 compares the validation losses generated by variousintegral time steps d t based on ﬁxed dataset time spans T train = 0 .
01 , 0 . 1 and 0 .
2 respectively in thetraining process.
The validation loss converges to a similar degree with various d t based on ﬁxed
T train = 0 .
01 and T train = 0 .
1 in 4(a) and (b), while it increases signiﬁcantly as d t increases basedon ﬁxed T train = 0 . 02 in 4(c). Thus, we should take relatively small d t for the dataset with largertime span T train  C OMPARISONS WITH OTHER METHODS
[Figure] 5. M ETHODOLOGIES [Figure] We compare our method with other recently proposed methods, such as HNN (Greydanus et al., 2019),
NeuralODE (Chen et al., 2018), TaylorNet (Tong et al., 2020), SSINN (DiPietro et al., 2020), SRNN(Chen et al., 2020), and SympNet (Jin et al., 2020). There are several features distinguishing ourmethod with others, as shown in Table 1. HNN ﬁrst enforces conservative features of a Hamiltoniansystem by reformulating its loss function, which incurs two main shortcomings. On the one hand,it requires the temporal derivatives of the momentum and the position of the systems to calculatethe loss function, which is difﬁcult to obtain from real-world systems.
On the other hand, HNNdoesn’t strictly preserve the symplectic structure, because its symplectomorphism is realized by itsloss function rather than its intrinsic network architecture. NeuralODE successfully bypasses thetime derivatives of the datasets by incorporating an integrator solver into the network architecture. Embedding the Hamiltonian prior into the NeuralODE, a series of methods are proposed, such as SRNN, SSINN, and TaylorNet, to predict the continuous trajectory of system variables; however,presently these methods are only designed to solve separable Hamiltonian systems.
Instead ofupdating the continuous dynamics by integrating the neural networks in NeuralODE, SympNet adoptsa symplectomorphism composed of well-designed both linear and non-linear matrices to intrinsicallymap the system variables within neighbouring time steps. However, the parameters scale in thematrix map for trainning N dimensional Hamiltonian system in SympNet is O ( N 2 ) , which makes ithard to generalize to the high dimensional N-body problems. For example, in Section 6, we predictthe dynamic evolution of 6000 vortex particles, which is challenging for the training process of the
SympNet on the level of O (6000 2 ) . NSSNN overcomes the weaknesses mentioned above. Under the framework of NeuralODE, NSSNNutilizes continuously-deﬁned dynamics in the neural networks, which gives it the capability to learn Methods
NSSNN HNN NeuralODETaylorNet SSINN SRNNSympNet
Solve nonseparable systems (cid:88) (cid:88) (cid:88) (cid:88)
Solve separable systems (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
Preserve
(cid:88) Partially (cid:88)
Partially (cid:88) (cid:88)
Utilize continuous dynamics (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
No need for derivatives in dataset (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
Long-term predictability (cid:88)
Partially (cid:88) (cid:88) (cid:88) (cid:88)
[Figure] (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
the continuous time evolution of dynamical systems.
Based on Tao (2016), NSSNN embeds thesymplectic proir into the nonseparable symplectic integrator to ensure the strict symplectomorphism,thereby guaranteeing the property of long-term predictability.
In addition, unlike SympNet, NSSNNis highly ﬂexible and can be generalized to high dimensional N-body problems by involving theinteraction networks (Sanchez-Gonzalez et al., 2019), which will be further discussed in Section 6.
We compare ﬁve implementations that learn and predict Hamiltonian systems.
The ﬁrst one is
NeuralODE, which trains the system by embedding the network f θ → ( d q / d t, d p / d t ) into the
Runge-Kutta (RK) intergrator.
The other four, however, achieve the goal by ﬁtting the Hamiltonian
H θ → H based on (1).
Speciﬁcally, HNN trains the network with the constraints of Hamiltoniansymplectic gradient along with the time derivative of system variables and then embeds the welltrained H [-( q 1 , q 2 , p 1 , p 2 ) = (cid:2) ( q 21 + p 21 ) 
 + ( q 22 + p 22 ) 2 (cid:3) / 4 − ( q 21 q 22 + p 21 p 22 −q 21 p 22 − p 21 q 22 + 4 q 1 q 2 p 1 p 2 ) .-]θ into the RK integrator for predicting the system.
The third and fourth implementations areablation tests.
One of them is improved HNN (IHNN), which embeds the well-trained H θ into thenonseparable symplectic integrator (Tao’s integrator) for predicting.
The other is to directly embed H θ into the RK integrator for training, which we call it HRK.
The ﬁfth method is NSSNN, whichembeds H θ into the nonseparable symplectic integrator for training.
For fair comparison, we adopt the same network structure (except that the dimension of output layerin NeuralODE is two times larger than that in the other four), the same L 1 loss function and samesize of the dataset, and the precision of all integral schemes is second order, and the other parameterskeep consistent with the one in Section 4.
The time derivative in the dataset for training HNN and IHNN is obtained by the ﬁrst difference method
[Equation]
Figure 5 demonstrates the differences between the ﬁve methods using a spring system H = 0 . 5( q 2 +p 2 ) with different time span T train = 0 .
4 , 1 and same time step d t = 0 .
We can see thatby introducing the nonseparable symplectic integrator into the prediction of Hamiltonian system, NSSNN has a stronger long-term predicting ability than all the other methods.
In addition, theprediction of HNN and IHNN lies in the dataset with time derivative; consequently, it will lead to alarger error when the given time span T train is large.
Moreover, the datasets obtained by (11) in HNN and IHNN are sensitive to noise.
Figure 6 comparesthe predictions of ( q, p ) for the system H = 0 .
5( q 2 + 1)( p 2 + 1) , where the network is trained bythe dataset with noise ∼ 0 .
05 U ( − 1 , 1) , along with time span T train = 0 .
2 and time step d t = 0 .
[-= (cid:107) q truth − q predict (cid:107) 
 + (cid:107) p truth − p predict (cid:107) 1 . It is clearly from-]Under the condition with noise, NSSNN still performs well compared with other methods. Also, wecompare the convergent error of a series of Hamiltonian systems with different H trained with noisydata in Appendix D, which generally shows better robustness than HNN does.
M ODELING VORTEX DYNAMICS OF MULTI - PARTICLE SYSTEM
For two-dimensional vortex particle systems, the dynamical equations of particle positions( x j , y j ) , j = 1 , 2 , · · · , N v with particle strengths Γ j can be written in the generalized Hamiltonian form as
[Equation]
[Equation]
By including the given particle strengths Γ j in Algorithm 1, we can still adopt the method mentionedabove to learn the Hamiltonian in (9) when there are less particles.
However, considering a systemwith N v (cid:29) 2 particles, the cost to collect training data from all N v particles might be high, and thetraining process can be time-consuming.
Thus, instead of collecting information from all N v particlesto train our model, we only use data collected from two bodies as training data to make prediction ofthe dynamics of N v particles. Speciﬁcally, we assume the interactive models between particle pairs with unit particle strengthsΓ
j = 1 are the same, and their corresponding Hamiltonian can be represented as network ˆ H θ ( x j , x k ) ,based on which the corresponding Hamiltonian of N v particles can be written as (Battaglia et al.
,2016; Sanchez-Gonzalez et al., 2019)
[Equation]
We embed (10) into the symplectic integrator that includes Γ j to obtain the ﬁnal network architecture.
The setup of the multi-particle problem is similar to the previous problems.
The training time spanis T train = 0 .
01 while the prediction period can be up to T predict =
We use 2048 clean datasamples to train our model.
The training process takes about 100 epochs for the loss to converge.
In Figure 7, we use our trained model to predict the dynamics of 6000-particle systems, including Taylorand Leapfrog vortices.
We generate results of Taylor vortex and Leapfrop vortex using NSSNN and HNN and compare them with the ground truth.
Vortex elements are used with corresponding initialvorticity conditions of Taylor vortex and Leapfrop vortex (Qu et al., 2019).
The difﬁculty of thenumerical modeling of these two systems lies in the separation of different dynamical vortices insteadof having them merging into a bigger structure.
In both cases, the vortices evolved using NSSNN areseparated nicely as the ground truth shows, while the vortices merge together using HNN.  L IMITATIONS
The network with the embeded integrator is often more time-consuming to train than the one basedon the dataset with time derivative.
For example, the ratio of training time of the methods HNN and NSSNN is 1 : 3 when d t = T train , and the training time of the recurrent networks further increaseswith the decreasing of d t . Although a smaller d t often has higher discretization accuracy, there is atradeoff between training cost and predicating accuracy. Additionally, a smaller d t may potentiallycause gradient explosion.
In this case, we may want to use adjoint method instead.
Another limitationlies on the assumption that the symplectic structure is conserved.
In real-world systems, there couldbe dissipation that makes this assumption unsatisﬁed.
C ONCLUSIONS
We incorporate a classic ideal that maps a nonseparable system to a higher dimensional space makingit quasi-separable to construct sympletic networks.
With the intrinsic symplectic structure, NSSNNpossesses many beneﬁts compared with other methods.
In particular, NSSNN is the ﬁrst methodthat can learn the vortex dynamical system, and accurately predict the evolution of complex vortexstructures, such as Taylor and Leapfrog vortices.
NSSNN, based on the ﬁrst principle learningcomplex systems, has potential applications in ﬁelds of physics, astronomy, and weather forecast, etc.
We will further explore the possibilities of neural networks with inherent structure preserving abilityin ﬁelds like 3D vortex dynamics and quantum turbulence.
In addition, we will also work on generalapplications of NSSNN with datasets based on images or other real scenes through automaticallyidentifying coordinate variables of Hamiltonian systems based on neural networks. A N ETWORK ARCHITECTURE Figure 8(a) shows the forward pass of NSSNN is composed of a forward pass through a differentiablesymplectic integrator as well as a backpropagation step through the model. Figure 8(b) plots theschematic diagram of NSSNN.
For the constructed network H θ ( q , p ) , we integrate (4) by usingthe second-order symplectic integrator (Tao, 2016). Speciﬁcally, The input layer of the integratoris ( q , p , x , y ) = ( q 0 , p 0 , q 0 , p 0 ) at t = t 0 and the output layer is ( q , p , x , y ) =
( q n , p n , x n , y n )at t = t 0 + n d t . The recursive relations of ( q i , p i , x i , y i ) , i = 1 , 2 , · · · , n , are expressed by thealgorithm 1. B S YMPLECTOMORPHISMS One of the most important features of the time evolution of Hamilton’s equations is that it is asymplectomorphism, representing a transformation of phase space that is volume-preserving.
In thesetting of canonical coordinates, symplectomorphism means the transformation of the phase ﬂow of a Hamiltonian system conserves the symplectic two-form
[Equation]
where ∧ denotes the wedge product of two differential forms.
The rules of wedge product can befound in Lee (2010).
In the two dimensional case, (11) can be understood as the area element ofsurface.
In this case, the symplectomorphism can be interpreted as the area element of the surface isconstant.
As proved below, our constructed network structure intrinsically preserves Hamiltonianstructure.
Theorem B.1.
For a given δ , the mapping φ δ 1 , φ δ 2 , and φ δ 3 in (5) are symplectomorphisms.
Proof.
Let
[Equation]
From the ﬁrst equation of (5), we have
[Equation]
Similarly, we can prove that d t q 2 ∧ d t p 2 + d t x 2 ∧ d t y 2 =
d q ∧ d p + d x ∧ d y .
In addition, from the thirdequation of (5), we can directly deduce that d t q 3 ∧ d t p 3 + d t x 3 ∧ d t y
=
d q ∧ d p + d x ∧ d y .
Suppose that Φ 1 and Φ 2 are two symplectomorphisms.
Then, it is easy to show that their compositemap Φ 2 ◦ Φ 1 is also symplectomorphism due to the chain rule.
Thus, the symplectomorphism ofalgorithm 1 can be guaranteed by the theorem B.1.
C D ETERMINING C OEFFICIENT ω
To further elucidation, the Hamiltonian H A + H B without the binding, i.e., H with ω = 0 , in extendedphase space ( q , p , x , y ) may not be integrable, even if H ( q , p ) is integrable in the original phasespace ( q , p ) .
However, H C is integrable.
Thus, as ω increases, a larger proportion in the phase spacefor H corresponds to regular behaviors (Kolmogorov, 1954).
For H ( q, p ) =
( q 2 + 1)( p 2 + 1) / 2 ,shown in Fig.
9, we compare the trajectories starting from [ q (0) , p (0) , x (0) , y (0)]
= ( − 3 , 0 , − 3 , 0)calculated by the symplectic integrator (Tao, 2016) with different ω , where the calculation accuracyis second order accuracy and the time interval is 0.001.
As Figs. 9(a), (b), (c), and (d) shown, thechaotic region in phase space is signiﬁcantly decreasing until forming a stable limit cycle.
We deﬁne(cid:15) =
(cid:107) ( q, p ) − ( x, y ) (cid:107) 2 as the calculation error of this system, shown in Figs.
(e), (f), (g), and (h) thatthe error is decreasing with ω increasing, which ﬁts the quantitative results of phase trajectory well. D O THER EXPERIMENTS
We consider the pendulum, the Lotka–Volterra, the Spring, the Hénon–Heiles, the Tao’s example(Tao, 2016), the Fourier form of nonlinear Schrödinger and the vortex particle systems in ourimplementation.
The Hamiltonian energies of these systems (except vortex particle system) aresummarized as follows:
Pendulum system: H ( q, p ) = 3(1 − cos( q ))
+ p 2 .
Lotka–Volterra system: H ( q, p ) =
p − e p +2 q − e q .
Spring system: H ( q, p ) =
q 2 + p 2 .
Hénon–Heiles system: H ( q 1 , q 2 , p 1 , p 2 ) =
( p 21 + p 22 ) / 2 + ( q 21 +q 22 ) +
( q 21 q 2 − q 32 / 3) / 2 .
Tao’s example (Tao, 2016): H ( q, p ) =
( q 2 + 1)( p 2 + 1) / 2 .
{+Fourier form ofnonlinear Schrödinger equation: H ( q 1 , q 2 , p 1 , p 2 ) = (cid:2) ( q 21 + p 21 ) 
+}
+ ( q 22 + p 22 ) 2 (cid:3) / 4 − ( q 21 q 22 + p 21 p 22 −q 21 p 22 − p 21 q 22 + 4 q 1 q 2 p 1 p 2 ) .
[Table]
The network is trained by the dataset with noise ∼ 0 .
1 U ( − 1 , 1) .
The training time span, integral timestep
[Table]
and validation time span are 0 .
[Table]
and 0 . 1 , respectively.
Table 2 compares the Hamiltoniandeviation (cid:15) H
=
(cid:107)H ( q truth , p truth ) − H ( q predict , p predict )
(cid:107) 2 / (cid:107)H ( q truth , p truth ) (cid:107) 2 and the prediction error(cid:15) p
{+= (cid:107) q truth − q predict (cid:107) 
+}
+ (cid:107) p truth − p predict (cid:107) 1 .
It is clearly from the Table 2 that NSSNN eitheroutperforms or has similar performances as NeuralODE and HNN do.