Paper: Learning Dynamical Systems with Helmholtz-Hodge Decomposition and Gaussian Processes (ICLR-2024)

Reviewer comment (Theory):
As suggested by the authors, the decomposition is not unique due to the unidentifiability of the harmony component. The authors address this issue by imposing symmetry constraints; however, the uniqueness and characteristics of the resulting decomposition are not investigated.

Author response:
We thank the reviewer for giving us the opportunity to clarify the the uniqueness of the resulting decomposition. A unique decomposition makes our model identifiable. To show the identifiability of our model, an additional experiment on the damped pendulum system was perfomed to demonstrate the ***consistency*** of our model. Consistency is a key property of identifiable models [C1, C2]. If the model is identifiable, then the parameter estimates should converge to the true values as the amount of data increases. Therefore, we added experiments of evaluating the models performance with increasing number of training data (n=20, 100, 260, 420) in the Appendix A.8.3 of the updated draft (colored by blue). We invite the reviewer to review Fig.9(c) and Table 4, which shows that models without symmetry constraints (HHD-GP and D-HNN) consistently generate significant errors in predicting energy evolution, and the errors are almost unaffected by the number of training data, meaning that the model parameters cannot converge as the amount of data increases. In contrast, the energy errors of SPHHD-GP (Symmetry-preserving HHD-GP) exhibit a decreasing trend with increasing number of training data. The experimental results proves the consistency of our model, but since consistency is only a necessary condition for identifiability, we can only conclude that symmetry constraints improves the model identifiability, i.e. reducing the space of harmonic components. To prove that the existence of harmonic components are completely eliminated by the symmetry constraints, we added a theoretical verification that for the three dynamical systems in our experiments, the non-uniqueness of HHD has been solved through forced symmetries. We invite the reviewer to review this analysis in Appendix A.6 (colored by blue) of the updated draft.

We would like to further clarify that although we currently do not have a theoretical result to characterize the conditions for symmetry to effectively eliminate harmonic components in HHD, we have demonstrated the potential of symmetry in enhancing model identifiability, an area that has been relatively unexplored before. Current research mainly focuses on using symmetry to improve the predictive power of models. Therefore, we believe that studying the effect of symmetry on model identifiability is a very valuable future direction, which is important for model interpretability.

Reviewer comment (Novelty):
Another concern is the novelty of the approach compared to the multiple kernel learning literature. The proposed method appears to be an instance of learning a combination of two pre-defined kernels.

Author response:
We appreciate the reviewer for raising this concern. We would like to clarify the difference between our method and multiple kernel learning (MKL) in terms of their underlying principles and objectives. MKL focuses on learning the optimal combination of predefined kernels [C3], while our model assumes an additive structure in the data and model it by combining independent kernels in an additive manner. This additive structure in dynamical systems is guaranteed by the Helmholtz-Hodge decomposition (HHD), which states that a dynamical system can be decomposed into a sum of curl-free and div-free components. Therefore, we would like to emphasize that our main contribution lies in the contruction of divergence-free kernels and curl-free kernels, as well as their symmetry-preserving extensions,  which not only improve the prediction performance of the model but also address the identifiability problem caused by harmonic components in additive structures.

Reviewer comment (Theory):
What is the computational complexity of the method in terms of the sample size and input dimension?

Author response:
Thanks to the reviewer for raising this concern. Computational complexity is indeed one of the main drawbacks of GPs. Define $m$ as the sample size and $n$ as the number of the input dimension, then the time complexity of our method is dominated by the Cholesky decomposition to invert the $mn \times mn$ covariance matrix, which takes $\left ( m n \right )^3/6$ operations, and all other operations are at most quadratic in $m$ and $n$. So the time complexity of our model is $\mathcal{O} \left ( m^3 n^3 \right )$. We understand the reviewer's concern about computational complexity, but at the expense of this computational cost, our model achieves high data efficiency and inherent regularization. We thank the reviewer for your inspiration in leveraging low-rank structures in dynamical systems to approximate the covariance matrix, and we are very intrigued about extensions of our work in reducing the computational complexity in future work. And we now discussed this limitation of computational complexity in the section of Conlusion in the updated draft (colored by blue).

Reviewer comment (Novelty):
How does the method differ from multiple kernel learning with two given kernels?

Author response:
We appreciate the reviewer for raising this concern. We would like to clarify the difference between our method and multiple kernel learning (MKL) in terms of their underlying principles and objectives. MKL focuses on learning the optimal combination of predefined kernels [C3], while our model assumes an additive structure in the data and model it by combining independent kernels in an additive manner. This additive structure in dynamical systems is guaranteed by the Helmholtz-Hodge decomposition (HHD), which states that a dynamical system can be decomposed into a sum of curl-free and div-free components. Therefore, we would like to emphasize that our main contribution lies in the contruction of divergence-free kernels and curl-free kernels, as well as their symmetry-preserving extensions,  which not only improve the prediction performance of the model but also address the identifiability problem caused by harmonic components in additive structures.

Reviewer comment (Theory):
The non-uniqueness of HHD is well-known. The authors use it as in section 3.3 to introduce the enforcement of the symmetries of the GP model. However, it's unclear that the enforcement of the symmetries can completely eliminate the non-uniqueness of HHD. (My understanding is negative). Although the preservation of symmetries is usually a desirable property to model dynamical systems, either a theoretical guarantee or empirical validation that the non-uniqueness can be addressed or mitigated is highly desirable.

Author response:
We thank the reviewer for giving us the opportunity to present a theoretical analysis that for the three dynamical systems in our experiments the non-uniqueness of HHD is solved through forced symmetries. The non-uniqueness of HHD is caused by the presence of harmonic components (vector fields that are both curl-free and divergence-free). To make HHD unique, we propose to imposing symmetry constraints to the space of divergence-free vector fields and the space of curl-free vector fields respectively, with the corresponding symmetry groups defined as $G_{div}$ and $G_{curl}$. So the HHD will be unique if there are no harmonic vector fields respecting the union of $G_{div}$ and $G_{curl}$. 

A harmonic vector field on $\mathbb{R}^n$ can be represented by the gradient field of a harmonic function $h$, which is a scalar function satisfying $\nabla \cdot \nabla h = 0$ (Laplace's equation). For the three dynamical systems used in our experiments, it can be easily verified that the existence of $h$ can be eliminated through forced symmetries. The symmetry group union of a damped mass-spring system consists of a rotation group $G_{div} = SO \left ( 2 \right )$ and a translation group $G_{curl} = \left \\{ \left ( g, 0 \right ) \mid g \in \mathbb{R} \right \\}$. A harmonic vector field $\mathbf{f} = \nabla h$ that satisfies this translation symmetry implies that its harmonic function $h \left ( q, p \right )$ is independent of the variable $q$, so the harmonic vector field is given by $\mathbf{f} = \left (0, \partial_p h \right )$ and $\partial_p h$ should be a constant to satisfy the Laplace’s equation, but the harmonic vector field in the form of constant clearly contradicts rotation symmetry $G_{div}$. Therefore, there is no harmonic vector field that respects both the symmetry groups $G_{div}$ and $G_{curl}$. Similar conclusions can be drawn for the damped pendulum and the Chua circuit. The Laplace's equation and translation symmetry imply that harmonic vector fields can only exist in the form of constant vector fields. However, constant vector fields obviously contradict odd symmetry or mirror symmetry. We added this theoretical analysis to the Section A.6 in the appendix of the updated draft.

Although we currently do not have a theoretical result to characterize the conditions for symmetry to effectively eliminate harmonic components in HHD, we have demonstrated the potential of symmetry in enhancing model identifiability, an area that has been relatively unexplored before. Current research mainly focuses on using symmetry to improve the predictive power of models. Therefore, we believe that studying the effect of symmetry on model identifiability is a very valuable future direction, which is important for model interpretability.

Reviewer comment (Experiments):
Real data are never clean. The authors added noise into the training data, but the noise magnitude is too small (0.01, only 1% of the data range). I would like to see how well the method behaves when larger magnitudes of noise are added, for instance, with standard deviations of 0.01, 0.05, 0.10, and 0.20.

Author response:
Thank the reviewer for this valuable suggestions. We added the experiments of evaluating the models performance with increasing standard deviation (0.01, 0.05, 0.10 and 0.20) of Gaussian noise in training data. We find that our model (SPHHD-GP) performs best at every noise level relative to the baseline models, meaning that our model is more robust to data noise. We invite the reviewer to Fig.8 and Table 3 in the updated draft to check the experimental results. In addition, according to the suggestion of Reviewer e4ns, we added the experiments of investigating the effect of the number of training data on the model performance. The results are shown in Fig.9 and Table 4, where we can observe that our model also shows the best performance on any number of training data.

Reviewer comment (Reproducibility):
In Table 1, what are the standard deviation values collected from? My understanding is that random initial values are used in the GP model. Are the randomness of the initial values cause the derivations?

Author response:
In order to mitigate the influence of the distribution of training data and the initial parameters of the model on the experimental results, all experimental results are based on 10 independent experiments conducted by resampling the training set and the initial parameters of the model. Sorry for causing this confusion, we now highlight this setting in the caption of Table 1 (colored by blue).

Reviewer comment (Evaluation):
I am trying to understand the reported RMSE and VPT in Table 1. For each system, 20 pairs of data are generated for training; where are the testing data generated? In the temporal sense, do the testing data lie in the future of the training data, or are the training and testing data pairs intermingled with respect to time?

Author response:
Sorry for causing this confusion. In the original draft, the generation of test data in placed in the Appendix. To address this confusion, we now included the generation of test data in the first paragraph of Section 6 in the updated draft (colored by blue). For the RMSE, the test set (state-derivative pairs) was generated by sampling a grid with a resolution of 10 points along each dimension of the dynamical systems. And for the VPT, the test set consists of 50 trajectories simulated from randomly sampled initial states and each trajectory was generated by integrating an initial state forward in time at a frequency of 25 Hz for 15 seconds. There is no connection between the sampling of training data and test data.

Reviewer comment (Novelty):
There are several works in the literature which demonstrate how the Lagrangian, (port-)Hamiltonian, and neural ODE based NN frameworks can be used to model dissipative dynamical systems. Authors have not given any mention of such frameworks in the introduction, which provides a feeling that there is no prior work in this area. The reference comes much later when discussing the baselines. This should be clearly included in the introduction. Some relevant works are as follows.  
* Desai, S.A., Mattheakis, M., Sondak, D., Protopapas, P. and Roberts, S.J., 2021. Port-Hamiltonian neural networks for learning explicit time-dependent dynamical systems. Physical Review E, 104(3), p.034312.  
* Sosanya, A. and Greydanus, S., 2022. Dissipative hamiltonian neural networks: Learning dissipative and conservative dynamics separately. arXiv preprint arXiv:2201.10085.  
* Drgoňa, J., Tuor, A., Vasisht, S. and Vrabie, D., 2022. Dissipative deep neural dynamical systems. IEEE Open Journal of Control Systems, 1, pp.100-112.  
* Gruver, N., Finzi, M., Stanton, S. and Wilson, A.G., 2022. Deconstructing the inductive biases of hamiltonian neural networks. arXiv preprint arXiv:2202.04836.  
* Bhattoo, R., Ranu, S. and Krishnan, N.A., 2023. Learning the dynamics of particle-based systems with Lagrangian graph neural networks. Machine Learning: Science and Technology, 4(1), p.015003.

Author response:
We thank the reviewer for the suggestion and for providing these reference methods. We now include these works in the Introduction of the updated draft (colored by blue). We illustrate the essential differences between our approach and these methods. We emphasize that our focus is on designing an additive GP model whose each component satisfies certain differential invariants (either free of divergence or of curl), rather than constructing ML models according to some physically governing equations (e.g. Hamiltonian or Lagrangian equations). Therefore, our model has a wider scope of applicability. For example, in our experiments, the methods mentioned by the reviewer are difficult to apply to the Chua circuit system, which is a 3D chaotic system that is hard to describe using Hamiltonian or Lagrangian equations.

Reviewer comment (Evaluation):
The claim on the interpretability presented in the abstract is not substantiated later on in the empirical experiments or results. It is not clear how the framework is interpretable especially for a system with larger number of degrees of freedom.

Author response:
Thanks to the reviewer for recommending two such interesting works. Indeed, these deep symbolic regression methods are known to have high interpretability, because they can recover symbolic expressions of dynamical systems from learned models. Your comment prompt us to further clarify and demonstrate interpretability of our model. While our model cannot extract explicit analytical forms, it instead provides a unique and valuable interpretation by decomposing the underlying system into a divergence-free component and a curl-free component, which represent the distinct physical phenomena at play within the system. The current work emphasizes the connection between HHD and the generalized Hamiltonian formalism. As described in Appendix A, our model can learn a generalized Hamiltonian function for any dimensional system, as long as the correct form of HHD is learned. To demonstrate this interpretability, the experiments detailed in Section 6.2 shows that the div-free features learned by the model with symmetry constraints (SPHHD-GP) can be used to accurately predict the energy evolution of dynamical systems. In contrast, models without symmetry constraints (HHD-GP and D-HNN) cannot learn the correct HHD decomposition due to model identifiability issues, so the features they learn are not physically interpretable. To further show the interpretability of our model, we add experiments of generalizing the learned models to predict dynamics with unseen friction coefficients, which demonstrate the interpretability of learned curl-free features. We invite the reviewer to view the experimental results in the Appendix A.8.4 (colored by blue) of the updated draft. We state that the interpretability of our model is guaranteed by the identifiability of the model achieved by symmetry constraints. Therefore, in addition to these experimental results, we have now added theoretical verification (Appendix A.6 in the updated draft) for the three demonstration examples that the non-identifiability of our model (i.e. non-uniqueness of the HHD) is solved through forced symmetries.

Reviewer comment (Experiments):
The experiments performed are on very (very) simple systems such as a damped spring and damped pendulum. The community has moved forward from these experiments. Please see the experiments in the references mentioned against Point 1. Especially, the experiments on simple one-degree of freedom toy examples are not enough to show the applicability of the approach to any realistic problems. For this, some demonstration on larger systems with more degrees of freedom (~50-100) should be conducted. Note that realistic systems can have much higher degrees of freedom.

Author response:
Although there are now several works dedicated to modeling multi-object dynamical systems, two-dimensional systems including mass springs, pendulums, etc. are still among the main benchmark systems for studying and testing data-driven dynamic models, as in the references [1,2,3,4] provided by the reviewer in W1 and some more recent representative works [B1, B2, B3]. And in addition to the damped mass-spring system and the damped pendulum, our model has also been tested on a three-dimensional chaotic system, which is generally regarded as a system with complex behavior in existing works. However, we fully agree with the reviewer's concern about computational complexity of the proposed model, and this is indeed the most significant drawback of GP models. Despite the high computational complexity, GP models have several distinctive advantages, such as high data efficiency and inherent regularization, which have been verified in our experiments. Therefore, we believe that the proposed model has potential applications for many realistic systems, especially those where only sparse noise observations are available. We now add a discussion of computational complexity limitations in the section of Conclusion (Section.7 in the updated draft). To reduce the computational complexity, combining the proposed model with methods such as sparse variational inference is a fruitful future direction of inquiry.

Reviewer comment (Experiments):
Baselines are not appropriately chosen. References provided in Point 1 should be used for baselines. Some of the baselines that can be included are graph neural ODE, Lagrangian and Hamiltonian NN with dissipative terms, and Lagrangian and Hamiltonian graph NNs with the dissipative terms, to name a few.

Author response:
Thanks again to the reviewer for providing these reference methods. In our response to W1, we described the essential difference between our model and these approaches. Moreover, considering the distinctive characteristics of GPs, we have chosen more GP-based models as baseline models. However, we completely agree with the reviewer that physical equations-governed NNs should be incorporated into the comparisons. Therefore, the submitted draft included D-HNN (Dissipative Hamiltonian Neural Network, [2] in W1) as one of the baseline models. As expected, it tends to overfit noisy data compared to nonparametric models. At the same time, the lack of consideration of model identifiability prevents the D-HNN results from providing a physically plausible explanation. We believe that this observation is sufficient to emphasize the advantages of our model compared to physical equations-governed NNs.

Reviewer comment (Evaluation):
The evaluation metrics are important and should preferably be included in the main manuscript and not the appendix. Moreover, why are other metrics such as energy error, momentum error etc. not included? These metrics allow meaningful interpretation of the error in the learned dynamics.

Author response:
We thank the reviewer for your valuable comments. We now include the definitions of the evaluation metrics in the Section 6 of the updated main manuscript (colored by blue). As a way to assess the interpretability of the learning model, Fig.2 gives a visual comparison of the energy predictions, but due to the limited space in the main manuscript, the numerical comparison of the energy predictions is shown in Table 2 in Appendix A.8.1. We apologize for the confusion caused by placing them in the appendix. We appreciate the reviewer's suggestion to include momentum error, but current work compares the predicted state derivatives to the true derivatives (RMSE in Table 1), which can reflect the momentum error. Furthermore, following the suggestion of Reviewer e4ns, we added an additional evaluation metric, the mean negative log likelihood (MNLL), which can evaluate the predicted uncertainty provided by the GP models. We invite the reviewer to browse the additional experimental results (colored by blue) with these metrics in Appendix A.8.3 of the updated draft.

Reviewer comment (Theory):
It is not clear what the authors mean by the claim that the framework is interpretable. Do they mean that the dissipative and non-dissipative terms are learned separately? This is not necessarily interpretable. There are interesting works on interpretability. For instance, see:  
* Cranmer, M., Sanchez Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D. and Ho, S., 2020. Discovering symbolic models from deep learning with inductive biases. Advances in Neural Information Processing Systems, 33, pp.17429-17442.  
* Cranmer, M.D., Xu, R., Battaglia, P. and Ho, S., 2019. Learning symbolic physics with graph networks. arXiv preprint arXiv:1909.05862.

Author response:
Thanks to the reviewer for recommending two such interesting works. Indeed, these deep symbolic regression methods are known to have high interpretability, because they can recover symbolic expressions of dynamical systems from learned models. Your comment prompt us to further clarify and demonstrate interpretability of our model. While our model cannot extract explicit analytical forms, it instead provides a unique and valuable interpretation by decomposing the underlying system into a divergence-free component and a curl-free component, which represent the distinct physical phenomena at play within the system. The current work emphasizes the connection between HHD and the generalized Hamiltonian formalism. As described in Appendix A, our model can learn a generalized Hamiltonian function for any dimensional system, as long as the correct form of HHD is learned. To demonstrate this interpretability, the experiments detailed in Section 6.2 shows that the div-free features learned by the model with symmetry constraints (SPHHD-GP) can be used to accurately predict the energy evolution of dynamical systems. In contrast, models without symmetry constraints (HHD-GP and D-HNN) cannot learn the correct HHD decomposition due to model identifiability issues, so the features they learn are not physically interpretable. To further show the interpretability of our model, we add experiments of generalizing the learned models to predict dynamics with unseen friction coefficients, which demonstrate the interpretability of learned curl-free features. We invite the reviewer to view the experimental results in the Appendix A.8.4 (colored by blue) of the updated draft. We state that the interpretability of our model is guaranteed by the identifiability of the model achieved by symmetry constraints. Therefore, in addition to these experimental results, we have now added theoretical verification (Appendix A.6 in the updated draft) for the three demonstration examples that the non-identifiability of our model (i.e. non-uniqueness of the HHD) is solved through forced symmetries.

Reviewer comment (Experiments):
How does the system perform on more complex systems such as 50-mass spring systems or 5-pendulum systems? GPs are known to have issues with larger input features. Can the present approach be extended to such realistic systems?

Author response:
Although there are now several works dedicated to modeling multi-object dynamical systems, two-dimensional systems including mass springs, pendulums, etc. are still among the main benchmark systems for studying and testing data-driven dynamic models, as in the references [1,2,3,4] provided by the reviewer in W1 and some more recent representative works [B1, B2, B3]. And in addition to the damped mass-spring system and the damped pendulum, our model has also been tested on a three-dimensional chaotic system, which is generally regarded as a system with complex behavior in existing works. However, we fully agree with the reviewer's concern about computational complexity of the proposed model, and this is indeed the most significant drawback of GP models. Despite the high computational complexity, GP models have several distinctive advantages, such as high data efficiency and inherent regularization, which have been verified in our experiments. Therefore, we believe that the proposed model has potential applications for many realistic systems, especially those where only sparse noise observations are available. We now add a discussion of computational complexity limitations in the section of Conclusion (Section.7 in the updated draft). To reduce the computational complexity, combining the proposed model with methods such as sparse variational inference is a fruitful future direction of inquiry.