From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper uses a neural network based approach to discover the structure of dynamical systems in terms of first integrals. A discrete and a continuous time version is presented. The paper also presents a nice set of experiments and benchmark for this domain.

Strengths + Weakneeses: The paper presents a solid framework for learning the structure of dynamical systems or ODEs. Most reviewers found the approach very interesting and the experiments exhaustive and compelling. On the negative side, reviewer pYdf had issues regarding the clarity of the paper and also misunderstood some of the results wrong. These issues have been properly addressed, yet, the reviewer did not respond to the rebuttal, hence I weight his review less. Overall, the remaining reviewers were enthusiastic about the paper and I recommend acceptance.
justification_for_why_not_higher_score: Reviewers mention issues in terms of presentation of the results as well as the importance of correctly chosen hyper-parameters.
justification_for_why_not_lower_score: The points of the negative review (score 3) were addressed in the rebuttal. Unfortunately, the corresponding reviewer was not responding and did not update his scores. All other reviewer have a good opinion about the paper (average score of 8) and were much more proactive in the discussion, hence, I weight there opinion higher in my evaluation. 
summary_of_AC-reviewer_meeting: N/A
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper1165/Authors
title: Thank you!
comment: We are pleased that our response answered your questions. Your questions helped us to improve our manuscript. Thank you very much.

From: ICLR.cc/2023/Conference/Paper1165/Reviewer_RgJp
title: Reply to the authors' response
comment: I appreciate the authors’ response to my questions. 

The addtional experimental results and other revisions make me feel more comfortable accepting this work.

With this in mind, I will update my score.

From: ICLR.cc/2023/Conference/Paper1165/Authors
title: Thank you very much for your detailed review and deep understanding
comment: Thank you very much for your detailed review and deep understanding. We are glad that you found our paper well-organized and providing a relevant set of experiments.

> The authors could further clarify potential sources of error in FINDE. The current version explains how dFinde resolve the time discretization error in cFINDE, but it would be good to have a more information on other sources of error created by the approximation and they can potentially be resolved. E.g. In any finite difference scheme there is approximation error, such as truncation errors coming from the discretization of the equation.

Thank you for your suggestion.

For ODEs, there are three error sources (modeling error, temporal discretization error, and rounding error), and the latter two errors are also collectively referred to as numerical errors.
We have clarified this point in the 2nd paragraph on page 3 and corrected some ambiguous use of these terms in Appendices D.1 and D.3 (highlighted in red).

The proposed dFINDE eliminates the temporal discretization errors of first integrals but still suffers from the other errors, as discussed in Appendix D.1.

For PDEs, we created the dataset by spatially discretizing the equation and integrating it numerically.
Hence, the spatial truncation errors were already introduced into the dataset.
When the neural network learns this dataset, no spatial truncation errors are additionally introduced.
An evaluation using the analytical solution as a dataset or datasets created with different spatial resolutions is included in future work.
We have added the above explanation to Appendix C on page 20.

> The paper could be improved by having more detailed Figure captions across all figures, especially the one outlining the results (e.g. Figure 1, Figure 4 and Figure 5). More detailed captions would provide greater clarity on what the data show and how to interpret the results.

Thank you for your suggestion.
We have added explanations of panels in Figures 1--6 and A1 to their captions.

> You mention that FINDE preserves multiple quantities at the end of Section 2. Could you explain more how that occurs? Specifically how does preserving multiple quantities (e.g. energy and mass) differ from preserving just one quantity (energy) related to your method?

Thank you for your suggestion.

FINDE can assume one or more first integrals without changing anything.
This is a significant advantage.

Existing methods (e.g., HNN and CHNN) assume geometric structures (e.g., Hamiltonian structure) described in Appendix A in order to guarantee conservation laws.
When multiple structures are assumed at the same time, they must be integrated using appropriate prior knowledge.
If it is possible, it would achieve extremely high performance.
Otherwise, the geometric structures would conflict with each other and would not produce an appropriate model.
This is the reason why CHNN failed to learn the two-body problem and HNN+FINDE failed to learn the double pendulum in Appendix D.3.
In contrast, NODE with cFINDE or dFINDE does not assume any geometric structure but assumes first integrals in the most general way.

We have added this explanation to Appendix D.3.

> Given the additional advantages of dFINDE, in what situations would someone choose cFINDE over dFINDE?

Thank you for your suggestion.
cFINDE can be used with an explicit method and computationally cheaper than dFINDE, which is an implicit method.
Thus, cFINDE is more preferable when strict first integral conservation and high prediction accuracy are not required.
We have added an explanation about this trade-off at the end of page 5 and the 1st paragraph of Section 4.3.

From: ICLR.cc/2023/Conference/Paper1165/Authors
title: Thank you very much for your deep understanding and valuable suggestions.
comment: Thank you very much for your deep understanding and valuable suggestions.
We are pleased that you recognized technical soundness, clarity, and reproducibility.

> the authors do not clearly mention how one can determine such a critical hyper-parameter for unknown dynamical systems. It will be nice if the authors elarborate it.

Thank you for your important remark.
Because $K$ is a hyperparameter, it is basically a subject to be adjusted through evaluations on a validation set.

However, when $K$ is inappropriately large, the prediction performance on the *training set* also dropped significantly.
We added a new result for the double pendulum, where the performance dropped with $K=6$ for both training and test sets (see the table below).
This is because FINDE for $K=6$ assumes an inappropriately low-dimensional submanifold $\mathcal M'$; hence, it is incapable of learning the dynamics and performs poorly even on the training set.
This result suggests that the training set is enough to avoid a fatally inappropriate $K$.

Alternatively, other methods (e.g., [1], which you introduced) are complementarily available to determine $K$.
We have added the above explanation and related experimental results to Appendix D.4.

Results of Double Pendulum

|          |  $K$  | Train 1-step $\downarrow$ | Train VPT $\uparrow$ | Test 1-step $\downarrow$ | Test VPT $\uparrow$ |
| :------- | :---: | ------------------------: | -------------------: | -----------------------: | ------------------: |
| NODE     |   -   |                    $0.76$ |              $0.966$ |                   $0.82$ |             $0.110$ |
| + cFINDE |   1   |                    $0.72$ |              $0.974$ |                   $0.75$ |             $0.156$ |
|          |   2   |                    $0.69$ |              $0.981$ |                   $0.73$ |             $0.198$ |
|          |   3   |                    $0.63$ |              $0.994$ |                   $0.69$ |             $0.411$ |
|          |   4   |                    $0.67$ |              $0.990$ |                   $0.77$ |             $0.395$ |
|          |   5   |                    $0.65$ |              $0.998$ |                   $0.80$ |             $0.585$ |
|          |   6   |                    $9.93$ |              $0.126$ |                  $12.53$ |             $0.005$ |

> It will be nice if the authors cite [1] and compare pros and cons of the proposed method with [1] briefly.

Thank you for your introduction.
We have cited [1] at the bottom of Appendix A on page 15 and added the following discussion.

The method proposed in [1] specializes in finding first integrals, whereas FINDE learns dynamics and first integrals jointly.

In addition, [1] assumes that all data points share the same submanifold (permissible state manifold), and hence it can only process a single long time series with fixed first integrals.
In contrast, our proposed FINDE can leverage a dataset of multiple time series with different values of the first integrals.

Nonetheless, the method [1] can be used complementarily with our proposed FINDE, as discussed above.

From: ICLR.cc/2023/Conference/Paper1165/Authors
title: Thank you for your review, but we would like to confirm one thing first; this is about NODEs, but not PINNs.
comment: Thank you for taking time to review our manuscript.
We are pleased that you appreciate the importance of conserved quantities.

First of all, we would like to confirm that our manuscript investigated learning ODEs from observed data using neural nets (such as neural ordinary differential equations (NODEs) and Hamiltonian neural networks (HNNs)), but not solving PDEs given as symbolic equations using neural nets (such as physics-informed neural networks (PINNs)).
Even though both families aim to integrate physics and neural networks, their goals and experimental settings are quite distinct.
To avoid confusion, we have revised the very first paragraph to emphasize the difference and added a paragraph to introduce related works at the end of page 19.

Having clarified this point, we believe that many of your questions below may be resolved.

> a typo in equation A18 when quoting the KdV

Thank you for your point.
We have corrected (A18) by adding the missing $u$.

> Further KdV has infinite number of conserved quantities. The authors claim to have cut that down to some finite number by some discretization. This is entirely unexplained in the paper - and potentially a very fatal issue since that sounds like trying to entirely change the target from trying to solve KdV to something else!

Section 4.1, Table 2, and the paragraph at the top of page 20 explained that KdV equation was discretized in space into 50 elements and that it no longer had infinitely many first integrals.
Whereas PINNs attempted to solve PDEs without spacial discretization, NODEs and HNNs typically learn spatially discretized PDEs as ODEs.
Because our purpose is to extend NODEs and HNNs, our problem setting is appropriate.

> is V being trained? If yes, how? If I understand correctly -- the function f in Theorem 1 is also a neural net. Is that being trained? [...] If there is a loss function somewhere that is getting minimized then that is extremely unclear.

The base model $f$ and FINDE $V$ were neural networks and jointly trained by the Adam optimizer, as written in the last and second last paragraphs on page 6.
The loss functions were briefly described in the last paragraph on page 6 and discussed in detail in Appendix B.3, as referenced on page 7, line 2.
We did nothing special to make FINDE learn first integrals, but just made it learn dynamics under the constraints that there are first integrals.
It can be seen as a new finding that such an approach is enough to find first integrals.

> The testing of preservation of the conserved quantities looks very ambiguous to me. [...] it seems that at no value of K is the new method beating the benchmarks in this 1-step criteria for this 2-body example. Isn't that a strong argument against the new method?

As mentioned in the 2nd paragraph on page 7 (and in the cited references), the errors in state are misleading indicators, so only VPT was used to evaluate methods.
The 1-step error results were merely informative and did not serve as a criterion.
This is why its detailed definition was given in Appendix B.3.
To avoid confusion, we have removed 1-step errors from the discussions in Section 4, and added the above explanation at the end of the 1st paragraph of Section 4.3.

> Further, in Appendix D.2 the authors seem to clearly state that their approach of symbolic regression is not leading to a correct identification of the conserved quantities for their 2-pendulum and the FitzHugh-Nagumo datasets.

As discussed in Appendix D.2, any functions of only first integrals and arbitrary constants are again first integrals.
Hence, without constraints, it is in principle impossible to re-discover a first integral as a well-known symbolic expression, and a failure in symbolic regression is not a problem in any way.
For example, [1] uses "gauge fixing" as a constraint; a combination of such method may improve the results.
We have added the above explanation to the 3rd paragraph of Appendix D.2.

[1] Liu, Z., & Tegmark, M. (2021). Machine learning conservation laws from trajectories. Physical Review Letters, 126(18), 180604.

From: ICLR.cc/2023/Conference/Paper1165/Authors
title: Thank you very much for your detailed review and deep understanding
comment: Thank you very much for your detailed review and deep understanding. We are very pleased that you find our method to make a solid contribution and weaknesses "could make the paper better but should not overshadow the strength".

> The numerical results can be explained and presented better. For example, the switch between dFINDE and cFINDE could be explained better as they often make a bit confusion in the text; the plots can be improved with better presentation: for instance, Figure 3 - it can be shown as four subplots, each of which can compare HNN vs HNN+cFINDE; the plots could confuse audience a bit: for instance, it is hard to see the ground truth.

Thank you for your suggestion.

We replaced FINDE with cFINDE or dFINDE to avoid confusion (as denoted by blue text).
We have made the ground truth in Fig. 6 easier to see by using dotted lines.

In Fig. 3, we aimed to compare methods qualitatively rather than quantitatively.
In particular, when using cFINDE, the error in the center-of-gravity position is smaller than the error in the mass position, suggesting not only a higher accuracy in state prediction but also conservation of linear momentum.
To make it clearer, we have added an explanation in the last paragraph of Section 4.3.
We have never changed any contents or discussions.

> It is very interesting to see the big difference induced by using different values of $K$. As discovering the invariant quantities is one of the cores of this approach and also as author addressed in the conclusion, conducting more comprehensive and in-depth numeric investigation on the role of $K$, with more detailed discussion, along with more analysis in theory, would benefit a lot in helping audience understand and the field leverage the proposed methods.

Thank you for important suggestion.

The theoretical explanation for the high performance of neural networks (e.g., HNN) that assume first integrals for physical phenomena is an open question.
It has been theoretically shown that neural networks (e.g., CNNs and GNNs) with symmetry have faster learning convergence (as partly mentioned in the 2nd last paragraph on page 3), and we consider this approach can be applied to the above question.

At least for cFINDE and dFINDE, we have an intuitive but not rigorous explanation; assuming one more first integral (i.e., increasing $K$ by 1) reduces the number of degrees of freedom in the dynamics by 1, narrows the hypothesis space, accelerates learning convergence, and suppresses generalization errors.

This fact also implies that cFINDE and dFINDE with inappropriately large $K$ have insufficient degrees of freedom and cannot learn dynamics, resulting in poor performance even on the training set.
We added a new numerical result for the double pendulum, where the performance dropped with $K=6$ for both training and test sets (see the table below).

We have added the above explanations and results to Appendix D.4.

Results of Double Pendulum

|          |  $K$  | Train 1-step $\downarrow$ | Train VPT $\uparrow$ | Test 1-step $\downarrow$ | Test VPT $\uparrow$ |
| :------- | :---: | ------------------------: | -------------------: | -----------------------: | ------------------: |
| NODE     |   -   |                    $0.76$ |              $0.966$ |                   $0.82$ |             $0.110$ |
| + cFINDE |   1   |                    $0.72$ |              $0.974$ |                   $0.75$ |             $0.156$ |
|          |   2   |                    $0.69$ |              $0.981$ |                   $0.73$ |             $0.198$ |
|          |   3   |                    $0.63$ |              $0.994$ |                   $0.69$ |             $0.411$ |
|          |   4   |                    $0.67$ |              $0.990$ |                   $0.77$ |             $0.395$ |
|          |   5   |                    $0.65$ |              $0.998$ |                   $0.80$ |             $0.585$ |
|          |   6   |                    $9.93$ |              $0.126$ |                  $12.53$ |             $0.005$ |

From: ICLR.cc/2023/Conference/Paper1165/Reviewer_USE3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a neural network based approach to preserve and discover first integrals in the underlying target systems. In this paper, authors propose two instances of the approach, namely cFINDE and dFINDE, which works in continuous-time and discrete-time respectively. The proposed FINDE method is a great exploration in the area of preserving and/or discovering invariant quantities when learning dynamical systems with deep learning methods, and it is a generalization of existing deep learning based methods. As I see, the FINDE method makes a solid contribution.  
strength_and_weaknesses: The main strengths of this paper are 

$\mathbf{1}.$ on the proposal of enhancing the neural network based approach to learn dynamical systems by proposing an additional structure/equation to preserve and/or discover invariant quantities. 

$\mathbf{2}.$ This paper is very well organized, written and illustrated with graphic results. The background knowledge is adequate for readers (relevant in deep learning fields) to understand the rationale and the intuition driving the proposal, and the paper is very readable.  

There are a few weakness, which could make the paper better but should not overshadow the strength

$\mathbf{1}.$ The numerical results can be explained and presented better. For example, the switch between dFINDE and cFINDE could be explained better as they often make a bit confusion in the text; the plots can be improved with better presentation: for instance, Figure 3 - it can be shown as four subplots, each of which can compare HNN vs HNN+cFINDE; the plots could confuse audience a bit: for instance, it is hard to see the ground truth. 

$\mathbf{2}.$ It is very interesting to see the big difference induced by using different values of $K$. As discovering the invariant quantities is one of the cores of this approach and also as author addressed in the conclusion, conducting more comprehensive and in-depth numeric investigation on the role of $K$, with more detailed discussion, along with more analysis in theory, would benefit a lot in helping audience understand and the field leverage the proposed methods.    
clarity,_quality,_novelty_and_reproducibility: $\cdot$ This paper is very well clearly written and presented, with a great deal of details in introducing the background, literature review, intuitive explanation, theoretical analysis and numerical study. 

$\cdot$ The novelty of this paper is obvious as it proposed novel approach to generalize and enhance the existing literature to capture and discover first integral of the neural-network based dynamic system learning methods. The implementation is straightforward.

$\cdot$ This paper has detailed explanation of theoretical analysis, implementations, and experiments. And, it also listed most (if not all) of the used libraries that author used in the implementation.  
summary_of_the_review: As stated in the above comment sections, this paper has a solid contribution with a novel approach. It is a good paper.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: There is no ethics concerns.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1165/Reviewer_pYdf
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper asks a very pertinent question as to how we can incorporate knowledge of conserved quantities when solving PDEs via nets. The suggested method is quite unclear. 
strength_and_weaknesses: The main strength of the paper is that it raises and important question and that it has tried some serious benchmark tests. Also interestingly this paper tries to marry the ideas of symbolic regression with ideas of neural training to discover conserved quantities. Definitely a worthwhile attempt. 

Prima facie there seem to be major weaknesses in the paper.  

For a start V seems to be a neural net whose output coordinates encode the conserved quantities. But is V being trained? If yes, how? If I understand correctly -- the function f in Theorem 1 is also a neural net. Is that being trained? Its entirely unclear to me as to how these nets are getting determined. If there is a loss function somewhere that is getting minimized then that is extremely unclear. The paper is in dire need of a pseudocode! 

The testing of preservation of the conserved quantities looks very ambiguous to me. While the authors want to use this as one of the main tests it doesn't seem to have a clear definition till Appendix A (Equation A13?) - and even then it seems to be a function of time while the Table 3 seems to suggest that this is a number that characterizes the whole solution.  And if we focus on Table 3 then it seems that at no value of K is the new method beating the benchmarks in this 1-step criteria for this 2-body example. Isn't that a strong argument against the new method? 

Further, in Appendix D.2 the authors seem to clearly state that their approach of symbolic regression is not leading to a correct identification of the conserved quantities for their 2-pendulum and the FitzHugh-Nagumo datasets. So what exactly is the claim of the paper at all? Doesn't this beat the central claim in the paper? 

Using KdV as a test-case is a quite tricky. Firstly there seems to have be a typo in equation A18 when quoting the KdV. Further KdV has infinite number of conserved quantities. The authors claim to have cut that down to some finite number by some discretization. This is entirely unexplained in the paper - and potentially a very fatal issue since that sounds like trying to entirely change the target from trying to solve KdV to something else! 
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is extremely low. 
I am unable to figure out even basic things here like explained in the above point. 

  

summary_of_the_review: Given the weaknesses described above, I feel there are major gaps here - either very fundamental issues have gotten overlooked from the study or the presentation has failed to communicate them to the reader.  The paper needs a major rewrite in the very least.  
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1165/Reviewer_RgJp
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes first integral-preserving neural differential equation (FINDE) that can learn a dynamical system and its invariant quantities (i.e., first integrals) from observed data *without* prior knowledge (e.g., conservations of energy and momentum) or assumed geometric structures (e.g., Hamiltonian or Lagrangian) on the target system. FINDE consists of two neural networks; a neural network $\hat{f}(u) \in \mathbb{R}^N$ for modeling the target dynamics (i.e., neural ODE) and another neural network $V(u) \in \mathbb{R}^K$ that outputs $K$ first integrals. Then, FINDE projects the neural vector field $\hat{f}(u)$ to the directions in which $V(u)$ is preserved, by using the projection method. Thus, it guarantees the state $u$ evolves on $N-K$ dimensional sub-manifold (level set) which preserves first integrals. The optimization procedure of the projection method is well-formulated by using Lagrange multipliers, and it yields a novel type of neural ODE (continuous FINDE, cFINDE) that can learn conservative dynamics with repsect to the modeled frist integrals. The authors also present a discrete version of the proposed FINDE (dFINDE) to preserve the learned first integral under the discretization of numerical integrators. The authors theoretically show cFINDE and dFINDE exactly preserve the modeled first integrals $V(u)$ under the continuous and dicrete cases, repectively. The authors validate their approach with four dynamical systems including simple two-body problem and complicated KdV system.
strength_and_weaknesses: Strength:

1. The motivation of this paper is clear. Extracting underlying invariant quantities as well as learning dynamics *without* prior knoweldge on the target system is definitely important and challenging problem in physics. This paper provides a solid contribution to such an important problem. It might be one of good references for automated scientific discovery via machine learning tools.

2. The paper is technically sound. The used projection method seems to be a principled way to preserve arbitrary first integrals. The authors properly modifies the projection method optimization as a learnable model (FINDE) based on neural ODE framework. 

3. The paper is clearly written and easy-to-follow.

4. This work is highly reproducible. All used model architecture and hyper-parameters are well summarized in the main text of this paper. Experimental codes were made available.

Weakness:
1.  $K$, the output dimension of $V(u)$, is very important hyper-parameter because it directly determines the number of invariant quantities. Inappropriate setting of $K$ rather deteriorates the predictive performance of the model as depicted in the experimental results. In the current version of this paper, the authors do not clearly mention how one can determine such a critical hyper-parameter for unknown dynamical systems. It will be nice if the authors elarborate it.

2. I found a prior work [1] that seems to be very relevant to the proposed method. [1] uses PCA to find $N-K$ dimensional sub-manifold of trajectories and extracts $K$ invariant quantities. It will be nice if the authors cite [1] and compare pros and cons of the proposed method with [1] briefly.

***
[1] Liu, Z., & Tegmark, M. (2021). Machine learning conservation laws from trajectories. Physical Review Letters, 126(18), 180604.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well-written with sufficient details on the proposed algorithm.

Quality:

This paper tackles an important problem of physics and proposes a simple solution to resolve it. However, I have a concern regarding a hyper-parameter $K$ which might be very critical.

Novelty:

The use of the projection method for learning invariant quantities is straightforward (principled) yet novel to me. While the authors provide a thorough list of related literatures, however, [1] might be an important yet uncited work that very relevant to this work.

Reproducibility:

The authors provide a runnable code as well as a detaild description on the experimental setting.
summary_of_the_review: Overall, I believe the paper is very interesting, tackles an important problem, and thus is worthy of publication. However, the concern on $K$ tuning prevents me from recommending a clear accpetance of the paper in its current form. Currently, my evaluation is weak accept, however, I would like to increase my score if my concern is well addressed.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1165/Reviewer_vr6Y
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper introduces a new method to preserver first integrals in neural network modeling called first integral-preserving neural differential equation (FINDE) with the claim that FINDE can preserve and discover first integral in data without needing explicit information about the underlying structure. The authors propose two versions of FINDE: cFINDE (continuous integration) and dFINDE (discrete integration). In its introduction, the paper outlines the importance and challenges related to finding first integrals from data and which first integrals can be especially relevant for computer simulations (Energy, momentum, mass, etc). Next, the authors formally define first integrals and describe related work of first integrals in numerical analysis and neural networks that preserve first integrals, including Neural Ordinary Differential Equations, as well as works that have built on top of it and different kind of architectures (holonomic neural networks - HNN). The paper then introduces and defines the two versions of FINDE: cFINDE and dFINDE with relevant theorems explaining important preservation properties related to first integrals for both formulations.

In their experiments, the authors evaluate cFINDE and dFINDE on relevant datasets that contain different kinds of first integrals. In their implementation, the authors leverage an HNN,  apply related loss formulations to train cFINDE and dFINDE, and define valid prediction time (VPT) as an evaluation metric. The first set of experiments investigate whether first integrals are preserved in FINDE via a mass spring example. The second set of experiments show results on finding known integrals with FINDE compared to using vanilla HNN with the results indicating that FINDE methods generally perform better. The third set of experiments deal with finding first integrals in unknown systems using a set of examples (KdV, pendulum). The results generally indicate that FINDE methods are able to reproduce the ground truth compared to neural ODEs.
strength_and_weaknesses: **Strengths**

* The paper introduces a novel formulation for a relevant problem in applying neural network to physical systems and computer simulations thereof. The FINDE methods are described and derived with detail and their properties tested as part of the experiments.
* The paper provides a relevant set of experiments to assess the properties and performance of FINDE, both cFINDE and dFINDE and seems to achieves superior performance compared to other methods.

**Weaknesses**

* The paper could be improved by having more detailed Figure captions across all figures, especially the one outlining the results (e.g. Figure 1, Figure 4 and Figure 5). More detailed captions would provide greater clarity on what the data show and how to interpret the results.
* The authors could further clarify potential sources of error in FINDE. The current version explains how dFinde resolve the time discretization error in cFINDE, but it would be good to have a more information on other sources of error created by the approximation and they can potentially be resolved. E.g. In any finite difference scheme there is approximation error, such as truncation errors coming from the discretization of the equation.

**Additional Questions**

* You mention that FINDE preserves multiple quantities at the end of Section 2. Could you explain more how that occurs? Specifically how does preserving multiple quantities (e.g. energy and mass) differ from preserving just one quantity (energy) related to your method?
* Given the additional advantages of dFINDE, in what situations would someone choose cFINDE over dFINDE?

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is generally well-written with relevant details described. Figures could be improved with more detailed captions to guide the reader.

**Quality**

The quality of the writing and experiments is generally strong.

**Novelty**

The novelty is generally strong and tackles a relevant problem.

**Reproducibility**

The authors provide relevant detail and code for their experiments, as well as a reproducibility statement.
summary_of_the_review: The paper introduces a novel method and provides a relevant set of experiments to showcase the abilities of the method. Improvement can be made to further strengthen the paper in subsequent revisions.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper